Multi-rail ofa fabric not available (Intel MPI Library Version 5.0 Update 3)

Hello,

there is a problem with the latest MPI library and multi-rail support over OFA:

Version 5.0 Update 2 runs just fine:

$ mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 2 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 2  Build 20141030 (build id: 10994)
[0] MPI startup(): Copyright (C) 2003-2014 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[0] MPI startup(): Open 0 IB device: mlx5_0
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 2 ports per adapter
[1] MPI startup(): Start 2 ports per adapter
[0] MPID_nem_ofacm_init(): Init
[1] MPID_nem_ofacm_init(): Init
[0] MPI startup(): shm and ofa data transfer modes
[1] MPI startup(): shm and ofa data transfer modes
[0] MPI startup(): Device_reset_idx=3
[0] MPI startup(): Allgather: 3: 0-1 & 0-4
[0] MPI startup(): Allgather: 1: 1-1 & 0-4
[0] MPI startup(): Allgather: 4: 2-4 & 0-4
[0] MPI startup(): Allgather: 1: 5-92 & 0-4
[0] MPI startup(): Allgather: 4: 93-161 & 0-4
[0] MPI startup(): Allgather: 1: 162-33532 & 0-4
[0] MPI startup(): Allgather: 4: 33533-102618 & 0-4
[0] MPI startup(): Allgather: 1: 102619-546855 & 0-4
[0] MPI startup(): Allgather: 4: 0-2147483647 & 0-4
[0] MPI startup(): Allgather: 4: 1-839 & 5-8
[0] MPI startup(): Allgather: 2: 840-4170 & 5-8
[0] MPI startup(): Allgather: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgather: 1: 1-5 & 9-16
[0] MPI startup(): Allgather: 4: 6-8 & 9-16
[0] MPI startup(): Allgather: 1: 9-440 & 9-16
[0] MPI startup(): Allgather: 2: 441-2083 & 9-16
[0] MPI startup(): Allgather: 4: 2084-9050 & 9-16
[0] MPI startup(): Allgather: 1: 9050-65530 & 9-16
[0] MPI startup(): Allgather: 4: 65530-218986 & 9-16
[0] MPI startup(): Allgather: 1: 0-2147483647 & 9-16
[0] MPI startup(): Allgather: 1: 1-247 & 17-32
[0] MPI startup(): Allgather: 2: 248-1043 & 17-32
[0] MPI startup(): Allgather: 4: 1044-3132 & 17-32
[0] MPI startup(): Allgather: 1: 3133-3010585 & 17-32
[0] MPI startup(): Allgather: 3: 0-2147483647 & 17-32
[0] MPI startup(): Allgather: 1: 1-176 & 33-2147483647
[0] MPI startup(): Allgather: 4: 177-1818 & 33-2147483647
[0] MPI startup(): Allgather: 1: 1819-2278 & 33-2147483647
[0] MPI startup(): Allgather: 4: 2279-4129 & 33-2147483647
[0] MPI startup(): Allgather: 1: 4130-1555968 & 33-2147483647
[0] MPI startup(): Allgather: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-261835 & 0-2
[0] MPI startup(): Allgatherv: 4: 261835-611669 & 0-2
[0] MPI startup(): Allgatherv: 3: 611669-3196878 & 0-2
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 0-2
[0] MPI startup(): Allgatherv: 1: 0-370 & 3-4
[0] MPI startup(): Allgatherv: 2: 370-838 & 3-4
[0] MPI startup(): Allgatherv: 1: 838-3403 & 3-4
[0] MPI startup(): Allgatherv: 4: 3403-4808 & 3-4
[0] MPI startup(): Allgatherv: 1: 4808-8294 & 3-4
[0] MPI startup(): Allgatherv: 3: 8294-130946 & 3-4
[0] MPI startup(): Allgatherv: 4: 130946-208466 & 3-4
[0] MPI startup(): Allgatherv: 3: 208466-393216 & 3-4
[0] MPI startup(): Allgatherv: 4: 393216-617063 & 3-4
[0] MPI startup(): Allgatherv: 3: 617063-1398101 & 3-4
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 3-4
[0] MPI startup(): Allgatherv: 1: 0-25 & 5-8
[0] MPI startup(): Allgatherv: 4: 25-46 & 5-8
[0] MPI startup(): Allgatherv: 1: 46-1403 & 5-8
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 5-8
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 5-8
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 5-8
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-3 & 9-16
[0] MPI startup(): Allgatherv: 1: 3-23 & 9-16
[0] MPI startup(): Allgatherv: 4: 23-106 & 9-16
[0] MPI startup(): Allgatherv: 1: 106-797 & 9-16
[0] MPI startup(): Allgatherv: 2: 797-3141 & 9-16
[0] MPI startup(): Allgatherv: 3: 3141-5874 & 9-16
[0] MPI startup(): Allgatherv: 2: 5874-9274 & 9-16
[0] MPI startup(): Allgatherv: 1: 9274-17847 & 9-16
[0] MPI startup(): Allgatherv: 3: 17847-84012 & 9-16
[0] MPI startup(): Allgatherv: 4: 84012-225929 & 9-16
[0] MPI startup(): Allgatherv: 3: 225929-450203 & 9-16
[0] MPI startup(): Allgatherv: 4: 450203-545471 & 9-16
[0] MPI startup(): Allgatherv: 3: 545471-1767416 & 9-16
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allgatherv: 1: 0-25 & 17-32
[0] MPI startup(): Allgatherv: 4: 25-46 & 17-32
[0] MPI startup(): Allgatherv: 1: 46-1403 & 17-32
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 17-32
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 17-32
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 17-32
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-1 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 1-3 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 3-6 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 6-96 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 96-535 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 535-1255 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1255-3148 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 3148-5455 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 5455-17721 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 17721-41571 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 41571-110427 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 110427-596874 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 596874-1273594 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1273594-2420103 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allreduce: 7: 0-20 & 0-2
[0] MPI startup(): Allreduce: 1: 20-905 & 0-2
[0] MPI startup(): Allreduce: 4: 905-2058 & 0-2
[0] MPI startup(): Allreduce: 1: 2058-17240 & 0-2
[0] MPI startup(): Allreduce: 8: 17240-36706 & 0-2
[0] MPI startup(): Allreduce: 3: 36706-777395 & 0-2
[0] MPI startup(): Allreduce: 8: 777395-1247501 & 0-2
[0] MPI startup(): Allreduce: 2: 1247501-3149905 & 0-2
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 0-2
[0] MPI startup(): Allreduce: 1: 0-597 & 3-4
[0] MPI startup(): Allreduce: 7: 597-1755 & 3-4
[0] MPI startup(): Allreduce: 1: 1755-16932 & 3-4
[0] MPI startup(): Allreduce: 6: 16932-34438 & 3-4
[0] MPI startup(): Allreduce: 8: 34438-81725 & 3-4
[0] MPI startup(): Allreduce: 3: 81725-396925 & 3-4
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 3-4
[0] MPI startup(): Allreduce: 1: 0-55 & 5-8
[0] MPI startup(): Allreduce: 6: 55-74 & 5-8
[0] MPI startup(): Allreduce: 4: 74-213 & 5-8
[0] MPI startup(): Allreduce: 1: 213-1765 & 5-8
[0] MPI startup(): Allreduce: 4: 1765-2065 & 5-8
[0] MPI startup(): Allreduce: 1: 2065-16561 & 5-8
[0] MPI startup(): Allreduce: 2: 16561-39707 & 5-8
[0] MPI startup(): Allreduce: 8: 39707-168698 & 5-8
[0] MPI startup(): Allreduce: 7: 168698-570628 & 5-8
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 5-8
[0] MPI startup(): Allreduce: 6: 0-5 & 9-16
[0] MPI startup(): Allreduce: 1: 5-935 & 9-16
[0] MPI startup(): Allreduce: 4: 935-1105 & 9-16
[0] MPI startup(): Allreduce: 1: 1105-17349 & 9-16
[0] MPI startup(): Allreduce: 4: 17349-53440 & 9-16
[0] MPI startup(): Allreduce: 8: 53440-315615 & 9-16
[0] MPI startup(): Allreduce: 7: 315615-1140547 & 9-16
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allreduce: 6: 0-74 & 17-32
[0] MPI startup(): Allreduce: 4: 74-199 & 17-32
[0] MPI startup(): Allreduce: 6: 199-392 & 17-32
[0] MPI startup(): Allreduce: 4: 392-750 & 17-32
[0] MPI startup(): Allreduce: 6: 750-2087 & 17-32
[0] MPI startup(): Allreduce: 1: 2087-14841 & 17-32
[0] MPI startup(): Allreduce: 2: 14841-20338 & 17-32
[0] MPI startup(): Allreduce: 6: 20338-124787 & 17-32
[1] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): Allreduce: 8: 124787-646663 & 17-32
[0] MPI startup(): Allreduce: 7: 646663-2129006 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 0-245 & 0-2
[0] MPI startup(): Alltoall: 2: 245-276 & 0-2
[0] MPI startup(): Alltoall: 3: 0-2147483647 & 0-2
[0] MPI startup(): Alltoall: 2: 0-3878 & 3-4
[0] MPI startup(): Alltoall: 3: 3879-6492 & 3-4
[0] MPI startup(): Alltoall: 4: 6493-9680 & 3-4
[0] MPI startup(): Alltoall: 3: 9681-24681 & 3-4
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 3-4
[0] MPI startup(): Alltoall: 2: 0-3082 & 5-8
[0] MPI startup(): Alltoall: 3: 3083-20124 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2235 & 9-16
[0] MPI startup(): Alltoall: 3: 2236-26946 & 9-16
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 9-16
[0] MPI startup(): Alltoall: 1: 0-329 & 17-32
[0] MPI startup(): Alltoall: 2: 330-4661 & 17-32
[0] MPI startup(): Alltoall: 3: 4662-282206 & 17-32
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 17-32
[0] MPI startup(): Alltoall: 1: 0-217 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 218-409 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 410-4415 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 4416-32356 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 32357-36297 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 36298-65861 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 65862-866313 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoallv: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Alltoallw: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Barrier: 6: 0-2147483647 & 0-2147483647
[0] MPI startup(): Bcast: 2: 0-1 & 0-2
[0] MPI startup(): Bcast: 7: 1-402 & 0-2
[0] MPI startup(): Bcast: 5: 402-682 & 0-2
[0] MPI startup(): Bcast: 4: 682-1433 & 0-2
[0] MPI startup(): Bcast: 7: 1433-5734 & 0-2
[0] MPI startup(): Bcast: 1: 5734-21845 & 0-2
[0] MPI startup(): Bcast: 6: 21845-95963 & 0-2
[0] MPI startup(): Bcast: 5: 95963-409897 & 0-2
[0] MPI startup(): Bcast: 1: 0-2147483647 & 0-2
[0] MPI startup(): Bcast: 7: 0-2147483647 & 3-4
[0] MPI startup(): Bcast: 7: 1-1 & 5-8
[0] MPI startup(): Bcast: 1: 2-254 & 5-8
[0] MPI startup(): Bcast: 5: 255-261 & 5-8
[0] MPI startup(): Bcast: 1: 262-752 & 5-8
[0] MPI startup(): Bcast: 6: 753-1254 & 5-8
[0] MPI startup(): Bcast: 1: 1255-4032 & 5-8
[0] MPI startup(): Bcast: 4: 4033-19764 & 5-8
[0] MPI startup(): Bcast: 1: 19765-121346 & 5-8
[0] MPI startup(): Bcast: 6: 121347-201114 & 5-8
[0] MPI startup(): Bcast: 1: 201115-267790 & 5-8
[0] MPI startup(): Bcast: 2: 267791-588703 & 5-8
[0] MPI startup(): Bcast: 3: 588704-1341590 & 5-8
[0] MPI startup(): Bcast: 5: 1341591-2139116 & 5-8
[0] MPI startup(): Bcast: 6: 0-2147483647 & 5-8
[0] MPI startup(): Bcast: 7: 1-84 & 9-16
[0] MPI startup(): Bcast: 1: 85-137 & 9-16
[0] MPI startup(): Bcast: 6: 138-475 & 9-16
[0] MPI startup(): Bcast: 1: 476-1496 & 9-16
[0] MPI startup(): Bcast: 6: 1497-10273 & 9-16
[0] MPI startup(): Bcast: 1: 10274-106199 & 9-16
[0] MPI startup(): Bcast: 6: 106200-323972 & 9-16
[0] MPI startup(): Bcast: 2: 0-2147483647 & 9-16
[0] MPI startup(): Bcast: 7: 1-103 & 17-32
[0] MPI startup(): Bcast: 5: 104-180 & 17-32
[0] MPI startup(): Bcast: 7: 181-590 & 17-32
[0] MPI startup(): Bcast: 5: 591-1170 & 17-32
[0] MPI startup(): Bcast: 6: 1171-3459 & 17-32
[0] MPI startup(): Bcast: 1: 3460-52532 & 17-32
[0] MPI startup(): Bcast: 2: 52533-239896 & 17-32
[0] MPI startup(): Bcast: 5: 239897-262474 & 17-32
[0] MPI startup(): Bcast: 2: 0-2147483647 & 17-32
[0] MPI startup(): Bcast: 7: 1-935 & 33-2147483647
[0] MPI startup(): Bcast: 6: 936-1185 & 33-2147483647
[0] MPI startup(): Bcast: 1: 1186-4873 & 33-2147483647
[0] MPI startup(): Bcast: 6: 4874-9664 & 33-2147483647
[0] MPI startup(): Bcast: 1: 9665-85765 & 33-2147483647
[0] MPI startup(): Bcast: 2: 85766-250963 & 33-2147483647
[0] MPI startup(): Bcast: 5: 250964-262409 & 33-2147483647
[0] MPI startup(): Bcast: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Exscan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Gather: 3: 1-8 & 0-4
[0] MPI startup(): Gather: 1: 9-29 & 0-4
[0] MPI startup(): Gather: 3: 30-25688 & 0-4
[0] MPI startup(): Gather: 2: 25689-56274 & 0-4
[0] MPI startup(): Gather: 3: 56275-68784 & 0-4
[0] MPI startup(): Gather: 2: 68785-1304326 & 0-4
[0] MPI startup(): Gather: 3: 1304327-2684607 & 0-4
[0] MPI startup(): Gather: 2: 0-2147483647 & 0-4
[0] MPI startup(): Gather: 3: 1-10278 & 5-8
[0] MPI startup(): Gather: 2: 10279-22432 & 5-8
[0] MPI startup(): Gather: 3: 22433-51123 & 5-8
[0] MPI startup(): Gather: 2: 51124-93434 & 5-8
[0] MPI startup(): Gather: 3: 93435-667907 & 5-8
[0] MPI startup(): Gather: 2: 667908-1262190 & 5-8
[0] MPI startup(): Gather: 3: 1262191-2879271 & 5-8
[0] MPI startup(): Gather: 2: 0-2147483647 & 5-8
[0] MPI startup(): Gather: 3: 1-4004 & 9-16
[0] MPI startup(): Gather: 2: 4005-4168 & 9-16
[0] MPI startup(): Gather: 3: 4169-20254 & 9-16
[0] MPI startup(): Gather: 2: 20255-41713 & 9-16
[0] MPI startup(): Gather: 3: 41714-793099 & 9-16
[0] MPI startup(): Gather: 2: 793100-1272717 & 9-16
[0] MPI startup(): Gather: 3: 0-2147483647 & 9-16
[0] MPI startup(): Gather: 3: 1-2047 & 17-32
[0] MPI startup(): Gather: 2: 2048-16131 & 17-32
[0] MPI startup(): Gather: 3: 16132-25688 & 17-32
[0] MPI startup(): Gather: 2: 25689-56274 & 17-32
[0] MPI startup(): Gather: 3: 56275-68784 & 17-32
[0] MPI startup(): Gather: 2: 68785-1304326 & 17-32
[0] MPI startup(): Gather: 3: 1304327-2684607 & 17-32
[0] MPI startup(): Gather: 2: 0-2147483647 & 17-32
[0] MPI startup(): Gather: 3: 1-51845 & 33-2147483647
[0] MPI startup(): Gather: 2: 51846-76744 & 33-2147483647
[0] MPI startup(): Gather: 3: 76745-721936 & 33-2147483647
[0] MPI startup(): Gather: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Gatherv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-5 & 0-2
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 0-2
[0] MPI startup(): Reduce_scatter: 1: 0-29 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 29-48 & 3-8
[0] MPI startup(): Reduce_scatter: 1: 48-2805 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 2805-5556 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 5556-11732 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 11732-35932 & 3-8
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 3-8
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 9-16
[0] MPI startup(): Reduce_scatter: 1: 4-2807 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 2807-6883 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 6883-39653 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 39653-513184 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 513184-966175 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 17-32
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 4-14959 & 17-32
[0] MPI startup(): Reduce_scatter: 5: 14959-19362 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 19362-49970 & 17-32
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 17-32
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 1: 4-79521 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 79521-520875 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 5: 520875-1101846 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Reduce: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scatter: 3: 1-8 & 0-4
[0] MPI startup(): Scatter: 1: 9-31 & 0-4
[0] MPI startup(): Scatter: 3: 32-4867 & 0-4
[0] MPI startup(): Scatter: 2: 4868-18877 & 0-4
[0] MPI startup(): Scatter: 3: 18878-854395 & 0-4
[0] MPI startup(): Scatter: 2: 854396-3121304 & 0-4
[0] MPI startup(): Scatter: 3: 0-2147483647 & 0-4
[0] MPI startup(): Scatter: 3: 1-1 & 5-16
[0] MPI startup(): Scatter: 2: 2-3 & 5-16
[0] MPI startup(): Scatter: 3: 4-8 & 5-16
[0] MPI startup(): Scatter: 1: 9-31 & 5-16
[0] MPI startup(): Scatter: 3: 32-1902 & 5-16
[0] MPI startup(): Scatter: 2: 1903-4895 & 5-16
[0] MPI startup(): Scatter: 3: 4896-27566 & 5-16
[0] MPI startup(): Scatter: 2: 27567-39885 & 5-16
[0] MPI startup(): Scatter: 3: 39886-126420 & 5-16
[0] MPI startup(): Scatter: 2: 126421-1096865 & 5-16
[0] MPI startup(): Scatter: 3: 0-2147483647 & 5-16
[0] MPI startup(): Scatter: 3: 1-71026 & 17-32
[0] MPI startup(): Scatter: 2: 71027-339781 & 17-32
[0] MPI startup(): Scatter: 3: 339782-1290592 & 17-32
[0] MPI startup(): Scatter: 2: 1290593-3878829 & 17-32
[0] MPI startup(): Scatter: 3: 0-2147483647 & 17-32
[0] MPI startup(): Scatter: 3: 1-1 & 33-2147483647
[0] MPI startup(): Scatter: 2: 2-3 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4-8 & 33-2147483647
[0] MPI startup(): Scatter: 1: 9-31 & 33-2147483647
[0] MPI startup(): Scatter: 3: 32-1902 & 33-2147483647
[0] MPI startup(): Scatter: 2: 1903-4895 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4896-27566 & 33-2147483647
[0] MPI startup(): Scatter: 2: 27567-39885 & 33-2147483647
[0] MPI startup(): Scatter: 3: 39886-126420 & 33-2147483647
[0] MPI startup(): Scatter: 2: 126421-1096865 & 33-2147483647
[0] MPI startup(): Scatter: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Scatterv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       6281     e5-2670-1  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): 1       1972     e5-2670-2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): I_MPI_DEBUG=6
[0] MPI startup(): I_MPI_FABRICS=shm:ofa
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:1,mlx5_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_OFA_ADAPTER_NAME=mlx5_0
[0] MPI startup(): I_MPI_OFA_NUM_PORTS=2
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
[0] MPI startup(): PMI_RANK=0
# OSU MPI Bandwidth Test v4.3
# Size      Bandwidth (MB/s)
1                       0.99
2                       2.02
4                       4.06
8                       8.28
16                     16.53
32                     31.99
64                     63.87
128                   128.85
256                   254.65
512                   495.51
1024                  949.66
2048                 1720.14
4096                 3043.21
8192                 4893.90
16384                5793.38
32768                7280.63
65536               10894.34
131072              11713.48
262144              12230.60
524288              12462.42
1048576             12613.03
2097152             12653.87
4194304             12701.80

Version 5.0 Update 3 doesn't run:

mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 2 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 3  Build 20150128 (build id: 11250)
[0] MPI startup(): Copyright (C) 2003-2015 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 2 ports per adapter
[1] MPI startup(): Start 2 ports per adapter
[0] MPI startup(): ofa fabric is not available and fallback fabric is not enabled
[1] MPI startup(): ofa fabric is not available and fallback fabric is not enabled

Single-rail with OFA is not a problem for both versions:

$ mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 1 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 2  Build 20141030 (build id: 10994)
[0] MPI startup(): Copyright (C) 2003-2014 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[0] MPI startup(): Open 0 IB device: mlx5_0
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 1 ports per adapter
[1] MPI startup(): Start 1 ports per adapter
[0] MPID_nem_ofacm_init(): Init
[1] MPID_nem_ofacm_init(): Init
[0] MPI startup(): shm and ofa data transfer modes
[1] MPI startup(): shm and ofa data transfer modes
[0] MPI startup(): Device_reset_idx=3
[0] MPI startup(): Allgather: 3: 0-1 & 0-4
[0] MPI startup(): Allgather: 1: 1-1 & 0-4
[0] MPI startup(): Allgather: 4: 2-4 & 0-4
[0] MPI startup(): Allgather: 1: 5-92 & 0-4
[0] MPI startup(): Allgather: 4: 93-161 & 0-4
[0] MPI startup(): Allgather: 1: 162-33532 & 0-4
[0] MPI startup(): Allgather: 4: 33533-102618 & 0-4
[0] MPI startup(): Allgather: 1: 102619-546855 & 0-4
[0] MPI startup(): Allgather: 4: 0-2147483647 & 0-4
[0] MPI startup(): Allgather: 4: 1-839 & 5-8
[0] MPI startup(): Allgather: 2: 840-4170 & 5-8
[0] MPI startup(): Allgather: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgather: 1: 1-5 & 9-16
[0] MPI startup(): Allgather: 4: 6-8 & 9-16
[0] MPI startup(): Allgather: 1: 9-440 & 9-16
[0] MPI startup(): Allgather: 2: 441-2083 & 9-16
[0] MPI startup(): Allgather: 4: 2084-9050 & 9-16
[0] MPI startup(): Allgather: 1: 9050-65530 & 9-16
[0] MPI startup(): Allgather: 4: 65530-218986 & 9-16
[0] MPI startup(): Allgather: 1: 0-2147483647 & 9-16
[0] MPI startup(): Allgather: 1: 1-247 & 17-32
[0] MPI startup(): Allgather: 2: 248-1043 & 17-32
[0] MPI startup(): Allgather: 4: 1044-3132 & 17-32
[0] MPI startup(): Allgather: 1: 3133-3010585 & 17-32
[0] MPI startup(): Allgather: 3: 0-2147483647 & 17-32
[0] MPI startup(): Allgather: 1: 1-176 & 33-2147483647
[0] MPI startup(): Allgather: 4: 177-1818 & 33-2147483647
[0] MPI startup(): Allgather: 1: 1819-2278 & 33-2147483647
[0] MPI startup(): Allgather: 4: 2279-4129 & 33-2147483647
[0] MPI startup(): Allgather: 1: 4130-1555968 & 33-2147483647
[0] MPI startup(): Allgather: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-261835 & 0-2
[0] MPI startup(): Allgatherv: 4: 261835-611669 & 0-2
[0] MPI startup(): Allgatherv: 3: 611669-3196878 & 0-2
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 0-2
[0] MPI startup(): Allgatherv: 1: 0-370 & 3-4
[0] MPI startup(): Allgatherv: 2: 370-838 & 3-4
[0] MPI startup(): Allgatherv: 1: 838-3403 & 3-4
[0] MPI startup(): Allgatherv: 4: 3403-4808 & 3-4
[0] MPI startup(): Allgatherv: 1: 4808-8294 & 3-4
[0] MPI startup(): Allgatherv: 3: 8294-130946 & 3-4
[0] MPI startup(): Allgatherv: 4: 130946-208466 & 3-4
[0] MPI startup(): Allgatherv: 3: 208466-393216 & 3-4
[0] MPI startup(): Allgatherv: 4: 393216-617063 & 3-4
[0] MPI startup(): Allgatherv: 3: 617063-1398101 & 3-4
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 3-4
[0] MPI startup(): Allgatherv: 1: 0-25 & 5-8
[0] MPI startup(): Allgatherv: 4: 25-46 & 5-8
[0] MPI startup(): Allgatherv: 1: 46-1403 & 5-8
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 5-8
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 5-8
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 5-8
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-3 & 9-16
[0] MPI startup(): Allgatherv: 1: 3-23 & 9-16
[0] MPI startup(): Allgatherv: 4: 23-106 & 9-16
[0] MPI startup(): Allgatherv: 1: 106-797 & 9-16
[0] MPI startup(): Allgatherv: 2: 797-3141 & 9-16
[0] MPI startup(): Allgatherv: 3: 3141-5874 & 9-16
[0] MPI startup(): Allgatherv: 2: 5874-9274 & 9-16
[0] MPI startup(): Allgatherv: 1: 9274-17847 & 9-16
[0] MPI startup(): Allgatherv: 3: 17847-84012 & 9-16
[0] MPI startup(): Allgatherv: 4: 84012-225929 & 9-16
[0] MPI startup(): Allgatherv: 3: 225929-450203 & 9-16
[0] MPI startup(): Allgatherv: 4: 450203-545471 & 9-16
[0] MPI startup(): Allgatherv: 3: 545471-1767416 & 9-16
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allgatherv: 1: 0-25 & 17-32
[0] MPI startup(): Allgatherv: 4: 25-46 & 17-32
[0] MPI startup(): Allgatherv: 1: 46-1403 & 17-32
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 17-32
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 17-32
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 17-32
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-1 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 1-3 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 3-6 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 6-96 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 96-535 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 535-1255 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1255-3148 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 3148-5455 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 5455-17721 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 17721-41571 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 41571-110427 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 110427-596874 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 596874-1273594 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1273594-2420103 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allreduce: 7: 0-20 & 0-2
[0] MPI startup(): Allreduce: 1: 20-905 & 0-2
[0] MPI startup(): Allreduce: 4: 905-2058 & 0-2
[0] MPI startup(): Allreduce: 1: 2058-17240 & 0-2
[0] MPI startup(): Allreduce: 8: 17240-36706 & 0-2
[0] MPI startup(): Allreduce: 3: 36706-777395 & 0-2
[0] MPI startup(): Allreduce: 8: 777395-1247501 & 0-2
[0] MPI startup(): Allreduce: 2: 1247501-3149905 & 0-2
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 0-2
[0] MPI startup(): Allreduce: 1: 0-597 & 3-4
[0] MPI startup(): Allreduce: 7: 597-1755 & 3-4
[0] MPI startup(): Allreduce: 1: 1755-16932 & 3-4
[0] MPI startup(): Allreduce: 6: 16932-34438 & 3-4
[0] MPI startup(): Allreduce: 8: 34438-81725 & 3-4
[0] MPI startup(): Allreduce: 3: 81725-396925 & 3-4
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 3-4
[0] MPI startup(): Allreduce: 1: 0-55 & 5-8
[0] MPI startup(): Allreduce: 6: 55-74 & 5-8
[0] MPI startup(): Allreduce: 4: 74-213 & 5-8
[0] MPI startup(): Allreduce: 1: 213-1765 & 5-8
[0] MPI startup(): Allreduce: 4: 1765-2065 & 5-8
[0] MPI startup(): Allreduce: 1: 2065-16561 & 5-8
[0] MPI startup(): Allreduce: 2: 16561-39707 & 5-8
[0] MPI startup(): Allreduce: 8: 39707-168698 & 5-8
[0] MPI startup(): Allreduce: 7: 168698-570628 & 5-8
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 5-8
[0] MPI startup(): Allreduce: 6: 0-5 & 9-16
[0] MPI startup(): Allreduce: 1: 5-935 & 9-16
[0] MPI startup(): Allreduce: 4: 935-1105 & 9-16
[0] MPI startup(): Allreduce: 1: 1105-17349 & 9-16
[0] MPI startup(): Allreduce: 4: 17349-53440 & 9-16
[0] MPI startup(): Allreduce: 8: 53440-315615 & 9-16
[0] MPI startup(): Allreduce: 7: 315615-1140547 & 9-16
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allreduce: 6: 0-74 & 17-32
[0] MPI startup(): Allreduce: 4: 74-199 & 17-32
[0] MPI startup(): Allreduce: 6: 199-392 & 17-32
[0] MPI startup(): Allreduce: 4: 392-750 & 17-32
[0] MPI startup(): Allreduce: 6: 750-2087 & 17-32
[0] MPI startup(): Allreduce: 1: 2087-14841 & 17-32
[0] MPI startup(): Allreduce: 2: 14841-20338 & 17-32
[0] MPI startup(): Allreduce: 6: 20338-124787 & 17-32
[0] MPI startup(): Allreduce: 8: 124787-646663 & 17-32
[0] MPI startup(): Allreduce: 7: 646663-2129006 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 33-2147483647
[1] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): Alltoall: 3: 0-245 & 0-2
[0] MPI startup(): Alltoall: 2: 245-276 & 0-2
[0] MPI startup(): Alltoall: 3: 0-2147483647 & 0-2
[0] MPI startup(): Alltoall: 2: 0-3878 & 3-4
[0] MPI startup(): Alltoall: 3: 3879-6492 & 3-4
[0] MPI startup(): Alltoall: 4: 6493-9680 & 3-4
[0] MPI startup(): Alltoall: 3: 9681-24681 & 3-4
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 3-4
[0] MPI startup(): Alltoall: 2: 0-3082 & 5-8
[0] MPI startup(): Alltoall: 3: 3083-20124 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2235 & 9-16
[0] MPI startup(): Alltoall: 3: 2236-26946 & 9-16
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 9-16
[0] MPI startup(): Alltoall: 1: 0-329 & 17-32
[0] MPI startup(): Alltoall: 2: 330-4661 & 17-32
[0] MPI startup(): Alltoall: 3: 4662-282206 & 17-32
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 17-32
[0] MPI startup(): Alltoall: 1: 0-217 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 218-409 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 410-4415 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 4416-32356 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 32357-36297 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 36298-65861 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 65862-866313 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoallv: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Alltoallw: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Barrier: 6: 0-2147483647 & 0-2147483647
[0] MPI startup(): Bcast: 2: 0-1 & 0-2
[0] MPI startup(): Bcast: 7: 1-402 & 0-2
[0] MPI startup(): Bcast: 5: 402-682 & 0-2
[0] MPI startup(): Bcast: 4: 682-1433 & 0-2
[0] MPI startup(): Bcast: 7: 1433-5734 & 0-2
[0] MPI startup(): Bcast: 1: 5734-21845 & 0-2
[0] MPI startup(): Bcast: 6: 21845-95963 & 0-2
[0] MPI startup(): Bcast: 5: 95963-409897 & 0-2
[0] MPI startup(): Bcast: 1: 0-2147483647 & 0-2
[0] MPI startup(): Bcast: 7: 0-2147483647 & 3-4
[0] MPI startup(): Bcast: 7: 1-1 & 5-8
[0] MPI startup(): Bcast: 1: 2-254 & 5-8
[0] MPI startup(): Bcast: 5: 255-261 & 5-8
[0] MPI startup(): Bcast: 1: 262-752 & 5-8
[0] MPI startup(): Bcast: 6: 753-1254 & 5-8
[0] MPI startup(): Bcast: 1: 1255-4032 & 5-8
[0] MPI startup(): Bcast: 4: 4033-19764 & 5-8
[0] MPI startup(): Bcast: 1: 19765-121346 & 5-8
[0] MPI startup(): Bcast: 6: 121347-201114 & 5-8
[0] MPI startup(): Bcast: 1: 201115-267790 & 5-8
[0] MPI startup(): Bcast: 2: 267791-588703 & 5-8
[0] MPI startup(): Bcast: 3: 588704-1341590 & 5-8
[0] MPI startup(): Bcast: 5: 1341591-2139116 & 5-8
[0] MPI startup(): Bcast: 6: 0-2147483647 & 5-8
[0] MPI startup(): Bcast: 7: 1-84 & 9-16
[0] MPI startup(): Bcast: 1: 85-137 & 9-16
[0] MPI startup(): Bcast: 6: 138-475 & 9-16
[0] MPI startup(): Bcast: 1: 476-1496 & 9-16
[0] MPI startup(): Bcast: 6: 1497-10273 & 9-16
[0] MPI startup(): Bcast: 1: 10274-106199 & 9-16
[0] MPI startup(): Bcast: 6: 106200-323972 & 9-16
[0] MPI startup(): Bcast: 2: 0-2147483647 & 9-16
[0] MPI startup(): Bcast: 7: 1-103 & 17-32
[0] MPI startup(): Bcast: 5: 104-180 & 17-32
[0] MPI startup(): Bcast: 7: 181-590 & 17-32
[0] MPI startup(): Bcast: 5: 591-1170 & 17-32
[0] MPI startup(): Bcast: 6: 1171-3459 & 17-32
[0] MPI startup(): Bcast: 1: 3460-52532 & 17-32
[0] MPI startup(): Bcast: 2: 52533-239896 & 17-32
[0] MPI startup(): Bcast: 5: 239897-262474 & 17-32
[0] MPI startup(): Bcast: 2: 0-2147483647 & 17-32
[0] MPI startup(): Bcast: 7: 1-935 & 33-2147483647
[0] MPI startup(): Bcast: 6: 936-1185 & 33-2147483647
[0] MPI startup(): Bcast: 1: 1186-4873 & 33-2147483647
[0] MPI startup(): Bcast: 6: 4874-9664 & 33-2147483647
[0] MPI startup(): Bcast: 1: 9665-85765 & 33-2147483647
[0] MPI startup(): Bcast: 2: 85766-250963 & 33-2147483647
[0] MPI startup(): Bcast: 5: 250964-262409 & 33-2147483647
[0] MPI startup(): Bcast: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Exscan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Gather: 3: 1-8 & 0-4
[0] MPI startup(): Gather: 1: 9-29 & 0-4
[0] MPI startup(): Gather: 3: 30-25688 & 0-4
[0] MPI startup(): Gather: 2: 25689-56274 & 0-4
[0] MPI startup(): Gather: 3: 56275-68784 & 0-4
[0] MPI startup(): Gather: 2: 68785-1304326 & 0-4
[0] MPI startup(): Gather: 3: 1304327-2684607 & 0-4
[0] MPI startup(): Gather: 2: 0-2147483647 & 0-4
[0] MPI startup(): Gather: 3: 1-10278 & 5-8
[0] MPI startup(): Gather: 2: 10279-22432 & 5-8
[0] MPI startup(): Gather: 3: 22433-51123 & 5-8
[0] MPI startup(): Gather: 2: 51124-93434 & 5-8
[0] MPI startup(): Gather: 3: 93435-667907 & 5-8
[0] MPI startup(): Gather: 2: 667908-1262190 & 5-8
[0] MPI startup(): Gather: 3: 1262191-2879271 & 5-8
[0] MPI startup(): Gather: 2: 0-2147483647 & 5-8
[0] MPI startup(): Gather: 3: 1-4004 & 9-16
[0] MPI startup(): Gather: 2: 4005-4168 & 9-16
[0] MPI startup(): Gather: 3: 4169-20254 & 9-16
[0] MPI startup(): Gather: 2: 20255-41713 & 9-16
[0] MPI startup(): Gather: 3: 41714-793099 & 9-16
[0] MPI startup(): Gather: 2: 793100-1272717 & 9-16
[0] MPI startup(): Gather: 3: 0-2147483647 & 9-16
[0] MPI startup(): Gather: 3: 1-2047 & 17-32
[0] MPI startup(): Gather: 2: 2048-16131 & 17-32
[0] MPI startup(): Gather: 3: 16132-25688 & 17-32
[0] MPI startup(): Gather: 2: 25689-56274 & 17-32
[0] MPI startup(): Gather: 3: 56275-68784 & 17-32
[0] MPI startup(): Gather: 2: 68785-1304326 & 17-32
[0] MPI startup(): Gather: 3: 1304327-2684607 & 17-32
[0] MPI startup(): Gather: 2: 0-2147483647 & 17-32
[0] MPI startup(): Gather: 3: 1-51845 & 33-2147483647
[0] MPI startup(): Gather: 2: 51846-76744 & 33-2147483647
[0] MPI startup(): Gather: 3: 76745-721936 & 33-2147483647
[0] MPI startup(): Gather: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Gatherv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-5 & 0-2
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 0-2
[0] MPI startup(): Reduce_scatter: 1: 0-29 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 29-48 & 3-8
[0] MPI startup(): Reduce_scatter: 1: 48-2805 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 2805-5556 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 5556-11732 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 11732-35932 & 3-8
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 3-8
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 9-16
[0] MPI startup(): Reduce_scatter: 1: 4-2807 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 2807-6883 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 6883-39653 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 39653-513184 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 513184-966175 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 17-32
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 4-14959 & 17-32
[0] MPI startup(): Reduce_scatter: 5: 14959-19362 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 19362-49970 & 17-32
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 17-32
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 1: 4-79521 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 79521-520875 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 5: 520875-1101846 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Reduce: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scatter: 3: 1-8 & 0-4
[0] MPI startup(): Scatter: 1: 9-31 & 0-4
[0] MPI startup(): Scatter: 3: 32-4867 & 0-4
[0] MPI startup(): Scatter: 2: 4868-18877 & 0-4
[0] MPI startup(): Scatter: 3: 18878-854395 & 0-4
[0] MPI startup(): Scatter: 2: 854396-3121304 & 0-4
[0] MPI startup(): Scatter: 3: 0-2147483647 & 0-4
[0] MPI startup(): Scatter: 3: 1-1 & 5-16
[0] MPI startup(): Scatter: 2: 2-3 & 5-16
[0] MPI startup(): Scatter: 3: 4-8 & 5-16
[0] MPI startup(): Scatter: 1: 9-31 & 5-16
[0] MPI startup(): Scatter: 3: 32-1902 & 5-16
[0] MPI startup(): Scatter: 2: 1903-4895 & 5-16
[0] MPI startup(): Scatter: 3: 4896-27566 & 5-16
[0] MPI startup(): Scatter: 2: 27567-39885 & 5-16
[0] MPI startup(): Scatter: 3: 39886-126420 & 5-16
[0] MPI startup(): Scatter: 2: 126421-1096865 & 5-16
[0] MPI startup(): Scatter: 3: 0-2147483647 & 5-16
[0] MPI startup(): Scatter: 3: 1-71026 & 17-32
[0] MPI startup(): Scatter: 2: 71027-339781 & 17-32
[0] MPI startup(): Scatter: 3: 339782-1290592 & 17-32
[0] MPI startup(): Scatter: 2: 1290593-3878829 & 17-32
[0] MPI startup(): Scatter: 3: 0-2147483647 & 17-32
[0] MPI startup(): Scatter: 3: 1-1 & 33-2147483647
[0] MPI startup(): Scatter: 2: 2-3 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4-8 & 33-2147483647
[0] MPI startup(): Scatter: 1: 9-31 & 33-2147483647
[0] MPI startup(): Scatter: 3: 32-1902 & 33-2147483647
[0] MPI startup(): Scatter: 2: 1903-4895 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4896-27566 & 33-2147483647
[0] MPI startup(): Scatter: 2: 27567-39885 & 33-2147483647
[0] MPI startup(): Scatter: 3: 39886-126420 & 33-2147483647
[0] MPI startup(): Scatter: 2: 126421-1096865 & 33-2147483647
[0] MPI startup(): Scatter: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Scatterv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       6303     e5-2670-1  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): 1       2013     e5-2670-2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): I_MPI_DEBUG=6
[0] MPI startup(): I_MPI_FABRICS=shm:ofa
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:1,mlx5_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_OFA_ADAPTER_NAME=mlx5_0
[0] MPI startup(): I_MPI_OFA_NUM_PORTS=1
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
[0] MPI startup(): PMI_RANK=0
# OSU MPI Bandwidth Test v4.3
# Size      Bandwidth (MB/s)
1                       0.97
2                       1.98
4                       4.06
8                       8.06
16                     16.28
32                     31.57
64                     62.85
128                   126.21
256                   250.36
512                   489.73
1024                  939.56
2048                 1705.57
4096                 3024.29
8192                 4840.15
16384                5495.84
32768                5529.70
65536                5952.25
131072               6190.62
262144               6300.36
524288               6362.90
1048576              6390.01
2097152              6405.34
4194304              6414.40

$ mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 1 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 3  Build 20150128 (build id: 11250)
[0] MPI startup(): Copyright (C) 2003-2015 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[0] MPI startup(): Open 0 IB device: mlx5_0
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 1 ports per adapter
[1] MPI startup(): Start 1 ports per adapter
[0] MPID_nem_ofacm_init(): Init
[1] MPID_nem_ofacm_init(): Init
[0] MPI startup(): shm and ofa data transfer modes
[1] MPI startup(): shm and ofa data transfer modes
[0] MPI startup(): Device_reset_idx=3
[0] MPI startup(): Allgather: 3: 0-1 & 0-4
[0] MPI startup(): Allgather: 1: 1-1 & 0-4
[0] MPI startup(): Allgather: 4: 2-4 & 0-4
[0] MPI startup(): Allgather: 1: 5-92 & 0-4
[0] MPI startup(): Allgather: 4: 93-161 & 0-4
[0] MPI startup(): Allgather: 1: 162-33532 & 0-4
[0] MPI startup(): Allgather: 4: 33533-102618 & 0-4
[0] MPI startup(): Allgather: 1: 102619-546855 & 0-4
[0] MPI startup(): Allgather: 4: 0-2147483647 & 0-4
[0] MPI startup(): Allgather: 4: 1-839 & 5-8
[0] MPI startup(): Allgather: 2: 840-4170 & 5-8
[0] MPI startup(): Allgather: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgather: 1: 1-5 & 9-16
[0] MPI startup(): Allgather: 4: 6-8 & 9-16
[0] MPI startup(): Allgather: 1: 9-440 & 9-16
[0] MPI startup(): Allgather: 2: 441-2083 & 9-16
[0] MPI startup(): Allgather: 4: 2084-9050 & 9-16
[0] MPI startup(): Allgather: 1: 9050-65530 & 9-16
[0] MPI startup(): Allgather: 4: 65530-218986 & 9-16
[0] MPI startup(): Allgather: 1: 0-2147483647 & 9-16
[0] MPI startup(): Allgather: 1: 1-247 & 17-32
[0] MPI startup(): Allgather: 2: 248-1043 & 17-32
[0] MPI startup(): Allgather: 4: 1044-3132 & 17-32
[0] MPI startup(): Allgather: 1: 3133-3010585 & 17-32
[0] MPI startup(): Allgather: 3: 0-2147483647 & 17-32
[0] MPI startup(): Allgather: 1: 1-176 & 33-2147483647
[0] MPI startup(): Allgather: 4: 177-1818 & 33-2147483647
[0] MPI startup(): Allgather: 1: 1819-2278 & 33-2147483647
[0] MPI startup(): Allgather: 4: 2279-4129 & 33-2147483647
[0] MPI startup(): Allgather: 1: 4130-1555968 & 33-2147483647
[0] MPI startup(): Allgather: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-261835 & 0-2
[0] MPI startup(): Allgatherv: 4: 261835-611669 & 0-2
[0] MPI startup(): Allgatherv: 3: 611669-3196878 & 0-2
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 0-2
[0] MPI startup(): Allgatherv: 1: 0-370 & 3-4
[0] MPI startup(): Allgatherv: 2: 370-838 & 3-4
[0] MPI startup(): Allgatherv: 1: 838-3403 & 3-4
[0] MPI startup(): Allgatherv: 4: 3403-4808 & 3-4
[0] MPI startup(): Allgatherv: 1: 4808-8294 & 3-4
[0] MPI startup(): Allgatherv: 3: 8294-130946 & 3-4
[0] MPI startup(): Allgatherv: 4: 130946-208466 & 3-4
[0] MPI startup(): Allgatherv: 3: 208466-393216 & 3-4
[0] MPI startup(): Allgatherv: 4: 393216-617063 & 3-4
[0] MPI startup(): Allgatherv: 3: 617063-1398101 & 3-4
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 3-4
[0] MPI startup(): Allgatherv: 1: 0-25 & 5-8
[0] MPI startup(): Allgatherv: 4: 25-46 & 5-8
[0] MPI startup(): Allgatherv: 1: 46-1403 & 5-8
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 5-8
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 5-8
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 5-8
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-3 & 9-16
[0] MPI startup(): Allgatherv: 1: 3-23 & 9-16
[0] MPI startup(): Allgatherv: 4: 23-106 & 9-16
[0] MPI startup(): Allgatherv: 1: 106-797 & 9-16
[0] MPI startup(): Allgatherv: 2: 797-3141 & 9-16
[0] MPI startup(): Allgatherv: 3: 3141-5874 & 9-16
[0] MPI startup(): Allgatherv: 2: 5874-9274 & 9-16
[0] MPI startup(): Allgatherv: 1: 9274-17847 & 9-16
[0] MPI startup(): Allgatherv: 3: 17847-84012 & 9-16
[0] MPI startup(): Allgatherv: 4: 84012-225929 & 9-16
[0] MPI startup(): Allgatherv: 3: 225929-450203 & 9-16
[0] MPI startup(): Allgatherv: 4: 450203-545471 & 9-16
[0] MPI startup(): Allgatherv: 3: 545471-1767416 & 9-16
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allgatherv: 1: 0-25 & 17-32
[0] MPI startup(): Allgatherv: 4: 25-46 & 17-32
[0] MPI startup(): Allgatherv: 1: 46-1403 & 17-32
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 17-32
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 17-32
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 17-32
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-1 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 1-3 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 3-6 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 6-96 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 96-535 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 535-1255 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1255-3148 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 3148-5455 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 5455-17721 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 17721-41571 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 41571-110427 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 110427-596874 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 596874-1273594 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1273594-2420103 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allreduce: 7: 0-20 & 0-2
[0] MPI startup(): Allreduce: 1: 20-905 & 0-2
[0] MPI startup(): Allreduce: 4: 905-2058 & 0-2
[0] MPI startup(): Allreduce: 1: 2058-17240 & 0-2
[0] MPI startup(): Allreduce: 8: 17240-36706 & 0-2
[0] MPI startup(): Allreduce: 3: 36706-777395 & 0-2
[0] MPI startup(): Allreduce: 8: 777395-1247501 & 0-2
[0] MPI startup(): Allreduce: 2: 1247501-3149905 & 0-2
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 0-2
[0] MPI startup(): Allreduce: 1: 0-597 & 3-4
[0] MPI startup(): Allreduce: 7: 597-1755 & 3-4
[0] MPI startup(): Allreduce: 1: 1755-16932 & 3-4
[0] MPI startup(): Allreduce: 6: 16932-34438 & 3-4
[0] MPI startup(): Allreduce: 8: 34438-81725 & 3-4
[0] MPI startup(): Allreduce: 3: 81725-396925 & 3-4
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 3-4
[0] MPI startup(): Allreduce: 1: 0-55 & 5-8
[0] MPI startup(): Allreduce: 6: 55-74 & 5-8
[0] MPI startup(): Allreduce: 4: 74-213 & 5-8
[0] MPI startup(): Allreduce: 1: 213-1765 & 5-8
[0] MPI startup(): Allreduce: 4: 1765-2065 & 5-8
[0] MPI startup(): Allreduce: 1: 2065-16561 & 5-8
[0] MPI startup(): Allreduce: 2: 16561-39707 & 5-8
[0] MPI startup(): Allreduce: 8: 39707-168698 & 5-8
[0] MPI startup(): Allreduce: 7: 168698-570628 & 5-8
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 5-8
[0] MPI startup(): Allreduce: 6: 0-5 & 9-16
[0] MPI startup(): Allreduce: 1: 5-935 & 9-16
[0] MPI startup(): Allreduce: 4: 935-1105 & 9-16
[0] MPI startup(): Allreduce: 1: 1105-17349 & 9-16
[0] MPI startup(): Allreduce: 4: 17349-53440 & 9-16
[0] MPI startup(): Allreduce: 8: 53440-315615 & 9-16
[0] MPI startup(): Allreduce: 7: 315615-1140547 & 9-16
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allreduce: 6: 0-74 & 17-32
[0] MPI startup(): Allreduce: 4: 74-199 & 17-32
[0] MPI startup(): Allreduce: 6: 199-392 & 17-32
[0] MPI startup(): Allreduce: 4: 392-750 & 17-32
[0] MPI startup(): Allreduce: 6: 750-2087 & 17-32
[0] MPI startup(): Allreduce: 1: 2087-14841 & 17-32
[0] MPI startup(): Allreduce: 2: 14841-20338 & 17-32
[0] MPI startup(): Allreduce: 6: 20338-124787 & 17-32
[0] MPI startup(): Allreduce: 8: 124787-646663 & 17-32
[0] MPI startup(): Allreduce: 7: 646663-2129006 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 0-245 & 0-2
[0] MPI startup(): Alltoall: 2: 245-276 & 0-2
[0] MPI startup(): Alltoall: 3: 0-2147483647 & 0-2
[0] MPI startup(): Alltoall: 2: 0-3878 & 3-4
[0] MPI startup(): Alltoall: 3: 3879-6492 & 3-4
[0] MPI startup(): Alltoall: 4: 6493-9680 & 3-4
[0] MPI startup(): Alltoall: 3: 9681-24681 & 3-4
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 3-4
[0] MPI startup(): Alltoall: 2: 0-3082 & 5-8
[0] MPI startup(): Alltoall: 3: 3083-20124 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2235 & 9-16
[0] MPI startup(): Alltoall: 3: 2236-26946 & 9-16
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 9-16
[0] MPI startup(): Alltoall: 1: 0-329 & 17-32
[0] MPI startup(): Alltoall: 2: 330-4661 & 17-32
[0] MPI startup(): Alltoall: 3: 4662-282206 & 17-32
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 17-32
[0] MPI startup(): Alltoall: 1: 0-217 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 218-409 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 410-4415 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 4416-32356 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 32357-36297 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 36298-65861 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 65862-866313 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoallv: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Alltoallw: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Barrier: 6: 0-2147483647 & 0-2147483647
[0] MPI startup(): Bcast: 2: 0-1 & 0-2
[0] MPI startup(): Bcast: 7: 1-402 & 0-2
[0] MPI startup(): Bcast: 5: 402-682 & 0-2
[0] MPI startup(): Bcast: 4: 682-1433 & 0-2
[0] MPI startup(): Bcast: 7: 1433-5734 & 0-2
[0] MPI startup(): Bcast: 1: 5734-21845 & 0-2
[0] MPI startup(): Bcast: 6: 21845-95963 & 0-2
[0] MPI startup(): Bcast: 5: 95963-409897 & 0-2
[0] MPI startup(): Bcast: 1: 0-2147483647 & 0-2
[0] MPI startup(): Bcast: 7: 0-2147483647 & 3-4
[0] MPI startup(): Bcast: 7: 1-1 & 5-8
[0] MPI startup(): Bcast: 1: 2-254 & 5-8
[0] MPI startup(): Bcast: 5: 255-261 & 5-8
[0] MPI startup(): Bcast: 1: 262-752 & 5-8
[0] MPI startup(): Bcast: 6: 753-1254 & 5-8
[0] MPI startup(): Bcast: 1: 1255-4032 & 5-8
[0] MPI startup(): Bcast: 4: 4033-19764 & 5-8
[0] MPI startup(): Bcast: 1: 19765-121346 & 5-8
[0] MPI startup(): Bcast: 6: 121347-201114 & 5-8
[0] MPI startup(): Bcast: 1: 201115-267790 & 5-8
[0] MPI startup(): Bcast: 2: 267791-588703 & 5-8
[0] MPI startup(): Bcast: 3: 588704-1341590 & 5-8
[0] MPI startup(): Bcast: 5: 1341591-2139116 & 5-8
[0] MPI startup(): Bcast: 6: 0-2147483647 & 5-8
[0] MPI startup(): Bcast: 7: 1-84 & 9-16
[0] MPI startup(): Bcast: 1: 85-137 & 9-16
[0] MPI startup(): Bcast: 6: 138-475 & 9-16
[0] MPI startup(): Bcast: 1: 476-1496 & 9-16
[0] MPI startup(): Bcast: 6: 1497-10273 & 9-16
[0] MPI startup(): Bcast: 1: 10274-106199 & 9-16
[0] MPI startup(): Bcast: 6: 106200-323972 & 9-16
[0] MPI startup(): Bcast: 2: 0-2147483647 & 9-16
[0] MPI startup(): Bcast: 7: 1-103 & 17-32
[0] MPI startup(): Bcast: 5: 104-180 & 17-32
[0] MPI startup(): Bcast: 7: 181-590 & 17-32
[0] MPI startup(): Bcast: 5: 591-1170 & 17-32
[0] MPI startup(): Bcast: 6: 1171-3459 & 17-32
[0] MPI startup(): Bcast: 1: 3460-52532 & 17-32
[0] MPI startup(): Bcast: 2: 52533-239896 & 17-32
[0] MPI startup(): Bcast: 5: 239897-262474 & 17-32
[0] MPI startup(): Bcast: 2: 0-2147483647 & 17-32
[0] MPI startup(): Bcast: 7: 1-935 & 33-2147483647
[0] MPI startup(): Bcast: 6: 936-1185 & 33-2147483647
[0] MPI startup(): Bcast: 1: 1186-4873 & 33-2147483647
[0] MPI startup(): Bcast: 6: 4874-9664 & 33-2147483647
[0] MPI startup(): Bcast: 1: 9665-85765 & 33-2147483647
[0] MPI startup(): Bcast: 2: 85766-250963 & 33-2147483647
[0] MPI startup(): Bcast: 5: 250964-262409 & 33-2147483647
[0] MPI startup(): Bcast: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Exscan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Gather: 3: 1-8 & 0-4
[0] MPI startup(): Gather: 1: 9-29 & 0-4
[0] MPI startup(): Gather: 3: 30-25688 & 0-4
[0] MPI startup(): Gather: 2: 25689-56274 & 0-4
[0] MPI startup(): Gather: 3: 56275-68784 & 0-4
[0] MPI startup(): Gather: 2: 68785-1304326 & 0-4
[0] MPI startup(): Gather: 3: 1304327-2684607 & 0-4
[0] MPI startup(): Gather: 2: 0-2147483647 & 0-4
[0] MPI startup(): Gather: 3: 1-10278 & 5-8
[0] MPI startup(): Gather: 2: 10279-22432 & 5-8
[0] MPI startup(): Gather: 3: 22433-51123 & 5-8
[0] MPI startup(): Gather: 2: 51124-93434 & 5-8
[0] MPI startup(): Gather: 3: 93435-667907 & 5-8
[0] MPI startup(): Gather: 2: 667908-1262190 & 5-8
[0] MPI startup(): Gather: 3: 1262191-2879271 & 5-8
[0] MPI startup(): Gather: 2: 0-2147483647 & 5-8
[0] MPI startup(): Gather: 3: 1-4004 & 9-16
[0] MPI startup(): Gather: 2: 4005-4168 & 9-16
[0] MPI startup(): Gather: 3: 4169-20254 & 9-16
[0] MPI startup(): Gather: 2: 20255-41713 & 9-16
[0] MPI startup(): Gather: 3: 41714-793099 & 9-16
[0] MPI startup(): Gather: 2: 793100-1272717 & 9-16
[0] MPI startup(): Gather: 3: 0-2147483647 & 9-16
[0] MPI startup(): Gather: 3: 1-2047 & 17-32
[0] MPI startup(): Gather: 2: 2048-16131 & 17-32
[0] MPI startup(): Gather: 3: 16132-25688 & 17-32
[0] MPI startup(): Gather: 2: 25689-56274 & 17-32
[0] MPI startup(): Gather: 3: 56275-68784 & 17-32
[0] MPI startup(): Gather: 2: 68785-1304326 & 17-32
[0] MPI startup(): Gather: 3: 1304327-2684607 & 17-32
[0] MPI startup(): Gather: 2: 0-2147483647 & 17-32
[0] MPI startup(): Gather: 3: 1-51845 & 33-2147483647
[0] MPI startup(): Gather: 2: 51846-76744 & 33-2147483647
[0] MPI startup(): Gather: 3: 76745-721936 & 33-2147483647
[0] MPI startup(): Gather: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Gatherv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-5 & 0-2
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 0-2
[0] MPI startup(): Reduce_scatter: 1: 0-29 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 29-48 & 3-8
[0] MPI startup(): Reduce_scatter: 1: 48-2805 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 2805-5556 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 5556-11732 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 11732-35932 & 3-8
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 3-8
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 9-16
[0] MPI startup(): Reduce_scatter: 1: 4-2807 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 2807-6883 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 6883-39653 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 39653-513184 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 513184-966175 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 17-32
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 4-14959 & 17-32
[0] MPI startup(): Reduce_scatter: 5: 14959-19362 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 19362-49970 & 17-32
[1] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 17-32
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 1: 4-79521 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 79521-520875 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 5: 520875-1101846 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Reduce: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scatter: 3: 1-8 & 0-4
[0] MPI startup(): Scatter: 1: 9-31 & 0-4
[0] MPI startup(): Scatter: 3: 32-4867 & 0-4
[0] MPI startup(): Scatter: 2: 4868-18877 & 0-4
[0] MPI startup(): Scatter: 3: 18878-854395 & 0-4
[0] MPI startup(): Scatter: 2: 854396-3121304 & 0-4
[0] MPI startup(): Scatter: 3: 0-2147483647 & 0-4
[0] MPI startup(): Scatter: 3: 1-1 & 5-16
[0] MPI startup(): Scatter: 2: 2-3 & 5-16
[0] MPI startup(): Scatter: 3: 4-8 & 5-16
[0] MPI startup(): Scatter: 1: 9-31 & 5-16
[0] MPI startup(): Scatter: 3: 32-1902 & 5-16
[0] MPI startup(): Scatter: 2: 1903-4895 & 5-16
[0] MPI startup(): Scatter: 3: 4896-27566 & 5-16
[0] MPI startup(): Scatter: 2: 27567-39885 & 5-16
[0] MPI startup(): Scatter: 3: 39886-126420 & 5-16
[0] MPI startup(): Scatter: 2: 126421-1096865 & 5-16
[0] MPI startup(): Scatter: 3: 0-2147483647 & 5-16
[0] MPI startup(): Scatter: 3: 1-71026 & 17-32
[0] MPI startup(): Scatter: 2: 71027-339781 & 17-32
[0] MPI startup(): Scatter: 3: 339782-1290592 & 17-32
[0] MPI startup(): Scatter: 2: 1290593-3878829 & 17-32
[0] MPI startup(): Scatter: 3: 0-2147483647 & 17-32
[0] MPI startup(): Scatter: 3: 1-1 & 33-2147483647
[0] MPI startup(): Scatter: 2: 2-3 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4-8 & 33-2147483647
[0] MPI startup(): Scatter: 1: 9-31 & 33-2147483647
[0] MPI startup(): Scatter: 3: 32-1902 & 33-2147483647
[0] MPI startup(): Scatter: 2: 1903-4895 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4896-27566 & 33-2147483647
[0] MPI startup(): Scatter: 2: 27567-39885 & 33-2147483647
[0] MPI startup(): Scatter: 3: 39886-126420 & 33-2147483647
[0] MPI startup(): Scatter: 2: 126421-1096865 & 33-2147483647
[0] MPI startup(): Scatter: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Scatterv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       6405     e5-2670-1  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): 1       2108     e5-2670-2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): I_MPI_DEBUG=6
[0] MPI startup(): I_MPI_FABRICS=shm:ofa
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:1,mlx5_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_OFA_ADAPTER_NAME=mlx5_0
[0] MPI startup(): I_MPI_OFA_NUM_PORTS=1
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
[0] MPI startup(): PMI_RANK=0
# OSU MPI Bandwidth Test v4.3
# Size      Bandwidth (MB/s)
1                       0.97
2                       1.94
4                       3.99
8                       8.08
16                     16.05
32                     29.25
64                     58.31
128                   116.07
256                   229.98
512                   446.80
1024                  873.71
2048                 1564.83
4096                 2819.04
8192                 4638.51
16384                5488.30
32768                5518.08
65536                5942.60
131072               6175.84
262144               6291.72
524288               6358.98
1048576              6390.17
2097152              6406.38
4194304              6413.95

We are running CentOS 6.6 with Mellanox OFED 2.4-1.0.4 and Mellanox Connect-IB MCB194A-FCAT Dual-Port FDR PCIe 3.0 x16.

Best regards,

René

Multi-rail ofa fabric not available (Intel MPI Library Version 5.0 Update 3)

Trending Articles

Practice Sheet of Right form of verbs for HSC Students

Download: FK ft Shenky – Nakuyewa ”Prod by: Shenky”

How to win at Markstrat (Markstrat Tips and Tricks) – Vodites

Ominde Commission Report and Recommendations – Ominde Report of 1964

Bureau of Internal Revenue: Regional Offices (Directory)

GO 53 on Enhancement of Ex-gratia upto 5 Lakhs Toddy Tappers in Telangana

Cakewalk CA-2A Leveling Amplifier v2.0.1.97 WiN, v2.0.1.96 OSX Incl Keygen

Mp3 Download: Mdu - Kunjenjenjena

How the kill the job , when DTP request running for long hours.

Microsoft Intune から展開しているアプリのアップデートについて

18-year-old girl was beaten for half an hour by two Northampton men in 'an...

Car crash in Dunton Bassett leaves driver in critical condition

Macky 2, Two Others In Road Accident

Application log 00000000000000089514: Could not convert queue DLVST90CLNT

Detroit mafia: D’Anna Brothers agree to plea deal

Delivery block field greyed out using VA02

Muloraki Au

【個人撮影】スマホのプライベート映像♪「中に出さないで///」カラオケ屋での生ハメ撮りが流出ｗ【リベンジポルノ】＠PornHub

BREAKING NEWS: Diamond Platnumz Is Reported Dead After Ghastly Car Accident

FIAT 500 B0111 B0112