Quantcast
Channel: Clusters and HPC Technology
Viewing all articles
Browse latest Browse all 927
↧

MPI on two machines with different choices of I_MPI_FABRICS

June 5, 2017, 5:21 am
$
0
0

I am trying to run a simple "hello world" MPI program using two machines.

Here is the MPI program:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  MPI_Init(NULL, NULL);

  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  printf("Hello world from processor %s, rank %d out of %d processors\n",
         processor_name, world_rank, world_size);

  MPI_Finalize();
}

A few notes on my setup:

  • I am using Mellanox 3.18-2 OFED distribution, as directed by the Xeon Phi "MPSS user's guide"
  • For simplicity, I'm just trying to run MPI between two PC machines (no Xeon Phi involved)
  • Program is compiled using:
    mpiicpc helloMPI.cc -o helloMPI.XEON
  • Both machines are running bare-metal installations of CentOS 7 (i.e. no VMs)

I get different errors depending on some of the I_MPI_* environment variables.

The only configuration that works without any issues is:

export I_MPI_FABRICS=tcp
export I_MPI_MIC=1   # For some reason, I need this, even though I'm not running on a Xeon Phi
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

However, if I attempt to use other settings for I_MPI_FABRICS, then I get errors and it doesn't work.
For example, for the "dapl" fabric choice (which is my eventual desired setting):

export I_MPI_FABRICS=dapl
export I_MPI_MIC=1
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

 

host: 192.168.1.111
host: 192.168.1.222

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib
    MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl
    MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man
    XDG_SESSION_ID=150
    SPARK_HOME=/home/james/clone/benchmark_spark/spark
    HOSTNAME=xen2
    SELINUX_ROLE_REQUESTED=
    INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses
    IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp
    TERM=xterm-256color
    SHELL=/bin/zsh
    I_MPI_FABRICS=dapl
    HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1
    HISTSIZE=10000
    I_MPI_MIC=1
    KVM=/home/james/phi/modules/linux/custom/kvm
    SSH_CLIENT=10.70.2.94 46738 22
    LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4
    BENCH=/home/james/clone/benchmark_spark
    SELINUX_USE_CURRENT_RANGE=
    COI=/home/james/phi/src/mpss/mpss-coi-3.8.1
    MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468
    ZSH=/home/james/.oh-my-zsh
    QT_GRAPHICSSYSTEM_CHECKED=1
    MPSS=/home/james/phi/src/mpss/mpss-3.8.1
    USER=james
    C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot
    LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    I_MPI_MPIRUN=mpirun
    MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include
    PAGER=less
    MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m
    _PHI_ROOT=/home/james/phi
    LSCOLORS=Gxfxcxdxbxegedabagacad
    _INTEL_SOURCE_ME=yes
    NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
    MAIL=/var/spool/mail/james
    PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin
    FZF_COMPLETION_TRIGGER=##
    TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb
    I_MPI_HYDRA_DEBUG=on
    PHI=/home/james/phi
    PWD=/home/james
    JAVA_HOME=/home/james/java/jdk1.8.0_111
    EDITOR=vim
    HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop
    KERN=/home/james/phi/modules/linux
    LANG=en_CA.UTF-8
    NODE_PATH=/home/james/.jsctags/lib/jsctags/:
    SELINUX_LEVEL_REQUESTED=
    DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal
    HISTCONTROL=ignoredups
    MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1
    C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root
    SHLVL=2
    HOME=/home/james
    GOROOT=/home/james/golang
    I_MPI_DEBUG=6
    PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python
    LESS=-R
    LOGNAME=james
    CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar
    SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22
    LC_CTYPE=en_CA.UTF-8
    GOPATH=/home/james/gopath
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    _PHI_MPSS_SRC=/home/james/phi/src/mpss
    CMAKE_PREFIX_PATH=/home/james/phi/build:
    XDG_RUNTIME_DIR=/run/user/1000
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    _=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
    I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8
    DAPL_NETWORK_PROCESS_NUM=2

  Intel(R) MPI Library specific variables:
  ----------------------------------------
    I_MPI_PERHOST=allcores
    I_MPI_FABRICS=dapl
    I_MPI_MIC=1
    I_MPI_MPIRUN=mpirun
    I_MPI_HYDRA_DEBUG=on
    I_MPI_DEBUG=6
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8


    Proxy information:
    *********************
      [1] proxy: 192.168.1.111 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes);

      [2] proxy: 192.168.1.222 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes);


==================================================================================================

[mpiexec@xen2] Timeout set to -1 (-1 means infinite)
[mpiexec@xen2] Got a control port string of 192.168.1.222:35852

Proxy launch args: pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id

Arguments being passed to proxy 0:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.111 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_15973_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores''LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib''MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl''MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man''XDG_SESSION_ID=150''SPARK_HOME=/home/james/clone/benchmark_spark/spark''HOSTNAME=xen2''SELINUX_ROLE_REQUESTED=''INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses''IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp''TERM=xterm-256color''SHELL=/bin/zsh''I_MPI_FABRICS=dapl''HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1''HISTSIZE=10000''I_MPI_MIC=1''KVM=/home/james/phi/modules/linux/custom/kvm''SSH_CLIENT=10.70.2.94 46738 22''LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4''BENCH=/home/james/clone/benchmark_spark''SELINUX_USE_CURRENT_RANGE=''COI=/home/james/phi/src/mpss/mpss-coi-3.8.1''MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''SSH_TTY=/dev/pts/0''VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468''ZSH=/home/james/.oh-my-zsh''QT_GRAPHICSSYSTEM_CHECKED=1''MPSS=/home/james/phi/src/mpss/mpss-3.8.1''USER=james''C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot''LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:''I_MPI_MPIRUN=mpirun''MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include''PAGER=less''MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m''_PHI_ROOT=/home/james/phi''LSCOLORS=Gxfxcxdxbxegedabagacad''_INTEL_SOURCE_ME=yes''NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N''MAIL=/var/spool/mail/james''PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin''FZF_COMPLETION_TRIGGER=##''TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb''I_MPI_HYDRA_DEBUG=on''PHI=/home/james/phi''PWD=/home/james''JAVA_HOME=/home/james/java/jdk1.8.0_111''EDITOR=vim''HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop''KERN=/home/james/phi/modules/linux''LANG=en_CA.UTF-8''NODE_PATH=/home/james/.jsctags/lib/jsctags/:''SELINUX_LEVEL_REQUESTED=''DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal''HISTCONTROL=ignoredups''MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1''C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root''SHLVL=2''HOME=/home/james''GOROOT=/home/james/golang''I_MPI_DEBUG=6''PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python''LESS=-R''LOGNAME=james''CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar''SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22''LC_CTYPE=en_CA.UTF-8''GOPATH=/home/james/gopath''LESSOPEN=||/usr/bin/lesspipe.sh %s''_PHI_MPSS_SRC=/home/james/phi/src/mpss''CMAKE_PREFIX_PATH=/home/james/phi/build:''XDG_RUNTIME_DIR=/run/user/1000''I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi''_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1''GFORTRAN_UNBUFFERED_PRECONNECTED=y''I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8''DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON

Arguments being passed to proxy 1:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.222 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_15973_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores''LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib''MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl''MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man''XDG_SESSION_ID=150''SPARK_HOME=/home/james/clone/benchmark_spark/spark''HOSTNAME=xen2''SELINUX_ROLE_REQUESTED=''INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses''IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp''TERM=xterm-256color''SHELL=/bin/zsh''I_MPI_FABRICS=dapl''HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1''HISTSIZE=10000''I_MPI_MIC=1''KVM=/home/james/phi/modules/linux/custom/kvm''SSH_CLIENT=10.70.2.94 46738 22''LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4''BENCH=/home/james/clone/benchmark_spark''SELINUX_USE_CURRENT_RANGE=''COI=/home/james/phi/src/mpss/mpss-coi-3.8.1''MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''SSH_TTY=/dev/pts/0''VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468''ZSH=/home/james/.oh-my-zsh''QT_GRAPHICSSYSTEM_CHECKED=1''MPSS=/home/james/phi/src/mpss/mpss-3.8.1''USER=james''C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot''LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:''I_MPI_MPIRUN=mpirun''MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include''PAGER=less''MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m''_PHI_ROOT=/home/james/phi''LSCOLORS=Gxfxcxdxbxegedabagacad''_INTEL_SOURCE_ME=yes''NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N''MAIL=/var/spool/mail/james''PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin''FZF_COMPLETION_TRIGGER=##''TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb''I_MPI_HYDRA_DEBUG=on''PHI=/home/james/phi''PWD=/home/james''JAVA_HOME=/home/james/java/jdk1.8.0_111''EDITOR=vim''HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop''KERN=/home/james/phi/modules/linux''LANG=en_CA.UTF-8''NODE_PATH=/home/james/.jsctags/lib/jsctags/:''SELINUX_LEVEL_REQUESTED=''DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal''HISTCONTROL=ignoredups''MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1''C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root''SHLVL=2''HOME=/home/james''GOROOT=/home/james/golang''I_MPI_DEBUG=6''PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python''LESS=-R''LOGNAME=james''CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar''SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22''LC_CTYPE=en_CA.UTF-8''GOPATH=/home/james/gopath''LESSOPEN=||/usr/bin/lesspipe.sh %s''_PHI_MPSS_SRC=/home/james/phi/src/mpss''CMAKE_PREFIX_PATH=/home/james/phi/build:''XDG_RUNTIME_DIR=/run/user/1000''I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi''_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1''GFORTRAN_UNBUFFERED_PRECONNECTED=y''I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8''DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON

[mpiexec@xen2] Launch arguments: /usr/bin/ssh -x -q 192.168.1.111 sh -c 'export I_MPI_ROOT="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi" ; export PATH="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0""$@"' pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 0
[mpiexec@xen2] Launch arguments: pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 1
[proxy:0:1@xen2] Start PMI_proxy 1
[proxy:0:1@xen2] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@xen2] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@xen2] got pmi command (from 9): get_maxes

[proxy:0:1@xen2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[proxy:0:0@xen1] Start PMI_proxy 0
[proxy:0:0@xen1] STDIN will be redirected to 1 fd(s): 9
[proxy:0:0@xen1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@xen1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@xen1] got pmi command (from 6): get_maxes

[proxy:0:0@xen1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_ranks2hosts

[proxy:0:1@xen2] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1,
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_appnum

[proxy:0:1@xen2] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@xen1] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1,
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:0@xen1] got pmi command (from 6): get_appnum

[proxy:0:0@xen1] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded optimized library
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[0] MPI startup(): dapl data transfer mode
[mpiexec@xen2] [pgid: -1] got PMI command: cmd=put kvsname=kvs_15973_0 key=P0-businesscard-0 value=rdma_port0#23708$rdma_host0#0A0000000000029EFE80000000000000E61D2DFFFE129B4000000004$arch_code#6$fabrics_list#dapl$
[proxy:0:0@xen1] got pmi command (from 6): put
kvsname=kvs_15973_0 key=P0-businesscard-0 value=rdma_port0#23708$rdma_host0#0A0000000000029EFE80000000000000E61D2DFFFE129B4000000004$arch_code#6$fabrics_list#dapl$
[proxy:0:0@xen1] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-2u
[1] MPI startup(): DAPL provider ofa-v2-mlx4_0-2u
[1] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-2
[1] MPI startup(): secondary DAPL provider ofa-v2-mcm-2
[proxy:0:1@xen2] got pmi command (from 9): put
kvsname=kvs_15973_0 key=P1-businesscard-0 value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$
[proxy:0:1@xen2] PMI response: cmd=put_result rc=0 msg=success
[1] MPI startup(): dapl data transfer mode
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: -1] got PMI command: cmd=put kvsname=kvs_15973_0 key=P1-businesscard-0 value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 9: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 9: cmd=barrier_out
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): get
kvsname=kvs_15973_0 key=P1-businesscard-0
[proxy:0:0@xen1] PMI response: cmd=get_result rc=0 msg=success value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$
xen1:UCM:5c9c:194f1740: 21010 us(21010 us):  create_ah: ERR Success
xen1:UCM:5c9c:194f1740: 21019 us(9 us): UCM connect: snd ERR -> cm_lid 0 cm_qpn 327 r_psp 3e6e p_sz=24
libibverbs: GRH is mandatory For RoCE address handle
[0:192.168.1.111][../../src/mpid/ch3/channels/nemesis/netmod/dapl/dapl_conn_rc.c:247] error(0x30000): ofa-v2-mlx4_0-1u: could not connect DAPL endpoints: DAT_INSUFFICIENT_RESOURCES()
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=abort exitcode=68729104
Fatal error in MPI_Init: Internal MPI error!, error stack:
MPIR_Init_thread(805).................: fail failed
MPID_Init(1831).......................: channel initialization failed
MPIDI_CH3_Init(147)...................: fail failed
dapl_rc_setup_all_connections_20(1434): generic failure with errno = 16
(unknown)(): Internal MPI error!
[proxy:0:0@xen1] got pmi command (from 6): abort
exitcode=68729104
[proxy:0:0@xen1] we don't understand this command abort; forwarding upstream

 

 

The "ofa" fabric also does not work, although it has a different error message:

export I_MPI_FABRICS=ofa
export I_MPI_MIC=1
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

 

 

host: 192.168.1.111
host: 192.168.1.222

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib
    MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl
    MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man
    XDG_SESSION_ID=150
    SPARK_HOME=/home/james/clone/benchmark_spark/spark
    HOSTNAME=xen2
    SELINUX_ROLE_REQUESTED=
    INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses
    IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp
    TERM=xterm-256color
    SHELL=/bin/zsh
    I_MPI_FABRICS=ofa
    HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1
    HISTSIZE=10000
    I_MPI_MIC=1
    KVM=/home/james/phi/modules/linux/custom/kvm
    SSH_CLIENT=10.70.2.94 46738 22
    LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4
    BENCH=/home/james/clone/benchmark_spark
    SELINUX_USE_CURRENT_RANGE=
    COI=/home/james/phi/src/mpss/mpss-coi-3.8.1
    MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468
    ZSH=/home/james/.oh-my-zsh
    QT_GRAPHICSSYSTEM_CHECKED=1
    MPSS=/home/james/phi/src/mpss/mpss-3.8.1
    USER=james
    C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot
    LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    I_MPI_MPIRUN=mpirun
    MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include
    PAGER=less
    MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m
    _PHI_ROOT=/home/james/phi
    LSCOLORS=Gxfxcxdxbxegedabagacad
    _INTEL_SOURCE_ME=yes
    NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
    MAIL=/var/spool/mail/james
    PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin
    FZF_COMPLETION_TRIGGER=##
    TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb
    I_MPI_HYDRA_DEBUG=on
    PHI=/home/james/phi
    PWD=/home/james
    JAVA_HOME=/home/james/java/jdk1.8.0_111
    EDITOR=vim
    HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop
    KERN=/home/james/phi/modules/linux
    LANG=en_CA.UTF-8
    NODE_PATH=/home/james/.jsctags/lib/jsctags/:
    SELINUX_LEVEL_REQUESTED=
    DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal
    HISTCONTROL=ignoredups
    MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1
    C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root
    SHLVL=2
    HOME=/home/james
    GOROOT=/home/james/golang
    I_MPI_DEBUG=6
    PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python
    LESS=-R
    LOGNAME=james
    CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar
    SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22
    LC_CTYPE=en_CA.UTF-8
    GOPATH=/home/james/gopath
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    _PHI_MPSS_SRC=/home/james/phi/src/mpss
    CMAKE_PREFIX_PATH=/home/james/phi/build:
    XDG_RUNTIME_DIR=/run/user/1000
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    _=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
    I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8
    DAPL_NETWORK_PROCESS_NUM=2

  Intel(R) MPI Library specific variables:
  ----------------------------------------
    I_MPI_PERHOST=allcores
    I_MPI_FABRICS=ofa
    I_MPI_MIC=1
    I_MPI_MPIRUN=mpirun
    I_MPI_HYDRA_DEBUG=on
    I_MPI_DEBUG=6
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8


    Proxy information:
    *********************
      [1] proxy: 192.168.1.111 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes);

      [2] proxy: 192.168.1.222 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes);


==================================================================================================

[mpiexec@xen2] Timeout set to -1 (-1 means infinite)
[mpiexec@xen2] Got a control port string of 192.168.1.222:36282

Proxy launch args: pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id

Arguments being passed to proxy 0:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.111 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_16060_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores''LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib''MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl''MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man''XDG_SESSION_ID=150''SPARK_HOME=/home/james/clone/benchmark_spark/spark''HOSTNAME=xen2''SELINUX_ROLE_REQUESTED=''INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses''IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp''TERM=xterm-256color''SHELL=/bin/zsh''I_MPI_FABRICS=ofa''HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1''HISTSIZE=10000''I_MPI_MIC=1''KVM=/home/james/phi/modules/linux/custom/kvm''SSH_CLIENT=10.70.2.94 46738 22''LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4''BENCH=/home/james/clone/benchmark_spark''SELINUX_USE_CURRENT_RANGE=''COI=/home/james/phi/src/mpss/mpss-coi-3.8.1''MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''SSH_TTY=/dev/pts/0''VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468''ZSH=/home/james/.oh-my-zsh''QT_GRAPHICSSYSTEM_CHECKED=1''MPSS=/home/james/phi/src/mpss/mpss-3.8.1''USER=james''C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot''LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:''I_MPI_MPIRUN=mpirun''MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include''PAGER=less''MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m''_PHI_ROOT=/home/james/phi''LSCOLORS=Gxfxcxdxbxegedabagacad''_INTEL_SOURCE_ME=yes''NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N''MAIL=/var/spool/mail/james''PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin''FZF_COMPLETION_TRIGGER=##''TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb''I_MPI_HYDRA_DEBUG=on''PHI=/home/james/phi''PWD=/home/james''JAVA_HOME=/home/james/java/jdk1.8.0_111''EDITOR=vim''HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop''KERN=/home/james/phi/modules/linux''LANG=en_CA.UTF-8''NODE_PATH=/home/james/.jsctags/lib/jsctags/:''SELINUX_LEVEL_REQUESTED=''DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal''HISTCONTROL=ignoredups''MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1''C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root''SHLVL=2''HOME=/home/james''GOROOT=/home/james/golang''I_MPI_DEBUG=6''PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python''LESS=-R''LOGNAME=james''CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar''SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22''LC_CTYPE=en_CA.UTF-8''GOPATH=/home/james/gopath''LESSOPEN=||/usr/bin/lesspipe.sh %s''_PHI_MPSS_SRC=/home/james/phi/src/mpss''CMAKE_PREFIX_PATH=/home/james/phi/build:''XDG_RUNTIME_DIR=/run/user/1000''I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi''_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1''GFORTRAN_UNBUFFERED_PRECONNECTED=y''I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8''DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON

Arguments being passed to proxy 1:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.222 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_16060_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores''LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib''MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl''MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man''XDG_SESSION_ID=150''SPARK_HOME=/home/james/clone/benchmark_spark/spark''HOSTNAME=xen2''SELINUX_ROLE_REQUESTED=''INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses''IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp''TERM=xterm-256color''SHELL=/bin/zsh''I_MPI_FABRICS=ofa''HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1''HISTSIZE=10000''I_MPI_MIC=1''KVM=/home/james/phi/modules/linux/custom/kvm''SSH_CLIENT=10.70.2.94 46738 22''LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4''BENCH=/home/james/clone/benchmark_spark''SELINUX_USE_CURRENT_RANGE=''COI=/home/james/phi/src/mpss/mpss-coi-3.8.1''MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''SSH_TTY=/dev/pts/0''VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468''ZSH=/home/james/.oh-my-zsh''QT_GRAPHICSSYSTEM_CHECKED=1''MPSS=/home/james/phi/src/mpss/mpss-3.8.1''USER=james''C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot''LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:''I_MPI_MPIRUN=mpirun''MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic''CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include''PAGER=less''MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m''_PHI_ROOT=/home/james/phi''LSCOLORS=Gxfxcxdxbxegedabagacad''_INTEL_SOURCE_ME=yes''NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N''MAIL=/var/spool/mail/james''PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin''FZF_COMPLETION_TRIGGER=##''TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb''I_MPI_HYDRA_DEBUG=on''PHI=/home/james/phi''PWD=/home/james''JAVA_HOME=/home/james/java/jdk1.8.0_111''EDITOR=vim''HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop''KERN=/home/james/phi/modules/linux''LANG=en_CA.UTF-8''NODE_PATH=/home/james/.jsctags/lib/jsctags/:''SELINUX_LEVEL_REQUESTED=''DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal''HISTCONTROL=ignoredups''MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1''C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root''SHLVL=2''HOME=/home/james''GOROOT=/home/james/golang''I_MPI_DEBUG=6''PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python''LESS=-R''LOGNAME=james''CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar''SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22''LC_CTYPE=en_CA.UTF-8''GOPATH=/home/james/gopath''LESSOPEN=||/usr/bin/lesspipe.sh %s''_PHI_MPSS_SRC=/home/james/phi/src/mpss''CMAKE_PREFIX_PATH=/home/james/phi/build:''XDG_RUNTIME_DIR=/run/user/1000''I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi''_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1''GFORTRAN_UNBUFFERED_PRECONNECTED=y''I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8''DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON

[mpiexec@xen2] Launch arguments: /usr/bin/ssh -x -q 192.168.1.111 sh -c 'export I_MPI_ROOT="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi" ; export PATH="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0""$@"' pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 0
[mpiexec@xen2] Launch arguments: pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 1
[proxy:0:1@xen2] Start PMI_proxy 1
[proxy:0:1@xen2] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@xen2] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@xen2] got pmi command (from 9): get_maxes

[proxy:0:1@xen2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:0@xen1] Start PMI_proxy 0
[proxy:0:0@xen1] STDIN will be redirected to 1 fd(s): 9
[proxy:0:0@xen1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@xen1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@xen1] got pmi command (from 6): get_maxes

[proxy:0:0@xen1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_ranks2hosts

[proxy:0:1@xen2] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1,
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_appnum

[proxy:0:1@xen2] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@xen1] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1,
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[proxy:0:0@xen1] got pmi command (from 6): get_appnum

[proxy:0:0@xen1] PMI response: cmd=appnum appnum=0
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded optimized library
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[1] MPI startup(): Found 1 IB devices
[1] MPI startup(): Open 0 IB device: mlx4_0
[proxy:0:1@xen2] got pmi command (from 9): put
kvsname=kvs_16060_0 key=OFA_Init_fail value=1
[proxy:0:1@xen2] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@xen2] forwarding command (cmd=put kvsname=kvs_16060_0 key=OFA_Init_fail value=1) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=put kvsname=kvs_16060_0 key=OFA_Init_fail value=1
[1] MPI startup(): ofa fabric is not available and fallback fabric is not enabled

I don't have access to the Intel MPI source code, so sadly I cannot figure out where exactly these errors are manifesting.

Let me know if there's any other information I can provide.
Cheers,
James

Thread Topic: 

Help Me
↧
Search
Viewing all articles
Browse latest Browse all 927

Trending Articles

Practice Sheet of Right form of verbs for HSC Students

September 22, 2019, 11:40 pm

Download: FK ft Shenky – Nakuyewa ”Prod by: Shenky”

February 16, 2017, 4:24 pm

How to win at Markstrat (Markstrat Tips and Tricks) – Vodites

January 5, 2014, 10:34 pm

Ominde Commission Report and Recommendations – Ominde Report of 1964

March 16, 2015, 5:14 am

Bureau of Internal Revenue: Regional Offices (Directory)

January 9, 2014, 11:06 pm

GO 53 on Enhancement of Ex-gratia upto 5 Lakhs Toddy Tappers in Telangana

March 26, 2017, 11:23 pm

Cakewalk CA-2A Leveling Amplifier v2.0.1.97 WiN, v2.0.1.96 OSX Incl Keygen

October 17, 2016, 7:20 am

Mp3 Download: Mdu - Kunjenjenjena

December 7, 2017, 8:16 am

How the kill the job , when DTP request running for long hours.

July 26, 2013, 2:41 am

Microsoft Intune から展開しているアプリのアップデートについて

October 17, 2016, 4:11 am

18-year-old girl was beaten for half an hour by two Northampton men in 'an...

September 1, 2017, 10:00 pm

Car crash in Dunton Bassett leaves driver in critical condition

October 7, 2014, 7:51 am

Macky 2, Two Others In Road Accident

March 29, 2015, 5:34 am

Application log 00000000000000089514: Could not convert queue DLVST90CLNT

May 14, 2015, 11:27 pm

Detroit mafia: D’Anna Brothers agree to plea deal

April 21, 2016, 6:56 am

Delivery block field greyed out using VA02

January 26, 2016, 2:52 pm

Muloraki Au

June 22, 2016, 1:44 am

【個人撮影】スマホのプライベート映像♪「中に出さないで///」カラオケ屋での生ハメ撮りが流出ｗ【リベンジポルノ】＠PornHub

October 12, 2017, 2:23 pm

BREAKING NEWS: Diamond Platnumz Is Reported Dead After Ghastly Car Accident

February 9, 2018, 4:56 am

FIAT 500 B0111 B0112

July 5, 2018, 10:31 am
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>