Using a small cluster of Skylake-based systems, as below:
- 2019 beta update 1
- Red Hat 7.6 beta x86_64 (3.10.0-938.el7.x86_64)
- Systems include Intel Omni-Path HFAs in addition to an onboard gigabit Ethernet nic
- Systems are using the the RH7.6 inbox Omni-Path support
Attempting to run the included IMB-MPI1 binary over the OPA HFAs specifying psm2 as the transport appears to work correctly. However, trying to run it across the onboard Ethernet network specifying tcp as the transport generates the below message from MPI startup:
MPI startup(): tcp fabric is unknown or has been removed from the product, please use ofi or shm:ofi instead
The job does execute, but over the OPA fabric instead of the Ethernet network. If the OPA HFA is disconnected, the job fails.
fi_info -l
psm2:
version: 1.6
ofi_rxm:
version: 1.0
sockets:
version: 2.0