We use Intel MPI by default on our clusters, and we've recently run into a problem which, I think, is due to some problem with the way it passes environment variables through to the tasks.
If you export a shell function in bash:
function hw() { echo "Hello world"; } export -f hw
That is available to child processes, and can be seen with env:
$ env | grep -A1 hw BASH_FUNC_hw()=() { echo 'Hello world!' }
Using `mpirun` with this set, without any modifications to the environment, gives the following error:
bash: hw: line 1: syntax error: unexpected end of file bash: error importing function definition for `BASH_FUNC_hw'
We see this in 2018.3 and 2019.4, the most recent versions we have installed on our clusters. I tried to get a better handle on what it was doing with strace but I couldn't manage to find how it was going wrong -- like I said above, I'm guessing the way the environment variables are passed through wasn't written with these functions in mind and can't handle them.
Has anyone else seen this problem? I'd guess the ability to export shell functions doesn't get much use, but the module command in Environment Modules 4.4.x does it, so I would have thought it would come up elsewhere.