I am using Intel trace collector. For small simulations it works fine but for large runs I get following error:
[20] Intel(R) Trace Collector ERROR: Failed writing buffer to flush file "/tmp/VT-flush-blueDetector_scalasca_itac_x86.rts_0020-008123.dat": No space left on device
From Intel documentation I see that by default traces are written to /tmp and we are supposed to set VT_LOGFILE_PREFIX. But even if I set this environmental variable to directory under lustre file system and pass -x option for mpiexec, I still get the same error.
$ export VT_LOGFILE_PREFIX=/lustre/jhome7/jicg41/jicg4110/some_dir_path
$ LD_PRELOAD=/usr/local/intel/itac/8.1.2.033/itac/slib_impi4/libVT.so mpiexec -x -trace -np 48 ./app_exe
Note:
- with above settings, only first file i.e. app__itac_x86.rts.prot is written to VT_LOGFILE_PREFIX directory
- I am sure that -x option exports all env variables to all mpi processes, I have tested this
Am I missing something?