I am using the Trace Analyzer for an MPI job running on 4 nodes (80 physical cores total, 80 MPI threads).
When I run 'mpirun -trace ...' the job takes roughly 10 times longer than the same job running without tracing because processes are being suspended when trace routines dump data from memory to disk.
My goal is to identify the relative share of time spent in most active MPI routines but, In this situation, how much one can trust timing statistics displayed by the Trace Analyzer? Isn't it possible that a process starts an MPI_SEND when the target process is suspended by the tracer, so that the transfer does not start until the tracer in the target process has completed the dump and all this artificial wait time is added by the Trace Analyzer to the time spent in the MPI_SEND call?
Thank you for your attention.