The matched MPI Send/Recv pairs are mixed up

Hi,

Each MPI process has dedicated thread for each send/receive operation (I use MPI_Send/MPI_Recv calls). Because the data size is too big, I made each thread iterate over a for loop sending fixed sized data (a chunk) at a time.

My Expectation:

 SendThread (rank 0)    ReceiveThread (rank 1)
         chunk[i]    ->       chunk[i]
         chunk[i+1]  ->       chunk[i+1]
                     ...

Actually what happened is that:

 SendThread (rank 0)    ReceiveThread (rank 1)
         chunk[i]    ->       chunk[i+1]
         chunk[i+1]  ->       chunk[i]
                     ...

The data of chunk[i] on the sender side has sent to

How can this happen? AFAIK, MPI_Send/Recv call is blocking call that can not progress unless there is any matched call (Send <---> Recv).

Does it mean that Sending chunk[i] has been matched with Receiving chunk[i], but when the data is actually being delivered they got screwed up?

Is it expected behavior? thanks,