More error checking in `MPI_Neighbour_*` functions?

[eschnett]

In this PR JuliaParallel/MPI.jl#958 we encounter a segfault in MPICH's MPI_Neighbour_alltoall. The reason seems to be that we are passing MPI_IN_PLACE as argument, and that MPICH does not handle this argument (because it is not required by the standard). It would be convenient for us if there was an explicit error check in MPICH that detected this error.

(The MPI standard doesn't say that MPI_IN_PLACE is unsupported, it simply doesn't mention this constant at all in the MPI_Neighbour_* definitions, and this makes it difficult to realize that this mechanism is indeed unsupported.)

[hzhou]

Depend on a topology, the Neighbor_alltoall may not send its own data to itself, right? That will make MPI_IN_PLACE meaningless. I guess for the topology that do contain self edge, MPI_IN_PLACE can work. But even in that case, the topology is not explicit (such as MPI_Alltoallv where all displacements are explicit in the parameters). All the other use case of MPI_IN_PLACE are explicit rather than depending on a implicit topology.

Anyway, until the forum make an update, we'll treat this as error. I'll double check how we handle it.

[eschnett]

MPI_IN_PLACE is unrelated to whether a processor sends data to itself. It only means that the received data (from other processors) are to be stored in the same buffer that holds the send data. However, I don't think the standard allows MPI_IN_PLACE here, to MPICH is right in not handling it, it would only be convenient if there was an explicit error.