Install MPI ABI into separate prefix

[eschnett]

MPICH (head of the main branch) installs the header and library file for the new MPI ABI into the same prefix as the standard MPICH header and library files. This makes it difficult to configure other libraries or applications to build with the new MPI ABI. It would be preferable to either have separate prefixes, or to install only either the MPI ABI or the MPICH ABI but not both together.

In MPICH's current setup, one needs to pass the flag -DMPI_ABI at compile time to select the new ABI (or using mpicc_abi instead of mpicc). These are new mechanisms, and neither autoconf nor cmake know about this. Many existing packages have (for better or worse) their own hand-rolled configure system (e.g. PETSc), and they also do not know about this. This makes it very easy to accidentally compile or link against the wrong ABI. Unix is know to be abysmally bad at detecting such inconsistencies.

Having separate install directories would eliminate this risk. For example, installing all the files for the MPI ABI into $prefix/mpiabi (include/mpi.h, lib/libmpi_abi.so, and bin/mpicc) would do so.

[hzhou]

...or to install only either the MPI ABI or the MPICH ABI but not both together.

In the next release we will only install a single ABI.

[dalcinl]

This makes it difficult to configure other libraries or applications to build with the new MPI ABI.

How so? You either:

a) Use the mpicc_abi compiler wrapper.
b) Use mpicc -mpi-abi ... or mpicc -mpi_abi ..., that is, pass an extra argument to mpicc to compile/link with the new ABI
c) Sort of a hack, but you can use any compiler, passing a cpp flag -DMPI_ABI plus include/library dirs and -lmpi_abi.

Or, for maximum ABI compatibility of C-only code... you build your code with the MPI ABI stubs https://github.com/mpi-forum/mpi-abi-stubs. Then you can use either MPICH or Open MPI (main branch) or Intel MPI (latest release) at runtime. This workflow has been validated thougly with mpi4py. There is an open merge request for PETSc/petsc4py that confirms that the approach also works there [link].

In the next release we will only install a single ABI.

Which is just great. In the meantime, users have many options that I just elaborated.