MPI_File_write_all / read_all collective buffering does not scale for 4D block-decomposed subarray views (Lustre): 6x slower than a user-level MPI_Alltoallv + POSIX transposition

[paboyle]

MPI_File_write_all / read_all collective buffering does not scale for 4D block-decomposed subarray views (Lustre): 6x slower than a user-level MPI_Alltoallv + POSIX transposition

Summary

For the canonical "N-D distributed array written in global lexicographic order" pattern —
MPI_Type_create_subarray + MPI_File_set_view + MPI_File_write_all — collective
buffering plateaus at ~30 GB/s from 64 nodes upward on Frontier's Orion Lustre, and does
not improve with client count. Both MPI_File_write_all and MPI_File_read_all are
affected; reads are worse, flat at 30–33 GB/s while POSIX pread of the same file on the
same ranks reaches 375 GB/s.

We replaced the MPI-IO collective with an explicit user-level two-phase implementation
(MPI_Alltoallv inside a row subcommunicator, followed by independent POSIX
lseek+write of a handful of large extents per rank) and obtained 6.0x on write and
8.4x on read at 128 nodes / 1024 ranks on the identical file layout, decomposition, and
byte ordering. This is in production in the Grid lattice QCD library as of commit
41f5a022047049ce61d4761897810bf8620470f5 and is now the default path
(d68b111d0645a0254078d1c33bf7a73079cbf0ac).

We would rather delete our workaround than maintain it, hence this report.
It would also benefit everyone if default MPI behavior is higher performance as this is a common pattern.

Environment

Machine	OLCF Frontier
MPI	Cray MPICH from cpe/26.03 (compilers cce/21.0.0, rocm/7.2.0)
Filesystem	Lustre (Orion), directory default layout — no explicit lfs setstripe
Ranks	8 per node (1 per GCD), 4 → 128 nodes
Hints	MPI_INFO_NULL throughout (no cb_nodes, cb_buffer_size, romio_cb_write set)
Application	Grid, https://github.com/paboyle/Grid, branch develop

The access pattern

A 4D lattice of sizeof(fobj) = 576 byte site objects is block-decomposed over a 4D
process grid. The file must be in global lexicographic order so that it is independent
of the decomposition used to write it — a different job with a different process grid
must be able to read it. This is exactly the case MPI_Type_create_subarray exists for:

/* fobj = 72 doubles per lattice site */
MPI_Type_contiguous(numword, MPI_DOUBLE, &mpiObject);
MPI_Type_commit(&mpiObject);

/* file view: my block inside the global lattice */
MPI_Type_create_subarray(ndim, gLattice, lLattice, gStart,
                         MPI_ORDER_FORTRAN, mpiObject, &fileArray);
MPI_Type_commit(&fileArray);

/* memory: the whole local block, contiguous */
MPI_Type_create_subarray(ndim, lLattice, lLattice, lStart,
                         MPI_ORDER_FORTRAN, mpiObject, &localArray);
MPI_Type_commit(&localArray);

MPI_File_open(comm, file, MPI_MODE_RDWR | MPI_MODE_CREATE, MPI_INFO_NULL, &fh);
MPI_File_set_view(fh, disp, mpiObject, fileArray, "native", MPI_INFO_NULL);
MPI_File_write_all(fh, iodata, 1, localArray, &status);   /* <-- plateaus */

(Grid/parallelIO/BinaryIO.h, lines ~723–870.)

Concretely, at the 128-node point of the scan below:

• global lattice 32.32.256.1024, process grid 4.4.8.8, local block 8.8.32.128
• 151 MB per rank, 154.6 GB per record, 1024 ranks
• the file view gives each rank 32768 non-contiguous runs of 4608 bytes
  (lLattice[0] * 576), strided through a 154.6 GB file

Small contiguous runs are intrinsic to the pattern, not a pathological choice: the
fastest-varying local extent is what it is. Rearranging them into large contiguous
extents is precisely the job of two-phase collective buffering.

Observed behaviour

Weak scan, local volume and therefore the file-view structure held identical at every
rung (151 MB/rank, 4608 B contiguous runs); only the number of Lustre clients changes.
Best of 3 repetitions, MB/s. raw is a non-lexicographic control in which each rank
writes one disjoint contiguous segment with POSIX — the zero-transposition upper bound.

Write

nodes	ranks	record	POSIX raw (control)	MPI_File_write_all	user-level two-phase
4	32	4.8 GB	6232	6483	6743
8	64	9.7 GB	9681	12013	13605
16	128	19.3 GB	19351	17618	26671
32	256	38.6 GB	37634	27808	50962
64	512	77.3 GB	73213	29539	103769
128	1024	154.6 GB	97673	31675	189535

Read

nodes	ranks	POSIX raw (control)	MPI_File_read_all	user-level two-phase
4	32	26630	20040	17918
8	64	24986	28018	32246
16	128	52049	25693	59576
32	256	91583	36134	98925
64	512	199053	30495	209587
128	1024	375135	33419	279106

MPI-IO tracks the other two paths up to ~16 nodes and then stops scaling entirely. The
raw control shows the filesystem and the clients still have headroom of an order of
magnitude at that point. The read plateau is the more striking of the two: the file was
written by the same ranks moments earlier, so the client page cache is warm, and the
POSIX paths see it (375 GB/s) while MPI_File_read_all does not (33 GB/s).

Interaction with Lustre striping

Repeating the 128-node point with lfs setstripe -c -1 -S 8M on the directory:

	default layout	-c -1 -S 8M
POSIX raw	97673	81728
MPI_File_write_all	31675	77534
user-level two-phase	189535	86063

Maximum-width striping is what rescues the collective, 32 → 78 GB/s. It costs every
other path a factor of 1.2–2.2. So the collective's performance is strongly coupled to a
file layout that has to be chosen out of band, by the user, before the file is created,
and which is the wrong layout for everything else. Users who have "tuned MPI-IO" on
Lustre are, we suspect, mostly discovering this.

The workaround

We do the transposition ourselves. The key observation is that un-splitting the k
fastest dimensions of the process grid makes each rank's file contribution contiguous
in large pieces: rank r in a row subcommunicator of size R ends up owning whole
extents of prod_{d<k} gLattice[d] * lLattice[k] sites. k is chosen as the smallest
value that reaches a target extent size (default 4 MB). The row index is monotone in the
local lexicographic index, so the send buffer is the untouched local array and only one
MPI_Alltoallv is required:

/* phase 1: redistribute within the row subcommunicator.
   sendcounts/senddispls derived analytically from the plan; iodata untouched. */
MPI_Alltoallv(iodata,     sendcounts, senddispls, MPI_BYTE,
              aggregated, recvcounts, recvdispls, MPI_BYTE, rowcomm);

/* phase 2: a handful of large contiguous writes per rank, no MPI-IO involved */
for (e = 0; e < nextent; e++) {
    lseek(fd, offset + extentGsite[e]*sizeof(fobj), SEEK_SET);
    write(fd, aggregated + extentLocal[e], extentSites[e]*sizeof(fobj));
}

At the 128-node point this gives 8 extents of 18 MB per rank instead of 32768 runs of
4608 B, at a memory cost of 144 MB/rank. Measured stage breakdown for the 154.6 GB
record (max over ranks):

aggregate exchange (write) stages: alloc 0.035 s  permute 0.027 s  alltoallv 0.062 s
aggregate write stages: create 0.0018  barrier 0.0043  open 0.020
                        seek+write 1.246  close 0.0009  skew 0.654  truncate 0.0043

The entire redistribution — an all-to-all of 154.6 GB of payload — costs 62 ms, i.e.
4% of the 1.49 s the record takes end to end. Whatever MPI_File_write_all is spending
its time on, it is not taking the most efficient approach for this pattern.

That is really the substance of the report. An optimal two-phase implementation of this
pattern should cost the network exchange plus the streaming write, and the network
exchange is nearly free and substantially better I/O is obtainable.

Questions / requests

1. Is the ROMIO Lustre ADIO driver's aggregator selection and cb_buffer_size policy
   expected to plateau this way for MPI_ORDER_FORTRAN subarray views with few-KB
   contiguous runs? Is there a hint combination that recovers the missing 6x on the
   default layout — we tried none, deliberately, because out-of-the-box behaviour is
   what our users get.
2. Is the strong dependence on lfs setstripe -c -1 intended? If the collective needs
   wide striping, could ROMIO set the layout itself at MPI_MODE_CREATE, or at least
   warn?

Methodology caveats, stated up front

• No fsync / MPI_File_sync anywhere. All three paths are measured identically —
  open, write, close — so the comparison is fair, but the absolute write figures include
  whatever the client page cache absorbs. Compute nodes have 512 GB and the record is
  1.2 GB/node, so absorption is possible at the small rungs.
• Reads are warm, for all three paths equally, for the reason given above.
• Best-of-3 is quoted. Lustre variability and contention from other jobs are real;
  the qualitative result (MPI-IO flat from 64 nodes, the other two still climbing)
  is reproducible across jobs and is visible in the means as well as the bests.
• Correctness is cross-checked: the aggregate path is verified byte-identical to the
  MPI-IO path, and all three paths agree on a layout-independent checksum computed from
  the global site index.

Reproducer

Grid, https://github.com/paboyle/Grid, branch develop:

commit	content
41f5a022047049ce61d4761897810bf8620470f5	aggregation implementation, MPI_Alltoallv in the communicator layer
d68b111d0645a0254078d1c33bf7a73079cbf0ac	made the aggregate path the default
1f55c13bc4fc84a1959b09b196f884661980e2a4	Frontier SLURM driver for the scan above

• Grid/parallelIO/BinaryIO.h — IOobject() holds both paths, selected by the
  BINARYIO_AGGREGATE control bit; BuildAggregationPlan() is the plan; the MPI-IO path
  is the !aggregate branch quoted above.
• tests/IO/Test_aggregate_io.cc — standalone benchmark; times write and read for the
  POSIX control, MPI-IO and the aggregate path, with correctness checking. Options
  --aggregate-target <bytes>, --io-reps <n>, --io-no-correctness, --io-read-only.
• systems/Frontier/aggregate_io_scaling.slurm — the exact weak scan whose numbers are
  tabulated here; systems/Frontier/config-command is the build.

Raw job output for the scan is available on request (OLCF job 5234336).

• A simpler implementation that is standalone to MPI and C++ could be developed upon request if this is not all clear.

[paboyle]

Full disclosure, I made use of Claude developing the patch to grid, fully tested it, and got my Claude session to summarize the output. Claude has developed a standalone reproducer in pure MPI + C++ without my library that is untested at present, but could be finalized.

[paboyle]

Finally, I realize this is Cray MPICH so a forked version of ROMIO. As far as I have seen this has been a persistent issue across many version of MPI and many versions of ROMIO. I could, of course, conduct the comparisons on Aurora and a new variant MPICH if that is more helpful.

[wkliao]

Is the ROMIO Lustre ADIO driver's aggregator selection and cb_buffer_size policy
expected to plateau this way for MPI_ORDER_FORTRAN subarray views with few-KB
contiguous runs? Is there a hint combination that recovers the missing 6x on the
default layout — we tried none, deliberately, because out-of-the-box behaviour is
what our users get.

In ROMIO's Lustre ADIO driver, hint cb_buffer_size is always set to the Lustre file stripping size.
Thus, adjusting it will take no effect. There may be a few places inside of ROMIO that handle
memory copy of user buffers can be improved, (well, based on my experience).

Is the strong dependence on lfs setstripe -c -1 intended? If the collective needs
wide striping, could ROMIO set the layout itself at MPI_MODE_CREATE, or at least
warn?

The default Lustre file striping setting of Frontier is using Progressive File Layout (PFL),
which is not the best for the I/O pattern used in your test case. Instead of using the
entire OSTs available on the system, can you try setting the striping count to be equal
to the number of compute nodes. It may perform better. This can be done by setting hint
"striping_factor" into an MPI info object and passing it to MPI_File_open when creating
a new file.

[wkliao]

/* memory: the whole local block, contiguous */
MPI_Type_create_subarray(ndim, lLattice, lLattice, lStart, MPI_ORDER_FORTRAN, mpiObject, &localArray);

Just FYI, If the write buffer is contiguous in memory, then it is not necessary to create an MPI derived datatype.
But there is also nothing wrong of doing that either.

[wkliao]

"Warm read" usually cannot be used in a fair performance comparison, and it can be avoided by
first duplicating the input files into several copies, and then each job reads a different one.