bodner but good - #86
Draft
JorgeG94 wants to merge 11 commits into
Draft
Conversation
Rewrite of mixedlayer_restrat_Bodner as do concurrent loops inside a single persistent device data region, replacing the earlier naive port. * The mixed-layer density integral evaluates the equation of state a block of MLE_NKBLOCK layers at a time rather than hoisting it to a whole-column 3D call. The block scratch is (SZI,SZJ,nkblock), so the CPU default of 1 keeps it to a single slab; the GPU default of 0 puts the column in one block. The row-at-a-time early exit becomes a slab-wide one, which only changes work that the htot < big_H guard already discarded. * No shadow copies of CS/GV/US scalars or of the input fields. The kernels read CS% and tv% directly, with those parents mapped alongside their components as nvfortran requires once any component is mapped. * One target data region covers the whole calculation, so nothing round-trips between phases. Arrays that only one branch or diagnostic touches (the block scratch, uhml/vhml, the host-side big_H and wpup variants, the chksums) are mapped or synced inside that branch. * dmu is stashed in uhml/vhml instead of a per-column array, which do concurrent cannot give each iteration without a per-thread allocation. mu is now pure so it can be called from a concurrent loop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
h_MLD, BLD and bflux were left to implicit copyin, which the mapped-component switch disables along with derived-type copyin, so the kernels read uninitialized device memory and little_h/wpup came back NaN. BLD and bflux are mapped around the w'u' branches that dereference them; h_MLD joins the outer region. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mapping a derived-type component makes nvfortran create a parent entry that supersedes an explicit map(to: CS) in the same directive. The component descriptors still attach -- CS%Cr_space read correctly -- but the struct body is never copied, so CS%mstar, CS%nstar and CS%min_wstar2 silently read as zero. wpup then collapsed to exactly zero, r_wpup went infinite, and psi_mag came out 0*Inf = NaN. Mapping the parents in their own enclosing region copies the struct body. benchmark_ALE now reproduces repro_ocean.stats bitwise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With MLE_NKBLOCK=1 the equation-of-state scratch was still a full (SZI,SZJ) slab, where upstream used a single row rho_ml(SZI). That loses the locality the original had, where the EOS fills a row and the integral consumes it straight back out of cache. Add MLE_NJBLOCK alongside MLE_NKBLOCK, following PGF_PLM_NJBLOCK: 1 and 1 on the CPU reduce the scratch to the one row upstream used, 0 and 0 on the GPU make it the whole halo-1 j domain by the whole column, which is a single kernel. Rml_int goes back to being the 1D host array upstream declares, with the tiled path carrying its running totals in Rml_blk. The early-exit test compares htot against big_H on the host, so big_H is fetched once when it can actually run. Without that it read a stale big_H, concluded every column was full and stopped calling the equation of state early, which truncated the integral -- benchmark_ALE energy moved in the 7th digit at NJBLOCK=NKBLOCK=1 while the untiled default stayed bitwise. benchmark_ALE reproduces repro_ocean.stats bitwise at both 0/0 and 1/1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dropping it reproduces the zeroed-scalar NaN even with the parent in its own enclosing region. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous form leaned on `map(always, to: CS, tv)` in an enclosing structured region, a construct used nowhere else in MOM6. Replace it with what the rest of the GPU port does: unstructured `target enter data`, the control structure's array components mapped once in the init routine as hor_visc_init and friends do, and no structured `target data` regions anywhere. CS's scalars cannot be mapped alongside its arrays. Mapping any component of a derived type costs that type its implicit copyin, after which its scalars read as zero on the device while its array components still read correctly -- silent, and it surfaces only as a NaN much further downstream. So the eight scalars the kernels need are hoisted to plain locals, which is the practice the rest of the port already follows. Leaving the arrays unmapped instead is not an option: that faults with an illegal address on the first CS%MLD_filtered_slow read. benchmark_ALE reproduces repro_ocean.stats bitwise at MLE_NJBLOCK/NKBLOCK of both 0/0 and 1/1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The solo driver puts the whole MOM_control_struct on the device with `enter data map(alloc: MOM_CSp)`: the address range is claimed but its contents are never copied. Any later map(to:) of anything inside that range -- including the implicit copyin every kernel gets -- is then a present-table no-op, so kernels read zeros for the scalars of any control structure held as an inline member. Mapped array components were unaffected because their maps copy the heap data and attach the descriptor into the parent's device block; scalars have no attach equivalent. This was the mechanism behind the l_*/tau_* hoists. `target update` always copies regardless of presence, so issue `target update to(CS)` in mixedlayer_restrat_init once the parameters are set, BEFORE the component maps whose attach then fixes this struct's array descriptors in the device copy. With that, delete the eight hoisted locals and read CS% directly in the kernels, like every other module in the tree. Verified bitwise against repro_ocean.stats at both MLE_NJBLOCK/NKBLOCK 0/0 and 1/1 (overrides confirmed applied in MOM_parameter_doc.all). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GPU Port CoverageOverall: 2129 / 5675 portable executed lines ported (37.5%) Files touched by this PR: 84 / 98 portable executed lines ported (85.7%) Full per-file / per-routine breakdown: see the "gpu-port-report" job summary and artifact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a fresh implementation of the Bodner mixed layer restrat since I completely destroyed my previous branch when implementing blocking.