Skip to content

Map thermodynamic vars to device to prevent lots of small transfers - #84

Open
uwagura wants to merge 1 commit into
MOM6-GPU:dev/gpufrom
uwagura:cleanup/reduce-sum-output-transfers
Open

Map thermodynamic vars to device to prevent lots of small transfers#84
uwagura wants to merge 1 commit into
MOM6-GPU:dev/gpufrom
uwagura:cleanup/reduce-sum-output-transfers

Conversation

@uwagura

@uwagura uwagura commented Jul 28, 2026

Copy link
Copy Markdown

write_energy in MOM_sum_output.F90 currently pushes tv%s and tv%t to the device using and !$omp target update to( directive. However, since the thermodynamic variables are not yet persistently on the device, I don't think this update actually does anything, and the compiler instead inserts a bunch of small transfers:

upload CUDA data  file=/scratch/cimes/uw4770/MOM6-examples/src/MOM6/src/diagnostics/MOM_sum_output.F90 function=write_energy line=763 device=0 threadid=1 variable=tv%s(is:ie,js:je,1:nz) bytes=2880
upload CUDA data  file=/scratch/cimes/uw4770/MOM6-examples/src/MOM6/src/diagnostics/MOM_sum_output.F90 function=write_energy line=763 device=0 threadid=1 variable=tv%s(is:ie,js:je,1:nz) bytes=2880
upload CUDA data  file=/scratch/cimes/uw4770/MOM6-examples/src/MOM6/src/diagnostics/MOM_sum_output.F90 function=write_energy line=763 device=0 threadid=1 variable=tv%s(is:ie,js:je,1:nz) bytes=2880
upload CUDA data  file=/scratch/cimes/uw4770/MOM6-examples/src/MOM6/src/diagnostics/MOM_sum_output.F90 function=write_energy line=763 device=0 threadid=1 variable=tv%s(is:ie,js:je,1:nz) bytes=2880

This pr changes the update to to a target enter data map(to directive and then releases the arrays, consistent to how we've handled them in other parts of the code, which gets rid of the transfers.

Given that this is the second time this has happend, we may want to start thinking about whether or not we want to push the tv arrays to the device at initialization and then update them as need.

@github-actions

Copy link
Copy Markdown

GPU Port Coverage

Overall: 2043 / 5662 portable executed lines ported (36.1%)
Since base branch: +0 ported lines (+-0.0 pp)

Files touched by this PR: 21 / 107 portable executed lines ported (19.6%)

Full per-file / per-routine breakdown: see the "gpu-port-report" job summary and artifact.

Full line-by-line coverage report

@JorgeG94

Copy link
Copy Markdown

Given that this is the second time this has happend, we may want to start thinking about whether or not we want to push the tv arrays to the device at initialization and then update them as need.

I think we should yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants