Skip to content

ch4/shm/posix: avoiding device id lookup for ZE registered host memory #7939

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
colleeneb wants to merge 1 commit into
base: main
Choose a base branch
from
+14 −9
Open

ch4/shm/posix: avoiding device id lookup for ZE registered host memory #7939

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions src/mpid/ch4/shm/posix/posix_rma.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_POSIX_do_put(const void *origin_addr,

#ifdef MPIDI_CH4_SHM_ENABLE_GPU
MPIR_GPU_query_pointer_attr(origin_addr, &origin_attr);
if (MPL_gpu_attr_is_dev(&origin_attr))
if (MPL_gpu_attr_is_strict_dev(&origin_attr))
origin_dev_id = MPL_gpu_local_to_global_dev_id(MPL_gpu_get_dev_id_from_attr(&origin_attr));
#endif

Expand All @@ -148,7 +148,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_POSIX_do_put(const void *origin_addr,
disp_unit = win->disp_unit;
#ifdef MPIDI_CH4_SHM_ENABLE_GPU
MPIR_GPU_query_pointer_attr(base, &target_attr);
if (MPL_gpu_attr_is_dev(&target_attr))
if (MPL_gpu_attr_is_strict_dev(&target_attr))
target_dev_id =
MPL_gpu_local_to_global_dev_id(MPL_gpu_get_dev_id_from_attr(&target_attr));
#endif
Expand All @@ -168,8 +168,8 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_POSIX_do_put(const void *origin_addr,
#ifdef MPIDI_CH4_SHM_ENABLE_GPU
if (MPIR_CVAR_CH4_IPC_GPU_RMA_ENGINE_TYPE != MPIR_CVAR_CH4_IPC_GPU_RMA_ENGINE_TYPE_yaksa) {
MPL_gpu_engine_type_t engine_type =
MPIDI_RMA_choose_engine(MPL_gpu_attr_is_dev(&origin_attr), origin_dev_id,
MPL_gpu_attr_is_dev(&target_attr), target_dev_id);
MPIDI_RMA_choose_engine(MPL_gpu_attr_is_strict_dev(&origin_attr), origin_dev_id,
MPL_gpu_attr_is_strict_dev(&target_attr), target_dev_id);
/* try to use cached local mmap for fast_memcpy */
MPI_Aint copy_sz = MPL_MIN(origin_data_sz, target_data_sz);
void *put_origin = (void *) origin_addr;
Expand Down Expand Up @@ -241,7 +241,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_POSIX_do_get(void *origin_addr,

#ifdef MPIDI_CH4_SHM_ENABLE_GPU
MPIR_GPU_query_pointer_attr(origin_addr, &origin_attr);
if (MPL_gpu_attr_is_dev(&origin_attr))
if (MPL_gpu_attr_is_strict_dev(&origin_attr))
origin_dev_id = MPL_gpu_local_to_global_dev_id(MPL_gpu_get_dev_id_from_attr(&origin_attr));
#endif

Expand All @@ -250,7 +250,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_POSIX_do_get(void *origin_addr,
disp_unit = win->disp_unit;
#ifdef MPIDI_CH4_SHM_ENABLE_GPU
MPIR_GPU_query_pointer_attr(base, &target_attr);
if (MPL_gpu_attr_is_dev(&target_attr))
if (MPL_gpu_attr_is_strict_dev(&target_attr))
target_dev_id =
MPL_gpu_local_to_global_dev_id(MPL_gpu_get_dev_id_from_attr(&target_attr));
#endif
Expand All @@ -270,8 +270,8 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_POSIX_do_get(void *origin_addr,
#ifdef MPIDI_CH4_SHM_ENABLE_GPU
if (MPIR_CVAR_CH4_IPC_GPU_RMA_ENGINE_TYPE != MPIR_CVAR_CH4_IPC_GPU_RMA_ENGINE_TYPE_yaksa) {
MPL_gpu_engine_type_t engine_type =
MPIDI_RMA_choose_engine(MPL_gpu_attr_is_dev(&origin_attr), origin_dev_id,
MPL_gpu_attr_is_dev(&target_attr), target_dev_id);
MPIDI_RMA_choose_engine(MPL_gpu_attr_is_strict_dev(&origin_attr), origin_dev_id,
MPL_gpu_attr_is_strict_dev(&target_attr), target_dev_id);

@hzhou hzhou Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MPL_gpu_attr_is_dev and MPL_gpu_attr_is_strict_dev are so confusing to any one who is not familiar with ZE. I think I we should have MPL_gpu_attr_is_dev and MPL_gpu_attr_is_host. !MPL_gpu_attr_is_host() would be so much clearer than the current MPL_gpu_attr_is_dev.

@colleeneb colleeneb Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed! I can change it here or in another PR unless you want to.

For local_ze_device_count is an unsigned int32,, same I can change it here or in another PR.

@hzhou hzhou Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR is good as is. Let's make a separate PR for refactoring.

/* try to use cached local mmap for fast_memcpy */
MPI_Aint copy_sz = MPL_MIN(origin_data_sz, target_data_sz);
void *get_origin = origin_addr;
Expand Down
7 changes: 6 additions & 1 deletion src/mpl/src/gpu/mpl_gpu_ze.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1293,7 +1293,12 @@ int MPL_gpu_global_to_local_dev_id(int global_dev_id)

int MPL_gpu_local_to_global_dev_id(int local_dev_id)
{
assert(local_dev_id < local_ze_device_count);
/* Host-owned allocations (e.g. ZE registered host memory) have no
* associated device. */
if (local_dev_id == -1) {
return -1;
}
assert(local_dev_id >= 0 && (uint32_t) local_dev_id < local_ze_device_count);
return local_to_global_map[local_dev_id];
}

Expand Down