Skip to content

feat(processor): enable EFA for whole-host reservations (single-node, not just multinode) - #221

Merged
wdvr merged 1 commit into
wdvr:mainfrom
d4l3k:efa-whole-host-reservations
Jul 16, 2026
Merged

feat(processor): enable EFA for whole-host reservations (single-node, not just multinode)#221
wdvr merged 1 commit into
wdvr:mainfrom
d4l3k:efa-whole-host-reservations

Conversation

@d4l3k

@d4l3k d4l3k commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

EFA was only attached to multinode full-node pods. Enable it for any whole-host reservation (gpu_count == max_gpus) on an EFA-capable SKU, including single-node ones — a whole-host pod owns the node, so the existing hostNetwork + sshd:2222 model stays conflict-free.

  • Centralize the gate in _pod_uses_efa (efa_count > 0 and gpu_count == max_gpus); get_pod_resource_limits/requests now call it instead of duplicating the condition. Drops the is_multinode requirement.
  • Subhost (partial-node) pods still get NO EFA: they share a node and would collide on hostNetwork port 2222 — deferred.
  • Fix b200 efa_count 32 -> 8 to match the node's advertised efa_network_cards (p6-b200.48xlarge caps at 8), else whole-host b200 pods can't schedule.

Needs a reservation_processor lambda deploy to take effect.

… not just multinode)

EFA was only attached to multinode full-node pods. Enable it for any
whole-host reservation (gpu_count == max_gpus) on an EFA-capable SKU,
including single-node ones — a whole-host pod owns the node, so the
existing hostNetwork + sshd:2222 model stays conflict-free.

- Centralize the gate in _pod_uses_efa (efa_count > 0 and gpu_count ==
  max_gpus); get_pod_resource_limits/requests now call it instead of
  duplicating the condition. Drops the is_multinode requirement.
- Subhost (partial-node) pods still get NO EFA: they share a node and
  would collide on hostNetwork port 2222 — deferred.
- Fix b200 efa_count 32 -> 8 to match the node's advertised
  efa_network_cards (p6-b200.48xlarge caps at 8), else whole-host b200
  pods can't schedule.

Needs a reservation_processor lambda deploy to take effect.
@huydhn

huydhn commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

cc @wdvr I just realize that I don't have owner permission to the repo T_T

@huydhn
huydhn requested a review from wdvr July 16, 2026 22:37
@wdvr
wdvr merged commit 1c5db23 into wdvr:main Jul 16, 2026
2 checks passed
@d4l3k
d4l3k deleted the efa-whole-host-reservations branch July 17, 2026 16:40
@huydhn

huydhn commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

@wdvr Ty for your help! Are u planning to do a release for this change? I don't think I have the permission to run https://github.com/wdvr/osdc/actions/workflows/publish.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants