Skip to content

topology-aware: restore DRA claims after a restart - #859

Open
bart0sh wants to merge 6 commits into
mainfrom
stacked/bart0sh/DRA/014-restore-claims
Open

bart0sh wants to merge 6 commits into
mainfrom
stacked/bart0sh/DRA/014-restore-claims

Conversation

@bart0sh

@bart0sh bart0sh commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

topology-aware forgets its DRA claims when the plugin restarted. Setup() clears the policy's cache entries, and Sync() places every running container again, so containers holding a claim lost its CPUs, and new claims can get them. The kubelet does not prepare a claim again after our restart, so nothing rebuilt a claim once it was gone.

Now Setup() keeps the cache entries and restores the saved claim grants. Container grants are not restored: Sync() allocates every running container again, and finds the claims by then. A container which holds a claim the policy does not know gets a warning in the log.

If the new configuration cannot give a claim its CPUs, because they are reserved or no longer available or the pool is gone, the claim gets other CPUs of the same NUMA nodes, and Sync() moves its containers to them. A claim result is only a number of CPUs of a node, so the kubelet and the scheduler see no change. The claims which keep their CPUs are restored first, so a moved claim cannot take the CPUs of another claim. The start fails only when a node has too few CPUs for a claim. The error names the claim, and the agent writes it into the status of the config CR. Reverting the config lets the plugin start again with the claim restored. A runtime reconfiguration which cannot hold a claim is still refused.

Two fixes on the side: Reserve() took a reserved CPU which is also isolated for the isolated part of a claim, which also affected runtime reconfiguration on main. And the topology-aware policy tests unpacked the sysfs tarball once per test, about 20s each with -race, which made the package hit the 10m go test timeout. With one shared copy the package runs in 42s with -race.

Testing:

  • go test -race -count=1 ./... passes with the default 10m timeout.
  • golangci-lint v2.11.4 reports no issues.
  • e2e on Kubernetes 1.37 with DRAConsumableCapacity, with images built from this branch: test42 (topology-aware, also with DRANodeAllocatableResources) and test01-dra (template) pass. test42 restarts the plugin and checks that running containers keep their claimed CPUs and a new claim gets only free CPUs, that a config which reserves a CPU of a claim moves the claim and its containers to the free CPU of the same node, and that a config which leaves too few CPUs fails the start with the claim named in the config status. test01-dra restarts template with fewer allowed CPUs: the claim keeps its CPUs with a warning, and a new claim does not get them.

@bart0sh
bart0sh added this pull request to stack #807 October 18, 2026 14:55
@bart0sh bart0sh changed the title stacked/bart0sh/DRA/014 restore claims topology-aware: restore DRA claims after a restart Oct 18, 2026
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch 2 times, most recently from 3cec18b to 4f183af Compare October 18, 2026 15:44
@bart0sh
bart0sh requested a balanced review from Copilot October 18, 2026 15:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Restored claims can overlap reserved workloads when a claimed isolated CPU becomes reserved.

1 open issue
What changed in this PR

Restores topology-aware DRA claims from cache before running containers are reallocated after a plugin restart.

Changes:

  • Preserves cached claims and reports restoration failures.
  • Warns when containers reference unknown claims.
  • Adds restart tests and updates recovery documentation.
File Description
test/​e2e/​policies.test-suite/​topology-aware/​n4c16/​test42-dra-devices/​code.var.sh Tests claim persistence and failed-start recovery.
test/​e2e/​policies.test-suite/​template/​n4c16/​test01-dra/​code.var.sh Tests restart with fewer allowed CPUs.
docs/​resource-policy/​policy/​topology-aware.md Documents restart behavior and recovery.
cmd/​plugins/​topology-aware/​policy/​topology-aware-policy.go Preserves cache and restores claims during setup.
cmd/​plugins/​topology-aware/​policy/​dra.go Warns about unknown container claims.
cmd/​plugins/​topology-aware/​policy/​dra_test.go Adds restoration and failure-recovery unit tests.
cmd/​plugins/​topology-aware/​policy/​cache.go Restores claim grants and tracks unresolved pools.

Review effort: Balanced

Comment thread cmd/plugins/topology-aware/policy/cache.go Outdated
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 4f183af to 2dbe255 Compare October 18, 2026 16:07
@bart0sh
bart0sh requested a balanced review from Copilot October 18, 2026 18:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved issues remain, and regression coverage includes claim persistence, configuration conflicts, and recovery.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@bart0sh
bart0sh marked this pull request as ready for review October 18, 2026 18:27
@bart0sh
bart0sh marked this pull request as draft October 18, 2026 18:35
@bart0sh
bart0sh requested a balanced review from Copilot October 19, 2026 06:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Persisted CPU assignment migration and failed-start recovery warrant final human validation.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved blocking issues remain, and regression coverage addresses restoration, relocation, and configuration-failure recovery.

No open issues 🎉

Review effort: Balanced

@bart0sh
bart0sh marked this pull request as ready for review October 19, 2026 06:49
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from fe333f4 to 6a2e470 Compare October 19, 2026 11:33
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 6a2e470 to 17a1fd0 Compare October 19, 2026 12:15
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 17a1fd0 to 3be5773 Compare October 19, 2026 12:38
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch 2 times, most recently from a72a7b2 to fef55fd Compare October 19, 2026 12:59
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from fef55fd to 8d506eb Compare October 19, 2026 13:06
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch 2 times, most recently from 24d8ca2 to 8aaeec0 Compare October 19, 2026 13:11
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 8aaeec0 to ea848ca Compare October 19, 2026 13:30
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch 2 times, most recently from a0e4bef to 578db14 Compare October 19, 2026 14:29
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 578db14 to 9eb1683 Compare October 19, 2026 14:45
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 9eb1683 to 3b29f8f Compare October 19, 2026 14:50
@klihub
klihub force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 3b29f8f to 3129c87 Compare October 19, 2026 15:04
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 3129c87 to 983142e Compare October 20, 2026 06:22
@askervin
askervin force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 983142e to 3987fe9 Compare October 20, 2026 10:16
Base automatically changed from stacked/bart0sh/DRA/013-prepare-for-release to main October 20, 2026 11:18
@askervin
askervin force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 3987fe9 to 40c78a9 Compare October 20, 2026 11:18
Setup() no longer clears our cache entries. It restores the saved
claim grants, and fails on a claim whose pool is gone or whose CPUs
we cannot grant any more. Container grants are not restored, since
Sync() allocates every running container again.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
A reserved CPU can also be isolated, and Reserve() then takes it for
the isolated part of a claim, so reserved workloads could run on the
claim's CPUs after a restart or a reconfiguration.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
When a restart with a new configuration cannot give a claim its old
CPUs, give it other CPUs of the same NUMA nodes, and Sync() moves its
containers to them. The start fails only when a node has too few CPUs
for the claim. Before, such a claim failed the start, and the node ran
without the plugin until the configuration was reverted.

Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/014-restore-claims branch from 40c78a9 to 2f744b7 Compare October 23, 2026 12:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants