Atomic: ue ramdisk spec - #13278
Draft
github-actions[bot] wants to merge 17 commits into
Draft
Conversation
…HA pinning, RAM-ceiling; verify baseline
…ed PRs maxRunners:2 gated on workflow rewrite (path namespacing, prune removal), corrected RAM budget (counts work+shared-tmp emptyDirs), shared-dind DaemonSet (socket gid/PDB/2-build envelope), Talos tmpfs spike, and a deploy-order/rollback runbook incl. KubeVirt gate revert.
… separate PRs All work ships in one PR (#13278); items are sequenced workstreams within it. WS-1 (ci-unreal-build.yml rewrite) gates the maxRunners:2 flip.
…er spec All inert/safe-to-merge — nothing auto-activates (new manifests kept out of kustomization, values-ue.yaml unchanged, maxRunners stays 1): - WS-2 RAM-budget worksheet (the maxRunners:2 gate) + inert PrometheusRule - WS-3 shared-dind DaemonSet (+PDB, host-local socket gid 1000, tmpfs init) - WS-4 Talos decision record (reverses 2.1: DaemonSet over machine-config) - WS-5 cutover + rollback runbook (deploy order, KubeVirt gate revert) - WS-1 workflow edits pinned in runbook, applied at cutover (not blind-edited) Topology clarified: 2 Linux builders = one arc-runner-ue scale set; Windows = KubeVirt VM (UE5-Win); ue-win/ue-mac dormant, untouched.
…chine-config route - LFS: prepare does 'git lfs fetch --all' once; builders point lfs.storage at the shared repo.git/lfs (no per-job re-pull). Windows shares LFS readonly, not git alternates. §4 alternate-loss guard now covers LFS blobs. WS-1 commands pinned in runbook. - Safety: WS-4 decision made authoritative in §2.1 (was stale 'prefer Talos machine-config'). Implemented plan has NO Talos/OS change, NO reboot. - Added blast-radius inventory: only ONE cluster-wide change (KubeVirt virtiofs gate, Windows-only); Linux dedup needs zero cluster/Talos changes.
…RO cluster-wide changes Windows no longer file-shares the repo. Instead: prepare packs the populated repo (git+LFS) into a read-only disk image on tmpfs (repo.img); win_vm_boot attaches it RO (safe RO-everywhere block exception); the guest plain-clones into its own RAM work disk. Sole-writer-of-own-disk => no cross-OS corruption. - Removes ExperimentalVirtiofsSupport / filesystems device / WinFsp entirely. kubevirt/cr/kubevirt.yaml is untouched -> blast-radius inventory now ZERO cluster-wide changes (was 1). Only per-VM disk attach on vm-windows-builder. - Explains why block sharing corrupts (two kernels, no coordinator; FAT no help) and why RO-everywhere is the one safe exception. - prepare gains repo.img pack step; ordering prepare->win_vm_boot->build (disk at boot => no mount-readiness race). VM snapshot now optional (no guest mod). - WS-2 budget adds repo_img + win_clone_copy terms (full copy, RAM while VM up).
… modes - init + dind: assert /var/mnt/ramdisk is genuinely tmpfs before dockerd, refusing to put the 40GB data-root on the node OS disk (etcd's disk) on a mount/propagation gap (hostPath DirectoryOrCreate would mask it silently) - init: hard-fail if <CEILING> placeholder is unsubstituted instead of reaching mount(8) at runtime past every CI/Argo gate - dind: make the socket-gid fixup a persistent loop so a dockerd socket recreate cannot leave runners with permission denied and no self-heal - alert: correct the stale '<CEILING> threshold' comment (exprs are ratio/absolute, self-scaling) and flag the ruleSelector label-match gap - runbook: add cutover step 0 (substitute <CEILING>), a tmpfs assertion to step 1, a node-maintenance PDB-delete note, the atomic-auto-merge-lane prohibition for activation, and incident rows for the new fail-fast paths
…nners:2 viable The earlier specs assumed a ~109 GB node (mislabeled 'verified against the repo' — physical RAM is not in talos-worker.yaml). Operator-confirmed the node is ~500 GB. This flips the WS-2 headline: the budget closes with ~190 GB to spare, so maxRunners:2 (server + game Linux pods, + Windows VM in parallel, Mac on a separate box) is the safe target. Concurrency is now gated only by the functional B1/B2/B3 workflow blockers, not RAM. - daemonset: bake size=150G (was <CEILING> placeholder); drop the now-moot placeholder-not-substituted guard; keep the tmpfs-type assertions (the real silent-OS-disk-fallback guard) - ram-budget: rewrite for 500GB — two-ceiling model (node RAM vs the binding 150G tmpfs cap), reworked worked example that closes, and a 'Scaling maxRunners' table (3-4 needs a proportionally bigger ceiling) - design + talos-decision + runbook: 109->500, size=150G, plugin/engine are separate CI that share arc-runner-ue, refreshed preconditions/incident rows
…200G Operator chose maxRunners:3 — server + game + one plugin/engine Linux build concurrent (plugins are a separate CI that also lands on arc-runner-ue), with the Windows VM in parallel and Mac on a separate box. Per the WS-2 scaling rule, raise the tmpfs ceiling in lockstep: 150G -> 200G (engine 45 + mirror 7 + 3x~49 scratch). Budget still closes with ~100 GB free on the ~500 GB node. - daemonset: size=200G; dind cpu limit 8 -> 12 (CPU is the tighter term at 3 concurrent cooks co-resident with etcd); comments -> THREE cooks / maxRunners:3 - ram-budget: worked example + scaling table + decision rule reworked for 3 cooks / 200G; maxRunners:4 now the 'validate before going further' row - design + talos-decision + runbook: target 2 -> 3, ceiling 150G -> 200G, cutover 'flip maxRunners: 3 / three concurrent builds' - values-ue.yaml still UNCHANGED (inert maxRunners:1 baseline); the 3 is the cutover target, never applied live in this PR
Collapse the design / RAM-budget / Talos-decision / cutover-runbook docs into a single docs/superpowers/plans/2026-06-24-ue-ramdisk-shared-build.md written with the writing-plans skill: global constraints, file structure, ordered right-sized tasks with verify-commands + expected output, and RAM-sizing / guards references. Removes the audit-reconciliation, accepted/rejected, and WS-1..WS-5 meta framing — facts and actions only. Repoint the two manifest comments/annotations at the new plan; drop WS- jargon from them.
…mark per dev feedback Consolidation dropped a few load-bearing facts; restore them without the meta framing: - new 'Verified repo state' subsection: KubeVirt VM disk inventory, the enabled feature-gate list (ExperimentalVirtiofsSupport NOT among them), the intel-nuc labels are template strings (one node, not a fleet), single-node co-location - TCP-dind fallback guard (--tls + client certs + NetworkPolicy) back on the socket note - branch-drift reconciliation warning back on the baseline constraint - the deferred all-in-RAM vs image-on-disk benchmark restored as an explicit open question - reboot-drill validation step added to the DaemonSet bring-up
…IO counterweight The deferred note sold disk-backed too favorably (only the RAM it frees). Add why all-in-RAM is the default: build data lives in RAM to keep IO off etcd's shared SSD (#12987); engine-on-disk reintroduces extraction/eviction IO on that disk, and re-pulling per build re-pays the 40GB extraction every run. Bar for switching is measured + etcd-safe, not just 'frees RAM'.
…idle (configurable) Don't pin the ~52GB engine+mirror cache in RAM forever. Add an inert CronJob (ue-shared-dind-idle-reaper) that frees it after IDLE_TTL_SECONDS (default 4h, one env var to retune) of no UE builds; next build cold-starts via prepare, steady-state bursts still reuse the hot cache. - frees in-daemon (docker image prune -a -f + rm -rf mirror/scratch), NOT by deleting the pod: the tmpfs is host-ns mounted and survives pod deletion, so clearing contents is what returns the RAM (empty tmpfs costs ~0) - two guards prevent mid-build reaping: skip if any arc-runner-ue runner pod is Running, and skip if the shared daemon has any running container - prepare touches /var/mnt/ramdisk/.last-build so idle is measured from last build start; reaper seeds it if absent - plan: File Structure row, Task 6b (activate + verify guarded), prepare marker step, Reference: lifecycle (kept-vs-freed table), rollback line - inert: not in kustomization.yaml; activated at cutover
h0lybyte
marked this pull request as draft
July 20, 2026 22:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Atomic Branch:
atom-06240827-ue-ramdisk-specThis is an automated pull request for atomic changes.
Changes
Branch Info
atom-06240827-ue-ramdisk-specdevThis PR was automatically created by the atomic branch workflow. - KBVE Studio