Conversation
DockerBackend measured an EWMA of commit/flatten throughput after every snapshot, persisted it to a state file, and loaded it on worker startup — but no production call site ever passed the measured value into _snapshot_timeout_s. The throughput_bytes_per_second branch was dead in production; save_image/load_image always used the fixed ns_per_byte rate, while a whole persisted-calibration subsystem (config fields, load-on- startup, record-on-commit/flatten, state file) was maintained for a consumer that did not exist. The sandbox_snapshot_timeout_ns_per_byte description promised "until measured throughput is available" — but measured throughput was never made available. Introduced in b315836 (PR #2032), which rewired the snapshot-verb timeout to a size_walk_seconds branch and dropped the sole call site that passed the EWMA into _snapshot_timeout_s, leaving the measurement/persistence intact but unused. Removed rather than re-wired: the EWMA is measured from docker commit/flatten (single-layer compression, often CPU-bound), whose throughput profile differs from docker image save/load (multi-layer tar export/import of the whole image); a naive wiring could produce tighter budgets than the fixed rate and regress the exact mis-sized-timeout failure the calibration was meant to absorb. Restoring adaptive save/load budgets would require measuring save/load throughput directly and is left as separate work. Gone: _load_throughput, _record_throughput, self._throughput_bytes_per_second, the throughput_bytes_per_second parameter and dead branch of _snapshot_timeout_s, the size_rw/timing parameters of _commit/_flatten that existed only to feed it, and the sandbox_snapshot_throughput_ewma_alpha / sandbox_snapshot_throughput_state_path config fields. __init__ no longer reads the state file on startup; the file is never written. The snapshot verb's size_walk_seconds budget and the retry escalation are unchanged. Tests: rewrote test_snapshot_timeout_calibration.py to pin both live branches and add contract coverage for the previously-untested save_image/load_image timeout call sites, plus guards against re-introducing the dead seam. Existing snapshot-verb, disk-gate, registry-breaker, and snapshot-store suites unchanged and green. Co-authored-by: Detail <detail@users.noreply.github.com>
Code reviewVerdict: pass No blocking findings. The removed EWMA state/configuration was not consumed by production timeout call sites, and the fixed-rate save/load plus walk-derived commit/flatten timeout behavior remains intact. Repository-wide references confirm no production users of the deleted API remain. Validated at
|
…pshot-throughput-calibr-fb3b73
Code reviewVerdict: pass Reviewed Scope. Head is a merge of What I verified on this head
Non-blocking observations (no fix required; recorded so a later reader does not mistake them for oversights):
Not evaluated (and not required to be, per verification-proportionality): Docker-marked E2E suites and the gVisor runtime variant, both reported in the PR body. They exercise the unchanged walk-derived and disk-gate paths; the unit-level contract tests above cover the behavior this diff actually touches, and no removed code had a runtime-only effect that unit tests could hide. |
…pshot-throughput-calibr-fb3b73
Code reviewVerdict: pass Reviewed Scope of this round. Head is a new merge of Standing properties re-checked on this head Prior rounds asserted no blocking properties (both earlier eumemic-bot reviews passed). I re-verified the implicit acceptance criteria of a deletion PR against the current tree rather than trusting the earlier passes:
Non-blocking observations (carried forward from the prior round; still true, still not worth a fix cycle):
Not evaluated (and not required, per verification proportionality): the Docker-marked E2E suites and the gVisor runtime variant reported in the PR body. They exercise the walk-derived snapshot verb and disk-gate paths, which this diff does not modify; no removed code had a runtime-only effect that unit-level tests could conceal, since the deleted branch was unreachable from every production call site. |
…pshot-throughput-calibr-fb3b73
Code reviewVerdict: pass Reviewed Scope of this round. Head is a new merge of Standing properties re-checked on this head Prior rounds asserted no blocking properties (all three earlier reviews passed). I re-verified the acceptance criteria implicit in a deletion PR against the current tree rather than inheriting the earlier passes:
Non-blocking observations (carried forward; still true, still not worth a fix cycle):
Not evaluated (and not required, per verification proportionality): the Docker-marked E2E suites and the gVisor runtime variant reported in the PR body. They exercise the walk-derived snapshot verb and disk-gate paths, which this diff does not modify; no removed code had a runtime-only effect that unit-level tests could conceal, since the deleted branch was unreachable from every production call site. The lanes changes carried in by the master merge are outside this PR's diff and were reviewed under #2381. |
…pshot-throughput-calibr-fb3b73
Code reviewVerdict: pass Reviewed Scope of this round. Head is a new merge of Standing properties re-checked on this head Prior rounds asserted no blocking properties (all four earlier reviews passed). I re-verified the acceptance criteria implicit in a deletion PR against the current tree rather than inheriting the earlier passes:
Non-blocking observations (carried forward; still true, still not worth a fix cycle):
Not evaluated (and not required, per verification proportionality): the Docker-marked E2E suites and the gVisor runtime variant reported in the PR body. They exercise the walk-derived snapshot verb and disk-gate paths, which this diff does not modify; no removed code had a runtime-only effect that unit-level tests could conceal, since the deleted branch was unreachable from every production call site. The wake/lint changes carried in by the master merge are outside this PR's diff. |
…pshot-throughput-calibr-fb3b73
Code reviewVerdict: pass Reviewed Scope of this round. Head is a new merge of Standing properties re-checked on this head Prior rounds asserted no blocking properties (all five earlier reviews passed). I re-verified the acceptance criteria implicit in a deletion PR against the current tree rather than inheriting the earlier passes:
Non-blocking observations (carried forward; still true, still not worth a fix cycle):
Not evaluated (and not required, per verification proportionality): the Docker-marked E2E suites and the gVisor runtime variant reported in the PR body. They exercise the walk-derived snapshot verb and disk-gate paths, which this diff does not modify; no removed code had a runtime-only effect that unit-level tests could conceal, since the deleted branch was unreachable from every production call site. The |
Detail bug report: View on Detail
Summary
DockerBackendmeasured an EWMA of snapshot throughput after everydocker commit/flatten, persisted it to a state file, and loaded it on worker startup — but no production call site ever passed the measured value into_snapshot_timeout_s. Every caller leftthroughput_bytes_per_second=None, so the adaptive branch was dead code:save_image/load_imagealways fell back to the fixedsandbox_snapshot_timeout_ns_per_byterate, while a whole persisted-calibration subsystem (config fields, load-on-startup, record-on-commit/flatten, thesnapshot-throughput.jsonstate file) was maintained for a consumer that never existed. Thens_per_bytedescription promised "until measured throughput is available" — but measured throughput was never made available.size_walk_secondsbranch and dropped the sole call site that passed the EWMA into_snapshot_timeout_s, leaving the measurement/persistence intact but with no remaining consumer (andsave_image/load_image, added later, inherited the omission)._load_throughput,_record_throughput,self._throughput_bytes_per_second, thethroughput_bytes_per_secondparameter and dead branch of_snapshot_timeout_s, thesize_rw/timing parameters of_commit/_flattenthat existed only to feed it, and thesandbox_snapshot_throughput_ewma_alpha/sandbox_snapshot_throughput_state_pathconfig fields.__init__no longer reads the state file; it is never written. The snapshot verb'ssize_walk_secondsbudget and the retry escalation are unchanged. Thens_per_bytedescription now honestly describes the fixed save/load budget.docker commit/flatten (single-layer compression, often CPU-bound); its throughput profile differs fromdocker image save/load(multi-layer tar export/import of the whole image). A naive wiring could produce tighter budgets than today's fixed rate and regress the exact mis-sized-timeout failure the calibration was meant to absorb. Restoring adaptive save/load budgets would require measuring save/load throughput directly and is left as separate work.Substrate state changes
Removed two env vars for dead subsystem config:
AIOS_SANDBOX_SNAPSHOT_THROUGHPUT_EWMA_ALPHAandAIOS_SANDBOX_SNAPSHOT_THROUGHPUT_STATE_PATH. BecauseSettingsusesextra="ignore", deployments that still set them will silently drop them on startup (no validation failure). The persisted state file at/var/lib/aios/snapshot-throughput.json(default) is no longer read or written.Post-merge ops checklist:
Test plan
test_snapshot_timeout_calibration.pyto pin both live branches (fixed-rate save/load; walk-derived snapshot verb), add contract coverage for the previously-untestedsave_image/load_imagetimeout call sites, and guard against re-introducing the dead seam (passingthroughput_bytes_per_second=now raises; the config fields and backend calibration attrs/methods are gone). Snapshot-verb, disk-gate, registry-salvage/breaker, snapshot-store, and config suites are unchanged and green; fulltests/unitis green.ghcr.io/eumemic/aios-sandbox:latest):test_sandbox_salvage.py(crash-corpse + running-corpse salvage/resume), the fulltest_sandbox_persistence.pysuite (realdocker commit,docker export|importflatten, and resume-from-snapshot-tag), andtest_sandbox_image_contract.py(43 tests) all pass. Verified after these runs that nosnapshot-throughput.jsonfile is written anywhere (including/var/lib/aios/).[flatten]persistence variant failed the flatten disk-gate on a 20 GB loop FS that can't satisfy the default 15 GBsandbox_flatten_disk_floor_bytes(passes with the floor lowered), and twotest_browser_network_isolationcases failed on a host cgroupv2 "domain threaded mode" limitation that rejects the browser spec's--cpusconfig. Both reproduced identically against HEAD before the fix.AIOS_SANDBOX_RUNTIME=runsc ... pytest -m docker):runscwas downloaded and verified working, but the Docker daemon config lives on a read-only bootstrap filesystem, sorunsccould not be registered as a runtime (andSIGHUPreload doesn't register runtimes). It re-runs the samedocker-marked E2E suite under a second runtime; the snapshot-verb paths it would exercise already passed under the defaultruncruntime above.Risk / rollback
Low. The removed machinery was dead — no timeout behavior changes (save/load already used the fixed rate; the snapshot verb already used the walk-based budget). Roll back by reverting the commit; the deleted state file, if a worker had one, becomes stale and is simply ignored.
Automatic Fixes PRs can be configured here.