Skip to content

ci: keep the crash harness's host records on every run and free its memory first - #55

Merged
stephane-segning merged 1 commit into
mainfrom
claude/emulator-pin-evidence
Oct 9, 2026
Merged

stephane-segning merged 1 commit into
mainfrom
claude/emulator-pin-evidence

Conversation

@stephane-segning

@stephane-segning stephane-segning commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What this does

The crash harness stays on Android emulator 37.1.11.0 (build 15917651). Moving it to 37.2.12.0 (build 16428233) was tried three ways and none of them worked, so this PR only keeps what those attempts showed was worth keeping:

  • host.txt and the kernel log are uploaded on every run (if: always()), not only on a failure. A green run is what a red one is read against.
  • A "Free the host" step stops the Gradle and Kotlin compile daemons before the emulator starts. They hold about 4.8 GB on the 16 GB runner (free -m went from 5757 MB used to 1120 MB used on this PR's own API 34 job, in the log).
  • The comment on EMULATOR_BUILD records the cause. EMULATOR_BUILD itself is unchanged at "15917651", and the guard that reads Pkg.BuildId from the SDK's own record is unchanged.

Why 37.2.12.0 cannot be adopted yet

The hang is host memory exhaustion, and the emulator process causes it. Its resident size grows with every app launch and never comes back:

API 34, one run emulator process lowest memory available swap
37.1.11.0 control, run 37873735095 3.1 GB at the first sample, 3.9 GB through all 20 repeats, 4.05 GB at the end 9717 MB never used
37.2.12.0, run 37869721841 3.0 GB growing to 13.6 GB 459 MB full from the 14th minute
37.2.12.0, Vulkan off, run 37873739944 13.5 GB 468 MB full from the 14th minute
37.2.12.0, -gpu guest, run 37873744804 13.2 GB 93 MB full from the 15th minute

On 37.2.12.0 the growth is 50 to 300 MB per app launch, crashed or not: the release-build refusal phase, which never crashes anything, still grows by about 60 MB per launch. On 37.1.11.0 the same phases grow by 0 to 16 MB per launch after the first few. The sampler's smaps_rollup shows the growth is anonymous native heap (Pss_Anon 3.3 GB on 37.1.11.0, 8.0 to 8.4 GB on 37.2.12.0 at the end of API 30), on top of a fixed 2 GB guest RAM mapping. The AVD is already at the emulator's own 2560 MB minimum, so a lower AVD RAM changes nothing.

The "hanging thread" watchdog, the QEMU stalls of 15 to 24 s and the lost runner are what the kernel does once physical memory and the 3 GB of swap are gone. Two 37.2.12.0 runs show it directly:

  • Run 37871662706: every harness stage passed (native: 20/20, native: 6/6, PASS release refuses the crash channel), then the runner received a shutdown signal during the final flutter test Gradle build, which is the last memory spike. That run's artifact was lost with the runner.
  • Run 37873744804: green, but detected a hanging thread 'QEMU2 main loop' (17.7 s), 'QEMU2 CPU0 thread' (21.4 s) and 'QEMU2 CPU1 thread' (22.8 s) were logged in that same final Gradle build, with 93 MB available.

The three attempts, one change each, all measured on the same sampler:

Attempt API 34 runs Result
Stop the build daemons before the emulator (this PR keeps it) 37869721841 green, 37871662706 runner lost Frees 4.8 GB at the start, but the leak still spends it. One pass at 459 MB available is luck.
-feature -Vulkan 37873739944 green Same growth, 13.5 GB.
-gpu guest 37873744804 green with three hanging-thread lines Same growth, 13.2 GB.

API 30 under the same workload (no 20-repeat stage) ends at 3.4 GB on 37.1.11.0 (run 37873735095) and at 8.0 to 8.6 GB on 37.2.12.0 in all three variants (runs 37871662706, 37873739944, 37873744804).

A green run on 37.2.12.0 here means the run finished before the memory ran out, by 93 to 468 MB. That is not a proof the harness works on it, so main stays pinned. Bump EMULATOR_BUILD once the emulator's size stays level in host.txt.

Verification

  • actionlint, zizmor --offline (no findings) and yamllint -d relaxed on .github/workflows/crash-harness.yml; Super-linter is green on the PR.
  • The squash message parses with @conventional-commits/parser 0.4.1, assembled as ci/commit-message-parse/parse.mjs assembles it, and the commit-message-parses check passes.
  • Actions stay SHA-pinned, nothing is interpolated into a run: block, and permissions are unchanged.
  • This PR's own crash-harness run, 37875987419, on the pinned emulator: both jobs green. API 34 logs Android emulator version 37.1.11.0 (build_id 15917651), the guard prints Pkg.BuildId=15917651, native: 20/20 runs passed, native: 6/6 runs passed, PASS release refuses the crash channel, 1 test passed, and no hanging thread. The emulator stayed at 4.2 GB at most, 9502 MB of memory were available at the lowest, and swap was never used. host.txt was uploaded by the green run, which is what if: always() is for.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XLy7TiPAxtVjPKEwQ9iwBa


Generated by Claude Code

…emory first

The harness uploaded host.txt and the kernel log only when a job failed, so
a green run left nothing to read a red one against. They are kept on every
run now.

The Gradle and Kotlin daemons the builds leave behind hold about 4.8 GB on
the 16 GB runner while the emulator runs, so they are stopped before it
starts, with free -m before and after in the log.

The pin on emulator 37.1.11.0 stays. Its comment now records why: the
37.2.12.0 emulator process grows by 50 to 300 MB for every app launch until
the runner is out of memory, and none of the three workarounds tried
removed that.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLy7TiPAxtVjPKEwQ9iwBa
@stephane-segning
stephane-segning marked this pull request as ready for review October 9, 2026 03:11
@stephane-segning
stephane-segning merged commit 20b8ac6 into main Oct 9, 2026
21 checks passed
@vaam-apps vaam-apps Bot mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants