Skip to content

ci: pin the Android emulator to 37.1.11 for the crash harness - #52

Merged
stephane-segning merged 2 commits into
mainfrom
claude/pin-emulator
Oct 9, 2026
Merged

stephane-segning merged 2 commits into
mainfrom
claude/pin-emulator

Conversation

@stephane-segning

@stephane-segning stephane-segning commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The crash harness's API 34 job has hung since 2026-10-02. The cause is an unpinned emulator: reactivecircus/android-emulator-runner installs the latest stable Android emulator when it is given no emulator-build, and emulator 37.2.12.0 (build 16428233) went stable on 2026-10-01. The first red run is the next day.

Evidence, on API 34:

  • Emulator 37.1.11.0 (build 15917651): 19 of 20 runs green. The one red is the known first-native "received []" flake (fix: cap dartastic_opentelemetry_api below 1.0.0-rc.4 #49), which also failed on 37.1.11.0 before the change and is a different mechanism.
  • Emulator 37.2.12.0 (build 16428233): 10 of 10 runs that reached the repeat stage hung. 9 were QEMU watchdog hangs and 1 lost the runner. They die 11 to 13.5 minutes after boot, after 16 to 19 repeat passes: QEMU threads stall 15 to 24 seconds, crashpad reports ptrace "no such process", and the harness dies on adb force-stop with the device offline (exit 255).
  • Not the cause: the runner image (the same image was green and red), the system image (byte-identical, same vbmeta.digest), the AVD config (identical), and the guest (no lmkd kills).

What this PR changes, in .github/workflows/crash-harness.yml only:

  • env.EMULATOR_BUILD: "15917651" (37.1.11.0), passed to the action as emulator-build.
  • A guard as the first script line: grep -Fx "Pkg.BuildId=$EMULATOR_BUILD" "$ANDROID_HOME/emulator/source.properties". It fails the step if the installed emulator is another build. It reads the SDK's own record of the build because running emulator -version fails on the runner (see Verification).
  • A second script line logs the system image's Pkg.Revision.
  • Host evidence: a 15 second sampler of free -m, df -h and the five largest processes by RSS, plus dmesg -T on failure, both kept in the harness-logs artifact.
  • The comments say why the build is pinned and when to bump it.

Nothing else moves: API 34 is not skipped or quarantined, the 20 native repeats stay, the boot is not split, and no run is retried. All actions stay SHA-pinned; values reach run: blocks through env:.

Intent

The owner asked: "Fix the otel-zone crash harness emulator hang."

The harness exists to measure native crash capture on a real runtime. Hiding the API 34 job would remove that measurement, so the fix keeps the job and pins the one input that changed underneath it.

Verification

Local:

  • actionlint, zizmor and yamllint are clean on the workflow. The only zizmor findings, at the pedantic persona, are the three pre-existing ${{ env.OTEL_PORT }} uses in run: lines I did not touch.
  • The pinned zip emulator-linux_x64-15917651.zip downloads (334378080 bytes) and its emulator/source.properties holds Pkg.Revision=37.1.11 and Pkg.BuildId=15917651. The guard line exits 0 against that file, and 1 for build 16428233 and for the prefix 1591765.
  • The sampler and dmesg steps run under bash -e.
  • The squash message built from both commits parses with @conventional-commits/parser 0.4.1.

On CI, first run (commit 40b631f, run 37859252308, not counted): the pin worked. The action fetched the 15917651 zip and the emulator booted as Android emulator version 37.1.11.0 (build_id 15917651). The first version of the guard ran emulator -version, which loads QEMU, and the runner has no libpulse.so.0, so the step failed before the harness started on both API levels. My local check passed because it ran on a machine that has the library. The guard now reads source.properties instead. That run also exercised the host evidence: its harness-logs artifact held host.txt (a sample every 15 seconds) and host-dmesg.txt.

The proof is the runs of this workflow on this branch after that fix. This PR's own run counts if it ran the harness; further runs are workflow_dispatch runs on claude/pin-emulator, one at a time because the concurrency group cancels an in-progress run. Four clean API 34 runs in total, each showing:

  • Android emulator version 37.1.11.0 (build_id 15917651) in the boot log, and the guard printing Pkg.BuildId=15917651;
  • no hanging thread;
  • native: 20/20 runs passed and native: 6/6 runs passed;
  • PASS release refuses the crash channel and the integration test passing.

API 30 must stay green. A red caused only by the "received []" flake is hang-free but does not count toward the four. If the guard fails because another emulator was installed, the fallback is to install the build in pre-emulator-launch-script (replace $ANDROID_HOME/emulator from the same zip) and restart the count.

Why four: if 37.1.11.0 were still hanging at the rate 37.2.12.0 did (10 of 10, a 95% lower bound of about 0.74), four clean runs by luck would have a chance of at most 0.5%.

Risk

  • Low. Only the workflow changes; the library, the harness and the example are untouched.
  • The pin is to a build that is no longer the newest. A future emulator fix is not picked up until someone bumps EMULATOR_BUILD, which the comment says to do only after the harness passes on the new build.
  • The guard can only fail loudly. If the action's sdkmanager system-image install ever replaces the pinned emulator, the step fails on its first line instead of running on the wrong build. It checks the installed emulator, not the process; the boot log's first line names the build that actually ran.
  • The pinned zip is fetched from dl.google.com by the action. If Google withdrew it, the job would fail at install, not hang.
  • The sampler is a background loop and is stopped with the job. It writes one small text file.
  • Out of scope: the "received []" first-native flake (fix: cap dartastic_opentelemetry_api below 1.0.0-rc.4 #49), and whether to file a Google issue-tracker bug for 37.2.12.0. A single dispatch with EMULATOR_BUILD=16428233 and the sampler would capture the evidence for that.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XLy7TiPAxtVjPKEwQ9iwBa

claude added 2 commits October 8, 2026 23:23
The API 34 job has hung since 2026-10-02. The emulator-runner action
installs the latest stable emulator when it is given no build, and
37.2.12.0 (build 16428233) became stable on 2026-10-01. On API 34, runs on
37.1.11.0 (build 15917651) passed 19 of 20 times, the one red being the
known first-native "received []" flake. Of the runs on 37.2.12.0 that reached
the repeat stage, 10 of 10 hung: QEMU stalls about twelve minutes after boot
and the harness dies when adb goes offline. The runner image and the system
image are the same on the green and the red runs.

Pin the emulator to build 15917651 through EMULATOR_BUILD and the action's
emulator-build input, fail the step when the running emulator is another
build, and log the system image's revision. Sample the host's memory and
disk every 15 seconds, and keep dmesg on failure, in the harness-logs
artifact, so a further stall is read from evidence.

Nothing is skipped, retried or split: API 34 still runs, still repeats
native twenty times and still boots once.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLy7TiPAxtVjPKEwQ9iwBa
The first run on this branch showed that the pin works and the guard did
not. The emulator downloaded and booted as 37.1.11.0, build 15917651, but
the guard ran emulator -version, which loads QEMU, and the runner has no
libpulse for it, so the step failed before the harness started. The boot
itself does not take that path.

The guard now matches Pkg.BuildId in the emulator's source.properties as a
whole line. The pinned zip carries that file with the build id in it, and
the step still fails if the installed emulator is any other build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLy7TiPAxtVjPKEwQ9iwBa
@stephane-segning
stephane-segning marked this pull request as ready for review October 9, 2026 01:11
@stephane-segning
stephane-segning merged commit 39e6ef6 into main Oct 9, 2026
22 checks passed
@vaam-apps vaam-apps Bot mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants