Human test: the example app's crashes are reported once each from real Android phones and a real iPhone #56

1. Type

• Operational task — manual verification by a person, in a real scenario

2. Summary

We need a person to crash the example app on purpose on real phones (low-end Transsion and Samsung Android phones, an Android 10 phone, and an iPhone) and confirm what reaches a collector, because native crash capture has only ever run on x86_64 Google emulators at API 30 and 34. No CI job crashes the app on iOS at all, and MetricKit is never delivered on a simulator. The OS records this package reads (ApplicationExitInfo, tombstones, MetricKit) are written by the phone maker's build of the OS, and only a real phone has one.

Expected result:

On each real phone, every crash, ANR and uncaught exception the example is told to die of arrives at the collector exactly once, as a FATAL record of the right kind, on the next launch, including when that launch was offline.

3. Intent

The Vaam app ships this capture to buyers and vendors on Tecno, Infinix, itel and Samsung phones. If those phones' OS records differ from the emulator's, crashes in the field are lost or doubled and nobody knows. A report sent twice, or every background kill reported as a crash, is also paid for out of the user's data bundle.

4. Source of Truth

• ci: pin the Android emulator to 37.1.11 for the crash harness #52 (merged 39e6ef60) — pins the Android emulator to 37.1.11.0, because 37.2.12.0 hung the crash harness: CI's proof runs on one emulator build.
• ci: wait out the uninstall before the crash harness reinstalls #54 (merged c1ec6c69) — waits out the uninstall before reinstalling, after the emulator's removal broadcast erased a record (received []); "Not seen on a device by hand: the OS's removal timing is read from the CI emulators' logs only."
• ci: keep the crash harness's host records on every run and free its memory first #55 (merged 20b8ac69) — frees the runner's memory and keeps host records; the harness stays on 37.1.11.0.
• README.md, "Crash harness" → "Manual device verification (not run in CI; a real-device farm is out of scope)" — the iOS steps this test runs.
• vaam-apps/vaam-apps#598 and docs/mobile/observability.md in vaam-apps, "Native crashes" — "What no build can show: that a crash is really reported … otel_zone's own example/ carries a debug-only crash harness for that … this repository does not re-run it."

5. Current Behavior

What the automated gates already prove:

• crash-harness.yml, on API 30 and API 34 x86_64 google_apis emulators: jvm, native and anr each arrive exactly once after a relaunch and never again; an update over a crashed build names the crashed build; native passes 20 of 20 repeats on API 34; a release build refuses the crash channel.
• Kotlin unit tests for the exit-record and tombstone decoding; swift test against fixture MetricKit payloads; a simulator build of the example; a check that the harness is in Debug and absent from Release.

What no gate can see:

• A real phone maker's OS: whether a Transsion (HiOS, XOS, itel OS) or Samsung (One UI) phone files ApplicationExitInfo and tombstones the way the emulator does, on arm64.
• The ANR dialog on a real phone, which CI hides with hide_error_dialogs.
• Android 10, where the OS keeps no exit records and only the JVM handler can report.
• Any iOS crash at all: MetricKit is not delivered on a simulator, and no CI job runs tool/ios-crash-harness.dart.
• Background kills by memory pressure or by the phone's battery manager, which an emulator does not produce.

6. Expected Behavior

• jvm, native and anr on Android, and nsexception and signal on iOS, each produce one FATAL device.crash or device.anr record on the next launch, with its device.crash.kind, and none on the launch after.
• Android 10 reports jvm and nothing for the kinds it has no OS record of, and still starts normally.
• A crash recovered while offline is delivered on the next online launch, once.

7. Acceptance Criteria

• Given the example's Debug build on a Tecno, Infinix or itel phone (Android 13–14), when the tester taps "Crash: jvm" and reopens the app, then the collector receives exactly one FATAL record with device.crash.kind = jvm and the exception's stack; a further reopen brings none.
• Given the same phone, when the tester taps "Crash: native" and reopens, then exactly one FATAL record with device.crash.kind = native arrives, with the crashing thread's frames; a further reopen brings none.
• Given the same phone, when the tester taps "Crash: anr", taps the screen a few times until Android shows "isn't responding", chooses "Close app" and reopens, then exactly one FATAL device.anr record arrives; a further reopen brings none.
• Given the same phone, when the tester taps "Crash: anr" and chooses "Wait" instead, then the app comes back, and no record arrives on the next reopen.
• Given a Samsung A-series phone (Android 14–15), when the jvm, native and anr steps are repeated, then each arrives exactly once.
• Given an Android 10 phone, when "Crash: jvm" is tapped and the app reopened, then exactly one jvm record arrives; and "Crash: native" leaves the app starting normally next time (no record is expected below API 30).
• Given airplane mode on, when the tester taps "Crash: native", reopens the app offline, closes it, turns data on and reopens it, then exactly one native record arrives, marked otel_zone.replayed = true.
• Given the example in the background on a 2–3 GB phone while the tester opens the camera, a game and several other apps until Android removes it from memory, when the example is reopened, then at most one record arrives for that death and it names its exit reason; the number of such records over one day of normal use is written down.
• Given the example's Debug build installed from Xcode on an iPhone (iOS 17 or later) and then launched from the home screen with Xcode detached, when the tester taps "Crash: nsexception" and reopens, then exactly one FATAL device.crash with device.crash.kind = nsexception, the exception's reason and its throw-site stack arrives; a further reopen brings none.
• Given the app running from Xcode on that iPhone, when the developer chooses Debug → Simulate MetricKit Payloads, stops it and relaunches, then the sample crash and hang diagnostics arrive once each; a further launch brings none.
• Given Xcode detached, when the tester taps "Crash: signal" and then opens the app at least twice over the next 26 hours, then exactly one signal record arrives.
• Verification evidence below is attached to this issue.

8. Out of Scope

• The Vaam app's own telemetry from a real phone (route spans, redaction, the floor): its own human test in vaam-apps/vaam-apps, filed with this one.
• Release builds: the example's crash channel is Debug-only by design, and CI already proves a release build refuses it.
• setEndUser (feat: setEndUser stamps enduser.id on spans and log records #51): Dart-only, wire-tested against a loopback collector, and not yet called by the Vaam app.
• Symbolication of native frames.

9. Technical Context

Build: the example app at main (at or after c1ec6c69), built in Debug by a developer with --dart-define=OTEL_EXPORTER_OTLP_ENDPOINT=<a collector the phone can reach and the developer can read>. Android: flutter build apk --debug in example/, installed by cable. iOS: run once from Xcode in the Debug configuration, then stop Xcode and launch from the home screen, because a debugger catches the crash before the OS records it.

Devices: a Tecno, Infinix or itel phone on Android 13–14 with 2–3 GB of RAM; a Samsung A-series on Android 14–15; an Android 10 phone; an iPhone on iOS 17 or later.

Network: Wi-Fi to the collector; airplane mode where a step says so.

Preconditions: a developer with the repository, Flutter at the version ci.yml pins, Xcode and a Mac, prepares the builds and reads the collector; the tester follows the phone steps. The collector is any OTLP/HTTP receiver the phone can reach: tool/otlp-log-sink.dart on a laptop on the same Wi-Fi works. This test needs nothing from Vaam's production, which the owner took down on 2026-10-09. Once Vaam's OpenObserve is back, its operator may count the device.crash and device.anr records production vaam-mobile builds already sent (they carry this capture since otel_zone 0.3.0); any from these phone models already answer part of this test.

Relevant code (for whoever fixes a failure): android/src/main/kotlin/com/vaam/otel_zone/ExitInfoSource.kt, ExitRecords.kt, JvmCrashHandler.kt, TombstoneDecoder.kt, CrashStore.kt; ios/otel_zone/Sources/otel_zone/MetricKitSource.swift, Crash/NSExceptionSource.swift, Crash/NativeCrashRepository.swift; lib/src/native-crash.dart, lib/src/spool.dart; example/lib/crash-harness.dart.

Security impact (SSDLC):

• Authentication / authorization
• Secrets or credentials
• PII or other sensitive data
• A new or changed external interface (API, webhook, import/export)

10. Risks

• Crashes on the phones Vaam's users actually own go unreported, or are reported twice, and the team reads the field wrong.
• Every background kill on a low-RAM phone exported as a FATAL record costs users data and buries real crashes.
• iOS crash capture has never been seen working anywhere; it may not work at all.

11. Implementation Plan

Test steps:

1. Developer: start the collector, build the example's Debug APK with its address, install it on each Android phone, and confirm the app says "Exporting to …".
2. On each Android phone: tap "Crash: jvm", reopen, reopen again; then "Crash: native", and "Crash: anr" (tap the screen until the dialog shows, choose "Close app"), each followed by two reopens.
3. Tap "Crash: anr" once more and choose "Wait".
4. Turn on airplane mode, tap "Crash: native", reopen, close, turn data on, reopen.
5. On the low-RAM phone, leave the example in the background, open the camera, a game and other apps, then reopen the example; keep the phone in normal use for a day and reopen the example in the evening.
6. On the Android 10 phone: "Crash: jvm" and "Crash: native", each followed by a reopen.
7. iPhone: run the example from Xcode once, stop Xcode, launch from the home screen; tap "Crash: nsexception", reopen twice.
8. Developer: run it from Xcode, choose Debug → Simulate MetricKit Payloads, stop, relaunch twice.
9. iPhone, Xcode detached: tap "Crash: signal"; open the app at least twice over the next 26 hours.
10. Developer: after each step, read the records that arrived and paste them into this issue.

12. Test Plan

• Manual verification
• Regression verification (tick if the steps also re-check old behaviour)

Expected result:

Each induced death arrives once, with its kind, on the next launch, on every phone;
none arrives twice; the offline one arrives replayed.

13. Verification Evidence

The tester attaches:

• Device model, OS version, app version and build number (the example reports service.version and app.build_id from --dart-define; note the commit it was built from)
• Network used (Wi-Fi / MTN / Orange / offline)
• A screen recording of the steps, and screenshots of any failure
• For each step, the records the collector received (kind, severity, event.name, the record's time), and the count of background-kill records over the day

14. AI Usage Declaration

AI was used for:

• Drafting the ticket
• Understanding code

Human verification completed:

• I understood the intent
• I checked the source of truth
• I verified the implementation manually

Human accountable owner: the tester assigned by @stephane-segning

17. Definition of Ready

• A tester is assigned and has the device classes listed above
• The build named under Technical Context is installed

18. Definition of Done

• Every acceptance criterion is ticked, or a failure is filed as its own bug linking back here
• Verification evidence is attached

Generated by Claude Code