Skip to content

Read the detector-done poll by index, and give it a quality floor - #724

Merged
xmap merged 1 commit into
mainfrom
fix/detector-poll-decode-and-quality
Aug 24, 2026
Merged

xmap merged 1 commit into
mainfrom
fix/detector-poll-decode-and-quality

Conversation

@xmap

@xmap xmap commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Why

Acquire_RBV decided "the acquisition is over" with value in (0, "Done"). A real areaDetector Acquire_RBV is a bi record, so it arrives as kind="Categorical" carrying whatever ZNAM/ONAM the IOC declares, with the index it resolved from on Measurement.ordinal. "Done" is ADCore's default label, not a guarantee: a relabelled record, or a build whose defaults differ, produces a value the check cannot recognise, and the poll loop waits forever on a detector that already finished.

This is the fourth instance of a defect family already fixed three times this month:

consumer fixed
hutch permit (SecureM) 2026-08-09
BLEPS interlock flags 2026-08-23
beam-availability gate 2026-08-24
detector-done poll here

The comparison also read reading.value with no quality floor at all. A Bad reading whose stale value happened to decode to Done would have recorded a capture as finished when the value meant nothing — a fail-open into the recorded evidence.

Not reachable at 2-BM (CONTROL_WRITES_ENABLED=false), so this is a correctness fix for the first deployment that drives a detector, caught by review rather than by a facility.

What

_acquisition_finished reads the ordinal via the same binary_code decoder the other three fixes use, and floors on believable, not actionable. That choice is the interesting part: a detector carrying a standing Uncertain alarm for a reason unrelated to whether it finished (a temperature warning, a nearly-full file-writer disk) must still be readable as Done, or every acquisition on it would hang. An alarm on Acquire_RBV says nothing about whether the acquisition is over; only Bad may withhold a conclusion.

The two duplicated poll loops (_run_collect_cycle, continuous) are now one shared _await_acquire_done — they had carried identical copies of the loop, which is how the same bug needed fixing in two places.

stream's NumCaptured_RBV comparison had the same missing floor, worse in one way: a non-numeric reading raises TypeError, which is not one of the Conductor's closed _CONTROL_ERRORS, so it would escape the Conductor uncaught rather than being recorded as a step failure.

Scope, confirmed before starting: decode + quality only. The poll loop's wait stays unbounded, which is a documented v1 choice (_POLL_INTERVAL_S: "relies on caller-side cancellation (Procedure abort) for hard timeout"), consistent with the cancellation model locked in the scan-primitives design (Conductor halts, records, operator/saga decides). This fix stops an ordinary label mismatch or a stray alarm from making that wait silent and indistinguishable from a slow detector; it does not change when the wait ends. An unreadable reading now logs once per continuous stretch (not once per 50ms tick) and recovers with an info line.

cora/shared/quality.py gains a "per QUESTION, not per component" section: the Conductor itself now uses both floors (actionable for check steps, believable for this poll), so the earlier framing that one BC means one floor was no longer accurate.

Test fixtures were part of the bug

Both softIOC-backed test tiers seeded Acquire_RBV as a bare Scalar 0, a shape the substrate cannot produce — the same blind spot that hid the previous three occurrences. Fixed:

  • tests/integration/_softioc.py: cam1:Acquire_RBV is now a real bi (ZNAM="Done" / ONAM="Acquiring"), not a longin.
  • All four action-body unit test files now build the Categorical + ordinal shape (_acquire_rbv_reading() in test_collect_action_body.py, mirrored into continuous/discrete).

Verification

Full suite: architecture 34,121 passed, unit 14,333 passed. Pyright, ruff, tach clean.

Five mutations run, five caught, each isolated with a small asyncio.wait_for-bounded standalone script (pytest-timeout's thread method doesn't actually cancel a genuinely-hung while True: await asyncio.sleep() loop, so mutations that hang the poll were verified outside pytest to keep the loop bounded):

mutation result
drop the ordinal path relabelled-record case hangs
drop the believable guard Bad-quality Done reading wrongly terminates after 1 read
tighten to actionable Uncertain-quality Done reading wrongly hangs
drop the once-per-episode log flag 3 warnings instead of 1
drop stream's guard Bad-quality count wrongly terminates after 1 read

🤖 Generated with Claude Code

`Acquire_RBV` decided "the acquisition is over" with `value in (0, "Done")`.
A real areaDetector `Acquire_RBV` is a `bi` record, so it arrives as
`kind="Categorical"` carrying whatever ZNAM/ONAM the IOC declares, with the
index it resolved from on `Measurement.ordinal`. "Done" is ADCore's default
label, not a guarantee: a relabelled record, or a build whose defaults
differ, produces a value the check cannot recognise, and the poll loop then
waits forever on a detector that already finished. This is the fourth
instance of a defect family already fixed three times this month (the hutch
permit, the BLEPS interlock flags, the beam-availability gate), and the
comparison also read `reading.value` with no quality floor at all, so a
`Bad` reading whose stale value happened to decode to Done would have
recorded a capture as finished when the value meant nothing.

`_acquisition_finished` reads the ordinal via the same `binary_code`
decoder the other three fixes use, and floors on `believable`, not
`actionable`. That choice matters here specifically: a detector carrying a
standing `Uncertain` alarm for a reason unrelated to whether it finished (a
temperature warning, a nearly-full file-writer disk) must still be readable
as Done, or every acquisition on it would hang instead of finish. An alarm
on `Acquire_RBV` says nothing about whether the acquisition is over; only
`Bad` may withhold a conclusion. The two duplicated poll loops
(`_run_collect_cycle`, `continuous`) are now one shared `_await_acquire_done`,
which also closes the reason the bug existed in two copies to begin with.

`stream`'s `NumCaptured_RBV` comparison had the same missing floor, worse in
one way: a non-numeric reading raises `TypeError`, which is not one of the
Conductor's closed `_CONTROL_ERRORS`, so it would have escaped the Conductor
uncaught rather than being recorded as a step failure.

None of this bounds the poll loop's wait, which stays the documented v1
choice (Procedure abort is the caller-side timeout). The fix stops an
ordinary label mismatch or a stray alarm from making that wait silent and
indistinguishable from a slow detector; it does not change when the wait
ends. Not reachable at 2-BM (`CONTROL_WRITES_ENABLED=false`), so this is a
correctness fix for the first deployment that drives a detector, caught by
review rather than by a facility.

`quality.py` gains a "per QUESTION, not per component" section: the
Conductor itself now uses both floors (`actionable` for check steps,
`believable` for this poll), so the module's earlier framing that one BC
means one floor was no longer accurate.

Both softIOC-backed test tiers previously seeded `Acquire_RBV` as a bare
Scalar 0, a shape the substrate cannot produce, which is why neither could
see this bug. The integration fixture is now a real `bi` (ZNAM=Done); the
unit fixtures across all four action-body test files now build the
Categorical + ordinal shape. Five mutations (drop the ordinal path, drop the
believable guard, tighten to actionable, drop the once-per-episode log
flag, drop stream's guard) were each run and each caught by exactly the
test built for it.
@github-actions

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  apps/api/src/cora/operation
  acquisitions.py
  apps/api/src/cora/shared
  quality.py
Project Total  

This report was generated by python-coverage-comment-action

@xmap
xmap merged commit 18341da into main Aug 24, 2026
19 checks passed
@xmap
xmap deleted the fix/detector-poll-decode-and-quality branch August 24, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant