Skip to content

Share the chunk-dim resolution rules and the axis deprecation mechanism - #15

Merged
cboulay merged 1 commit into
devfrom
cboulay/chunk-dim-helpers
Sep 6, 2026
Merged

Share the chunk-dim resolution rules and the axis deprecation mechanism#15
cboulay merged 1 commit into
devfrom
cboulay/chunk-dim-helpers

Conversation

@cboulay

@cboulay cboulay commented Sep 6, 2026

Copy link
Copy Markdown
Member

Foundation for the chunk_dim work landing across the ezmsg packages. Companion to ezmsg-sigproc#239, which currently carries private copies of all of this.

Why here

An AxisArray names its dimensions, but nothing about a position says what a dimension means. dims[0] is not "the streaming axis" and dims[-1] is not "the channel axis" — both break under transpose and downstream of any windowing stage, where (time, ch) becomes (win, time, ch). chunk_dim is the producer's declaration of which dimension messages accumulate along.

_message_hash already resolves that for its own purposes. Keeping the rules in one consumer package invites exactly the failure this work is fixing: a processor whose arithmetic disagrees with its state-reset logic about which dimension is which resets on the wrong changes and caches state along the wrong axis.

It's also the only place every package can reach. ezmsg-blackrock doesn't depend on sigproc (only transitively via ezmsg-event), and ezmsg-tools has it in a dev group, not at runtime — but all of them depend on baseproc.

util/chunkdim.py

Three rules, because one doesn't fit every case:

helper for rule
resolve_chunk_dim state carried between messages — filter zi, running mean, buffer, previous-sample cache chunk_dimSTREAMING_DIMSdims[0]
resolve_feature_dim a static axis (channels, components) position −1 or 0, skipping the chunk dim
resolve_transform_dim a transform consuming a regularly-sampled axis innermost non-chunk LinearAxis, else the chunk dim

The third exists because a spectrum needs time on both (time, ch) and (win, time, ch)win accumulates, but each window's spectrum is taken over time.

resolve_configured_chunk_dim honours an explicit setting, and warns once when it disagrees with a declared chunk_dim — or, via legacy_default, when a caller relied on a default that used to be hardcoded and whose resolution is now different.

STREAMING_DIMS moves here too, so the BaseStatefulTransformer class default and the resolver can't drift apart.

util/deprecation.py

The machinery for retiring an axis setting. Policy — which settings, which release — stays with each package, hence the package and removal parameters.

  • FutureWarning, not DeprecationWarning. The latter is suppressed by default outside __main__, so pipelines — library code — would never see it. Matches what ezmsg core already does for its own deprecations.
  • The stacklevel is found by walking to the first frame outside ezmsg. A fixed level can't work: the depth differs between constructing the settings directly, going through _unify_settings, and calling a factory that builds them inside the library. Nor can a filename test — the dataclass __init__ is exec-generated, so co_filename is "<string>" for the library and for python -c alike.
  • A frame with no __name__ keeps the walk going. dataclasses builds that __init__ with globals from sys.modules[cls.__module__]; a class whose module isn't in sys.modules leaves it with no module identity at all. Such a frame is synthetic, never the user's, so treating "unknown" as "user code" would blame a frame with no source line to show. Found by testing rather than by reasoning — see TestItBlamesTheCaller.
  • suppress_axis_deprecation() for a stage forwarding a setting to a child it built itself, which would otherwise warn on the user's behalf — and, from a _reset_state, would warn mid-stream naming whatever drives the pipeline.

Testing

239 passed, 1 skipped. Two new modules cover the three rules and the full deprecation contract, including the flip side worth recording: a settings class defined outside an ezmsg. module correctly blames its own __post_init__, because nothing distinguishes it from the caller.

Sequencing

This needs to land and release before the dependent PRs can bump their pins:

  1. this PR → baseproc release
  2. ezmsg-sigproc#239 — drops its private copies, re-exports from here
  3. ezmsg-event, ezmsg-learn, ezmsg-panel, ezmsg-simbiophys, ezmsg-blackrock

An AxisArray names its dimensions, but nothing about a *position* says what a
dimension means. `dims[0]` is not "the streaming axis" and `dims[-1]` is not
"the channel axis"; both break under transpose and downstream of a windowing
stage, where `(time, ch)` becomes `(win, time, ch)`. `chunk_dim` is the
producer's declaration of which dimension messages accumulate along.

`_message_hash` already resolves that for its own purposes, so the rules
belong here rather than in one consumer: a processor whose arithmetic
disagreed with its state-reset logic about which dimension is which would
reset on the wrong changes and cache state along the wrong axis. Every
downstream package depends on baseproc, but not all of them depend on any
single one of the others, so this is also the only place they can all reach.

util/chunkdim.py -- three rules, because one does not fit every case:

* resolve_chunk_dim: the accumulating dimension, for state carried between
  messages (filter zi, a running mean, a buffer, a previous-sample cache).
* resolve_feature_dim: position -1 or 0 skipping the chunk dim, for a static
  axis. dims[-1] can silently *be* the chunk dim on a (ch, time) stream.
* resolve_transform_dim: the innermost non-chunk LinearAxis, else the chunk
  dim -- a spectrum needs "time" on both (time, ch) and (win, time, ch).

resolve_configured_chunk_dim honours an explicit setting and warns once when
it disagrees with a declared chunk_dim, or when a caller relied on a default
that used to be hardcoded (legacy_default).

util/deprecation.py -- the machinery for retiring an `axis` setting, with the
policy (which settings, which release) left to each package:

* FutureWarning, not DeprecationWarning: the latter is suppressed by default
  outside __main__, so pipelines -- library code -- would never see it.
* The stacklevel is found by walking to the first frame outside ezmsg. A fixed
  level cannot work, since the depth differs between constructing the settings
  directly, going through _unify_settings, and calling a factory that builds
  them inside the library. Nor can a filename test: the dataclass __init__ is
  exec-generated, so co_filename is "<string>" for the library and for
  `python -c` alike. Frames are identified by module instead -- and a frame
  with no __name__ at all is synthetic, never the user's, so it keeps the walk
  going rather than being blamed for a line it cannot show.
* suppress_axis_deprecation() for a stage forwarding a setting to a child it
  built itself, which would otherwise warn on the user's behalf -- and, from a
  _reset_state, would warn mid-stream naming the pipeline driver.

STREAMING_DIMS moves here too, so the class default and the resolver cannot
drift apart.
@cboulay
cboulay merged commit 9d360cc into dev Sep 6, 2026
14 checks passed
@cboulay
cboulay deleted the cboulay/chunk-dim-helpers branch September 6, 2026 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant