Share the chunk-dim resolution rules and the axis deprecation mechanism - #15
Merged
Conversation
An AxisArray names its dimensions, but nothing about a *position* says what a dimension means. `dims[0]` is not "the streaming axis" and `dims[-1]` is not "the channel axis"; both break under transpose and downstream of a windowing stage, where `(time, ch)` becomes `(win, time, ch)`. `chunk_dim` is the producer's declaration of which dimension messages accumulate along. `_message_hash` already resolves that for its own purposes, so the rules belong here rather than in one consumer: a processor whose arithmetic disagreed with its state-reset logic about which dimension is which would reset on the wrong changes and cache state along the wrong axis. Every downstream package depends on baseproc, but not all of them depend on any single one of the others, so this is also the only place they can all reach. util/chunkdim.py -- three rules, because one does not fit every case: * resolve_chunk_dim: the accumulating dimension, for state carried between messages (filter zi, a running mean, a buffer, a previous-sample cache). * resolve_feature_dim: position -1 or 0 skipping the chunk dim, for a static axis. dims[-1] can silently *be* the chunk dim on a (ch, time) stream. * resolve_transform_dim: the innermost non-chunk LinearAxis, else the chunk dim -- a spectrum needs "time" on both (time, ch) and (win, time, ch). resolve_configured_chunk_dim honours an explicit setting and warns once when it disagrees with a declared chunk_dim, or when a caller relied on a default that used to be hardcoded (legacy_default). util/deprecation.py -- the machinery for retiring an `axis` setting, with the policy (which settings, which release) left to each package: * FutureWarning, not DeprecationWarning: the latter is suppressed by default outside __main__, so pipelines -- library code -- would never see it. * The stacklevel is found by walking to the first frame outside ezmsg. A fixed level cannot work, since the depth differs between constructing the settings directly, going through _unify_settings, and calling a factory that builds them inside the library. Nor can a filename test: the dataclass __init__ is exec-generated, so co_filename is "<string>" for the library and for `python -c` alike. Frames are identified by module instead -- and a frame with no __name__ at all is synthetic, never the user's, so it keeps the walk going rather than being blamed for a line it cannot show. * suppress_axis_deprecation() for a stage forwarding a setting to a child it built itself, which would otherwise warn on the user's behalf -- and, from a _reset_state, would warn mid-stream naming the pipeline driver. STREAMING_DIMS moves here too, so the class default and the resolver cannot drift apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Foundation for the
chunk_dimwork landing across the ezmsg packages. Companion to ezmsg-sigproc#239, which currently carries private copies of all of this.Why here
An
AxisArraynames its dimensions, but nothing about a position says what a dimension means.dims[0]is not "the streaming axis" anddims[-1]is not "the channel axis" — both break undertransposeand downstream of any windowing stage, where(time, ch)becomes(win, time, ch).chunk_dimis the producer's declaration of which dimension messages accumulate along._message_hashalready resolves that for its own purposes. Keeping the rules in one consumer package invites exactly the failure this work is fixing: a processor whose arithmetic disagrees with its state-reset logic about which dimension is which resets on the wrong changes and caches state along the wrong axis.It's also the only place every package can reach.
ezmsg-blackrockdoesn't depend on sigproc (only transitively via ezmsg-event), andezmsg-toolshas it in a dev group, not at runtime — but all of them depend on baseproc.util/chunkdim.pyThree rules, because one doesn't fit every case:
resolve_chunk_dimzi, running mean, buffer, previous-sample cachechunk_dim→STREAMING_DIMS→dims[0]resolve_feature_dimresolve_transform_dimLinearAxis, else the chunk dimThe third exists because a spectrum needs
timeon both(time, ch)and(win, time, ch)—winaccumulates, but each window's spectrum is taken overtime.resolve_configured_chunk_dimhonours an explicit setting, and warns once when it disagrees with a declaredchunk_dim— or, vialegacy_default, when a caller relied on a default that used to be hardcoded and whose resolution is now different.STREAMING_DIMSmoves here too, so theBaseStatefulTransformerclass default and the resolver can't drift apart.util/deprecation.pyThe machinery for retiring an
axissetting. Policy — which settings, which release — stays with each package, hence thepackageandremovalparameters.FutureWarning, notDeprecationWarning. The latter is suppressed by default outside__main__, so pipelines — library code — would never see it. Matches what ezmsg core already does for its own deprecations.ezmsg. A fixed level can't work: the depth differs between constructing the settings directly, going through_unify_settings, and calling a factory that builds them inside the library. Nor can a filename test — the dataclass__init__is exec-generated, soco_filenameis"<string>"for the library and forpython -calike.__name__keeps the walk going.dataclassesbuilds that__init__with globals fromsys.modules[cls.__module__]; a class whose module isn't insys.modulesleaves it with no module identity at all. Such a frame is synthetic, never the user's, so treating "unknown" as "user code" would blame a frame with no source line to show. Found by testing rather than by reasoning — seeTestItBlamesTheCaller.suppress_axis_deprecation()for a stage forwarding a setting to a child it built itself, which would otherwise warn on the user's behalf — and, from a_reset_state, would warn mid-stream naming whatever drives the pipeline.Testing
239 passed, 1 skipped. Two new modules cover the three rules and the full deprecation contract, including the flip side worth recording: a settings class defined outside an
ezmsg.module correctly blames its own__post_init__, because nothing distinguishes it from the caller.Sequencing
This needs to land and release before the dependent PRs can bump their pins: