diff --git a/docs/design/ltm--loops-that-matter.md b/docs/design/ltm--loops-that-matter.md index f80b4fff0..98de73f1a 100644 --- a/docs/design/ltm--loops-that-matter.md +++ b/docs/design/ltm--loops-that-matter.md @@ -1067,13 +1067,17 @@ hoisted too: the `Iterated` axis carries the (target, source) dimension pair, the agg is arrayed over the TARGET dim (`State`), and each source row is remapped to the slot of its positionally-corresponding target element (`iterated_axis_slot_elements` -- the preimage of -`mapped_element_correspondence`, so the element-map/positional gate is -inherited). The only reducers *not* hoisted are the dynamic-index carve-out -(`SUM(pop[idx, *])`, `idx` non-literal -- not statically describable, -reclassified `DynamicIndex`) and the mapped sliced reducers the -correspondence declines -- an explicit element-mapped pair (execution -resolves positionally, GH #756) or a reverse-declared mapping (GH #757) -- -which keep the conservative cross-product; a bare non-literal index +`positional_correspondence`, which is the right rule here because +`matrix[State, *]` names the dimension the equation ITERATES and execution +folds that to an ordinal; an explicit element map is therefore honoured as a +DECLARED correspondence but not READ, GH #997). The only reducers *not* +hoisted are the dynamic-index carve-out (`SUM(pop[idx, *])`, `idx` +non-literal -- not statically describable, reclassified `DynamicIndex`), a +pair with no declared correspondence at all, and a `MappedRead` axis +(`SUM(matrix[Region, *])` naming a NON-iterated dimension, GH #997: its +executed rule admits a many-to-one correspondence that the one-slot-per-row +remap cannot express, so `compute_read_slice` declines it) -- all of which +keep the conservative cross-product; a bare non-literal index (`arr[i+1]`) is a dynamic reference, not a reducer, so it stays conservative. Variable-backed aggs (`total_population = SUM(population[*])`) are already real nodes -- their edges come from the normal arrayed→scalar / @@ -1173,13 +1177,14 @@ slot per `D1` element); `SUM(matrix3d[D1, NYC, *])` over an A2A-`D1` body ⇒ `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim, and the emitters remap each source row to the slot of its positionally-corresponding target element (`iterated_axis_slot_elements`, -the preimage inversion of `mapped_element_correspondence`, so the -positional-only gate is inherited). The carve-outs (tracked tech debt; +the preimage inversion of `positional_correspondence`, the rule the ITERATED +spelling gets). The carve-outs (tracked tech debt; the conservative cross-product / coarse link score stays in place) are: a reducer over a *dynamic index* (`SUM(pop[idx, *])`, `idx` non-literal -- the IR reclassifies its reference to `DynamicIndex`); a mapped sliced reducer -the correspondence declines -- an explicit element-mapped pair (execution -resolves positionally and ignores the map, GH #756) or a mapping declared +the correspondence declines -- a pair with no declared correspondence, or a +`MappedRead` axis whose executed rule the slot remap cannot invert +(GH #997) -- or a mapping declared only in the reverse direction (on the source's dimension; GH #757 tracks that direction's classification); and a multi-source reducer whose arrayed args read incompatible slices (`combined_read_slice` returns `None` on diff --git a/src/simlin-engine/CLAUDE.md b/src/simlin-engine/CLAUDE.md index 11d2864c3..5e4fddd2e 100644 --- a/src/simlin-engine/CLAUDE.md +++ b/src/simlin-engine/CLAUDE.md @@ -11,23 +11,24 @@ Equation text flows through these stages in order: 1. **`src/lexer/`** - Tokenizer for equation syntax. `KEYWORDS` is the equation language's reserved-word table (`if`/`then`/`else`/`not`/`mod`/`and`/`or`/`nan`; the units lexer shares it and differs only in also admitting `$` inside identifiers), and `identifierish` resolves a bare word against it BEFORE falling back to `Token::Ident`. `is_reserved_word` exposes that verdict so `ast::needs_quoting` can read this table rather than restate it -- without it a variable legally named `"if"` printed BARE and no longer re-parsed, so an unrelated `patch` rename persisted an unparseable equation (GH #976). 2. **`src/parser/`** - Recursive descent parser producing `Expr0` AST 3. **`src/ast/`** - AST type system with progressive lowering: `Expr0` (parsed) -> `Expr1` (modules expanded) -> `Expr2` (dimensions resolved) -> `Expr3` (subscripts expanded). `array_view.rs` tracks array dimensions and sparsity. Every layer's `Const` holds an `ast::Literal` (`literal.rs`) rather than a bare `f64`: a numeric literal is compared and hashed by BIT PATTERN, so `NaN == NaN` and a value holding one is equal to itself. Salsa backdates a re-executed query's memo by comparing old and new with `PartialEq`, and every stdlib SMOOTH/DELAY/TREND template declares `initial_value = NAN`, so a bare `f64` left every project carrying stage values and parse memos that were unequal to themselves and could never backdate (GH #987/#981). The four AST enums derive `Eq` as well, which is what keeps the fix STRUCTURAL *within the AST*: a new float-bearing variant or field on any type the AST reaches must use `Literal`, because a bare `f64` fails to satisfy `Eq` and that is a compile error rather than a silently reintroduced incrementality cliff (enforced transitively -- an `f64` added to `ArrayView`, which `Expr3` reaches only indirectly, fails the same way). Bit comparison costs this layer NOTHING, because both distinctions it draws are unreachable in an AST literal by construction rather than accepted: there is one `nan` spelling yielding one canonical `f64::NAN` (a practitioner cannot author a payload), and `lexer::scan_number` takes no leading sign, so `-0` is a negation of the literal `0` and never a `Const`. Both premises are tripwire tests. `Literal`'s rustdoc carries the PROJECT'S POSITION ON FLOAT EQUALITY -- wherever a float feeds a cache key we want bit equality -- and `src/float.rs`'s module docs carry the domain reason it is the right posture rather than a shortcut. Two families keep the derived IEEE `PartialEq` as an ACCEPTED state, not an open defect -- what is accepted is the cost of NOT converting them, not a semantic property worth keeping: the compiled bytecode types (`ByteCode::literals`, `ByteCodeContext::graphical_functions`, `results::Specs`, and the symbolic `SymbolicByteCode::literals` / `PerVarBytecodes::graphical_functions` -- GH #642, closed on this reasoning), and `variable::Table`'s `x`/`y: Vec` + `GraphicalFunctionScale`s, which ride into the SAME three memos so a lookup table with a NaN y-point still defeats their backdating. Each type carries a pointer to the position; the argument lives only on `Literal`. Two things recorded there rather than rediscovered: #642's own "the only consumer is non-tracked" premise is FALSE one level down (`PerVarBytecodes` is the value of the tracked `compile_var_fragment`, read by the tracked `assemble_module`), so the missed backdate is real and knowingly accepted; and IEEE equality is also LOOSER on signed zeros, so a pool differing only in a zero's sign backdates and the stale one is kept (unlike the AST, `-0.0` IS reachable in a compiled pool -- `compiler::fold` folds `0 * -1` to it) -- pre-existing, corpus-unreached, and not what #642 describes, but it points the same way: converting those types closes both directions and gives up nothing. No other AST-equality consumer is affected either way (the numeric comparisons in `mdl::writer::exprs_equal`, `ltm::polarity`'s sign tests, and the compiler's static index resolution all read the value out with `Literal::value` and are unaffected). `Expr2Context` trait includes `has_mapping_to()` for cross-dimension mapping lookups during `find_matching_dimension`. `needs_quoting` is the single "can this canonical name be spelled bare in an EQUATION" predicate -- character classes, the leading-character rule, and `lexer::is_reserved_word` -- and it has THREE consumers, one per producer of equation text: `print_ident` (the `print_eqn` path), `ltm_augment::quote_ident` (LTM's generated guard forms), and `mdl::xmile_compat::XmileFormatter::quote_reference` (the MDL importer, which writes equation text into `datamodel::Equation` that our own lexer reads back; it excludes `nan` alone, a disclosed residual explained in the MDL module doc). `print_eqn_proptest::a_bare_spellable_name_lexes_as_one_identifier` pins its completeness against the lexer itself, which is what makes the restatement checkable. The MDL *writer*'s `needs_mdl_quoting` is a DIFFERENT language's rule and deliberately separate -- as is `quoted_space_to_underbar`, which spells a definition-side *ident*, not equation text. A canonical name containing `"` is the one shape with no spelling at all (the lexer's quoted identifier has no escape); `patch::apply_rename_variable` refuses to rename TO one rather than persisting an equation that cannot be re-read. -4. **`src/builtins.rs`** - Builtin function definitions (e.g. `MIN`, `PULSE`, `LOOKUP`, `QUANTUM`, `SSHAPE`, `VECTOR SELECT`, `VECTOR ELM MAP`, `VECTOR SORT ORDER`, `VECTOR RANK`, `ALLOCATE AVAILABLE`, `ALLOCATE BY PRIORITY`, `NPV`, `MODULO`, `PREVIOUS`, `INIT`). `is_stdlib_module_function()` is the authoritative predicate for deciding whether a function name expands to a stdlib module; the **module-function** generalization that also resolves project macros lives in **`src/module_functions.rs`** -- the pure (Functional-Core) `ModuleFunctionDescriptor`/`MacroRegistry` resolver+validator unifying stdlib functions and macros (`stdlib_descriptor`, `MacroRegistry::build`/`resolve_macro`, the shared `is_renamed_opcode_intrinsic`/`is_renamed_stdlib_module_builtin`/`is_renamed_builtin_macro_collision` precedence predicates). `MacroRegistry::build` validates in four passes: duplicate macro name and macro/model name collision (macros.AC5.3), macro-to-macro recursion (macros.AC5.2), and -- Pass 4 -- a `Variable::Module` inside a macro-marked model (macros.AC5.7, `ErrorCode::MacroContainsModule`). Pass 4 is a CYCLE-SAFETY rule and is not redundant with Pass 3: `db::project_module_graph` records only EXPLICIT module edges, so a cycle `mac ->(explicit module)-> u ->(macro call, an IMPLICIT edge)-> mac` was invisible to the gate and drove the recursive queries into salsa's dependency-graph cycle panic, while Pass 3's macro-to-macro graph cannot even express the edge through the non-macro `u`. The rejection is deliberately broader than the cycle -- an acyclic module inside a macro is rejected too -- and that cost is REAL, not zero: the shape is reachable from an ordinary XMILE file (the `` content model is shared with ``, so the reader passes a `` through unfiltered), our own XMILE writer round-trips it, and an acyclic instance compiles and SIMULATES correctly today. It is rejected anyway because narrowing to only-when-cyclic needs a second reachability analysis that must agree with `project_module_graph`'s, whose back edge is the macro CALL -- discoverable only by parsing every model's equations, which is the dependency-list cost widening the graph was rejected for; and because a macro is a TEMPLATE, so instantiating a sub-model inside one is dubious on its own terms and reads as a language rule rather than a workaround. The MDL importer cannot produce it, but not because "Vensim macros have no modules": `src/mdl/convert/multi_output.rs` DOES mint a `Variable::Module` for a multi-output `:`-list invocation, and it is unreachable from a macro body only because the scoped body sub-context hard-codes an empty materialization (`src/mdl/convert/macros.rs`). A module *targeting* a macro (how multi-output invocations work) is untouched -- only a module *inside* a macro-marked model is rejected. A **genuine passthrough macro** -- a single-parameter, single-output macro whose body is exactly `out = BUILTIN(param)` self-calling its own renamed-builtin-collision name (`:MACRO: INIT(x) = INITIAL(x)`, stored after the importer's `INITIAL`->`INIT` rename as `init = init(x)`) -- is classified once at `MacroRegistry::build` time (`classify_passthrough`, the only place the body AST is available) into a `passthrough: Option` on the descriptor; `builtins_visitor.rs` reads it to *collapse the call to the builtin opcode* (`init`->`LoadInitial`) instead of expanding the buggy per-element synthetic module (#591), falling through to the same renamed-builtin intrinsic routing the #554 self-call exception takes. Strict criteria (no additional outputs, bare-parameter argument, self-call, renamed-builtin-collision name) keep it from misfiring on a near-miss like `INIT = INIT(x) + 1`. `equation_is_module_call()` (pre-scan, renamed from `equation_is_stdlib_call`) and `contains_module_call()` (walk-time, renamed from `contains_stdlib_call`) both consult the `MacroRegistry` (a passthrough caller is still classified module-backed here -- benign, since it collapses to a flat-slot variable). `builtins_visitor.rs` handles implicit module instantiation, single-output macro inlining (an arrayed macro invocation enters the per-element path), and PREVIOUS/INIT helper rewriting: unary `PREVIOUS(x)` desugars to `PREVIOUS(x, 0)`, direct scalar args compile to `LoadPrev`, and module-backed or expression args are first rewritten through synthesized scalar helper auxes. `INIT(x)` compiles to `LoadInitial`, using the same helper rewrite when needed. Tracks `module_idents` so `PREVIOUS(module_var)` never reads a multi-slot module directly. +4. **`src/builtins.rs`** - Builtin function definitions (e.g. `MIN`, `PULSE`, `LOOKUP`, `QUANTUM`, `SSHAPE`, `VECTOR SELECT`, `VECTOR ELM MAP`, `VECTOR SORT ORDER`, `VECTOR RANK`, `ALLOCATE AVAILABLE`, `ALLOCATE BY PRIORITY`, `NPV`, `MODULO`, `PREVIOUS`, `INIT`). `is_stdlib_module_function()` is the authoritative predicate for deciding whether a function name expands to a stdlib module; the **module-function** generalization that also resolves project macros lives in **`src/module_functions.rs`** -- the pure (Functional-Core) `ModuleFunctionDescriptor`/`MacroRegistry` resolver+validator unifying stdlib functions and macros (`stdlib_descriptor`, `MacroRegistry::build`/`resolve_macro`, the shared `is_renamed_opcode_intrinsic`/`is_renamed_stdlib_module_builtin`/`is_renamed_builtin_macro_collision` precedence predicates). `MacroRegistry::build` validates in four passes: duplicate macro name and macro/model name collision (macros.AC5.3), macro-to-macro recursion (macros.AC5.2), and -- Pass 4 -- a `Variable::Module` inside a macro-marked model (macros.AC5.7, `ErrorCode::MacroContainsModule`). Pass 4 is a CYCLE-SAFETY rule and is not redundant with Pass 3: `db::project_module_graph` records only EXPLICIT module edges, so a cycle `mac ->(explicit module)-> u ->(macro call, an IMPLICIT edge)-> mac` was invisible to the gate and drove the recursive queries into salsa's dependency-graph cycle panic, while Pass 3's macro-to-macro graph cannot even express the edge through the non-macro `u`. The rejection is deliberately broader than the cycle -- an acyclic module inside a macro is rejected too -- and that cost is REAL, not zero: the shape is reachable from an ordinary XMILE file (the `` content model is shared with ``, so the reader passes a `` through unfiltered), our own XMILE writer round-trips it, and an acyclic instance compiles and SIMULATES correctly today. It is rejected anyway because narrowing to only-when-cyclic needs a second reachability analysis that must agree with `project_module_graph`'s, whose back edge is the macro CALL -- discoverable only by parsing every model's equations, which is the dependency-list cost widening the graph was rejected for; and because a macro is a TEMPLATE, so instantiating a sub-model inside one is dubious on its own terms and reads as a language rule rather than a workaround. The MDL importer cannot produce it, but not because "Vensim macros have no modules": `src/mdl/convert/multi_output.rs` DOES mint a `Variable::Module` for a multi-output `:`-list invocation, and it is unreachable from a macro body only because the scoped body sub-context hard-codes an empty materialization (`src/mdl/convert/macros.rs`). A module *targeting* a macro (how multi-output invocations work) is untouched -- only a module *inside* a macro-marked model is rejected. A **genuine passthrough macro** -- a single-parameter, single-output macro whose body is exactly `out = BUILTIN(param)` self-calling its own renamed-builtin-collision name (`:MACRO: INIT(x) = INITIAL(x)`, stored after the importer's `INITIAL`->`INIT` rename as `init = init(x)`) -- is classified once at `MacroRegistry::build` time (`classify_passthrough`, the only place the body AST is available) into a `passthrough: Option` on the descriptor; `builtins_visitor.rs` reads it to *collapse the call to the builtin opcode* (`init`->`LoadInitial`) instead of expanding the buggy per-element synthetic module (#591), falling through to the same renamed-builtin intrinsic routing the #554 self-call exception takes. Strict criteria (no additional outputs, bare-parameter argument, self-call, renamed-builtin-collision name) keep it from misfiring on a near-miss like `INIT = INIT(x) + 1`. `equation_is_module_call()` (pre-scan, renamed from `equation_is_stdlib_call`) and `contains_module_call()` (walk-time, renamed from `contains_stdlib_call`) both consult the `MacroRegistry` (a passthrough caller is still classified module-backed here -- benign, since it collapses to a flat-slot variable). `builtins_visitor.rs` handles implicit module instantiation, single-output macro inlining (an arrayed macro invocation enters the per-element path), and PREVIOUS/INIT helper rewriting: unary `PREVIOUS(x)` desugars to `PREVIOUS(x, 0)`, direct scalar args compile to `LoadPrev`, and module-backed or expression args are first rewritten through synthesized scalar helper auxes. `INIT(x)` compiles to `LoadInitial`, using the same helper rewrite when needed. Tracks `module_idents` so `PREVIOUS(module_var)` never reads a multi-slot module directly. An **array-shaped** subscripted argument -- `arg_is_array_shaped`: every index either statically resolvable or leaving a whole dimension standing (a wildcard, a star-range, or a bare ACTIVE apply-to-all dimension name) -- is passed through UNTOUCHED instead: no per-element `substitute_dimension_refs`, no capture helper. That is what makes an array-valued `PREVIOUS` possible at all (GH #995), and it puts the decision where it can be made: `PREVIOUS(vals[D])` means the element in `y[D] = PREVIOUS(vals[D])` and the whole array in `y[D] = VECTOR SORT ORDER(PREVIOUS(vals[D]), 1)`, exactly as bare `vals[D]` does, and only `compiler::context`'s promotion knows which position it is in. Substituting here would pin it to one element before that context exists, and the helper path cannot hold it either (a scalar `Equation::Scalar` helper holding `vals[*]` does not compile). A mapped or otherwise FOREIGN dimension name is deliberately excluded from the predicate: those need the positional translation only this layer can do, so they keep the proven scalar path. 5. **`src/compiler/`** - Multi-pass compilation to bytecode: - `mod.rs` - Orchestration; includes A2A hoisting logic that detects array-producing builtins (VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) during array expansion, hoists them into `AssignTemp` pre-computations, and emits per-element `TempArrayElement` reads. Treats a **standalone lookup-only variable** -- a graphical-function holder whose equation is empty or the legacy MDL `"0+0"` sentinel (`crate::variable::var_is_lookup_only`/`is_empty_or_sentinel`, mirroring `mdl::writer::is_lookup_only_equation`'s "empty or sentinel" rule) -- as a non-value-bearing **static table** (`Variable::Var::is_table_only`): it is excluded from every runlist and from the saved output, so it produces NO series of its own (issue #606). Its data is reached only through `LOOKUP(table, x)` call sites (resolved by ident -> `base_gf`); a *bare* reference with no argument is a compile error (`ErrorCode::LookupReferencedWithoutArgument`). WITH LOOKUP (`var = WITH LOOKUP(input, table)`: tables present *and* a real input) is NOT lookup-only -- it is a value-bearing variable that lowers to `LOOKUP(self, input)` for every equation shape (`apply_implicit_with_lookup`): per element for arrayed variables, where a per-element gf applies each element's OWN table and a gf-less element keeps its raw input equation (GH #909) + - `array_operand.rs` - The last lowering pass, run from the `Var::new` chokepoint right after constant folding: it discharges codegen's contract that an **array-valued operand is a VIEW over storage** (`walk_expr_as_view` accepts `StaticSubscript | TempArray | Var | Subscript`, plus the array-valued `PREVIOUS`/`INIT` snapshot view described under `codegen.rs` below). A computed array -- `vals[D] * 2`, `NOT ...`, an `IF` over two arrays, an elementwise `ABS(...)`, a nested array-producing builtin -- is none of those, so it is moved into an `AssignTemp` of its own spliced in front of the expression that reads it, and codegen's `AssignTemp` -> `BeginIter` path evaluates it element by element. No new opcode, no VM change, no wasm change (GH #995). Pass 1 (`ast::expr3::Pass1Context`) materializes what it can, but it works on `Expr3`, BEFORE subscripts are resolved, and misses two shapes for the same underlying reason: an operand carrying an unresolved apply-to-all dimension reference is deferred to pass 2 (`vals[D]` only means "the whole array" after `context.rs`'s `with_vector_builtin_wildcards` promotion, which happens during lowering), and the type checker bounds `vals[D] * 2` as a SCALAR for that same reason, so `needs_decomposition` declines it before the deferral even matters. Running on the fully lowered fragment is what makes the pass simple: the promotion has already happened and every view is concrete. The temp's shape is the JOIN of every array shape in the operand, not the first one found: an elementwise operand is evaluated by codegen's `AssignTemp` -> `BeginIter` loop, which broadcasts each source view onto the ITERATION by dimension id, so a source dimension the iteration does not have reads NaN. `compiler::join_array_views` picks the view every other one is CONTAINED IN (by dimension name and size), which makes the answer independent of operand order -- `VECTOR SORT ORDER(small[d] + wide[e,d], 1)` used to iterate `small`'s three elements and return the sort order of three NaNs while the commuted spelling returned the right answer. Two shapes neither of which contains the other DECLINE (the union's AXIS ORDER is not determined by the operand, and axis order is which axis VECTOR SORT ORDER sorts along), which leaves codegen to reject the operand with a diagnostic attributed to the variable, exactly as the two deliberately unmaterialized positions below do. A view REPEATING a dimension name (`matrix[d,d]`) declines wherever it appears -- mixed with another shape AND as the operand's sole shape (`view_repeats_a_dimension`, checked by the MATERIALIZER on the join's answer rather than inside the join, and by `codegen::snapshot_static_view` on the snapshot route; those are the two positions that can refuse loudly, and putting it in the join instead reached the three silent hoisting sites, sizing `out[d] = SUM(VECTOR SORT ORDER(matrix[d,d], 1))`'s temp at three slots for a nine-element sort order and indexing the VM past it -- a shape that returns numbers at the merge base). `[d,d]` can say "contains `d` at size 3" but not WHICH `d`, and every layer projecting between an array and a temp matches by name and takes the first hit (`project_var_index_to_temp` gives both axes one coordinate, so `out[i,j]` would read `temp[i,i]`; `codegen::array_view_to_static_temp` keys `DimId`s the same way), so there is no shape to give. The refusal costs nothing that worked: measured at the MERGE BASE `ccf7ed34`, neither `VECTOR SORT ORDER(matrix[d,d] * 2, 1)` nor the `PREVIOUS`/`INIT` spelling compiles -- both became compilable on this branch and both produced first-axis-wins garbage until refused, the second through `codegen::snapshot_static_view`, which rejects a repeated-dimension snapshot source on the same grounds. Reading a repeated dimension DIRECTLY is untouched and is a disclosed PRE-EXISTING residual: `out[d,d] = matrix[d,d]` returns `[11,11,11,22,22,22,33,33,33]` and `VECTOR SORT ORDER(matrix[d,d], 1)` returns `[0,0,0,1,1,1,2,2,2]` (per-row orders are `[0,1,2]`), identically at the merge base and here, pinned by `a_repeated_dimension_read_directly_is_a_pre_existing_residual`. Its fix is to give the projection an axis identity rather than a dimension name -- the same root cause as `expand_same_element`'s repeated-target residual. Vensim REJECTS the declaration outright ("DimA appears more than once on LHS", Vensim DSS 2026-08-04), so no MDL-imported model reaches the residual; the XMILE v1.0 spec exemplifies the declaration, so it stays reachable from conformant XMILE and stays worth fixing. The spec settles only the DECLARATION -- what a REFERENCE like `sq[X,X]` means is asked of Stella by `vensim-probes/stella_repeated_dimension.stmx`. The defect is narrower than it looks: Simlin's STORAGE is a correct 2-D array (measured `SUM(sq[X,*])` = 36/66/96, `SUM(sq[*,*])` = 198); only the subscripted reference collapses to the first axis. The join is `find_expr_array_view` itself rather than a second rule, so the shape of the hoisted array-producing builtin's own temp -- derived from the same function before this pass runs -- moves with it. Only the materializer's decline is LOUD, and the asymmetry is written out on `find_expr_array_view`: the three apply-to-all/arrayed hoisting sites SUBSTITUTE the variable's own view on `None` with no diagnostic, which is shadowed rather than safe -- they ask about an array-producing builtin whose shape is its shaping argument's, so a `None` there means that same argument is an operand the materializer also declines and the fragment fails to compile first. Nothing enforces the shadowing. The **safety property** is that it rewrites only operands codegen would have rejected (`is_view` is the negation of `walk_expr_as_view`'s accepting arms), so a fragment that compiles today is untouched, temp count included -- measured: max temps/fragment across the corpus and C-LEARN is 21 before and after. It DOES double temp-id consumption on the per-element hoisting path (one temp per array element becomes two), which gives a **user-visible ceiling**: a computed array operand under per-element hoisting (an array-producing builtin whose *scalar* argument varies by element, so each element re-evaluates it) stops compiling above roughly **128 elements**, because two temps per element exhausts the `u8` `TempId` namespace. That is a loud `resolve_static_view` rejection rather than wrong numbers (see `symbolic.rs` below); #583, widening the namespace, is what lifts the ceiling. The shared (non-per-element) shape allocates two temps total and is unaffected at any array size. An array-valued `PREVIOUS`/`INIT` is NOT materialized, because it does not need to be: since GH #995's phase C3 it is itself a view, over a snapshot buffer rather than over `curr` (`is_snapshot_view`, sharing `compiler::snapshot_view_arg` with codegen so the pass and the emitter cannot disagree about which calls take the array route). Nested inside a computed operand it materializes like anything else and the `BeginIter` body reads the snapshot view per element -- which is what closed the phase C1+C2 decline that used to refuse the whole operand, because the argument was then lowered element-collapsed and the temp would have held ONE element's previous value broadcast across the array (measured `[0,2,1]` where the answer is `[2,0,1]`). A `PREVIOUS` of a genuinely scalar variable, or of a fixed element like `matrix[E,1]`, still carries no array shape, so it broadcasts and keeps materializing. The view-position enumeration is a single exhaustive `match` with no `_` arm, derived from codegen's `walk_expr_as_view` call sites. TWO positions are deliberately declined, each because materializing would be silently wrong rather than merely incomplete: the arrayed-GF `LOOKUP` table (codegen resolves it to a `base_gf` by ident, and a temp has no graphical functions -- not constructible from the equation language today either way); and `ALLOCATE AVAILABLE`'s priority profiles (its view is re-expanded by `context::expand_pp_view_for_allocate`, which only understands a direct variable reference, so a computed profile would reach the VM as a one-column-per-requester temp). Tests: `src/array_operand_materialization_tests.rs` (the derived position x shape x spelling matrix, the snapshot-view rows over a TIME-VARYING fixture, and GH #995's own table re-run), corpus fixtures `test/vector_computed_operand/` and `test/vector_snapshot_operand/`. - `context.rs` - Symbol tables and variable metadata; `lower_preserving_dimensions()` skips Pass 1 dimension resolution to keep full array views for array-producing builtins. Handles `@N` position syntax resolution: in scalar context (no active A2A dimension, not inside an array-reducing builtin), `DimPosition(@N)` resolves to a concrete element offset; inside array-reducing builtins (`preserve_wildcards_for_iteration`), dimension views are preserved for iteration. Two wildcard-preservation contexts: `with_preserved_wildcards()` for reducers (SUM, MEAN, etc.) where `ActiveDimRef` resolves to a concrete offset, and `with_vector_builtin_wildcards()` for array-producing builtins (VectorSortOrder, VectorElmMap, etc.) where `ActiveDimRef` is promoted to `Wildcard` to preserve the full array view - `expr.rs` - Expression compilation - `invariance.rs` - Run-invariance classification (time-invariant hoisting, GH #712, stage B1). The pure **functional core** `exprs_are_invariant(exprs, classify_offset)` walks a variable's lowered `Vec` and returns whether it is run-invariant (same value every timestep): exhaustive + default-variant over every `Expr`/`BuiltinFn` variant, parameterized by an offset-classification callback that resolves a referenced slot to its owner's verdict (`OffsetClass::{Invariant, Variant}`). Invariant = literals/`Dt`/`TimeStep`/`StartTime`/`FinalTime`/`Pi`/`Inf`, `Init(_)` of any var (init buffer frozen), GF lookups with invariant indices, pure builtins/reducers/array-ops of invariant args, and other invariant vars by offset. Variant (default) = `Time`/`Pulse`/`Ramp`/`Step`/`Previous`/`ModuleInput`/`EvalModule`, stocks, module-instance slots. The two compile paths (`compiler::Module` monolithic, test-only; salsa `db::invariance`) feed it different callbacks but share this walk, so they classify identically -- pinned by `db::invariance`'s `salsa_and_monolithic_paths_agree`. See [the design note](/docs/design-plans/2026-06-04-time-invariant-hoisting.md). - - `codegen.rs` - Bytecode emission; routes array-producing builtins through dedicated opcodes instead of element-wise iteration. Codegen emits **symbolic** opcodes: a variable operand is a `compiler::VarRef` (canonical name + element offset) copied straight out of the lowered `Expr`, never a slot, so a compiled fragment says nothing about where its variables live and `symbolic::resolve_module` assigns every address exactly once, at assembly. `ModuleCtx` is the compiler's WHOLE input contract -- a borrowed struct holding exactly what `Compiler` reads (`ident`, `inputs`, `temp_sizes`, the three runlists, `var_sizes`, `tables`, `dimensions`, `dimensions_ctx`) and nothing more; there is deliberately no offset map and no slot count, because the only thing codegen still needs from the symbol table is a variable's EXTENT (`full_source_len`, the VECTOR ELM MAP bound). `VarSizes` is keyed by the whole `VarRef`, not by name, and `context::whole_variable_extents` is its only constructor -- shared with lowering, which reads the same table through `ContextCore::var_sizes` for the GH #578 constant-offset ELM MAP fold, so the fold and the opcode cannot disagree about where a source's storage ends. Keying by reference is what makes a CROSS-MODULE source right: `m·x` lowers to `VarRef { name: m, element_offset: x's slot inside the instance }`, so a name-keyed lookup answered with the module instance's whole block size, and reads past `x`'s end silently landed on the next sub-model variable instead of yielding `:NA:`. A module instance therefore contributes no entry of its own and one per sub-model variable at that variable's slot, recursively through nested instances (`array_tests::cross_module_array_reference_tests`). Both callers build one: the `#[cfg(test)]` monolithic `Module::compile` borrows every field off its owned `Module` and then resolves the emitted symbolic module against its own layout, and the production per-variable fragment compiler borrows the salsa-cached project-global dimension context/converted dims and keeps the fragment symbolic until assembly. The `#[cfg(test)]` gate on `Module` is a load-bearing assertion, not tidiness: it makes "no production `compiler::Module` literal remains" (GH #964) a compile error rather than a claim. There is no un-fused `Opcode::AssignNext`: a stock update is emitted straight as the fused `BinOpAssignNext`, and `Var::new`'s `check_stock_updates_are_emittable` rejects any update whose lowered form does not end in an `Op2` with a per-variable `NotSimulatable` -- so an eventual `non_negative` (GH #545) clamp that wrapped the update in `MAX` surfaces as a diagnostic instead of a stock that silently never integrates. `emit_array_reduce()` is the shared helper for single-argument array builtins (SUM, SIZE, STDDEV, MIN, MAX, MEAN): pushes view, emits reduction opcode, pops view + - `codegen.rs` - Bytecode emission; routes array-producing builtins through dedicated opcodes instead of element-wise iteration. Codegen emits **symbolic** opcodes: a variable operand is a `compiler::VarRef` (canonical name + element offset) copied straight out of the lowered `Expr`, never a slot, so a compiled fragment says nothing about where its variables live and `symbolic::resolve_module` assigns every address exactly once, at assembly. `ModuleCtx` is the compiler's WHOLE input contract -- a borrowed struct holding exactly what `Compiler` reads (`ident`, `inputs`, `temp_sizes`, the three runlists, `var_sizes`, `tables`, `dimensions`, `dimensions_ctx`) and nothing more; there is deliberately no offset map and no slot count, because the only thing codegen still needs from the symbol table is a variable's EXTENT (`full_source_len`, the VECTOR ELM MAP bound). `VarSizes` is keyed by the whole `VarRef`, not by name, and `context::whole_variable_extents` is its only constructor -- shared with lowering, which reads the same table through `ContextCore::var_sizes` for the GH #578 constant-offset ELM MAP fold, so the fold and the opcode cannot disagree about where a source's storage ends. Keying by reference is what makes a CROSS-MODULE source right: `m·x` lowers to `VarRef { name: m, element_offset: x's slot inside the instance }`, so a name-keyed lookup answered with the module instance's whole block size, and reads past `x`'s end silently landed on the next sub-model variable instead of yielding `:NA:`. A module instance therefore contributes no entry of its own and one per sub-model variable at that variable's slot, recursively through nested instances (`array_tests::cross_module_array_reference_tests`). Both callers build one: the `#[cfg(test)]` monolithic `Module::compile` borrows every field off its owned `Module` and then resolves the emitted symbolic module against its own layout, and the production per-variable fragment compiler borrows the salsa-cached project-global dimension context/converted dims and keeps the fragment symbolic until assembly. The `#[cfg(test)]` gate on `Module` is a load-bearing assertion, not tidiness: it makes "no production `compiler::Module` literal remains" (GH #964) a compile error rather than a claim. There is no un-fused `Opcode::AssignNext`: a stock update is emitted straight as the fused `BinOpAssignNext`, and `Var::new`'s `check_stock_updates_are_emittable` rejects any update whose lowered form does not end in an `Op2` with a per-variable `NotSimulatable` -- so an eventual `non_negative` (GH #545) clamp that wrapped the update in `MAX` surfaces as a diagnostic instead of a stock that silently never integrates. `emit_array_reduce()` is the shared helper for single-argument array builtins (SUM, SIZE, STDDEV, MIN, MAX, MEAN): pushes view, emits reduction opcode, pops view. `walk_expr_as_view` accepts a FIFTH shape beyond the four storage views (GH #995): an array-valued `PREVIOUS`/`INIT`, which `snapshot_static_view` turns into the argument's view over a snapshot region. How permissive that is depends on the POSITION (`SnapshotPosition`), and the two arms are different questions. In a **view operand** the caller has already said an array is required, so every argument that lowered to a view goes through the view -- a single-element one included, which is what makes `VECTOR ELM MAP(PREVIOUS(vals[1]), offs)` behave exactly as `VECTOR ELM MAP(vals[1], offs)` does (the element establishes the base and the mapping ranges over the whole source variable; `full_source_len` looks THROUGH the `PREVIOUS` for that extent). The NUMERIC index is what that coherence claim is about: the same element spelled with its bare NAME (`vals[e1]`) never reaches this arm, because `builtins_visitor::index_is_static` will not accept an unqualified element name on the user-equation parse path, so `PREVIOUS` reads a scalar capture helper of extent ONE and the mapping is confined to it. Inside a **`BeginIter` body** the position is scalar, so only an array-valued call takes the view route and a single-element argument keeps compiling to `LoadPrev`/`LoadInitial` -- which is what lets `PREVIOUS(matrix[E,1])` go on broadcasting. `collect_iter_source_views_impl` must pre-push exactly the views the body reads, so it asks the same question the body's `walk_expr` will; a shape the region view cannot express (a temp, which has no snapshot) contributes no view there and surfaces as the propagated `Err` when `walk_expr` reaches the same node - `dimensions.rs` - Dimension checking/inference - `subscript.rs` - Array subscript expansion and iteration - `pretty.rs` - Debug pretty-printing - - `symbolic.rs` - Layout-independent symbolic bytecode layer (the backbone of incremental compilation): opcodes reference variables by name (`SymVarRef`, an alias for `compiler::VarRef { name: Ident, element_offset }`) rather than model-global offset, so salsa caches a per-variable `PerVarBytecodes` fragment that survives variable add/remove. Pipeline: lowered `Expr` (names) → codegen → `SymbolicByteCode` → `resolve` → concrete bytecode. Addresses travel in ONE direction and are assigned ONCE: there is no symbolization pass, no reverse offset map, and no per-fragment layout (GH #964). The bytecode builder and its peephole optimizer live here too (`SymbolicByteCodeBuilder`, tested in the sibling `symbolic_builder_tests.rs`): both are address-independent, so running them before resolution keeps `resolve_bytecode` a strict 1:1 mapping -- which the run-invariant flow-prefix boundary and the SCC per-element segmentation both depend on. `resolve_bytecode` is the SOLE producer of concrete bytecode and therefore where the VM's fixed-stack safety proof is discharged (a program deeper than `STACK_CAPACITY` is a compile error, not an abort). Sixteen `SymbolicOpcode` variants carry `#[allow(dead_code)]`: codegen constructs none of them, which the dead-code lint now proves (codegen is the only producer of a `SymbolicOpcode`); they are the superseded halves of incremental view-stack construction and broadcast iteration, and retiring them plus their `Opcode` twins, VM arms and wasm arms is sequenced as its own change. `FragmentMerger` is the shared core of `concatenate_fragments_with_gf` (the sequential phase concat) and `combine_scc_fragment` (the interleaved multi-member SCC fragment). Its contract is written out on the type as eight obligations M1-M8, stated as properties of the MERGED FRAGMENT rather than as parity with any other compiler -- referential integrity (a renumbered opcode names the same resource value it named in its own fragment), disjointness and tiling of the flat resources, id-type capacity, GF sharing by content only, temp non-aliasing, 1:1 opcode preservation, `SymVarRef`s untouched, and agreement between the per-phase renumber and the all-phases merge. M3 (id-type capacity) is about ASSIGNED ids, so the counts that become a later phase's base are carried in `usize` and the `u16` bound is discharged in exactly one function, `resource_base` -- the only point that sees both a base and the length of the fragment about to consume it. Stating it per place instead left the three places disagreeing: a table of exactly 65,536 entries is addressable and the merger accepted it, while the cross-phase and initials-phase narrowings rejected it, so a model whose every id was valid failed to assemble. "Assigned" also means assigned to something the module KEEPS, which is why the all-phases aggregation of the shared context tables is `merge_context_side_channels` rather than a full merge: each phase retains its own literal pool (every compiled initial keeps one; flows and stocks keep one each), so an aggregate pool over all three is discarded -- and bounding it failed assembly for models around 33k scalar stocks whose every retained pool was well inside the limit. Each names the test that pins it; the property tests are `src/compiler/symbolic_merge_proptest.rs` (sequential) and `src/db/combined_fragment_proptest.rs` (interleaved). `TempStrategy { Recycle, Sum }` is how a caller DECLARES its emission shape: `Recycle` collapses temps by identity (safe because the sequential concat emits each fragment as one contiguous run, and necessary because summing 0-based per-fragment counts overflows the `u8` `TempId`, #583), `Sum` gives each fragment a disjoint range (required because the SCC interleave makes members' live ranges overlap). `GfDedup` content-de-duplicates graphical-function blocks across fragments (#582), so a dependency arrayed GF re-extracted by N consumers is laid out once and every consumer's `base_gf` is remapped to it. -6. **`src/bytecode.rs`** - Instruction set definition, opcodes, type aliases (`LiteralId`, `ModuleId`, `DimId`, `TempId`, etc.). Includes `LoadPrev`/`LoadInitial` opcodes for `PREVIOUS()`/`INIT()` intrinsics, the scalar `Lookup` opcode plus `LookupArray` (per-element arrayed graphical function `g[D!](index)`: pops the shared scalar index, reads the arrayed GF's full storage view, evaluates `graphical_functions[base_gf + i]` per element into a temp array view -- so a wrapping reducer / vector op such as VECTOR SELECT applies over the arrayed-GF result; out-of-range element ⇒ NaN like scalar `Lookup`; GH #580), and vector operation opcodes (`VectorSelect`, `VectorElmMap`, `VectorSortOrder`, `Rank`, `AllocateAvailable`, `AllocateByPriority`) that operate on view-stack arrays and write results to temp storage. There is deliberately no un-fused `AssignNext` and no `PushVarView`: codegen cannot emit either (a stock update always ends in an `Op2`, so it is fused into `BinOpAssignNext` by `symbolic::SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next` at emit time; a full-array variable view is always a `PushStaticView` or `PushVarViewDirect`), and carrying them cost a VM arm, a wasm arm, and two `SymbolicOpcode` variants apiece for nothing. -7. **`src/vm.rs`** - Stack-based bytecode VM. Hot loop uses proven-safe unchecked array access validated at compile time by `compiler::symbolic::resolve_bytecode` -- the single place concrete bytecode is produced, so nothing reaches the VM unchecked. Both failure modes (over-depth, and a `stack_effect` metadata underflow) are reported as a compile `Err`, so an unprovable program is rejected rather than executed. Maintains `prev_values` and `initial_values` snapshot buffers for `LoadPrev`/`LoadInitial` opcodes. Implements vector operation dispatch (VectorSelect, VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) and the per-element arrayed-GF `LookupArray`. `Opcode::VectorElmMap` and `Opcode::VectorSortOrder` dispatch into the sibling helper modules below (extracted purely for the per-file line cap). Array reducers (ArrayMax, ArrayMin, ArrayMean, ArrayStddev) return NaN for empty views; ArraySum returns 0.0 (additive identity). When conveyor/queue plans are attached, the VM runs the special-stock side-table passes each step (`init_belts` / `publish_container_values` / `run_coupled_passes`); `get_value` mid-run therefore previews the resting `curr` on CLONED side tables rather than reading a pass-driven flow's placeholder zero. The reset/`run_to`/constant-override tests live in the sibling **`vm_reset_run_to_and_constants_tests.rs`** (split out for the per-file line cap). + - `symbolic.rs` - Layout-independent symbolic bytecode layer (the backbone of incremental compilation): opcodes reference variables by name (`SymVarRef`, an alias for `compiler::VarRef { name: Ident, element_offset }`) rather than model-global offset, so salsa caches a per-variable `PerVarBytecodes` fragment that survives variable add/remove. Pipeline: lowered `Expr` (names) → codegen → `SymbolicByteCode` → `resolve` → concrete bytecode. Addresses travel in ONE direction and are assigned ONCE: there is no symbolization pass, no reverse offset map, and no per-fragment layout (GH #964). The bytecode builder and its peephole optimizer live here too (`SymbolicByteCodeBuilder`, tested in the sibling `symbolic_builder_tests.rs`): both are address-independent, so running them before resolution keeps `resolve_bytecode` a strict 1:1 mapping -- which the run-invariant flow-prefix boundary and the SCC per-element segmentation both depend on. `resolve_bytecode` is the SOLE producer of concrete bytecode and therefore where the VM's fixed-stack safety proof is discharged (a program deeper than `STACK_CAPACITY` is a compile error, not an abort). `SymStaticViewBase` is where a view's base lives before layout assignment, and it has four arms: `Var`, `PrevVar`, `InitialVar` (GH #995 -- the same `SymVarRef` read out of `curr`, `prev_values` or `initial_values`, which share `curr`'s slot numbering, so one layout lookup serves all three) and `Temp`. Splitting the regions into distinct variants rather than pairing one `Var` with a storage field keeps `Temp` + a snapshot region unrepresentable: a temp has no snapshot. The sibling `resolve_static_view` discharges the other one-way narrowing: a static view's base is the ONLY place a temp id travels as a `u32`, while every other opcode naming a temp (`BeginIter`/`write_temp_id`, `LoadTempConst`) carries it as `TempId` (= `u8`), narrowed at emit time with a plain `as`, so a view over a temp above 255 reads storage no opcode wrote -- a well-formed program with wrong numbers and no diagnostic. It is rejected there. This was reachable and silently wrong before the guard: a per-element hoist allocates one temp per array ELEMENT, so `order[D] = VECTOR SORT ORDER(301 - vals[*], dir[D])` over a 130-element dimension returned a different array from element 128 on. The narrowing on the WRITE side is deliberately NOT guarded, but the reason is NARROWER than "truncation is harmless": a temp only ever written and read by `write_temp_id`/`LoadTempConst` truncates identically on both sides, and when the aliased temps are the SAME SIZE -- which is what a per-element hoist over one array produces -- each element's temp is written immediately before it is read and the aliasing is unobservable. It is NOT safe in general: temps of different sizes sharing a truncated id let a larger write run past the smaller slot into its neighbour's storage, which is in-bounds for the flat temp region and therefore silent. No lowering path emits that today (a fragment reaching 256 temps does so through one repeated per-element shape), and pinning what IS relied on -- pinned by `array_operand_materialization_tests::a_per_element_hoist_past_the_temp_namespace_without_a_temp_view_is_correct` -- is as far as this goes. #583 (widen the id namespace) is the real fix for both halves. Sixteen `SymbolicOpcode` variants carry `#[allow(dead_code)]`: codegen constructs none of them, which the dead-code lint now proves (codegen is the only producer of a `SymbolicOpcode`); they are the superseded halves of incremental view-stack construction and broadcast iteration, and retiring them plus their `Opcode` twins, VM arms and wasm arms is sequenced as its own change. `FragmentMerger` is the shared core of `concatenate_fragments_with_gf` (the sequential phase concat) and `combine_scc_fragment` (the interleaved multi-member SCC fragment). Its contract is written out on the type as eight obligations M1-M8, stated as properties of the MERGED FRAGMENT rather than as parity with any other compiler -- referential integrity (a renumbered opcode names the same resource value it named in its own fragment), disjointness and tiling of the flat resources, id-type capacity, GF sharing by content only, temp non-aliasing, 1:1 opcode preservation, `SymVarRef`s untouched, and agreement between the per-phase renumber and the all-phases merge. M3 (id-type capacity) is about ASSIGNED ids, so the counts that become a later phase's base are carried in `usize` and the `u16` bound is discharged in exactly one function, `resource_base` -- the only point that sees both a base and the length of the fragment about to consume it. Stating it per place instead left the three places disagreeing: a table of exactly 65,536 entries is addressable and the merger accepted it, while the cross-phase and initials-phase narrowings rejected it, so a model whose every id was valid failed to assemble. "Assigned" also means assigned to something the module KEEPS, which is why the all-phases aggregation of the shared context tables is `merge_context_side_channels` rather than a full merge: each phase retains its own literal pool (every compiled initial keeps one; flows and stocks keep one each), so an aggregate pool over all three is discarded -- and bounding it failed assembly for models around 33k scalar stocks whose every retained pool was well inside the limit. Each names the test that pins it; the property tests are `src/compiler/symbolic_merge_proptest.rs` (sequential) and `src/db/combined_fragment_proptest.rs` (interleaved). `TempStrategy { Recycle, Sum }` is how a caller DECLARES its emission shape: `Recycle` collapses temps by identity (safe because the sequential concat emits each fragment as one contiguous run, and necessary because summing 0-based per-fragment counts overflows the `u8` `TempId`, #583), `Sum` gives each fragment a disjoint range (required because the SCC interleave makes members' live ranges overlap). `GfDedup` content-de-duplicates graphical-function blocks across fragments (#582), so a dependency arrayed GF re-extracted by N consumers is laid out once and every consumer's `base_gf` is remapped to it. +6. **`src/bytecode.rs`** - Instruction set definition, opcodes, type aliases (`LiteralId`, `ModuleId`, `DimId`, `TempId`, etc.). `ViewStorage` is which of the VM's parallel f64 regions a view reads -- `Curr`, `Temp`, `Prev`, `Initial` -- carried on both `StaticArrayView` and `RuntimeView`. It is also what decides whether `StaticArrayView::to_runtime_view` adds the executing instance's `module_off` to the base: a static view's `base_off` comes from the FRAGMENT'S OWN model layout (`symbolic::resolve_static_view`) and is module-relative like every other offset in that fragment, so the three chunk-shaped regions take the addend and `Temp` -- a temp id, not a slab slot, in per-evaluation storage every instance shares -- does not. Without it every array reduction inside a sub-model instance read the ROOT's slots (both instances of the same sub-model returning `time + dt + initial_time`); the defect predates the snapshot regions and both backends had it, which is why the pins assert absolute series rather than VM/wasm parity. What was broken is narrower than "array views in models with modules": a view pushed while EXECUTING INSIDE an instance. A cross-module read taken FROM THE ROOT (a root equation reading `m·x`) was always correct, because the root's `module_off` is 0 and the dropped addend is invisible there -- which is why `array_tests::cross_module_array_reference_tests` passed throughout. The pins therefore drive two INSTANCES and, through the nested twin, two HOPS, since a one-hop fixture cannot distinguish accumulating the addend down the chain from applying only the last hop (`array_operand_materialization_tests::an_array_view_inside_a_module_instance_reads_that_instance`, `wasmgen::module_tests::compile_simulation_arrayed_submodel_views_address_their_instance`). It replaced an `is_temp: bool` when GH #995 added the snapshot regions, which made every dereference site a compile error until it said which region it meant; the three chunk-shaped regions share `curr`'s slot numbering (each is an `n_slots` copy), so only the backing slice changes and `Temp` stays the odd one out (its `base_off` is a temp id resolved through `temp_offsets`). Includes `LoadPrev`/`LoadInitial` opcodes for the SCALAR `PREVIOUS()`/`INIT()` intrinsics -- the array forms are a `PushStaticView` over `Prev`/`Initial` instead -- the scalar `Lookup` opcode plus `LookupArray` (per-element arrayed graphical function `g[D!](index)`: pops the shared scalar index, reads the arrayed GF's full storage view, evaluates `graphical_functions[base_gf + i]` per element into a temp array view -- so a wrapping reducer / vector op such as VECTOR SELECT applies over the arrayed-GF result; out-of-range element ⇒ NaN like scalar `Lookup`; GH #580), and vector operation opcodes (`VectorSelect`, `VectorElmMap`, `VectorSortOrder`, `Rank`, `AllocateAvailable`, `AllocateByPriority`) that operate on view-stack arrays and write results to temp storage. There is deliberately no un-fused `AssignNext` and no `PushVarView`: codegen cannot emit either (a stock update always ends in an `Op2`, so it is fused into `BinOpAssignNext` by `symbolic::SymbolicByteCodeBuilder::fuse_trailing_op2_into_assign_next` at emit time; a full-array variable view is always a `PushStaticView` or `PushVarViewDirect`), and carrying them cost a VM arm, a wasm arm, and two `SymbolicOpcode` variants apiece for nothing. +7. **`src/vm.rs`** - Stack-based bytecode VM. Hot loop uses proven-safe unchecked array access validated at compile time by `compiler::symbolic::resolve_bytecode` -- the single place concrete bytecode is produced, so nothing reaches the VM unchecked. Both failure modes (over-depth, and a `stack_effect` metadata underflow) are reported as a compile `Err`, so an unprovable program is rejected rather than executed. Maintains `prev_values` and `initial_values` snapshot buffers for `LoadPrev`/`LoadInitial` and for the `ViewStorage::Prev`/`Initial` views an ARRAY-valued `PREVIOUS`/`INIT` pushes (GH #995). `PushStaticView` folds the runtime `module_off` into the view's base for those regions and for `Curr`, mirroring `LoadVar`/`LoadPrev`, so a view inside a sub-model instance addresses that instance (see `bytecode.rs` above). `ChunkRegions` is the read-only bundle of the three chunk-shaped regions plus the two pieces of run state that say what a snapshot read means before its snapshot exists; `temp_storage` is deliberately not in it, since an array-producing opcode writes through it while reading views. Its `backing` is the single place a view's region is resolved, and its two branches MIRROR the scalar opcodes rather than restating them: a `Prev` view while `use_prev_fallback` is set reads the `PREVIOUS` fallback for every element (`None`, i.e. 0 -- which is why `codegen::is_default_previous_fallback` refuses any other fallback on an array `PREVIOUS`), and an `Initial` view during the initials phase reads `curr`, because the snapshot has not been captured yet. Making the fallback a BRANCH rather than a reliance on `prev_values` being zero-filled is what lets the wasm backend -- whose `reset` does not clear the snapshot regions -- mirror it with the same `select` its scalar `LoadPrev` emits. Implements vector operation dispatch (VectorSelect, VectorElmMap, VectorSortOrder, Rank, AllocateAvailable, AllocateByPriority) and the per-element arrayed-GF `LookupArray`. `Opcode::VectorElmMap` and `Opcode::VectorSortOrder` dispatch into the sibling helper modules below (extracted purely for the per-file line cap). Array reducers (ArrayMax, ArrayMin, ArrayMean, ArrayStddev) return NaN for empty views; ArraySum returns 0.0 (additive identity). When conveyor/queue plans are attached, the VM runs the special-stock side-table passes each step (`init_belts` / `publish_container_values` / `run_coupled_passes`); `get_value` mid-run therefore previews the resting `curr` on CLONED side tables rather than reading a pass-driven flow's placeholder zero. The reset/`run_to`/constant-override tests live in the sibling **`vm_reset_run_to_and_constants_tests.rs`** (split out for the per-file line cap). - **`src/vm_vector_sort_order.rs`** - Genuine-Vensim VECTOR SORT ORDER. Ranks WITHIN each currently-iterated source slice (the innermost/last-declared dim is the sorted axis; outer dims select independent rows), 0-based: result position `j` of a row holds the 0-based source index *within that row* of its `j`-th element in sorted order (`direction == 1` ascending, else descending; stable ties). A 1-D view is the degenerate single-row case (in-row ranks == whole-view ranks). The prior whole-flattened-view absolute-index behavior (GH #585) made a multi-row source feed out-of-range flat indices into a downstream single-column ELM MAP; ground truth is real Vensim DSS `/test/test-models/tests/vector_order/output.tab` (ranks include `0`, impossible for a 1-based permutation). RANK is a distinct, correctly 1-based opcode. - - **`src/vm_vector_elm_map.rs`** - Genuine-Vensim VECTOR ELM MAP: result element `i` = `source[base_i + round(offset[i])]` over the source variable's FULL row-major contiguous storage, where `base_i` is the flat position arg-1's element reference establishes and the offset steps the source's innermost dim (stride 1). An offset+base outside `[0, full_source_len)`, or a NaN offset, yields genuine IEEE NaN (the out-of-range result Vensim documents as `:NA:`; this is the absorbing NaN, NOT the finite `crate::float::NA` sentinel). NO modulo / NO wraparound (the bug the prior sliced-view-no-base implementation had). + - **`src/vm_vector_elm_map.rs`** - Genuine-Vensim VECTOR ELM MAP: result element `i` = `source[base_i + round(offset[i])]` over the source variable's FULL row-major contiguous storage, where `base_i` is the flat position arg-1's element reference establishes and the offset steps the source's innermost dim (stride 1). That rule is DOCUMENTED and ground-truthed, not inferred: the Vensim reference page (retrieved 2026-08-02) says the function "returns the value of the variable that is offset from vec by the specified amount" and that an offset "outside the range of the variable" yields `:NA:`, and its multi-subscript example spells the offset as a flat index over the whole variable; real Vensim output in `test/sdeverywhere/models/vector/` confirms the cross-row read (`f[DimA,DimB] = VECTOR ELM MAP(d[DimA,B1], a[DimA])` prints `1,1,5,5,6,6`, where `f[A2,B1] = 5 = d[A2,B2]` -- past its own `B1` slice). A COMPUTED source is a Simlin EXTENSION, and measurement settled that it is one rather than a guess at Vensim's rule: run in Vensim DSS 2026-08-04, `vensim-probes/elm_map_computed_source.mdl` refuses to simulate with "Argument 1 to function VECTOR ELM MAP must be a normal variable". With no Vensim behaviour to match, the extension is DEFINED by helper-equivalence -- an inline expression means what the same values pre-assigned to a named variable mean, the spelling that IS legal Vensim -- and `compiler::array_operand` implements exactly that by confining a materialized source to its (full-array-by-construction) temp. One variable-source spelling is still open and is NOT covered by `vector.dat`: for the whole-variable `VECTOR ELM MAP(helper[DimA], offs)` this engine collapses the base to 0, where base-from-reference predicts per-element bases and two `:NA:`s; `vensim-probes/elm_map_variable_sources.mdl` asks it. An offset+base outside `[0, full_source_len)`, or a NaN offset, yields genuine IEEE NaN (the out-of-range result Vensim documents as `:NA:`; this is the absorbing NaN, NOT the finite `crate::float::NA` sentinel). NO modulo / NO wraparound (the bug the prior sliced-view-no-base implementation had). 8. **`src/alloc.rs`** - Allocation helpers for VM priority allocation: `allocate_available()` (bisection-based priority allocation), `alloc_curve()` (per-requester allocation curves for 6 profile types), `normal_cdf()`/`erfc_approx()`. -9. **`src/wasmgen/`** - WebAssembly code-generation backend: an alternative execution path to the bytecode VM (item 7) that lowers the salsa-compiled `CompiledSimulation` to one self-contained wasm module (no host imports), mirroring the VM opcode-for-opcode. Intended for fast repeated re-simulation (e.g. interactive parameter scrubbing): a host instantiates the blob once and calls its exported `run` on every change. **The bytecode VM remains the correctness oracle** -- every emitted module is executed under the pure-Rust DLR-FT `wasm-interpreter` in tests and compared against `Vm::run_to_end`. Entry point `compile_simulation(&CompiledSimulation) -> WasmArtifact { wasm: Vec, layout: WasmLayout }`; the blob exports `memory`, `run`, the resumable run pair `run_to(target)`/`run_initials` (mirroring `vm.rs::run_to`/`run_initials`; `run` re-expresses as `reset` then `run_to(stop)`), the geometry globals `n_slots`/`n_chunks`/`results_offset` (step-major results), `set_value`/`reset`/`clear_values` (constant-override semantics matching the VM, sourced from a mutable const-override region indexed by absolute slot), and `get_error()` (the runtime error channel, GH #921). The per-step run cursor (`saved`/`step_accum`/`did_initials`) lives in *internal* mutable wasm globals (not exported) so a run survives across separate exported calls -- `run_initials` runs once and each `run_to` resumes from where the prior call stopped (a `run_to` that resumes on an already-complete slab is a no-op: the loop breaks at the top when `saved >= n_chunks`, so it can never write past the `n_chunks`-row results region). The blob owns the live `curr` chunk's presentation end-to-end, so a host needs no shadow writes: `set_value` mirrors the override into curr (like the VM's `set_value_now`), and `reset` clears the cursor AND re-establishes the fresh pre-run curr (zeroed, with explicitly-overridden constants reapplied -- tracked by a `const_override_set` marker region distinct from the validity region), matching libsimlin's recreate-and-reapply reset; constant overrides persist across `reset` and are dropped by `clear_values`. `WasmLayout` (canonical-name -> slot offset) lets a host read one variable's series by striding the results region. Coverage is the full core-simulation surface: every scalar opcode + builtin (transcendentals open-coded as wasm helpers, so the blob needs no math imports), arrays (subscripts, iteration, reducers, dynamic subscripts with OOB->NaN), graphical-function lookups (scalar + per-element `LookupArray`), the vector ops (`VectorSelect`/`VectorElmMap`/`VectorSortOrder`/`Rank`) and market-clearing allocation (`AllocateAvailable`/`AllocateByPriority`), Euler/RK2/RK4 integration, `PREVIOUS`/`INIT`, and nested modules (one set of initials/flows/stocks functions per `(model, input_set)` instance, addressed by a runtime `module_off`). LTM is lowered alongside the core simulation, so an `enableLtm` wasm sim produces the same per-step link-score slab the VM does and the shared analytic core (`simlin_analyze_links_from_wasm_results`) reads it (see [`src/engine/CLAUDE.md`](../engine/CLAUDE.md) for the @simlin/engine wiring); a genuinely-unlowerable LTM (or core-simulation) construct surfaces as a `WasmGenError::Unsupported` error with **no silent VM fallback**, identical to the non-LTM path. The remaining unlowerable constructs are: a true-runtime-range subscript (`ViewRangeDynamic`, GH #612) returns `Unsupported`; array unrolling is bounded by `MAX_UNROLL_UNITS` (65,536 elements/function), above which a model returns `Unsupported` for the caller to surface as an explicit error (instead of emitting a multi-megabyte function body that a wasm engine would reject); and a conveyor inflow carrying a non-default `isee:spreadflow` -- `even`, `dest`, `dist`, or `source` (GH #946) -- returns `Unsupported` from `belt::reject_unsupported` rather than mis-lowering. **Both special stock types lower** (GH #884, closed by #924): `compile_datamodel_to_artifact`/`compile_datamodel_to_wasm` route a conveyor or queue model through the same unified dispatch the VM uses (`queue_compile::compile_sim`), so its driven flows and container auxes are compiled from the *expanded* project and its per-step pass is emitted as unrolled, plan-specialized wasm (`belt.rs` for the belt, `passes.rs` for the FIFO). There is no up-front marker reject and no `#[cfg(test)]` seam: the entry `libsimlin`'s `simlin_model_compile_to_wasm` calls is the entry the parity tests exercise, and `test/conveyors/`+`test/queues/` are gated end-to-end on the two backends agreeing (`tests/integration/simulate.rs`, `simulate_special_path`). +9. **`src/wasmgen/`** - WebAssembly code-generation backend: an alternative execution path to the bytecode VM (item 7) that lowers the salsa-compiled `CompiledSimulation` to one self-contained wasm module (no host imports), mirroring the VM opcode-for-opcode. Intended for fast repeated re-simulation (e.g. interactive parameter scrubbing): a host instantiates the blob once and calls its exported `run` on every change. **The bytecode VM remains the correctness oracle** -- every emitted module is executed under the pure-Rust DLR-FT `wasm-interpreter` in tests and compared against `Vm::run_to_end`. Entry point `compile_simulation(&CompiledSimulation) -> WasmArtifact { wasm: Vec, layout: WasmLayout }`; the blob exports `memory`, `run`, the resumable run pair `run_to(target)`/`run_initials` (mirroring `vm.rs::run_to`/`run_initials`; `run` re-expresses as `reset` then `run_to(stop)`), the geometry globals `n_slots`/`n_chunks`/`results_offset` (step-major results), `set_value`/`reset`/`clear_values` (constant-override semantics matching the VM, sourced from a mutable const-override region indexed by absolute slot), and `get_error()` (the runtime error channel, GH #921). The per-step run cursor (`saved`/`step_accum`/`did_initials`) lives in *internal* mutable wasm globals (not exported) so a run survives across separate exported calls -- `run_initials` runs once and each `run_to` resumes from where the prior call stopped (a `run_to` that resumes on an already-complete slab is a no-op: the loop breaks at the top when `saved >= n_chunks`, so it can never write past the `n_chunks`-row results region). The blob owns the live `curr` chunk's presentation end-to-end, so a host needs no shadow writes: `set_value` mirrors the override into curr (like the VM's `set_value_now`), and `reset` clears the cursor AND re-establishes the fresh pre-run curr (zeroed, with explicitly-overridden constants reapplied -- tracked by a `const_override_set` marker region distinct from the validity region), matching libsimlin's recreate-and-reapply reset; constant overrides persist across `reset` and are dropped by `clear_values`. `WasmLayout` (canonical-name -> slot offset) lets a host read one variable's series by striding the results region. Coverage is the full core-simulation surface: every scalar opcode + builtin (transcendentals open-coded as wasm helpers, so the blob needs no math imports), arrays (subscripts, iteration, reducers, dynamic subscripts with OOB->NaN), graphical-function lookups (scalar + per-element `LookupArray`), the vector ops (`VectorSelect`/`VectorElmMap`/`VectorSortOrder`/`Rank`) and market-clearing allocation (`AllocateAvailable`/`AllocateByPriority`), Euler/RK2/RK4 integration, `PREVIOUS`/`INIT` in both their scalar and their ARRAY forms (GH #995: `views::ViewBase` gained `Prev`/`Initial` and every addressing site takes the region bases as one `RegionBases`, so a new region is added in one place. All three chunk-shaped bases (`Curr`/`Prev`/`Initial`) are MODULE-RELATIVE -- the emitter adds `module_off * 8`, mirroring the VM's fold at push time -- and `Temp` is not; the arms are named for their region rather than for an absolute/relative split because there is no longer a static view that bakes an absolute base. `Initial` resolves its "during initials the snapshot does not exist yet, read `curr`" branch at COMPILE time from `EmitCtx::step_part`, exactly as `emit_load_initial` does; `Prev` carries a RUNTIME `select` on `use_prev_fallback`, the same one `emit_load_prev` emits, and that select is load-bearing rather than defensive -- the blob's `reset` deliberately does not clear the snapshot regions, so without it a SECOND `run` reads the first run's final `prev_values` on step 0. Pinned by `module_tests::compile_simulation_repeated_run_resets_previous_fallback_for_an_array_view`, which the corpus gate cannot reach because a first run over zeroed memory looks correct either way), and nested modules (one set of initials/flows/stocks functions per `(model, input_set)` instance, addressed by a runtime `module_off`). LTM is lowered alongside the core simulation, so an `enableLtm` wasm sim produces the same per-step link-score slab the VM does and the shared analytic core (`simlin_analyze_links_from_wasm_results`) reads it (see [`src/engine/CLAUDE.md`](../engine/CLAUDE.md) for the @simlin/engine wiring); a genuinely-unlowerable LTM (or core-simulation) construct surfaces as a `WasmGenError::Unsupported` error with **no silent VM fallback**, identical to the non-LTM path. The remaining unlowerable constructs are: a true-runtime-range subscript (`ViewRangeDynamic`, GH #612) returns `Unsupported`; array unrolling is bounded by `MAX_UNROLL_UNITS` (65,536 elements/function), above which a model returns `Unsupported` for the caller to surface as an explicit error (instead of emitting a multi-megabyte function body that a wasm engine would reject); and a conveyor inflow carrying a non-default `isee:spreadflow` -- `even`, `dest`, `dist`, or `source` (GH #946) -- returns `Unsupported` from `belt::reject_unsupported` rather than mis-lowering. **Both special stock types lower** (GH #884, closed by #924): `compile_datamodel_to_artifact`/`compile_datamodel_to_wasm` route a conveyor or queue model through the same unified dispatch the VM uses (`queue_compile::compile_sim`), so its driven flows and container auxes are compiled from the *expanded* project and its per-step pass is emitted as unrolled, plan-specialized wasm (`belt.rs` for the belt, `passes.rs` for the FIFO). There is no up-front marker reject and no `#[cfg(test)]` seam: the entry `libsimlin`'s `simlin_model_compile_to_wasm` calls is the entry the parity tests exercise, and `test/conveyors/`+`test/queues/` are gated end-to-end on the two backends agreeing (`tests/integration/simulate.rs`, `simulate_special_path`). **Two error channels, at two different times.** `WasmGenError` is COMPILE-time: the backend refuses to emit a module it cannot lower correctly. `errors.rs` is the emitted module's RUN-time channel (GH #921), the prerequisite for lowering the conveyor belt pass (the only pass with a per-step runtime error, `ConveyorTransitTooLong`; `init_belts` additionally raises `ConveyorTransitNotPositive`). Every blob carries two internal mutable i32 globals -- `G_ERR_CODE` (0 = ok, else `ErrorCode as i32`; `NoError` is discriminant 0 and never produced) and `G_ERR_BELT` (plan-list index) -- and exports `get_error() -> i64` = `(belt << 32) | code`. The blob builds no strings; the HOST rebuilds the VM's exact message with `wasmgen::reconstruct_error(word, plans, dt, read_curr_slot)`, which reads the belt's name from `ConveyorPlan::name` and its transit time from `curr[plan.len_off]` (no pass writes that slot) and delegates to the *same* `conveyor_compile::transit_too_long_error`/`transit_not_positive_error` constructors the VM calls -- so the two backends' text cannot drift. That transit slot is a synthesized belt-parameter aux, absent from the initials runlist and written only by the Flows phase, so `run_initials` evaluates Flows before the side-table init hook whenever a pass declares `Passes::needs_flows_before_init` (mirroring `vm.rs:1508-1514`); without it a belt would read a transit of 0 and raise a spurious `ConveyorTransitNotPositive`. Semantics: a raising pass sets the globals and `br`s out of the pass block (`errors::ErrorScope`, the unwind contract #922 is written against -- `br`, not `return`, because the pass body is emitted at both the real step site and the mid-run preview site, which must undo its clone). An `ErrorScope` **cannot be forged**: its sole constructor is `errors::open_pass_block`, which emits the block in the same breath, and a pass body receives `Option` (`None` where no block was opened) so that `errors::expect_scope` turns a driver that forgot to set `pass_can_error` into a loud emit-time panic. That matters because a `br` with no enclosing pass block still *validates* -- it resolves to `run_to`'s step loop, and the blob spins forever with no diagnostic at all. `run_to`'s step site then returns *before* Stocks, the `prev_values` snapshot, and the save/advance tail, so the failing step saves no row (matching `vm.rs:958-972`); `run_initials` returns before the container publish, the reconciliation, and setting `G_DID_INITIALS` (matching `vm.rs:1542-1548`); and the mid-run preview restores `curr` from a snapshot and CLEARS the channel, swallowing the failure exactly as `vm.rs:1187-1216` does. One deliberate divergence from the VM: the channel is **sticky** -- `run_initials`/`run_to` are no-ops until `reset` clears it -- because a wasm export cannot return a `Result` to force the caller's hand, and the VM's own resume-after-error re-runs Phase A over belts it already advanced. Since GH #924 lifted the conveyor reject, a shipped model CAN set the channel (a belt with a non-positive or out-of-bound transit); a queue-only or ordinary model cannot, and its blob elides the guards and the preview save region entirely, since they are emitted only when a pass can raise. `errors_tests.rs` proves the whole mechanism independently by splicing a synthetic raising pass (`FaultInjection`, uninhabited outside a test build) into the same two hook points. Hosts that RUN a blob must poll the getter, since the export cannot return a `Result`: the engine's integration harness does (`assert_no_runtime_error` after every `run`/`run_to` segment), and so does the TypeScript `DirectBackend` (`throwIfWasmRuntimeError` after each `run_to`, via the pure `decodeWasmError`). libsimlin compiles blobs but never runs one, so it polls nothing. A failed `memory.grow` in the queue pass's bump allocator still **traps** rather than routing through this channel -- see `passes::emit_alloc` for the reasoning (an OOM is resource exhaustion, not a model diagnostic; the VM aborts on it too; and `br` cannot cross the helper-call boundary). Files: - **`mod.rs`** - the `WasmGenError` error type + module re-exports. @@ -91,7 +92,7 @@ Diagnostics (`collect_all_diagnostics`) run on the user's `SourceProject`, so sy - **`src/errors.rs`** - Human-readable error formatting: `FormattedError`/`FormattedErrors`, `FormattedErrorKind`, `UnitErrorKind`. `format_diagnostic()` converts a salsa `Diagnostic` to `FormattedError`; `format_diagnostic_with_datamodel()` adds source snippets from the datamodel. `collect_formatted_errors()` is the bulk entry point that aggregates all diagnostics into a `FormattedErrors` value. `FormattedError.message` is TERMINAL-formatted (source snippet + `~~~` underline + model/variable summary line); the separate `details` field carries just the bare reason for unit errors (e.g. "computed units 'x' don't match specified units") so GUI consumers (via libsimlin's `SimlinErrorDetail.details`) can render it without the snippet noise. `FormattedError.severity` carries the originating `db::DiagnosticSeverity`, and the summary line's severity word is rendered FROM it (`severity_word`) rather than hardcoded (GH #919): a `Warning` -- the conveyor/queue LTM-degraded advisory, the conveyor spec advisories, a unit *consistency*/*inference* mismatch -- reads "warning in model ..." / "units warning in model ...", while an `Error` (an equation error, a unit *definition* syntax error) keeps "error in model ...". The word therefore tracks the diagnostic's severity, not the arm it came from, so no consumer that renders `message` verbatim can present an advisory as a compilation failure. `FormattedErrors::push` is the SOLE place `has_model_errors`/`has_variable_errors` are set, and it counts `Error` severity only -- those flags gate failure-shaped decisions (the CLI's redundant-`NotSimulatable` suppression), so a warning must never flip them. `format_simulation_error` has no `Diagnostic` behind it and is unconditionally `Error`. Canonical implementation shared by both `simlin-mcp` and `libsimlin` (which re-exports from here). - **`src/datamodel.rs`** - Core structures: `Project`, `Model`, `Variable`, `Equation` (including `Arrayed` variant with `default_equation` for EXCEPT semantics and `has_except_default` bool flag), `Dimension` (with `mappings: Vec` replacing the old `maps_to` field, and `parent: Option` for indexed subdimension relationships), `DimensionMapping`, `DataSource`/`DataSourceKind`, `UnitMap`, `MacroSpec` (a macro-marked `Model`'s calling convention: `parameters`/`primary_output`/`additional_outputs`; `Model.macro_spec` is `None` for every ordinary model; carried directly on the `SourceModel` input). `Model::new_macro(name, params, additional_outputs, body_variables)` is the shared port-synthesis + `MacroSpec`-construction step used by *both* the MDL converter and the XMILE reader: it sets `can_be_module_input` on each formal-parameter body variable (synthesizing a `Flow`/`Aux` placeholder port when absent) so `collect_module_idents` treats the macro as an ordinary sub-model. View element types (`Aux`, `Stock`, `Flow`, `Alias`, `Cloud`) carry an optional `ViewElementCompat` with original Vensim sketch dimensions/bits for MDL roundtrip fidelity. `StockFlow` has an optional `font` string for the Vensim default font spec. - **`src/variable.rs`** - Variable variants (`Stock`, `Flow`, `Aux`, `Module`), `ModuleInput`, `Table` (graphical functions). `classify_dependencies()` is the primary API for extracting dependency categories from an AST in a single walk, returning a `DepClassification` with five sets: `all` (every referenced ident), `init_referenced`, `previous_referenced`, `previous_only` (idents only inside PREVIOUS), and `init_only` (idents only inside INIT/PREVIOUS). `parse_var_with_module_context` accepts a `module_idents` set so `PREVIOUS(module_var)` rewrites through a scalar helper aux instead of `LoadPrev`. Per-element graphical functions: `build_tables` materializes one `Table` per element of an arrayed GF and `reorder_arrayed_element_tables` places each at the element's *flat row-major declared-dimension index*, NOT its `Equation::Arrayed` `elems` Vec position -- because the runtime selects a per-element table by the row-major dimension offset (`vm.rs` `Lookup`/`LookupArray`: `graphical_functions[base_gf + element_offset]`). Elements lacking a GF get an empty placeholder so the index stays aligned. The salsa dependency-table path mirrors this in `db.rs::extract_tables_from_source_var`. -- **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `resolve_axis_index_name(name, axis_dim, target_iterates)` is the engine's SINGLE element-vs-dimension-name precedence for a bare-identifier subscript index -- the axis's own declared elements first, then a dimension the enclosing equation iterates -- matching `compiler::subscript::normalize_subscripts3`; three LTM consumers read it -- `ltm_agg::classify_axis_access`, the pin rule (`ltm_augment_post_transform::pin_dimension_name_indices`), and SOME of the ceteris-paribus wrap's subscript-index guard (`ltm_augment::index_axis_verdict`, GH #986). The wrap was the last to be unified and the unification is PARTIAL, which is the important part: it had two PROJECT-WIDE predicates (`dimension_uniquely_containing_element` and `is_element_of_any_dimension`, ranging over every dimension in the project) where the compiler ranges over the indexed variable's own axis, so a model variable whose canonical name happens to be an element of an UNRELATED dimension is a runtime read to the simulation and a static element selector to the wrap -- the partial leaves it LIVE and moves with it (a link score reporting real influence for an edge with no causal dependence at all), and the qualification step rewrites the index to `otherdim·name`, an element of a dimension the indexed variable is not declared over, which still compiles and reads a different slot than the `PREVIOUS(target)` anchor did. Declaring a dimension that NO equation references is enough to change a score, with no diagnostic in either mode. The wrap is now THREE call-site families with TWO answers: the arrayed / scalar / A2A / stock-flow emitters resolve against the axis (pinned by `db::ltm_tests::a_colliding_index_name_is_resolved_against_the_axis_it_indexes`, on the emitted text AND the simulated series), while **`generate_per_element_link_equation` and every `LOOKUP` table index still take the project-wide fallbacks and still reproduce the defect** -- the first because it threads `dep_dims: None`, the second because a graphical-function holder is by construction absent from `dep_dims` (`variable::classify_dependencies`' `LookupTable` arm records it in `referenced_tables` and keeps it off the dependency graph, GH #606), so `axis_dim_at` can never resolve its axis. Both gaps are named at their call sites; GH #984 stays open on the second. The axis comes from `IteratedDimCtx::dep_dims`, the dep -> declared-dimensions table already threaded for the GH #526 other-dep correspondence check -- reusing it avoids a second table built by a second route, but it is a table of the TARGET's array dependencies rather than of declared dimensions, and that mismatch is exactly what bounds the fix. A separate, older defect at the same site is also open: an index frozen inside an ALREADY-frozen head is DOUBLE-lagged (the partial reads the head at `t-1` indexed at `t-2`, where the anchor indexed at `t-1`), so such a partial is not fully ceteris-paribus even where every name resolves correctly -- and the wrap's own GH #759 comment calls reading an index two steps back "semantically wrong for a genuinely-dynamic index". It is deferred because it is a semantics question that interacts with GH #975's own head-lag pin, NOT because the current behaviour has been adjudicated: `db::ltm_char_tests::per_element_dynamic_index_scores_preserve_head_lag` freezes the lag but was written to catch a blanket skip of the whole index pass, and uses the lag only as its discriminator. `mapped_element_correspondence(iterated_dim, source_dim)` (GH #527) is the reusable element-level correspondence for a mapped dimension pair -- per iterated element, the source element the executed simulation reads (both declaration directions, single-hop only, POSITIONAL mappings only; an explicit element map returns `None` because the executed A2A lowering resolves positionally and ignores it -- GH #753 and the tracked positional-vs-element-map execution inconsistency (GH #756) are the gate for re-enabling; `None` ⇒ callers keep their conservative broadcast, a superset of the true edges). It is what keeps the LTM element graph (`expand_same_element`'s mapped diagonal) and link-score dimensions (`link_score_dimensions` -- whose mapped arm is additionally gated on the edge having a `Bare`-classified site via `model_edge_shapes`, so the arrayed retarget fires exactly when the element graph expands the diagonal rather than the element-mapped `DynamicIndex` cross-product) in lockstep with the classifier (`classify_iterated_dim_shape`, whose mapped arm gates on this same correspondence in BOTH declaration directions since GH #757 -- via `ltm_agg::classify_axis_access` / `iterated_axis_slot_elements` -- so a reverse-declared positional pair classifies `Bare` and gets the diagonal) and -- for positional mappings -- the simulation; the agg machinery's mapped sliced reducers (GH #534) consume it through `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion), so the same gate governs hoisting and the emitters' slot remap. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions +- **`src/dimensions.rs`** - `DimensionsContext` for dimension matching, subdimension detection, and element-level mappings. Supports indexed subdimensions via `parent` field (child maps to first N elements of parent). `has_mapping_to()` checks for element-level dimension mappings between two dimensions. `resolve_axis_index_name(name, axis_dim, target_iterates)` is the engine's SINGLE element-vs-dimension-name precedence for a bare-identifier subscript index -- the axis's own declared elements first, then a dimension the enclosing equation iterates -- matching `compiler::subscript::normalize_subscripts3`; three LTM consumers read it -- `ltm_agg::classify_axis_access`, the pin rule (`ltm_augment_post_transform::pin_dimension_name_indices`), and SOME of the ceteris-paribus wrap's subscript-index guard (`ltm_augment::index_axis_verdict`, GH #986). The wrap was the last to be unified and the unification is PARTIAL, which is the important part: it had two PROJECT-WIDE predicates (`dimension_uniquely_containing_element` and `is_element_of_any_dimension`, ranging over every dimension in the project) where the compiler ranges over the indexed variable's own axis, so a model variable whose canonical name happens to be an element of an UNRELATED dimension is a runtime read to the simulation and a static element selector to the wrap -- the partial leaves it LIVE and moves with it (a link score reporting real influence for an edge with no causal dependence at all), and the qualification step rewrites the index to `otherdim·name`, an element of a dimension the indexed variable is not declared over, which still compiles and reads a different slot than the `PREVIOUS(target)` anchor did. Declaring a dimension that NO equation references is enough to change a score, with no diagnostic in either mode. The wrap is now THREE call-site families with TWO answers: the arrayed / scalar / A2A / stock-flow emitters resolve against the axis (pinned by `db::ltm_tests::a_colliding_index_name_is_resolved_against_the_axis_it_indexes`, on the emitted text AND the simulated series), while **`generate_per_element_link_equation` and every `LOOKUP` table index still take the project-wide fallbacks and still reproduce the defect** -- the first because it threads `dep_dims: None`, the second because a graphical-function holder is by construction absent from `dep_dims` (`variable::classify_dependencies`' `LookupTable` arm records it in `referenced_tables` and keeps it off the dependency graph, GH #606), so `axis_dim_at` can never resolve its axis. Both gaps are named at their call sites; GH #984 stays open on the second. The axis comes from `IteratedDimCtx::dep_dims`, the dep -> declared-dimensions table already threaded for the GH #526 other-dep correspondence check -- reusing it avoids a second table built by a second route, but it is a table of the TARGET's array dependencies rather than of declared dimensions, and that mismatch is exactly what bounds the fix. A separate, older defect at the same site is also open: an index frozen inside an ALREADY-frozen head is DOUBLE-lagged (the partial reads the head at `t-1` indexed at `t-2`, where the anchor indexed at `t-1`), so such a partial is not fully ceteris-paribus even where every name resolves correctly -- and the wrap's own GH #759 comment calls reading an index two steps back "semantically wrong for a genuinely-dynamic index". It is deferred because it is a semantics question that interacts with GH #975's own head-lag pin, NOT because the current behaviour has been adjudicated: `db::ltm_char_tests::per_element_dynamic_index_scores_preserve_head_lag` freezes the lag but was written to catch a blanket skip of the whole index pass, and uses the lag only as its discriminator. **Mapped-reference correspondence is keyed by the reference's SPELLING** (GH #997), because execution applies two different rules and the deciding fact belongs to the reference site, not to the dimension pair. `resolve_mapped_read(source_axis, active_dim, active_element)` is the engine's single statement of the map-following rule -- NAME on the source axis first, then `translate_via_mapping`, then a mapped PARENT of the active subdimension -- and all three executed sites call it: `compiler::context`'s `get_implicit_subscript_off` (subscript-less references reaching the implicit-axis allocator: a stock's flows, the stock self-reference, module input wiring), `compiler::subscript`'s `build_view_from_ops` on the `IndexOp::ActiveDimRef` arm (a subscript naming a NON-active dimension), and `compiler::context`'s `IndexExpr3::Dimension` arm (the dynamic-path twin of the second, which before #997 consulted the map WITHOUT trying the name first -- a latent divergence, measured to resolve nothing across the lib suite but reachable whenever a sibling index defeats static normalization). Two describers sit on top, and a caller picks the one its site's spelling gets: `positional_correspondence(iterated_dim, source_dim)` for the two spellings execution folds to an ORDINAL -- a subscript naming a dimension the equation ITERATES, and a bare in-equation reference, which `lower_pass0` rewrites into the first -- and `executed_read_correspondence(iterated_dim, source_dim)` (`resolve_mapped_read` per element) for the two that follow the map: a subscript naming a NON-active dimension, typically the source's own (`target[COP] = x[Aggregated Regions]`, C-LEARN's shape), and a stock's FLOW reference. Both need a mapping DECLARED in one direction or the other and are single-hop; the positional one additionally needs equal cardinality, while the executed one works at every cardinality including many-to-one. Neither declines an explicit element map -- the single pre-#997 `mapped_element_correspondence` did, not as a claim about execution but because one function could not see which spelling it was being asked about, and it therefore answered neither. Every row of both is measured against the VM in `crate::mapped_reference_semantics_tests` (4 spellings x 5 mapping kinds x 2 declaration directions). `mapped_read_partner_dim` is the PAIRING half, mirroring `compiler::subscript::normalize_subscripts3`'s `IndexExpr3::Dimension` arm: which of the target's iterated dimensions a non-active index dimension is matched with. It declines AMBIGUITY (two viable partners) where execution breaks the tie by position -- a describer that copied that would attribute influence along edges chosen by declaration order. Consumers: `ltm_agg::iterated_axis_slot_elements` (the per-source-element preimage inversion behind the agg machinery's mapped sliced reducers, GH #534) and `classify_axis_access`'s `Iterated` arm take the POSITIONAL one; `classify_axis_access`'s new `AxisRead::MappedRead` arm, `ltm_augment_post_transform::per_element_row_for_target`'s matching arm, and `dep_element_pins`' subscripted-reference row take the EXECUTED one. Two projected axes naming the SAME target dimension are DRIVEN TOGETHER, in the shared row derivation `db::ltm::read_slice_row_parts`: execution resolves both indices against the one active element, so `target[State] = matrix[Region1, Region2]` reads the diagonal `matrix[map1(s), map2(s)]` and crossing the axes emitted rows nothing reads. Its rows travel as STRUCTURED per-axis parts (`ReadSliceRowParts::slot_parts`/`row_parts`), never as a comma-joined string a consumer re-splits: a canonical element name can itself contain a comma (a quoted XMILE `"a,b"` canonicalizes to `a,b`, compiles and simulates), so a serialize-and-split round trip reads one coordinate as two -- the element-edge arm was fixed onto the structured form, and the comma-joined `read_slice_rows` is `#[cfg(test)]`-gated at the `db::ltm` boundary. The link-SCORE emitters still carry several `split(',')` sites of their own (a measured, disclosed residual -- see `mapped_reference_semantics_tests::a_comma_bearing_element_name_survives_the_mapped_projection`'s doc for the observed score damage); new code must not add more. That produced element edges and loop candidates the simulation never traverses (9 loops over a 3x3 source where there are 3) AND disagreed with `per_element_row_for_target`, which projects one target element through each axis and therefore always produced the diagonal -- so the link-score NAMES and the element EDGES described different graphs. The same shape reached through an all-`Iterated` subscript (`matrix[D,D]`, `matrix[State,State]`) used to classify as `RefShape::Bare`, whose `expand_same_element` sees only the two variables' dimension lists and cannot express a shared coordinate; `ltm_ir::classify_iterated_dim_shape` now retargets it to `PerElement` -- but only when the TARGET names that dimension once, since a target repeating it (`cube[D1,D1]`) has two coordinates for one name and every per-element derivation addresses a target axis by name. Measured against the VM in `crate::mapped_reference_semantics_tests`, which also pins the residual `expand_same_element` leaves on the repeated-TARGET shape. `dep_element_pins` carries BOTH rows on one `DepElementPin` -- `axes` for a `dep[]` subscript, `bare_row` for a bare `dep` -- which is #997 in miniature: one table answering two spellings with one rule is what made C-LEARN's element-mapped deps unpinnable. `db::analysis::bare_reference_correspondence` is the UNION of the two, used by `expand_same_element` and by `link_score_dimensions`' mapped gate, because a `RefShape::Bare` site can be an in-equation reference (positional) OR a structural flow-to-stock edge (map-following) and `ltm_finding::expand_a2a_link_offsets` re-derives the from-node with no site information at all; the union is exact wherever the two rules agree (every positional mapping between disjointly-named dimensions) and a two-edge superset where they differ, which keeps the element graph and discovery in lockstep (GH #754) and never emits fewer edges than execution reads -- with ONE measured exception, which is a defect in `expand_same_element` rather than in the correspondence it consumes: a variable REPEATING a dimension on the target side (`cube[D1,D1] = pop[D1,D1]`). `to_dim_positions` is a `HashMap<&str, usize>`, so a target naming `D1` twice records only its LAST axis and both source axes claim it; over a 2x2 fixture the arm emits 12 edges covering 2 of the 4 reads the simulation makes, so it is a superset in neither direction. Pinned in both directions (a missing real edge and a phantom one) by `mapped_reference_semantics_tests::a_repeated_target_dimension_reads_the_first_axis_on_both_sides`, which a fix must red and restate. Blast radius MEASURED: Vensim rejects a repeated-dimension declaration ("DimA appears more than once on LHS", `vensim-probes/repeated_dimension.mdl` in Vensim DSS 2026-08-04), so no MDL-imported model reaches it and the residual is confined to hand-authored XMILE/JSON/protobuf -- which the XMILE v1.0 spec does sanction by example ("A 2D non-apply-to-all array with dimensions X by X", verified in `docs/reference/xmile-v1.0.html`), so the shape stays legitimate and the residual stays worth fixing. The fix belongs here rather than in the reference classifier: `ltm_ir::classify_iterated_dim_shape` deliberately does NOT retarget this shape to `PerElement` (which would emit those same 2 edges with no phantoms) because `RefShape` also selects the SCORE emitter, and `emit_per_element_link_scores` refuses a repeated-dimension target -- so the retarget would silently trade an emitted link score for a loud skip on every edge touching such a variable. `SubdimensionRelation` caches parent-child offset mappings for both named (element containment) and indexed (declared parent) dimensions - **`src/model.rs`** - Model compilation stages (`ModelStage0` -> `ModelStage1` -> `ModuleStage2`). It performs NO dependency analysis of its own: `ModelStage1::set_dependencies` reads the production dependency graph (`db::dep_graph::model_dependency_graph`) once per module instantiation and copies its runlists into `ModuleStage2`. Until GH #568 it ran a second, independent walk here -- its own transitive closure (`all_deps`), its own cross-model output resolution (`module_output_deps`), its own `topo_sort` runlists and its own `CircularDependency` -- and that second gate genuinely disagreed with production's, rejecting the element-acyclic recurrence SCCs `resolve_recurrence_sccs` resolves. There is one gate now, pinned by `project.rs`'s `the_circular_dependency_gate_is_the_production_one` in both the resolves and the rejects direction. `collect_module_idents` pre-scans datamodel variables to identify which names will expand to modules (preventing incorrect `LoadPrev` compilation). Unit checking uses salsa tracked functions in `db.rs`. The two `#[cfg(test)]` `ModelStage0` constructors are the salsa-FREE twin of `db::stages::model_stage0`: `new_in_project(project_models, x_model, ..)` builds a stage from a `datamodel::Model` with no database, resolving module-function calls against the whole project's `MacroRegistry` (which is what makes it a faithful oracle for a model that CALLS a macro defined in a sibling model), and `new(x_model, ..)` is the single-model wrapper the many one-model fixtures use. They derive the module-ident set, the macro registry and the duplicate-ident errors along completely different routes than the query, which is exactly why `db::stages_tests` can use them as an oracle. The former salsa-cached `ModelStage0::new_cached` -- a test-only third copy of the query's construction -- was deleted; its coverage now points at the live query. `enumerate_modules_inner` records a model's instantiation BEFORE descending into it, so that a module cycle terminates: the "have I seen this model" test is the recursion guard, so a model still being walked has to count as seen. (A cycle through `main` already terminated, because `enumerate_modules` records `main` up front; one below `main` did not.) This matches its salsa twin `db::assemble::enumerate_module_instances_inner`. Recording early cannot LOSE an instantiation because the insert is unconditional at every module site -- only the recursion is guarded; the set-of-input-sets value is a separate fact, and it is why the ORDER instantiations arrive in is unobservable. - **`src/project.rs`** - `Project` struct aggregating models. `from_salsa(datamodel, db, source_project, cb)` builds a Project from a pre-synced salsa database: it READS `db::stages::model_stage1` for every project model and clones each memo (the clone is mandatory, not incidental -- `model_deps.take()`, `set_dependencies` and the `model_cb` all mutate it, while a `returns(ref)` memo is shared with every other reader). Each stage is kept paired with the `SourceModel` handle it came from, because `set_dependencies` reads the production dependency graph and that query is keyed on the handle, not on the model's name. It used to build its own whole-project `ModelStage0` map and lower it inline, a second salsa-native copy that had silently drifted from `db::stages` on three fields. Its model order is deterministic because the `topo_sort` seed is sorted first -- `topo_sort` breaks ties by visit order and the seed came from a `HashMap`'s keys (the `Project`-path twin of the GH #595 fix in `db::dep_graph`); the Initials runlists inherit the dependency graph's own determinism since GH #568 unified the two gates. `compiler::Module::new` is the sole reader of `ModelStage1::instantiations`; its callers are `#[cfg(test)] TestProject::build_module` and three direct calls in the `compiler`'s dimension tests, all `#[cfg(test)]`, while production `src/db/assemble.rs` never builds a `compiler::Module` at all. It REFUSES a model whose dependency graph carries a resolved recurrence SCC (`ModelStage1::set_dependencies` records a `NotSimulatable` for one): GH #568 unified the cycle GATE but not the EMITTER, and the per-element interleave `db::assemble::combine_scc_fragment` performs has no monolithic equivalent, so emitting the members whole would read a co-member's element before assigning it -- a silent wrong answer where the second gate used to give an accidental refusal. A model that can REACH a module cycle is gated out of dependency resolution first, exactly as the production entry points gate it (`db::diagnostic::module_cycle_diagnostic`, GH #806): the dependency graph's recurrence-SCC refinement descends into the recursive `model_module_map`, and salsa turns that into an unrecoverable dependency-graph panic -- a process abort under `panic = abort`, reachable from the public `From`. Such a model records the cycle as its error and keeps empty runlists. `from_datamodel(datamodel)` is a convenience wrapper that creates a local DB and syncs. **Neither is reachable from production**: inside the crate every caller is a test, and the only in-repo user of the public `From` impl is the engine's own `tests/integration/simulate_ltm.rs`, feeding the `ltm_finding::discover_loops` convenience wrapper (the shipped analysis path, libsimlin's `simlin_analyze_discover_loops`, goes through `discover_loops_with_graph` and builds no `Project`). Treat it as a monolith kept honest as a test oracle, not as live code. Production compiles via `db::compile_project_incremental` with `ltm_enabled`/`ltm_discovery_mode` on `SourceProject`. - **`src/results.rs`** - `Results` (variable offsets + timeseries data), `Specs` (time/integration config) @@ -182,7 +183,7 @@ The unit subsystem is partial-result throughout: a single bad declaration or one - `tests.rs` - integration-style tests spanning all submodules; preserved as a single file to avoid splitting shared test fixtures. - **`src/ltm_finding.rs`** - Strongest-path loop discovery algorithm (Eberlein & Schoenberg 2020). Post-processes simulation results containing link score synthetic variables to find the most important loops at each timestep. `discover_loops_with_graph(results, causal_graph, stocks, ltm_vars, dims, expansion, sub_model_output_ports, budget)` is the primary entry point: `ltm_vars` and `dims` enable A2A link score expansion (per-element edges from `parse_link_offsets`); when empty, all link scores are treated as scalar. A Bare A2A link score is dimensioned over the TARGET's dims, so `expand_a2a_link_offsets` subscripts the TARGET node per element but PROJECTS the SOURCE node onto `from`'s OWN declared dims -- bare for a scalar feeder (`scale → growth`, GH #790), the same-element diagonal for an equal-dim feeder, the broadcast/partial-collapse form for a lower-dim feeder, and the positionally-mapped diagonal for a `State→Region` pair (GH #527) -- by reusing the element graph's own `db::expand_same_element` rule, so the discovery search graph's node names match `model_element_causal_edges` node-for-node (GH #754; before this, subscripting BOTH endpoints with the score's full tuple minted phantom from-nodes like `scale[a]`/`boost[r,a]`/`x[s]` that named no real node, so every loop through such a feeder dangled and was silently undiscoverable). The from-var declared dims + dimension-mapping context ride in a `LinkExpansionContext` (`declared_dims` + `dim_ctx`) that `analyze_model` builds via the public `analysis::build_link_expansion_context` (the SAME `variable_dimensions` / `project_dimensions_context` queries the element graph reads); the db-less `discover_loops(&Results, &Project)` convenience path passes `LinkExpansionContext::default()` (no A2A expansion runs there). Element-mapped (non-positional) pairs never reach the projection: `db::ltm::link_score_dimensions` declines to retarget them (no Bare A2A score; the GH #758 loud skip fires instead) under the GH #756 positional-only gate. `SearchGraph` provides DFS-based strongest-path traversal from stock nodes. The post-simulation score recompute (path → `FoundLoop`) applies the **per-exit-port pathway selection** (`recompute_module_input_edge_series`, GH #698): a loop edge `x → m` into a module is recomputed by max-abs-selecting over the sub-model's `m·$⁚ltm⁚path⁚{entry}⁚{idx}` pathway scores that end at the exit port the loop reads (recovered from the next link `m → y`), instead of reading the module *composite* (which max-abs-selects across ALL ports and so can pick a wrong-signed port for a multi-output module -- flipping the loop polarity vs exhaustive). The pathway indices come from the module's sub-graph via the same `enumerate_pathways_to_outputs_with_truncation` over the same sorted output-port set the sub-model emitted against, so they match index-for-index (the discovery-mode equivalent of exhaustive's `compute_module_link_overrides`). That port set is NOT re-derived parent-scoped (which would shift indices when another project model reads an extra output port -- GH #698 / PR #705 r3353097150): `discover_loops_with_graph` takes a `SubModelOutputPorts` map (sub-model canonical name -> emitted sorted port set) that `analyze_model` builds from the SAME emission decision via `analysis::build_sub_model_output_ports` -> `db::ltm::sub_model_output_ports` (identity-by-construction); the db-less `discover_loops(&Results, &Project)` convenience path reconstructs the same project-wide-union + stdlib-`output` semantics in `project_sub_model_output_ports`. Falls back to the base composite offset for single-output / pathless / indeterminate-port / sub-model-absent-from-map edges. Because the discovery graph is element-level, the recompute `strip_subscript`s `link.from`/`link.to`/`next.from`/`next.to` before its name matches (arrayed loop nodes carry `[elem]`; mirrors the exhaustive twin's stripping -- PR #705 r3353758167); that stripping is LIVE rather than latent since GH #716 closed: a scalar module output feeding an arrayed reader used to emit ONE scalar constant-0 link score, which dropped the loop, and is now scored per target element by `db::ltm::link_scores::try_implicit_scalar_to_arrayed_link_scores` (which also owns the per-element module INSTANCES a per-element expansion mints, whose partials were previously `scalarize`d onto element 0's arm), so an arrayed loop through a multi-output module is discoverable end-to-end -- `analysis::tests::analyze_model_arrayed_module_loop_is_discovered_per_element`. Returns a `DiscoveryResult` whose `FoundLoop`s carry per-timestep link/loop/pathway scores, ranked **competitive-first** (`rank_and_filter`): loops sharing their cycle partition with at least one other discovered loop come first, ordered by mean partition-relative importance; loops trivially ALONE in their partition (relative score exactly +/-1 by construction -- zero information, e.g. C-LEARN's isolated gas-uptake decay loops) sort after all competing loops and are dropped first under the `MAX_LOOPS` cap. Each `FoundLoop` carries a result-scoped dense `partition` index into `DiscoveryResult::partitions` (`DiscoveredPartition { stocks, loop_count }`, first-appearance order; NOT stable across runs/edits -- key on the stock set for durable identity), threaded through `analysis::ModelAnalysis::partitions` / `LoopSummary::partition`, the FFI `SimlinDiscoveredPartition` / `SimlinDiscoveredLoop.partition`, and pysimlin `Analysis.partitions` / `Loop.partition`. Also hosts the link-set synthetic-node collapse: `trim_synthetic_aggs_from_loop_links` collapses `$⁚ltm⁚agg⁚{n}` nodes out of a single loop's link *cycle*, and the public `collapse_synthetic_links(Vec)` generalizes that to ALL synthetic/macro/module-internal nodes (`ltm::is_synthetic_node_name`) over an arbitrary link *set* -- each chain `X -> $⁚…internal… -> Y` collapses to one composite edge `X -> Y` with product polarity and the per-timestep max-magnitude path score (the composite link score, LTM ref 6.3/6.4); a purely-internal cycle/source is dropped. `CollapsibleLink { from, to, polarity, score: Option> }` is the abstract shape, so structural-only callers (`score = None`) and LTM-run callers share one impl. libsimlin's `simlin_analyze_get_links(.., include_internal=false)` collapses the `get_links()` set through this; `include_internal=true` returns the raw graph. The `#[cfg(test)]` tests live in the sibling `src/ltm_finding_tests.rs` (mounted via `#[cfg(test)] #[path]`, split out for the per-file line cap). - **`src/ltm_agg.rs`** - Aggregate-node enumeration for LTM. `enumerate_agg_nodes` (salsa-tracked) walks every variable's `Expr2` AST left-to-right depth-first and identifies each maximal array-reducer subexpression (`SUM`/`MEAN`/`MIN`/`MAX`/`STDDEV`/`RANK`/`SIZE`); AST-identical subexpressions (keyed by canonical printed equation text) dedupe to one `AggNode`. Two kinds: **synthetic** (`is_synthetic == true`, the reducer is a sub-expression of a larger equation -- a `$⁚ltm⁚agg⁚{n}` aux is minted) and **variable-backed** (`is_synthetic == false`, the reducer is the entire dt-equation of a scalar/A2A variable like `total_population = SUM(pop[*])` -- the variable itself is the agg; EXCEPT a whole-RHS reducer whose shape the variable-backed machinery cannot express -- a MAPPED iterated axis (GH #534: the name-based link-score path cannot remap, so its `Wildcard` partial would silently stub to 0) or NON-ALIGNED result dims (GH #764 / shape-expressiveness T4: a broadcast over extra owner dims `out[D1,D3] = SUM(matrix[D1,*])` or permuted axes `out[D2,D1] = SUM(cube[D1,D2,*])`, where a per-`(row, slot)` slot cannot name a complete owner element) -- which mints a synthetic agg instead (`variable_backed_shape_is_expressible`, the ONE minting condition), riding the two-half emitters + the GH #528 projection). Each `AggNode` carries a `sources: Vec` -- one entry per source variable, SORTED by canonical name and deduped (salsa cache-equality + emission-order determinism; T2 of the shape-expressiveness design), each with its OWN `read_slice: Vec` (one `AxisRead ∈ {Pinned(elem), Iterated{dim, source_dim}, Reduced{subset}}` per THAT source's axes -- which rows of it the reducer actually reads; a scalar feeder like `scale` in `SUM(pop[*] * scale)` carries an empty slice; `Iterated` carries the (target, source) canonical dim pair, equal for the literal case; `Reduced.subset` is `None` for the full extent or the proper-subdimension element subset for a `SUM(arr[*:Sub])` StarRange, GH #766, decided per axis by `classify_axis_access` -- the single per-axis classifier of the shape-expressiveness design) -- and a `result_dims` (the `Iterated` axes' TARGET dims, datamodel-cased -- empty for a whole-extent or pinned-slice reduce). Under the I1 acceptance (`accept_source_slices`, T5 of the shape-expressiveness design / GH #767) every arrayed CO-SOURCE (`Reduced`-bearing slice) carries the identical *canonical* slice (`AggNode::canonical_read_slice` -- the first `Reduced`-bearing source slice, falling back to the first non-empty for the degenerate no-co-source agg), while an ITERATED-DIM PROJECTION FEEDER -- a source whose slice is all-`Iterated` over exactly the canonical slice's iterated target dims, in order, unmapped (`frac[D1]` in `SUM(matrix[D1,*] * frac[D1])`, per-result-slot constant) -- is accepted with ITS OWN slice (`AggNode::source_is_projection_feeder` is the discriminator); per-source consumers (`emit_agg_routed_edges`, `emit_source_to_agg_link_scores`) read `AggNode::source_read_slice(from)`, and name-keyed consumers use `AggNode::reads_var`. A feeder's link-score half is the per-`(row, slot)` CHANGED-LAST equation (`ltm_augment::generate_iterated_feeder_to_agg_equation`, emitted via `link_scores::iterated_feeder_row_scores` from both the synthetic source-half and `try_cross_dimensional_link_scores`' variable-backed feeder branch): the reducer text pinned to the slot with only the feeder frozen -- the arrayed generalization of the GH #737 scalar-feeder convention, exactly complementary per slot to the co-source rows' changed-first numerators for a bilinear body; the co-source rows' changed-first body partial pins the mismatched-arity feeder dep BY DIM NAME (`pin_body_to_row`'s GH #767 extension) so `PREVIOUS(frac[d1·r])` is held frozen at the row instead of bailing to the delta-ratio fallback. `compute_read_slice` decides hoistability: a whole-extent reduce (`SUM(pop[*])` ⇒ all-`Reduced`), a sliced one (`SUM(pop[NYC,*])` ⇒ `[Pinned(nyc), Reduced]`, `SUM(matrix[D1,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Reduced]` → an arrayed agg over `D1`), a mixed one (`SUM(matrix3d[D1,NYC,*])` over an A2A-`D1` body ⇒ `[Iterated(d1,d1), Pinned(nyc), Reduced]`), or a positionally-MAPPED sliced one (GH #534: `SUM(matrix[State,*])` over `matrix[Region,D2]` with a positional `State→Region` mapping ⇒ `[Iterated{state, region}, Reduced]`, `result_dims = [State]` -- the agg is arrayed over the TARGET's iterated dim and the three `Iterated`-axis consumers remap each source row to the slot of its positionally-corresponding target element via `iterated_axis_slot_elements`, the preimage inversion of `mapped_element_correspondence`, so the positional-only/element-map gate is inherited) is hoisted; the carve-outs are (a) a reducer over a *dynamic index* (`SUM(pop[idx,*])`, `idx` non-literal ⇒ not statically describable ⇒ not hoisted, reference stays on the conservative path -- `db::ltm_ir` reclassifies it as `DynamicIndex`), (b) an ELEMENT-mapped sliced reducer the correspondence declines (execution resolves positionally and ignores the map, GH #756; a POSITIONAL mapping is accepted in either declaration direction since GH #757) ⇒ `compute_read_slice` returns `None`, conservative -- (c) a multi-source slice combination outside the I1 acceptance -- co-sources with differing slices, one variable read with two different slices (I3b), or a no-`Reduced` source that is not the pure iterated projection (a Pinned-axis mix like `SUM(matrix[D1,*] * w[D1, c1])`, a dim-subset/permuted feeder, or any mapped Iterated axis in a feeder combination) -- `combined_read_slice`/`accept_source_slices` return `None`, (d) a StarRange naming a NON-subdimension of its axis (a mid-edit inconsistency; declined rather than silently widened to the full extent), and (e) `RANK` (GH #771: array-valued, so `reducer_is_hoistable` requires `reducer_collapses_to_scalar` and RANK references stay `Direct` -- a bare arg classifies `Bare`, scored by the GH #742 arrayed-capture path; loops through the rank ORDERING are a documented residual). A multi-source reducer whose arrayed args *agree* (`SUM(a[*] + b[*])`, `a`, `b` over the same dim) mints one agg with one `AggSource` per variable, each carrying the shared canonical slice. `AggNodesResult` exposes `aggs` (first-encounter order), `agg_for_key` (by canonical reducer text), and `aggs_in_var` (which aggs occur in a variable's equation, so the element-graph reroute can ask "which agg of `to` reads `from`?"). A variable-backed reduce with a NON-TRIVIAL statically-describable slice is gated by `variable_backed_reduce_agg` (GH #752, generalized by GH #765 / T3 of the shape-expressiveness design) -- the SAME gate `model_element_causal_edges`' dispatch, `build_element_level_loops`' per-circuit routing, and `try_cross_dimensional_link_scores`' row derivation consume, so edges, loop routing, and scores always cover the identical read rows (all three derive from `read_slice_rows`, invariant I4). Accepted: an ALIGNED partial reduce (`row_sum[D1] = SUM(matrix[D1,*])`, `result_dims` equal to the variable's own dims in order -- Pinned-mixed `outf[D1] = MEAN(cube[D1,x,*])` and subset `out[D1] = MEAN(matrix[D1,*:Sub])` slices included: the divisor is the true read count and unread rows get neither edges nor scores) gets read-slice element edges straight onto the variable's element nodes and per-circuit scalar loops whose both-subscripted `matrix[d1,d2]→row_sum[d1]` links resolve the per-`(row, slot)` scores; a scalar-result Pinned/subset slice on a SCALAR owner (`total = SUM(pop[nyc,*])`, `total = SUM(arr[*:Sub])`) routes the read rows into the bare `to` node with matching per-read-row scores; and the ARRAYED-owner scalar-result Pinned/subset BROADCAST slice (`share[Region] = SUM(pop[nyc,*])`, no `Iterated` axis -- GH #777) fans each read row across the FULL target element set (`emit_agg_routed_edges`' broadcast arm emits `pop[nyc,d2] → share[e]` for every `e`; `emit_broadcast_reduce_link_scores` emits the matching per-(read-row, full-target-element) scalar scores `pop[nyc,d2]→share[e]`, the section-3 `PerElement` rule applied to a variable-backed reducer owner), with loop circuits routed to the per-circuit scalar path via `is_broadcast_reduce_edge` -- the read rows are independent of `to`'s dims, so the RELATED-dim (`share[Region]`) and DISJOINT-dim (`share[D9]`) spellings emit identically. Declined: a pure full-extent variable-backed agg keeps the normal reference walker's edges (the true reads for that shape, inert skip). The GH #764 broadcast/permuted result shapes never reach this gate since T4 -- they mint synthetic aggs at enumeration -- so the gate's Iterated-arm alignment check is defense-in-depth. `scalar_feeder_of_variable_backed_agg` (GH #790) is the scalar sibling of `source_is_projection_feeder`: it composes `variable_backed_reduce_agg` with an empty-slice + genuine-`Reduced`-canonical check to recognize a SCALAR FEEDER of a whole-RHS variable-backed reduce (`scale` in `growth[D1] = SUM(matrix[D1,*] * scale)`), so `try_scalar_to_arrayed_link_scores` can route it to the single Bare A2A changed-last feeder score instead of the uncompilable per-target-element partials. Two predicates here answer what LOOKS like one question -- "is this reference inside a reducer?" -- and their answers are inverted on exactly `SIZE` and `RANK`; the inversion is the DEFINITION of the difference, not a disagreement (GH #982, assessed and left as two predicates). `reducer_collapses_to_scalar` is about the reducer's RESULT TYPE (does the subtree fit in a scalar slot?) and is read by the two freeze/capture gates plus the GH #779 bare-reducer-feeder decline: `SIZE` is a count so it collapses, `RANK` is array-valued so it does not. `builtin_routes_through_agg` is about LTM ROUTING (did `enumerate_agg_nodes` mint a node for this call?) and sets `db::ltm_ir::OccurrenceSite::in_reducer`: `SIZE` is `Constant` and is never hoisted, `RANK` gets an array-valued agg. Both read the ONE `reducer_kind_from_name` table, and `builtin_routes_through_agg` is the disjunction of the enumerator's own two minting branches (`reducer_is_hoistable` and `array_valued_rank_arg`) rather than a restatement of them. The `#[cfg(test)]` `REDUCER_DECISION_TABLE` pins all three derived predicates row by row over every arm of the kind table -- including the agreement gate's name-keyed twin of the routing predicate -- so no cell can move silently. `is_synthetic_agg_name` / `synthetic_agg_name` are the `$⁚ltm⁚agg⁚{n}` name helpers. `classify_axis_access` resolves a bare-identifier index through the shared `dimensions::resolve_axis_index_name` (element-first, GH #986), so it and `ltm_augment_post_transform::pin_dimension_name_indices` cannot disagree about which row a colliding name selects. The `#[cfg(test)]` tests live in the sibling `src/ltm_agg_tests.rs` (split out for the per-file line cap). Each node also carries `reducer: BuiltinFn` -- the reducer call the enumerator classified when it decided the hoist, of which `equation_text` is the printed rendering (GH #983). It is what makes the link-score and polarity emitters parse-free: `ltm_augment::classify_reducer_in_builtin` reads the kind/name/body off it and `ltm::CausalGraph::source_to_agg_polarity` analyses it directly, where both used to print `equation_text`, re-parse it, re-lower it against a freshly built scope and re-derive the classification -- per (agg, source) pair, with both fallible steps returning early and silently zeroing the agg's loop score. It is stored in `Expr2::strip_loc_and_bounds` form, which removes two of the three ways this field could make the salsa-cached `AggNodesResult` compare unequal to an identical rebuild: `Loc` (load-bearing -- two AST-identical occurrences differ in byte offsets, so raw storage would make the dedup winner observable and would stop `enumerate_agg_nodes` backdating across an offset-only edit) and `ArrayBounds` (a guard -- inert today, since `reconstruct_model_variables` lowers against an empty model scope and allocates no bound). Neither reader looks at either. It cannot remove the third -- dropping a `nan` literal would change what the equation means -- so that one is closed at the ROOT instead: `Expr2::Const` holds an `ast::Literal`, compared by BIT PATTERN, so a model whose hoisted reducer contains a `nan` literal backdates like any other (GH #987/#981; with a bare `f64` it never could, since `NaN != NaN`). Pinned by `a_nan_literal_in_a_reducer_does_not_defeat_agg_backdating`. The stored builtin is read only for SYNTHETIC aggs (both readers filter to those); the variable-backed arm's copy is unread today, kept so `AggNode` has one shape. `AggNode`/`AggNodesResult` derive `Eq` (reflexivity is a compile-checked property now that the literal is not a bare `f64`) and `Debug` only under `debug-derive`. -- **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. Every `PREVIOUS` the wrap SYNTHESIZES goes through `freeze_at_previous`, which chooses the call's first-DT initial value from the position being frozen: a VALUE position keeps the unary spelling (desugared to `0`, the XMILE-documented default, and unobservable behind the guard form's `TIME = INITIAL_TIME` arm), while a SUBSCRIPT INDEX names its own un-lagged operand -- `PREVIOUS(idx, idx)` -- because `0` is out of range for every 1-based dimension, so the frozen read yielded NaN at t=0 and `make_temp_arg`'s capture helper served that NaN as the score's FIRST LIVE step (GH #975). Only the two walkers that descend into indices need it (`wrap_non_matching_in_previous` via `wrap_index_non_matching_in_previous`, and `wrap_matching_in_previous`); `wrap_live_shaped_in_previous` and `freeze_pinned_body` document that they never do. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, and that precedence is not the pin's own: it is the shared `dimensions::resolve_axis_index_name`, which `ltm_agg::classify_axis_access` reads too (GH #986 closed the divergence -- the classifier had the opposite order, so a mapped collision described the axis as `Iterated` over a dimension the compiler never iterates there). Element-first is what `compiler::subscript`'s `normalize_subscripts3` does ("First check if it's a named dimension element (takes priority)"), and the simulation is the authority: the two readings collide when a dimension declares an element whose name is also a dimension name (`Bucket = [old, region]` beside a `Region` dimension), and a describer that breaks the tie the other way names rows the simulation never reads (`a_colliding_index_name_reads_the_axis_element_in_the_simulation` is the numeric oracle; the XMILE spec's footnote [9] settles the adjacent VARIABLE-vs-element pair outright and sections 2.1/3.7.1 argue this pair from the namespace rule -- `resolve_axis_index_name`'s rustdoc says which is which). Everything that is NOT a bare identifier is left verbatim, needing no pin: a numeric literal, arithmetic over literals, an `@N` position (which `compiler::context` resolves to a concrete element offset in scalar context -- spelling it out here would be a second implementation of position syntax), and a compound expression selecting the element at RUNTIME. That last one used to be a conditional REFUSAL, and deleting it is GH #984: the wrap now freezes a `LOOKUP` table argument's index reads itself (`freeze_lookup_table_indices`), so a runtime index arrives here already lagged and the rule keeps it. That freeze WIDENS its own descent's dep set with the index idents, and without that it would not fire at all -- `variable::classify_dependencies`' `BuiltinContents::LookupTable` arm records the table's ident and never walks the table expression, so an index variable referenced only there is not a dependency and the wrap's `other_deps` freeze could never reach it. The widened set is scoped to that argument's indices, and the element / dimension-name guards run before the dep check, so it cannot make a selector wrap. (Leaving the dep set itself alone is deliberate: an index dropped from a variable's dependencies is a runlist-ordering question, not an LTM one.) What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The verdict space (`Pinned`, `Keep`, and the one loud `Unspellable`) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s three verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). `subscript_idents_at_element` pins a target's arrayed deps for a per-element scalar partial, and it pins each one over the dimensions THAT DEP declares rather than over the target's element tuple (GH #974): a bare arrayed reference in an apply-to-all body reads its own axes' coordinates matched by dimension NAME, so a subset-dims dep (`w[Age]` under `growth[Region,Age]`) got an over-arity subscript whose fragment failed to compile, and a REORDERED one (`w[Age,Region]`) got a subscript that compiled and silently read the transposed element. The projection is `post_transform::dep_element_pins`/`dep_row_for_target`, reused by `pin_bare_source_ref` for a bare reference to the LIVE SOURCE (which is why a positionally-MAPPED bare source reference resolves through `mapped_element_correspondence` instead of being left bare and frozen into an uncompilable multi-slot `PREVIOUS`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Five more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above), and **`ltm_augment_freeze.rs`** (the GH #975 first-DT initial value of every synthesized `PREVIOUS`). +- **`src/ltm_augment.rs`** - Equation generators for LTM synthetic variables: `generate_link_score_equation_for_link` (ceteris-paribus link scores; takes `RefShape` and source dimension elements to drive per-shape PREVIOUS wrapping), `generate_loop_score_variables` (emits one `loop_score` per loop as a dimension-shaped `datamodel::Equation`: `Scalar` for scalar loops, `ApplyToAll` for dimensioned loops whose links resolve through Bare A2A names, and per-slot `Equation::Arrayed` for dimensioned loops backed by per-element circuits via `Loop.slot_links` -- GH #653; relative loop scores are computed post-simulation in `ltm_post.rs`), `build_partial_equation_shaped` (the `#[cfg(test)]` TEXT entry point for the ceteris-paribus wrap; arrayed-per-element-equation (`Ast::Arrayed`) targets get one partial per element assembled into an `Equation::Arrayed`). **Production never parses a target equation**: `wrap_changed_first_ast` takes an `Expr0` lowered straight from the target's `Expr2` by `patch::expr2_to_expr0` (which is what `expr2_to_string` prints, so the former print->reparse was a parse of our own output), and every per-occurrence decision -- access shape, the GH #526 other-dep verdict, the literal-element index guard, and the `PerElement` row pinning -- is a lookup into the `db::ltm_ir` occurrence IR by the structural child-index path the wrap tracks, which equals the occurrence's `SiteId` BY CONSTRUCTION now that both walk the same tree. Every `PREVIOUS` the wrap SYNTHESIZES goes through `freeze_at_previous`, which chooses the call's first-DT initial value from the position being frozen: a VALUE position keeps the unary spelling (desugared to `0`, the XMILE-documented default, and unobservable behind the guard form's `TIME = INITIAL_TIME` arm), while a SUBSCRIPT INDEX names its own un-lagged operand -- `PREVIOUS(idx, idx)` -- because `0` is out of range for every 1-based dimension, so the frozen read yielded NaN at t=0 and `make_temp_arg`'s capture helper served that NaN as the score's FIRST LIVE step (GH #975). Only the two walkers that descend into indices need it (`wrap_non_matching_in_previous` via `wrap_index_non_matching_in_previous`, and `wrap_matching_in_previous`); `wrap_live_shaped_in_previous` and `freeze_pinned_body` document that they never do. A subscripted source reference the IR did NOT record -- reachable under a `LOOKUP` TABLE argument, which the walker skips as static data ("not a causal edge", which is right about ATTRIBUTION) -- still has to COMPILE, so the pin-only descent discharges it by NAME (`pin_dimension_name_indices`: an index naming one of the TARGET's iterated dimensions becomes the source element this target element reads on that axis, the same structural substitution `pin_bare_source_ref` performs for a bare `Var`). That is a lowering-completeness rule, not a second classifier: it consults no occurrence, infers no shape, and never makes the reference live-selectable -- it asks the SHARED row derivation `per_element_row_for_target` (hence `DimensionsContext::mapped_element_correspondence`) which element an axis reads, so the identity axis and a positionally-MAPPED one (`effect[State, old]` over an `effect[Region, Age]` source, either declaration direction) are one arm and it accepts exactly the mapped pairs `ltm_agg::classify_axis_access` accepts. An index the source's axis DECLARES as an element is resolved BEFORE that dimension-name reading, and that precedence is not the pin's own: it is the shared `dimensions::resolve_axis_index_name`, which `ltm_agg::classify_axis_access` reads too (GH #986 closed the divergence -- the classifier had the opposite order, so a mapped collision described the axis as `Iterated` over a dimension the compiler never iterates there). Element-first is what `compiler::subscript`'s `normalize_subscripts3` does ("First check if it's a named dimension element (takes priority)"), and the simulation is the authority: the two readings collide when a dimension declares an element whose name is also a dimension name (`Bucket = [old, region]` beside a `Region` dimension), and a describer that breaks the tie the other way names rows the simulation never reads (`a_colliding_index_name_reads_the_axis_element_in_the_simulation` is the numeric oracle; the XMILE spec's footnote [9] settles the adjacent VARIABLE-vs-element pair outright and sections 2.1/3.7.1 argue this pair from the namespace rule -- `resolve_axis_index_name`'s rustdoc says which is which). Everything that is NOT a bare identifier is left verbatim, needing no pin: a numeric literal, arithmetic over literals, an `@N` position (which `compiler::context` resolves to a concrete element offset in scalar context -- spelling it out here would be a second implementation of position syntax), and a compound expression selecting the element at RUNTIME. That last one used to be a conditional REFUSAL, and deleting it is GH #984: the wrap now freezes a `LOOKUP` table argument's index reads itself (`freeze_lookup_table_indices`), so a runtime index arrives here already lagged and the rule keeps it. That freeze WIDENS its own descent's dep set with the index idents, and without that it would not fire at all -- `variable::classify_dependencies`' `BuiltinContents::LookupTable` arm records the table's ident and never walks the table expression, so an index variable referenced only there is not a dependency and the wrap's `other_deps` freeze could never reach it. The widened set is scoped to that argument's indices, and the element / dimension-name guards run before the dep check, so it cannot make a selector wrap. (Leaving the dep set itself alone is deliberate: an index dropped from a variable's dependencies is a runlist-ordering question, not an LTM one.) What the SHARED derivation declines -- an unmapped or element-mapped pair (GH #756), a transposition, a dimension the target does not iterate -- is declined LOUDLY (`WrapOutcome::missing_occurrence` -> warned skip) rather than emitted with its dimension-name subscript intact, which would not resolve in a scalar fragment. The verdict space (`Pinned`, `Keep`, and the one loud `Unspellable`) is ENUMERATED cell by cell in `ltm_augment_pin_tests.rs`'s three verdict enumerations rather than sampled. There is ONE access-shape classifier family, on `Expr2`; the Expr0 mirror is test-support only and lives in `ltm_augment_wrap_test_support.rs` (kept because three wrap unit tests -- unparseable text, empty text, and the Fig. 2 Q4 `SUM(w[from]) + from` shape the engine REJECTS as a model -- cannot be db-backed fixtures), with `ltm_classifier_agreement_tests.rs` proving it matches the IR field for field (`SiteId` path, `shape`, `axes`, `in_reducer`) corpus-wide, `link_score_var_name` (synthetic name helper: Bare gets the canonical `{from}\u{2192}{to}` form, FixedIndex prepends `[elem]` to from; the obsolete per-shape `\u{205A}wildcard`/`\u{205A}dynamic` Wildcard/DynamicIndex suffixes were retired -- those shapes now collapse onto the Bare name, since *every statically-describable* inlined reducer (whole-extent or sliced) is hoisted into a `$⁚ltm⁚agg⁚{n}` node and only a `DynamicIndex` reference -- `arr[i+1]`, a range, or the not-hoistable dynamic-index reducer carve-out `SUM(pop[idx,*])` -- a whole-RHS variable-backed reducer's `Wildcard` argument, or a de-hoisted array-valued reducer's `Wildcard` arg (`RANK(pop[*], 1)`, GH #771) reach this function), `quote_ident` (identifier quoting for equations). Array support: `classify_reducer` (walks target Expr2 AST to identify reducing builtins -- Linear for SUM/MEAN, Nonlinear for MIN/MAX/STDDEV/RANK, Constant for SIZE -- a thin reader of `ltm_agg::reducer_kind`; it also hands back the reducer's array-argument AST as `ClassifiedReducer::body`, lowered from `Expr2` rather than printed, so the body-aware row partials never re-parse it), `generate_element_to_scalar_equation` (per-element link score equations for arrayed-to-scalar edges, used by both the variable-backed-reducer path and the `source[d] → $⁚ltm⁚agg⁚{n}` half) which dispatches on `ReducerKind` -- `generate_linear_partial` (SUM/MEAN algebraic shortcut), `generate_nonlinear_partial` (MIN/MAX nested binary calls; STDDEV the unrolled population-variance `sqrt` ceteris-paribus partial -- divisor `N`, matching `vm.rs::Opcode::ArrayStddev`, with the mean string-inlined; RANK the documented delta-ratio stand-in pinned by `test_generate_rank_keeps_delta_ratio` -- an order statistic, non-differentiable and unreachable via a real model RHS), `generate_scalar_to_element_equation` (per-element link score for the `$⁚ltm⁚agg⁚{n} → target[e]` half; takes a `source_ref_override: Option<&str>` so a multi-slot arrayed agg's `Δsource` denominator carries the projected `agg[]` subscript instead of the bare agg name, which wouldn't compile as a scalar), `substitute_reducers_in_expr0` (textually replaces a recognized reducer subexpression in an `Expr0` with its agg name, for the `$⁚ltm⁚agg⁚{n} → target` link score), `resolve_link_score_name_for_loop` (picks the Bare-or-FixedIndex link-score name a loop-score reference should target). Module link score formulas (black-box delta-ratio and composite-ref) are inlined directly into `module_link_score_equation` in `db.rs` (called by the per-shape `link_score_equation_text_shaped`). `subscript_idents_at_element` pins a target's arrayed deps for a per-element scalar partial, and it pins each one over the dimensions THAT DEP declares rather than over the target's element tuple (GH #974): a bare arrayed reference in an apply-to-all body reads its own axes' coordinates matched by dimension NAME, so a subset-dims dep (`w[Age]` under `growth[Region,Age]`) got an over-arity subscript whose fragment failed to compile, and a REORDERED one (`w[Age,Region]`) got a subscript that compiled and silently read the transposed element. The projection is `post_transform::dep_element_pins`/`dep_row_for_target`, reused by `pin_bare_source_ref` for a bare reference to the LIVE SOURCE (which is why a positionally-MAPPED bare source reference resolves through `mapped_element_correspondence` instead of being left bare and frozen into an uncompilable multi-slot `PREVIOUS`). The partial-equation builders (`build_partial_equation_shaped`/`_with_live_ref`, `subscript_idents_at_element`) and every link-score equation generator return `Result<_, PartialEquationError>`: a parse failure (genuine `Err`, or an empty `Ok(None)` equation) has no AST to PREVIOUS-wrap, so emitting the unwrapped input would silently produce a non-ceteris-paribus "partial" identical to the full equation (link score magnitude constant |Δz/Δz| = 1) -- a hidden attribution error that compiles cleanly (GH #311). The db-bearing callers (`link_score_equation_text_shaped` and the `src/db/ltm/link_scores.rs` emitters) convert the error into a `Warning` (`emit_ltm_partial_equation_warning`, naming the variable + offending equation text) and skip the variable -- distinct from `model_ltm_fragment_diagnostics`, which only catches *compile* failures. The failure is effectively unreachable in production (the text is always a `print_eqn` re-print; an empty equation is rejected as an `EmptyEquation` Error upstream), so this is defense-in-depth. The `#[cfg(test)]` tests live in the sibling `src/ltm_augment_tests.rs` (split out for the per-file line cap). Six more siblings are `#[path]`-mounted into `ltm_augment` purely for that cap, so every caller still names their items `crate::ltm_augment::*`: **`ltm_augment_partial_error.rs`** (the `PartialEquationError`/`PartialEquationErrorKind` loud-failure vocabulary plus the `contains_rank_like_builtin` walk the `RankLikePartial` class is decided by), **`ltm_augment_occurrence.rs`** (the wrap's read side of the occurrence IR -- `SlotOccurrences` groups a target's stream by slot ONCE and is the only way to obtain an `OccurrenceLookup`, so the borrow forces callers to hoist it out of their per-element loop), **`ltm_augment_post_transform.rs`** (the concrete-form lowerings: the agg-name substitution, and the `PerElement` row pinning the wrap calls AS IT DESCENDS -- the wrap is the only place that knows both the occurrence and whether it is about to FREEZE the reference, which is what picks the bare row for the live occurrence over the qualified row for every other one), **`ltm_augment_with_lookup.rs`** (the GH #910 implicit-WITH-LOOKUP rules), **`ltm_augment_wrap_test_support.rs`** (the `#[cfg(test)]` occurrence reconstruction + the Expr0 classifier mirror described above), and **`ltm_augment_freeze.rs`** (the GH #975 first-DT initial value of every synthesized `PREVIOUS`). - **`src/ltm_augment_with_lookup.rs`** - The implicit-WITH-LOOKUP rules for LTM (GH #910), re-exported from `ltm_augment` so every `crate::ltm_augment::*` path is unchanged. A `v = WITH LOOKUP(input, table)` variable is lowered by the compiler to `LOOKUP(v, input)` (`compiler::apply_implicit_with_lookup`), so a link-score partial that RE-EVALUATES such a target's equation is in gf-INPUT units while the guard form's `PREVIOUS(target)` anchor is in gf-OUTPUT units. `is_implicit_with_lookup` carries the coverage doc: every partial is either a **full re-evaluation** (class 1 -- must be wrapped in the gf application) or a **delta-ratio stand-in** (class 2 -- the RANK arm and the nested-arithmetic arm, already in output units, must NEVER be wrapped or a gf output is fed back through the gf). `WithLookupSlotRefs` resolves the target's table reference ONCE per target (`NoGf` / `Shared` / `PerElement`), so a per-element-gf target costs one row-major `SubscriptIterator` walk rather than one per element; an arrayed target's shared table is pinned as `to[1,...]` because a bare arrayed reference resolves each iterated element's own table offset, which the VM reads as NaN past `table_count`. `compose_with_lookup_polarity` (in `src/ltm/polarity.rs`) is the polarity twin, mirroring `apply_implicit_with_lookup`'s placeholder-Positive and zero-point-table rules. The polarity tests live in `src/ltm/with_lookup_tests.rs`, a child module of `ltm::tests`. - **`src/ltm_post.rs`** - Post-simulation relative loop *and link* score computation. `compute_rel_loop_scores(results, loop_partitions)` normalizes each loop's `loop_score` series against the sum of absolute scores within its cycle partition, using SAFEDIV-0 semantics (zero denominator -> zero result). Called after simulation rather than emitted as synthetic equations to avoid O(P^2) equation-text growth on models with dense partitions. `loop_partitions` is an `IndexMap` (re-exported `engine::indexmap`). `compute_rel_loop_scores*` walk its **emission** order rather than re-sorting the loop ids: the partition-sum denominator accumulates `|loop_score|` in that order, and emission order keeps the IEEE-754 (non-associative) sum bit-for-bit identical to the pre-#461 compile-time emitter (GH #468). Emission order is itself deterministic across salsa cache invalidations / processes because `assign_loop_ids` orders loops by a content-derived key (`ltm::graph::loop_id_sort_key`), so it never flaps even though `IndexMap`'s `PartialEq` (salsa cache equality) is order-insensitive. `compute_rel_link_scores(links, step_count)` is the link-level analogue (GH #652): raw link scores divide by the change in the *target*, so they are not comparable across targets and ranking by raw magnitude surfaces numerically-degenerate links (near-constant targets blow up the score). It groups the input `RelLinkInput { to, score }` links by their `to` target and normalizes each link's score by the per-target, per-timestep sum of `|score|` over all that target's *scored* inputs -- a **signed** value in `[-1, 1]` (sign kept like `compute_rel_loop_scores`), with the same `denom_summand` NaN-exclusion / Inf-retention / SAFEDIV-0 semantics. Denominator scope is the scored inputs only: complete in discovery mode (every causal edge scored) but covering just the in-loop subset in exhaustive mode (documented caveat on the fn). libsimlin's `analyze_links_core` calls it over the final (post-synthetic-collapse) link set so the per-target denominator matches the links the caller receives. - **`src/ltm_dominance.rs`** - Dominant-period selection over LTM loop importance series (GH #998): `FeedbackLoop` (a loop + its signed partition-relative importance series), `DominantPeriod`, the coarse 3-way display `LoopPolarity`, and `calculate_dominant_periods` -- the per-cycle-partition Praxis-style selection, parameterized by the caller-declared `PartitionSurface` (`PartitionBearing`: each partition-`None` loop is its own solo dominance group, mirroring discovery's `NormGroup::Solo`; `NoMetadata`: the flat legacy group, used only by layout's persisted-loop-metadata fallback). Two consumer families: `analysis::analyze_model` (the discovery surface, reaching FFI/pysimlin/MCP/TS via `ModelAnalysis::dominant_loops_by_period`) and the layout pipeline (`layout::detect_ltm_loops` + `layout::metadata::ComputedMetadata`). These types historically lived in `layout::metadata`; they were moved out so the LTM/analysis surface does not depend on a layout submodule -- layout consuming LTM is the right dependency direction, the reverse was not. diff --git a/src/simlin-engine/examples/layout_probe_models.rs b/src/simlin-engine/examples/layout_probe_models.rs new file mode 100644 index 000000000..11f8a3953 --- /dev/null +++ b/src/simlin-engine/examples/layout_probe_models.rs @@ -0,0 +1,63 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! One-shot helper: give each vensim-probes/*.mdl a generated diagram so the +//! probes open with a visible view in Vensim, WITHOUT touching the equations. +//! +//! The obvious route -- read, lay out, and re-serialize the whole project +//! through the MDL writer -- rewrites the equation section too, and the writer +//! spells an apply-to-all equation per element. That changes exactly what a +//! probe asks Vensim to parse (an element-pinned left-hand side over a +//! right-hand side naming subscript ranges), so instead the generated output is +//! used only as a donor: the sketch block between the `\\\---///` and +//! `///---\\\` markers is spliced into the original file, whose hand-written +//! equation text stays byte-identical. Sketch entries reference variables by +//! name, so a donor sketch over the original equations is well-formed. + +use std::fs; + +const SKETCH_START: &str = "\\\\\\---///"; +const SKETCH_END: &str = "///---\\\\\\"; + +fn sketch_block(mdl: &str) -> &str { + let start = mdl.find(SKETCH_START).expect("no sketch start marker"); + let end = mdl.find(SKETCH_END).expect("no sketch end marker") + SKETCH_END.len(); + &mdl[start..end] +} + +fn main() { + for path in [ + "vensim-probes/elm_map_computed_source.mdl", + "vensim-probes/elm_map_variable_sources.mdl", + "vensim-probes/repeated_dimension.mdl", + ] { + let original = fs::read_to_string(path).unwrap_or_else(|e| panic!("read {path}: {e}")); + let mut project = simlin_engine::compat::open_vensim(&original) + .unwrap_or_else(|e| panic!("open {path}: {e}")); + let model_name = project.models[0].name.clone(); + let view = simlin_engine::layout::generate_best_layout(&project, &model_name, None) + .unwrap_or_else(|e| panic!("layout {path}: {e}")); + project.models[0].views = vec![simlin_engine::datamodel::View::StockFlow(view)]; + let (rendered, warnings) = simlin_engine::compat::to_mdl_with_warnings(&project) + .unwrap_or_else(|e| panic!("render {path}: {e}")); + for w in &warnings { + eprintln!("warning ({path}): {}", w.message); + } + + let donor = sketch_block(&rendered); + let start = original.find(SKETCH_START).expect("no sketch in original"); + let end = original + .find(SKETCH_END) + .expect("no sketch end in original") + + SKETCH_END.len(); + let spliced = format!("{}{}{}", &original[..start], donor, &original[end..]); + assert_eq!( + &spliced[..start], + &original[..start], + "equation text must be untouched" + ); + fs::write(path, spliced).unwrap_or_else(|e| panic!("write {path}: {e}")); + println!("spliced sketch into {path}"); + } +} diff --git a/src/simlin-engine/examples/ltm_declined_edges.rs b/src/simlin-engine/examples/ltm_declined_edges.rs new file mode 100644 index 000000000..8090e510a --- /dev/null +++ b/src/simlin-engine/examples/ltm_declined_edges.rs @@ -0,0 +1,106 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Every LTM link score a model DECLINES to emit, bucketed by the reason the +//! generator gave. +//! +//! `examples/ltm_fragment_failures.rs` counts fragments that fail to COMPILE; +//! this counts the ones never generated at all -- the `PartialEquationError` +//! family (unprojectable dep, rank-like partial, unfreezable partial, bare +//! reducer feeder, parse failure) plus the GH #758 loud skip. Those are +//! invisible to the fragment count precisely because nothing was emitted. +//! +//! Usage: +//! cargo run --release -p simlin-engine --example ltm_declined_edges +//! LTM_DECLINE_MODEL=path/to/model.mdl cargo run --release ... --example ltm_declined_edges + +use std::collections::BTreeMap; +use std::path::PathBuf; + +use simlin_engine::db::{ + SimlinDb, collect_all_diagnostics, set_project_ltm_enabled, sync_from_datamodel_incremental, +}; +use simlin_engine::{open_vensim, open_xmile}; + +/// Which decline this diagnostic reports, keyed off the message's own wording +/// (the messages are the only channel `collect_all_diagnostics` exposes). +fn bucket(msg: &str) -> Option<&'static str> { + if !msg.contains("could not be generated") && !msg.contains("no link score") { + return None; + } + let kinds = [ + ( + "cannot be projected onto that target element", + "unprojectable-dep", + ), + ("array-producing", "rank-like-partial"), + ("freeze an array slice", "unfreezable-partial"), + ("inside an array-reducer argument", "bare-reducer-feeder"), + ("did not parse", "parse-failure"), + ]; + for (needle, name) in kinds { + if msg.contains(needle) { + return Some(name); + } + } + Some("other") +} + +fn main() { + let model_path = std::env::var("LTM_DECLINE_MODEL") + .map(PathBuf::from) + .unwrap_or_else(|_| { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../test/xmutil_test_models/C-LEARN v77 for Vensim.mdl") + }); + + let contents = std::fs::read_to_string(&model_path).expect("read model"); + let datamodel = if model_path.extension().is_some_and(|e| e == "mdl") { + open_vensim(&contents).expect("import vensim model") + } else { + open_xmile(&mut contents.as_bytes()).expect("import xmile model") + }; + println!("model: {}", model_path.display()); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &datamodel, None); + set_project_ltm_enabled(&mut db, sync.project, true); + + let diags = collect_all_diagnostics(&db, sync.project); + let mut by_bucket: BTreeMap<&'static str, Vec> = BTreeMap::new(); + for d in &diags { + let msg = format!("{:?}", d.error); + if let Some(b) = bucket(&msg) { + // The link-score variable name is the quoted ident right after + // "variable '". + let name = msg + .split_once("variable '") + .and_then(|(_, rest)| rest.split_once('\'')) + .map(|(n, _)| n.to_string()) + .unwrap_or_else(|| msg.clone()); + // The offending dep / equation text, the second quoted run. + let detail = msg + .split_once("dependency '") + .or_else(|| msg.split_once("equation '")) + .and_then(|(_, rest)| rest.split_once('\'')) + .map(|(d, _)| d.to_string()) + .unwrap_or_default(); + by_bucket + .entry(b) + .or_default() + .push(format!("{name} [{detail}]")); + } + } + + let total: usize = by_bucket.values().map(Vec::len).sum(); + println!("declined link scores: {total}"); + for (b, names) in &by_bucket { + println!("\n=== {b}: {} ===", names.len()); + let mut names = names.clone(); + names.sort(); + for n in &names { + println!(" {n}"); + } + } +} diff --git a/src/simlin-engine/examples/ltm_slot_width.rs b/src/simlin-engine/examples/ltm_slot_width.rs new file mode 100644 index 000000000..dd29b6a29 --- /dev/null +++ b/src/simlin-engine/examples/ltm_slot_width.rs @@ -0,0 +1,67 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! The two numbers `simulate::clearn_ltm_var_count_guardrail` pins: a model's +//! emitted LTM variable COUNT and its per-step result-row WIDTH in slots (the +//! GH #654 resource, against the VM's 65,536 u16 slot ceiling). +//! +//! The guard's rustdoc requires re-measuring BOTH whenever the count moves, and +//! the width is not derivable from the count -- an arrayed variable occupies one +//! slot per element. This is the harness that produces them, so the numbers in +//! that rustdoc are regenerable rather than folklore. +//! +//! Usage: +//! cargo run --release -p simlin-engine --example ltm_slot_width +//! LTM_WIDTH_MODEL=path/to/model.mdl cargo run --release ... --example ltm_slot_width + +use std::path::PathBuf; + +use simlin_engine::db::{ + SimlinDb, model_ltm_variables, set_project_ltm_enabled, sync_from_datamodel_incremental, +}; +use simlin_engine::queue_compile::compile_sim; +use simlin_engine::{open_vensim, open_xmile}; + +fn main() { + let model_path = std::env::var("LTM_WIDTH_MODEL") + .map(PathBuf::from) + .unwrap_or_else(|_| { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../test/xmutil_test_models/C-LEARN v77 for Vensim.mdl") + }); + + let contents = std::fs::read_to_string(&model_path).expect("read model"); + let datamodel = if model_path.extension().is_some_and(|e| e == "mdl") { + open_vensim(&contents).expect("import vensim model") + } else { + open_xmile(&mut contents.as_bytes()).expect("import xmile model") + }; + println!("model: {}", model_path.display()); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &datamodel, None); + set_project_ltm_enabled(&mut db, sync.project, true); + + let total: usize = sync + .models + .values() + .map(|m| { + model_ltm_variables(&db, m.source_model, sync.project) + .vars + .len() + }) + .sum(); + println!("emitted LTM variables: {total}"); + + let main_name = datamodel + .models + .iter() + .find(|m| m.name == "main") + .map(|m| m.name.clone()) + .unwrap_or_else(|| datamodel.models[0].name.clone()); + let build = compile_sim(&mut db, sync.project, &datamodel, &main_name).expect("compile"); + let width = build.compiled.n_slots(); + println!("per-step result-row width: {width} slots"); + println!("free against the 65,536-slot ceiling: {}", 65536 - width); +} diff --git a/src/simlin-engine/examples/ltm_var_dump.rs b/src/simlin-engine/examples/ltm_var_dump.rs new file mode 100644 index 000000000..2df751668 --- /dev/null +++ b/src/simlin-engine/examples/ltm_var_dump.rs @@ -0,0 +1,43 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Every LTM variable name a model emits, `modelname`, sorted -- the +//! instrument behind `simulate::clearn_ltm_var_count_guardrail`'s derivation. +//! +//! The guardrail pins a COUNT, which says a number moved but not which names +//! moved or in which direction. Diffing two runs of this does: +//! +//! ```text +//! cargo run --release -p simlin-engine --example ltm_var_dump > after.txt +//! # (revert the change under test) +//! cargo run --release -p simlin-engine --example ltm_var_dump > before.txt +//! comm -13 <(sort before.txt) <(sort after.txt) # added +//! comm -23 <(sort before.txt) <(sort after.txt) # removed +//! ``` +//! +//! That is how the MDL apply-to-all import fix was shown to be strictly +//! additive (315 added, 0 removed) rather than a wash of gains and losses. +use simlin_engine::db::{ + SimlinDb, model_ltm_variables, set_project_ltm_enabled, sync_from_datamodel_incremental, +}; +fn main() { + let path = concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../test/xmutil_test_models/C-LEARN v77 for Vensim.mdl" + ); + let contents = std::fs::read_to_string(path).expect("read model"); + let datamodel = simlin_engine::open_vensim(&contents).expect("import"); + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &datamodel, None); + set_project_ltm_enabled(&mut db, sync.project, true); + let mut names: Vec = Vec::new(); + for (model_name, m) in sync.models.iter() { + let ltm = model_ltm_variables(&db, m.source_model, sync.project); + names.extend(ltm.vars.iter().map(|v| format!("{model_name}\t{}", v.name))); + } + names.sort_unstable(); + for n in &names { + println!("{n}"); + } +} diff --git a/src/simlin-engine/examples/mdl_compile_census.rs b/src/simlin-engine/examples/mdl_compile_census.rs new file mode 100644 index 000000000..e86df4ff7 --- /dev/null +++ b/src/simlin-engine/examples/mdl_compile_census.rs @@ -0,0 +1,71 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Import every `.mdl` under `test/` and report which ones fail, and why. +//! +//! Prints one `IMPORT-FAIL` / `COMPILE-FAIL` line per failing model with its +//! diagnostics, and a summary to stderr. Diffing two runs is how a change to the +//! MDL importer is shown not to regress the corpus: the apply-to-all import fix +//! moved exactly one model (`sdeverywhere/models/vector/vector.mdl`, which had +//! been failing codegen on `y`'s dimension arithmetic) from fail to ok, and +//! moved none the other way, across 262 files. +//! +//! The remaining failures are pre-existing and unrelated -- unimplemented Vensim +//! builtins dominate -- so the summary counts are a ratchet, not a target. +use std::path::{Path, PathBuf}; +fn walk(dir: &Path, out: &mut Vec) { + let Ok(rd) = std::fs::read_dir(dir) else { + return; + }; + for e in rd.flatten() { + let p = e.path(); + if p.is_dir() { + walk(&p, out); + } else if p.extension().is_some_and(|x| x == "mdl") { + out.push(p); + } + } +} +fn main() { + let root = PathBuf::from(concat!(env!("CARGO_MANIFEST_DIR"), "/../../test")); + let mut files = Vec::new(); + walk(&root, &mut files); + files.sort(); + let (mut ok, mut import_err, mut compile_err) = (0, 0, 0); + for f in &files { + let rel = f.strip_prefix(&root).unwrap_or(f).display().to_string(); + let Ok(contents) = std::fs::read_to_string(f) else { + continue; + }; + let dm = match simlin_engine::open_vensim(&contents) { + Ok(d) => d, + Err(e) => { + import_err += 1; + println!("IMPORT-FAIL\t{rel}\t{e}"); + continue; + } + }; + // Use the production incremental path: sync + collect diagnostics. + let mut db = simlin_engine::db::SimlinDb::default(); + let sync = simlin_engine::db::sync_from_datamodel_incremental(&mut db, &dm, None); + let diags = simlin_engine::db::collect_all_diagnostics(&db, sync.project); + let mut msgs: Vec = diags + .iter() + .filter(|d| d.severity == simlin_engine::db::DiagnosticSeverity::Error) + .map(|d| format!("{}:{:?}", d.variable.as_deref().unwrap_or("-"), d.error)) + .collect(); + msgs.sort(); + msgs.dedup(); + if msgs.is_empty() { + ok += 1; + } else { + compile_err += 1; + println!("COMPILE-FAIL\t{rel}\t{}", msgs.join(" | ")); + } + } + eprintln!( + "total={} ok={ok} import_err={import_err} compile_err={compile_err}", + files.len() + ); +} diff --git a/src/simlin-engine/src/array_operand_materialization_tests.rs b/src/simlin-engine/src/array_operand_materialization_tests.rs new file mode 100644 index 000000000..a7f48377e --- /dev/null +++ b/src/simlin-engine/src/array_operand_materialization_tests.rs @@ -0,0 +1,2837 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Codegen consumes an array-valued operand as a **view over storage** +//! (`compiler::codegen::walk_expr_as_view`), so a *computed* array -- `vals[d] +//! + bump[d]`, `NOT ...`, an `IF` over two arrays -- has to be evaluated into a +//! temp of its own before the builtin that reads it. GH #995: the shapes below +//! failed to compile at all, in ordinary hand-written apply-to-all models, with +//! LTM disabled. `compiler::array_operand` is the fix. +//! +//! # The enumeration these rows are derived from +//! +//! **Positions.** Every `walk_expr_as_view` call site in `codegen.rs` is one +//! row of the position axis; there is no other way for an operand to be +//! required to be a view. In source order: +//! +//! | call site | position | covered by | +//! |---|---|---| +//! | `emit_array_reduce` | `SUM`/`SIZE`/`STDDEV`/`MIN`(1-arg)/`MAX`(1-arg)/`MEAN`(1-arg) arg0 | [`reducer_positions`] | +//! | `VectorSelect` | arg0 (selection), arg1 (values) | [`vector_select_positions`] | +//! | `VectorElmMap` | arg0 (source), arg1 (offsets) | [`vector_elm_map_positions`] | +//! | `VectorSortOrder` | arg0 (array) | [`vector_sort_order_positions`] | +//! | `Rank` | arg0 (array) | [`rank_positions`] | +//! | `Lookup`/`LookupForward`/`LookupBackward` | arg0 (arrayed GF table) | [`deliberately_unmaterialized_positions`] | +//! | `AllocateAvailable` | arg0 (requests) | [`allocate_positions`] | +//! | `AllocateAvailable` | arg1 (priority profiles) | [`deliberately_unmaterialized_positions`] | +//! | `AllocateByPriority` | arg0 (requests), arg1 (priorities) | [`allocate_positions`] | +//! +//! Two positions are deliberately **not** materialized, and each is pinned as +//! still-failing rather than left unstated -- see +//! [`deliberately_unmaterialized_positions`] for the reasons, which live next +//! to the code in `compiler::array_operand`. +//! +//! **Shapes.** The materializer's decision is `is_view`, the negation of +//! `walk_expr_as_view`'s four storage-view arms, plus "an array view can be +//! derived for it", minus "it already IS a view over a snapshot buffer" +//! (`is_snapshot_view`, the C3 shape). The shape axis is therefore the set of +//! *rejected* `compiler::Expr` variants that can carry an array: `Op2`, `Op1`, +//! `If`, and `App` (an elementwise builtin -- +//! [`elementwise_builtin_operands_materialize`] covers the two families +//! `find_expr_array_view` recognises -- or a nested array-producing builtin). +//! +//! **The collapse.** Position and shape are decided by two separate, singly +//! implemented pieces of code -- one `match` over `BuiltinFn` naming the view +//! positions, and one shared `materialize_view_operand` that all of them call. +//! So the matrix is covered as a cross rather than as a full product: *every* +//! position is exercised with one computed shape (an `Op2`) plus its +//! already-compiling control, and *every* shape is exercised at one position +//! (`VECTOR SORT ORDER` arg0). The two spellings of an apply-to-all reference +//! -- `vals[d]`, which only means "the whole array" after `context.rs`'s +//! `with_vector_builtin_wildcards` promotion, and `vals[*]` -- are a third +//! axis, covered at `VECTOR SORT ORDER` arg0 and `RANK` arg0, the two arms the +//! issue reports separately. +//! +//! **Why some rows carry a `+ SUM(VECTOR SORT ORDER(vals[*], 1))` tail.** A +//! reducer or `VECTOR SELECT` argument only survives Pass 1 unmaterialized +//! when the equation *also* holds an array-producing builtin: that is what +//! makes `compiler::mod`'s apply-to-all hoister lower through +//! `lower_preserving_dimensions`, whose `Pass1Context` has no apply-to-all +//! context and so defers every operand carrying a dimension reference. The +//! tail is the smallest thing that forces that path; it contributes the +//! constant 1 + 2 + 0 = 3. +//! +//! **Values.** Every compiling row asserts VM numbers, chosen so that reading +//! the *wrong* array gives a different answer than reading the computed one: +//! `vals = [30, 10, 20]` and `vals + bump = [30, 110, 20]` have different sort +//! orders, different ranks and different element-map results, and each wrong +//! rule (read `vals` raw, read `bump` raw, collapse the operand to its first +//! element) lands on its own distinct answer. The value each wrong rule would +//! produce is written next to the assertion. +//! +//! `PREVIOUS`/`INIT` of an arrayed reference is Phase C3 (GH #995's option D), +//! and it is a FIFTH view shape rather than a computed array: the call reads its +//! argument's view out of one of the VM's snapshot buffers. Its rows live in +//! their own section below, over a TIME-VARYING fixture -- a constant fixture +//! cannot tell a previous value from a current one, so every row there asserts +//! series rather than single arrays. The complement stays a green row: +//! [`a_scalar_previous_beside_an_array_operand_still_materializes`], without +//! which "array-valued" could widen to "any `PREVIOUS`" unnoticed and a scalar +//! `PREVIOUS(s)` would stop broadcasting. + +use crate::common::ErrorCode; +use crate::test_common::TestProject; + +/// The shared fixture. Values are chosen so a computed operand's answer +/// differs from the answer produced by reading either raw input -- see the +/// module docs. +/// +/// * `vals = [30, 10, 20]` +/// * `bump = [0, 100, 0]` (so `vals + bump = [30, 110, 20]`) +/// * `offs = [2, 0, 1]` +/// * `shift = [-2, 1, 0]` (so `offs + shift = [0, 1, 1]`) +/// * `sel = [1, 1, 0]` +/// * `mask = [1, 0, 0]` (so `sel - mask = [0, 1, 0]`) +/// * `matrix = [[1, 2, 3], [10, 20, 30]]` +fn fixture(name: &str) -> TestProject { + TestProject::new(name) + .indexed_dimension("d", 3) + .indexed_dimension("e", 2) + .array_with_ranges("vals[d]", vec![("1", "30"), ("2", "10"), ("3", "20")]) + .array_with_ranges("bump[d]", vec![("1", "0"), ("2", "100"), ("3", "0")]) + .array_with_ranges("offs[d]", vec![("1", "2"), ("2", "0"), ("3", "1")]) + .array_with_ranges("shift[d]", vec![("1", "-2"), ("2", "1"), ("3", "0")]) + .array_with_ranges("sel[d]", vec![("1", "1"), ("2", "1"), ("3", "0")]) + .array_with_ranges("mask[d]", vec![("1", "1"), ("2", "0"), ("3", "0")]) + .array_with_ranges( + "matrix[e,d]", + vec![ + ("1,1", "1"), + ("1,2", "2"), + ("1,3", "3"), + ("2,1", "10"), + ("2,2", "20"), + ("2,3", "30"), + ], + ) +} + +/// Compile `out[d] = ` against the shared fixture and return `out`. +fn out_of(name: &str, eqn: &str) -> Vec { + let project = fixture(name).array_aux("out[d]", eqn); + project.assert_compiles_incremental(); + project.vm_result_incremental("out") +} + +/// Compile `out[e] = ` against the shared fixture and return `out`. Used +/// by the reducer rows, whose argument is a row slice `matrix[e,*]`. +fn row_out_of(name: &str, eqn: &str) -> Vec { + let project = fixture(name).array_aux("out[e]", eqn); + project.assert_compiles_incremental(); + project.vm_result_incremental("out") +} + +fn assert_close(actual: &[f64], expected: &[f64], what: &str) { + assert_eq!( + actual.len(), + expected.len(), + "{what}: length mismatch, got {actual:?}" + ); + for (i, (a, e)) in actual.iter().zip(expected.iter()).enumerate() { + assert!( + (a - e).abs() < 1e-9, + "{what}: element {i} expected {e}, got {a} (whole array {actual:?})" + ); + } +} + +/// A row that is still expected to fail: it must fail, and it must still name +/// the variable it failed for (GH #994's attribution), not fail anonymously. +fn assert_fails_attributed(project: TestProject, what: &str) { + let err = project + .compile_incremental() + .err() + .unwrap_or_else(|| panic!("{what}: compiled, but this shape is not supposed to")); + assert_eq!( + err.code, + ErrorCode::NotSimulatable, + "{what}: expected a NotSimulatable rejection, got {err:?}" + ); + let details = err.get_details().unwrap_or_default(); + assert!( + details.contains("out"), + "{what}: the rejection must name the variable it belongs to, got {details:?}" + ); +} + +/// A row that must fail for a STATED reason, checked against the per-variable +/// diagnostic (the surface a user reads) rather than against the aggregate +/// assembly error, which names only the variable. +fn assert_declines_because(project: TestProject, variable: &str, reason: &str) { + use crate::db::{DiagnosticError, SimlinDb, collect_all_diagnostics, sync_from_datamodel}; + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let diags = collect_all_diagnostics(&db, sync.project); + let matched = diags.iter().any(|d| { + d.variable.as_deref() == Some(variable) + && matches!(&d.error, DiagnosticError::Assembly(msg) if msg.contains(reason)) + }); + assert!( + matched, + "expected a diagnostic for '{variable}' containing {reason:?}; got: {diags:?}" + ); +} + +// =========================================================================== +// Position axis: one computed operand (an `Op2`) per `walk_expr_as_view` call +// site, plus the already-compiling control. +// =========================================================================== + +#[test] +fn vector_sort_order_positions() { + // Control: a plain reference is already a view, and already compiled. + // `vals = [30, 10, 20]` ascending: 10@1, 20@2, 30@0. + assert_close( + &out_of("vso_ctl", "VECTOR SORT ORDER(vals[d], 1)"), + &[1.0, 2.0, 0.0], + "control: VECTOR SORT ORDER over a direct reference", + ); + + // Computed: `vals + bump = [30, 110, 20]` ascending: 20@2, 30@0, 110@1. + // Reading `vals` raw would give [1, 2, 0]; reading `bump` raw would give + // [0, 2, 1]; collapsing to a 1-element view would give [0, 0, 0]. + assert_close( + &out_of("vso_arg0", "VECTOR SORT ORDER(vals[d] + bump[d], 1)"), + &[2.0, 0.0, 1.0], + "VECTOR SORT ORDER arg0, computed", + ); +} + +#[test] +fn rank_positions() { + // Control. RANK is 1-based: `vals = [30, 10, 20]` ascending ranks are + // 30 -> 3, 10 -> 1, 20 -> 2. + assert_close( + &out_of("rank_ctl", "RANK(vals[d], 1)"), + &[3.0, 1.0, 2.0], + "control: RANK over a direct reference", + ); + + // Computed: `[30, 110, 20]` ascending ranks are 30 -> 2, 110 -> 3, + // 20 -> 1. Reading `vals` raw would give [3, 1, 2]; reading `bump` raw + // would give [1, 3, 2]; a collapsed view would give [1, 1, 1]. + assert_close( + &out_of("rank_arg0", "RANK(vals[d] + bump[d], 1)"), + &[2.0, 3.0, 1.0], + "RANK arg0, computed", + ); +} + +#[test] +fn vector_elm_map_positions() { + // Control: `offs = [2, 0, 1]` maps `vals = [30, 10, 20]` to + // [vals[2], vals[0], vals[1]] = [20, 30, 10]. + assert_close( + &out_of("elm_ctl", "VECTOR ELM MAP(vals[d], offs[d])"), + &[20.0, 30.0, 10.0], + "control: VECTOR ELM MAP over direct references", + ); + + // arg0 computed: the source becomes [30, 110, 20], mapped by + // offs = [2, 0, 1] to [20, 30, 110]. Reading `vals` raw would give + // [20, 30, 10]; reading `bump` raw would give [0, 0, 100]. + assert_close( + &out_of("elm_arg0", "VECTOR ELM MAP(vals[d] + bump[d], offs[d])"), + &[20.0, 30.0, 110.0], + "VECTOR ELM MAP arg0 (source), computed", + ); + + // arg1 computed: the offsets become `offs + shift = [0, 1, 1]`, so the + // result is [vals[0], vals[1], vals[1]] = [30, 10, 10]. Reading `offs` raw + // would give [20, 30, 10]; reading `shift` raw would put element 0 at + // offset -2, which is out of range and yields NaN. + assert_close( + &out_of("elm_arg1", "VECTOR ELM MAP(vals[d], offs[d] + shift[d])"), + &[30.0, 10.0, 10.0], + "VECTOR ELM MAP arg1 (offsets), computed", + ); +} + +#[test] +fn vector_select_positions() { + // VECTOR SELECT reduces to a scalar, so every element of `out` holds the + // same value; the `+ SUM(VECTOR SORT ORDER(vals[*], 1))` tail adds 3 and + // is what forces the lowering path these rows are about (module docs). + // + // Control: `sel = [1, 1, 0]` selects vals[0] + vals[1] = 40, plus 3. + assert_close( + &out_of( + "vsel_ctl", + "VECTOR SELECT(sel[d], vals[d], 0, 0, 0) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[43.0, 43.0, 43.0], + "control: VECTOR SELECT over direct references", + ); + + // arg0 computed: `sel - mask = [0, 1, 0]` selects vals[1] = 10, plus 3. + // Reading `sel` raw would give 43; reading `mask` raw would give 33; a + // collapsed 1-element view would select nothing and fall back to the + // max_value argument, giving 3. + assert_close( + &out_of( + "vsel_arg0", + "VECTOR SELECT(sel[d] - mask[d], vals[d], 0, 0, 0) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[13.0, 13.0, 13.0], + "VECTOR SELECT arg0 (selection array), computed", + ); + + // arg1 computed: `sel = [1, 1, 0]` over `vals + bump = [30, 110, 20]` + // selects 30 + 110 = 140, plus 3. Reading `vals` raw would give 43; + // reading `bump` raw would give 103. + assert_close( + &out_of( + "vsel_arg1", + "VECTOR SELECT(sel[d], vals[d] + bump[d], 0, 0, 0) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[143.0, 143.0, 143.0], + "VECTOR SELECT arg1 (value array), computed", + ); +} + +#[test] +fn reducer_positions() { + // The five `emit_array_reduce` arms, over the row slice `matrix[e,*]` + // (rows [1, 2, 3] and [10, 20, 30]). A reducer keeps its argument as a row + // slice -- `with_preserved_wildcards` does NOT promote an active-dimension + // reference -- so `matrix[e,*] * 2` is [2, 4, 6] and [20, 40, 60]. Each + // row carries the `+ SUM(VECTOR SORT ORDER(vals[*], 1))` tail, worth 3. + + // Control: SUM of the raw rows is 6 and 60. + assert_close( + &row_out_of( + "red_ctl", + "SUM(matrix[e,*]) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[9.0, 63.0], + "control: SUM over a direct row slice", + ); + + // SUM: 12 and 120. Reading `matrix` raw would give 9 and 63. + assert_close( + &row_out_of( + "red_sum", + "SUM(matrix[e,*] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[15.0, 123.0], + "SUM over a computed array", + ); + + // MAX: 6 and 60. Reading `matrix` raw would give 6 and 33. + assert_close( + &row_out_of( + "red_max", + "MAX(matrix[e,*] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[9.0, 63.0], + "MAX over a computed array", + ); + + // MIN: 2 and 20. Reading `matrix` raw would give 4 and 13. + assert_close( + &row_out_of( + "red_min", + "MIN(matrix[e,*] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[5.0, 23.0], + "MIN over a computed array", + ); + + // SIZE counts elements: 3 either way. A collapsed operand would give 1, + // which is the failure mode this row rules out. + assert_close( + &row_out_of( + "red_size", + "SIZE(matrix[e,*] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[6.0, 6.0], + "SIZE over a computed array", + ); + + // STDDEV is the POPULATION deviation (`ArrayStddev` divides by n, not + // n - 1): sqrt(8/3) for [2, 4, 6] and sqrt(800/3) for [20, 40, 60]. + // Reading `matrix` raw would halve both. + let pop_stddev = |xs: [f64; 3]| -> f64 { + let mean = (xs[0] + xs[1] + xs[2]) / 3.0; + (xs.iter().map(|x| (x - mean).powi(2)).sum::() / 3.0).sqrt() + }; + assert_close( + &row_out_of( + "red_stddev", + "STDDEV(matrix[e,*] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[ + pop_stddev([2.0, 4.0, 6.0]) + 3.0, + pop_stddev([20.0, 40.0, 60.0]) + 3.0, + ], + "STDDEV over a computed array", + ); + + // Single-argument MEAN: means of [2,4,6] and [20,40,60] are 4 and 40. + // Reading `matrix` raw would give 2 and 20. This row is the one that + // makes MEAN agree with its four sibling reducers: before, an array-shaped + // MEAN argument fell through codegen's scalar fallback and failed to + // compile, so the `[*]` spelling array-meaned through Pass 1 while the + // `[e,*]` spelling did not compile at all. + assert_close( + &row_out_of( + "red_mean", + "MEAN(matrix[e,*] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[7.0, 43.0], + "MEAN over a computed array", + ); + // The variadic form has no view position and must be untouched: MEAN of + // three scalars is their average, 2, plus the tail. + assert_close( + &row_out_of( + "red_mean_variadic", + "MEAN(matrix[e,1], matrix[e,2], matrix[e,3]) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[5.0, 23.0], + "variadic MEAN is a scalar average and takes no view", + ); +} + +/// The ALLOCATE fixture: three requesters, a rectangular priority-profile +/// array, and the two "bump" arrays the computed-operand rows add in. +fn allocate_fixture(name: &str) -> TestProject { + TestProject::new(name) + .indexed_dimension("d", 3) + .indexed_dimension("xp", 4) + .array_with_ranges("request[d]", vec![("1", "10"), ("2", "20"), ("3", "30")]) + .array_with_ranges("extra[d]", vec![("1", "20"), ("2", "0"), ("3", "0")]) + .array_with_ranges("priority[d]", vec![("1", "3"), ("2", "1"), ("3", "2")]) + .array_with_ranges("prio_bump[d]", vec![("1", "0"), ("2", "5"), ("3", "0")]) + .array_with_ranges( + "pp[d,xp]", + vec![ + ("1,1", "1"), + ("1,2", "3"), + ("1,3", "1"), + ("1,4", "0"), + ("2,1", "1"), + ("2,2", "1"), + ("2,3", "1"), + ("2,4", "0"), + ("3,1", "1"), + ("3,2", "2"), + ("3,3", "1"), + ("3,4", "0"), + ], + ) + .scalar_const("supply", 35.0) + .scalar_const("width", 1.0) +} + +/// `ALLOCATE AVAILABLE` and `ALLOCATE BY PRIORITY` run a bisection over +/// per-requester allocation curves, so their element values are not +/// hand-computable the way a sort order is. Each row instead pins the +/// computed-operand model against the model that materializes the same array +/// into a named variable first -- the path that already compiled -- and +/// separately asserts it differs from the raw-operand model, so "the +/// computation was actually read" is asserted rather than assumed. +#[test] +fn allocate_positions() { + struct Row { + what: &'static str, + computed: &'static str, + helper: (&'static str, &'static str), + reference: &'static str, + raw: &'static str, + } + + let rows = [ + Row { + what: "allocate_available arg0 (requests)", + computed: "allocate_available(request[d] + extra[d], pp[d,1], supply)", + helper: ("req2[d]", "request[d] + extra[d]"), + reference: "allocate_available(req2[d], pp[d,1], supply)", + raw: "allocate_available(request[d], pp[d,1], supply)", + }, + Row { + what: "allocate_by_priority arg0 (requests)", + computed: "allocate_by_priority(request[d] + extra[d], priority[d], 0, width, supply)", + helper: ("req2[d]", "request[d] + extra[d]"), + reference: "allocate_by_priority(req2[d], priority[d], 0, width, supply)", + raw: "allocate_by_priority(request[d], priority[d], 0, width, supply)", + }, + Row { + what: "allocate_by_priority arg1 (priorities)", + computed: "allocate_by_priority(request[d], priority[d] + prio_bump[d], 0, width, supply)", + helper: ("prio2[d]", "priority[d] + prio_bump[d]"), + reference: "allocate_by_priority(request[d], prio2[d], 0, width, supply)", + raw: "allocate_by_priority(request[d], priority[d], 0, width, supply)", + }, + ]; + + for (i, row) in rows.iter().enumerate() { + let computed = allocate_fixture(&format!("alloc_c{i}")).array_aux("out[d]", row.computed); + computed.assert_compiles_incremental(); + let computed = computed.vm_result_incremental("out"); + + let reference = allocate_fixture(&format!("alloc_r{i}")) + .array_aux(row.helper.0, row.helper.1) + .array_aux("out[d]", row.reference); + reference.assert_compiles_incremental(); + let reference = reference.vm_result_incremental("out"); + + let raw = allocate_fixture(&format!("alloc_w{i}")).array_aux("out[d]", row.raw); + raw.assert_compiles_incremental(); + let raw = raw.vm_result_incremental("out"); + + assert_close( + &computed, + &reference, + &format!( + "{}: the inline computed operand must allocate exactly as the \ + pre-materialized helper does", + row.what + ), + ); + assert_ne!( + computed, raw, + "{}: the fixture must make the computed operand change the answer, \ + otherwise this row proves nothing (computed {computed:?}, raw {raw:?})", + row.what + ); + } +} + +/// The one view position the materializer deliberately declines, plus the two +/// arms that decline for a reason other than the position. The reasons live on +/// the arms in `compiler::array_operand::materialize_view_operands`; what is +/// pinned here is that each still fails loudly, or keeps its existing meaning, +/// rather than compiling to something wrong. +/// +/// The arrayed graphical-function table has no row: it is not constructible as +/// a computed expression from the equation language. `Lookup`'s table argument +/// is synthesized -- by `apply_implicit_with_lookup` for WITH LOOKUP and by the +/// table-holder resolution for `g[D!](x)` -- and is always a bare reference, so +/// the declining arm guards against a future producer rather than a shape +/// reachable today. (`SUM(LOOKUP(vals[d] * 2, 1))` is rejected earlier, at +/// table resolution: `vals` is not a graphical function at all.) +#[test] +fn deliberately_unmaterialized_positions() { + // ALLOCATE AVAILABLE's priority-profile argument: its view is rewritten + // by `context::expand_pp_view_for_allocate`, which re-expands a collapsed + // `pp[d,1]` to the variable's full requester x XPriority array. That + // helper only understands a direct variable reference, so materializing a + // computed profile would silently hand the VM a one-column-per-requester + // temp. + let pp_computed = TestProject::new("unmat_pp") + .indexed_dimension("d", 3) + .indexed_dimension("xp", 4) + .array_with_ranges("request[d]", vec![("1", "10"), ("2", "20"), ("3", "30")]) + .array_const("pp[d,xp]", 1.0) + .array_const("pp_bump[d,xp]", 0.0) + .scalar_const("supply", 35.0) + .array_aux( + "out[d]", + "allocate_available(request[d], pp[d,1] + pp_bump[d,1], supply)", + ); + assert_fails_attributed( + pp_computed, + "ALLOCATE AVAILABLE priority profiles, computed", + ); + + // A GENUINELY scalar argument is left alone, because no array view can be + // derived for it -- `matrix[e,1] * 2` is two scalars. MEAN is the only + // reduce arm this is observable through: codegen's `Mean` arm emits a + // plain scalar walk for anything that is not one of the four view shapes, + // where `emit_array_reduce` (SUM/SIZE/STDDEV/MIN/MAX) pushes a view + // unconditionally and rejects a scalar expression with or without this + // pass. So the row that matters is MEAN's, and it must keep its value. + assert_close( + &row_out_of( + "unmat_mean_scalar", + "MEAN(matrix[e,1] * 2) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ), + &[5.0, 23.0], + "MEAN of a computed SCALAR stays a scalar mean", + ); + + // The SAME position also refuses a `PREVIOUS`/`INIT` (GH #995 phase C3), + // and for the same reason -- see + // `a_snapshot_priority_profile_declines_rather_than_allocating_over_one_column` + // for the wrong allocation it prevents and the workaround that compiles. + for (name, eqn) in [ + ( + "unmat_pp_prev", + "allocate_available(request[d], PREVIOUS(pp[d,1]), supply)", + ), + ( + "unmat_pp_init", + "allocate_available(request[d], INIT(pp[d,1]), supply)", + ), + ] { + assert_fails_attributed( + TestProject::new(name) + .indexed_dimension("d", 3) + .indexed_dimension("xp", 4) + .array_with_ranges("request[d]", vec![("1", "10"), ("2", "20"), ("3", "30")]) + .array_const("pp[d,xp]", 1.0) + .scalar_const("supply", 35.0) + .array_aux("out[d]", eqn), + eqn, + ); + } +} + +/// The pp-position decline, with the fixture that shows what it prevents. +/// +/// `pp` is CONSTANT here, so `PREVIOUS(pp[d,1])` and `pp[d,1]` hold the same +/// numbers at every step after the first: any difference in the allocation is +/// therefore a SHAPE defect and nothing else. Without the guard the frozen form +/// compiled and allocated over a one-column-per-requester profile -- a silently +/// wrong allocation where HEAD failed loudly, which is the regression this row +/// exists to keep out. +/// +/// The workaround is asserted too, so the decline is a redirection rather than a +/// dead end: capturing the profile into a variable of its own gives the expander +/// the direct reference it needs, and the allocation then matches the unfrozen +/// model exactly (`pp` being constant is what makes that the RIGHT answer to +/// compare against). +#[test] +fn a_snapshot_priority_profile_declines_rather_than_allocating_over_one_column() { + let fixture = |name: &str| { + TestProject::new(name) + .with_sim_time(0.0, 2.0, 1.0) + .indexed_dimension("d", 3) + .indexed_dimension("xp", 4) + .array_with_ranges("request[d]", vec![("1", "10"), ("2", "20"), ("3", "30")]) + .array_with_ranges( + "pp[d,xp]", + vec![ + ("1,1", "1"), + ("1,2", "3"), + ("1,3", "1"), + ("1,4", "0"), + ("2,1", "1"), + ("2,2", "1"), + ("2,3", "1"), + ("2,4", "0"), + ("3,1", "1"), + ("3,2", "2"), + ("3,3", "1"), + ("3,4", "0"), + ], + ) + .scalar_const("supply", 35.0) + }; + + assert_declines_because( + fixture("pp_prev_reject").array_aux( + "out[d]", + "allocate_available(request[d], PREVIOUS(pp[d,1]), supply)", + ), + "out", + "would allocate over one column", + ); + + let series = |project: TestProject| -> Vec> { + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + (1..=3) + .map(|k| all.get(&format!("out[{k}]")).unwrap().clone()) + .collect() + }; + let unfrozen = series( + fixture("pp_unfrozen") + .array_aux("out[d]", "allocate_available(request[d], pp[d,1], supply)"), + ); + let captured = series( + fixture("pp_captured") + .array_aux("frozen[d,xp]", "PREVIOUS(pp[d,xp])") + .array_aux( + "out[d]", + "allocate_available(request[d], frozen[d,1], supply)", + ), + ); + // Step 0 differs: the capture reads the PREVIOUS fallback (an all-zero + // profile), which is a legitimate allocation over a degenerate profile + // rather than a shape defect. From step 1 on the frozen profile IS `pp`, + // so the two models must agree element for element. + for (k, (a, b)) in captured.iter().zip(unfrozen.iter()).enumerate() { + assert_close( + &a[1..], + &b[1..], + &format!( + "the per-element capture workaround must allocate exactly as the \ + unfrozen model does once the snapshot exists (element {})", + k + 1 + ), + ); + } +} + +// =========================================================================== +// Shape axis: every rejected `Expr` variant that can carry an array, at one +// position (VECTOR SORT ORDER arg0). +// =========================================================================== + +#[test] +fn computed_operand_shapes() { + // Op2 -- also covered by `vector_sort_order_positions`, restated here so + // the shape enumeration is complete in one place. + assert_close( + &out_of("shape_op2", "VECTOR SORT ORDER(vals[d] + bump[d], 1)"), + &[2.0, 0.0, 1.0], + "shape Op2", + ); + + // Op1. Unary minus lowers to `Op2(Sub, 0, x)` and `Transpose` is folded + // during lowering, so `NOT` is the only `Expr::Op1` a fragment can carry. + // `NOT (vals[d] > 15)` = [0, 1, 0]; ascending with stable ties that is + // 0@0, 0@2, 1@1 -> [0, 2, 1]. Reading `vals` raw would give [1, 2, 0]. + assert_close( + &out_of("shape_op1", "VECTOR SORT ORDER(NOT (vals[d] > 15), 1)"), + &[0.0, 2.0, 1.0], + "shape Op1 (NOT)", + ); + + // If, selecting between two arrays. `IF sel[d] > 0 THEN bump[d] ELSE + // vals[d]` = [0, 100, 20]; ascending that is 0@0, 20@2, 100@1 -> + // [0, 2, 1]. Reading `vals` alone would give [1, 2, 0]. + assert_close( + &out_of( + "shape_if", + "VECTOR SORT ORDER(IF sel[d] > 0 THEN bump[d] ELSE vals[d], 1)", + ), + &[0.0, 2.0, 1.0], + "shape If", + ); + + // App, elementwise: `ABS(vals[d] - 25)` = [5, 15, 5]; ascending with + // stable ties that is 5@0, 5@2, 15@1 -> [0, 2, 1]. Reading `vals` raw + // would give [1, 2, 0]. + assert_close( + &out_of("shape_app", "VECTOR SORT ORDER(ABS(vals[d] - 25), 1)"), + &[0.0, 2.0, 1.0], + "shape App (elementwise builtin)", + ); + + // App, a nested array-producing builtin. The inner ELM MAP yields + // [vals[2], vals[0], vals[1]] = [20, 30, 10]; ascending that is + // 10@2, 20@0, 30@1 -> [2, 0, 1]. + assert_close( + &out_of( + "shape_nested", + "VECTOR SORT ORDER(VECTOR ELM MAP(vals[d], offs[d]), 1)", + ), + &[2.0, 0.0, 1.0], + "shape App (nested array-producing builtin)", + ); +} + +/// The elementwise scalar builtins `find_expr_array_view` recognises: an +/// operand whose outermost node is one of them takes the shape of whichever +/// argument has one, so it materializes like any other computed array. Two +/// representatives of the two families it grew for this work -- a +/// single-argument one (`SIGN`) and a multi-argument one (two-argument `MAX`). +/// +/// `SIGN(vals[d] - 15)` = [1, -1, 1]; ascending with stable ties that is +/// -1@1, 1@0, 1@2 -> [1, 0, 2]. Reading `vals` raw would give [1, 2, 0]. +/// `MAX(bump[d], vals[d])` = [30, 100, 20]; ascending -> [2, 0, 1]. Reading +/// `vals` raw gives [1, 2, 0], reading `bump` raw gives [0, 2, 1]. +#[test] +fn elementwise_builtin_operands_materialize() { + assert_close( + &out_of("elemwise_sign", "VECTOR SORT ORDER(SIGN(vals[d] - 15), 1)"), + &[1.0, 0.0, 2.0], + "SIGN operand (single-argument elementwise)", + ); + assert_close( + &out_of( + "elemwise_max2", + "VECTOR SORT ORDER(MAX(bump[d], vals[d]), 1)", + ), + &[2.0, 0.0, 1.0], + "two-argument MAX operand (multi-argument elementwise)", + ); +} + +/// Materializing a `VECTOR ELM MAP` **source** changes which storage the +/// mapping ranges over, and this pins the choice rather than leaving it +/// accidental. It pins it against OURSELVES: the variable-source half is +/// documented and ground-truthed, the computed-source half is not. +/// +/// **Documented.** The Vensim reference page for `VECTOR ELM MAP` (retrieved +/// 2026-08-02) says the function "returns the value of the variable that is +/// offset from vec by the specified amount", and that an offset "outside the +/// range of the variable" yields `:NA:`. Real Vensim output agrees: in +/// `test/sdeverywhere/models/vector/`, +/// `f[DimA,DimB] = VECTOR ELM MAP(d[DimA,B1], a[DimA])` prints `1,1,5,5,6,6`, +/// and `f[A2,B1] = 5 = d[A2,B2]` -- the mapping read past its own `B1` slice +/// into the next row of `d`'s storage. `vm_vector_elm_map.rs` implements that +/// with its `source_is_full_array` test: a strict slice such as `matrix[1,*]` +/// keeps a per-element base and CAN read across rows. +/// +/// **A DEFINED EXTENSION, not a match.** Vensim rejects a computed source +/// outright: run in Vensim DSS on 2026-08-04, +/// `vensim-probes/elm_map_computed_source.mdl` refuses to simulate with +/// "Argument 1 to function VECTOR ELM MAP must be a normal variable". There is +/// therefore no Vensim behaviour for these numbers to agree or disagree with, +/// and the shape is one Simlin accepts and Vensim does not. +/// +/// What it MEANS is defined by helper-equivalence: an inline expression behaves +/// exactly as the same values pre-assigned to a named variable -- the spelling +/// that IS legal Vensim. A materialized operand is a fresh contiguous temp, so +/// it is full-array by construction and the mapping is confined to the computed +/// array, which is exactly `VECTOR ELM MAP(helper[A1], offs)` for a `helper` +/// holding those values. The rows below pin that definition. +/// +/// `matrix` is [[1,2,3],[10,20,30]] (flat storage of 6) and `far` is [3,4,5]. +/// Over the row-1 slice those offsets run off the end of row 1 and into row 2; +/// over a 3-element temp they are all out of range and yield `:NA:`. +#[test] +fn materializing_an_elm_map_source_confines_the_mapping_to_the_temp() { + let far = || { + fixture("elm_base").array_with_ranges("far[d]", vec![("1", "3"), ("2", "4"), ("3", "5")]) + }; + + // The direct slice keeps genuine Vensim's full-variable rule. Recorded as + // the contrast, and as a tripwire if that rule ever moves. + let slice = far().array_aux("out[d]", "VECTOR ELM MAP(matrix[1,*], far[d])"); + slice.assert_compiles_incremental(); + let slice = slice.vm_result_incremental("out"); + assert_eq!(slice.len(), 3); + assert!( + (slice[0] - 10.0).abs() < 1e-9 && (slice[1] - 30.0).abs() < 1e-9 && slice[2].is_nan(), + "a direct strict-slice source maps over the whole variable, got {slice:?}" + ); + + // The computed source is a temp of its own, so every offset is out of its + // range. + let computed = far().array_aux("out[d]", "VECTOR ELM MAP(matrix[1,*] * 1, far[d])"); + computed.assert_compiles_incremental(); + let computed = computed.vm_result_incremental("out"); + assert!( + computed.len() == 3 && computed.iter().all(|v| v.is_nan()), + "a materialized source confines the mapping to the computed array, got {computed:?}" + ); +} + +/// C1: `Pass1Context`'s `Rank` arm decomposes its array argument like all five +/// of its siblings. +/// +/// This is not observable from VM values -- the post-lowering pass materializes +/// the same operand either way, and the numbers agree -- so it is pinned at the +/// lowered-fragment level, through the production `Var::new` lowering that +/// `build_module` drives. What differs is WHERE the temp is allocated: Pass 1 +/// numbers the operand's temp before the apply-to-all hoister numbers the +/// builtin's result, while the post-lowering pass continues past the highest id +/// the fragment already uses, so the two temps come out in the opposite order. +/// +/// Stating it as "RANK's fragment has the same temp structure as VECTOR SORT +/// ORDER's" is the actual C1 claim (arm consistency) and reds if the arm +/// reverts to `transform_inner`. +#[test] +fn the_rank_arm_decomposes_its_array_argument_like_its_siblings() { + fn assign_temp_ids(name: &str, eqn: &str) -> Vec { + use crate::compiler::expr::Expr; + fixture(name) + .array_aux("out[d]", eqn) + .build_module() + .unwrap_or_else(|e| panic!("{name} should build: {e}")) + .runlist_flows + .iter() + .filter_map(|e| match e { + Expr::AssignTemp(id, _, _) => Some(*id), + _ => None, + }) + .collect() + } + + let vso = assign_temp_ids("c1_vso", "VECTOR SORT ORDER(vals[*] + bump[*], 1)"); + let rank = assign_temp_ids("c1_rank", "RANK(vals[*] + bump[*], 1)"); + assert_eq!( + vso, + vec![0, 1], + "the sibling arm decomposes in Pass 1: operand temp 0, then the \ + builtin's own result temp 1" + ); + assert_eq!( + rank, vso, + "RANK must decompose its array argument in Pass 1 like VECTOR SORT \ + ORDER does; a fragment numbered the other way means the arm fell \ + through to the post-lowering pass instead" + ); +} + +// =========================================================================== +// Spelling axis: `vals[d]` (needs the ActiveDimRef -> Wildcard promotion) and +// `vals[*]`, at the two arms the issue reports separately. +// =========================================================================== + +#[test] +fn both_apply_to_all_spellings_materialize() { + // `VECTOR SORT ORDER`'s star spelling reaches Pass 1 with no unresolved + // dimension reference, so it already compiled; the active-dimension + // spelling did not. Both must now agree. + assert_close( + &out_of("spell_vso_star", "VECTOR SORT ORDER(vals[*] + bump[*], 1)"), + &[2.0, 0.0, 1.0], + "VECTOR SORT ORDER, star spelling", + ); + assert_close( + &out_of("spell_vso_dim", "VECTOR SORT ORDER(vals[d] + bump[d], 1)"), + &[2.0, 0.0, 1.0], + "VECTOR SORT ORDER, active-dimension spelling", + ); + + // `RANK` is the arm whose Pass-1 recursion never called + // `maybe_decompose_array_arg_inner`, unlike all five of its siblings, so + // BOTH spellings failed. Ranks of [30, 110, 20] ascending: 2, 3, 1. + assert_close( + &out_of("spell_rank_star", "RANK(vals[*] + bump[*], 1)"), + &[2.0, 3.0, 1.0], + "RANK, star spelling", + ); + assert_close( + &out_of("spell_rank_dim", "RANK(vals[d] + bump[d], 1)"), + &[2.0, 3.0, 1.0], + "RANK, active-dimension spelling", + ); +} + +// =========================================================================== +// Phase C3: `PREVIOUS`/`INIT` of an arrayed reference (GH #995, option D). +// +// An array-valued `PREVIOUS`/`INIT` is a VIEW over one of the VM's snapshot +// buffers -- the same `prev_values` / `initial_values` the scalar `LoadPrev` / +// `LoadInitial` read, addressed with the argument's own geometry. So it is a +// view position like any other, not a computed array that has to be +// materialized first. +// +// These rows were the red half of Phase C1+C2's decline. Each now asserts VM +// NUMBERS over a TIME-VARYING fixture, because a constant fixture cannot tell a +// previous value from a current one. +// =========================================================================== + +/// The time-varying fixture. Everything the C3 rows read moves, so reading +/// `curr` where `prev` was meant is a different answer at every step but the +/// first. +/// +/// Three saved steps (t = 0, 1, 2): +/// +/// | variable | t=0 | t=1 | t=2 | +/// |------------|----------------|----------------|-----------------| +/// | `vals[d]` | `[30, 10, 20]` | `[5, 20, 20]` | `[-20, 30, 20]` | +/// | `offs[d]` | `[2, 0, 1]` | `[1, 0, 1]` | `[0, 0, 1]` | +/// | `sel[d]` | `[1, 1, 0]` | `[0, 1, 0]` | `[0, 1, 0]` | +/// | `matrix[1,*]` | `[1, 2, 3]` | `[2, 2, 3]` | `[3, 2, 3]` | +/// | `matrix[2,*]` | `[10,20,30]`| `[10, 20, 40]` | `[10, 20, 50]` | +/// +/// `fixed[d] = [30, 10, 20]` is deliberately CONSTANT: it is the second operand +/// of the nested rows and the source of the `+ SUM(VECTOR SORT ORDER(fixed[*], +/// 1))` tail, which must contribute the same 3 at every step so the tail does +/// not smear the value being asserted. (That tail is what forces the lowering +/// path the `VECTOR SELECT` rows are about -- see the module docs.) +fn moving_fixture(name: &str) -> TestProject { + TestProject::new(name) + .with_sim_time(0.0, 2.0, 1.0) + .indexed_dimension("d", 3) + .indexed_dimension("e", 2) + .array_with_ranges( + "vals[d]", + vec![ + ("1", "30 - 25 * TIME"), + ("2", "10 + 10 * TIME"), + ("3", "20"), + ], + ) + .array_with_ranges("offs[d]", vec![("1", "2 - TIME"), ("2", "0"), ("3", "1")]) + .array_with_ranges( + "sel[d]", + vec![("1", "IF TIME > 0.5 THEN 0 ELSE 1"), ("2", "1"), ("3", "0")], + ) + .array_with_ranges("fixed[d]", vec![("1", "30"), ("2", "10"), ("3", "20")]) + .array_with_ranges( + "matrix[e,d]", + vec![ + ("1,1", "1 + TIME"), + ("1,2", "2"), + ("1,3", "3"), + ("2,1", "10"), + ("2,2", "20"), + ("2,3", "30 + 10 * TIME"), + ], + ) + // A NAMED row dimension, so the qualified `row·r1` spelling exists, and + // rows two orders of magnitude apart so reading the WRONG row cannot be + // mistaken for reading the wrong step. Row sums: r1 = 6, 7, 8; + // r2 = 600, 610, 620. + .named_dimension("row", &["r1", "r2"]) + .array_with_ranges( + "wide[row,d]", + vec![ + ("r1,1", "1 + TIME"), + ("r1,2", "2"), + ("r1,3", "3"), + ("r2,1", "100"), + ("r2,2", "200"), + ("r2,3", "300 + 10 * TIME"), + ], + ) +} + +/// Run ` = ` against the moving fixture and return each element's +/// series, element-major: `series[k]` is `out[k+1]` over t = 0, 1, 2. +fn moving_series(name: &str, lhs: &str, eqn: &str, n_elements: usize) -> Vec> { + let project = moving_fixture(name).array_aux(lhs, eqn); + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + (1..=n_elements) + .map(|k| { + all.get(&format!("out[{k}]")) + .unwrap_or_else(|| panic!("out[{k}] missing from {:?}", all.keys())) + .clone() + }) + .collect() +} + +fn assert_series(actual: &[Vec], expected: &[[f64; 3]], what: &str) { + assert_eq!( + actual.len(), + expected.len(), + "{what}: element-count mismatch, got {actual:?}" + ); + for (k, (a, e)) in actual.iter().zip(expected.iter()).enumerate() { + assert_close(a, e, &format!("{what}: out[{}] over time", k + 1)); + } +} + +/// The position axis, bare operand: every `walk_expr_as_view` call site from +/// the module-doc table, with `PREVIOUS()` in it. +/// +/// The whole point of the fixture moving is that each row's expected series +/// distinguishes four readings: the correct previous array, the CURRENT array, +/// one element's previous value broadcast, and the all-zero stub a failed +/// fragment leaves behind. Each row says which. +#[test] +fn previous_operands_are_views_over_the_prev_snapshot() { + // VECTOR SORT ORDER arg0. prev(vals) is [0,0,0], [30,10,20], [5,20,20]; + // ascending sort orders are [0,1,2], [1,2,0], [0,1,2]. Reading `vals` + // CURRENT would give [1,2,0], [0,1,2], [0,2,1]; broadcasting element 0's + // previous value gives [0,1,2] at every step, which differs at t=1. + assert_series( + &moving_series( + "c3_vso", + "out[d]", + "VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)", + 3, + ), + &[[0.0, 1.0, 0.0], [1.0, 2.0, 1.0], [2.0, 0.0, 2.0]], + "VECTOR SORT ORDER arg0", + ); + + // RANK arg0, 1-based. [0,0,0] ties to [1,2,3] under the stable sort; + // [30,10,20] ranks [3,1,2]; [5,20,20] ranks [1,2,3]. Reading CURRENT would + // give [3,1,2], [1,2,3], [1,3,2]. + assert_series( + &moving_series("c3_rank", "out[d]", "RANK(PREVIOUS(vals[d]), 1)", 3), + &[[1.0, 3.0, 1.0], [2.0, 1.0, 2.0], [3.0, 2.0, 3.0]], + "RANK arg0", + ); + + // VECTOR ELM MAP arg0 (source). The prev view spans the whole variable, so + // it is a full-array source and `result[i] = prev_vals[offs[i]]` with the + // CURRENT offsets: [0,0,0] mapped by [2,0,1]; [30,10,20] by [1,0,1] -> + // [10,30,10]; [5,20,20] by [0,0,1] -> [5,5,20]. This row is also the one + // that pins `full_source_len` looking THROUGH the `PREVIOUS`: bounding the + // source at 1 element instead of 3 turns every mapped offset but 0 into + // `:NA:` (measured: `[NaN, NaN, 5]` for out[1]). + assert_series( + &moving_series( + "c3_elm_src", + "out[d]", + "VECTOR ELM MAP(PREVIOUS(vals[d]), offs[d])", + 3, + ), + &[[0.0, 10.0, 5.0], [0.0, 30.0, 5.0], [0.0, 10.0, 20.0]], + "VECTOR ELM MAP arg0 (source)", + ); + + // VECTOR ELM MAP arg1 (offsets): `result[i] = vals[prev_offs[i]]` over the + // CURRENT vals. prev_offs [0,0,0] over [30,10,20] -> [30,30,30]; + // [2,0,1] over [5,20,20] -> [20,5,20]; [1,0,1] over [-20,30,20] -> + // [30,-20,30]. Reading `offs` current would give [20,30,10], [20,5,20], + // [-20,-20,30]. + assert_series( + &moving_series( + "c3_elm_off", + "out[d]", + "VECTOR ELM MAP(vals[d], PREVIOUS(offs[d]))", + 3, + ), + &[[30.0, 20.0, 30.0], [30.0, 5.0, -20.0], [30.0, 20.0, 30.0]], + "VECTOR ELM MAP arg1 (offsets)", + ); + + // VECTOR SELECT reduces to a scalar, so every element of `out` holds the + // same value; the constant tail adds 3. + // + // arg0 (selection): prev(sel) [0,0,0] selects nothing -> the max_value + // argument 0; [1,1,0] selects vals[0]+vals[1] = 5+20 = 25; [0,1,0] selects + // vals[1] = 30. Reading `sel` current would give 43, 23, 33. + assert_series( + &moving_series( + "c3_sel_sel", + "out[d]", + "VECTOR SELECT(PREVIOUS(sel[d]), vals[d], 0, 0, 0) \ + + SUM(VECTOR SORT ORDER(fixed[*], 1))", + 3, + ), + &[[3.0, 28.0, 33.0], [3.0, 28.0, 33.0], [3.0, 28.0, 33.0]], + "VECTOR SELECT arg0 (selection array)", + ); + + // arg1 (values): the CURRENT sel over prev(vals). [1,1,0] over [0,0,0] -> 0; + // [0,1,0] over [30,10,20] -> 10; [0,1,0] over [5,20,20] -> 20. Reading + // `vals` current would give 43, 23, 33. + assert_series( + &moving_series( + "c3_sel_val", + "out[d]", + "VECTOR SELECT(sel[d], PREVIOUS(vals[d]), 0, 0, 0) \ + + SUM(VECTOR SORT ORDER(fixed[*], 1))", + 3, + ), + &[[3.0, 13.0, 23.0], [3.0, 13.0, 23.0], [3.0, 13.0, 23.0]], + "VECTOR SELECT arg1 (value array)", + ); +} + +/// The six `emit_array_reduce` arms plus `MEAN`, over the row slice +/// `PREVIOUS(matrix[e,*])`. +/// +/// A reducer's argument is lowered with `with_preserved_wildcards`, which does +/// NOT promote an active-dimension reference -- so `matrix[e,*]` stays a ROW +/// slice and the prev view is that row of the snapshot, not the whole matrix. +/// That is the ELM MAP coherence rule stated positively: a prev view of a strict +/// slice behaves exactly like the curr view of the same slice. +/// +/// prev rows are `[0,0,0]`/`[0,0,0]`, then `[1,2,3]`/`[10,20,30]`, then +/// `[2,2,3]`/`[10,20,40]`. Reading the CURRENT rows would shift each series one +/// step earlier, which every row below distinguishes except `SIZE` -- whose +/// point is the count, and whose wrong answer (a collapsed 1-element view) is 1. +#[test] +fn previous_reducer_operands_read_the_previous_row() { + assert_series( + &moving_series("c3_sum", "out[e]", "SUM(PREVIOUS(matrix[e,*]))", 2), + &[[0.0, 6.0, 7.0], [0.0, 60.0, 70.0]], + "SUM", + ); + assert_series( + &moving_series("c3_max", "out[e]", "MAX(PREVIOUS(matrix[e,*]))", 2), + &[[0.0, 3.0, 3.0], [0.0, 30.0, 40.0]], + "MAX (1-arg)", + ); + assert_series( + &moving_series("c3_min", "out[e]", "MIN(PREVIOUS(matrix[e,*]))", 2), + &[[0.0, 1.0, 2.0], [0.0, 10.0, 10.0]], + "MIN (1-arg)", + ); + // SIZE counts elements of the prev view: 3 always. A collapsed operand + // would give 1, which is the failure this row rules out. + assert_series( + &moving_series("c3_size", "out[e]", "SIZE(PREVIOUS(matrix[e,*]))", 2), + &[[3.0, 3.0, 3.0], [3.0, 3.0, 3.0]], + "SIZE", + ); + // MEAN's single-argument form is an array reduction, and its codegen arm + // enumerates the view shapes rather than pushing a view unconditionally -- + // so the snapshot view had to be added there too, or an array-valued + // PREVIOUS would have fallen through to the scalar walk and failed to + // compile (measured before the arm was extended). + assert_series( + &moving_series("c3_mean", "out[e]", "MEAN(PREVIOUS(matrix[e,*]))", 2), + &[[0.0, 2.0, 7.0 / 3.0], [0.0, 20.0, 70.0 / 3.0]], + "MEAN (1-arg)", + ); + // STDDEV is the POPULATION deviation (`ArrayStddev` divides by n). + let pop_stddev = |xs: [f64; 3]| -> f64 { + let mean = (xs[0] + xs[1] + xs[2]) / 3.0; + (xs.iter().map(|x| (x - mean).powi(2)).sum::() / 3.0).sqrt() + }; + assert_series( + &moving_series("c3_stddev", "out[e]", "STDDEV(PREVIOUS(matrix[e,*]))", 2), + &[ + [ + 0.0, + pop_stddev([1.0, 2.0, 3.0]), + pop_stddev([2.0, 2.0, 3.0]), + ], + [ + 0.0, + pop_stddev([10.0, 20.0, 30.0]), + pop_stddev([10.0, 20.0, 40.0]), + ], + ], + "STDDEV", + ); +} + +/// A prev view of a strict row slice must read THAT ROW of the snapshot. +/// +/// The reducer rows above pin the LAG (they would catch reading `curr`) but are +/// weak on the ROW, because their two matrix rows are only an order of magnitude +/// apart and both are read by the same apply-to-all iteration. `wide`'s rows are +/// two orders of magnitude apart, so the four readings are unmistakable: +/// previous r1 is `[0, 6, 7]` and previous r2 is `[0, 600, 610]`, against the +/// `curr` controls `[6, 7, 8]` and `[600, 610, 620]`. Reading the wrong row +/// lands on the other row's series; reading the wrong step lands on its own +/// control. +/// +/// SPELLING, disclosed rather than assumed. This uses the ACTIVE-DIMENSION +/// spelling (`wide[row,*]` under `out[row]`), which resolves per element. The +/// QUALIFIED spelling the LTM wrap generates -- +/// `PREVIOUS(matrix[region·nyc,*])` -- pins the row by NAME instead, and it +/// reaches this SAME view route from an ordinary APPLY-TO-ALL user equation: +/// `arg_is_array_shaped` accepts it (the qualified index is static, the `*` +/// spans), so no capture helper is synthesized and the argument passes through +/// to lowering. Measured: `out[row] = SUM(PREVIOUS(wide[row·r1,*]))` compiles +/// with ZERO synthesized helpers and reads r1's previous row at every step, +/// while with `arg_is_array_shaped` reverted (HEAD's visitor) the same equation +/// declines through a capture helper. So the qualified spelling is neither +/// LTM-only nor pre-existing -- C3 is what routes it here. +/// +/// What IS split is `Ast::Scalar` vs `Ast::ApplyToAll` inside +/// `builtins_visitor::instantiate_implicit_modules`, not user-vs-LTM: both +/// `variable.rs` and `db::ltm::parse` pass `Some(dimensions)`. In a SCALAR +/// equation the qualified index is not accepted as static and +/// `SUM(PREVIOUS(wide[row·r1,*]))` still declines through a helper; that half is +/// unchanged by C3 and is a front-end residual, not a view question. +/// +/// The row property asserted below is the VIEW ARITHMETIC's, which both +/// spellings share, and the loop pins BOTH: the active-dimension rows read +/// each iteration's own row, while the qualified rows pin one row by NAME for +/// every element. The LTM-side witness that the qualified spelling compiles +/// there too is `db::ltm_char_tests::char_agg_nested_reducer`, whose partial +/// embeds `previous(matrix[region·boston,*])` -- but that fixture cannot +/// discriminate a ROW (every element of its `matrix` and `other` is the +/// constant 1), which is why the numeric row property lives here. +#[test] +fn a_prev_view_of_a_row_slice_reads_that_row_of_the_snapshot() { + for (name, eqn, expected) in [ + ( + "c3_row_prev", + "SUM(PREVIOUS(wide[row,*]))", + [[0.0, 6.0, 7.0], [0.0, 600.0, 610.0]], + ), + // The `curr` controls, which are what a lost lag would return. + ( + "c3_row_now", + "SUM(wide[row,*])", + [[6.0, 7.0, 8.0], [600.0, 610.0, 620.0]], + ), + // The QUALIFIED spelling: one row pinned by name, read for EVERY + // element of the iteration. Reading the wrong row lands on r2's + // unmistakable series; losing the lag lands on the curr control above. + ( + "c3_row_prev_qual", + "SUM(PREVIOUS(wide[row\u{B7}r1,*]))", + [[0.0, 6.0, 7.0], [0.0, 6.0, 7.0]], + ), + ( + "c3_row_prev_qual2", + "SUM(PREVIOUS(wide[row\u{B7}r2,*]))", + [[0.0, 600.0, 610.0], [0.0, 600.0, 610.0]], + ), + ] { + let project = moving_fixture(name).array_aux("out[row]", eqn); + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + for (elem, want) in ["r1", "r2"].into_iter().zip(expected.iter()) { + assert_close( + all.get(&format!("out[{elem}]")) + .unwrap_or_else(|| panic!("out[{elem}] missing")), + want, + &format!("{eqn} at row {elem}"), + ); + } + } +} + +/// The `INIT` twins. `initial_values` is the post-initials snapshot, so an +/// `INIT` view is the t=0 array at EVERY step -- including t=0 itself, where +/// `PREVIOUS` reads its fallback instead. +/// +/// `INIT(vals)` is `[30, 10, 20]` throughout, so the sort order is `[1, 2, 0]` +/// at every step. That is distinct from the `PREVIOUS` series above at t=0 and +/// t=2, and from reading `vals` current at t=1 and t=2. +#[test] +fn init_operands_are_views_over_the_initial_snapshot() { + assert_series( + &moving_series( + "c3_init_vso", + "out[d]", + "VECTOR SORT ORDER(INIT(vals[d]), 1)", + 3, + ), + &[[1.0, 1.0, 1.0], [2.0, 2.0, 2.0], [0.0, 0.0, 0.0]], + "VECTOR SORT ORDER arg0, INIT", + ); + // A reducer over an INIT row slice: `matrix` row sums at t=0 are 6 and 60, + // held for the whole run. The PREVIOUS twin above reads 0, then 6/60, then + // 7/70. + assert_series( + &moving_series("c3_init_sum", "out[e]", "SUM(INIT(matrix[e,*]))", 2), + &[[6.0, 6.0, 6.0], [60.0, 60.0, 60.0]], + "SUM over an INIT row slice", + ); + // An INIT view in the initials phase reads `curr` rather than the snapshot + // (the snapshot does not exist yet), exactly as `Opcode::LoadInitial` does. + // An arrayed stock whose INITIAL equation reduces an INIT view is the shape + // that exercises it: `SUM(INIT(matrix[e,*]))` at t=0 is 6 and 60, and the + // stock never changes, so a broken initials branch (reading an all-zero + // snapshot) would leave it at 0. + let init_phase = moving_fixture("c3_init_phase").array_stock( + "lvl[e]", + "SUM(INIT(matrix[e,*]))", + &[], + &[], + None, + ); + init_phase.assert_compiles_incremental(); + let all = init_phase.run_vm_incremental(); + assert_close(all.get("lvl[1]").unwrap(), &[6.0, 6.0, 6.0], "lvl[1]"); + assert_close(all.get("lvl[2]").unwrap(), &[60.0, 60.0, 60.0], "lvl[2]"); +} + +/// The nested rows Phase C1+C2 declined: an array-valued `PREVIOUS`/`INIT` +/// under arithmetic that is itself the operand. +/// +/// C1+C2 refused these because the argument was lowered element-collapsed, so +/// materializing would have produced ONE element's previous value broadcast +/// across the array -- measured `[0, 2, 1]` where the answer was `[2, 0, 1]`. +/// The argument now keeps its array shape, `find_expr_array_view` gives the call +/// its argument's shape, and the operand materializes like any other computed +/// array: the `BeginIter` body reads the snapshot view per element. +/// +/// `fixed = [30, 10, 20]`, so `prev(vals) + fixed` is `[30,10,20]`, `[60,20,40]`, +/// `[35,30,40]` and the ascending orders are `[1,2,0]`, `[1,2,0]`, `[1,0,2]`. +/// Reading `vals` CURRENT would give `[1,2,0]`, `[0,1,2]`, `[0,1,2]`; a +/// broadcast of element 0's previous value gives `[1,2,0]` at every step. +#[test] +fn nested_previous_and_init_operands_materialize() { + assert_series( + &moving_series( + "c3n_vso", + "out[d]", + "VECTOR SORT ORDER(PREVIOUS(vals[d]) + fixed[d], 1)", + 3, + ), + &[[1.0, 1.0, 1.0], [2.0, 2.0, 0.0], [0.0, 0.0, 2.0]], + "VECTOR SORT ORDER arg0, nested PREVIOUS", + ); + // Operand order must not matter: `find_expr_array_view` on an `Op2` takes + // `lhs.or_else(rhs)`, and both sides now carry the same shape. + assert_series( + &moving_series( + "c3n_vso_rhs", + "out[d]", + "VECTOR SORT ORDER(fixed[d] + PREVIOUS(vals[d]), 1)", + 3, + ), + &[[1.0, 1.0, 1.0], [2.0, 2.0, 0.0], [0.0, 0.0, 2.0]], + "VECTOR SORT ORDER arg0, nested PREVIOUS on the right", + ); + // INIT nested: `[30,10,20] + [30,10,20] = [60,20,40]` at every step, so the + // order is `[1,2,0]` throughout -- which the PREVIOUS row above differs from + // at t=2. + assert_series( + &moving_series( + "c3n_vso_init", + "out[d]", + "VECTOR SORT ORDER(INIT(vals[d]) + fixed[d], 1)", + 3, + ), + &[[1.0, 1.0, 1.0], [2.0, 2.0, 2.0], [0.0, 0.0, 0.0]], + "VECTOR SORT ORDER arg0, nested INIT", + ); + // RANK, the sibling arm Phase C1 fixed: ranks of the same three arrays are + // [3,1,2], [3,1,2], [2,1,3]. + assert_series( + &moving_series( + "c3n_rank", + "out[d]", + "RANK(PREVIOUS(vals[d]) + fixed[d], 1)", + 3, + ), + &[[3.0, 3.0, 2.0], [1.0, 1.0, 1.0], [2.0, 2.0, 3.0]], + "RANK arg0, nested PREVIOUS", + ); + // VECTOR ELM MAP, both positions. Source: `prev(vals) + fixed` mapped by the + // current `offs` -- and the materialized source is a fresh contiguous temp, + // so the mapping is confined to it (`materializing_an_elm_map_source_...`). + // [30,10,20] by [2,0,1] -> [20,30,10]; [60,20,40] by [1,0,1] -> [20,60,20]; + // [35,30,40] by [0,0,1] -> [35,35,30]. + assert_series( + &moving_series( + "c3n_elm_src", + "out[d]", + "VECTOR ELM MAP(PREVIOUS(vals[d]) + fixed[d], offs[d])", + 3, + ), + &[[20.0, 20.0, 35.0], [30.0, 60.0, 35.0], [10.0, 20.0, 30.0]], + "VECTOR ELM MAP arg0, nested PREVIOUS", + ); + // Offsets: `prev(offs) + 0` -- `fixed` would swamp the index range, so the + // nested arithmetic here is `PREVIOUS(offs[d]) * 1`, an `Op2` all the same. + // Same values as the bare arg1 row. + assert_series( + &moving_series( + "c3n_elm_off", + "out[d]", + "VECTOR ELM MAP(vals[d], PREVIOUS(offs[d]) * 1)", + 3, + ), + &[[30.0, 20.0, 30.0], [30.0, 5.0, -20.0], [30.0, 20.0, 30.0]], + "VECTOR ELM MAP arg1, nested PREVIOUS", + ); + // VECTOR SELECT, both positions. Selection: `prev(sel) * 1` is the same + // array, so the values match the bare arg0 row. + assert_series( + &moving_series( + "c3n_sel_sel", + "out[d]", + "VECTOR SELECT(PREVIOUS(sel[d]) * 1, vals[d], 0, 0, 0) \ + + SUM(VECTOR SORT ORDER(fixed[*], 1))", + 3, + ), + &[[3.0, 28.0, 33.0], [3.0, 28.0, 33.0], [3.0, 28.0, 33.0]], + "VECTOR SELECT arg0, nested PREVIOUS", + ); + // Values: current `sel` over `prev(vals) + fixed`. [1,1,0] over [30,10,20] + // -> 40; [0,1,0] over [60,20,40] -> 20; [0,1,0] over [35,30,40] -> 30. + assert_series( + &moving_series( + "c3n_sel_val", + "out[d]", + "VECTOR SELECT(sel[d], PREVIOUS(vals[d]) + fixed[d], 0, 0, 0) \ + + SUM(VECTOR SORT ORDER(fixed[*], 1))", + 3, + ), + &[[43.0, 23.0, 33.0], [43.0, 23.0, 33.0], [43.0, 23.0, 33.0]], + "VECTOR SELECT arg1, nested PREVIOUS", + ); + // The reducer positions, which C1+C2 could not even reach: the argument + // died in `builtins_visitor`'s capture-helper synthesis before the + // materializer saw it. It now passes through, so the operand materializes. + // `SUM(prev(matrix[e,*]) + matrix[e,*])` is the previous row's sum plus this + // row's. Row 1: 0+6, 6+7, 7+8 -> 6, 13, 15. Row 2: 0+60, 60+70, 70+80 -> + // 60, 130, 150. Reading `matrix` current on BOTH sides would double the + // current row (12, 14, 16 and 120, 140, 160). + assert_series( + &moving_series( + "c3n_sum", + "out[e]", + "SUM(PREVIOUS(matrix[e,*]) + matrix[e,*])", + 2, + ), + &[[6.0, 13.0, 15.0], [60.0, 130.0, 150.0]], + "SUM over a nested PREVIOUS", + ); + assert_series( + &moving_series( + "c3n_mean", + "out[e]", + "MEAN(PREVIOUS(matrix[e,*]) + matrix[e,*])", + 2, + ), + &[[2.0, 13.0 / 3.0, 5.0], [20.0, 130.0 / 3.0, 50.0]], + "MEAN over a nested PREVIOUS", + ); +} + +/// `ALLOCATE AVAILABLE` / `ALLOCATE BY PRIORITY`, the two positions the +/// materializer does hoist. +/// +/// A bisection over allocation curves is not hand-computable the way a sort +/// order is, so -- exactly as [`allocate_positions`] does for the computed rows +/// -- each row is pinned against the model that captures the same array into a +/// variable of its own first. That reference is the PER-ELEMENT `LoadPrev` +/// route, which is the oracle this whole phase has to agree with, and the row +/// separately asserts it differs from the unfrozen model so "the previous values +/// were actually read" is asserted rather than assumed. +#[test] +fn allocate_previous_operands_agree_with_the_per_element_capture() { + struct Row { + what: &'static str, + inline: &'static str, + helper: (&'static str, &'static str), + reference: &'static str, + raw: &'static str, + } + let rows = [ + Row { + what: "allocate_available arg0 (requests)", + inline: "allocate_available(PREVIOUS(request[d]), pp[d,1], supply)", + helper: ("prev_req[d]", "PREVIOUS(request[d])"), + reference: "allocate_available(prev_req[d], pp[d,1], supply)", + raw: "allocate_available(request[d], pp[d,1], supply)", + }, + Row { + what: "allocate_by_priority arg0 (requests)", + inline: "allocate_by_priority(PREVIOUS(request[d]), priority[d], 0, width, supply)", + helper: ("prev_req[d]", "PREVIOUS(request[d])"), + reference: "allocate_by_priority(prev_req[d], priority[d], 0, width, supply)", + raw: "allocate_by_priority(request[d], priority[d], 0, width, supply)", + }, + Row { + what: "allocate_by_priority arg1 (priorities)", + inline: "allocate_by_priority(request[d], PREVIOUS(priority[d]), 0, width, supply)", + helper: ("prev_pri[d]", "PREVIOUS(priority[d])"), + reference: "allocate_by_priority(request[d], prev_pri[d], 0, width, supply)", + raw: "allocate_by_priority(request[d], priority[d], 0, width, supply)", + }, + ]; + + // A moving ALLOCATE fixture: both the requests and the priorities change + // every step, so freezing either one changes the allocation. + let fixture = |name: &str| { + TestProject::new(name) + .with_sim_time(0.0, 2.0, 1.0) + .indexed_dimension("d", 3) + .indexed_dimension("xp", 4) + .array_with_ranges( + "request[d]", + vec![("1", "10 + 10 * TIME"), ("2", "20"), ("3", "30 - 5 * TIME")], + ) + .array_with_ranges( + "priority[d]", + vec![("1", "3"), ("2", "1 + TIME"), ("3", "2")], + ) + .array_with_ranges( + "pp[d,xp]", + vec![ + ("1,1", "1"), + ("1,2", "3"), + ("1,3", "1"), + ("1,4", "0"), + ("2,1", "1"), + ("2,2", "1"), + ("2,3", "1"), + ("2,4", "0"), + ("3,1", "1"), + ("3,2", "2"), + ("3,3", "1"), + ("3,4", "0"), + ], + ) + .scalar_const("supply", 35.0) + .scalar_const("width", 1.0) + }; + + let series = |project: TestProject| -> Vec> { + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + (1..=3) + .map(|k| all.get(&format!("out[{k}]")).unwrap().clone()) + .collect() + }; + + for (i, row) in rows.iter().enumerate() { + let inline = series(fixture(&format!("c3_alloc_i{i}")).array_aux("out[d]", row.inline)); + let reference = series( + fixture(&format!("c3_alloc_r{i}")) + .array_aux(row.helper.0, row.helper.1) + .array_aux("out[d]", row.reference), + ); + let raw = series(fixture(&format!("c3_alloc_w{i}")).array_aux("out[d]", row.raw)); + + for (k, (a, e)) in inline.iter().zip(reference.iter()).enumerate() { + assert_close( + a, + e, + &format!( + "{}: the inline array PREVIOUS must allocate exactly as the \ + per-element capture helper does (element {})", + row.what, + k + 1 + ), + ); + } + assert_ne!( + inline, raw, + "{}: the fixture must make freezing the operand change the answer, \ + otherwise this row proves nothing (frozen {inline:?}, raw {raw:?})", + row.what + ); + } +} + +/// The equivalence the whole design rests on, asserted directly: an array-valued +/// `PREVIOUS` reads, element for element and step for step, exactly what a +/// per-element `LoadPrev` with the same fallback reads. +/// +/// The reference model captures `PREVIOUS(vals[d])` into an ordinary arrayed aux +/// -- which compiles to one `LoadPrev` per element, the route that has always +/// worked -- and then feeds THAT array to the same builtin. The two must agree +/// at every step, including the first, where the view route reads no buffer at +/// all and the scalar route returns its fallback. +/// +/// Rows cover the two snapshot regions and both a whole-array and a strict-slice +/// argument, since those reach different parts of the view arithmetic. +#[test] +fn an_array_snapshot_view_agrees_with_the_per_element_capture() { + struct Row { + what: &'static str, + lhs: &'static str, + inline: &'static str, + helper: (&'static str, &'static str), + reference: &'static str, + n: usize, + } + let rows = [ + Row { + what: "PREVIOUS of a whole array, VECTOR SORT ORDER", + lhs: "out[d]", + inline: "VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)", + helper: ("cap[d]", "PREVIOUS(vals[d])"), + reference: "VECTOR SORT ORDER(cap[d], 1)", + n: 3, + }, + Row { + what: "INIT of a whole array, VECTOR SORT ORDER", + lhs: "out[d]", + inline: "VECTOR SORT ORDER(INIT(vals[d]), 1)", + helper: ("cap[d]", "INIT(vals[d])"), + reference: "VECTOR SORT ORDER(cap[d], 1)", + n: 3, + }, + Row { + what: "PREVIOUS of a row slice, SUM", + lhs: "out[e]", + inline: "SUM(PREVIOUS(matrix[e,*]))", + helper: ("cap[e,d]", "PREVIOUS(matrix[e,d])"), + reference: "SUM(cap[e,*])", + n: 2, + }, + Row { + what: "INIT of a row slice, SUM", + lhs: "out[e]", + inline: "SUM(INIT(matrix[e,*]))", + helper: ("cap[e,d]", "INIT(matrix[e,d])"), + reference: "SUM(cap[e,*])", + n: 2, + }, + ]; + + for (i, row) in rows.iter().enumerate() { + let inline = moving_series(&format!("c3_eq_i{i}"), row.lhs, row.inline, row.n); + let reference_project = moving_fixture(&format!("c3_eq_r{i}")) + .array_aux(row.helper.0, row.helper.1) + .array_aux(row.lhs, row.reference); + reference_project.assert_compiles_incremental(); + let all = reference_project.run_vm_incremental(); + let reference: Vec> = (1..=row.n) + .map(|k| all.get(&format!("out[{k}]")).unwrap().clone()) + .collect(); + for (k, (a, e)) in inline.iter().zip(reference.iter()).enumerate() { + assert_close(a, e, &format!("{}: element {}", row.what, k + 1)); + } + } +} + +/// FIRST-STEP SEMANTICS, stated as its own row rather than left implicit in the +/// series above. +/// +/// `Opcode::LoadPrev` returns its caller-supplied fallback while +/// `use_prev_fallback` is set -- i.e. until the first snapshot is taken at the +/// end of step 0 -- and unary `PREVIOUS(x)` desugars to `PREVIOUS(x, 0)`. The +/// view route reproduces that by reading the fallback 0 for every element +/// (`vm::ChunkRegions::backing`'s `None` arm, and the wasm backend's `select` on +/// the same flag), which is why an array-valued `PREVIOUS` may carry no other +/// fallback. +/// +/// `SUM(PREVIOUS(vals[d]))` at t=0 must therefore be 0, not `SUM(vals)` = 60 and +/// not a NaN from an unwritten buffer. `MIN` and `MAX` are included because they +/// would surface a stale or uninitialized buffer as an out-of-range extremum +/// rather than as a plausible zero. +#[test] +fn the_first_step_of_an_array_previous_is_the_scalar_fallback() { + assert_series( + &moving_series("c3_first_sum", "out[d]", "SUM(PREVIOUS(vals[*]))", 3), + &[[0.0, 60.0, 45.0], [0.0, 60.0, 45.0], [0.0, 60.0, 45.0]], + "SUM of a PREVIOUS view", + ); + assert_series( + &moving_series("c3_first_min", "out[d]", "MIN(PREVIOUS(vals[*]))", 3), + &[[0.0, 10.0, 5.0], [0.0, 10.0, 5.0], [0.0, 10.0, 5.0]], + "MIN of a PREVIOUS view", + ); + assert_series( + &moving_series("c3_first_max", "out[d]", "MAX(PREVIOUS(vals[*]))", 3), + &[[0.0, 30.0, 20.0], [0.0, 30.0, 20.0], [0.0, 30.0, 20.0]], + "MAX of a PREVIOUS view", + ); + // The explicit spelling of the default fallback is the same value and must + // stay accepted: `PREVIOUS(x)` desugars to exactly this. + assert_series( + &moving_series( + "c3_first_explicit", + "out[d]", + "SUM(PREVIOUS(vals[*], 0))", + 3, + ), + &[[0.0, 60.0, 45.0], [0.0, 60.0, 45.0], [0.0, 60.0, 45.0]], + "SUM of a PREVIOUS view with an explicit 0 fallback", + ); +} + +/// The VM's half of the first-step semantics across a RESET, which one run +/// cannot reach: `Vm::reset` clears `prev_values_valid`, and a snapshot view +/// must go back to reading the fallback rather than the finished run's last +/// snapshot. +/// +/// The VM is doubly protected here (it also zero-fills `prev_values` on reset), +/// which is exactly why this is asserted rather than assumed: the wasm backend +/// deliberately does NOT clear its snapshot regions and reproduces the semantics +/// with a `select` instead +/// (`wasmgen::module_tests::compile_simulation_repeated_run_resets_previous_fallback_for_an_array_view`, +/// which fails without it). The two backends must agree, so both sides of the +/// axis carry a row. +#[test] +fn a_reset_run_reads_the_fallback_again() { + let project = moving_fixture("c3_reset").array_aux("out[d]", "SUM(PREVIOUS(vals[*]))"); + let compiled = project + .compile_incremental() + .expect("the fixture should compile"); + let mut vm = crate::vm::Vm::new(compiled).expect("VM creation should succeed"); + vm.run_to_end().expect("first run"); + let first = vm + .get_series(&crate::common::Ident::new("out[1]")) + .expect("out[1] series"); + vm.reset(); + vm.run_to_end().expect("second run"); + let second = vm + .get_series(&crate::common::Ident::new("out[1]")) + .expect("out[1] series"); + assert_close(&first, &[0.0, 60.0, 45.0], "first run"); + assert_close(&second, &first, "a reset run must reproduce the first run"); +} + +/// The pinned decline: a NON-default fallback on an array-valued `PREVIOUS`. +/// +/// A view carries no per-call-site scalar, so the array route can only reproduce +/// the fallback the snapshot buffer already reads as before its first snapshot, +/// which is 0. Approximating -- silently reading 0 where the model asked for 5 +/// -- would be a wrong number on the first step of every run, so the shape is +/// refused instead. The scalar spelling is unaffected, and the row below shows +/// the workaround: capture the array into a variable of its own, where each +/// element's `LoadPrev` carries the fallback. +#[test] +fn a_non_default_array_previous_fallback_declines_loudly() { + assert_fails_attributed( + moving_fixture("c3_fb_reject") + .array_aux("out[d]", "VECTOR SORT ORDER(PREVIOUS(vals[d], 5), 1)"), + "array PREVIOUS with a non-zero fallback", + ); + // The rejection must name the FALLBACK, not merely fail: this construct is + // one the practitioner can fix, and the message says how. Asserted through + // the per-variable diagnostic, which is the surface a user reads. + assert_declines_because( + moving_fixture("c3_fb_reason") + .array_aux("out[d]", "VECTOR SORT ORDER(PREVIOUS(vals[d], 5), 1)"), + "out", + "nowhere to carry a fallback", + ); + // `-0.0` is not the default either, and the check compares BIT PATTERNS so + // that it is not. The spelling matters: `-0` is a negation of the literal + // `0`, which constant folding turns into `+0.0`, so it is accepted and IS + // the default. `0 * -1` folds to a genuine `-0.0` (the shape + // `compiler::fold` is documented to produce), and `1 / PREVIOUS(x, 0 * -1)` + // is negative infinity where `1 / PREVIOUS(x, 0)` is positive -- a value + // comparison would silently accept it and read the wrong sign of infinity + // on the first step. + assert_fails_attributed( + moving_fixture("c3_fb_negzero") + .array_aux("out[d]", "VECTOR SORT ORDER(PREVIOUS(vals[d], 0 * -1), 1)"), + "array PREVIOUS with a -0.0 fallback", + ); + // `-0` IS accepted, and that is the other half of the bit-pattern claim: + // the literal is a negation of `0`, which constant folding turns back into + // `+0.0`, so it IS the default and must not be refused. Only the folded + // `0 * -1` above produces a genuine `-0.0`. Same series as the bare + // `PREVIOUS(vals[d])` row, since the fallback is the default either way. + assert_series( + &moving_series( + "c3_fb_negzero_ok", + "out[d]", + "VECTOR SORT ORDER(PREVIOUS(vals[d], -0), 1)", + 3, + ), + &[[0.0, 1.0, 0.0], [1.0, 2.0, 1.0], [2.0, 0.0, 2.0]], + "a `-0` fallback folds to the default and is accepted", + ); + + // The workaround compiles and is per-element correct: at t=0 every element + // reads 5, so the sort order is the identity under stable ties; afterwards + // it is the previous array's order, matching the bare row above. + let workaround = moving_fixture("c3_fb_helper") + .array_aux("cap[d]", "PREVIOUS(vals[d], 5)") + .array_aux("out[d]", "VECTOR SORT ORDER(cap[d], 1)"); + workaround.assert_compiles_incremental(); + let all = workaround.run_vm_incremental(); + assert_close(all.get("out[1]").unwrap(), &[0.0, 1.0, 0.0], "out[1]"); + assert_close(all.get("out[2]").unwrap(), &[1.0, 2.0, 1.0], "out[2]"); + assert_close(all.get("out[3]").unwrap(), &[2.0, 0.0, 2.0], "out[3]"); +} + +/// The three spellings of an arrayed reference, at `VECTOR SORT ORDER` arg0. +/// +/// They arrive from three different directions and only one of them ever +/// reached lowering intact before: `vals` (a bare name) already lowered to a +/// whole-array view; `vals[*]` and `vals[d]` were claimed by +/// `builtins_visitor`'s capture-helper synthesis, which cannot hold an array +/// (`vals[*]` in a scalar `Equation::Scalar` helper does not compile) or pinned +/// them to one element (`substitute_dimension_refs` rewriting `d` to `d·elem`). +/// All three must now mean the same array. +#[test] +fn all_three_arrayed_previous_spellings_agree() { + let expected = [[0.0, 1.0, 0.0], [1.0, 2.0, 1.0], [2.0, 0.0, 2.0]]; + for (name, eqn) in [ + ("c3_sp_dim", "VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)"), + ("c3_sp_star", "VECTOR SORT ORDER(PREVIOUS(vals[*]), 1)"), + ("c3_sp_bare", "VECTOR SORT ORDER(PREVIOUS(vals), 1)"), + ] { + assert_series( + &moving_series(name, "out[d]", eqn, 3), + &expected, + &format!("spelling: {eqn}"), + ); + } + + // The other side of the boundary, unchanged: `PREVIOUS` of a SINGLE element + // is still a scalar that broadcasts, not an array. `matrix[e,1]` pins the + // trailing index, so the reference collapses to one slot and the operand is + // `prev(matrix[e,0]) + matrix[e,*]` per element. + // + // The scalar is BROADCAST across the three reduced elements, so the sum is + // `3 * prev_element + row_sum`. Row 1: prev(matrix[1,1]) is 0, 1, 2 and the + // row sums are 6, 7, 8 -> 6, 10, 14. Row 2: prev(matrix[2,1]) is 0, 10, 10 + // over rows summing 60, 70, 80 -> 60, 100, 110. (Treating the element as an + // ARRAY instead would give the row-slice sums 0, 6, 7 added to 6, 7, 8.) + assert_series( + &moving_series( + "c3_sp_element", + "out[e]", + "SUM(PREVIOUS(matrix[e,1]) + matrix[e,*])", + 2, + ), + &[[6.0, 10.0, 14.0], [60.0, 100.0, 110.0]], + "PREVIOUS of a fixed element still broadcasts", + ); +} + +/// The DEGENERATE half of the view-operand rule, pinned so GH #995's "do NOT +/// simply make everything compile" section can be checked against it. +/// +/// An element-collapsed `PREVIOUS`/`INIT` in a rank-like position now compiles, +/// to a one-element view -- a constant `0` sort order, a constant `1` rank. That +/// is the trap the issue names, and what makes it acceptable here is that it is +/// EXACTLY what the non-`PREVIOUS` twin already produced: `VECTOR SORT +/// ORDER(vals[1], 1)` is the same constant 0 at HEAD, and has been. C3 did not +/// create a degenerate answer; it stopped `PREVIOUS` from being the one operand +/// that behaved differently from its own argument in the same position. +/// +/// The half the issue actually warns about -- the LTM ceteris-paribus wrap +/// pinning a rank-like builtin's ARGUMENT down to one element, turning a loud +/// drop into a plausible constant-0 score -- is unaffected: `ltm_agg`'s +/// rank-like decline is independent of compilability, and C-LEARN's five +/// `rank-like-partial` declines are byte-identical before and after C3. +#[test] +fn an_element_collapsed_snapshot_in_a_rank_like_position_matches_its_curr_twin() { + for (name, eqn, expected) in [ + ("c3_degen_vso", "VECTOR SORT ORDER(vals[1], 1)", 0.0), + ( + "c3_degen_vso_prev", + "VECTOR SORT ORDER(PREVIOUS(vals[1]), 1)", + 0.0, + ), + ("c3_degen_rank", "RANK(vals[1], 1)", 1.0), + ("c3_degen_rank_prev", "RANK(PREVIOUS(vals[1]), 1)", 1.0), + ] { + let series = moving_series(name, "out[d]", eqn, 3); + for (k, s) in series.iter().enumerate() { + assert_close( + s, + &[expected; 3], + &format!( + "{eqn}: a one-element view is degenerate at element {}", + k + 1 + ), + ); + } + } +} + +/// The shape PR #1001 was written against, verbatim: a per-row `VECTOR SELECT` +/// over the previous step's matrix rows. +/// +/// `sel2` selects columns 1 and 3 of row 1 and column 2 of row 2, over +/// `PREVIOUS(matrix[Row,*])`. The previous rows are `[0,0,0]`/`[0,0,0]`, then +/// `[1,2,3]`/`[10,20,30]`, then `[2,2,3]`/`[10,20,40]`, so the selected sums are +/// 0/0, 1+3=4 / 20, and 2+3=5 / 20. +#[test] +fn the_gh_1001_user_shape_compiles_and_reads_the_previous_row() { + let project = moving_fixture("c3_user_shape") + .array_with_ranges( + "sel2[e,d]", + vec![ + ("1,1", "1"), + ("1,2", "0"), + ("1,3", "1"), + ("2,1", "0"), + ("2,2", "1"), + ("2,3", "0"), + ], + ) + .array_aux( + "picked[e]", + "VECTOR SELECT(sel2[e,*], PREVIOUS(matrix[e,*]), 0, 0, 0)", + ); + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + assert_close(all.get("picked[1]").unwrap(), &[0.0, 4.0, 5.0], "picked[1]"); + assert_close( + all.get("picked[2]").unwrap(), + &[0.0, 20.0, 20.0], + "picked[2]", + ); +} + +/// The other side of that boundary: a `PREVIOUS`/`INIT` of a genuinely SCALAR +/// variable beside an array operand lowers to an `Expr::Var`, carries no +/// per-element identity, and broadcasts correctly -- so it must keep +/// materializing. Without this row the decline could be widened to "contains +/// any PREVIOUS" and nothing would notice. +/// +/// `PREVIOUS(s)` is the constant 5 at every step, so the operand keeps `vals`' +/// ascending order, `[1, 2, 0]`. +#[test] +fn a_scalar_previous_beside_an_array_operand_still_materializes() { + let project = fixture("scalar_prev") + .scalar_const("s", 5.0) + .array_aux("out[d]", "VECTOR SORT ORDER(vals[d] + PREVIOUS(s), 1)"); + project.assert_compiles_incremental(); + assert_close( + &project.vm_result_incremental("out"), + &[1.0, 2.0, 0.0], + "a scalar PREVIOUS beside an array operand", + ); + + let init = fixture("scalar_init") + .scalar_const("s", 5.0) + .array_aux("out[d]", "VECTOR SORT ORDER(vals[d] + INIT(s), 1)"); + init.assert_compiles_incremental(); + assert_close( + &init.vm_result_incremental("out"), + &[1.0, 2.0, 0.0], + "a scalar INIT beside an array operand", + ); + + // The boundary case the predicate has to get right: a PREVIOUS of a fixed + // element spelled `matrix[e,1]` (2-D, literal trailing index) lowers to an + // `Expr::Var` at that element's slot, and broadcasting it across the + // reduced row is exactly what the equation says. Declining on "contains + // any PREVIOUS" would break this. Measured for THIS spelling only: the + // 1-D literal-index spelling `PREVIOUS(vals[2])` lowers to a + // `StaticSubscript` instead, so the predicate declines it -- loud, and no + // regression, since an operand containing an App was never a view shape + // and did not compile before this pass either. + // + // Row 0: prev(matrix[0,0]) = 1, so SUM over [1,2,3] of (1 + x) = 9; + // row 1: prev(matrix[1,0]) = 10, so SUM over [10,20,30] of (10 + x) = 90. + // Plus the array-producing tail, worth 3. NOTE the value assertion here is + // a compile-boundary pin, not a discriminator: matrix is constant, so the + // same equation without the PREVIOUS returns the identical [12, 93] -- what + // this row defends against is the decline being widened until the shape + // stops compiling, and the values merely confirm the broadcast reading. + let fixed_element = fixture("scalar_prev_elem").array_aux( + "out[e]", + "SUM(PREVIOUS(matrix[e,1]) + matrix[e,*]) + SUM(VECTOR SORT ORDER(vals[*], 1))", + ); + fixed_element.assert_compiles_incremental(); + assert_close( + &fixed_element.vm_result_incremental("out"), + &[12.0, 93.0], + "a PREVIOUS of a fixed element broadcasts correctly and must not decline", + ); + + // A SCALAR `PREVIOUS`/`INIT` directly in a view position. `SUM(s)` for a + // scalar `s` has always compiled -- `walk_expr_as_view`'s `Expr::Var` arm + // pushes a one-element view -- while `SUM(PREVIOUS(s))` did not, which was + // the same incoherence as the array rows. Both now take the same route, so + // the reduce reads one element of the snapshot: `s = 10 + 5 * TIME`, so + // `SUM(PREVIOUS(s))` is 0 (the fallback), 10, 15 and `SUM(INIT(s))` is 10 + // throughout. + for (name, eqn, expected) in [ + ("scalar_view_prev", "SUM(PREVIOUS(s))", [0.0, 10.0, 15.0]), + ("scalar_view_init", "SUM(INIT(s))", [10.0, 10.0, 10.0]), + ] { + let project = moving_fixture(name) + .aux("s", "10 + 5 * TIME", None) + .array_aux("out[d]", eqn); + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + assert_close(all.get("out[1]").unwrap(), &expected, eqn); + } +} + +// =========================================================================== +// The safety property the whole pass rests on. +// =========================================================================== + +/// The materializer must fire *only* where codegen would have rejected the +/// operand, so a fragment that compiled before is unchanged after. The +/// observable version of that claim: an operand that is already one of +/// `walk_expr_as_view`'s four accepted shapes consumes no extra temp, and a +/// computed one costs exactly one. +/// +/// `temp_sizes` is derived from the lowered expressions, so it is the direct +/// readout of how many temps a fragment allocates. +#[test] +fn a_computed_operand_costs_exactly_one_temp_and_a_view_costs_none() { + let temps = |name: &str, eqn: &str| -> usize { + fixture(name) + .array_aux("out[d]", eqn) + .build_module() + .unwrap_or_else(|e| panic!("{name} should build: {e}")) + .temp_sizes + .len() + }; + + let control = temps("temp_ctl", "VECTOR SORT ORDER(vals[d], 1)"); + assert_eq!( + control, 1, + "a direct-reference VECTOR SORT ORDER needs exactly the one temp its \ + own result lives in" + ); + assert_eq!( + temps("temp_computed", "VECTOR SORT ORDER(vals[d] + bump[d], 1)"), + control + 1, + "materializing a computed operand costs exactly one temp beyond the \ + builtin's own result" + ); + assert_eq!( + temps( + "temp_shape_nested", + "VECTOR SORT ORDER(ABS(vals[d] - 25), 1)" + ), + control + 1, + "an elementwise builtin operand costs the same one temp -- the \ + materializer allocates per operand, not per node" + ); +} + +/// The hoisted `AssignTemp` must be spliced in FRONT of the expression that +/// reads it. Nothing about a constant model can tell: a temp written after its +/// reader still holds the right value from the previous step, and at step 0 a +/// zeroed temp can coincide with the answer. So this row makes the operand +/// vary with time and reads the per-element series, where a stale temp is a +/// visibly different array at every step but the last. +/// +/// `vals + bump * TIME` is [30,10,20] at t=0, [30,110,20] at t=1 and +/// [30,210,20] at t=2, so the ascending order moves from [1,2,0] to [2,0,1] +/// and stays. Read one step late it would be [0,1,2] (a zeroed temp sorts to +/// the identity under stable ties), then [1,2,0], then [2,0,1]. +#[test] +fn the_hoisted_assignment_is_emitted_before_its_reader() { + let project = fixture("hoist_order") + .with_sim_time(0.0, 2.0, 1.0) + .array_aux("out[d]", "VECTOR SORT ORDER(vals[d] + bump[d] * TIME, 1)"); + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + let series = |elem: usize| -> Vec { + all.get(&format!("out[{elem}]")) + .unwrap_or_else(|| panic!("out[{elem}] missing from {:?}", all.keys())) + .clone() + }; + // Element-major: out[1] over t = 0, 1, 2 and so on. Written out as the + // three per-step arrays for readability: [1,2,0], [2,0,1], [2,0,1]. + assert_close(&series(1), &[1.0, 2.0, 2.0], "out[1] over time"); + assert_close(&series(2), &[2.0, 0.0, 0.0], "out[2] over time"); + assert_close(&series(3), &[0.0, 1.0, 1.0], "out[3] over time"); +} + +// =========================================================================== +// The `TempId` namespace (GH #583). +// +// The per-element hoisting path allocates one temp per array ELEMENT -- each +// element re-evaluates the builtin with its own scalar argument -- and +// materializing a computed operand doubles that. `TempId` is a `u8`, so a few +// hundred elements is past the namespace, and BOTH tests below live at that +// boundary. +// =========================================================================== + +/// A per-element hoist over `sort_project`'s dimension: `vals` descends and the +/// operand `301 - vals[d]` ascends, so the two readings are exact swaps at +/// every element. Element `k` sorts ascending when `k` is odd. +fn sort_project(name: &str, n: usize, eqn: &str) -> TestProject { + fn refs(v: &[(String, String)]) -> Vec<(&str, &str)> { + v.iter().map(|(a, b)| (a.as_str(), b.as_str())).collect() + } + // vals[j] = n - 1 - j, i.e. [n-1, ..., 0]; `301 - vals[d]` is increasing. + let vals: Vec<(String, String)> = (0..n) + .map(|j| ((j + 1).to_string(), (n - 1 - j).to_string())) + .collect(); + let dir: Vec<(String, String)> = (0..n) + .map(|k| { + ( + (k + 1).to_string(), + if k % 2 == 1 { "1" } else { "-1" }.to_string(), + ) + }) + .collect(); + TestProject::new(name) + .indexed_dimension("d", n as u32) + .array_with_ranges("vals[d]", refs(&vals)) + .array_with_ranges("dir[d]", refs(&dir)) + .array_aux("out[d]", eqn) +} + +/// A per-element hoist that consumes MORE than 256 temp ids but never uses a +/// temp as a view SOURCE still produces the right numbers, and must keep doing +/// so: every writer and every reader of such a temp narrows the same id the +/// same way (`write_temp_id: id as TempId` against `LoadTempConst`'s +/// `temp_id: id as TempId`), and each element's temp is written immediately +/// before it is read, so the aliasing is unobservable. +/// +/// That reasoning is bounded, and the bound is the fixture: the aliased temps +/// here are all the SAME SIZE, because a per-element hoist over one array +/// repeats one shape. Truncation is NOT safe in general -- temps of different +/// sizes sharing a truncated id let the larger write run past the smaller slot +/// into its neighbour's storage, in-bounds for the flat temp region and +/// therefore silent. No lowering path emits that today; #583 is the fix for +/// both halves. +/// +/// So this is a property of the emission pattern, not of the namespace, and it +/// is pinned here rather than assumed: the moment a change makes those two +/// narrowings disagree, this returns a different array instead of failing. +#[test] +fn a_per_element_hoist_past_the_temp_namespace_without_a_temp_view_is_correct() { + const N: usize = 300; + let project = sort_project("temp_namespace_ok", N, "VECTOR SORT ORDER(vals[d], dir[d])"); + project.assert_compiles_incremental(); + + // Over the DEcreasing `vals`, ascending is the reversal and descending is + // the identity. + let expected: Vec = (0..N) + .map(|k| if k % 2 == 1 { N - 1 - k } else { k } as f64) + .collect(); + assert_close( + &project.vm_result_incremental("out"), + &expected, + "300-element per-element hoist, no temp read as a view", + ); +} + +/// The same hoist WITH a materialized operand puts a temp in a view position, +/// and a view base is the one place a temp id is carried as a `u32` while +/// every writer narrows it to `u8` -- so above 255 the view reads storage no +/// opcode wrote. `symbolic::resolve_static_view` rejects that rather than +/// emitting a well-formed program with wrong numbers. +/// +/// Both spellings are covered because they arrive from different directions, +/// and only one of them is new: the `vals[*]` spelling ALREADY put a Pass-1 +/// temp in a view position, so at HEAD it returned a silently wrong array from +/// element 128 on (a pre-existing #583 instance, not caused by this work); the +/// `vals[d]` spelling did not compile at all before this module's fix, and +/// would have joined it. Both are now loud. +/// +/// 130 elements is the smallest round size past the boundary (two temps per +/// element, so ids cross 255 at element 128). +#[test] +fn a_temp_read_as_a_view_past_the_temp_namespace_is_rejected() { + for (name, eqn) in [ + ("temp_view_dim", "VECTOR SORT ORDER(301 - vals[d], dir[d])"), + ("temp_view_star", "VECTOR SORT ORDER(301 - vals[*], dir[d])"), + ] { + let err = sort_project(name, 130, eqn) + .compile_incremental() + .err() + .unwrap_or_else(|| panic!("{eqn}: a view over a temp above 255 must be rejected")); + let details = err.get_details().unwrap_or_default(); + assert!( + details.contains("TempId capacity"), + "{eqn}: expected the temp-namespace rejection, got {err:?}" + ); + } +} + +/// A residual this work does NOT fix, pinned so it is loud rather than +/// silently rediscovered: an array-producing builtin nested inside +/// *arithmetic* that is itself an array operand. +/// +/// `VECTOR SORT ORDER(VECTOR ELM MAP(a, b) + c, 1)` materializes the `Op2` into +/// a temp correctly -- for the star spelling that already happened in Pass 1, +/// before this work -- but the resulting `AssignTemp` body still holds the +/// inner `App(VectorElmMap)`, and codegen's `AssignTemp` arm only routes an +/// array-producing builtin to its opcode when the builtin is the body's *root*. +/// Anywhere else in the body it reaches the `BeginIter` loop and is rejected +/// with "array-producing builtin outside AssignTemp context". +/// +/// That is a different contract from the one this module is about -- where an +/// array-producing `App` may APPEAR, not whether an operand is a view -- and +/// fixing it needs a notion of "this subexpression is array-valued" that the +/// lowered `Expr` tree does not carry locally. The bare nested form +/// (`VECTOR SORT ORDER(VECTOR ELM MAP(a, b), 1)`, no arithmetic) does work; see +/// [`computed_operand_shapes`]. +#[test] +fn a_nested_array_producing_builtin_inside_arithmetic_is_a_separate_residual() { + for (name, eqn) in [ + ( + "residual_dim", + "VECTOR SORT ORDER(VECTOR ELM MAP(vals[d], offs[d]) + bump[d], 1)", + ), + ( + "residual_star", + "VECTOR SORT ORDER(VECTOR ELM MAP(vals[*], offs[*]) + bump[*], 1)", + ), + ] { + assert_fails_attributed(fixture(name).array_aux("out[d]", eqn), eqn); + } +} + +/// GH #995's own table, re-run. Every row the issue reported as failing now +/// compiles, and each is checked against the reading it is supposed to have. +/// +/// Two rows resolve by COHERENCE rather than by gaining an array meaning, and +/// they are the ones worth stating: a single element stays a single element +/// under `PREVIOUS`, so in an array-operand position it pushes a ONE-ELEMENT +/// view -- a legitimate `VECTOR ELM MAP` base (the mapping ranges over the whole +/// source variable) and a degenerate one-element `VECTOR SELECT`. +/// +/// The SPELLING decides which route the element takes, and the two are +/// different: a NUMERIC index (`vals[1]`) reaches the view over the snapshot, +/// while the bare element NAME the issue's table uses (`vals[e1]`) is not +/// accepted as a static index on the user-equation parse path, so it is read +/// through a scalar capture helper of extent one instead. Both are asserted +/// below, each against its own oracle, rather than one standing in for the +/// other -- the comments on each `compare` call carry the difference. +#[test] +fn every_row_of_the_issue_995_table_compiles() { + // The issue's own dimension: element names, so `vals[e1]` is a literal + // element rather than an index. + let base = |name: &str| { + TestProject::new(name) + .with_sim_time(0.0, 2.0, 1.0) + .named_dimension("d", &["e1", "e2", "e3"]) + .array_with_ranges( + "vals[d]", + vec![ + ("e1", "30 - 25 * TIME"), + ("e2", "10 + 10 * TIME"), + ("e3", "20"), + ], + ) + .array_with_ranges( + "offs[d]", + vec![("e1", "2 - TIME"), ("e2", "0"), ("e3", "1")], + ) + }; + + // Every row of the table, in the issue's order. The first three were + // reported as compiling and are the controls; the rest were reported as + // failing. + let rows = [ + "VECTOR SORT ORDER(vals[d], 1)", + "VECTOR ELM MAP(vals[e1], offs[d])", + "RANK(vals[*], 1)", + "VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)", + "VECTOR ELM MAP(PREVIOUS(vals[e1]), offs[d])", + "VECTOR ELM MAP(vals[e1], PREVIOUS(offs[d]))", + "VECTOR SORT ORDER(INIT(vals[d]), 1)", + "VECTOR SORT ORDER(vals[d] * 2, 1)", + "VECTOR SELECT(PREVIOUS(offs[d]), vals[d], 0, 1, 0)", + "RANK(vals[*] * 2, 1)", + ]; + for (i, eqn) in rows.iter().enumerate() { + base(&format!("t995_{i}")) + .array_aux("out[d]", eqn) + .assert_compiles_incremental(); + } + + // The two coherence rows, against the per-element capture. `cap` holds + // `PREVIOUS(vals[d])` / `PREVIOUS(offs[d])` element by element -- one + // `LoadPrev` per slot -- so substituting it for the inline `PREVIOUS` must + // not change a number. + let compare = |what: &str, capture: (&str, &str), inline: &str, reference: &str| { + let run = |name: &str, project: TestProject| -> Vec> { + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + ["e1", "e2", "e3"] + .into_iter() + .map(|k| { + all.get(&format!("out[{k}]")) + .unwrap_or_else(|| panic!("{name}: out[{k}] missing")) + .clone() + }) + .collect() + }; + let a = run(what, base("t995_i").array_aux("out[d]", inline)); + let b = run(what, { + // The captures are a mix of arrayed and scalar helpers, so pick + // the constructor from the name rather than hard-coding one. + let p = base("t995_r"); + let p = if capture.0.contains('[') { + p.array_aux(capture.0, capture.1) + } else { + p.aux(capture.0, capture.1, None) + }; + p.array_aux("out[d]", reference) + }); + for (k, (x, y)) in a.iter().zip(b.iter()).enumerate() { + // NaN-tolerant: a mapped offset outside the source's extent is a + // genuine `:NA:`, and two runs agreeing on WHERE the NaNs fall is + // part of what is being checked. + assert_eq!(x.len(), y.len(), "{what}: element {} length", k + 1); + for (step, (p, q)) in x.iter().zip(y.iter()).enumerate() { + assert!( + (p.is_nan() && q.is_nan()) || (p - q).abs() < 1e-9, + "{what}: element {} step {step} -- inline {p}, per-element capture {q} \ + (inline {x:?}, reference {y:?})", + k + 1 + ); + } + } + }; + // A single-element PREVIOUS base for VECTOR ELM MAP, spelled with a NUMERIC + // index: the argument reaches lowering as the same collapsed + // `StaticSubscript` its `curr` twin does, so the source keeps the whole + // variable's extent and the mapping ranges over the previous array. + compare( + "VECTOR ELM MAP with a single-element PREVIOUS base", + ("cap[d]", "PREVIOUS(vals[d])"), + "VECTOR ELM MAP(PREVIOUS(vals[1]), offs[d])", + "VECTOR ELM MAP(cap[1], offs[d])", + ); + // The SAME element spelled with its bare NAME takes a different route, and + // that is the spelling the issue's table uses. `index_is_static` will not + // accept an unqualified element name on the user-equation parse path (such a + // name can be shadowed by a variable, and the disambiguating check is + // deliberately disabled there to stay incremental under renames), so + // `builtins_visitor` synthesizes a scalar capture helper and `PREVIOUS` + // reads THAT. The source is then the helper -- one slot -- and ELM MAP's + // "range over the source variable's full storage" rule applies to it. That + // is the same rule a materialized operand follows + // (`materializing_an_elm_map_source_confines_the_mapping_to_the_temp`) and + // the same answer a practitioner gets by writing the capture out, so it is + // self-consistent rather than a second semantics -- but the two spellings DO + // mean different things, and only the front end decides which, so both are + // pinned rather than one standing in for the other. + compare( + "VECTOR ELM MAP with a bare-element-name PREVIOUS base", + ("h", "PREVIOUS(vals[e1])"), + "VECTOR ELM MAP(PREVIOUS(vals[e1]), offs[d])", + "VECTOR ELM MAP(h, offs[d])", + ); + compare( + "VECTOR SELECT over a single-element PREVIOUS selection", + ("cap[d]", "PREVIOUS(offs[d])"), + "VECTOR SELECT(PREVIOUS(offs[d]), vals[d], 0, 1, 0)", + "VECTOR SELECT(cap[d], vals[d], 0, 1, 0)", + ); +} + +// =========================================================================== +// Shape axis, second dimension: an operand mixing arrays of DIFFERENT shapes. +// =========================================================================== + +/// A wider companion for the shared fixture: `matrix[e,d]` is already there, +/// and `rowv[e]` is the shape that is incomparable with `vals[d]`. +/// +/// * `rowv = [5, 50]` +/// * `matrixt[d,e]` is `matrix` transposed, the shape that ties with +/// `matrix[e,d]` on containment while disagreeing on axis order. +fn wide_fixture(name: &str) -> TestProject { + fixture(name) + .array_with_ranges("rowv[e]", vec![("1", "5"), ("2", "50")]) + .array_with_ranges( + "matrixt[d,e]", + vec![ + ("1,1", "1"), + ("1,2", "10"), + ("2,1", "2"), + ("2,2", "20"), + ("3,1", "3"), + ("3,2", "30"), + ], + ) +} + +/// Compile `out[e,d] = ` against the wide fixture and return `out`, +/// row-major (`[e1d1, e1d2, e1d3, e2d1, e2d2, e2d3]`). +fn wide_out_of(name: &str, eqn: &str) -> Vec { + let project = wide_fixture(name).array_aux("out[e,d]", eqn); + project.assert_compiles_incremental(); + project.vm_result_incremental("out") +} + +/// Both spellings of a commutative mixed-shape operand must produce the same +/// array -- the property the first-wins shape rule broke. +/// +/// A computed operand is evaluated by codegen's `AssignTemp` -> `BeginIter` +/// loop, which broadcasts each source view onto the ITERATION by dimension id +/// (`vm`'s `LoadIterViewAt` -> `dimensions::match_dimensions_two_pass`), and a +/// source dimension the iteration does not have reads NaN. Shaping the temp by +/// the first array in the operand therefore made +/// `VECTOR SORT ORDER(vals[d] + matrix[e,d], 1)` iterate over `vals`'s three +/// elements, read `matrix` as three NaNs and return the sort order of NaNs +/// (measured `[0,1,2, 0,1,2]`), while the commuted `matrix[e,d] + vals[d]` -- +/// the same array -- returned the right answer. `compiler::join_array_views` +/// picks the shape by CONTAINMENT instead, which has no left-to-right in it. +/// +/// The values: `vals = [30,10,20]` and `matrix = [[1,2,3],[10,20,30]]`, so the +/// sum is `[[31,12,23],[40,30,50]]` and the in-row ascending orders are +/// `[1,2,0]` and `[1,0,2]`. +#[test] +fn a_mixed_shape_operand_agrees_with_its_commuted_spelling() { + let expected = [1.0, 2.0, 0.0, 1.0, 0.0, 2.0]; + // Narrow first -- the spelling that read NaNs. + assert_close( + &wide_out_of("mix_narrow", "VECTOR SORT ORDER(vals[d] + matrix[e,d], 1)"), + &expected, + "mixed-shape operand, narrow array first", + ); + // Wide first -- the spelling that happened to work. + assert_close( + &wide_out_of("mix_wide", "VECTOR SORT ORDER(matrix[e,d] + vals[d], 1)"), + &expected, + "mixed-shape operand, wide array first", + ); + + // A DIMENSIONLESS subexpression is the degenerate case of the same rule: a + // subscript collapsed to one element carries no dimensions, so it + // broadcasts and constrains nothing. Reading the first view blind made the + // two orders disagree about whether the equation compiles AT ALL -- + // `vals[1] + bump[d]` was rejected while `bump[d] + vals[1]` compiled. + // `vals[1] + bump = [30, 130, 30]`, ascending with stable ties `[0, 2, 1]`. + assert_close( + &out_of("mix_elem_lhs", "VECTOR SORT ORDER(vals[1] + bump[d], 1)"), + &[0.0, 2.0, 1.0], + "collapsed element first", + ); + assert_close( + &out_of("mix_elem_rhs", "VECTOR SORT ORDER(bump[d] + vals[1], 1)"), + &[0.0, 2.0, 1.0], + "collapsed element second", + ); +} + +/// The join is over ALL the shapes, not just two, and it is order-independent +/// in the strong sense: no permutation of a three-array operand may change the +/// answer. +/// +/// This is what makes the choice a maximum rather than a left-to-right fold. A +/// fold over `[d], [d], [e,d]` is fine, but a fold over `[e], [d], [e,d]` -- +/// the shape of an operand mixing a row vector, a column vector and the matrix +/// they broadcast into -- would call the first two incomparable and decline +/// before ever seeing the third. +/// +/// `vals + bump = [30,110,20]`, plus `matrix` rows gives `[[31,112,23], +/// [40,130,50]]`, whose in-row ascending orders are `[2,0,1]` and `[0,2,1]`. +#[test] +fn a_three_array_operand_joins_regardless_of_order() { + let expected = [2.0, 0.0, 1.0, 0.0, 2.0, 1.0]; + for (name, eqn) in [ + ( + "mix3_a", + "VECTOR SORT ORDER(vals[d] + bump[d] + matrix[e,d], 1)", + ), + ( + "mix3_b", + "VECTOR SORT ORDER(matrix[e,d] + vals[d] + bump[d], 1)", + ), + ( + "mix3_c", + "VECTOR SORT ORDER(vals[d] + matrix[e,d] + bump[d], 1)", + ), + ] { + assert_close(&wide_out_of(name, eqn), &expected, eqn); + } + // The row/column/matrix mix a fold would decline on its second step. Only + // `matrix` is maximal, so the join is `[e,d]`. + // `rowv = [5,50]` broadcast down the rows plus `vals = [30,10,20]` across + // them plus `matrix` gives `[[36,17,28],[90,80,100]]`; in-row ascending + // orders `[1,2,0]` and `[1,0,2]`. + assert_close( + &wide_out_of( + "mix3_rcm", + "VECTOR SORT ORDER(rowv[e] + vals[d] + matrix[e,d], 1)", + ), + &[1.0, 2.0, 0.0, 1.0, 0.0, 2.0], + "row + column + matrix", + ); +} + +/// Every shape-carrying `Expr` variant reaches the same join, checked at the +/// one position (`VECTOR SORT ORDER` arg0) the shape axis is exercised at -- +/// the mixed-shape twin of [`computed_operand_shapes`]. +/// +/// The `If` row is the one that is not merely a repeat of the `Op2` rule: the +/// CONDITION is a fourth operand, and it is read by the `BeginIter` body +/// (`codegen::collect_iter_source_views_impl` pushes its view) even though it +/// contributes nothing to an `IF` whose arms already agree. A shape derivation +/// that skipped it sized the temp from the arms alone and the condition read +/// NaN, which compares false, so the `IF` silently collapsed to its ELSE arm +/// for every element (measured `[0,2,1, 0,2,1]`). +#[test] +fn every_operand_shape_reaches_the_mixed_shape_join() { + // Op2: covered by `a_mixed_shape_operand_agrees_with_its_commuted_spelling`. + + // Op1 (`NOT`, the only one a fragment can carry -- see + // `computed_operand_shapes`). `vals < matrix` is [F,F,F] on row 1 and + // [F,T,T] on row 2, so `NOT` gives [[1,1,1],[1,0,0]] and the in-row + // ascending orders are [0,1,2] and [1,2,0]. + assert_close( + &wide_out_of( + "mixs_op1", + "VECTOR SORT ORDER(NOT (vals[d] < matrix[e,d]), 1)", + ), + &[0.0, 1.0, 2.0, 1.0, 2.0, 0.0], + "Op1 over a mixed-shape comparison", + ); + + // If, with the wide array in the CONDITION and both arms narrow. + // `matrix > 5` is false across row 1 and true across row 2, so the result + // is `bump = [0,100,0]` then `vals = [30,10,20]`; in-row ascending orders + // [0,2,1] and [1,2,0]. + assert_close( + &wide_out_of( + "mixs_if_cond", + "VECTOR SORT ORDER(IF matrix[e,d] > 5 THEN vals[d] ELSE bump[d], 1)", + ), + &[0.0, 2.0, 1.0, 1.0, 2.0, 0.0], + "If with a wider condition than its arms", + ); + + // App, a multi-argument elementwise builtin: `MAX(vals[d], matrix[e,d])` is + // [[30,10,20],[30,20,30]], in-row ascending orders [1,2,0] and [1,0,2]. + // Both argument orders, since this arm has its own first-wins rule. + for (name, eqn) in [ + ( + "mixs_max_a", + "VECTOR SORT ORDER(MAX(vals[d], matrix[e,d]), 1)", + ), + ( + "mixs_max_b", + "VECTOR SORT ORDER(MAX(matrix[e,d], vals[d]), 1)", + ), + ] { + assert_close( + &wide_out_of(name, eqn), + &[1.0, 2.0, 0.0, 1.0, 0.0, 2.0], + eqn, + ); + } + + // App, a single-argument elementwise builtin wrapping the mix. + // `ABS(vals[d] - matrix[e,d])` is [[29,8,17],[20,10,10]]; in-row ascending + // orders [1,2,0] and [1,2,0] (stable tie between the two 10s). + assert_close( + &wide_out_of( + "mixs_abs", + "VECTOR SORT ORDER(ABS(vals[d] - matrix[e,d]), 1)", + ), + &[1.0, 2.0, 0.0, 1.0, 2.0, 0.0], + "elementwise builtin over a mixed-shape difference", + ); +} + +/// Two shapes neither of which contains the other DECLINE, and the decline is +/// the same loud, variable-attributed codegen rejection the two deliberately +/// unmaterialized positions produce. +/// +/// The union `[e] u [d] = [e,d]` would compile, and that is exactly why it is +/// refused: nothing in `rowv[e] + vals[d]` says whether the result is `[e,d]` +/// or `[d,e]`, and the temp's axis order is the axis `VECTOR SORT ORDER` sorts +/// along. Guessing it produces a plausible array of wrong numbers -- which is +/// what the first-wins rule did, returning the sort order of `[e]`-shaped NaNs +/// (`[0,0,0, 1,1,1]`) with no diagnostic at all. +/// +/// The transposed row is the same refusal reached from the other side: `[e,d]` +/// and `[d,e]` CONTAIN each other, so containment alone leaves two maximal +/// candidates and picking either would reintroduce the operand-order +/// dependence in the axis order. +#[test] +fn incomparable_operand_shapes_decline_loudly() { + for (name, eqn) in [ + ("incomp_rc", "VECTOR SORT ORDER(rowv[e] + vals[d], 1)"), + ("incomp_cr", "VECTOR SORT ORDER(vals[d] + rowv[e], 1)"), + ( + "incomp_transpose", + "VECTOR SORT ORDER(matrix[e,d] + matrixt[d,e], 1)", + ), + ] { + assert_fails_attributed(wide_fixture(name).array_aux("out[e,d]", eqn), eqn); + assert_declines_because( + wide_fixture(name).array_aux("out[e,d]", eqn), + "out", + "Cannot push view for expression type", + ); + } +} + +// =========================================================================== +// Module instances: a static view is addressed at the executing INSTANCE's +// slot base, not the root's. +// =========================================================================== + +/// An array view inside a sub-model instance reads that instance's own slots. +/// +/// A static view's `base_off` is resolved out of the FRAGMENT'S OWN model +/// layout (`symbolic::resolve_static_view`), so it is module-relative -- exactly +/// like the offset `Opcode::LoadVar` reads as `curr[module_off + off]`. The +/// executing instance's `module_off` has to be added at push time, and it was +/// not: `StaticArrayView::to_runtime_view` copied `base_off` verbatim, so every +/// array reduction inside a sub-model read the ROOT's slots. Over this fixture +/// both instances returned `[1, 2, 3, 4]` -- the sum of the first three global +/// slots, `time + dt + initial_time` -- instead of their own arrays. +/// +/// This predates the array-valued `PREVIOUS`/`INIT` route (`ViewStorage::Curr` +/// has always gone through the same push), so the `out_curr` rows are the +/// control and the `out_prev`/`out_init` rows are what GH #995 added on top of +/// it. All three regions are `n_slots` copies of `curr` and share its slot +/// numbering, which is why one addend serves all three -- and why a fix that +/// covered only the two new ones would have left the oldest one broken. +#[test] +fn an_array_view_inside_a_module_instance_reads_that_instance() { + use crate::db::{SimlinDb, compile_project_incremental, sync_from_datamodel_incremental}; + use crate::vm::Vm; + + let project = crate::test_common::two_instance_arrayed_submodel_project(); + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + let compiled = + compile_project_incremental(&db, sync.project, "main").expect("two-instance compile"); + let mut vm = Vm::new(compiled).expect("vm"); + vm.run_to_end().expect("run"); + let results = crate::test_common::collect_results(&vm.into_results()); + + for (name, expected) in crate::test_common::two_instance_arrayed_submodel_expected() { + let actual = results + .get(name) + .unwrap_or_else(|| panic!("no series for {name}")); + assert_close(actual, &expected, name); + } +} + +/// Every operand carrying a REPEATED-dimension view declines -- mixed with +/// another shape or as the operand's SOLE shape -- and the reason is that this +/// branch is what made them compilable in the first place. +/// +/// Measured at the MERGE BASE `ccf7ed34`, not at a branch commit: the computed +/// spelling `VECTOR SORT ORDER(matrix[d,d] * 2, 1)` FAILS to compile there. So +/// the numbers it produced mid-branch were a regression this work introduced, +/// not a pre-existing wrong answer inherited from main -- an earlier revision of +/// this test said the latter, having measured at `b45a0ca1`, which already +/// carries the materializer that makes it compile. +/// +/// `[d,d]` names one dimension twice, and every layer that projects between an +/// array and a temp matches BY NAME and takes the first hit +/// (`compiler::project_var_index_to_temp` gives both axes the same coordinate, +/// so `out[i,j]` reads `temp[i,i]`; `codegen::array_view_to_static_temp` keys +/// `DimId`s the same way). There is no answer to give, so the pass gives none. +/// What that costs is nothing: at the merge base none of these compiled. +#[test] +fn a_repeated_dimension_operand_declines_rather_than_guessing_which_axis() { + let square = |name: &str| { + fixture(name).array_with_ranges( + "square[d,d]", + vec![ + ("1,1", "11"), + ("1,2", "12"), + ("1,3", "13"), + ("2,1", "21"), + ("2,2", "22"), + ("2,3", "23"), + ("3,1", "31"), + ("3,2", "32"), + ("3,3", "33"), + ], + ) + }; + for (name, eqn) in [ + // Mixed with a different shape, both operand orders -- the join has no + // containment relation to work with. + ("sqmix_lhs", "VECTOR SORT ORDER(square[d,d] + vals[d], 1)"), + ("sqmix_rhs", "VECTOR SORT ORDER(vals[d] + square[d,d], 1)"), + // The SOLE shape. A single view needs no join, so nothing about + // containment refuses this one; it is refused because the shape itself + // cannot be projected into a temp + // (`compiler::view_repeats_a_dimension`, checked by the materializer + // after the join rather than inside it). Without this row that check + // can be deleted with every other row still green. + ("sqmix_alone", "VECTOR SORT ORDER(square[d,d] * 2, 1)"), + ( + "sqmix_two", + "VECTOR SORT ORDER(square[d,d] + square[d,d], 1)", + ), + ] { + assert_fails_attributed(square(name).array_aux("out[d,d]", eqn), eqn); + assert_declines_because( + square(name).array_aux("out[d,d]", eqn), + "out", + "Cannot push view for expression type", + ); + } + + // The array-valued `PREVIOUS`/`INIT` route reaches the same shape by a + // different door -- it pushes a view over a snapshot region rather than over + // a temp -- and it is equally new here: this did not compile at the merge + // base either. `codegen::snapshot_static_view` refuses it, with its own + // message rather than the generic view rejection. + for (name, eqn) in [ + ("sqprev", "VECTOR SORT ORDER(PREVIOUS(square[d,d]), 1)"), + ("sqinit", "VECTOR SORT ORDER(INIT(square[d,d]), 1)"), + ] { + assert_fails_attributed(square(name).array_aux("out[d,d]", eqn), eqn); + assert_declines_because( + square(name).array_aux("out[d,d]", eqn), + "out", + "names one dimension twice", + ); + } +} + +/// The complement, and the boundary of the refusal above: reading a repeated +/// dimension DIRECTLY still compiles, and still returns the wrong numbers it +/// returned at the merge base. +/// +/// This is a **disclosed pre-existing residual**, pinned in both directions so +/// it is loud rather than silently rediscovered, and deliberately not fixed +/// here. Measured at `ccf7ed34` and identical on this branch: +/// +/// | equation | result | correct | +/// |---|---|---| +/// | `out[d,d] = square[d,d]` | `[11,11,11, 22,22,22, 33,33,33]` | the matrix | +/// | `out[d,d] = VECTOR SORT ORDER(square[d,d], 1)` | `[0,0,0, 1,1,1, 2,2,2]` | `[0,1,2]` per row | +/// +/// Both are the same first-axis-wins projection: `out[i,j]` reads `[i,i]`. The +/// fix is to give the projection an axis identity rather than a dimension name +/// -- the same root cause as `db::analysis::expand_same_element`'s +/// repeated-target residual, and its own change. +/// +/// **Blast radius, measured.** Vensim REJECTS the declaration: run in Vensim DSS +/// 2026-08-04, `vensim-probes/repeated_dimension.mdl` refuses to simulate with +/// "DimA appears more than once on LHS". No MDL-imported model can carry the +/// shape, so this residual is confined to hand-authored XMILE/JSON/protobuf. +/// It is not illegitimate, though -- the XMILE v1.0 spec exemplifies the +/// declaration ("A 2D non-apply-to-all array with dimensions X by X, where X is +/// size 2", verified in `docs/reference/xmile-v1.0.html`) -- so the shape must +/// keep working and this test still pins OUR behaviour with no claim it is +/// right. What the spec exemplifies is only the DECLARATION; what a REFERENCE +/// like `sq[X,X]` means is the open part, and +/// `vensim-probes/stella_repeated_dimension.stmx` asks Stella. Note the defect +/// is narrower than "repeated dimensions are broken": on that probe Simlin's +/// STORAGE is a correct 2-D array (`SUM(sq[X,*])` gives the true row sums +/// 36/66/96 and `SUM(sq[*,*])` gives 198, both measured); only the subscripted +/// reference collapses. +#[test] +fn a_repeated_dimension_read_directly_is_a_pre_existing_residual() { + let square = |name: &str| { + fixture(name).array_with_ranges( + "square[d,d]", + vec![ + ("1,1", "11"), + ("1,2", "12"), + ("1,3", "13"), + ("2,1", "21"), + ("2,2", "22"), + ("2,3", "23"), + ("3,1", "31"), + ("3,2", "32"), + ("3,3", "33"), + ], + ) + }; + let copy = square("sqdirect_copy").array_aux("out[d,d]", "square[d,d]"); + copy.assert_compiles_incremental(); + assert_close( + ©.vm_result_incremental("out"), + &[11.0, 11.0, 11.0, 22.0, 22.0, 22.0, 33.0, 33.0, 33.0], + "residual: a direct repeated-dimension read projects to [i,i]", + ); + + let sorted = square("sqdirect_sort").array_aux("out[d,d]", "VECTOR SORT ORDER(square[d,d], 1)"); + sorted.assert_compiles_incremental(); + assert_close( + &sorted.vm_result_incremental("out"), + &[0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0], + "residual: the per-row orders are [0,1,2] throughout, not this", + ); + + // The direct read NESTED IN A REDUCER, which is the row that says WHERE the + // refusal above may live. These reach a hoisting site + // (`compiler::replace_nested_builtins_for_element` and its two siblings), + // which sizes the hoisted temp from `find_expr_array_view` and SUBSTITUTES + // the variable's own view when that is `None` -- no diagnostic, and at a + // different size. Putting the repeated-dimension refusal inside + // `compiler::join_array_views` therefore sized these temps at `out3`'s three + // slots while the builtin still wrote nine elements, and the VM indexed past + // the temp: a panic, and under `panic = abort` a dead host process. All four + // return these numbers at the merge base `ccf7ed34`, so that would have been + // a regression, not a newly-refused shape. (The numbers themselves are the + // same first-axis-wins residual as above: nine ranks over the whole square, + // broadcast to every element of `out3`.) + for (name, eqn, expected) in [ + ("sqred_sum", "SUM(VECTOR SORT ORDER(square[d,d], 1))", 9.0), + ("sqred_mean", "MEAN(RANK(square[d,d], 1))", 5.0), + ("sqred_max", "MAX(VECTOR SORT ORDER(square[d,d], 1))", 2.0), + ("sqred_size", "SIZE(VECTOR SORT ORDER(square[d,d], 1))", 9.0), + ] { + let p = square(name).array_aux("out3[d]", eqn); + p.assert_compiles_incremental(); + assert_close( + &p.vm_result_incremental("out3"), + &[expected; 3], + "a repeated-dimension read inside a reducer must keep its merge-base value", + ); + } +} + +/// The two-HOP twin: `main` -> `mid` (twice) -> `inner`. +/// +/// A one-hop fixture cannot distinguish the rule the VM actually uses -- +/// ACCUMULATE `module_off + decl.off` at each `EvalModule` -- from "apply the +/// last hop only" or "re-base from the root at each hop", because at one hop all +/// three agree. `mid` carries a scalar ahead of its module declaration so +/// `inner`'s block does not start at its parent's base, which makes the two +/// hops' offsets distinct non-zero numbers that have to sum. +#[test] +fn an_array_view_inside_a_nested_module_instance_reads_that_instance() { + use crate::db::{SimlinDb, compile_project_incremental, sync_from_datamodel_incremental}; + use crate::vm::Vm; + + let project = crate::test_common::nested_instance_arrayed_submodel_project(); + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + let compiled = compile_project_incremental(&db, sync.project, "main").expect("nested compile"); + let mut vm = Vm::new(compiled).expect("vm"); + vm.run_to_end().expect("run"); + let results = crate::test_common::collect_results(&vm.into_results()); + + for (name, expected) in crate::test_common::nested_instance_arrayed_submodel_expected() { + let actual = results + .get(name) + .unwrap_or_else(|| panic!("no series for {name}")); + assert_close(actual, &expected, name); + } +} diff --git a/src/simlin-engine/src/ast/expr3.rs b/src/simlin-engine/src/ast/expr3.rs index dcdde85d7..641bb6b75 100644 --- a/src/simlin-engine/src/ast/expr3.rs +++ b/src/simlin-engine/src/ast/expr3.rs @@ -936,8 +936,15 @@ impl<'a> Pass1Context<'a> { a_has_a2a || b_has_a2a || c_has_a2a, ) } + // RANK's first argument is an ARRAY the opcode reads as a view, + // exactly like VectorSortOrder's -- so it decomposes through + // `maybe_decompose_array_arg_inner` like all five of its siblings + // below. Recursing with the plain `transform_inner` left + // `RANK(vals[*] * 2, 1)` as an un-viewable `Op2` for codegen to + // reject, while the identical VECTOR SORT ORDER spelling compiled + // (GH #995). Rank(e, direction) => { - let (new_e, e_has_a2a) = self.transform_inner(*e); + let (new_e, e_has_a2a) = self.maybe_decompose_array_arg_inner(*e); let (new_direction, direction_has_a2a) = self.transform_inner(*direction); ( Rank(Box::new(new_e), Box::new(new_direction)), diff --git a/src/simlin-engine/src/builtins_visitor.rs b/src/simlin-engine/src/builtins_visitor.rs index 9b5ea2de0..b31a98bd2 100644 --- a/src/simlin-engine/src/builtins_visitor.rs +++ b/src/simlin-engine/src/builtins_visitor.rs @@ -543,6 +543,61 @@ impl<'a> BuiltinVisitor<'a> { } } + /// Does this subscript index leave a whole dimension standing, rather than + /// selecting one element of it? + /// + /// True for a wildcard or star-range, and for a bare reference to one of the + /// ACTIVE apply-to-all dimensions -- the spelling `context.rs` resolves per + /// element in scalar position and promotes to the whole array inside a + /// vector builtin's array-operand position + /// (`with_vector_builtin_wildcards`). A mapped or otherwise foreign + /// dimension name is deliberately NOT included: those need + /// `substitute_dimension_refs`' positional translation, which is only + /// available here. + fn index_spans_a_dimension(&self, idx: &IndexExpr0) -> bool { + match idx { + IndexExpr0::Wildcard(_) | IndexExpr0::StarRange(_, _) => true, + IndexExpr0::Expr(Expr0::Var(ident, _)) => { + let canonical = CanonicalDimensionName::from_raw(ident.as_str()); + self.dimension_names.iter().any(|d| d == &canonical) + } + _ => false, + } + } + + /// Is `arg` a subscripted reference that is ARRAY-shaped -- one whose + /// indices leave at least one dimension standing, and where every index is + /// either that or statically resolvable (GH #995)? + /// + /// Such an argument is passed through to lowering untouched: no per-element + /// substitution, and no synthesized capture helper. That is what lets an + /// array-valued `PREVIOUS`/`INIT` exist at all, and it puts the decision in + /// the one place that can make it. `PREVIOUS(vals[d])` means the element in + /// `y[d] = PREVIOUS(vals[d])` and the whole array in + /// `y[d] = VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)`, exactly as bare + /// `vals[d]` does -- and only `compiler::context` knows which position it + /// is in. Substituting here would pin it to one element before that context + /// exists; the helper path cannot hold it either, since a scalar + /// `Equation::Scalar` helper holding `vals[*]` does not compile. + /// + /// The scalar routing is unchanged for every other shape: an all-static + /// subscript still substitutes and compiles to `LoadPrev`/`LoadInitial` + /// against a fixed slot, and anything with a dynamic index still gets its + /// capture helper (which is also what gives a dynamic index the correct + /// lagged semantics). + fn arg_is_array_shaped(&self, arg: &Expr0) -> bool { + match arg { + Expr0::Subscript(id, indices, _) => { + !self.is_module_backed_ident(id) + && indices.iter().any(|i| self.index_spans_a_dimension(i)) + && indices + .iter() + .all(|i| self.index_is_static(i) || self.index_spans_a_dimension(i)) + } + _ => false, + } + } + /// Substitute dimension references in the expression with concrete element names. /// For example, if we're processing element "A2" of dimension "SubA", /// transform `input[SubA]` to `input[A2]`. @@ -1162,18 +1217,30 @@ impl<'a> BuiltinVisitor<'a> { // makes their indices statically resolvable); other shapes // keep their original form so behavior is unchanged for // them (`make_temp_arg` substitutes internally, and the - // substitution is idempotent). + // substitution is idempotent). An ARRAY-shaped subscript is + // the exception: substituting would pin it to one element + // before lowering can tell whether the position wants the + // element or the whole array (`arg_is_array_shaped`). let arg0 = match arg0 { - Subscript(_, _, _) if self.active_subscript.is_some() => { + Subscript(_, _, _) + if self.active_subscript.is_some() + && !self.arg_is_array_shaped(&arg0) => + { self.substitute_dimension_refs(arg0) } other => other, }; + // An index that leaves a dimension standing needs no + // helper either: it resolves statically too, just to a VIEW + // over the argument's storage rather than to a single slot + // (codegen's `snapshot_static_view`). let needs_temp_arg = match &arg0 { Var(ident, _) => self.is_module_backed_ident(ident), Subscript(id, indices, _) => { self.is_module_backed_ident(id) - || !indices.iter().all(|idx| self.index_is_static(idx)) + || !indices.iter().all(|idx| { + self.index_is_static(idx) || self.index_spans_a_dimension(idx) + }) } _ => true, }; diff --git a/src/simlin-engine/src/bytecode.rs b/src/simlin-engine/src/bytecode.rs index 56d234b96..e4638f68e 100644 --- a/src/simlin-engine/src/bytecode.rs +++ b/src/simlin-engine/src/bytecode.rs @@ -161,6 +161,36 @@ impl SubdimensionRelation { // Runtime View (for view stack during VM execution) // ============================================================================ +/// Which of the VM's parallel f64 regions a view's elements are read from. +/// +/// `curr`, the `PREVIOUS` snapshot and the `INIT` snapshot are three buffers +/// with **identical slot numbering** (each is an `n_slots` copy of a chunk), so +/// a view's `base_off`/`strides`/`offset` arithmetic is the same for all three +/// and only the backing slice changes. `Temp` is the odd one out: its +/// `base_off` is a temp id, resolved through `ByteCodeContext::temp_offsets`. +/// +/// This replaces an `is_temp: bool`, so that adding the snapshot regions +/// (GH #995) made every dereference site a compile error until it said which +/// region it meant, rather than silently keeping the old two-way split. +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +pub enum ViewStorage { + /// `curr[base_off + ..]` -- this timestep's values. + Curr, + /// `temp_storage[temp_offsets[base_off] + ..]`. + Temp, + /// `prev_values[base_off + ..]` -- the snapshot taken after the previous + /// step's stocks, i.e. what `PREVIOUS()` reads. While no snapshot has been + /// taken yet, every element reads the scalar `PREVIOUS` fallback instead; + /// see [`Opcode::LoadPrev`] and `vm::ViewRegions::read`. + Prev, + /// `initial_values[base_off + ..]` -- the snapshot taken after the initials + /// phase, i.e. what `INIT()` reads. During the initials phase itself the + /// snapshot does not exist yet and the read falls back to `curr`, exactly + /// as [`Opcode::LoadInitial`] does. + Initial, +} + /// Sparse mapping for a single dimension in a RuntimeView. /// Used when iterating over non-contiguous elements. #[derive(Clone, Debug, PartialEq, Eq, Hash)] @@ -176,10 +206,11 @@ pub struct RuntimeSparseMapping { #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq)] pub struct RuntimeView { - /// Base offset: either variable offset in curr[] or temp_id for temps + /// Base offset: a slot offset for the three chunk-shaped regions + /// (`Curr`/`Prev`/`Initial`), a temp id for `Temp` pub base_off: u32, - /// true = base_off is a temp_id, false = base_off is offset in curr[] - pub is_temp: bool, + /// Which region `base_off` addresses + pub storage: ViewStorage, /// Dimension sizes for this view pub dims: SmallVec<[u16; 4]>, /// Strides for each dimension (signed to support transpose) @@ -208,7 +239,7 @@ impl RuntimeView { RuntimeView { base_off, - is_temp: false, + storage: ViewStorage::Curr, dims, strides, offset: 0, @@ -225,7 +256,7 @@ impl RuntimeView { dim_ids: SmallVec<[DimId; 4]>, ) -> Self { let mut view = Self::for_var(temp_id as u32, dims, dim_ids); - view.is_temp = true; + view.storage = ViewStorage::Temp; view } @@ -234,7 +265,7 @@ impl RuntimeView { pub fn invalid() -> Self { RuntimeView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::new(), strides: SmallVec::new(), offset: 0, @@ -1585,10 +1616,11 @@ pub struct ArrayDefinition { #[cfg_attr(feature = "debug-derive", derive(Debug))] #[derive(Clone, PartialEq, Eq, Hash)] pub struct StaticArrayView { - /// Base variable offset in curr[] + /// Base variable offset within the region named by `storage` (a temp id + /// when that is `ViewStorage::Temp`) pub base_off: u32, - /// true = base_off is a temp_id, false = base_off is offset in curr[] - pub is_temp: bool, + /// Which region `base_off` addresses + pub storage: ViewStorage, /// Dimension sizes pub dims: SmallVec<[u16; 4]>, /// Strides for each dimension @@ -1616,15 +1648,26 @@ impl StaticArrayView { /// (non-star-range) view -- the overwhelmingly common case -- so we take a /// free fresh empty `SmallVec` then and only fall back to a real clone for a /// genuinely sparse view. - pub fn to_runtime_view(&self) -> RuntimeView { + pub fn to_runtime_view(&self, module_off: u32) -> RuntimeView { let sparse = if self.sparse.is_empty() { SmallVec::new() } else { self.sparse.clone() }; RuntimeView { - base_off: self.base_off, - is_temp: self.is_temp, + // The three chunk-shaped regions are addressed by the executing + // INSTANCE's slot base, exactly as `Opcode::LoadVar` and + // `Opcode::LoadPrev` are: a fragment's `base_off` comes from its own + // model's layout and is module-relative. A temp id is not -- temp + // storage is per-evaluation, shared by whichever instance is running + // -- so it is the one base the instance offset must NOT touch. + base_off: match self.storage { + ViewStorage::Curr | ViewStorage::Prev | ViewStorage::Initial => { + self.base_off + module_off + } + ViewStorage::Temp => self.base_off, + }, + storage: self.storage, dims: SmallVec::from_slice(&self.dims), strides: SmallVec::from_slice(&self.strides), offset: self.offset, @@ -2582,7 +2625,7 @@ mod tests { let view = RuntimeView::for_var(100, dims, dim_ids); assert_eq!(view.base_off, 100); - assert!(!view.is_temp); + assert_eq!(view.storage, ViewStorage::Curr); assert_eq!(view.dims.as_slice(), &[5]); assert_eq!(view.strides.as_slice(), &[1]); assert_eq!(view.offset, 0); @@ -2622,7 +2665,7 @@ mod tests { let view = RuntimeView::for_temp(3, dims, dim_ids); assert_eq!(view.base_off, 3); - assert!(view.is_temp); + assert_eq!(view.storage, ViewStorage::Temp); } #[test] @@ -3043,7 +3086,7 @@ mod tests { let view = StaticArrayView { base_off: 100, - is_temp: false, + storage: ViewStorage::Curr, dims: smallvec::smallvec![3, 4], strides: smallvec::smallvec![4, 1], offset: 0, @@ -3063,7 +3106,7 @@ mod tests { fn test_static_view_to_runtime() { let static_view = StaticArrayView { base_off: 100, - is_temp: false, + storage: ViewStorage::Curr, dims: smallvec::smallvec![3, 4], strides: smallvec::smallvec![4, 1], offset: 8, @@ -3071,10 +3114,10 @@ mod tests { dim_ids: smallvec::smallvec![0, 1], }; - let runtime = static_view.to_runtime_view(); + let runtime = static_view.to_runtime_view(0); assert_eq!(runtime.base_off, 100); - assert!(!runtime.is_temp); + assert_eq!(runtime.storage, ViewStorage::Curr); assert_eq!(runtime.dims.as_slice(), &[3, 4]); assert_eq!(runtime.strides.as_slice(), &[4, 1]); assert_eq!(runtime.offset, 8); @@ -3216,7 +3259,7 @@ mod tests { // 1D array with non-zero offset (sliced view) let view = RuntimeView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: smallvec::smallvec![3], strides: smallvec::smallvec![1], offset: 5, // Start at element 5 @@ -3239,7 +3282,7 @@ mod tests { // 2D array with column-major strides (not contiguous) let view = RuntimeView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: smallvec::smallvec![3, 4], // 3 rows, 4 cols strides: smallvec::smallvec![1, 3], // Column-major: stride[0]=1, stride[1]=3 offset: 0, @@ -3264,7 +3307,7 @@ mod tests { // 1D sparse array: elements at indices [1, 3, 7] of parent let view = RuntimeView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: smallvec::smallvec![3], // 3 sparse elements strides: smallvec::smallvec![1], offset: 0, @@ -3289,7 +3332,7 @@ mod tests { // Scalar view (0 dimensions) let view = RuntimeView { base_off: 10, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::new(), strides: SmallVec::new(), offset: 5, diff --git a/src/simlin-engine/src/compiler/array_operand.rs b/src/simlin-engine/src/compiler/array_operand.rs new file mode 100644 index 000000000..47de4b59d --- /dev/null +++ b/src/simlin-engine/src/compiler/array_operand.rs @@ -0,0 +1,426 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! Materialization of computed array operands (GH #995). +//! +//! Codegen consumes an array-valued operand as a **view over storage** +//! ([`super::codegen::Compiler::walk_expr_as_view`]): a `StaticSubscript`, a +//! `TempArray`, a whole `Var`, or a dynamic `Subscript`, and nothing else. A +//! *computed* array -- `vals[D] * 2`, `NOT ...`, an `IF` selecting between two +//! arrays, an elementwise `ABS(...)`, a nested array-producing builtin -- is +//! none of those, so it has to be evaluated into a temp of its own before the +//! builtin that reads it. Codegen already knows how to do that: an +//! `AssignTemp` whose body is not one of the array-producing opcodes lowers to +//! a `BeginIter` loop that evaluates the body element by element. +//! +//! [`super::context::Context::lower`]'s Pass 1 +//! ([`crate::ast::Pass1Context`]) materializes the operands it can see, +//! but it works on `Expr3`, *before* subscripts are resolved, so two shapes get +//! past it: +//! +//! * an operand still carrying an unresolved apply-to-all dimension reference. +//! `vals[D]` inside a vector builtin only means "the whole array" after +//! [`super::context::Context::with_vector_builtin_wildcards`] promotes its +//! `ActiveDimRef` to a `Wildcard`, which happens during lowering, after Pass +//! 1; Pass 1 sees an unresolved reference and defers to pass 2. +//! * an operand the *type checker* bounded as a scalar for the same reason: +//! `vals[D] * 2` carries `ArrayBounds: None`, so Pass 1's +//! `needs_decomposition` declines it before the deferral even matters. +//! +//! This pass is the backstop, and it runs on the fully lowered fragment, where +//! the promotion has happened and every view is concrete. +//! +//! # Why it is safe +//! +//! It rewrites **only** operands codegen would have rejected: [`is_view`] is +//! the negation of `walk_expr_as_view`'s accepting arms, so a fragment that +//! compiles today passes through untouched, temp count included. +//! +//! Where it does fire it costs one temp per materialized operand, which on the +//! per-element hoisting path (one temp per array ELEMENT already) doubles temp +//! consumption. That runs into the `u8` `TempId` namespace above ~128 elements +//! -- and a materialized operand is read through a static VIEW, the one place a +//! temp id is not narrowed to `u8`. `symbolic::resolve_static_view` rejects +//! that combination loudly rather than letting the two narrowings disagree; +//! #583 is the real fix. Measured max temps per fragment across the checked-in +//! corpus and C-LEARN: 21, unchanged by this pass. +//! +//! # What still declines +//! +//! Four limits are worth knowing before reading a "this shape does not +//! compile" report as a bug in this pass: +//! +//! * An operand only materializes if [`super::find_expr_array_view`] can +//! derive a shape for it. That function's `App` arm is an exhaustive match +//! naming exactly which builtins propagate an array shape; the ones that do +//! not (the reducers, `VECTOR SELECT`, the `Lookup` family) are listed there +//! with the reason. +//! * That shape is the JOIN of every array in the operand -- the view they all +//! broadcast into -- so an operand mixing INCOMPARABLE shapes (`row[e]` and +//! `col[d]`, neither containing the other) has none, and declines. The union +//! `[e,d]` would compile, but nothing in the operand says whether it is +//! `[e,d]` or `[d,e]`, and the temp's axis order is the axis +//! `VECTOR SORT ORDER` sorts along. Declining leaves the loud codegen +//! rejection; guessing would leave a plausible array of wrong numbers. +//! * An operand carrying a REPEATED-dimension view (`matrix[d,d]`) declines -- +//! mixed with another shape, and as the operand's SOLE shape. `[d,d]` can say +//! "contains `d` at size 3" but not WHICH `d`, and every layer that projects +//! between an array and a temp matches by name and takes the first hit: +//! `super::project_var_index_to_temp` gives both axes the same coordinate, so +//! `out[i,j]` would read `temp[i,i]`, and `codegen::array_view_to_static_temp` +//! keys `DimId`s the same way. There is no shape to give. +//! +//! The refusal lives HERE and in `codegen::snapshot_static_view` -- the two +//! positions that can be loud about it -- and deliberately not inside +//! `super::join_array_views`, even though that reads as the tidier home for +//! it. `super::find_expr_array_view` has four consumers and the other three +//! substitute the VARIABLE's own view for a `None`, silently and at a +//! possibly different SIZE: refusing in the join sized the temp of +//! `out[d] = SUM(VECTOR SORT ORDER(matrix[d,d], 1))` at `out`'s three slots +//! while the sort order still wrote nine, and the VM indexed past the temp. +//! That equation returns numbers at the merge base, so the tidier home cost a +//! process abort on a shape that worked. +//! +//! This costs nothing that worked: measured at the MERGE BASE `ccf7ed34`, +//! `VECTOR SORT ORDER(matrix[d,d] * 2, 1)` does not compile, and neither does +//! the `PREVIOUS`/`INIT` spelling `codegen::snapshot_static_view` refuses on +//! the same grounds. Both became compilable on this branch, and both compiled +//! to first-axis-wins garbage until this refusal. Reading a repeated dimension +//! DIRECTLY is a different matter and is untouched: `out[d,d] = matrix[d,d]` +//! and `VECTOR SORT ORDER(matrix[d,d], 1)` compile at the merge base, to those +//! same wrong numbers, and remain exactly as they were -- a disclosed residual +//! whose blast radius is now MEASURED: Vensim REJECTS the declaration -- run in Vensim DSS 2026-08-04, `vensim-probes/repeated_dimension.mdl` refuses to simulate with "DimA appears more than once on LHS" -- so no MDL-imported model can contain this shape and the residual is confined to hand-authored XMILE/JSON/protobuf. It is NOT illegitimate, though: the XMILE v1.0 spec exemplifies the declaration (`docs/reference/xmile-v1.0.html`, "A 2D non-apply-to-all array with dimensions X by X, where X is size 2", verified in-repo), so a conformant file may carry it and Simlin must keep reading it. The spec exemplifies only the DECLARATION, with per-element equations; it says nothing about what a REFERENCE such as `sq[X,X]` means, which is the part that is wrong here. Pinned by +//! `array_operand_materialization_tests::a_repeated_dimension_read_directly_is_a_pre_existing_residual`, +//! whose fix belongs in the projection rather than here. + +use crate::ast::ArrayView; +use crate::compiler::expr::{BuiltinFn, Expr, SubscriptIndex}; + +/// Rewrite `exprs` so every array operand codegen reads as a view actually is +/// one, splicing an `AssignTemp` in front of the expression that needs it. +/// +/// Temp ids continue past the highest one the fragment already uses, so the +/// new temps cannot collide with the per-element ids the apply-to-all hoister +/// assigns (which restart at 0 for each `lower()` call and are remapped there). +pub(super) fn materialize_computed_array_operands(exprs: Vec) -> Vec { + let mut next_temp_id = super::next_available_temp_id(&exprs); + let mut out: Vec = Vec::with_capacity(exprs.len()); + for expr in exprs { + // Hoisted assignments are emitted immediately before the expression + // that reads them, and in the order they were allocated, so a nested + // materialization's temp is always written before the outer one that + // consumes it. + let mut hoisted = Vec::new(); + let expr = rewrite(expr, &mut next_temp_id, &mut hoisted); + out.extend(hoisted); + out.push(expr); + } + out +} + +fn rewrite(expr: Expr, next_temp_id: &mut u32, hoisted: &mut Vec) -> Expr { + match expr { + Expr::App(builtin, loc) => { + // Bottom-up: an operand that is itself a builtin call is rewritten + // (and, if it needs it, materialized) before this level looks at + // it, so a nested array-producing builtin arrives here as a + // `TempArray` rather than as an un-viewable `App`. + let builtin = builtin.map(|arg| rewrite(arg, next_temp_id, hoisted)); + Expr::App( + materialize_view_operands(builtin, next_temp_id, hoisted), + loc, + ) + } + Expr::Op1(op, inner, loc) => { + Expr::Op1(op, Box::new(rewrite(*inner, next_temp_id, hoisted)), loc) + } + Expr::Op2(op, lhs, rhs, loc) => Expr::Op2( + op, + Box::new(rewrite(*lhs, next_temp_id, hoisted)), + Box::new(rewrite(*rhs, next_temp_id, hoisted)), + loc, + ), + Expr::If(cond, then_expr, else_expr, loc) => Expr::If( + Box::new(rewrite(*cond, next_temp_id, hoisted)), + Box::new(rewrite(*then_expr, next_temp_id, hoisted)), + Box::new(rewrite(*else_expr, next_temp_id, hoisted)), + loc, + ), + Expr::Subscript(base, indices, bounds, loc) => { + let indices = indices + .into_iter() + .map(|idx| match idx { + SubscriptIndex::Single(e) => { + SubscriptIndex::Single(rewrite(e, next_temp_id, hoisted)) + } + SubscriptIndex::Range(start, end) => SubscriptIndex::Range( + rewrite(start, next_temp_id, hoisted), + rewrite(end, next_temp_id, hoisted), + ), + }) + .collect(); + Expr::Subscript(base, indices, bounds, loc) + } + Expr::EvalModule(ident, model_name, input_set, args) => Expr::EvalModule( + ident, + model_name, + input_set, + args.into_iter() + .map(|arg| rewrite(arg, next_temp_id, hoisted)) + .collect(), + ), + Expr::AssignCurr(dst, rhs) => { + Expr::AssignCurr(dst, Box::new(rewrite(*rhs, next_temp_id, hoisted))) + } + Expr::AssignNext(dst, rhs) => { + Expr::AssignNext(dst, Box::new(rewrite(*rhs, next_temp_id, hoisted))) + } + Expr::AssignTemp(id, rhs, view) => { + Expr::AssignTemp(id, Box::new(rewrite(*rhs, next_temp_id, hoisted)), view) + } + leaf @ (Expr::Const(_, _) + | Expr::Var(_, _) + | Expr::StaticSubscript(_, _, _) + | Expr::TempArray(_, _, _) + | Expr::TempArrayElement(_, _, _, _) + | Expr::Dt(_) + | Expr::ModuleInput(_, _)) => leaf, + } +} + +/// The single enumeration of view-requiring operand positions, derived from +/// codegen's `walk_expr_as_view` call sites. Written as an exhaustive match +/// with no `_` arm so a new `BuiltinFn` variant is a compile error here rather +/// than a silently unconsidered position. +fn materialize_view_operands( + builtin: BuiltinFn, + next_temp_id: &mut u32, + hoisted: &mut Vec, +) -> BuiltinFn { + use crate::builtins::BuiltinFn::*; + let mut mat = |arg: Box| materialize_view_operand(arg, next_temp_id, hoisted); + match builtin { + // `emit_array_reduce`: pushes the argument as a view unconditionally. + Sum(a) => Sum(mat(a)), + Size(a) => Size(mat(a)), + Stddev(a) => Stddev(mat(a)), + // One-argument MIN/MAX are the array reductions; the two-argument + // forms are scalar and take no view. + Min(a, b) => match b { + None => Min(mat(a), None), + Some(b) => Min(a, Some(b)), + }, + Max(a, b) => match b { + None => Max(mat(a), None), + Some(b) => Max(a, Some(b)), + }, + + // The scalar-reducing selector and the five array-producing opcodes. + VectorSelect(sel, values, max_val, action, err) => { + VectorSelect(mat(sel), mat(values), max_val, action, err) + } + // Materializing an ELM MAP *source* deliberately changes which storage + // the mapping ranges over, and the choice is this: the temp. + // + // The rule for a VARIABLE source is DOCUMENTED and ground-truthed. The + // Vensim reference page for VECTOR ELM MAP (retrieved 2026-08-02) says + // the function "returns the value of the variable that is offset from + // vec by the specified amount", and that an offset "outside the range of + // the variable" yields `:NA:`; its multi-subscript example spells the + // offset as a flat index over the whole variable + // (`(sub-1)*ELMCOUNT(tub)*ELMCOUNT(gub) + ...`). Real Vensim output + // agrees: in `test/sdeverywhere/models/vector/`, + // `f[DimA,DimB] = VECTOR ELM MAP(d[DimA,B1], a[DimA])` prints + // `1,1,5,5,6,6`, and `f[A2,B1] = 5 = d[A2,B2]` -- the mapping read past + // its own `B1` slice into the next row. `vm_vector_elm_map.rs` + // implements exactly that with a `source_is_full_array` test: a strict + // slice keeps a per-element base and can read across rows, a full + // contiguous source has `base_i == 0`. + // + // A COMPUTED source is a Simlin EXTENSION, and it is now settled that + // it is one. Vensim rejects the shape outright -- run in Vensim DSS on + // 2026-08-04, `vensim-probes/elm_map_computed_source.mdl` refuses to + // simulate with "Argument 1 to function VECTOR ELM MAP must be a normal + // variable". So there is no Vensim behaviour to match here, and the + // question is not "which rule does Vensim use" but "what shall this mean + // in Simlin". + // + // It means the HELPER-EQUIVALENT thing: an inline expression behaves + // exactly as the same values pre-assigned to a named variable, which is + // the spelling that IS legal Vensim. A materialized operand is a fresh + // contiguous temp and so is full-array by construction, which confines + // the mapping to the computed array -- exactly what + // `VECTOR ELM MAP(helper[A1], offs)` does when `helper` holds those + // values. That definition is deliberate and no longer provisional; the + // temp has no "rest of the variable" to run into, so nothing else is + // even expressible. Pinned by + // `array_operand_materialization_tests::materializing_an_elm_map_source_confines_the_mapping_to_the_temp`. + VectorElmMap(source, offsets) => VectorElmMap(mat(source), mat(offsets)), + VectorSortOrder(array, direction) => VectorSortOrder(mat(array), direction), + Rank(array, direction) => Rank(mat(array), direction), + // ALLOCATE AVAILABLE's priority-profile argument is deliberately NOT + // materialized. Its view is rewritten during lowering by + // `context::Context::expand_pp_view_for_allocate`, which re-expands a + // collapsed reference such as `pp[D,1]` back to the variable's full + // requester x XPriority array because the allocator always reads all + // four profile columns. That helper only understands a direct variable + // reference, so a computed profile array has no defined shape here: + // materializing `pp[D,1] + adj[D,1]` would silently hand the VM a + // one-column-per-requester temp. Leaving it alone keeps the loud + // codegen rejection instead. + AllocateAvailable(requests, profiles, available) => { + AllocateAvailable(mat(requests), profiles, available) + } + AllocateByPriority(requests, priorities, size, width, supply) => { + AllocateByPriority(mat(requests), mat(priorities), size, width, supply) + } + + // An arrayed graphical-function apply reads its table as a view, but + // the table must name a whole *variable*: codegen resolves it to a + // `base_gf` by ident (`arrayed_lookup_table_info`), and a temp has no + // graphical functions attached to it. + Lookup(_, _, _) | LookupForward(_, _, _) | LookupBackward(_, _, _) => builtin, + // MEAN is the one reduce that is variadic. Only its single-argument + // form is an array reduction and only that form reaches + // `emit_array_reduce`; the multi-argument form averages scalars and + // has no view position at all. Codegen's `Mean` arm matches the four + // view shapes and emits a plain scalar `walk_expr` otherwise -- which + // is right for `MEAN(a * b)` over two scalars, and is why a + // scalar-shaped argument must keep passing through untouched. That + // fallback is NOT a licence to leave an array-shaped argument alone: + // `MEAN(matrix[E,*] * 2)` reaches the fallback, emits a scalar walk + // over an array expression, and fails to compile. `mat` declines + // anything with no derivable array view, so the scalar form is + // unaffected and the array form now agrees with every other reducer. + Mean(args) => { + if args.len() == 1 { + Mean(args.into_iter().map(|a| *mat(Box::new(a))).collect()) + } else { + Mean(args) + } + } + + // No view-requiring operand. + other @ (Abs(_) + | Arccos(_) + | Arcsin(_) + | Arctan(_) + | Cos(_) + | Exp(_) + | Inf + | Int(_) + | IsModuleInput(_, _) + | Ln(_) + | Log10(_) + | Pi + | Pulse(_, _, _) + | Quantum(_, _) + | Ramp(_, _, _) + | SafeDiv(_, _, _) + | Sign(_) + | Sshape(_, _, _) + | Sin(_) + | Sqrt(_) + | Step(_, _) + | Tan(_) + | Time + | TimeStep + | StartTime + | FinalTime + | Previous(_, _) + | Init(_)) => other, + } +} + +/// Move `operand` into a temp of its own and return the `TempArray` reference +/// that replaces it, or return it unchanged when it is already a view or when +/// no array shape can be derived for it. +/// +/// A bare array-valued `PREVIOUS`/`INIT` is left alone for the same reason a +/// `StaticSubscript` is: it already IS a view, over a snapshot buffer rather +/// than over `curr` (GH #995, [`is_snapshot_view`]). Materializing it would +/// spend a temp to copy an array that codegen can address directly. +fn materialize_view_operand( + operand: Box, + next_temp_id: &mut u32, + hoisted: &mut Vec, +) -> Box { + if is_view(&operand) { + return operand; + } + if is_snapshot_view(&operand) { + return operand; + } + // The operand's shape is the JOIN of its subexpressions' shapes, which is + // what makes `small[d] + wide[e,d]` and `wide[e,d] + small[d]` the same + // array (`super::find_expr_array_view`). `None` covers both "no shape" and + // "two shapes neither of which contains the other"; declining leaves the + // operand for codegen to reject, exactly as the two deliberately + // unmaterialized positions above do, rather than guessing an axis order. + let Some(source_view) = super::find_expr_array_view(&operand) else { + return operand; + }; + // A repeated dimension name is refused as a TEMP's shape even when it is the + // operand's sole shape (`super::view_repeats_a_dimension`). The refusal lives + // here rather than inside the join because `find_expr_array_view` has four + // consumers and only this one is loud: the three hoisters in `super` fall + // back to the VARIABLE's own view, so a `None` there silently reshapes a + // temp instead of declining -- measured, `out[d] = SUM(VECTOR SORT + // ORDER(matrix[d,d], 1))` sized a 9-element sort order's temp at 3 and the + // VM indexed past it. Refusing at the one site that produces a diagnostic + // keeps the direct spellings byte-identical to the merge base, which is all + // this refusal ever claimed. + if super::view_repeats_a_dimension(&source_view) { + return operand; + } + if source_view.dims.is_empty() { + return operand; + } + + // The temp is fresh compact storage; only the SHAPE of the producing + // expression carries over. (Codegen normalizes a temp's view to compact + // row-major strides anyway -- `array_view_to_static_temp` -- so passing a + // sliced source view would merely be misleading, not wrong.) + let view = if source_view.dim_names.len() == source_view.dims.len() { + ArrayView::contiguous_with_names(source_view.dims, source_view.dim_names) + } else { + ArrayView::contiguous(source_view.dims) + }; + + let loc = operand.get_loc(); + let temp_id = *next_temp_id; + *next_temp_id += 1; + hoisted.push(Expr::AssignTemp(temp_id, operand, view.clone())); + Box::new(Expr::TempArray(temp_id, view, loc)) +} + +/// True when `expr` is an array-valued `PREVIOUS`/`INIT` -- a fifth view shape +/// `codegen::walk_expr_as_view` accepts, over one of the VM's snapshot buffers +/// rather than over `curr` (GH #995). +/// +/// Kept separate from [`is_view`] because the two are decided differently: +/// `is_view` is a pure shape test on the `Expr` variant, while this one depends +/// on the ARGUMENT's shape -- `PREVIOUS(vals[D])` is a view and +/// `PREVIOUS(matrix[E,1])` is a scalar. Both are decided by +/// [`super::snapshot_view_arg`], shared with codegen, so the pass and the +/// emitter cannot disagree about which calls take the array route. An argument +/// codegen cannot express as a snapshot view is a loud rejection THERE, and +/// declining to materialize here is what keeps it loud: a temp built around a +/// `PREVIOUS` its `BeginIter` body cannot emit would be a wrong number in place +/// of a diagnostic. +fn is_snapshot_view(expr: &Expr) -> bool { + matches!(expr, Expr::App(builtin, _) if super::snapshot_view_arg(builtin).is_some()) +} + +/// The four expression shapes `codegen::walk_expr_as_view` accepts. Anything +/// else is a codegen error, which is exactly the set this pass rewrites. +fn is_view(expr: &Expr) -> bool { + matches!( + expr, + Expr::StaticSubscript(_, _, _) + | Expr::TempArray(_, _, _) + | Expr::Var(_, _) + | Expr::Subscript(_, _, _, _) + ) +} diff --git a/src/simlin-engine/src/compiler/codegen.rs b/src/simlin-engine/src/compiler/codegen.rs index 5534629ab..9f03a405c 100644 --- a/src/simlin-engine/src/compiler/codegen.rs +++ b/src/simlin-engine/src/compiler/codegen.rs @@ -80,6 +80,96 @@ impl<'a> ModuleCtx<'a> { } } +/// Where a `PREVIOUS`/`INIT` call sits, which decides how permissive the +/// snapshot-view route is (`Compiler::snapshot_static_view`). +#[derive(Clone, Copy, PartialEq, Eq)] +enum SnapshotPosition { + /// The array operand of a builtin: `walk_expr_as_view` is emitting a view + /// here, so a view is what the call must produce. + ViewOperand, + /// A per-element read inside a `BeginIter` body: a scalar position, where + /// only an array-valued call needs the view route. + IterationBody, +} + +/// The single slot a `LoadPrev`/`LoadInitial` can address, if `arg` resolved to +/// one: a scalar variable (`Expr::Var`), or an array reference whose subscripts +/// collapsed the view to one element (`arr[Dim.elem]`, `arr[2]`). +/// +/// This is the ARRAY route's negation as well as the scalar route's predicate +/// (`Compiler::snapshot_static_view`), so the two partition the argument shapes. +fn static_slot(arg: &Expr) -> Option { + match arg { + Expr::Var(var, _) => Some(var.clone()), + Expr::StaticSubscript(base, view, _) if view.dims.is_empty() => { + Some(base.offset_by(view.offset)) + } + _ => None, + } +} + +/// `ALLOCATE AVAILABLE`'s priority-profile position refuses a `PREVIOUS`/`INIT` +/// (GH #995 phase C3), for the same reason it refuses a computed profile. +/// +/// The allocator reads ALL of a requester's XPriority columns, but the Vensim +/// convention writes the argument collapsed (`pp[D,1]` -- "the priority vector +/// starting at column 1"). `context::expand_pp_view_for_allocate` is what +/// re-expands that back to the variable's full requester x XPriority array, and +/// it only understands a direct variable reference: everything else falls +/// through its `_ => Ok(lowered)` arm untouched. Before C3 that was harmless, +/// because a `PREVIOUS` of an array did not compile at all and the position's +/// only reachable non-reference shape (a computed profile) was rejected by +/// `walk_expr_as_view`. Once `PREVIOUS(pp[D,1])` became a legal view, it started +/// compiling to a ONE-COLUMN-per-requester profile and the allocator bisected +/// over it -- a silently wrong allocation where HEAD had a loud failure, which +/// is strictly worse. Rejected here instead. +/// +/// Declining rather than fixing is the proportionate move: it restores exactly +/// what HEAD did for this position. Making it CORRECT is option (b) -- teach +/// `expand_pp_view_for_allocate` to look through a `PREVIOUS`/`INIT`, rebuild +/// the full-variable view underneath it, and re-wrap -- which is a lowering +/// change with its own allocator-semantics question (whether a frozen profile +/// should freeze the whole array or only the referenced column) and belongs +/// with a fixture that can tell the two apart. The workaround needs no engine +/// change at all: capture the profile into a variable of its own +/// (`frozen[D,XP] = PREVIOUS(pp[D,XP])`) and pass `frozen[D,1]`, which is a +/// direct reference the expander does understand. +fn reject_snapshot_priority_profile(profile: &Expr) -> Result<()> { + let is_snapshot = matches!( + profile, + Expr::App(BuiltinFn::Previous(_, _) | BuiltinFn::Init(_), _) + ); + if is_snapshot { + return sim_err!( + NotSimulatable, + "ALLOCATE AVAILABLE reads every priority column, and its profile \ + argument is re-expanded to the whole array from a direct variable \ + reference -- a PREVIOUS/INIT there would allocate over one column. \ + Capture the frozen profile in a variable of its own and pass that." + .to_string() + ); + } + Ok(()) +} + +/// Is this `PREVIOUS` fallback the default the unary spelling desugars to? +/// +/// `builtins_visitor` rewrites `PREVIOUS(x)` to `PREVIOUS(x, 0)`, and the array +/// route can only reproduce that one value (see +/// `Compiler::snapshot_static_view`). +/// +/// Compared by BIT PATTERN, which is not a formality here: `1 / PREVIOUS(x, 0)` +/// and `1 / PREVIOUS(x, -0)` differ in the sign of the infinity they yield on +/// the first step, and a `-0.0` fallback IS reachable -- not as the literal +/// `-0`, which is a negation of `0` that constant folding turns back into +/// `+0.0`, but as `0 * -1`, the shape `compiler::fold` is documented to produce. +/// See the float-equality position on [`crate::ast::Literal`]; both spellings +/// are pinned by +/// `array_operand_materialization_tests::a_non_default_array_previous_fallback_declines_loudly`. +fn is_default_previous_fallback(fallback: &Expr) -> bool { + matches!(fallback, Expr::Const(value, _) if value.to_bits() == 0.0f64.to_bits()) +} + pub(super) struct Compiler<'module> { module: ModuleCtx<'module>, module_decls: Vec, @@ -332,6 +422,15 @@ impl<'module> Compiler<'module> { } Expr::Var(base, _) => (Some(base), 1usize), Expr::TempArray(_, view, _) => (None, view.dims.iter().product::().max(1)), + // An array-valued `PREVIOUS`/`INIT` source (GH #995) is a view over + // the SAME variable's storage in a snapshot buffer, so its full + // extent is its argument's -- the snapshot regions are `n_slots` + // copies of a chunk. Without this arm the source fell to the `_` + // case and reported an extent of 1, and every mapped offset but 0 + // was reported out of range (`:NA:`). + Expr::App(BuiltinFn::Previous(arg, _), _) | Expr::App(BuiltinFn::Init(arg), _) => { + return self.full_source_len(arg); + } _ => (None, 1usize), }; @@ -343,8 +442,34 @@ impl<'module> Compiler<'module> { view_len as u32 } - /// Convert an ArrayView to a SymbolicStaticView for a variable + /// Convert an ArrayView to a SymbolicStaticView reading a variable out of + /// `curr`. fn array_view_to_static(&mut self, base: &VarRef, view: &ArrayView) -> SymbolicStaticView { + self.array_view_to_static_in(SymStaticViewBase::Var(base.clone()), view) + } + + /// The same view geometry over one of the snapshot regions (GH #995): an + /// array-valued `PREVIOUS`/`INIT` is the argument's view read out of + /// `prev_values` / `initial_values`, which share `curr`'s slot numbering, + /// so only the base tag changes. + fn array_view_to_snapshot_static( + &mut self, + base: &VarRef, + view: &ArrayView, + storage: super::SnapshotRegion, + ) -> SymbolicStaticView { + let base = match storage { + super::SnapshotRegion::Prev => SymStaticViewBase::PrevVar(base.clone()), + super::SnapshotRegion::Initial => SymStaticViewBase::InitialVar(base.clone()), + }; + self.array_view_to_static_in(base, view) + } + + fn array_view_to_static_in( + &mut self, + base: SymStaticViewBase, + view: &ArrayView, + ) -> SymbolicStaticView { // Convert sparse info let sparse: SmallVec<[RuntimeSparseMapping; 2]> = view .sparse @@ -371,7 +496,7 @@ impl<'module> Compiler<'module> { .collect(); SymbolicStaticView { - base: SymStaticViewBase::Var(base.clone()), + base, dims: view.dims.iter().map(|&d| d as u16).collect(), strides: view.strides.iter().map(|&s| s as i32).collect(), offset: view.offset as u32, @@ -479,9 +604,164 @@ impl<'module> Compiler<'module> { Ok((base_gf, table_count)) } + /// Build the snapshot-region static view for an **array-valued** + /// `PREVIOUS`/`INIT` (GH #995), or `Ok(None)` when `builtin` is not one, or + /// is the ordinary scalar form that compiles to `LoadPrev`/`LoadInitial` + /// against a single slot. + /// + /// This is the single place the array route is decided, so the three + /// consumers -- `walk_expr_as_view` (bare operand), `walk_expr` (inside a + /// `BeginIter` body) and `collect_iter_source_views_impl` (which must + /// pre-push exactly the views the body reads) -- cannot disagree about + /// which shapes take it. + /// + /// A shape the region view cannot express is a loud `Err`, never a silent + /// fall-through to the scalar route: reading one element's snapshot and + /// broadcasting it where an array was written is a plausible array of wrong + /// numbers, which is strictly worse than not compiling. + fn snapshot_static_view( + &mut self, + builtin: &BuiltinFn, + position: SnapshotPosition, + ) -> Result> { + let (arg, region) = match position { + // The caller has already said an array is required here, so every + // argument that lowered to a view takes the snapshot route -- + // including one that collapsed to a SINGLE element. That is what + // makes `VECTOR ELM MAP(PREVIOUS(vals[1]), offs)` behave exactly as + // `VECTOR ELM MAP(vals[1], offs)` does: the element establishes the + // base and the mapping ranges over the whole source variable. A + // `PREVIOUS` that behaved differently from its own argument in the + // same position would be the anomaly, degenerate answers included -- + // and degenerate is what a one-element view means in a RANK-LIKE + // position: `VECTOR SORT ORDER(PREVIOUS(vals[1]), 1)` is a constant + // 0 sort order and `RANK(...)` a constant 1, exactly what + // `VECTOR SORT ORDER(vals[1], 1)` already produced at HEAD. GH #995 + // warns against making that shape compile, and the warning is about + // the LTM wrap pinning a rank-like builtin's ARGUMENT to one element + // (a loud drop becoming a plausible constant-0 score); that half is + // untouched -- `ltm_agg`'s rank-like decline does not key on + // compilability, and C-LEARN's five `rank-like-partial` declines are + // byte-identical across this change. Pinned by + // `array_operand_materialization_tests::an_element_collapsed_snapshot_in_a_rank_like_position_matches_its_curr_twin`. + // + // The NUMERIC index is what the sentence above is about. The same + // element spelled with its bare NAME (`vals[e1]`) never reaches here: + // `builtins_visitor::index_is_static` will not accept an unqualified + // element name on the user-equation parse path, so `PREVIOUS` reads a + // scalar capture helper whose extent is ONE and the mapping is + // confined to it -- measured, and pinned by + // `array_operand_materialization_tests::every_row_of_the_issue_995_table_compiles`. + SnapshotPosition::ViewOperand => match builtin { + BuiltinFn::Previous(arg, _) => (arg.as_ref(), super::SnapshotRegion::Prev), + BuiltinFn::Init(arg) => (arg.as_ref(), super::SnapshotRegion::Initial), + _ => return Ok(None), + }, + // A SCALAR position (a `BeginIter` body element read): only an + // array-valued call takes the view route. A single-element argument + // is a slot and keeps compiling to `LoadPrev`/`LoadInitial`, which + // is what lets `PREVIOUS(matrix[E,1])` go on broadcasting. + SnapshotPosition::IterationBody => match super::snapshot_view_arg(builtin) { + Some(pair) => pair, + None => return Ok(None), + }, + }; + let fallback = match builtin { + BuiltinFn::Previous(_, fallback) => Some(fallback.as_ref()), + _ => None, + }; + + // A `PREVIOUS` fallback is per-call-site scalar state and a view carries + // none. Before the first snapshot exists, a snapshot view yields 0 for + // every element (`vm::ChunkRegions::backing`, and the wasm backend's + // `select` on the same flag) -- which IS the default fallback the unary + // spelling desugars to, so that case needs nothing. Any other fallback + // would be silently ignored on the first step of every run, so reject it + // rather than approximate it. + if let Some(fallback) = fallback + && !is_default_previous_fallback(fallback) + { + return sim_err!( + NotSimulatable, + "an array-valued PREVIOUS has nowhere to carry a fallback, so it \ + can only take the default of 0; give the fallback to a scalar \ + PREVIOUS, or capture the array in a variable of its own first" + .to_string() + ); + } + + match arg { + Expr::StaticSubscript(base, view, _) if super::view_repeats_a_dimension(view) => { + // A source naming one dimension twice (`matrix[d,d]`) has no + // usable projection between the array and the consumer of this + // view -- every layer that does it matches by dimension NAME and + // takes the first hit, so `out[i,j]` reads element `[i,i]` (see + // `compiler::view_repeats_a_dimension`). The array route is new + // with GH #995: `VECTOR SORT ORDER(PREVIOUS(matrix[d,d]), 1)` + // did not compile at the merge base, and letting it compile here + // buys a plausible array of wrong numbers. Refuse it, exactly as + // the temp and non-view arms below do. The DIRECT spelling + // (`VECTOR SORT ORDER(matrix[d,d], 1)`) is untouched: it + // compiles at the merge base, to those same wrong numbers, and + // fixing that is a pre-existing defect in the projection rather + // than something to bolt onto this route. + sim_err!( + NotSimulatable, + "PREVIOUS/INIT of an array that names one dimension twice \ + cannot be read as an array: the element-to-temp projection \ + matches dimensions by name and cannot tell the two apart" + .to_string() + ) + } + Expr::StaticSubscript(base, view, _) => { + Ok(Some(self.array_view_to_snapshot_static(base, view, region))) + } + // A bare variable reference in a view position, mirroring + // `walk_expr_as_view`'s own `Expr::Var` arm: a one-element view. + // Unreachable from `IterationBody`, whose classifier calls it scalar. + Expr::Var(base, _) => { + let view = ArrayView::contiguous(vec![1]); + Ok(Some( + self.array_view_to_snapshot_static(base, &view, region), + )) + } + // A temp has no snapshot: nothing copies `temp_storage` into + // `prev_values`, so a computed array's previous value is simply not + // recorded anywhere. Everything else here is an expression that + // `find_expr_array_view` gave a shape but that did not lower to a + // view over storage -- there is no snapshot of an expression either. + // Both are loud rather than approximated. + Expr::TempArray(_, _, _) => sim_err!( + NotSimulatable, + "PREVIOUS/INIT of a computed array has no snapshot to read: only \ + a stored variable's values are captured each step" + .to_string() + ), + other => sim_err!( + NotSimulatable, + format!( + "PREVIOUS/INIT used where an array is required needs a \ + statically resolvable array reference, got {:?}", + std::mem::discriminant(other) + ) + ), + } + } + /// Emit bytecode to push an expression's view onto the view stack. /// This is used for array operations that need to iterate over arrays. fn walk_expr_as_view(&mut self, expr: &Expr) -> Result<()> { + // An array-valued `PREVIOUS`/`INIT` is its argument's view read out of + // a snapshot region (GH #995), so it is a view position like any other + // rather than a computed array that must be materialized first. + if let Expr::App(builtin, _) = expr + && let Some(static_view) = + self.snapshot_static_view(builtin, SnapshotPosition::ViewOperand)? + { + let view_id = self.add_static_view(static_view); + self.push(SymbolicOpcode::PushStaticView { view_id }); + return Ok(()); + } match expr { Expr::StaticSubscript(base, view, _) => { // Create a static view and push it @@ -919,18 +1199,35 @@ impl<'module> Compiler<'module> { // scalar -- e.g. `arr[Dim.elem]` or `arr[2]`). The latter is // what the builtins-visitor lets through instead of // synthesizing a helper aux when every subscript index is a - // compile-time constant. Anything else (dynamic indices, - // expressions) was rewritten through a helper aux at parse - // time, so reaching here with one is a compiler bug. - let static_slot = |arg: &Expr| -> Option { - match arg { - Expr::Var(var, _) => Some(var.clone()), - Expr::StaticSubscript(base, view, _) if view.dims.is_empty() => { - Some(base.offset_by(view.offset)) - } - _ => None, + // compile-time constant (`static_slot`). + // + // An ARRAY-valued argument takes the other route (GH #995): + // `snapshot_static_view` turns it into a view over the same + // snapshot buffer the opcodes read, which inside a `BeginIter` + // body is one of the pre-pushed source views and is loaded per + // element like any other array operand. Outside an iteration + // there is no per-element context to load into, so that shape is + // reachable only through `walk_expr_as_view`. + if let Some(static_view) = + self.snapshot_static_view(builtin, SnapshotPosition::IterationBody)? + { + if !self.in_iteration { + return sim_err!( + NotSimulatable, + "an array-valued PREVIOUS/INIT is only meaningful where an \ + array is expected, not as a scalar operand" + .to_string() + ); } - }; + let offset = self.find_iter_view_offset(&static_view).unwrap_or_else(|| { + unreachable!( + "snapshot view not found in pre-pushed set - \ + collect_iter_source_views_impl and walk_expr should visit same nodes" + ) + }); + self.push(SymbolicOpcode::LoadIterViewAt { offset }); + return Ok(Some(())); + } match builtin { BuiltinFn::Previous(arg, fallback) => { self.walk_expr(fallback)?.unwrap(); @@ -1105,19 +1402,24 @@ impl<'module> Compiler<'module> { // builtins which take Box. Single-arg MEAN can receive // scalar expressions (Op2, etc.) that walk_expr_as_view // can't handle, so we match on expression type first. - match &args[0] { + // The five shapes `walk_expr_as_view` accepts: the + // four storage views, plus an array-valued + // `PREVIOUS`/`INIT` (a view over a snapshot buffer, + // GH #995). Anything else is a genuine scalar + // expression and averages as one. + let is_view = matches!( + &args[0], Expr::StaticSubscript(..) - | Expr::TempArray(..) - | Expr::Var(..) - | Expr::Subscript(..) => { - return self - .emit_array_reduce(&args[0], SymbolicOpcode::ArrayMean {}); - } - _ => { - self.walk_expr(&args[0])?.unwrap(); - return Ok(Some(())); - } + | Expr::TempArray(..) + | Expr::Var(..) + | Expr::Subscript(..) + ) || matches!(&args[0], Expr::App(b, _) if super::snapshot_view_arg(b).is_some()); + if is_view { + return self + .emit_array_reduce(&args[0], SymbolicOpcode::ArrayMean {}); } + self.walk_expr(&args[0])?.unwrap(); + return Ok(Some(())); } // Multi-argument scalar mean: (arg1 + arg2 + ... + argN) / N @@ -1424,6 +1726,7 @@ impl<'module> Compiler<'module> { return Ok(None); } BuiltinFn::AllocateAvailable(requests, profile, avail) => { + reject_snapshot_priority_profile(profile)?; self.walk_expr_as_view(requests)?; self.walk_expr_as_view(profile)?; self.walk_expr(avail)?.unwrap(); @@ -1600,8 +1903,22 @@ impl<'module> Compiler<'module> { self.collect_iter_source_views_impl(else_expr, views, seen); } Expr::App(builtin, _) => { - // Recurse into all arguments of the builtin function - self.collect_builtin_views(builtin, views, seen); + // An array-valued `PREVIOUS`/`INIT` reads its argument's view + // out of a snapshot region rather than out of `curr`, so it + // contributes THAT view and its argument is not walked -- the + // `curr` view of the same reference is a different source and + // would be pushed for nothing. A shape the region view cannot + // express contributes nothing here and surfaces as the + // propagated `Err` when `walk_expr` reaches the same node. + match self.snapshot_static_view(builtin, SnapshotPosition::IterationBody) { + Ok(Some(static_view)) => { + if seen.insert(static_view.clone()) { + views.push(static_view); + } + } + Ok(None) => self.collect_builtin_views(builtin, views, seen), + Err(_) => {} + } } // Leaf expressions that don't contain views Expr::Const(_, _) diff --git a/src/simlin-engine/src/compiler/context.rs b/src/simlin-engine/src/compiler/context.rs index ca2dc38aa..17f017eb2 100644 --- a/src/simlin-engine/src/compiler/context.rs +++ b/src/simlin-engine/src/compiler/context.rs @@ -255,14 +255,49 @@ impl Context<'_> { self.get_submodel_metadata(self.model_name, ident) } - /// The active subscript each of `dims` reads, for a BARE arrayed reference - /// inside an apply-to-all body. + /// The active subscript each of `dims` reads, for a subscript-less arrayed + /// reference inside an apply-to-all body. /// /// The axis ALLOCATION -- which active axis supplies which of `dims` -- is /// `dimensions::allocate_implicit_axes`, shared with the LTM per-element /// projection so a link-score pin cannot spell a row this reference does not /// read. See that function for the two properties (positional, one-to-one) /// that a name-keyed re-derivation gets wrong. + /// + /// **Which references arrive here** -- worth stating, because the obvious + /// guess is wrong and a GH #996 investigation lost time to it. A bare + /// arrayed reference in an EQUATION BODY does NOT: [`Self::lower_pass0`] + /// rewrites it into an explicit `Expr2::Subscript` before Expr3, so it + /// resolves through the subscript path and never reaches `var_ref`'s + /// arrayed branch. The invariant, measured by tagging each call with its + /// caller and running the whole lib suite: **exactly two production + /// callers, and both are wiring rather than expressions.** ZERO calls + /// arrive via `lower_from_expr3`, which is the one that would mean an + /// ordinary equation reference. + /// + /// - [`Self::fold_flows`] -- a stock's inflow/outflow references; + /// - `compiler::Var::new` -- the stock self-reference inside + /// `build_stock_update_expr`, plus module input wiring. + /// + /// The counts, since they are only reproducible with the condition + /// attached (`cargo test -p simlin-engine --lib -- --nocapture + /// --test-threads=1`, which `--nocapture` is required for -- without it + /// stderr is captured and the measurement reads zero): 477 + 438 = 915 + /// production calls on the current tree, or 441 + 402 = 843 when + /// `crate::mapped_reference_semantics_tests` is skipped, since that module + /// adds calls of its own. One further call in either condition comes from + /// `test_get_implicit_subscript_off_translates_through_mapping_parent`, + /// which invokes [`Self::get_implicit_subscript_off`] directly. + /// + /// That split is why the flow reference is the one subscript-less spelling + /// that can follow an explicit element map (through the + /// `translate_via_mapping` fallback in + /// [`Self::get_implicit_subscript_off`], though only when the source + /// dimension does not already contain an element of the same NAME) while + /// the bare in-equation one is positional; both halves are pinned in + /// `crate::mapped_reference_semantics_tests`. It is also why the GH #996 + /// hazard fixture there is built from a two-axis FLOW under a stock: no + /// ordinary expression can reach this allocation at all. fn get_implicit_subscripts(&self, dims: &[Dimension], ident: &str) -> Result> { if self.active_dimension.is_none() { return sim_err!(ArrayReferenceNeedsExplicitSubscripts, ident.to_owned()); @@ -287,38 +322,34 @@ impl Context<'_> { let mut off = 0_usize; for (dim, subscript) in dims.iter().zip(subscripts) { let element = CanonicalElementName::from_raw(subscript); - let element_off = dim.get_offset(&element).or_else(|| { - // The subscript comes from the active dimension but the source dimension - // uses different element names. Use dimension mapping to translate. - for active_dim in active_dims.iter() { - if active_dim.get_offset(&element).is_some() - && let Some(translated) = self.dimensions_ctx.translate_via_mapping( - dim.canonical_name(), - active_dim.canonical_name(), - &element, - ) - { - return dim.get_offset(&translated); - } - - // If dim maps to a parent of the active subdimension, translate through - // that mapped parent (active subdimension elements are a subset of parent). - if active_dim.get_offset(&element).is_some() - && let Some(parent_dim) = self.dimensions_ctx.find_mapping_parent_of( - dim.canonical_name(), - active_dim.canonical_name(), - ) - && let Some(translated) = - self.dimensions_ctx.translate_to_source_via_mapping( - dim.canonical_name(), - parent_dim, - &element, - ) - { - return dim.get_offset(&translated); - } - } - None + // The subscript comes from an active dimension; which element of + // THIS source axis it selects is + // `DimensionsContext::resolve_mapped_read` (GH #997) -- name + // first, then the declared mapping, then a mapped parent of the + // active subdimension. + // + // The `get_offset` guard is the SEARCH half and stays here: with + // several active dimensions, only the one that owns this element + // can supply it, and the shared rule is per (source axis, active + // dimension) pair rather than a search over candidates. At least + // one active dimension always passes it -- `get_implicit_subscripts` + // returns active SUBSCRIPTS, each an element of its own active + // dimension -- so the shared rule's name-first arm is reached for + // every element, exactly as the un-guarded `dim.get_offset` that + // used to precede this loop was. + // + // One behaviour widened when the loop became a `find_map`: a + // translation that resolves to an element this axis does not + // declare (a malformed element map) used to abort the whole + // resolution, and now falls through to the remaining active + // dimensions. That can only resolve a reference that previously + // failed to compile. + let element_off = active_dims.iter().find_map(|active_dim| { + active_dim.get_offset(&element)?; + let resolved = self + .dimensions_ctx + .resolve_mapped_read(dim, active_dim, &element)?; + dim.get_offset(&resolved) }); let element_off = element_off.ok_or_else(|| { crate::Error::new( @@ -2667,42 +2698,94 @@ impl Context<'_> { let active_dims = self.active_dimension.as_ref().unwrap(); let active_subscripts = self.active_subscript.as_ref().unwrap(); - // Find the matching active dimension (direct name match) - for (active_dim, active_subscript) in active_dims.iter().zip(active_subscripts) { - if &*canonicalize(active_dim.name()) == name.as_str() { - if let Some(offset) = dim.get_offset(active_subscript) { - return Ok(SubscriptIndex::Single(Expr::Const( - (offset + 1) as f64, - *dim_loc, - ))); - } else if let Ok(idx_val) = active_subscript.as_str().parse::() { - return Ok(SubscriptIndex::Single(Expr::Const( - idx_val as f64, - *dim_loc, - ))); - } - } - } - - // No direct match -- check dimension mappings. - // The subscript dimension (name) maps to an active dimension, or vice versa. + // Find the active dimension this index names, then resolve the + // element it selects on THIS source axis. Which active + // dimension: by name first, then through a declared mapping in + // either direction -- the pairing + // `compiler::subscript::normalize_subscripts3` makes on the + // static path. Which element: the shared executed rule + // (`DimensionsContext::resolve_mapped_read`, GH #997). + // + // Both halves used to be spelled out here as two separate + // loops, and the second one consulted the mapping WITHOUT + // trying the active element's own name against this axis + // first -- a divergence from the two static sites that would + // have read a different element for a mapped pair whose two + // dimensions share element names. Instrumenting all three + // arms found this one resolving nothing across the lib suite, + // but it is reachable -- measured at 8 references in the + // integration corpus -- so the divergence was latent rather + // than absent. Routing it through the shared rule removes it. + // + // One behaviour changed for a reference that ALREADY compiled, + // and it is a fix rather than a wash: where a source axis's + // dimension maps to two active dimensions and the index names + // one of them, the old second loop could pair it with the OTHER + // (it tested only that a mapping existed, in either direction, + // and took the first active dimension that had one). The + // candidate order below names the one the index spells first, + // matching what `normalize_subscripts3` picks on the static + // path for the same reference. + // Candidates in `normalize_subscripts3`'s order -- every active + // dimension the index NAMES, then every one it reaches through + // a declared mapping -- and the first that resolves wins. + // + // The two used to be separate passes distinguished by a + // `Pairing` enum whose only reader was a numeric fallback: for + // an INDEXED active dimension whose numeral the source axis did + // not declare, the by-name pass emitted the raw 1-based index. + // Both are gone. The fallback is measured DEAD -- zero + // executions across the lib and integration corpora, where the + // by-name candidate is reached 8 times and every one resolves + // by name identity -- and its static twin `build_view_from_ops` + // has no such fallback at all, so keeping it was the same class + // of latent divergence GH #997 removed from the rest of this + // arm. Both paths now REFUSE an unresolvable subscript rather + // than one of them inventing an index -- the codes still + // differ (`MismatchedDimensions` here, `Generic` there), which + // is worth tidying but is not what the fallback was about. + // (The gate was not structurally vacuous: a NAMED dimension may + // declare a mapping toward an indexed one, which puts an + // indexed active dimension in the mapping candidates. It is + // empirically dead, which is the stronger reason to drop it.) + // + // The ORDER survives as a chained iterator rather than as a + // documented property, because it costs nothing and mirrors + // `normalize_subscripts3`. No reference in either corpus has + // two candidates: this arm is entered 12 times, 8 with a single + // by-name candidate (every one resolving by name identity) and + // 4 with none at all (the `no_mapping_*` refusal cells of + // `crate::mapped_reference_semantics_tests`). The two-candidate + // shape -- a target iterating both a dimension and something + // mapped to it -- is nevertheless REACHABLE, and not by the + // route one would guess: Pass 1 folds an active dimension's + // name to an ordinal only when it runs, and + // `lower_preserving_dimensions` skips it, which is exactly how + // all 8 corpus references (`LOOKUP` table arguments with an + // `@N` sibling) arrive here naming an ACTIVE dimension. A + // fixture of that shape reaches this loop with two candidates. + // What is unmeasured is whether the two ever resolve to + // DIFFERENT elements in a model that compiles; the order is + // chosen to match the static path either way. let sub_dim_name = CanonicalDimensionName::from_raw(name.as_str()); - for (active_dim, active_subscript) in active_dims.iter().zip(active_subscripts) { - let active_dim_name = - CanonicalDimensionName::from_raw(&canonicalize(active_dim.name())); - let has_forward = self - .dimensions_ctx - .has_mapping_to(&sub_dim_name, &active_dim_name); - let has_reverse = self - .dimensions_ctx - .has_mapping_to(&active_dim_name, &sub_dim_name); - if (has_forward || has_reverse) - && let Some(translated) = self.dimensions_ctx.translate_via_mapping( - dim.canonical_name(), - active_dim.canonical_name(), - active_subscript, - ) - && let Some(offset) = dim.get_offset(&translated) + let by_name = active_dims + .iter() + .zip(active_subscripts) + .filter(|(ad, _)| ad.canonical_name().as_str() == name.as_str()); + let by_mapping = active_dims + .iter() + .zip(active_subscripts) + .filter(|(ad, _)| ad.canonical_name().as_str() != name.as_str()) + .filter(|(ad, _)| { + let adn = ad.canonical_name(); + self.dimensions_ctx.has_mapping_to(&sub_dim_name, adn) + || self.dimensions_ctx.has_mapping_to(adn, &sub_dim_name) + }); + for (active_dim, active_subscript) in by_name.chain(by_mapping) { + if let Some(resolved) = + self.dimensions_ctx + .resolve_mapped_read(dim, active_dim, active_subscript) + && let Some(offset) = dim.get_offset(&resolved) { return Ok(SubscriptIndex::Single(Expr::Const( (offset + 1) as f64, diff --git a/src/simlin-engine/src/compiler/dimensions.rs b/src/simlin-engine/src/compiler/dimensions.rs index 55d4340c9..7ba2a035e 100644 --- a/src/simlin-engine/src/compiler/dimensions.rs +++ b/src/simlin-engine/src/compiler/dimensions.rs @@ -42,7 +42,29 @@ use crate::dimensions::{Dimension, DimensionsContext}; /// exact identity name matches, and only one reaches the mapping branch at all, /// with a single active axis where no reordering is possible. The shapes that /// could distinguish the orders are barely exercised, so "no disagreement" is -/// mostly a statement about the corpus. +/// mostly a statement about the corpus. Re-measuring on the compiler path +/// alone found the same thing more sharply: over the lib suite every shape +/// but two is `dims == active_dims` by name and the two exceptions are +/// single-axis, and on C-LEARN the mapping pass is never consulted at all -- +/// 278 calls spanning 4 shapes, ALL identity (`["scenario"]`, `["cop"]`, +/// `["scenario","layers"]`, `["hfc_type"]`, each against itself). To +/// reproduce the C-LEARN figure, print `dims`/`active_dims` at the top of +/// `compiler::context`'s `get_implicit_subscripts` and run +/// `cargo run --release -p simlin-engine --example ltm_fragment_failures`, +/// which compiles the model with LTM enabled. (The lib-suite call counts +/// and the two-caller invariant behind them are recorded on that same +/// function, with the condition they need to be reproducible.) +/// +/// The hazard is nevertheless REACHABLE from a real model, which is the part +/// the corpus does not show: +/// `mapped_reference_semantics_tests::the_996_hazard_shape_compiles_and_reads_name_first` +/// is a stock over `[Line, Shift]` fed by a flow over `[Board Type, Line]`, +/// where `Board Type` maps to both `Line` and `Shift`. Under this flat +/// staging it compiles and reads the element map; under the per-dimension +/// staging it does not compile at all. An ordinary expression cannot reach +/// this function (see `compiler::context`'s `get_implicit_subscripts`), so a +/// stock's flow is the way in and a fixture built from an aux equation will +/// silently exercise nothing. /// /// Both were live silent-wrong-row defects in the LTM per-element projection /// (P2-1 / P2-2 of the whole-branch review) precisely because that projection diff --git a/src/simlin-engine/src/compiler/mod.rs b/src/simlin-engine/src/compiler/mod.rs index ba974f897..93d6f7ec1 100644 --- a/src/simlin-engine/src/compiler/mod.rs +++ b/src/simlin-engine/src/compiler/mod.rs @@ -2,6 +2,7 @@ // Use of this source code is governed by the Apache License, // Version 2.0, that can be found in the LICENSE file. +mod array_operand; mod codegen; pub mod context; pub mod dimensions; @@ -998,6 +999,11 @@ impl Var { // and the salsa per-variable fragment path) funnels through -- so both // backends (VM and wasmgen) see the folded form. let ast: Vec = ast.into_iter().map(fold::fold_constants).collect(); + // Discharge codegen's "an array operand is a view over storage" + // contract (GH #995). Runs at the same chokepoint and after folding, + // so it sees the final tree both backends consume and never + // materializes something folding would have collapsed. + let ast = array_operand::materialize_computed_array_operands(ast); check_stock_updates_are_emittable(&ast, var.ident())?; Ok(Var { ident: Ident::new(var.ident()), @@ -1188,21 +1194,275 @@ fn is_array_producing_builtin(expr: &Expr) -> bool { ) } +/// Which snapshot buffer an array-valued `PREVIOUS`/`INIT` reads (GH #995). +/// +/// Deliberately narrower than [`crate::bytecode::ViewStorage`]: a temp array has +/// no snapshot, so that pairing cannot be expressed here at all. +#[derive(Clone, Copy, PartialEq, Eq)] +pub(super) enum SnapshotRegion { + Prev, + Initial, +} + +/// Classify a `PREVIOUS`/`INIT` call: `Some((argument, region))` when it is +/// ARRAY-valued, `None` when it is the ordinary scalar form (or not a +/// `PREVIOUS`/`INIT` at all). +/// +/// This is the single definition of "this call takes the array route" for every +/// SCALAR position: `walk_expr` (inside a `BeginIter` body), +/// `collect_iter_source_views_impl` (which must pre-push exactly the views the +/// body reads), codegen's one-argument `Mean` arm, and [`array_operand`]'s +/// materializer, which must NOT move an array-valued `PREVIOUS` into a temp. +/// Those disagreeing is how a view gets pushed that nothing reads, or read that +/// nothing pushed. `walk_expr_as_view` deliberately does NOT ask this question +/// -- an explicit view operand takes ANY `PREVIOUS`/`INIT` that lowered to a +/// view, single-element included -- so it goes straight to +/// `Compiler::snapshot_static_view`'s `SnapshotPosition::ViewOperand` arm, whose +/// rustdoc carries the reason. +/// +/// Array-valuedness is decided by the ARGUMENT's shape rather than by the call: +/// `PREVIOUS(vals[D])` inside a vector builtin is the whole array while +/// `PREVIOUS(matrix[E,1])` is one element, and only lowering knows which. An +/// argument that carries an array shape but did not lower to a view over storage +/// still classifies as array-valued -- `snapshot_static_view` then rejects it +/// loudly, which is the point: silently falling back to the scalar route would +/// read one element and broadcast it. +pub(super) fn snapshot_view_arg(builtin: &BuiltinFn) -> Option<(&Expr, SnapshotRegion)> { + let (arg, region) = match builtin { + BuiltinFn::Previous(arg, _) => (arg.as_ref(), SnapshotRegion::Prev), + BuiltinFn::Init(arg) => (arg.as_ref(), SnapshotRegion::Initial), + _ => return None, + }; + let is_array = find_expr_array_view(arg).is_some_and(|view| !view.dims.is_empty()); + is_array.then_some((arg, region)) +} + /// Extract the output ArrayView from an expression. For array-producing builtins, the /// output dimensions come from the builtin's "shaping" argument: /// VectorElmMap(_, offset) -> offset's view /// VectorSortOrder(arr, _) -> arr's view /// AllocateAvailable(req,_,_) -> req's view -fn find_expr_array_view(expr: &Expr) -> Option { +/// +/// Everything else is elementwise, and an elementwise expression's shape is the +/// JOIN of its subexpressions' shapes -- the narrowest view they all broadcast +/// into ([`join_array_views`]), not whichever one is written first. Both consumers +/// need the join and for the same reason: an elementwise expression is evaluated +/// by codegen's `AssignTemp` -> `BeginIter` loop, which broadcasts each source +/// view onto the ITERATION by dimension id, so a source dimension the iteration +/// does not have reads NaN. Taking the first view instead made +/// `VECTOR SORT ORDER(small[d] + wide[e,d], 1)` iterate over `small`'s three +/// elements and return the sort order of three NaNs, while the commuted +/// `wide[e,d] + small[d]` -- the same array -- returned the right answer. +/// +/// `None` means one of two things: no subexpression carries an array shape, or +/// two of them carry shapes neither of which contains the other. A view that +/// REPEATS a dimension name (`matrix[d,d]`) is NOT one of them -- as an +/// expression's sole shape it is returned like any other. That shape is refused +/// as a TEMP's shape, but the refusal lives at the one call site that can be +/// loud about it ([`array_operand::materialize_view_operand`]) rather than here; +/// see [`view_repeats_a_dimension`] for why the difference matters. +/// +/// The four call sites do NOT all treat `None` the same way, and only one of +/// them is loud: +/// +/// * [`array_operand::materialize_view_operand`] declines to materialize, which +/// leaves codegen to reject the operand with a diagnostic attributed to the +/// variable. This is the loud one. +/// * the three apply-to-all / arrayed hoisting sites SUBSTITUTE the variable's +/// own view (`unwrap_or_else(|| var_view.clone())`) with no diagnostic, and +/// the substituted view can be a DIFFERENT SIZE from the array the hoisted +/// builtin writes. That is a live hazard, not a latent one: while this +/// function refused a sole repeated-dimension view, `out[d] = +/// SUM(VECTOR SORT ORDER(matrix[d,d], 1))` sized the sort order's temp at +/// `out`'s three slots and the VM indexed past it -- a panic, which under +/// `panic = abort` takes the host process with it. Any future `None` this +/// function learns to return must be checked against these three sites, or +/// given to them as an `Err` instead. +/// * [`snapshot_view_arg`] reads only `is_empty()`, so a `None` classifies the +/// call as SCALAR and it compiles to `LoadPrev`/`LoadInitial`. Reaching it +/// needs a `PREVIOUS`/`INIT` whose argument survived `builtins_visitor`'s +/// helper rewriting as a multi-shape expression, which the array-shaped +/// passthrough predicate does not admit. +pub(super) fn find_expr_array_view(expr: &Expr) -> Option { + let mut views = Vec::new(); + collect_expr_array_views(expr, &mut views); + join_array_views(views) +} + +/// The narrowest of `views` that every one of them broadcasts into, or `None` +/// when there is no such view. +/// +/// A single view is returned unchanged, dimensionless (a subscript collapsed to +/// one element) included, so this is a no-op wherever the shapes already agreed. +/// +/// The widest view is CHOSEN rather than accumulated left to right: a fold would +/// call `[e], [d], [e,d]` incomparable on its second step even though the third +/// contains both. Two maximal views that disagree on AXIS ORDER (`[e,d]` and +/// `[d,e]` contain each other) are `None` rather than a coin flip -- the axis +/// order is the one `VECTOR SORT ORDER` sorts along and the layout every +/// consumer projects through, so guessing it is exactly the silently-wrong +/// answer this function exists to stop producing. +fn join_array_views(views: Vec) -> Option { + if views.len() <= 1 { + return views.into_iter().next(); + } + let maximal: Vec = (0..views.len()) + .filter(|&i| views.iter().all(|other| view_contains(&views[i], other))) + .collect(); + let &widest = maximal.first()?; + if maximal + .iter() + .any(|&i| views[i].dim_names != views[widest].dim_names) + { + return None; + } + Some(views[widest].clone()) +} + +/// True when `view` names one dimension more than once (`matrix[d,d]`). +/// +/// Such a view is a perfectly good ARRAY -- nine well-defined cells -- and this +/// says nothing about reading it directly. What it cannot be is the shape of a +/// temp that a computed operand is evaluated into, or of a snapshot region a +/// `PREVIOUS`/`INIT` view addresses, because every layer that projects between +/// an array and a temp does so BY DIMENSION NAME and takes the first match: +/// [`project_var_index_to_temp`] gives both `d` axes the same coordinate (so +/// `out[i,j]` reads `temp[i,i]`), and `codegen::array_view_to_static_temp` keys +/// `DimId`s by name, so the runtime broadcast has the same blind spot. Neither +/// can say WHICH `d` is meant. +/// +/// It has exactly TWO callers, and both are positions that can refuse LOUDLY: +/// [`array_operand::materialize_view_operand`] (which leaves codegen to reject +/// the operand) and `codegen::snapshot_static_view` (which returns an `Err` of +/// its own). It deliberately does NOT live inside [`join_array_views`], because +/// [`find_expr_array_view`]'s other three consumers turn a `None` into a silent +/// substitution of the variable's own view -- and for `out[d] = +/// SUM(VECTOR SORT ORDER(matrix[d,d], 1))` that substituted a three-slot temp +/// for a nine-element sort order and the VM indexed past it. Refusing at the +/// loud sites refuses exactly the same equations and costs no others. +/// +/// Scope, deliberately: this refuses only what GH #995 newly made compilable. +/// A repeated dimension read DIRECTLY (`out[d,d] = VECTOR SORT ORDER(matrix[d,d], +/// 1)`, or even `out[d,d] = matrix[d,d]`) compiles at the merge base and still +/// does, to the same first-axis-wins numbers -- measured, and pinned as a +/// disclosed residual by +/// `array_operand_materialization_tests::a_repeated_dimension_read_directly_is_a_pre_existing_residual`. +/// Widening the refusal to cover it would be a fix to a pre-existing defect +/// riding on an unrelated change, and the right fix is to make the projection +/// axis-identity-aware rather than to refuse the shape -- the more so because +/// the shape is legitimate: Vensim REJECTS the declaration -- run in Vensim DSS 2026-08-04, `vensim-probes/repeated_dimension.mdl` refuses to simulate with "DimA appears more than once on LHS" -- so no MDL-imported model can contain this shape and the residual is confined to hand-authored XMILE/JSON/protobuf. It is NOT illegitimate, though: the XMILE v1.0 spec exemplifies the declaration (`docs/reference/xmile-v1.0.html`, "A 2D non-apply-to-all array with dimensions X by X, where X is size 2", verified in-repo), so a conformant file may carry it and Simlin must keep reading it. The spec exemplifies only the DECLARATION, with per-element equations; it says nothing about what a REFERENCE such as `sq[X,X]` means, which is the part that is wrong here. +pub(super) fn view_repeats_a_dimension(view: &ArrayView) -> bool { + (1..view.dim_names.len()) + .any(|i| !view.dim_names[i].is_empty() && view.dim_names[..i].contains(&view.dim_names[i])) +} + +/// True when an iteration shaped like `outer` can read every element of `inner`. +/// +/// The first branch is an IDENTICAL-shape test, and it carries both families +/// [`named_dims`] refuses. An UNNAMED view (a temp's `dim_names` are empty +/// strings) has no name to compare and needs none against a copy of itself. A +/// REPEATED name is the same: `square[d,d] + square[d,d]` is a well-defined +/// elementwise expression and joins to that shape, and the join is the right +/// answer to give -- it is the MATERIALIZER that then refuses to build a temp of +/// that shape ([`view_repeats_a_dimension`]), because the refusal is about +/// projecting into a temp rather than about comparing two views. +/// +/// Beyond identity the relation is by dimension NAME and size, because that is +/// what the runtime broadcast matches on (`vm`'s `LoadIterViewAt` -> +/// [`crate::dimensions::match_dimensions_two_pass`]): a source dimension the +/// iteration cannot match by id reads NaN, so placing one positionally would be +/// a guess. A dimensionless view is contained by everything, which is how a +/// collapsed element such as `vals[1]` broadcasts without constraining the shape +/// around it. +fn view_contains(outer: &ArrayView, inner: &ArrayView) -> bool { + if outer.dims == inner.dims && outer.dim_names == inner.dim_names { + return true; + } + let (Some(outer), Some(inner)) = (named_dims(outer), named_dims(inner)) else { + return false; + }; + inner + .iter() + .all(|(name, size)| outer.iter().any(|(o, s)| o == name && s == size)) +} + +/// A view's `(dimension name, size)` pairs, or `None` when it does not name +/// every dimension or names one TWICE. +/// +/// Both refusals are the same point: containment is decided by name, and +/// neither shape can answer it. An unnamed axis has nothing to match; a +/// `matrix[d,d]` view can say "contains `d` at size 3" but not WHICH `d`, so +/// `[d,d] contains [d]` is unanswerable rather than true. Both families still +/// reach [`view_contains`]'s identical-shape branch, which needs no name; what +/// refuses a repeated name as an expression's SOLE shape is +/// [`view_repeats_a_dimension`], at the materializer. See `array_operand`'s +/// "What still declines". +fn named_dims(view: &ArrayView) -> Option> { + if view.dim_names.len() != view.dims.len() || view.dim_names.iter().any(|n| n.is_empty()) { + return None; + } + if (1..view.dim_names.len()).any(|i| view.dim_names[..i].contains(&view.dim_names[i])) { + return None; + } + Some( + view.dim_names + .iter() + .map(|n| n.as_str()) + .zip(view.dims.iter().copied()) + .collect(), + ) +} + +/// Every array view the subexpressions of `expr` carry. +/// +/// Split out from [`find_expr_array_view`] so the enumeration of which +/// positions carry a shape lives in exactly one place: which arguments an +/// array-producing builtin takes its shape from, which builtins are elementwise +/// (and so contribute every argument's shape), and which are scalar-valued and +/// contribute nothing. +/// +/// The `If` CONDITION is visited. It contributes nothing to an `IF` whose arms +/// already agree, but the `BeginIter` body READS it +/// (`codegen::collect_iter_source_views_impl` pushes its view), so +/// `IF wide[e,d] > 0 THEN a[d] ELSE b[d]` does vary over `e` and the iteration +/// evaluating it has to as well. +/// +/// Written as an exhaustive match with no `_` arm over `BuiltinFn` so a new +/// builtin is a compile error here rather than a silently unshaped operand. +fn collect_expr_array_views(expr: &Expr, out: &mut Vec) { match expr { - Expr::StaticSubscript(_, view, _) | Expr::TempArray(_, view, _) => Some(view.clone()), + Expr::StaticSubscript(_, view, _) | Expr::TempArray(_, view, _) => out.push(view.clone()), Expr::App(builtin, _) => match builtin { - BuiltinFn::VectorElmMap(_, offset) => find_expr_array_view(offset), + BuiltinFn::VectorElmMap(_, offset) => collect_expr_array_views(offset, out), BuiltinFn::VectorSortOrder(arr, _) | BuiltinFn::Rank(arr, _) => { - find_expr_array_view(arr) + collect_expr_array_views(arr, out) } BuiltinFn::AllocateAvailable(req, _, _) - | BuiltinFn::AllocateByPriority(req, _, _, _, _) => find_expr_array_view(req), + | BuiltinFn::AllocateByPriority(req, _, _, _, _) => collect_expr_array_views(req, out), + // Elementwise scalar builtins: applied per iteration inside a + // `BeginIter` body, so their result has the shape of whichever + // argument carries one -- every one of them, for the join; the + // first, for `find_expr_array_view`, matching the `Op2` rule below. + // + // Deliberately absent: `Mean` (variadic -- its single-argument + // form is a REDUCTION to a scalar, not elementwise), the reducers + // (`Sum`/`Size`/`Stddev` and one-argument `Min`/`Max`, also + // scalar-valued), `VectorSelect` (scalar-valued), the `Lookup` + // family (`LookupArray`'s shape is the TABLE array's, which this + // would have to reach through the gf registry), and the 0-arity + // builtins. Their ARGUMENTS are not walked either: a reducer + // collapses whatever it reads to one number, so + // `vals[d] + SUM(wide[*,*])` is `[d]`-shaped and a walk that let + // `wide`'s view through would widen the temp to a shape the + // operand does not have. + // + // `PREVIOUS`/`INIT` DO carry a shape (GH #995): codegen reads an + // array-valued one as its argument's view over a snapshot buffer, + // so the result has exactly the argument's shape -- and an argument + // that collapsed to a single element yields none, which is what + // keeps a scalar `PREVIOUS(s)` broadcasting instead of reshaping + // the operand around it. + BuiltinFn::Previous(a, _) | BuiltinFn::Init(a) => collect_expr_array_views(a, out), BuiltinFn::Abs(e) | BuiltinFn::Arccos(e) | BuiltinFn::Arcsin(e) @@ -1212,17 +1472,60 @@ fn find_expr_array_view(expr: &Expr) -> Option { | BuiltinFn::Int(e) | BuiltinFn::Ln(e) | BuiltinFn::Log10(e) + | BuiltinFn::Sign(e) | BuiltinFn::Sin(e) | BuiltinFn::Sqrt(e) - | BuiltinFn::Tan(e) => find_expr_array_view(e), - _ => None, + | BuiltinFn::Tan(e) => collect_expr_array_views(e, out), + // Two-argument MIN/MAX are the scalar (elementwise) forms; the + // one-argument forms are array reductions and yield no array. + BuiltinFn::Min(a, Some(b)) | BuiltinFn::Max(a, Some(b)) => { + collect_expr_array_views(a, out); + collect_expr_array_views(b, out); + } + BuiltinFn::Min(_, None) | BuiltinFn::Max(_, None) => {} + BuiltinFn::Quantum(a, b) | BuiltinFn::Step(a, b) => { + collect_expr_array_views(a, out); + collect_expr_array_views(b, out); + } + BuiltinFn::Sshape(a, b, c) => { + collect_expr_array_views(a, out); + collect_expr_array_views(b, out); + collect_expr_array_views(c, out); + } + BuiltinFn::Pulse(a, b, c) | BuiltinFn::Ramp(a, b, c) | BuiltinFn::SafeDiv(a, b, c) => { + collect_expr_array_views(a, out); + collect_expr_array_views(b, out); + if let Some(c) = c.as_ref() { + collect_expr_array_views(c, out); + } + } + BuiltinFn::Lookup(_, _, _) + | BuiltinFn::LookupForward(_, _, _) + | BuiltinFn::LookupBackward(_, _, _) + | BuiltinFn::Mean(_) + | BuiltinFn::Sum(_) + | BuiltinFn::Size(_) + | BuiltinFn::Stddev(_) + | BuiltinFn::VectorSelect(_, _, _, _, _) + | BuiltinFn::IsModuleInput(_, _) + | BuiltinFn::Inf + | BuiltinFn::Pi + | BuiltinFn::Time + | BuiltinFn::TimeStep + | BuiltinFn::StartTime + | BuiltinFn::FinalTime => {} }, - Expr::Op1(_, inner, _) => find_expr_array_view(inner), + Expr::Op1(_, inner, _) => collect_expr_array_views(inner, out), Expr::Op2(_, lhs, rhs, _) => { - find_expr_array_view(lhs).or_else(|| find_expr_array_view(rhs)) + collect_expr_array_views(lhs, out); + collect_expr_array_views(rhs, out); } - Expr::If(_, t, f, _) => find_expr_array_view(t).or_else(|| find_expr_array_view(f)), - _ => None, + Expr::If(cond, t, f, _) => { + collect_expr_array_views(t, out); + collect_expr_array_views(f, out); + collect_expr_array_views(cond, out); + } + _ => {} } } @@ -1908,7 +2211,7 @@ fn replace_nested_builtins_for_element( /// Find the next available temp ID by scanning existing expressions for /// AssignTemp nodes. Uses the existing extract_temp_sizes infrastructure /// which already walks the full expression tree. -fn next_available_temp_id(exprs: &[Expr]) -> u32 { +pub(super) fn next_available_temp_id(exprs: &[Expr]) -> u32 { let mut temp_sizes_map = HashMap::new(); for expr in exprs { extract_temp_sizes(expr, &mut temp_sizes_map); diff --git a/src/simlin-engine/src/compiler/subscript.rs b/src/simlin-engine/src/compiler/subscript.rs index 4c220cbb9..c0aa79a09 100644 --- a/src/simlin-engine/src/compiler/subscript.rs +++ b/src/simlin-engine/src/compiler/subscript.rs @@ -367,17 +367,24 @@ pub(crate) fn build_view_from_ops( let dim = &config.dims[i]; let offset = dim.get_offset(subscript).or_else(|| { - // The subscript from the active dimension doesn't exist in this - // variable's dimension. Try dimension mapping translation. + // The active element's own name is not declared on this + // source axis, so the reference resolves through the + // shared executed rule (GH #997): the declared mapping, + // then a mapped parent of the active subdimension. + // `normalize_subscripts3` already picked the active + // dimension, so this is one call rather than a search. + // + // The mapped-parent step is new here (it was already in + // `get_implicit_subscript_off`, the other executed site). + // Unifying can only resolve a reference that previously + // failed to compile: the shared rule tries name and then + // mapping first, which is exactly what this arm did, and + // reaches the parent step only where both missed. let dims_ctx = config.dimensions_ctx?; let active_dims = config.active_dimension?; let active_dim = &active_dims[*active_idx]; - let translated = dims_ctx.translate_via_mapping( - dim.canonical_name(), - active_dim.canonical_name(), - subscript, - )?; - dim.get_offset(&translated) + let resolved = dims_ctx.resolve_mapped_read(dim, active_dim, subscript)?; + dim.get_offset(&resolved) }); if let Some(offset) = offset { diff --git a/src/simlin-engine/src/compiler/symbolic.rs b/src/simlin-engine/src/compiler/symbolic.rs index f2324323f..e40f4a0fa 100644 --- a/src/simlin-engine/src/compiler/symbolic.rs +++ b/src/simlin-engine/src/compiler/symbolic.rs @@ -34,7 +34,7 @@ use crate::bytecode::{ BuiltinId, ByteCode, ByteCodeContext, CompiledInitial, CompiledModule, DimId, DimListId, GraphicalFunctionId, LiteralId, LookupMode, ModuleDeclaration, ModuleId, ModuleInputOffset, Op2, Opcode, PcOffset, RuntimeSparseMapping, STACK_CAPACITY, StaticArrayView, TempId, - VariableOffset, ViewId, + VariableOffset, ViewId, ViewStorage, }; use crate::common::{Canonical, Ident}; @@ -319,11 +319,23 @@ pub(crate) struct SymbolicStaticView { pub dim_ids: SmallVec<[DimId; 4]>, } +/// Where a symbolic static view's elements live, before layout assignment. +/// +/// The three variable-backed arms differ ONLY in which of the VM's parallel +/// chunk-shaped regions they read; they share `curr`'s slot numbering, so all +/// three resolve a `SymVarRef` through the same layout lookup. Splitting them +/// into distinct variants rather than pairing one `Var` with a storage field +/// keeps `Temp` + a snapshot region -- a temp has no snapshot -- unrepresentable +/// (GH #995). #[derive(Clone, Debug, PartialEq, Eq, Hash)] pub(crate) enum SymStaticViewBase { - /// Model variable reference (replaces base_off when is_temp=false) + /// Model variable reference, read from `curr` Var(SymVarRef), - /// Temp array ID (kept as-is when is_temp=true) + /// Model variable reference, read from the `PREVIOUS` snapshot + PrevVar(SymVarRef), + /// Model variable reference, read from the `INIT` snapshot + InitialVar(SymVarRef), + /// Temp array ID Temp(u32), } @@ -1136,22 +1148,51 @@ pub(crate) fn resolve_static_view( sv: &SymbolicStaticView, layout: &VariableLayout, ) -> Result { - let (base_off, is_temp) = match &sv.base { - SymStaticViewBase::Var(var_ref) => { - let entry = layout.get(var_ref.name.as_str()).ok_or_else(|| { - format!( - "variable '{}' not found in layout during static view resolution", - var_ref.name - ) - })?; - ((entry.offset + var_ref.element_offset) as u32, false) + // The three chunk-shaped regions share `curr`'s slot numbering (each is an + // `n_slots` snapshot of it), so one layout lookup serves all three and only + // the region tag differs. + let resolve_var = |var_ref: &SymVarRef| -> Result { + let entry = layout.get(var_ref.name.as_str()).ok_or_else(|| { + format!( + "variable '{}' not found in layout during static view resolution", + var_ref.name + ) + })?; + Ok((entry.offset + var_ref.element_offset) as u32) + }; + let (base_off, storage) = match &sv.base { + SymStaticViewBase::Var(var_ref) => (resolve_var(var_ref)?, ViewStorage::Curr), + SymStaticViewBase::PrevVar(var_ref) => (resolve_var(var_ref)?, ViewStorage::Prev), + SymStaticViewBase::InitialVar(var_ref) => (resolve_var(var_ref)?, ViewStorage::Initial), + // A view base is the ONE place a temp id is carried as a `u32`. Every + // OTHER opcode that names a temp -- `BeginIter` and the + // array-producing opcodes' `write_temp_id`, `LoadTempConst`'s + // `temp_id` -- carries it as `TempId` (= `u8`), narrowed at emit time + // with a plain `as`. So a view over a temp above 255 reads storage + // nothing ever wrote: the writer's `as TempId` lands on `id % 256` + // while this read lands on `id`, and the program is well-formed either + // way -- wrong numbers with no diagnostic. Reject it in the resolution + // layer, where the concrete program is produced (#583 is the real fix: + // the id namespace is too small for a per-element hoist over a few + // hundred elements). The write-side narrowing is deliberately left + // unguarded; see the module note on this in the crate's CLAUDE.md. + SymStaticViewBase::Temp(id) => { + if *id > TempId::MAX as u32 { + return Err(format!( + "a view over temp {} exceeds TempId capacity (u8::MAX = {}); \ + every writer of a temp narrows its id to u8, so this view \ + would read storage no opcode writes", + id, + TempId::MAX + )); + } + (*id, ViewStorage::Temp) } - SymStaticViewBase::Temp(id) => (*id, true), }; Ok(StaticArrayView { base_off, - is_temp, + storage, dims: sv.dims.clone(), strides: sv.strides.clone(), offset: sv.offset, @@ -2007,7 +2048,13 @@ impl FragmentMerger { self.merged_views.extend(frag.static_views.iter().map(|sv| { let base = match &sv.base { SymStaticViewBase::Temp(id) => SymStaticViewBase::Temp(*id + temp_offset), - other => other.clone(), + // Variable-backed bases -- `curr` and both snapshot regions -- + // name a variable, not a merged resource, so they carry across + // untouched. Written out rather than caught by `_` so a future + // base variant has to state which side of M1 it falls on. + base @ (SymStaticViewBase::Var(_) + | SymStaticViewBase::PrevVar(_) + | SymStaticViewBase::InitialVar(_)) => base.clone(), }; SymbolicStaticView { base, ..sv.clone() } })); @@ -2924,7 +2971,7 @@ mod tests { let resolved = resolve_static_view(&sym, &layout).unwrap(); assert_eq!(resolved.base_off, 5); - assert!(!resolved.is_temp); + assert_eq!(resolved.storage, ViewStorage::Curr); assert_eq!(resolved.dims, sym.dims); assert_eq!(resolved.offset, 0); } @@ -2944,7 +2991,7 @@ mod tests { let resolved = resolve_static_view(&sym, &layout).unwrap(); assert_eq!(resolved.base_off, 7); - assert!(resolved.is_temp); + assert_eq!(resolved.storage, ViewStorage::Temp); } #[test] @@ -3128,7 +3175,14 @@ mod tests { ); } for sv in compiled.context.static_views.iter() { - if !sv.is_temp { + // The three chunk-shaped regions are all `n_slots` wide and share + // `curr`'s numbering, so the owned-slot check applies to each; only + // a temp base indexes something else. + let addresses_a_slot = match sv.storage { + ViewStorage::Curr | ViewStorage::Prev | ViewStorage::Initial => true, + ViewStorage::Temp => false, + }; + if addresses_a_slot { assert!( (sv.base_off as usize) < layout.n_slots && owned[sv.base_off as usize], "static view base {} is not an owned slot", @@ -3294,7 +3348,7 @@ mod tests { let resolved = resolve_static_view(&sym, &layout).unwrap(); assert_eq!(resolved.base_off, large_off as u32); - assert!(!resolved.is_temp); + assert_eq!(resolved.storage, ViewStorage::Curr); } #[test] diff --git a/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs b/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs index c015fb5eb..10cd030cb 100644 --- a/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs +++ b/src/simlin-engine/src/compiler/symbolic_merge_proptest.rs @@ -230,10 +230,21 @@ fn build_fragment(spec: &FragSpec) -> PerVarBytecodes { let on_temps = spec.views_on_temps && !spec.temp_sizes.is_empty(); let static_views: Vec = (0..spec.n_views) .map(|i| SymbolicStaticView { + // Cycle the three VARIABLE-backed bases (GH #995 added the two + // snapshot regions): all three name a variable rather than a merged + // resource, so M1 (referential integrity) and M5 (temp + // non-aliasing) must hold for each, and only `Temp` is renumbered. + // Generating just `Var` would leave the two new arms free to be + // renumbered like a temp with nothing to notice. base: if on_temps { SymStaticViewBase::Temp((i % spec.temp_sizes.len()) as u32) } else { - SymStaticViewBase::Var(SymVarRef::new(name.clone(), i)) + let var = SymVarRef::new(name.clone(), i); + match i % 3 { + 0 => SymStaticViewBase::Var(var), + 1 => SymStaticViewBase::PrevVar(var), + _ => SymStaticViewBase::InitialVar(var), + } }, dims: smallvec::smallvec![(spec.tag * 10 + i) as u16], strides: smallvec::smallvec![1], @@ -1085,7 +1096,11 @@ fn temp_uses(code: &[SymbolicOpcode], views: &[SymbolicStaticView]) -> Vec<(usiz }); match &view.base { SymStaticViewBase::Temp(id) => *id, - SymStaticViewBase::Var(_) => return None, + // Variable-backed bases -- `curr` and the two snapshot + // regions -- name no temp channel. + SymStaticViewBase::Var(_) + | SymStaticViewBase::PrevVar(_) + | SymStaticViewBase::InitialVar(_) => return None, } } _ => return None, diff --git a/src/simlin-engine/src/db/analysis.rs b/src/simlin-engine/src/db/analysis.rs index c0c577dac..b2e948a35 100644 --- a/src/simlin-engine/src/db/analysis.rs +++ b/src/simlin-engine/src/db/analysis.rs @@ -101,10 +101,12 @@ fn format_multi_element_name(var_name: &str, elements: &[&str]) -> String { /// A site that is *not* a hoisted reducer's argument -- a bare dynamic index /// (`arr[i+1]`, a range), the dynamic-index reducer carve-out /// (`SUM(pop[idx, *])`, `idx` non-literal, reclassified to `DynamicIndex`), -/// an ELEMENT-mapped sliced reducer the correspondence declines (GH #756; -/// `enumerate_agg_nodes` declines it, so the reference stays `Direct` and -/// is reclassified `DynamicIndex` -- the reverse-declared POSITIONAL pair -/// is accepted since GH #757), or a direct `pop[idx]` alongside a +/// a sliced reducer the correspondence declines -- an UNDECLARED pair, a +/// cardinality mismatch, or a `MappedRead` axis (GH #997), all of which +/// `enumerate_agg_nodes` refuses, so the reference stays `Direct` and is +/// reclassified `DynamicIndex` (a DECLARED mapping is accepted in either +/// direction since GH #757, an explicit element map included since GH #997) +/// -- or a direct `pop[idx]` alongside a /// `SUM(pop[*])` -- keeps a conservative edge and a Bare-named link score, /// EXCEPT when both endpoints are arrayed with non-corresponding dimensions /// (the declined mapped-reducer cases): no compilable conservative score @@ -234,10 +236,11 @@ fn dimension_element_names(dim: &crate::dimensions::Dimension) -> Vec { /// here is only a variable-backed WHOLE-EXTENT reducer's argument /// (`total = SUM(population[*])`, the broadcast `share[R] = SUM(pop[*])`, or /// a partial reduce whose result axes don't equal the target's dims), a -/// (rare) non-reducer whole-array reference, or an ELEMENT-mapped sliced -/// reducer the correspondence declines (GH #756; the reverse-declared -/// positional pair is hoisted since GH #757). The conservative cross -/// product is sound for the element +/// (rare) non-reducer whole-array reference, or a sliced reducer the +/// correspondence declines -- an UNDECLARED pair, a cardinality mismatch, or +/// a `MappedRead` axis (GH #997); a DECLARED mapping is hoisted in either +/// direction since GH #757, an explicit element map included since GH #997. +/// The conservative cross product is sound for the element /// EDGES in all of those (a superset, never fewer); the declined /// mapped-reducer cases' link SCORES have no compilable conservative shape, /// so the emitter skips them loudly and loop scores through the edge are @@ -380,7 +383,16 @@ fn emit_edges_for_reference( use crate::ltm_agg::AxisRead; let from_dim_element_lists: Vec> = from_dims.iter().map(dimension_element_names).collect(); - let rows = crate::db::ltm::read_slice_rows(axes, &from_dim_element_lists, dim_ctx); + // The STRUCTURED derivation, not the comma-joined projection: a + // canonical element name can itself contain a comma (a quoted XMILE + // element `"a,b"` canonicalizes to `a,b`, and such a model compiles + // and runs -- measured), so joining the slot coordinates and + // re-splitting them here would mis-read one coordinate as two. That + // would drop the real edge and mint one to a target element that + // does not exist. `emit_agg_routed_edges` below already reads the + // structured form; this is the same rule, and neither surface needs + // the string. + let rows = crate::db::ltm::read_slice_row_parts(axes, &from_dim_element_lists, dim_ctx); // Iterated target dims in slot order; every one must name a // target dim for the slot projection to be meaningful (true by // construction -- the classifier only mints `Iterated` for the @@ -390,8 +402,14 @@ fn emit_edges_for_reference( let iter_dims: Vec<&str> = axes .iter() .filter_map(|a| match a { - AxisRead::Iterated { dim, .. } => Some(dim.as_str()), - _ => None, + // Both projected axes contribute a slot coordinate, in axis + // order, matching what `read_slice_row_parts` pushes; only + // the RULE each uses to resolve its element differs + // (GH #997). + AxisRead::Iterated { dim, .. } | AxisRead::MappedRead { dim, .. } => { + Some(dim.as_str()) + } + AxisRead::Pinned(_) | AxisRead::Reduced { .. } => None, }) .collect(); let slots_resolve = !to_is_scalar @@ -406,9 +424,17 @@ fn emit_edges_for_reference( .map(|d| iter_dims.iter().position(|id| *id == d.name())) .collect(); let target_set: BTreeSet<&String> = target_nodes.iter().collect(); - for crate::db::ltm::ReadSliceRow { row, slot, .. } in &rows { - let from_node = format!("{from_name}[{row}]"); - let slot_parts: Vec<&str> = slot.split(',').collect(); + for crate::db::ltm::ReadSliceRowParts { + row_parts, + slot_parts, + } in &rows + { + let row_refs: Vec<&str> = row_parts.iter().map(String::as_str).collect(); + let from_node = if row_refs.len() == 1 { + format_element_name(from_name, row_refs[0]) + } else { + format_multi_element_name(from_name, &row_refs) + }; // Candidate elements per target-dim position: the slot // coordinate where the dim is iterated, every element // where it broadcasts. @@ -416,7 +442,7 @@ fn emit_edges_for_reference( .iter() .zip(&to_dim_slot_pos) .map(|(d, pos)| match pos { - Some(j) => vec![slot_parts[*j].to_string()], + Some(j) => vec![slot_parts[*j].clone()], None => dimension_element_names(d), }) .collect(); @@ -529,17 +555,129 @@ fn cartesian_element_names(var_name: &str, dims: &[crate::dimensions::Dimension] .collect() } +/// Per element of `to_dim` in declared order, the `from_dim` elements a +/// same-element (`RefShape::Bare`) reference may read -- the UNION of the two +/// spellings' correspondences (GH #527, re-keyed by GH #997). +/// +/// `RefShape::Bare` is the one shape covering references that resolve by two +/// different rules, and nothing downstream of the classification tells them +/// apart: +/// +/// * a bare or iterated-dimension reference in an equation body +/// (`target[State] = x` / `= x[State]`) resolves POSITIONALLY; +/// * a structural flow-to-stock edge (`level[State] = INTEG(x, 0)` with `x` +/// over `Region`) is labelled `Bare` by `model_edge_shapes` with no AST +/// reference behind it at all, and resolves name-first then through the +/// declared element map. +/// +/// Both are legal, both ship, and the element graph must not emit FEWER edges +/// than either reads. So this returns both answers and the expansion emits +/// their union: exact wherever the two rules agree -- which is every +/// positional mapping between dimensions with disjoint element names, i.e. +/// everything that resolved before GH #997 -- and a two-edge superset per +/// target element where they genuinely differ (an explicit element map, or a +/// pair sharing element names). +/// +/// Keeping the union in ONE function is also what keeps the element graph and +/// discovery's from-node projection in lockstep. `expand_same_element` has two +/// consumers, and only one of them can see the reference site: the element +/// graph does, `ltm_finding::expand_a2a_link_offsets` re-derives the from-node +/// from a link score's dimensions alone. A rule chosen per site would have put +/// them back out of step, which is the GH #754 failure -- a from-node naming +/// no real element node, so every loop through it dangles. +/// +/// # EDGES may be a union; SCORES may not +/// +/// The union is sound for edges precisely because an extra edge is the safe +/// direction. It is NOT sound for a link SCORE, and the two must be gated +/// separately -- [`mapped_pair_projects_uniquely`] is that gate. +/// +/// A Bare A2A link score is one arrayed variable with one slot per TARGET +/// element, and `ltm_finding::expand_a2a_link_offsets` maps every edge this +/// function emits for a target element onto that element's single slot. Where +/// the two rules DISAGREE, one of the two source elements is a phantom for the +/// site that actually exists -- and it would read the real edge's non-zero +/// score out of the shared slot. That is a compilable, confidently wrong +/// number, which this repo treats as worse than no score at all (GH #758): the +/// edge is denied the arrayed retarget and takes the loud skip instead. +/// +/// So the two questions are deliberately different: +/// * "which element edges exist?" -> every element in this function's answer; +/// * "may this edge carry an arrayed score?" -> only when every target +/// element's answer is a SINGLETON, i.e. the two rules agree everywhere and +/// no slot is shared by a phantom. +/// +/// Every mapping that resolved before GH #997 is a singleton (the two rules +/// coincide on a positional mapping between disjointly-named dimensions), and +/// so is C-LEARN's many-to-one element map -- there the positional rule +/// declines outright, leaving the executed rule alone in the union. What is +/// NOT a singleton is an equal-cardinality PERMUTED element map, or a pair +/// sharing element names in a different order. +/// +/// `None` when the two dimensions are not related by a declared mapping at +/// all; the caller then broadcasts. +pub(crate) fn bare_reference_correspondence( + dim_ctx: &crate::dimensions::DimensionsContext, + to_dim: &crate::common::CanonicalDimensionName, + from_dim: &crate::common::CanonicalDimensionName, +) -> Option>> { + let positional = dim_ctx.positional_correspondence(to_dim, from_dim); + let executed = dim_ctx.executed_read_correspondence(to_dim, from_dim); + let len = positional + .as_ref() + .or(executed.as_ref()) + .map(Vec::len) + .filter(|n| *n > 0)?; + Some( + (0..len) + .map(|i| { + let mut elems: Vec = Vec::with_capacity(2); + for corr in [positional.as_ref(), executed.as_ref()] + .into_iter() + .flatten() + { + if let Some(e) = corr.get(i) + && !elems.contains(e) + { + elems.push(e.clone()); + } + } + elems + }) + .collect(), + ) +} + +/// May a `Bare` edge across this dimension pair carry an ARRAYED (per-target- +/// element) link score? True only when every target element's +/// [`bare_reference_correspondence`] entry is a SINGLETON -- the two spellings +/// agree, so no target slot is shared by a source element that only one of them +/// reads. +/// +/// See [`bare_reference_correspondence`]'s "EDGES may be a union; SCORES may +/// not" section for why this is a separate, stricter question than which edges +/// exist. A pair this declines keeps its element-edge union (the never-fewer +/// direction is untouched) and loses only the retarget, which sends the edge to +/// the GH #758 loud skip: one Warning naming it, no link-score variable, and +/// loop scores through it dropped. +pub(crate) fn mapped_pair_projects_uniquely( + dim_ctx: &crate::dimensions::DimensionsContext, + to_dim: &crate::common::CanonicalDimensionName, + from_dim: &crate::common::CanonicalDimensionName, +) -> bool { + bare_reference_correspondence(dim_ctx, to_dim, from_dim) + .is_some_and(|corr| corr.iter().all(|elems| elems.len() == 1)) +} + /// Expand same-element edges with possible partial dimension collapse. /// /// For each source element tuple, constructs the target element tuple(s) by /// matching shared dimension names -- or, when names differ, a declared /// dimension MAPPING between a target dimension and a source dimension /// (GH #527; the correspondence comes from -/// [`crate::dimensions::DimensionsContext::mapped_element_correspondence`] -/// and is the diagonal WHEN a usable correspondence exists -- today, -/// positional mappings only -- else the conservative broadcast, a superset -/// of the simulation's true reads; see that helper's rustdoc for the -/// positional-only gate). Dimensions in the source that correspond to +/// [`bare_reference_correspondence`], which is the union of the two +/// spellings' diagonals -- see there for why a `Bare` site cannot pick one). +/// Dimensions in the source that correspond to /// no target dimension are collapsed (their elements are iterated but do /// not appear in the target subscript); target dimensions that correspond /// to no source dimension broadcast over all their elements. @@ -550,10 +688,11 @@ fn cartesian_element_names(var_name: &str, dims: &[crate::dimensions::Dimension] /// - `from[Region] -> to[State]` with a positional `State→Region` mapping: /// the mapping's diagonal -- `from[mapped(s)] -> to[s]` for each State /// element `s`. -/// - `from[Region] -> to[State]` with NO mapping -- or one declared via an -/// explicit element map (declined by the positional-only gate): the -/// conservative broadcast (every source element feeds every target -/// element). +/// - `from[Region] -> to[State]` with an explicit element map: the union of +/// the map's diagonal and the positional one -- at most two source elements +/// per target element, and one where the two agree. +/// - `from[Region] -> to[State]` with NO mapping: the conservative broadcast +/// (every source element feeds every target element). /// /// Besides the element graph (`emit_edges_for_reference`'s `Bare` arm), this /// is also consumed by discovery's `ltm_finding::expand_a2a_link_offsets` @@ -584,16 +723,15 @@ pub(crate) fn expand_same_element( /// Same-named target dimension at this position: target element = /// same index as the source element. SameName(usize), - /// Mapped target dimension at this position. The Vec is indexed by - /// TARGET element index and holds the corresponding SOURCE element - /// index (the diagonal direction `mapped_element_correspondence` - /// defines); the expansion below inverts it per source element. - /// Today the helper only returns positional (bijective) - /// correspondences, but the inversion below is written for the - /// general (many-to-one) shape so re-enabling element-map - /// diagonals (see the helper's positional-only gate) needs no - /// emitter change. - Mapped(usize, Vec), + /// Mapped target dimension at this position. The outer Vec is + /// indexed by TARGET element index and holds the SOURCE element + /// indices that target element may read -- the diagonal direction + /// [`bare_reference_correspondence`] defines, with one entry per + /// spelling whose answers differ. The expansion below inverts it per + /// source element, and is written for the general shape: a source + /// element may have no preimage (nothing maps to it) or several (a + /// many-to-one element map). + Mapped(usize, Vec>), /// No corresponding target dimension: collapse. Collapsed, } @@ -624,18 +762,23 @@ pub(crate) fn expand_same_element( continue; } let Some(elems) = - dim_ctx.mapped_element_correspondence(to_dim.canonical_name(), from_canon) + bare_reference_correspondence(dim_ctx, to_dim.canonical_name(), from_canon) else { continue; }; // Resolve the per-target-element source names to source element - // indices. `mapped_element_correspondence` only returns elements - // of the source dimension, so the lookup can't fail for a - // well-formed context; bail to Collapsed (broadcast) if it does. + // indices. The correspondence only returns elements of the source + // dimension, so the lookup can't fail for a well-formed context; + // bail to Collapsed (broadcast) if it does. let Some(idxs) = elems .iter() - .map(|e| from_dims[i].get_offset(e)) - .collect::>>() + .map(|per_target| { + per_target + .iter() + .map(|e| from_dims[i].get_offset(e)) + .collect::>>() + }) + .collect::>>>() else { continue; }; @@ -710,15 +853,14 @@ pub(crate) fn expand_same_element( to_elem_options[*pos].push(name); } Correspondence::Mapped(pos, target_to_source) => { - // Preimage: every target element whose mapped source - // element is this source element. With today's - // positional-only correspondences this is always a - // singleton, but the general form (empty for a source - // element nothing maps to; several elements for a - // many-to-one element map) is kept for the element-map - // re-enable gate. - for (target_idx, &src_idx) in target_to_source.iter().enumerate() { - if src_idx == src_elem_idx { + // Preimage: every target element that reads this source + // element under either spelling. A singleton for a + // positional mapping between disjointly-named dimensions; + // empty for a source element nothing maps to; several for + // a many-to-one element map, or where the two spellings + // disagree. + for (target_idx, src_idxs) in target_to_source.iter().enumerate() { + if src_idxs.contains(&src_elem_idx) { to_elem_options[*pos].push(to_dim_elements[*pos][target_idx].as_str()); } } @@ -3723,13 +3865,20 @@ mod emit_edges_for_reference_tests { } } - /// GH #527: a `Bare` edge between dimensions related by a POSITIONAL - /// mapping projects the diagonal; a pair related only by an EXPLICIT - /// element map keeps the conservative broadcast (the executed A2A - /// lowering resolves positionally, ignoring the element map -- see - /// `mapped_element_correspondence`'s positional-only gate / GH #753). + /// GH #527 / GH #997: a `Bare` edge between dimensions related by a + /// POSITIONAL mapping projects the single diagonal both spellings agree on; + /// a pair related by an EXPLICIT element map projects the UNION of the two + /// spellings' diagonals, because a `Bare` site can be an in-equation + /// reference (positional) or a structural flow-to-stock edge (map-following) + /// and nothing downstream of the shape tells them apart -- see + /// [`bare_reference_correspondence`]. + /// /// Exercised directly (no salsa pipeline) so both arms of the - /// correspondence decision are pinned at the emitter level. + /// correspondence decision are pinned at the emitter level. The 2-element + /// permuted case below produces the FULL broadcast, which is what this test + /// asserted before GH #997 and would still pass for the old reason; the + /// 3-element case after it is the one that distinguishes a union from a + /// broadcast. #[test] fn bare_mapped_dims_positional_diagonal_element_map_broadcast() { // Positional mapping: diagonal. @@ -3806,6 +3955,53 @@ mod emit_edges_for_reference_tests { Some(&broadcast), "element map: x[b] keeps the conservative broadcast" ); + + // Three elements, so the two rules' answers are a strict SUBSET of the + // broadcast and the union is observable. The map is the 3-cycle + // s1->b, s2->c, s3->a; the positional diagonal is s1->a, s2->b, s3->c. + // Each source element therefore feeds exactly TWO targets, never all + // three -- which is the property the flow-to-stock spelling needs (its + // map-following read must have an edge) without giving up the tightening + // (the third pair is a phantom either way). + let mut state3 = crate::datamodel::Dimension::named( + "State".to_string(), + vec!["s1".to_string(), "s2".to_string(), "s3".to_string()], + ); + state3.mappings = vec![crate::datamodel::DimensionMapping { + target: "Region".to_string(), + element_map: vec![ + ("s1".to_string(), "b".to_string()), + ("s2".to_string(), "c".to_string()), + ("s3".to_string(), "a".to_string()), + ], + }]; + let region3 = crate::datamodel::Dimension::named( + "Region".to_string(), + vec!["a".to_string(), "b".to_string(), "c".to_string()], + ); + let dim_ctx3 = + crate::dimensions::DimensionsContext::from(&[state3.clone(), region3.clone()]); + let mut edges3: HashMap> = HashMap::new(); + emit_edges_for_reference( + "x", + "target", + &[crate::dimensions::Dimension::from(®ion3)], + &[crate::dimensions::Dimension::from(&state3)], + &RefShape::Bare, + None, + &dim_ctx3, + &mut edges3, + ); + for (src, positional, mapped) in [("a", "s1", "s3"), ("b", "s2", "s1"), ("c", "s3", "s2")] { + assert_eq!( + edges3.get(&format!("x[{src}]")), + Some(&BTreeSet::from([ + format!("target[{positional}]"), + format!("target[{mapped}]") + ])), + "x[{src}] must feed exactly its positional and its mapped target" + ); + } } } diff --git a/src/simlin-engine/src/db/combined_fragment_proptest.rs b/src/simlin-engine/src/db/combined_fragment_proptest.rs index df7d68661..1fe1b7476 100644 --- a/src/simlin-engine/src/db/combined_fragment_proptest.rs +++ b/src/simlin-engine/src/db/combined_fragment_proptest.rs @@ -133,10 +133,19 @@ fn build_member(spec: &MemberSpec) -> PerVarBytecodes { .collect(); let static_views: Vec = (0..spec.n_views) .map(|i| SymbolicStaticView { + // Cycle the three VARIABLE-backed bases (GH #995 added the two + // snapshot regions): all three name a variable rather than a merged + // resource, so only `Temp` is renumbered and the interleaved merge + // must carry each of the others across untouched. base: if spec.n_temps > 0 && i % 2 == 0 { SymStaticViewBase::Temp((i % spec.n_temps) as u32) } else { - SymStaticViewBase::Var(vref(&name, i)) + let var = vref(&name, i); + match i % 3 { + 0 => SymStaticViewBase::Var(var), + 1 => SymStaticViewBase::PrevVar(var), + _ => SymStaticViewBase::InitialVar(var), + } }, dims: smallvec::smallvec![(tag * 10 + i) as u16], strides: smallvec::smallvec![1], @@ -300,7 +309,11 @@ fn temp_uses(code: &[SymbolicOpcode], views: &[SymbolicStaticView]) -> Vec<(usiz }); match &view.base { SymStaticViewBase::Temp(id) => *id, - SymStaticViewBase::Var(_) => return None, + // Variable-backed bases -- `curr` and the two snapshot + // regions -- name no temp channel. + SymStaticViewBase::Var(_) + | SymStaticViewBase::PrevVar(_) + | SymStaticViewBase::InitialVar(_) => return None, } } _ => return None, diff --git a/src/simlin-engine/src/db/dep_graph.rs b/src/simlin-engine/src/db/dep_graph.rs index c88454cc7..7a3a90356 100644 --- a/src/simlin-engine/src/db/dep_graph.rs +++ b/src/simlin-engine/src/db/dep_graph.rs @@ -685,7 +685,13 @@ pub(crate) fn dt_cycle_sccs_engine_consistent( /// addressable-only set, just made explicit in symbolic space.) fn static_view_element_offsets(view: &crate::compiler::symbolic::SymbolicStaticView) -> Vec { let base_elem = match &view.base { - crate::compiler::symbolic::SymStaticViewBase::Var(v) => v.element_offset, + // All three variable-backed regions share `curr`'s slot numbering, so + // the element set a view addresses is the same for each; WHETHER that + // set counts as an ordering edge is the caller's decision (see the + // `PushStaticView` arm of `symbolic_phase_element_order`). + crate::compiler::symbolic::SymStaticViewBase::Var(v) + | crate::compiler::symbolic::SymStaticViewBase::PrevVar(v) + | crate::compiler::symbolic::SymStaticViewBase::InitialVar(v) => v.element_offset, // A temp-backed view threads scratch storage, not a current-value // recurrence read (the prior `collect_read_slots` likewise did not // treat `TempArray*` as a read). @@ -982,9 +988,28 @@ fn symbolic_phase_element_order( // resolves). An out-of-range `view_id` is a malformed // fragment (loud-safe: unresolved). let view = frag.static_views.get(*view_id as usize)?; - if let crate::compiler::symbolic::SymStaticViewBase::Var(v) = &view.base { + // A view's base carries the SAME lagged/current + // classification as the scalar read opcodes above, and for + // the same reasons (GH #995 gave `PREVIOUS`/`INIT` array + // forms, which lower to a view over the snapshot region + // instead of to `SymLoadPrev`/`SymLoadInitial`): a `curr` + // view is a current-value read; a PREVIOUS view is a + // prior-timestep snapshot and is an ordering edge in + // NEITHER phase; an INIT view is an initial-snapshot read, + // an edge in `SccPhase::Initial` only. + use crate::compiler::symbolic::SymStaticViewBase as ViewBase; + let read_name = match &view.base { + ViewBase::Var(v) => Some(&v.name), + ViewBase::InitialVar(v) + if matches!(phase, crate::db::SccPhase::Initial) => + { + Some(&v.name) + } + ViewBase::InitialVar(_) | ViewBase::PrevVar(_) | ViewBase::Temp(_) => None, + }; + if let Some(name) = read_name { for elem in static_view_element_offsets(view) { - pending_reads.insert((v.name.clone(), elem)); + pending_reads.insert((name.clone(), elem)); } } } diff --git a/src/simlin-engine/src/db/dep_graph_tests.rs b/src/simlin-engine/src/db/dep_graph_tests.rs index a19a3c01d..f4026d659 100644 --- a/src/simlin-engine/src/db/dep_graph_tests.rs +++ b/src/simlin-engine/src/db/dep_graph_tests.rs @@ -3394,3 +3394,141 @@ fn initials_runlist_is_sorted_topological_order() { is not sorting its candidate set before topo_sort_str" ); } + +// ── the SNAPSHOT-VIEW arms of the element-graph read classification ───── +// +// `symbolic_phase_element_order`'s `PushStaticView` arm classifies a view's +// base exactly as the scalar read opcodes above it are classified, and GH #995 +// gave `PREVIOUS`/`INIT` array forms that lower to a VIEW instead of to +// `SymLoadPrev`/`SymLoadInitial`. Three arms, and the two fixtures below cover +// the two that the scalar tests cannot reach through a view: +// +// * `Var` (a current-value read) -- covered by every existing element-graph +// test, since that is what an ordinary array reference lowers to. +// * `PrevVar` -- an ordering edge in NEITHER phase. +// [`a_prev_view_is_not_a_same_step_element_edge`] exercises the `Dt` half +// only (an array `PREVIOUS` in an INITIAL equation reads the fallback, so no +// initials fixture can observe an ordering difference); the Initial half +// rests on the same argument. This is the view analogue of +// [`resolve_dt_sample_if_true_shaped_scc_resolves_despite_previous_self_read`], +// which pins the same rule for the scalar `SymLoadPrev`. +// * `InitialVar` -- an ordering edge in `SccPhase::Initial` ONLY. +// [`an_init_view_is_an_init_phase_element_edge`] covers the Initial half +// (where the edge must be present); the Dt half is the same +// "an initial-snapshot read is not a current-dt value" argument the scalar +// `SymLoadInitial` arm carries, and no fixture here separates it. + +/// A `PREVIOUS` VIEW must contribute no element edge, in either phase. +/// +/// `x` is a per-element forward recurrence whose FIRST element additionally +/// reads a whole-array `PREVIOUS` of `x` itself. The whole-variable relation +/// sees `x -> x` through the un-lagged `x[t1]`/`x[t2]` references, so the SCC IS +/// identified and the element graph is consulted; that graph is the chain +/// `(x,0) -> (x,1) -> (x,2)`, which is acyclic and resolves. +/// +/// `SUM(PREVIOUS(x[*]))` lowers to a `PushStaticView` over a `PrevVar` base +/// spanning EVERY element of `x`, so counting it as a current-value read mints +/// `(x,e) -> (x,0)` for every `e` -- a `(x,0) -> (x,0)` self-loop among them. +/// The verdict then flips to unresolved and the model is rejected with a +/// `CircularDependency` it does not have. Both halves are asserted, so a +/// mutation that makes the base unconditional reds here rather than passing. +#[test] +fn a_prev_view_is_not_a_same_step_element_edge() { + use crate::db::SccPhase; + + let project = TestProject::new("prev_view_element_edge") + .named_dimension("t", &["t1", "t2", "t3"]) + .array_with_ranges( + "x[t]", + vec![ + ("t1", "1 + SUM(PREVIOUS(x[*]))"), + ("t2", "x[t1] + 1"), + ("t3", "x[t2] + 1"), + ], + ); + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let result = sync_from_datamodel(&db, &datamodel); + let model = result.models["main"].source; + + let res = resolve_recurrence_sccs(&db, model, result.project, SccPhase::Dt); + assert!( + !res.has_unresolved, + "the element graph is the acyclic chain (x,0)->(x,1)->(x,2); the only \ + thing that can make it look cyclic is the whole-array PREVIOUS view in \ + (x,0)'s segment being counted as a current-value read" + ); + assert_eq!(res.resolved.len(), 1, "exactly one resolved SCC ({{x}})"); + assert_eq!(res.resolved[0].phase, SccPhase::Dt); + + // The user-visible consequence, so the rule is pinned at both levels: the + // model compiles and runs. `x[t1] = 1 + SUM(prev(x))` is 1 at t=0 (the + // PREVIOUS fallback makes the sum 0) and 1 + (1+2+3) = 7 at t=1. + project.assert_compiles_incremental(); + let all = project.run_vm_incremental(); + for (elem, expected) in [("t1", [1.0, 7.0]), ("t2", [2.0, 8.0]), ("t3", [3.0, 9.0])] { + let series = all + .get(&format!("x[{elem}]")) + .unwrap_or_else(|| panic!("x[{elem}] missing")); + assert_eq!(series.as_slice(), &expected, "x[{elem}]"); + } +} + +/// An `INIT` VIEW must contribute an element edge in the INITIAL phase. +/// +/// This is the direction the `PrevVar` fixture cannot test: an arm that ADDS an +/// edge can only be caught by a fixture the edge makes CYCLIC, since dropping it +/// makes graphs resolve more often rather than less. +/// +/// `s` is an arrayed stock (so its init equation is separate from its dt +/// equation, and only the init relation is exercised) whose `t2` arm reduces a +/// whole-array `INIT` of `s` itself. `INIT`-refs are NOT stripped from +/// `initial_deps`, so the `s -> s` init self-loop is identified; the element +/// graph then holds `(s,1) -> (s,1)` through that view, which is a genuine +/// same-element init cycle and must stay UNRESOLVED (loud-safe) and surface as +/// `CircularDependency`. +/// +/// Drop the `InitialVar`-in-`Initial` arm and the self-loop disappears: the SCC +/// "resolves" and the model compiles to a per-element order that reads `s`'s own +/// initial value before it exists. +#[test] +fn an_init_view_is_an_init_phase_element_edge() { + use crate::db::SccPhase; + + let datamodel = arrayed_init_recurrence_stock_project(vec![ + ("t1", "1"), + ("t2", "SUM(INIT(s[*]))"), + ("t3", "s[t2] + 1"), + ]); + let db = SimlinDb::default(); + let result = sync_from_datamodel(&db, &datamodel); + let model = result.models["main"].source; + + let res = resolve_recurrence_sccs(&db, model, result.project, SccPhase::Initial); + assert!( + res.has_unresolved, + "(s,1)'s init segment reads a whole-array INIT view of `s`, which is a \ + genuine same-element init cycle: the SCC must stay unresolved" + ); + assert!( + res.resolved.is_empty(), + "nothing to resolve -- the element graph is cyclic: {:?}", + res.resolved + ); + + // And the user-visible verdict, which is what the loud-safe posture buys. + let diags = crate::db::collect_all_diagnostics(&db, result.project); + let circular = diags.iter().any(|d| { + d.variable.as_deref() == Some("s") + && matches!( + &d.error, + crate::db::DiagnosticError::Model(e) + if e.code == crate::common::ErrorCode::CircularDependency + ) + }); + assert!( + circular, + "the unresolved init SCC must reach the user as a CircularDependency on \ + 's'; got: {diags:?}" + ); +} diff --git a/src/simlin-engine/src/db/diagnostic_tests.rs b/src/simlin-engine/src/db/diagnostic_tests.rs index 63e759448..6407537c3 100644 --- a/src/simlin-engine/src/db/diagnostic_tests.rs +++ b/src/simlin-engine/src/db/diagnostic_tests.rs @@ -3043,13 +3043,20 @@ fn variable_error_fields_are_the_lowering_channel() { /// variable and carrying codegen's reason. /// /// The shape: an array-valued operand of an array builtin must be a *view* -/// over storage (`codegen::walk_expr_as_view` accepts only -/// `StaticSubscript | TempArray | Var | Subscript`). `PREVIOUS(vals[d])` is an -/// `Expr::App`, so `VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)` reaches codegen -/// intact and is rejected with `Cannot push view for expression type -/// Discriminant(7)`. Nothing about this equation involves LTM -- it is the -/// same defect 244 LTM fragments on C-LEARN hit, reached from a model a user -/// can type. +/// over storage (`codegen::walk_expr_as_view`). `ALLOCATE AVAILABLE`'s +/// priority-profile argument is the one view position `compiler::array_operand` +/// deliberately declines to materialize -- its view is re-expanded by +/// `context::expand_pp_view_for_allocate`, which only understands a direct +/// variable reference -- so a computed profile lowers cleanly and is rejected by +/// codegen with `Cannot push view for expression type`. Nothing about this +/// equation involves LTM; it is reached from a model a user can type. +/// +/// (The shape this test was originally written against, +/// `VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)`, compiles since GH #995's Phase C3 +/// gave `PREVIOUS` a snapshot-buffer view. What this test is ABOUT is +/// attribution, not that particular construct, so it moved to a construct that +/// is still refused -- and `array_operand_materialization_tests:: +/// deliberately_unmaterialized_positions` pins the refusal itself.) /// /// Before this was wired up the failure was INVISIBLE: `compile_phase` is /// `compile_phase_to_per_var_bytecodes(..)`, which is `.ok()` over the @@ -3068,8 +3075,15 @@ fn variable_error_fields_are_the_lowering_channel() { fn codegen_rejection_of_an_ordinary_variable_names_the_variable_and_its_reason() { let project = crate::test_common::TestProject::new("codegen_reject") .named_dimension("d", &["e1", "e2", "e3"]) - .array_aux("vals[d]", "10") - .array_aux("out[d]", "VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)") + .indexed_dimension("xp", 4) + .array_const("request[d]", 10.0) + .array_const("pp[d,xp]", 1.0) + .array_const("pp_bump[d,xp]", 0.0) + .scalar_const("supply", 35.0) + .array_aux( + "out[d]", + "allocate_available(request[d], pp[d,1] + pp_bump[d,1], supply)", + ) .build_datamodel(); let db = SimlinDb::default(); @@ -3159,9 +3173,9 @@ fn a_model_that_compiles_gains_no_codegen_diagnostic() { /// /// `compile_var_fragment` calls `compile_phase` at three sites -- initials, /// flows, stocks -- and each discards its codegen `Err` independently, so one -/// wired-up site says nothing about the others. The test above covers flows -/// (the phase all 244 C-LEARN LTM fragments take); this one covers initials -/// via an arrayed stock whose INITIAL equation carries the same refused shape. +/// wired-up site says nothing about the others. The test above covers flows; +/// this one covers initials via an arrayed stock whose INITIAL equation carries +/// the same refused shape. /// /// The stocks phase is deliberately NOT covered, and its call site admits TWO /// kinds of variable -- `(is_stock || is_module) && membership.stocks` -- so @@ -3193,10 +3207,14 @@ fn a_model_that_compiles_gains_no_codegen_diagnostic() { fn codegen_rejection_in_the_initials_phase_is_attributable_too() { let project = crate::test_common::TestProject::new("codegen_reject_init") .named_dimension("d", &["e1", "e2", "e3"]) - .array_aux("vals[d]", "10") + .indexed_dimension("xp", 4) + .array_const("request[d]", 10.0) + .array_const("pp[d,xp]", 1.0) + .array_const("pp_bump[d,xp]", 0.0) + .scalar_const("supply", 35.0) .array_stock( "lvl[d]", - "VECTOR SORT ORDER(PREVIOUS(vals[d]), 1)", + "allocate_available(request[d], pp[d,1] + pp_bump[d,1], supply)", &[], &[], None, diff --git a/src/simlin-engine/src/db/element_graph_tests.rs b/src/simlin-engine/src/db/element_graph_tests.rs index 6742d1eff..bcd56fd8d 100644 --- a/src/simlin-engine/src/db/element_graph_tests.rs +++ b/src/simlin-engine/src/db/element_graph_tests.rs @@ -1660,7 +1660,7 @@ fn element_graph_mapped_reverse_declared_bare_is_diagonal() { /// iterated-dim reference whose POSITIONAL mapping is declared only in the /// reverse direction (`Region→State`, i.e. on the source's dimension) now /// classifies `Bare` -- `classify_iterated_dim_shape`'s mapped arm gates on -/// the same `mapped_element_correspondence` data `expand_same_element` +/// the same correspondence data `expand_same_element` /// consults (both declaration directions), so the subscripted form gets the /// same DIAGONAL the bare form (`element_graph_mapped_reverse_declared_bare_is_diagonal`) /// already got, matching the compiler's `translate_via_mapping` (which @@ -1747,16 +1747,19 @@ fn element_graph_per_element_broadcast_is_pinned_diagonal() { assert_no_edge(&result, "pop[b,old]", "mid[b,x]"); } -/// GH #527: an EXPLICIT element-level mapping (here the different- -/// cardinality many-to-one `State{s1,s2,s3}→Region{a,b}`: s1↦a, s2↦a, -/// s3↦b) keeps the conservative BROADCAST, not a map-following diagonal: -/// the engine's executed A2A lowering resolves mapped references -/// positionally, ignoring the element map (this 3→2 model doesn't compile -/// at all -- GH #753 -- so the graph-level pin is all we can have here; -/// the same positional-execution inconsistency is why -/// `mapped_element_correspondence` declines element maps wholesale, see -/// its rustdoc gate). The broadcast is a superset of whatever the engine -/// would read, so no true edge can be missing. +/// GH #527: a many-to-one `State{s1,s2,s3}→Region{a,b}` element map read on +/// the ITERATED spelling (`x[State]`) keeps the conservative BROADCAST. +/// +/// The reason is CARDINALITY, not the map. This spelling folds its index to an +/// ordinal (GH #997), and `positional_correspondence` needs equal extents -- +/// there is no third `Region` element for `s3`'s ordinal to read. The model +/// does not compile either (`mapped_reference_semantics_tests`' +/// `(ManyToOne, IteratedDim)` cell is a refusal, GH #753), so a graph-level pin +/// is all this shape can have; the broadcast is a superset of whatever the +/// engine would read, so no true edge can be missing. The map-following +/// spelling of the SAME cardinality is +/// `element_graph_many_to_one_mapped_read_emits_the_map_rows`, which compiles +/// and gets the map's rows. #[test] fn element_graph_mapped_element_map_stays_broadcast() { let project = TestProject::new("mapped_element_map_3_to_2") @@ -1826,21 +1829,20 @@ fn element_graph_mapped_diagonal_with_broadcast_dim() { /// On THIS fixture's spelling the executed A2A lowering resolves the /// reference POSITIONALLY, ignoring the explicit element map (target[s1] = /// x[a], the map notwithstanding), so a map-following diagonal would DROP -/// the true positionally-read edges -- which is why -/// `mapped_element_correspondence` declines explicit element maps -/// (conservative broadcast). Removing that decline reds this test by -/// dropping the `x[a] -> target[s1]` edge, the direction the LTM contract -/// forbids. +/// the true positionally-read edges -- which is why `expand_same_element` +/// projects the UNION of the two spellings' diagonals rather than picking one +/// (GH #997). Emitting only the map's diagonal reds this test by dropping the +/// `x[a] -> target[s1]` edge, the direction the LTM contract forbids. /// /// The qualifier is load-bearing and was missing: this claim is NOT /// universal. `x[State]` spells the dimension the equation ITERATES, which /// `ast::expr3` folds to an ordinal that indexes `x`'s storage raw; a /// subscript naming a NON-active dimension instead reaches /// `translate_via_mapping` and DOES follow the element map. Both spellings -/// are real and both ship -- see `DimensionsContext::mapped_element_correspondence` -/// for the fork, the Vensim `Ref.vdf` evidence that map-following is correct -/// where it happens, and why the decline stays anyway. This fixture builds -/// only the positional spelling, so it constrains only that half. +/// are real and both ship -- see `db::analysis::bare_reference_correspondence` +/// for the fork and the Vensim `Ref.vdf` evidence that map-following is correct +/// where it happens. This fixture builds only the positional spelling, so it +/// constrains only that half. /// /// The test derives the implied edges from the run itself, so it keeps /// passing if execution on this spelling later changes. @@ -1959,14 +1961,25 @@ fn element_graph_mapped_sliced_reducer_routes_through_remapped_agg() { } } -/// GH #534 (conservative gate): a sliced reducer over an EXPLICIT -/// element-mapped pair stays un-hoisted -- the engine's executed A2A -/// lowering resolves mapped references positionally, ignoring element maps -/// (GH #756), so `mapped_element_correspondence` declines and the reference -/// keeps the conservative full cross-product (a superset of the true reads). -/// No agg node appears in the element graph. +/// GH #997 (flipped from the GH #534-era conservative pin): a sliced reducer +/// over an EXPLICIT element-mapped pair is hoisted, and its slots are remapped +/// POSITIONALLY -- ignoring the declared element map. +/// +/// This test asserted the conservative cross-product until GH #997, on the +/// reasoning that the single correspondence declined the pair. The +/// decline was never a claim about this spelling: `matrix[State, *]` names the +/// dimension the equation ITERATES, and `mapped_reference_semantics_tests`' +/// `(Permuted, IteratedDim)` cell measures such a reference reading by ordinal +/// against the VM -- the map is not consulted. One function served both +/// spellings and could not answer either, so it answered neither; now +/// `positional_correspondence` answers this one and the reducer hoists with the +/// slots execution actually reads. +/// +/// The element map here (s1↦r2, s2↦r1) is the REVERSE of the positional +/// diagonal, so the two rules disagree on every slot and the assertions below +/// distinguish them. #[test] -fn element_graph_element_mapped_sliced_reducer_stays_cross_product() { +fn element_graph_element_mapped_sliced_reducer_remaps_positionally() { let project = TestProject::new("element_mapped_sliced") .named_dimension("Region", &["r1", "r2"]) .named_dimension("D2", &["x", "y"]) @@ -1985,12 +1998,27 @@ fn element_graph_element_mapped_sliced_reducer_stays_cross_product() { ); let result = element_edges(&project); + let agg = "$\u{205A}ltm\u{205A}agg\u{205A}0"; - // Conservative cross-product: every matrix row feeds every growth slot. + // POSITIONAL slots: `Region`'s first element feeds `State`'s first slot, + // which is what the ordinal fold reads. The declared map says the opposite + // (s1↦r2), so a map-following remap would land every row on the other slot. + assert_edge(&result, "matrix[r1,x]", &format!("{agg}[s1]")); + assert_edge(&result, "matrix[r1,y]", &format!("{agg}[s1]")); + assert_edge(&result, "matrix[r2,x]", &format!("{agg}[s2]")); + assert_edge(&result, "matrix[r2,y]", &format!("{agg}[s2]")); + assert_no_edge(&result, "matrix[r1,x]", &format!("{agg}[s2]")); + assert_no_edge(&result, "matrix[r2,x]", &format!("{agg}[s1]")); + // The agg fans into the target diagonally on the shared State axis. + assert_edge(&result, &format!("{agg}[s1]"), "growth[s1]"); + assert_edge(&result, &format!("{agg}[s2]"), "growth[s2]"); + assert_no_edge(&result, &format!("{agg}[s1]"), "growth[s2]"); + // The reference is fully routed through the agg: the pre-#997 conservative + // cross-product is gone. for r in ["r1", "r2"] { for d2 in ["x", "y"] { for s in ["s1", "s2"] { - assert_edge( + assert_no_edge( &result, &format!("matrix[{r},{d2}]"), &format!("growth[{s}]"), @@ -1998,21 +2026,12 @@ fn element_graph_element_mapped_sliced_reducer_stays_cross_product() { } } } - // No agg node was minted for the element-mapped sliced reducer. - assert!( - !result - .edges - .keys() - .any(|k| k.starts_with("$\u{205A}ltm\u{205A}agg\u{205A}")), - "an element-mapped sliced reducer must not route through an agg node; edges: {:?}", - result.edges.keys().collect::>() - ); } /// GH #757 (flipped from the GH #534-era conservative pin): a sliced /// reducer whose POSITIONAL mapping is declared only in the REVERSE /// direction (on the source's `Region` toward `State`) is now hoisted -- -/// `classify_axis_access` gates on `mapped_element_correspondence`, which +/// `classify_axis_access` gates on `positional_correspondence`, which /// accepts both declaration directions -- so the element graph routes it /// through the remapped agg slots exactly like the forward-declared twin. #[test] @@ -2217,3 +2236,377 @@ fn element_graph_projection_feeder_routes_by_own_slice() { assert_edge(&result, &format!("{agg}[r2]"), "out[r2]"); assert_no_edge(&result, &format!("{agg}[r1]"), "out[r2]"); } + +// ===== GH #997: the class-D shape -- a dep read through an element-mapped +// axis, spelled with the SOURCE's own dimension name ===== +// +// The four fixtures below are the shapes the executed rule can meet on this +// spelling, derived from `mapped_reference_semantics_tests`' mapping-kind +// enumeration rather than sampled: a single-axis many-to-one map (C-LEARN's), +// an equal-cardinality permuted map, a pair sharing element names (where NAME +// identity must beat the map), and an ambiguous pairing (which declines). +// `element_graph_element_mapped_sliced_reducer_remaps_positionally` covers the +// same mapping kinds on the OTHER spelling. + +/// The C-LEARN shape: three `Aggregated Regions` elements mapped onto seven +/// `COP` ones, read as `aggregated[Aggregated Regions]` inside a +/// `COP`-iterating equation. +/// +/// The subscript names the SOURCE's own dimension, which +/// `compiler::subscript::normalize_subscripts3` turns into an +/// `IndexOp::ActiveDimRef` and `build_view_from_ops` resolves through the +/// declared element map (`mapped_reference_semantics_tests`' `SourceOwnDim` +/// row, measured against the VM at exactly this cardinality). The element graph +/// must therefore emit the map's rows -- three source nodes fanning out over +/// seven target elements -- and NOT the 3x7 cross-product it emitted while the +/// reference classified `DynamicIndex`. +#[test] +fn element_graph_many_to_one_mapped_read_emits_the_map_rows() { + let project = TestProject::new("class_d_many_to_one") + .named_dimension("cop", &["c1", "c2", "c3", "c4"]) + .named_dimension_with_element_mapping( + "agg", + &["a1", "a2"], + "cop", + &[("a1", "c1"), ("a1", "c2"), ("a2", "c3"), ("a2", "c4")], + ) + .array_aux_direct("aggregated", vec!["agg".into()], "1", None) + .array_aux_direct("target", vec!["cop".into()], "aggregated[agg] * 2", None); + + let result = element_edges(&project); + + for (src, targets) in [("a1", ["c1", "c2"]), ("a2", ["c3", "c4"])] { + for t in targets { + assert_edge( + &result, + &format!("aggregated[{src}]"), + &format!("target[{t}]"), + ); + } + } + // The off-map pairs are what the pre-#997 cross-product added. + for (src, targets) in [("a1", ["c3", "c4"]), ("a2", ["c1", "c2"])] { + for t in targets { + assert_no_edge( + &result, + &format!("aggregated[{src}]"), + &format!("target[{t}]"), + ); + } + } +} + +/// An equal-cardinality PERMUTED element map: the diagonal must follow the map, +/// not the ordinal. This is the row that separates the two rules -- both are +/// one-to-one here, so only the element names distinguish them. +#[test] +fn element_graph_permuted_mapped_read_follows_the_map_not_the_ordinal() { + let project = TestProject::new("class_d_permuted") + .named_dimension("cop", &["c1", "c2"]) + .named_dimension_with_element_mapping( + "agg", + &["a1", "a2"], + "cop", + &[("a1", "c2"), ("a2", "c1")], + ) + .array_aux_direct("aggregated", vec!["agg".into()], "1", None) + .array_aux_direct("target", vec!["cop".into()], "aggregated[agg] * 2", None); + + let result = element_edges(&project); + + assert_edge(&result, "aggregated[a2]", "target[c1]"); + assert_edge(&result, "aggregated[a1]", "target[c2]"); + assert_no_edge(&result, "aggregated[a1]", "target[c1]"); + assert_no_edge(&result, "aggregated[a2]", "target[c2]"); +} + +/// NAME identity beats the declared element map. +/// +/// `agg` and `cop` declare the same element names in a different order, and the +/// map is a third permutation, so all three candidate answers are distinct. +/// `build_view_from_ops` looks the active element's own name up on the source +/// axis BEFORE consulting any mapping, so `target[e1]` reads `aggregated[e1]` +/// -- pinned against the VM by `mapped_reference_semantics_tests`' +/// `SharedElementNames` row. Vensim's Example 3 subrange idiom makes this an +/// ordinary shape rather than an oddity. +#[test] +fn element_graph_mapped_read_resolves_shared_element_names_by_name() { + let project = TestProject::new("class_d_shared_names") + .named_dimension("cop", &["e1", "e2", "e3"]) + .named_dimension_with_element_mapping( + "agg", + &["e3", "e1", "e2"], + "cop", + &[("e3", "e2"), ("e1", "e3"), ("e2", "e1")], + ) + .array_aux_direct("aggregated", vec!["agg".into()], "1", None) + .array_aux_direct("target", vec!["cop".into()], "aggregated[agg] * 2", None); + + let result = element_edges(&project); + + for e in ["e1", "e2", "e3"] { + assert_edge( + &result, + &format!("aggregated[{e}]"), + &format!("target[{e}]"), + ); + } + // The map's own diagonal (e1 -> e3) and the positional one (cop's first + // element -> agg's first, e3) are both wrong here. + assert_no_edge(&result, "aggregated[e1]", "target[e3]"); + assert_no_edge(&result, "aggregated[e3]", "target[e1]"); +} + +/// AMBIGUITY declines, keeping the conservative cross-product. +/// +/// `agg` maps to BOTH of the target's iterated dimensions, so +/// `DimensionsContext::mapped_read_partner_dim` cannot say which axis +/// `aggregated[agg]` is iterated over. Execution breaks the tie by position; +/// a describer that copied that would attribute influence along edges chosen by +/// declaration order, so the reference keeps its pre-#997 shape and the element +/// graph keeps the superset. +#[test] +fn element_graph_ambiguous_mapped_read_declines_to_the_cross_product() { + let project = TestProject::new("class_d_ambiguous") + .named_dimension("cop", &["c1", "c2"]) + .named_dimension("county", &["k1", "k2"]) + .named_dimension_with_mappings( + "agg", + &["a1", "a2"], + &[ + ("cop", &[("a1", "c1"), ("a2", "c2")]), + ("county", &[("a1", "k2"), ("a2", "k1")]), + ], + ) + .array_aux_direct("aggregated", vec!["agg".into()], "1", None) + .array_aux_direct( + "target", + vec!["cop".into(), "county".into()], + "aggregated[agg] * 2", + None, + ); + + let result = element_edges(&project); + + for a in ["a1", "a2"] { + for c in ["c1", "c2"] { + for k in ["k1", "k2"] { + assert_edge( + &result, + &format!("aggregated[{a}]"), + &format!("target[{c},{k}]"), + ); + } + } + } + + // Attribution: the cross-product above is caused by the AMBIGUITY, not by + // anything else about the fixture. Drop one of the two mappings and the + // same model gets the `cop` diagonal. + let unambiguous = TestProject::new("class_d_one_partner") + .named_dimension("cop", &["c1", "c2"]) + .named_dimension("county", &["k1", "k2"]) + .named_dimension_with_element_mapping( + "agg", + &["a1", "a2"], + "cop", + &[("a1", "c1"), ("a2", "c2")], + ) + .array_aux_direct("aggregated", vec!["agg".into()], "1", None) + .array_aux_direct( + "target", + vec!["cop".into(), "county".into()], + "aggregated[agg] * 2", + None, + ); + let result = element_edges(&unambiguous); + for k in ["k1", "k2"] { + assert_edge(&result, "aggregated[a1]", &format!("target[c1,{k}]")); + assert_no_edge(&result, "aggregated[a1]", &format!("target[c2,{k}]")); + } +} + +/// The premise that forced `expand_same_element` to emit a UNION rather than +/// pick a rule, reached through the production pipeline. +/// +/// A structural flow-to-stock edge is labelled `RefShape::Bare` by +/// `model_edge_shapes` with no AST reference behind it -- a stock's equation +/// holds only its initial value, so the flow's name never appears in it. The +/// flow reference is resolved by `Context::fold_flows` through +/// `get_implicit_subscript_off`, i.e. name-first then through the declared +/// element map (`mapped_reference_semantics_tests`' `StockFlow` row measures it +/// against the VM). An in-equation `Bare` reference on the same pair resolves +/// POSITIONALLY. Both wear one shape, so the element graph must cover both. +/// +/// This fixture builds BOTH on one element-mapped pair -- `level[State]` +/// integrating a `Region`-declared flow, and `readout[State]` reading the same +/// flow in an equation -- and asserts the union appears on each. +/// +/// THREE elements, with a 3-CYCLE map (s1↦b, s2↦c, s3↦a) against the positional +/// diagonal (s1↦a, s2↦b, s3↦c). Two is not enough: at two elements the union of +/// two disjoint permutations IS the full broadcast, so the test could not tell a +/// union from a decline -- deleting `bare_reference_correspondence` reds ten +/// other tests and would leave a 2-element version of this one green. At three +/// it is a strict subset: each source element feeds exactly TWO of the three +/// targets, and the third is asserted ABSENT. +/// +/// The hand-built `bare_mapped_dims_positional_diagonal_element_map_broadcast` +/// exercises `emit_edges_for_reference` directly with a `RefShape::Bare` it +/// supplies itself; this one derives the shape through `model_edge_shapes` and +/// the real `model_element_causal_edges` query, so it pins that production +/// actually classifies a structural stock edge that way. +#[test] +fn element_graph_flow_to_stock_across_an_element_map_gets_the_union() { + let project = TestProject::new("mapped_flow_to_stock") + .named_dimension("Region", &["a", "b", "c"]) + .named_dimension_with_element_mapping( + "State", + &["s1", "s2", "s3"], + "Region", + &[("s1", "b"), ("s2", "c"), ("s3", "a")], + ) + .array_flow("feed[Region]", "1", None) + .array_stock("level[State]", "0", &["feed"], &[], None) + .array_aux_direct("readout", vec!["State".into()], "feed * 2", None); + + let result = element_edges(&project); + + // Non-vacuity: the structural edge must really be there, and it is not one + // the AST walker could have produced -- `level`'s equation is `0`. + assert!( + result.edges.contains_key("feed[a]"), + "the structural flow->stock edge must reach the element graph; got: {:?}", + result.edges.keys().collect::>() + ); + + // Per target element: the POSITIONAL source, the MAPPED source, and the + // third element that neither spelling reads. + for target in ["level", "readout"] { + for (elem, positional, mapped, absent) in [ + ("s1", "a", "b", "c"), + ("s2", "b", "c", "a"), + ("s3", "c", "a", "b"), + ] { + assert_edge( + &result, + &format!("feed[{positional}]"), + &format!("{target}[{elem}]"), + ); + assert_edge( + &result, + &format!("feed[{mapped}]"), + &format!("{target}[{elem}]"), + ); + assert_no_edge( + &result, + &format!("feed[{absent}]"), + &format!("{target}[{elem}]"), + ); + } + } +} + +/// [`super::mapped_pair_projects_uniquely`], row by row, derived from the four +/// cases its own rustdoc enumerates rather than sampled. +/// +/// The function decides whether a `Bare` edge across a mapped pair may carry an +/// ARRAYED link score, and it is a STRICTER question than which element edges +/// exist -- a distinction no other test isolates. Two of the four rows also have +/// end-to-end pins (`a_disagreeing_mapped_pair_is_denied_the_arrayed_score` and +/// its agreeing companion in `tests/integration/ltm_array_agg.rs`), but the +/// MANY-TO-ONE admit row -- the one the rustdoc credits with keeping C-LEARN's +/// class-D edges working -- has none: C-LEARN's class-D references classify +/// `PerElement`, so they never consult this gate at all. Without this test that +/// row is an unexercised claim. +#[test] +fn mapped_pair_projects_uniquely_enumeration() { + use crate::common::CanonicalDimensionName; + use crate::dimensions::DimensionsContext; + + let region = |elems: &[&str]| { + crate::datamodel::Dimension::named( + "Region".to_string(), + elems.iter().map(|e| e.to_string()).collect(), + ) + }; + let state = |elems: &[&str]| { + crate::datamodel::Dimension::named( + "State".to_string(), + elems.iter().map(|e| e.to_string()).collect(), + ) + }; + let with_map = |mut d: crate::datamodel::Dimension, pairs: &[(&str, &str)]| { + d.mappings = vec![crate::datamodel::DimensionMapping { + target: "Region".to_string(), + element_map: pairs + .iter() + .map(|(a, b)| (a.to_string(), b.to_string())) + .collect(), + }]; + d + }; + let projects = |dims: &[crate::datamodel::Dimension]| -> bool { + let ctx = DimensionsContext::from(dims); + super::mapped_pair_projects_uniquely( + &ctx, + &CanonicalDimensionName::from_raw("State"), + &CanonicalDimensionName::from_raw("Region"), + ) + }; + + // ADMIT 1 -- a plain positional mapping. The two spellings coincide, which + // is why every pre-GH #997 mapped edge kept its score. + let mut positional = state(&["s1", "s2"]); + positional.set_maps_to("Region".to_string()); + assert!( + projects(&[positional, region(&["a", "b"])]), + "a positional mapping must admit" + ); + + // ADMIT 2 -- a MANY-TO-ONE element map (C-LEARN's shape). The positional + // rule declines outright at unequal cardinality, leaving the executed rule + // alone in the union, so every entry is still a singleton. + assert!( + projects(&[ + with_map( + state(&["s1", "s2", "s3"]), + &[("s1", "a"), ("s2", "a"), ("s3", "b")], + ), + region(&["a", "b"]), + ]), + "a many-to-one element map must admit -- the positional rule contributes \ + nothing to the union at unequal cardinality" + ); + + // REFUSE 1 -- an equal-cardinality PERMUTED element map. Both rules answer, + // and they disagree, so a target slot would be shared by two source + // elements. + assert!( + !projects(&[ + with_map(state(&["s1", "s2"]), &[("s1", "b"), ("s2", "a")]), + region(&["a", "b"]), + ]), + "an equal-cardinality permuted element map must refuse" + ); + + // REFUSE 2 -- a pair SHARING element names in a different order. Here the + // executed rule stops at name identity while the positional one reads by + // ordinal; no element map is involved at all, which is what makes this a + // separate row rather than a restatement of the one above. + let mut shared = state(&["b", "a"]); + shared.set_maps_to("Region".to_string()); + assert!( + !projects(&[shared, region(&["a", "b"])]), + "a mapped pair sharing element names in a different order must refuse" + ); + + // And the boundary the two REFUSE rows sit against: shared names in the + // SAME order agree, so they admit. Without this the refusals could be read + // as "shared names always refuse". + let mut agreeing = state(&["a", "b"]); + agreeing.set_maps_to("Region".to_string()); + assert!( + projects(&[agreeing, region(&["a", "b"])]), + "shared element names in the SAME order agree on both rules and admit" + ); +} diff --git a/src/simlin-engine/src/db/fragment_char_tests.rs b/src/simlin-engine/src/db/fragment_char_tests.rs index e61366c7e..26b45f447 100644 --- a/src/simlin-engine/src/db/fragment_char_tests.rs +++ b/src/simlin-engine/src/db/fragment_char_tests.rs @@ -411,6 +411,8 @@ fn render_opcode(op: &SymbolicOpcode, literals: &[f64]) -> String { fn render_static_view(idx: usize, sv: &SymbolicStaticView) -> String { let base = match &sv.base { SymStaticViewBase::Var(v) => render_var_ref(v), + SymStaticViewBase::PrevVar(v) => format!("prev({})", render_var_ref(v)), + SymStaticViewBase::InitialVar(v) => format!("initial({})", render_var_ref(v)), SymStaticViewBase::Temp(id) => format!("temp{id}"), }; let sparse: Vec = sv diff --git a/src/simlin-engine/src/db/ltm/link_scores.rs b/src/simlin-engine/src/db/ltm/link_scores.rs index 9123adf49..b46dbfe15 100644 --- a/src/simlin-engine/src/db/ltm/link_scores.rs +++ b/src/simlin-engine/src/db/ltm/link_scores.rs @@ -71,10 +71,11 @@ use super::parse::{ltm_equation_dimensions, retarget_ltm_equation_dims}; /// iterates it -- so the projection is the identity and no dimension mapping is /// consulted. That is a measured property of one corpus, not of the language: a /// model reading a table declared over a DIFFERENT axis than the target iterates -/// needs the mapped element correspondence, which -/// `DimensionsContext::mapped_element_correspondence` declines for an explicit -/// element map. Such a reference lands in `dep_element_pins`' incomplete arm and -/// keeps today's loud drop. +/// needs a mapped element correspondence, and it gets the one its SPELLING earns +/// (GH #997) -- `positional_correspondence` for an index naming an iterated +/// dimension, `executed_read_correspondence` for one naming the holder's own. +/// A pair with no DECLARED correspondence either way still lands in +/// `dep_element_pins`' incomplete arm and keeps today's loud drop. fn pinnable_arrayed_deps( db: &dyn Db, source_vars: &HashMap, @@ -321,22 +322,44 @@ pub(super) fn link_score_dimensions( // the edge having a `Bare`-classified reference site -- the exact // condition under which `expand_same_element` emits the mapped DIAGONAL // element edges, so "score arrayed over the target's dims" ⟺ "element - // edges are the diagonal". Since GH #757 the classifier - // (`classify_iterated_dim_shape` via `classify_axis_access`) gates its - // mapped arm on the SAME `mapped_element_correspondence` data, BOTH - // declaration directions, so a positionally-mapped subscripted + // edges are the diagonal". The gate therefore consults + // `db::analysis::bare_reference_correspondence`, the same helper + // `expand_same_element` does, so the two cannot disagree about which + // pairs project; since GH #757 the classifier + // (`classify_iterated_dim_shape` via `classify_axis_access`) accepts + // BOTH declaration directions, so a positionally-mapped subscripted // reference (forward- or reverse-declared) classifies `Bare` and passes - // this gate with the diagonal it deserves. The remaining shapes the - // gate excludes are the ELEMENT-mapped pairs (declined by the - // GH #756 positional-only rule, classified `DynamicIndex`, cross-product - // element edges): retargeting such an edge's (Bare-named, since - // Wildcard/DynamicIndex collapse onto the Bare name) score to the - // target's dims would shape per-slot DIAGONAL partials that the - // off-diagonal loop links then read by target-element subscript -- - // silent wrong-slot values. Denied the retarget, such an edge instead - // takes the GH #758 loud skip in `emit_per_shape_link_scores` (no - // link-score variable, loop scores through the edge dropped, one - // Warning). A mixed edge (a Bare site AND + // this gate with the diagonal it deserves. + // + // Since GH #997 an ELEMENT-mapped pair can project too, but on a STRICTER + // condition than the element graph's: `mapped_pair_projects_uniquely`, not + // `bare_reference_correspondence(..).is_some()`. The element graph emits + // the UNION of the two spellings' diagonals, which is sound there because + // an extra edge is the safe direction. A SCORE is one arrayed variable with + // one slot per target element, and `ltm_finding::expand_a2a_link_offsets` + // maps every union edge for a target element onto that one slot -- so where + // the two rules DISAGREE, the phantom from-node reads the real edge's + // non-zero score out of the shared slot. A compilable, confidently wrong + // number is the outcome this repo treats as worse than none (GH #758), so + // the retarget is denied unless every target element's correspondence is a + // SINGLETON. + // + // What that admits: every mapping that projected before GH #997 (the two + // rules coincide on a positional mapping between disjointly-named + // dimensions) plus C-LEARN's many-to-one element map, where the positional + // rule declines outright and leaves the executed rule alone in the union. + // What it refuses: an equal-cardinality PERMUTED element map, and a pair + // sharing element names in a different order. + // + // What still does NOT reach here at all is a reference the classifier gave + // a non-`Bare` shape -- a genuinely dynamic index, a transposition -- + // whose element edges stay the conservative cross-product; retargeting one + // of those to the target's dims would shape per-slot DIAGONAL partials + // that the off-diagonal loop links then read by target-element subscript, + // i.e. silent wrong-slot values. + // Denied the retarget, such an edge instead takes the GH #758 loud skip + // in `emit_per_shape_link_scores` (no link-score variable, loop scores + // through the edge dropped, one Warning). A mixed edge (a Bare site AND // a DynamicIndex site on the same `(from, to)`) keeps the arrayed score // -- the Bare site needs it -- while its cross-product links still read // diagonal slots; that is the pre-existing mixed-shape conservatism @@ -352,9 +375,11 @@ pub(super) fn link_score_dimensions( |td: &crate::dimensions::Dimension, fd: &crate::dimensions::Dimension| -> bool { td.name() == fd.name() || (edge_has_bare_site - && dim_ctx - .mapped_element_correspondence(td.canonical_name(), fd.canonical_name()) - .is_some()) + && crate::db::analysis::mapped_pair_projects_uniquely( + dim_ctx, + td.canonical_name(), + fd.canonical_name(), + )) }; let dims_compatible = from_dims == *to_dims || to_dims @@ -2063,8 +2088,10 @@ pub(super) fn emit_unscoreable_conservative_edge_warning( use salsa::Accumulator; let msg = format!( "LTM link score for edge {from} -> {to} could not be computed: both variables \ - are arrayed but their dimensions do not correspond (e.g. an element-mapped or \ - unmapped dimension pair), so the conservative score has no compilable shape; \ + are arrayed but their dimensions do not correspond -- an unmapped pair, or a \ + mapped one whose two reference spellings read different source elements, which \ + cannot share one per-target-element score slot -- so the conservative score has \ + no compilable shape; \ this edge will have no link-score variable and feedback loops through it will \ not be scored" ); @@ -2380,8 +2407,11 @@ pub(crate) fn ltm_partial_equation_warning_message( dropped -- rather than emitted with the dep's dimension-name subscript \ left in a scalar fragment, which compiles to a helper that reads a \ constant 0 while the score itself still compiles (a wrong number, not \ - an absent one). The reachable cause is a dep read across an EXPLICIT \ - element map, which `mapped_element_correspondence` declines." + an absent one). The reachable cause is a dep whose axis has no \ + DECLARED correspondence to any dimension the target iterates -- two \ + dimensions sharing element names, which the simulation resolves by \ + name while the pin's axis allocation needs a name match or a \ + declared mapping." ), PartialEquationErrorKind::RankLikePartial => format!( "LTM link-score variable '{variable_name}' could not be generated: the \ @@ -2775,9 +2805,14 @@ pub(super) fn emit_per_shape_link_scores( // cross-product, so per-slot diagonal partials would be read at wrong // slots). Degrade loudly instead: one Warning naming the edge, no // link-score variable, and (via `unscoreable_edges`) no loop scores - // through it. The declined ELEMENT-mapped sliced reducers (GH #756; - // reverse-declared positional pairs are hoisted since GH #757) land - // here, as do disjoint-dim ApplyToAll-target references whose sites + // through it. Two families land here: the sliced reducers the + // correspondence declines (an UNDECLARED pair, a cardinality mismatch, or + // a `MappedRead` axis -- GH #997; a DECLARED mapping is hoisted in either + // direction since GH #757, an explicit element map included since #997), + // and -- also since GH #997 -- a mapped pair whose two spellings DISAGREE, + // which `mapped_pair_projects_uniquely` denies the arrayed retarget + // because the two would share one score slot. So do disjoint-dim + // ApplyToAll-target references whose sites // are not all FixedIndex (the GH #769 widening recovers the // FixedIndex-only ones) and incompatible-dim dynamic-index reducers -- // all previously warned zero-stubs. @@ -3904,7 +3939,7 @@ pub(super) fn emit_agg_to_target_link_scores( .enumerate() .find_map(|(target_pos, target_dim)| { dim_ctx - .mapped_element_correspondence(target_dim.canonical_name(), &result_canon) + .positional_correspondence(target_dim.canonical_name(), &result_canon) .map(|mapped_elements| (target_pos, target_dim, mapped_elements)) }) { @@ -4002,16 +4037,20 @@ pub(super) fn emit_agg_to_target_link_scores( .map(|axis| axis.result_dim.clone()) .collect(); let qualified = crate::ltm_augment::qualify_element_csv(&slot, &result_dims); + let axes: Vec<(String, String)> = result_dims + .iter() + .zip(qualified.split(',')) + .map(|(dim, elem)| (dim.name().to_string(), elem.to_string())) + .collect(); Some(DepElementPin { - axes: result_dims - .iter() - .zip(qualified.split(',')) - .map(|(dim, elem)| (dim.name().to_string(), elem.to_string())) - .collect(), // An agg's slot space IS its `result_dims`, and the projection // either covers all of them or returned `None` above, so a bare - // agg reference is always spellable here. - complete: true, + // agg reference is always spellable here. There is only one + // spelling to answer for -- an agg ident carries no declared + // dimension a subscript could name -- so the two rows are the + // same row. + bare_row: Some(axes.iter().map(|(_, elem)| elem.clone()).collect()), + axes, }) }; // Every ARRAYED agg referenced by the substituted equation needs a body diff --git a/src/simlin-engine/src/db/ltm/loops.rs b/src/simlin-engine/src/db/ltm/loops.rs index 26343cfdb..fe1f13ae0 100644 --- a/src/simlin-engine/src/db/ltm/loops.rs +++ b/src/simlin-engine/src/db/ltm/loops.rs @@ -413,33 +413,156 @@ pub(crate) fn read_slice_row_parts( AxisRead::Reduced { subset } => { Some((subset.clone().unwrap_or_else(|| elems.clone()), None)) } + // A `MappedRead` axis (GH #997) enumerates the TARGET dimension's + // elements rather than the source's, and pairs each with the source + // element it reads. That direction is load-bearing: the executed + // correspondence need not be injective -- C-LEARN maps three + // `Aggregated Regions` elements onto seven `COP` ones -- so + // enumerating the SOURCE side would owe several slots to one row, + // which the one-slot-per-row shape below cannot express. Walking the + // target side keeps (row, slot) a function, at the cost of repeating + // a row under different slots, which is exactly what a many-to-one + // read is. + AxisRead::MappedRead { dim, source_dim } => { + let target_dim = + dim_ctx.get(&crate::common::CanonicalDimensionName::from_raw(dim))?; + let corr = dim_ctx.executed_read_correspondence( + &crate::common::CanonicalDimensionName::from_raw(dim), + &crate::common::CanonicalDimensionName::from_raw(source_dim), + )?; + let slots = crate::ltm_augment::dimension_element_names(target_dim); + if slots.len() != corr.len() { + return None; + } + Some(( + corr.iter().map(|e| e.as_str().to_string()).collect(), + Some(slots), + )) + } }) .collect::>>()?; - // Cartesian product, accumulating each row's element parts and its slot - // coordinate parts. - let mut rows: Vec = vec![ReadSliceRowParts { - row_parts: Vec::new(), - slot_parts: Vec::new(), - }]; - for (elems, slot_elems) in &per_axis { - let mut next: Vec = Vec::with_capacity(rows.len() * elems.len()); + // How many coordinates advance INDEPENDENTLY. A `Pinned` or `Reduced` axis + // is one of its own; every PROJECTED axis (`Iterated` or `MappedRead`) is + // driven by the target dimension it names, and two axes naming the SAME one + // are driven together. That is what execution does: both indices of + // `target[State] = matrix[Region1, Region2]` resolve against the one active + // `State` element, so the read is the DIAGONAL + // `matrix[map1(s), map2(s)]` -- three reads over a 3x3 source, measured in + // `mapped_reference_semantics_tests::two_axes_mapped_to_one_target_dimension_read_the_diagonal`. + // Crossing them instead emitted every off-diagonal pair as an element edge + // and a loop candidate the simulation never traverses (9 loops where there + // are 3), and it disagreed with `ltm_augment::per_element_row_for_target`, + // which projects one target element through each axis and therefore always + // produced the diagonal -- so the link-score NAMES and the element EDGES + // described different graphs. + // + // The groups are ordered by first member, so a slice with no repeated target + // dimension produces exactly the per-axis nesting (and therefore the exact + // row order) this was before the grouping existed. + let mut groups: Vec> = Vec::new(); + let mut group_of_dim: HashMap<&str, usize> = HashMap::new(); + for (i, axis) in read_slice.iter().enumerate() { + let driver = match axis { + AxisRead::Iterated { dim, .. } | AxisRead::MappedRead { dim, .. } => Some(dim.as_str()), + AxisRead::Pinned(_) | AxisRead::Reduced { .. } => None, + }; + match driver.and_then(|d| group_of_dim.get(d).copied()) { + Some(g) => groups[g].push(i), + None => { + if let Some(d) = driver { + group_of_dim.insert(d, groups.len()); + } + groups.push(vec![i]); + } + } + } + + // Cartesian product over the DRIVERS, accumulating each row's element parts + // and its slot coordinate parts. Parts are placed by axis index rather than + // pushed, because a shared driver fills several (possibly non-adjacent) + // axes at once; flattening at the end restores axis order. + let n_axes = read_slice.len(); + let mut rows: Vec<(Vec, Vec>)> = + vec![(vec![String::new(); n_axes], vec![None; n_axes])]; + for group in &groups { + let alternatives = driver_alternatives(group, &per_axis, read_slice, dim_ctx)?; + let mut next = Vec::with_capacity(rows.len() * alternatives.len()); for partial in &rows { - for (ei, e) in elems.iter().enumerate() { - let mut row_parts = partial.row_parts.clone(); - row_parts.push(e.clone()); - let mut slot_parts = partial.slot_parts.clone(); - if let Some(slots) = slot_elems { - slot_parts.push(slots[ei].clone()); + for alternative in &alternatives { + let mut row = partial.clone(); + for (axis, elem, slot) in alternative { + row.0[*axis] = elem.clone(); + row.1[*axis] = slot.clone(); } - next.push(ReadSliceRowParts { - row_parts, - slot_parts, - }); + next.push(row); } } rows = next; } - Some(rows) + Some( + rows.into_iter() + .map(|(row_parts, slot_parts)| ReadSliceRowParts { + row_parts, + slot_parts: slot_parts.into_iter().flatten().collect(), + }) + .collect(), + ) +} + +/// The alternatives one driver of [`read_slice_row_parts`] advances through: +/// per alternative, the `(axis, source element, slot coordinate)` triple for +/// every axis that driver fills. +/// +/// A single-axis driver walks that axis's own element list, which is what every +/// slice without a repeated target dimension is made of and is byte-for-byte +/// what this function replaced. +/// +/// A SHARED driver -- two or more projected axes naming one target dimension -- +/// walks the TARGET dimension's elements instead, and asks each member axis +/// which source element it reads for that coordinate. Its own slot list answers +/// that: an axis's `slots[k]` is the target element its `elems[k]` feeds, so the +/// source element for target element `t` is the `elems` entry whose slot is `t`. +/// Both projected kinds cover the target dimension exactly once (`Iterated` +/// inverts a bijection, `MappedRead` enumerates the target side directly), so +/// the lookup is total; a miss means a stale correspondence, and declining +/// degrades to the caller's conservative fallback rather than dropping rows. +#[allow(clippy::type_complexity)] +fn driver_alternatives( + group: &[usize], + per_axis: &[(Vec, Option>)], + read_slice: &[crate::ltm_agg::AxisRead], + dim_ctx: &crate::dimensions::DimensionsContext, +) -> Option)>>> { + use crate::ltm_agg::AxisRead; + if let [axis] = group { + let (elems, slots) = &per_axis[*axis]; + return Some( + elems + .iter() + .enumerate() + .map(|(k, e)| vec![(*axis, e.clone(), slots.as_ref().map(|s| s[k].clone()))]) + .collect(), + ); + } + let dim = match &read_slice[group[0]] { + AxisRead::Iterated { dim, .. } | AxisRead::MappedRead { dim, .. } => dim, + // Only a projected axis is ever grouped with another. + AxisRead::Pinned(_) | AxisRead::Reduced { .. } => return None, + }; + let target_dim = dim_ctx.get(&crate::common::CanonicalDimensionName::from_raw(dim))?; + crate::ltm_augment::dimension_element_names(target_dim) + .into_iter() + .map(|target_elem| { + group + .iter() + .map(|&axis| { + let (elems, slots) = &per_axis[axis]; + let k = slots.as_ref()?.iter().position(|s| *s == target_elem)?; + Some((axis, elems[k].clone(), Some(target_elem.clone()))) + }) + .collect::>>() + }) + .collect() } /// One source row a hoisted reducer reads, paired with the agg result slot it diff --git a/src/simlin-engine/src/db/ltm/mod.rs b/src/simlin-engine/src/db/ltm/mod.rs index 6cc97d4e4..a0c159dfe 100644 --- a/src/simlin-engine/src/db/ltm/mod.rs +++ b/src/simlin-engine/src/db/ltm/mod.rs @@ -59,13 +59,19 @@ pub(crate) use link_scores::emit_ltm_partial_equation_warning; pub(crate) use link_scores::ltm_partial_equation_warning_message; pub(crate) use loops::build_loops_from_tiered; // The single row/slot derivation (invariant I4 of the shape-expressiveness -// design), re-exported so every consumer derives rows from one function: -// `read_slice_row_parts` is the structured (per-axis parts) core that -// `db::analysis::emit_agg_routed_edges` reads directly (GH #783); the joined -// `read_slice_rows`/`ReadSliceRow` projection feeds -// `db::analysis::emit_edges_for_reference`'s `PerElement` arm (GH #525) and -// the link-score emitters. -pub(crate) use loops::{ReadSliceRow, ReadSliceRowParts, read_slice_row_parts, read_slice_rows}; +// design), re-exported so every consumer derives rows from one function. Only +// the STRUCTURED form crosses this boundary: both `db::analysis` consumers -- +// `emit_agg_routed_edges` (GH #783) and `emit_edges_for_reference`'s +// `PerElement` arm (GH #525) -- read the per-axis parts, because a canonical +// element name can contain a comma and a joined-then-split round-trip would +// mis-read one coordinate as two. +pub(crate) use loops::{ReadSliceRowParts, read_slice_row_parts}; +// The joined projection crosses this boundary for TESTS only -- and only to +// build expected node NAMES (`db::element_graph_proptest`), never to be split +// back apart. Production's remaining users are the link-score emitters inside +// `db::ltm`, which reach `loops` directly and likewise want a name. +#[cfg(test)] +pub(crate) use loops::read_slice_rows; // The cross-element-through-aggregate petal-stitching core, shared by the // exhaustive recovery (`recover_cross_agg_loops`) and discovery // (`ltm_finding`, GH #696) so both enumerate exactly the same cross-agg loops. diff --git a/src/simlin-engine/src/db/ltm_array_freeze_tests.rs b/src/simlin-engine/src/db/ltm_array_freeze_tests.rs index 96264ed23..e4386b55d 100644 --- a/src/simlin-engine/src/db/ltm_array_freeze_tests.rs +++ b/src/simlin-engine/src/db/ltm_array_freeze_tests.rs @@ -6,12 +6,15 @@ //! //! A ceteris-paribus partial that must freeze an ARRAY SLICE -- an other-dep //! (or, changed-last, the live source) referenced as `arr[pin, *]` or -//! `arr[pin, *:Sub]` inside a vector builtin -- cannot spell the freeze -//! inline: `PREVIOUS()` has no codegen path (an array-valued operand +//! `arr[pin, *:Sub]` inside a vector builtin -- did not spell the freeze +//! inline: `PREVIOUS()` had no codegen path (an array-valued operand //! must be a view over storage), so the wrap used to either decline the score //! loudly (`UnfreezablePartial`, the changed-first/changed-last doom) or emit //! a fragment that failed to compile (the per-target-element path, which -//! never doom-checked). +//! never doom-checked). GH #995 phase C3 has since given the inline form a +//! path of its own -- a view over `prev_values` -- so the helper is no longer +//! the only way to spell the freeze; it is retained for the name-correct row +//! rule below, and collapsing the two is tracked as follow-on work. //! //! The fix materializes the freeze as its own synthetic variable: an //! `Equation::Arrayed` aux `$⁚ltm⁚freeze⁚…` with one arm per slice row, each diff --git a/src/simlin-engine/src/db/ltm_char_tests.rs b/src/simlin-engine/src/db/ltm_char_tests.rs index bec256254..7485afef5 100644 --- a/src/simlin-engine/src/db/ltm_char_tests.rs +++ b/src/simlin-engine/src/db/ltm_char_tests.rs @@ -1400,12 +1400,16 @@ fn char_agg_to_scalar_target() { // clean-compiling structural zero (the frozen partial equals the frozen anchor). // HEAD instead recurses into the outer reducer, holds the agg live, and freezes // the co-source array slices: `sum(previous(matrix[region·, *]) * -// previous(other[region·nyc, *]) * "$⁚ltm⁚agg⁚0") * 0.001` -- which fails to -// compile (no LoadPrev-of-array-view path) and surfaces as a LOUD warned zero. -// The finding-2 gate teaches the GH #517 freeze about `live_reducer_text` so the -// enclosing reducer recurses, restoring HEAD's text (and its loud degradation -- -// a loud failure is strictly better than a silent structural zero). Byte- -// identical to HEAD `f057ef38`. +// previous(other[region·nyc, *]) * "$⁚ltm⁚agg⁚0") * 0.001`. The finding-2 gate +// teaches the GH #517 freeze about `live_reducer_text` so the enclosing reducer +// recurses, restoring HEAD's text; the golden below is byte-identical to HEAD +// `f057ef38`. +// +// That text used to have no codegen path (an array-valued `PREVIOUS`) and +// surfaced as a LOUD warned zero, which `b7898692` deliberately preserved over +// a silent structural zero. GH #995 phase C3 gave it one, so the equation is +// unchanged and now COMPILES -- see +// `agg_nested_reducer_partial_scores_full_attribution` for the number. // --------------------------------------------------------------------------- fn agg_nested_reducer_model() -> datamodel::Project { @@ -1429,39 +1433,32 @@ fn char_agg_nested_reducer() { "agg_nested_reducer", agg_nested_reducer_model(), "link_score", - FragmentExpectation::ExpectedFailures { - // The GH #517 nested-live-reducer degradation this fixture exists to - // pin: the hoisted `SUM(pop[*])` held live sits inside a DECLINED - // outer reducer, so the `agg -> growth[e]` partial embeds `PREVIOUS` - // of a wildcard slice, which has no LoadPrev-of-array-view codegen - // path. It surfaces as an Assembly warning -- the LOUD warned-zero - // `b7898692` deliberately PRESERVES rather than fixes. Each of the - // two target elements contributes its score plus the two - // PREVIOUS-capture helpers its partial synthesizes. - why: "GH #517 live-agg-inside-a-declined-outer-reducer: the partial \ - freezes a wildcard slice, which cannot compile; preserved as a \ - loud warned zero, not fixed", - vars: &[ - "$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[boston]", - "$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[nyc]", - "$\u{205A}$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[boston]\u{205A}0\u{205A}arg0", - "$\u{205A}$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[boston]\u{205A}1\u{205A}arg0", - "$\u{205A}$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[nyc]\u{205A}0\u{205A}arg0", - "$\u{205A}$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[nyc]\u{205A}1\u{205A}arg0", - ], - }, + // GH #995 Phase C3 closed the degradation this fixture was written to + // pin. The hoisted `SUM(pop[*])` held live sits inside a DECLINED outer + // reducer, so the `agg -> growth[e]` partial embeds `PREVIOUS` of a + // wildcard slice (visible in the golden). That used to have no codegen + // path and surfaced as a warned zero -- six Assembly warnings: the two + // scores plus the four `PREVIOUS`-capture helper auxes their partials + // synthesized. An array-valued `PREVIOUS` is now a view over + // `prev_values`, so the partial compiles and the capture helpers are no + // longer synthesized at all (the argument is array-shaped, so + // `builtins_visitor` passes it through). The GOLDEN TEXT is byte- + // identical across that change: what moved is compilability, not the + // emitted equation. `agg_nested_reducer_partial_scores_full_attribution` + // pins the resulting NUMBER. + FragmentExpectation::AllCompile, ); } -// Finding 2 materiality guard: unlike every other guard in this file (which -// asserts fragments MUST compile), the byte-parity contract here is to REPRODUCE -// HEAD's LOUD failure. HEAD emits the live-agg-inside-a-frozen-array-slice -// partial that cannot compile (six `Assembly` "failed to compile" warnings: the -// two `agg⁚0->growth[e]` scores plus their four `PREVIOUS`-capture helper auxes), -// producing a warned zero. The pre-fix transform-first freeze produced a SILENT -// clean-compiling zero (zero warnings) -- the worst failure class. This guard -// pins that the loud warnings ARE present: a regression that silently zeroes the -// score would drop them. `pop` is a stock so the edge is causally live. +// Finding 2 materiality guard, in its post-GH-#995 form. The pre-fix +// transform-first freeze produced a SILENT clean-compiling zero (zero warnings) +// -- the worst failure class -- and the guard originally pinned the LOUD +// warned zero that replaced it (six `Assembly` "failed to compile" warnings: the +// two `agg⁚0->growth[e]` scores plus their four `PREVIOUS`-capture helper auxes). +// Phase C3 gave an array-valued `PREVIOUS` a snapshot-buffer view, so that +// partial now compiles and the guard pins the NUMBER instead: a warned zero and +// a silent zero are both ruled out by asserting the score's hand-derived value. +// `pop` is a stock so the edge is causally live. fn agg_nested_reducer_feedback_model() -> datamodel::Project { TestProject::new("nested_reducer_feedback") .with_sim_time(0.0, 3.0, 1.0) @@ -1478,7 +1475,7 @@ fn agg_nested_reducer_feedback_model() -> datamodel::Project { } #[test] -fn agg_nested_reducer_preserves_loud_failure_not_silent_zero() { +fn agg_nested_reducer_partial_scores_full_attribution() { use crate::db::{DiagnosticError, DiagnosticSeverity, collect_model_diagnostics}; use salsa::Setter; @@ -1503,18 +1500,76 @@ fn agg_nested_reducer_preserves_loud_failure_not_silent_zero() { }) .map(|d| d.variable.clone().unwrap_or_default()) .collect(); - // The loud degradation must be present: the two agg⁚0->growth[e] scores must - // each fail to compile (the live-agg-inside-a-frozen-array-slice partial), - // matching HEAD. A silent-zero regression would report NO such failure. - let agg_growth_failures: Vec<&String> = frag_failures - .iter() - .filter(|v| v.contains("agg\u{205A}0\u{2192}growth")) - .collect(); assert!( - agg_growth_failures.len() >= 2, - "the nested-reducer agg->growth partial must LOUDLY fail to compile \ - (HEAD's warned zero), not silently zero; failed fragments: {frag_failures:?}" - ); + frag_failures.is_empty(), + "every fragment must compile now that an array-valued PREVIOUS has a \ + view; a warned zero would show up here: {frag_failures:?}" + ); + + // The number, derived by hand from the fixture rather than recorded from a + // run. `matrix` and `other` are constant 1 and Region has two elements, so + // + // growth[e] = SUM(matrix[e,*] * other[nyc,*] * agg) * 0.001 + // = (1*1*agg + 1*1*agg) * 0.001 = 0.002 * agg + // + // and the ceteris-paribus partial for `agg -> growth[e]` -- which freezes + // the two co-source slices at their PREVIOUS values and holds `agg` live -- + // is `SUM(PREV(matrix[e,*]) * PREV(other[nyc,*]) * agg) * 0.001`. The frozen + // slices are the same constant 1, so the partial equals `growth[e]` + // EXACTLY. The score is then + // + // SAFEDIV(partial - PREV(growth[e]), ABS(growth[e] - PREV(growth[e]))) + // * SIGN(agg - PREV(agg)) + // = SIGN(growth[e] - PREV(growth[e])) * SIGN(agg - PREV(agg)) + // = 1 + // + // because `pop` is a stock fed by `growth > 0`, so both `agg` and `growth` + // increase every step. 1.0 is full attribution, which is the right answer: + // `agg` is the only changing driver of `growth`. The first saved step is 0 + // by the score's own `TIME = INITIAL_TIME` guard. + // + // Every wrong reading lands somewhere else: a failed or stubbed fragment + // reads a constant 0, and freezing the whole declined outer reducer (the + // GH #517 arm) makes the partial equal the frozen anchor, i.e. also 0. + // + // TWO readings this fixture CANNOT tell apart, disclosed rather than left to + // be discovered, both because `matrix` and `other` are the constant 1: + // reading them at their CURRENT rather than their PREVIOUS values, and + // reading the WRONG ROW of the snapshot for a `region·` pin. Neither is + // uncovered -- `array_operand_materialization_tests`' + // `previous_operands_are_views_over_the_prev_snapshot` carries the lag over + // time-varying arrays and `a_prev_view_of_a_row_slice_reads_that_row_of_the_snapshot` + // carries the row over rows two orders of magnitude apart -- but they are + // covered THERE, over the view arithmetic, not here. What this fixture is + // for is the attribution value, which is the thing the LTM wrap decides. + let compiled = crate::db::compile_project_incremental(&db, source_project, "main") + .expect("the LTM-enabled fixture should compile"); + let mut vm = crate::vm::Vm::new(compiled.clone()).expect("VM creation should succeed"); + vm.run_to_end().expect("simulation should run"); + let results = vm.into_results(); + for elem in ["nyc", "boston"] { + let score = format!( + "$\u{205A}ltm\u{205A}link_score\u{205A}$\u{205A}ltm\u{205A}agg\u{205A}0\u{2192}growth[{elem}]" + ); + let offset = *compiled + .offsets + .get(score.as_str()) + .unwrap_or_else(|| panic!("{score} has no results offset")); + let series: Vec = (0..results.step_count) + .map(|step| results.data[step * results.step_size + offset]) + .collect(); + assert_eq!( + series[0], 0.0, + "{score}: the first step is guarded to 0 by the score equation" + ); + for (step, value) in series.iter().enumerate().skip(1) { + assert!( + (value - 1.0).abs() < 1e-12, + "{score}: step {step} must be full attribution (1.0), got {value}; \ + whole series {series:?}" + ); + } + } } // --------------------------------------------------------------------------- @@ -1623,9 +1678,13 @@ fn char_arrayed_target_slot_scores() { // `PREVIOUS(SUM(w[from, *])) + from` -- because `expr0_contains_live_match`'s // Subscript arm matches only a subscript whose HEAD is the live source, never // an index-nested occurrence (`ltm_augment.rs`). Recursing into the reducer -// instead would emit `SUM(PREVIOUS(w[from, *]))` (a PREVIOUS of an array view, -// which has no LoadPrev path -- a loud compile failure and a silently-zeroed -// score, GH #517). +// instead would emit `SUM(PREVIOUS(w[from, *]))`, which was a loud compile +// failure and a silently-zeroed score when written (GH #517: a PREVIOUS of an +// array view had no codegen path). GH #995 phase C3 gave it one, so that form +// compiles now -- but the SELECTION rule this golden pins is unchanged and is +// not about compilability: the changed-first partial freezes the whole reducer +// because the live occurrence is index-nested, and recursing would change which +// occurrence is held live. // // This pins the exact Fig. 2 Q4 selection semantics the stage-2 occurrence // switch must reproduce via `occ.index_nested`: an index-nested occurrence is diff --git a/src/simlin-engine/src/db/ltm_element_instance_tests.rs b/src/simlin-engine/src/db/ltm_element_instance_tests.rs index 455f7e097..da4849e58 100644 --- a/src/simlin-engine/src/db/ltm_element_instance_tests.rs +++ b/src/simlin-engine/src/db/ltm_element_instance_tests.rs @@ -629,11 +629,14 @@ fn an_arrayed_capture_helpers_scores_compile() { // SCOPE, stated rather than implied. This fixture still has failures, and // they are two OTHER root causes, both left for separate work: // - // * `PREVIOUS` of an array-valued reference has no codegen path (GH #995), - // so the helper's own partial cannot freeze its arrayed argument. Closing - // it means synthesizing an ARRAYED freeze helper (#995's option B) rather - // than declining -- the same contract behind the array-slice declines on - // C-LEARN's remaining loop-carrying edges. + // * an array-valued `PREVIOUS` in a SCALAR operand position. GH #995's + // Phase C3 gave `PREVIOUS` an array form, but only where an array is + // expected: this fixture's capture helper is `h[Region] = PREVIOUS(stock)` + // over a bare arrayed `stock`, whose right-hand side lowers to a + // whole-array view being assigned element by element. Making that work is + // a LOWERING question -- a bare arrayed name in an apply-to-all body + // should resolve per element -- not a view question, so C3 changed the + // message here and not the outcome. // * the loop builder subscripts `{helper}[elem]→growth` as though it were // dimensioned, while the emitter gives it none -- an emitter/consumer // shape disagreement that survives independently of the above. @@ -643,10 +646,11 @@ fn an_arrayed_capture_helpers_scores_compile() { // absorbing a new one. for msg in &failures { assert!( - msg.contains("PREVIOUS requires a variable reference") - || msg.contains("expected array variable '$⁚ltm⁚link_score⁚"), + msg.contains( + "an array-valued PREVIOUS/INIT is only meaningful where an array is expected" + ) || msg.contains("expected array variable '$⁚ltm⁚link_score⁚"), "unexpected residual failure class -- this test tolerates only the \ - GH #995 array-freeze class and the loop-builder shape \ + scalar-position array-PREVIOUS class and the loop-builder shape \ disagreement:\n{msg}" ); } diff --git a/src/simlin-engine/src/db/ltm_ir.rs b/src/simlin-engine/src/db/ltm_ir.rs index 4d209ae5b..2408d809c 100644 --- a/src/simlin-engine/src/db/ltm_ir.rs +++ b/src/simlin-engine/src/db/ltm_ir.rs @@ -234,10 +234,11 @@ fn classify_subscript_shape( /// /// A mapped iterated index (`State[i]` over a source declared with /// `Region[i]`) is accepted when `classify_axis_access`'s -/// `iterated_axis_slot_elements` / `mapped_element_correspondence` gate +/// `iterated_axis_slot_elements` / `positional_correspondence` gate /// yields a usable positional remap -- in EITHER declaration direction -/// (GH #757; explicit element maps decline per the GH #756 positional-only -/// gate, keeping the conservative shape). A position-mismatched subscript +/// (GH #757), an explicit element map included since GH #997 (this spelling +/// is folded to an ordinal and never reads the map). A position-mismatched +/// subscript /// like `row_sum[D2]` inside `growth[D1,D2]` where `row_sum` is over `D1` /// is a *genuine* cross-element reference -- no axis classifies -- so it /// returns `None` and keeps its `DynamicIndex` classification. @@ -271,18 +272,93 @@ fn classify_iterated_dim_shape( if axes.iter().any(|a| matches!(a, AxisRead::Reduced { .. })) { return None; } - let n_iterated = axes + let n_projected = axes .iter() - .filter(|a| matches!(a, AxisRead::Iterated { .. })) + .filter(|a| matches!(a, AxisRead::Iterated { .. } | AxisRead::MappedRead { .. })) .count(); - if n_iterated == 0 { + if n_projected == 0 { // All-`Pinned` canonicalizes to `FixedIndex` via the caller's // `classify_subscript_shape` fallback (identical resolution rules). return None; } - if n_iterated == axes.len() { - return Some(RefShape::Bare); + // Two projected axes naming the SAME target dimension are resolved from the + // one active element of it, so the reference reads that dimension's DIAGONAL + // (`target[D] = matrix[D,D]` reads `matrix[d,d]`, and + // `target[State] = matrix[State,State]` over a `matrix[Region1,Region2]` + // source reads `matrix[map1(s), map2(s)]` -- both measured in + // `mapped_reference_semantics_tests`). `Bare` cannot express that, for the + // same reason as the `MappedRead` note below: `expand_same_element` sees only + // the two variables' dimension lists, so a repeated target dimension either + // unions the two axes' candidates (`matrix[D,D]`, 15 edges over a 3x3 source + // where 3 are read) or claims the position for the first axis and leaves the + // second broadcasting (`matrix[State,State]`, 9 edges). `PerElement`'s + // `read_slice_rows` derivation drives axes sharing a target dimension from + // one coordinate and lands exactly the executed rows. + // + // The retarget is narrowed to a dimension the TARGET names ONCE, and the + // reason is the SCORE surface, not the edges. + // + // `RefShape` decides both. A target that repeats the dimension + // (`cube[D1,D1] = ... pop[D1,D1] ...`) is one `emit_per_element_link_scores` + // refuses outright -- every per-element derivation addresses a target axis by + // NAME and there are two coordinates for one name -- so retargeting such a + // reference would silently convert an EMITTED `Bare` link score into the loud + // per-element skip, on every edge into or out of a repeated-dimension + // variable (measured on a three-variable fixture: `pop -> cube` AND + // `cube -> grow` both flip, and loops through them stop being scored). That + // is a real product decision about a shape this change is not about, and it + // is not one to make as a side effect of an edge fix. + // + // What the narrowing COSTS is stated plainly because it is not zero: on + // EDGES the retarget would be better. Over `cube[D1,D1] = pop[D1,D1]` the + // simulation makes four reads (both indices resolve to the target's FIRST + // `D1` axis, so `cube[r1,r2]` reads `pop[r1,r1]`); `Bare` emits 12 edges + // covering 2 of them, and `PerElement` would emit 2 edges covering the SAME + // 2 with no phantoms. Both miss the same two real edges, so the retarget + // removes phantoms without fixing the missing half -- which is the half that + // breaks loop discovery. Fixing that half is `expand_same_element`'s + // name-keyed target positions, and doing it there lets the edges and the + // scores move together instead of trading one for the other. Pinned, both + // directions, by + // `mapped_reference_semantics_tests::a_repeated_target_dimension_reads_the_first_axis_on_both_sides`. + // + // Blast radius, measured: Vensim rejects a repeated-dimension declaration + // ("DimA appears more than once on LHS", `vensim-probes/repeated_dimension.mdl` + // in Vensim DSS 2026-08-04), so no MDL-imported model reaches this shape -- + // it is confined to hand-authored XMILE/JSON/protobuf, which the XMILE v1.0 + // spec does sanction by example. Bounded, not closed. + let all_iterated = axes.iter().all(|a| matches!(a, AxisRead::Iterated { .. })); + if all_iterated { + let mut seen = std::collections::HashSet::new(); + let repeats_a_singly_named_target_dim = axes + .iter() + .filter_map(|a| match a { + AxisRead::Iterated { dim, .. } => Some(dim.as_str()), + // Unreachable: `all_iterated` has just excluded all three. + AxisRead::Pinned(_) | AxisRead::Reduced { .. } | AxisRead::MappedRead { .. } => { + None + } + }) + .any(|dim| { + !seen.insert(dim) && target_iterated_dims.iter().filter(|t| *t == dim).count() == 1 + }); + if !repeats_a_singly_named_target_dim { + return Some(RefShape::Bare); + } } + // Everything else -- a mixed `Iterated`+`Pinned` subscript, and since + // GH #997 any subscript carrying a `MappedRead` axis -- is `PerElement`. + // + // A `MappedRead` axis deliberately does NOT collapse to `Bare` even when + // every axis is one. `Bare`'s element edges go through + // `db::analysis::expand_same_element`, which cannot see the reference site + // and so emits the union of both spellings' diagonals; `PerElement`'s go + // through the per-axis `read_slice_rows` derivation, which resolves each + // axis by ITS OWN rule and lands the exact rows execution reads. The + // per-(row, target-element) link scores follow the same derivation, so the + // names and the edges agree by construction -- which is the property a + // many-to-one element map needs, since several target elements then share + // one source row. Some(RefShape::PerElement { axes }) } @@ -1357,6 +1433,11 @@ pub(crate) enum OccurrenceAxis { /// Iterated over the target's dimension space, lined up by name or a /// positional mapping (`AxisRead::Iterated`). Iterated { dim: String, source_dim: String }, + /// Iterated over the target's dimension space, but spelled with a + /// NON-ACTIVE dimension name that execution pairs with a target-iterated + /// dimension through a declared mapping and resolves name-first then + /// through the element map (`AxisRead::MappedRead`, GH #997). + MappedRead { dim: String, source_dim: String }, /// A reduced axis (`*` / StarRange); present only inside reducer args /// (`AxisRead::Reduced`). Reduced { subset: Option> }, @@ -1377,6 +1458,9 @@ impl OccurrenceAxis { match ar { AxisRead::Pinned(e) => OccurrenceAxis::Pinned(e), AxisRead::Iterated { dim, source_dim } => OccurrenceAxis::Iterated { dim, source_dim }, + AxisRead::MappedRead { dim, source_dim } => { + OccurrenceAxis::MappedRead { dim, source_dim } + } AxisRead::Reduced { subset } => OccurrenceAxis::Reduced { subset }, } } @@ -1433,7 +1517,10 @@ pub(crate) fn derive_other_dep_verdict( // more indices than the target has iterated dims, and every index a bare // target-iterated-dim name -- i.e. every axis `Iterated` or // `MismatchedIterated` (a `Pinned`/`Reduced`/`Dynamic` axis is a literal, - // wildcard, or dynamic index). + // wildcard, or dynamic index; a `MappedRead` one spells a NON-iterated + // dimension name, and collapsing such a subscript to a bare `Var` would + // change its spelling from map-following to positional -- GH #997 -- so it + // is `NotIterated` and the subscript is kept and pinned instead). if axes.is_empty() || axes.len() > target_iterated_count { return OtherDepVerdict::NotIterated; } diff --git a/src/simlin-engine/src/db/ltm_ir_tests.rs b/src/simlin-engine/src/db/ltm_ir_tests.rs index b2c5a8aaa..7a09489c4 100644 --- a/src/simlin-engine/src/db/ltm_ir_tests.rs +++ b/src/simlin-engine/src/db/ltm_ir_tests.rs @@ -675,7 +675,7 @@ mod model_ltm_reference_sites_tests { /// GH #757 (T6 flip): a mapped iterated-dim subscript whose POSITIONAL /// mapping is declared only in the REVERSE direction (on the source's /// `Region` toward `State`) now classifies `Bare` too -- the mapped arm - /// gates on `mapped_element_correspondence` (both declaration + /// gates on `positional_correspondence` (both declaration /// directions, via `classify_axis_access`'s /// `iterated_axis_slot_elements`), matching the compiler's /// `translate_via_mapping`. diff --git a/src/simlin-engine/src/db/ltm_tests.rs b/src/simlin-engine/src/db/ltm_tests.rs index 1925f95ee..3cc31d97b 100644 --- a/src/simlin-engine/src/db/ltm_tests.rs +++ b/src/simlin-engine/src/db/ltm_tests.rs @@ -1569,11 +1569,32 @@ fn an_unrelated_equation_edit_does_not_regenerate_every_link_score() { /// An arrayed dep the per-element pin table cannot cover must make the edge /// decline LOUDLY, not produce a score computed around a hole. /// -/// The shape: `target[cop]` reads `aggregated[agg]`, where `agg` maps to `cop` -/// through an EXPLICIT element map. `mapped_element_correspondence` declines an -/// explicit map (the execution split is documented there), so -/// `dep_element_pins` can project no axis of `aggregated` onto a `cop` element -/// and the dep is absent from the pin table entirely. +/// The shape: `target[cop]` reads `aggregated[cop]`, where `aggregated` is +/// declared over `agg` -- a dimension with DISJOINT element names and NO +/// mapping to `cop`. +/// +/// It COMPILES because `cop` is the dimension the equation ITERATES, so +/// `ast::expr3`'s Pass 1 folds the index to that dimension's ordinal and it +/// indexes `agg`'s storage raw: `target[c1]` reads `agg`'s FIRST element, by +/// POSITION, consulting neither names nor mappings +/// (`mapped_reference_semantics_tests`' `no_mapping_equal_cardinality` measures +/// exactly this -- a cross-dimension read between two dimensions declared to +/// have nothing to do with each other compiles and produces numbers). +/// `build_view_from_ops` is never reached. The DESCRIBER declines because +/// `allocate_implicit_axes_partial` pairs axes by name or by a DECLARED +/// mapping and this pair has neither, so `dep_element_pins` can project no axis +/// of `aggregated` and the dep is absent from the pin table entirely. +/// +/// The element names are deliberately disjoint. An earlier revision used `cop`'s +/// own names on `agg`, which made the values look name-matched when the read is +/// positional -- true only by the coincidence that both lists were declared in +/// the same order, and a fixture that passes by coincidence records a mechanism +/// that is not the one running. +/// +/// This fixture used an EXPLICIT element map before GH #997. That shape is now +/// projectable -- it is exactly the class of dep C-LEARN reads, and +/// `an_element_mapped_arrayed_dep_is_scored_through_the_map` asserts the score +/// it gets -- so the guard needed a shape that still cannot project. /// /// What that produced before this guard: the dimension-name subscript survived /// into the scalar partial, `builtins_visitor` hoisted it into a @@ -1595,19 +1616,14 @@ fn an_uncoverable_arrayed_dep_declines_the_edge_loudly() { let project = TestProject::new("unpinnable_dep") .with_sim_time(0.0, 3.0, 1.0) .named_dimension("cop", &["c1", "c2"]) - .named_dimension_with_element_mapping( - "agg", - &["a1", "a2"], - "cop", - &[("a1", "c2"), ("a2", "c1")], - ) + .named_dimension("agg", &["x", "y"]) .array_aux("aggregated[agg]", "TIME * 2") .aux("switch", "1 + SUM(level[*]) * 0.01", None) .array_stock("level[cop]", "1", &["growth"], &[], None) .array_flow("growth[cop]", "target[cop] * 0.1", None) .array_aux( "target[cop]", - "if switch > 1.05 then aggregated[agg] else level[cop]", + "if switch > 1.05 then aggregated[cop] else level[cop]", ) .build_datamodel(); @@ -1616,7 +1632,26 @@ fn an_uncoverable_arrayed_dep_declines_the_edge_loudly() { source_project.set_ltm_enabled(&mut db).to(true); let source_model = sync_from_datamodel(&db, &project).models["main"].source; + // Non-vacuity: the MODEL must compile, or "no score emitted" would be + // satisfied by a project that never got as far as scoring anything. The + // simulation reads `aggregated[cop]` POSITIONALLY (see the fixture note), + // which is why an unmapped pair with unrelated element names is legal here. + let diags = crate::db::collect_all_diagnostics(&db, source_project); + let errors: Vec<_> = diags + .iter() + .filter(|d| d.severity == crate::db::DiagnosticSeverity::Error) + .map(|d| (&d.variable, &d.error)) + .collect(); + assert!( + errors.is_empty(), + "the fixture must compile; got: {errors:?}" + ); + let ltm = crate::db::model_ltm_variables(&db, source_model, source_project); + assert!( + !ltm.vars.is_empty(), + "the model must emit LTM variables, or the empty assertion below is vacuous" + ); let emitted: Vec<&String> = ltm .vars .iter() @@ -1625,14 +1660,13 @@ fn an_uncoverable_arrayed_dep_declines_the_edge_loudly() { .collect(); assert!( emitted.is_empty(), - "the switch->target edge cannot be scored (its `aggregated[agg]` dep is \ + "the switch->target edge cannot be scored (its `aggregated[cop]` dep is \ unpinnable), so NO link score may be emitted for it; got: {emitted:?}" ); // The decline must be LOUD, and it must be the only thing left: a helper // that fails while its parent compiles is exactly the silent zero this // guard exists to remove. - let diags = crate::db::collect_all_diagnostics(&db, source_project); let silently_degraded: Vec<&String> = diags .iter() .filter_map(|d| match &d.error { @@ -1664,6 +1698,180 @@ fn an_uncoverable_arrayed_dep_declines_the_edge_loudly() { ); } +/// GH #997: an arrayed dep read across an EXPLICIT element map IS scored, and +/// the pin follows the MAP. +/// +/// This is C-LEARN's shape in miniature, and the class of edge #997 exists to +/// recover: `target[cop]` reads `aggregated[agg]` -- the source's OWN dimension +/// name as the subscript -- with `agg` mapping onto `cop` through a declared +/// element map. `mapped_reference_semantics_tests`' `SourceOwnDim` row measures +/// that spelling against the VM: it resolves name-first and then through the +/// map, at every cardinality. Before #997 one correspondence served this +/// spelling and the positional one alike, declined both, and every such edge +/// took the loud unprojectable-dep skip -- 13 of them on C-LEARN. +/// +/// The map here is the reverse permutation (a1 -> c2), so the pinned element is +/// the discriminator: a positional pin would spell `agg\u{B7}a1` for `c1` where +/// the map says `agg\u{B7}a2`. +#[test] +fn an_element_mapped_arrayed_dep_is_scored_through_the_map() { + use salsa::Setter; + + let project = TestProject::new("element_mapped_dep") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("cop", &["c1", "c2"]) + .named_dimension_with_element_mapping( + "agg", + &["a1", "a2"], + "cop", + &[("a1", "c2"), ("a2", "c1")], + ) + .array_aux("aggregated[agg]", "TIME * 2") + .aux("switch", "1 + SUM(level[*]) * 0.01", None) + .array_stock("level[cop]", "1", &["growth"], &[], None) + .array_flow("growth[cop]", "target[cop] * 0.1", None) + .array_aux( + "target[cop]", + "if switch > 1.05 then aggregated[agg] else level[cop]", + ) + .build_datamodel(); + + let mut db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, &project).project; + source_project.set_ltm_enabled(&mut db).to(true); + let source_model = sync_from_datamodel(&db, &project).models["main"].source; + + let ltm = crate::db::model_ltm_variables(&db, source_model, source_project); + let scored: Vec<&crate::db::LtmSyntheticVar> = ltm + .vars + .iter() + .filter(|v| v.name.contains("link_score\u{205A}switch\u{2192}target")) + .collect(); + assert_eq!( + scored.len(), + 2, + "one scalar score per `cop` element; got: {:?}", + scored.iter().map(|v| &v.name).collect::>() + ); + + // The `c1` score must pin the dep to the element the MAP names (a2), not + // the one an ordinal would (a1). + let c1 = scored + .iter() + .find(|v| v.name.ends_with("target[c1]")) + .expect("a score for c1"); + let text = c1.equation.source_text(); + assert!( + text.contains("aggregated[agg\u{B7}a2]"), + "the dep must be pinned to the element the declared map names; got: {text}" + ); + assert!( + !text.contains("aggregated[agg\u{B7}a1]"), + "a positional pin would read the other element; got: {text}" + ); + + // And the decline is gone: no unprojectable-dep warning for this edge. + let diags = crate::db::collect_all_diagnostics(&db, source_project); + let declines: Vec<_> = diags + .iter() + .filter(|d| match &d.error { + crate::db::DiagnosticError::Assembly(m) => { + m.contains("cannot be projected onto that target element") + } + _ => false, + }) + .map(|d| (&d.variable, &d.error)) + .collect(); + assert!(declines.is_empty(), "got: {declines:?}"); +} + +/// GH #997: the class-D edge itself -- an arrayed SOURCE read through an +/// element-mapped axis -- is scored per (source row, target element), not with +/// the conservative cross-product it collapsed to while the reference +/// classified `DynamicIndex`. +/// +/// The map is MANY-TO-ONE (C-LEARN's shape at a smaller scale): two `agg` +/// elements onto four `cop` ones. That is the cardinality the positional rule +/// cannot describe at all, so every name below is evidence the map-following +/// rule produced it -- and each source row appears under two different target +/// elements, which is exactly what a many-to-one read is. +#[test] +fn a_many_to_one_mapped_read_is_scored_per_row_and_element() { + use salsa::Setter; + + let project = TestProject::new("class_d_scores") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("cop", &["c1", "c2", "c3", "c4"]) + .named_dimension_with_element_mapping( + "agg", + &["a1", "a2"], + "cop", + &[("a1", "c1"), ("a1", "c2"), ("a2", "c3"), ("a2", "c4")], + ) + // must sit INSIDE a feedback loop: exhaustive mode scores + // the edges loops traverse, so a feed-forward source gets no link score + // at all and the assertion below would be vacuous. + .array_aux("aggregated[agg]", "1 + SUM(level[*]) * 0.01") + .array_stock("level[cop]", "1", &["growth"], &[], None) + .array_flow("growth[cop]", "target[cop] * 0.1", None) + .array_aux("target[cop]", "aggregated[agg] * 2 + level[cop]") + .build_datamodel(); + + let mut db = SimlinDb::default(); + let source_project = sync_from_datamodel(&db, &project).project; + source_project.set_ltm_enabled(&mut db).to(true); + let source_model = sync_from_datamodel(&db, &project).models["main"].source; + + let diags = crate::db::collect_all_diagnostics(&db, source_project); + let errors: Vec<_> = diags + .iter() + .filter(|d| d.severity == crate::db::DiagnosticSeverity::Error) + .map(|d| (&d.variable, &d.error)) + .collect(); + assert!( + errors.is_empty(), + "the fixture must compile; got: {errors:?}" + ); + + let ltm = crate::db::model_ltm_variables(&db, source_model, source_project); + let mut scored: Vec<&String> = ltm + .vars + .iter() + .map(|v| &v.name) + .filter(|n| n.contains("link_score\u{205A}aggregated[") && n.contains("\u{2192}target[")) + .collect(); + scored.sort(); + let want: Vec = [("a1", "c1"), ("a1", "c2"), ("a2", "c3"), ("a2", "c4")] + .iter() + .map(|(row, elem)| { + format!("$\u{205A}ltm\u{205A}link_score\u{205A}aggregated[{row}]\u{2192}target[{elem}]") + }) + .collect(); + let want: Vec<&String> = want.iter().collect(); + assert_eq!( + scored, want, + "one score per (mapped source row, target element), and no off-map pair" + ); + + // The equation must READ the row its name claims: `target[c2]` reads + // `aggregated[a1]`, and a positional pin has no answer at all here (there + // is no fourth `agg` element for `c4`'s ordinal). + let c2 = ltm + .vars + .iter() + .find(|v| v.name.ends_with("aggregated[a1]\u{2192}target[c2]")) + .expect("the a1 -> c2 score"); + let text = c2.equation.source_text(); + assert!( + text.contains("aggregated[agg\u{B7}a1]"), + "the live source must be read at the mapped row; got: {text}" + ); + assert!( + !text.contains("aggregated[agg\u{B7}a2]"), + "no other row may appear in this element's partial; got: {text}" + ); +} + /// A `LOOKUP` table argument's dimension-name index must be element-pinned in a /// per-element scalar partial, exactly as every other arrayed dep of that target /// already is. @@ -1848,10 +2056,15 @@ fn a_lookup_table_index_is_element_pinned_in_a_per_element_partial() { /// This fixture reaches `emit_per_element_link_scores`: `out[region]` reads /// `src[region, t1]`, a 2-D source at a pinned column, which is the mixed /// iterated+literal `PerElement` shape (GH #525) that routes there. It also -/// reads `other[agg]` across an EXPLICIT element map, which -/// `mapped_element_correspondence` declines -- so that dep has no projectable -/// element and its dimension-name subscript would survive into the scalar -/// partial. Nothing exotic: a 2-D read and a mapped dep. +/// reads `other[region]`, where `other` is declared over `agg` -- a dimension +/// with DISJOINT element names and no declared mapping -- so that dep has no +/// projectable element and its dimension-name subscript would survive into the +/// scalar partial. The read itself is POSITIONAL (`region` is the iterated +/// dimension, folded to an ordinal by Pass 1), which is what makes an unrelated +/// pair legal; see `an_uncoverable_arrayed_dep_declines_the_edge_loudly` for +/// the full mechanism and for why this shape rather than an explicit element +/// map (GH #997 made the latter projectable). Nothing exotic: a 2-D read and an +/// undeclared-mapping dep. #[test] fn the_completeness_guard_holds_on_the_per_element_emitter() { use salsa::Setter; @@ -1860,12 +2073,7 @@ fn the_completeness_guard_holds_on_the_per_element_emitter() { .with_sim_time(0.0, 3.0, 1.0) .named_dimension("region", &["a", "b"]) .named_dimension("slot", &["t1", "t2"]) - .named_dimension_with_element_mapping( - "agg", - &["x", "y"], - "region", - &[("x", "b"), ("y", "a")], - ) + .named_dimension("agg", &["p", "q"]) .array_aux("other[agg]", "TIME * 2") .array_aux_direct( "src", @@ -1875,7 +2083,7 @@ fn the_completeness_guard_holds_on_the_per_element_emitter() { ) .array_stock("level[region]", "1", &["growth"], &[], None) .array_flow("growth[region]", "out[region] * 0.1", None) - .array_aux("out[region]", "src[region, t1] * 0.5 + other[agg]") + .array_aux("out[region]", "src[region, t1] * 0.5 + other[region]") .build_datamodel(); let mut db = SimlinDb::default(); @@ -1883,7 +2091,23 @@ fn the_completeness_guard_holds_on_the_per_element_emitter() { source_project.set_ltm_enabled(&mut db).to(true); let source_model = sync_from_datamodel(&db, &project).models["main"].source; + // Non-vacuity: the MODEL must compile (see the sibling guard's note). + let diags = crate::db::collect_all_diagnostics(&db, source_project); + let errors: Vec<_> = diags + .iter() + .filter(|d| d.severity == crate::db::DiagnosticSeverity::Error) + .map(|d| (&d.variable, &d.error)) + .collect(); + assert!( + errors.is_empty(), + "the fixture must compile; got: {errors:?}" + ); + let ltm = crate::db::model_ltm_variables(&db, source_model, source_project); + assert!( + !ltm.vars.is_empty(), + "the model must emit LTM variables, or the empty assertion below is vacuous" + ); let emitted: Vec<&String> = ltm .vars .iter() @@ -1892,11 +2116,10 @@ fn the_completeness_guard_holds_on_the_per_element_emitter() { .collect(); assert!( emitted.is_empty(), - "the src->out per-element edge cannot be scored (its `other[agg]` dep is \ + "the src->out per-element edge cannot be scored (its `other[region]` dep is \ unprojectable), so NO link score may be emitted; got: {emitted:?}" ); - let diags = crate::db::collect_all_diagnostics(&db, source_project); let silently_degraded: Vec<&String> = diags .iter() .filter_map(|d| match &d.error { @@ -1915,7 +2138,7 @@ fn the_completeness_guard_holds_on_the_per_element_emitter() { diags.iter().any(|d| match &d.error { crate::db::DiagnosticError::Assembly(m) => m.contains("cannot be projected onto that target element") - && m.contains("other[agg]"), + && m.contains("other[region]"), _ => false, }), "the decline must carry the unprojectable-dep warning naming the dep; got: {:?}", diff --git a/src/simlin-engine/src/db/vm_verification_tests.rs b/src/simlin-engine/src/db/vm_verification_tests.rs index 08c0affe1..ec8f32785 100644 --- a/src/simlin-engine/src/db/vm_verification_tests.rs +++ b/src/simlin-engine/src/db/vm_verification_tests.rs @@ -630,7 +630,7 @@ fn test_ltm_mapped_dimension_loop_scores_diagonal_and_nonzero() { // The mapped Bare edges' link scores carry the TARGET's dimensions // (the mapped pair counts as corresponding -- `link_score_dimensions` - // consults `mapped_element_correspondence`), so the per-slot + // consults `db::analysis::bare_reference_correspondence`), so the per-slot // references in the loop-score equations resolve. let ltm_vars = crate::db::model_ltm_variables(&db, source_model, source_project); let dims_of = |name: &str| -> &[String] { @@ -711,7 +711,7 @@ fn test_ltm_mapped_dimension_loop_scores_diagonal_and_nonzero() { /// x[State] * 0.1` over `x[Region]` with the mapping declared in the /// REVERSE direction (on `Region` toward `State`) now classifies `Bare` -- /// `classify_iterated_dim_shape` gates its mapped arm on the same -/// `mapped_element_correspondence` data `expand_same_element` consults +/// correspondence data `expand_same_element` consults /// (both declaration directions), matching the compiler's /// `translate_via_mapping`. The element graph emits the mapping DIAGONAL, /// `link_score_dimensions`' Bare-site gate passes, so the `x→inflow` score diff --git a/src/simlin-engine/src/dimensions.rs b/src/simlin-engine/src/dimensions.rs index 9b72d10f8..499682480 100644 --- a/src/simlin-engine/src/dimensions.rs +++ b/src/simlin-engine/src/dimensions.rs @@ -116,7 +116,24 @@ pub enum AxisIndexName { /// ITERATES -- the apply-to-all placeholder form, which stands for whatever /// element the current iteration selects. IteratedDim, - /// Neither: a variable read, or a name nothing in scope declares. + /// Neither of the two readings this function decides between. Three things + /// land here and the caller tells them apart, because they are its business + /// rather than this precedence rule's: + /// + /// * a NON-ACTIVE dimension name -- typically the source's own + /// (`x[Region]` under a `State`-iterating equation) -- which execution + /// pairs with an iterated dimension through a declared mapping and + /// resolves name-first, then through the element map (GH #997; see + /// [`DimensionsContext::mapped_read_partner_dim`], which + /// `ltm_agg::classify_axis_access` and + /// `ltm_augment_post_transform::pin_dimension_name_indices` both consult + /// from this arm); + /// * a VARIABLE read selecting the element at runtime (`pop[Region, idx]`); + /// * a name nothing in scope declares. + /// + /// Adding the first of those did not change what this function returns for + /// any index: it is a third reading BELOW both of this one's, checked only + /// after `Element` and `IteratedDim` have missed. Unresolved, } @@ -753,170 +770,268 @@ impl DimensionsContext { None } - /// Element-level correspondence between a target equation's iterated - /// dimension and a referenced source variable's (differently-named) - /// declared dimension, per the project's dimension mappings (GH #527). + /// The `source_axis` element the EXECUTED simulation reads when an + /// apply-to-all iteration sitting at `active_element` of `active_dim` + /// resolves a reference against a source axis declared over a DIFFERENT + /// dimension. + /// + /// This is the engine's single statement of the map-following resolution + /// rule (GH #997). It has three steps, tried in this order: + /// + /// 1. **NAME.** If `source_axis` itself declares an element called + /// `active_element`, that element is read -- whatever any declared + /// mapping says. Two dimensions sharing element names is an ordinary + /// modelling idiom (Vensim's Example 3 subrange copy), so this arm is + /// not a corner case: a declared element map is simply not consulted + /// where the names already line up. + /// 2. **MAPPING.** Otherwise [`Self::translate_via_mapping`], which + /// honours an explicit `element_map` when one is declared (in either + /// declaration direction) and falls back to positional correspondence + /// when it is not. + /// 3. **MAPPED PARENT.** Otherwise, when `source_axis` maps to a + /// dimension of which `active_dim` is a SUBDIMENSION, translate through + /// that parent -- the active subdimension's elements are a subset of + /// the parent's, so the parent-directed map applies unchanged. + /// + /// `None` means the reference does not resolve; all three executed callers + /// below turn that into a compile diagnostic. + /// + /// # The three executed call sites /// - /// Returns, for each element of `iterated_dim` in declared order, the - /// `source_dim` element the EXECUTED simulation reads for it -- i.e. - /// the per-element dataflow of an A2A reference like `target[State] = - /// x[State] * c` (or the bare `x * c`) where `x` is declared over - /// `Region` and a `State`/`Region` mapping exists. This is the - /// correspondence the LTM element-graph projection - /// (`expand_same_element`) and the link-score dimension rule - /// (`link_score_dimensions`) use; an LTM consumer falls back to the - /// conservative broadcast (a superset of the true edges) whenever this - /// returns `None`, so the classifier's mapped-`Bare` recognition stays - /// safe: a `Bare` classification yields the mapping diagonal WHEN a - /// usable correspondence exists, else the broadcast -- never fewer - /// edges than the simulation's reads. + /// * `compiler::context`'s `get_implicit_subscript_off` -- a reference + /// carrying NO subscript that reaches the implicit-axis allocator: a + /// stock's inflow/outflow, the stock self-reference, and module input + /// wiring. It calls this once per active dimension, having first + /// narrowed to the active dimensions that own `active_element` (that + /// narrowing is a SEARCH over candidate axes, not part of the rule, so + /// it stays at the call site). + /// * `compiler::subscript`'s `build_view_from_ops`, on the + /// `IndexOp::ActiveDimRef` arm -- a subscript naming a dimension the + /// equation does NOT iterate, typically the source's own + /// (`target[COP] = x[Region]`). Its active dimension is already chosen + /// by `normalize_subscripts3`, so it calls this exactly once. + /// * `compiler::context`'s `IndexExpr3::Dimension` arm, the dynamic-path + /// twin of the previous one (reached when some OTHER index of the same + /// subscript defeats static normalization). Instrumenting all three + /// showed this one resolving zero references across the lib suite, but + /// it is reachable by construction (`x[Region, i+1]` sends the whole + /// subscript down the dynamic path), and before GH #997 it consulted the + /// mapping WITHOUT trying the name first -- a latent divergence from the + /// other two, now removed by routing it here. /// - /// Semantics: - /// - **Positional mappings only**: a mapping with a non-empty explicit - /// `element_map` returns `None`. + /// Everything a reference does NOT get this rule is equally load-bearing + /// and is recorded on [`Self::positional_correspondence`]: a subscript + /// naming a dimension the equation ITERATES, and a bare reference inside + /// an equation body, are both resolved by ORDINAL and never reach here. + /// The whole 4-spelling x 5-mapping x 2-direction matrix is measured + /// against the VM in `crate::mapped_reference_semantics_tests`. + pub fn resolve_mapped_read( + &self, + source_axis: &Dimension, + active_dim: &Dimension, + active_element: &CanonicalElementName, + ) -> Option { + if source_axis.get_offset(active_element).is_some() { + return Some(active_element.clone()); + } + if let Some(translated) = self.translate_via_mapping( + source_axis.canonical_name(), + active_dim.canonical_name(), + active_element, + ) { + return Some(translated); + } + // Note what step 2 above does NOT do: it short-circuits. A mapping that + // translates to an element `source_axis` does not declare returns that + // element rather than falling through to step 3 below, so the caller's + // `get_offset` misses and the reference goes unresolved for THIS active + // dimension. That is the pre-GH #997 behaviour of + // `get_implicit_subscript_off`, preserved rather than chosen: only a + // malformed element map can produce it, and a caller that searches + // several active dimensions still tries the rest. + let parent = + self.find_mapping_parent_of(source_axis.canonical_name(), active_dim.canonical_name())?; + self.translate_to_source_via_mapping(source_axis.canonical_name(), parent, active_element) + } + + /// Per element of `iterated_dim` in declared order, the `source_dim` + /// element a POSITIONALLY-resolved reference reads (GH #527, re-keyed by + /// GH #997). /// - /// This gate's original justification -- "the engine's executed A2A - /// lowering resolves mapped references POSITIONALLY and ignores the - /// element map" -- is FALSE AS A UNIVERSAL, and the correction matters - /// because it is what made the gate blanket. Execution does both, and - /// which one you get depends on the SPELLING of the reference's - /// subscript, not on the mapping or the direction it was declared in: + /// This is the describer for the two spellings the executed lowering + /// resolves by ORDINAL, measured cell by cell in + /// `crate::mapped_reference_semantics_tests`: /// - /// * a subscript naming a dimension the enclosing equation ITERATES - /// (`target[State] = x[State]`, `State` active) is **positional**; - /// * a subscript naming a dimension that is NOT active -- typically the - /// referenced variable's OWN declared dimension (`target[COP] = - /// x[Region]`) -- **follows the element map**; - /// * a BARE reference (`target[COP] = x`) is **positional**, in either - /// declaration direction, and at unequal cardinality fails to compile - /// like the active-dimension spelling. It has no subscript, so the - /// mechanism below never fires for it -- worth stating because - /// `expand_same_element` names `emit_edges_for_reference`'s `Bare` arm - /// as a primary consumer of this correspondence. + /// * a subscript naming a dimension the equation ITERATES + /// (`target[State] = x[State]`, `x` over `Region`). `ast::expr3`'s + /// Pass 1 folds the active dimension's name to that dimension's ordinal, + /// which then indexes the source's storage raw. + /// * a BARE reference inside an equation body (`target[State] = x`). + /// `compiler::context`'s `lower_pass0` rewrites it into the spelling + /// above before anything resolves it, so the two are one rule. /// - /// The fork is `ast::expr3`'s `IndexExpr3::Dimension` arm: a name - /// matching an active dimension folds to that dimension's ordinal and - /// indexes the referenced variable's storage raw, never consulting a - /// mapping; a name matching none survives as a dimension reference and - /// reaches `translate_via_mapping`, which honours the map. + /// Neither consults a declared element map, which is why this function + /// returns the positional diagonal for an element-mapped pair rather than + /// declining it. It declined before GH #997 because ONE function served + /// both this rule and [`Self::resolve_mapped_read`]'s, and could not see + /// which was being asked; a caller now picks the one its site's spelling + /// gets and the conservative decline is no longer needed for either. /// - /// Map-following is the CORRECT behaviour where it happens, checked - /// against real Vensim rather than argued: C-LEARN's - /// `im_6_ff_co2[COP]` reads an `Aggregated Regions`-declared source on - /// the non-active spelling, and its source values and per-element - /// factors are distinct enough that the checked-in `Ref.vdf` uniquely - /// identifies which source region each of the 7 COP elements read -- - /// the declared element map every time (positional would put 9.7654 - /// where Vensim has 6.7294 for `oecd_eu`). `simulates_clearn` gates it. + /// A mapping must be DECLARED (in either direction) for the pair to + /// correspond at all. That is narrower than execution -- the iterated + /// spelling compiles and reads positionally between two dimensions + /// declared to have nothing to do with each other, which + /// `mapped_reference_semantics_tests::no_mapping_equal_cardinality` + /// measures -- and deliberately so: an undeclared pair keeps the + /// conservative broadcast, which is a SUPERSET of that read, and GH #527's + /// rule is that the diagonal follows a correspondence the model declares. /// - /// The positional half has NO such ground truth and is recorded here as - /// UNVERIFIED. The argument for doubting it: Vensim's documented trigger - /// for subscript mapping is a right-hand-side subscript absent from the - /// left-hand side, which the active-dimension spelling does not have, so - /// it may not be a mapping use in Vensim at all. Note the structure -- - /// that Vensim claim is the PREMISE for the doubt, and it is the weaker - /// of the two: it is PARAPHRASED from a summarising fetch of - /// , - /// not quoted from the page, and nobody here has read the raw text. So - /// treat neither the positional behaviour nor the reason for doubting it - /// as settled; the `Ref.vdf` evidence above is independent of both and - /// is the load-bearing half of this note. + /// Equal cardinality is likewise required. A larger source is read only up + /// to the target's extent and a smaller one does not compile at all, so + /// declining is again a superset rather than a wrong answer. /// - /// **The gate must nevertheless STAY until the spelling is threaded - /// in.** Deleting it is not the fix: with it removed, - /// `element_graph_mapped_element_map_edges_superset_of_simulation_reads` - /// loses a TRUE edge (the forbidden direction for the LTM contract -- - /// fewer edges than the simulation reads), and an integration fixture - /// starts reporting loop scores attributed along edges that do not - /// exist (9 lib + 2 integration tests red). A correct fix has to tell - /// the two spellings apart, and this predicate cannot: it is keyed by - /// the dimension PAIR, while the deciding fact -- which dimension name - /// the subscript spells -- belongs to the reference site. Threading it - /// reaches every call site (`db::analysis`, `ltm_agg`, - /// `db::ltm::link_scores`, `ltm_augment_post_transform`), which is why - /// it has not been done. + /// Returns `None` when no mapping is declared either way, when either + /// dimension is indexed, or when the cardinalities differ; callers keep + /// their conservative broadcast. /// - /// The trap, and the most useful thing to know before touching this: - /// **no simulation-level test can catch a mistake here.** The four - /// modules above are the only callers, and all are LTM/analysis -- - /// nothing in the compiler, the VM, or the wasm backend consults this, - /// and `DimensionsContext` is not reachable outside the crate at all - /// (`mod dimensions` is private in `lib.rs` and the type is never - /// re-exported), so libsimlin, pysimlin, the CLI and the wasm build - /// cannot reach it even in principle. `simulates_clearn` passes with the - /// gate removed. Only the element-graph parity and LTM tests gate this - /// behaviour. - /// - **Direction**: which dimension DECLARED the mapping. Not to be - /// confused with the spelling fork above -- declaration direction has no - /// bearing on whether execution follows the element map, and reading a - /// split in the fixtures as a direction effect is the wrong turn this - /// note exists to prevent. Both declaration directions are honored, - /// mirroring - /// [`Self::translate_via_mapping`] (which the compiler's subscript / - /// A2A resolution uses): a mapping declared on `iterated_dim` toward - /// `source_dim` or one declared on `source_dim` toward - /// `iterated_dim`. Since GH #757 the classifiers - /// (`classify_iterated_dim_shape` via - /// `ltm_agg::classify_axis_access`) gate their mapped arms on this - /// same correspondence, so subscripted references accept both - /// directions too -- classification, expansion, and the compiler - /// resolve the identical mapped set. - /// - **Transitivity**: single-hop only, matching `has_mapping_to`. A - /// chained `A→B→C` mapping yields `None` for the `(A, C)` pair, just - /// as the classifier declines it. - /// - **Cardinality**: equal sizes required, per - /// `translate_via_mapping`'s positional path -- a property of THIS - /// function, not of the language. GH #753 records that a - /// different-cardinality map "does not compile", and that too holds - /// only for the active-dimension spelling: on the map-following - /// spelling a many-to-one map compiles and runs correctly (C-LEARN's - /// 3-element `Aggregated Regions` onto 7-element `COP` is the shipped - /// case, and a 2-onto-4 fixture reproduces it). GH #756 tracks the - /// positional-vs-element-map execution split itself, which the - /// spelling fork above describes. - /// - **Fallback**: `None` when no direct mapping exists in either - /// direction, either dimension is indexed, a non-empty element map - /// is declared, or any iterated element fails to translate (a - /// positional size mismatch). Callers treat `None` as "no - /// correspondence" and keep their conservative broadcast. - pub fn mapped_element_correspondence( + /// GH #756 tracks the open question of whether VENSIM resolves the + /// iterated spelling positionally at all -- its own Example 3 cannot say, + /// because there the three candidate rules coincide. If that is ever + /// settled the other way, this function moves with execution; it describes + /// what the engine runs today. + pub fn positional_correspondence( &self, iterated_dim: &CanonicalDimensionName, source_dim: &CanonicalDimensionName, ) -> Option> { - // Cheap pre-check so the common no-mapping case doesn't pay the - // per-element translation attempts below. if !self.has_mapping_to(iterated_dim, source_dim) && !self.has_mapping_to(source_dim, iterated_dim) { return None; } - // Decline explicit (non-positional) element maps, keeping the - // broadcast superset. Execution follows the element map on one - // reference spelling and resolves positionally on the other, and this - // predicate is keyed by the dimension PAIR, so it cannot tell which - // one it is being asked about; answering either way is wrong for the - // other. Read the "Positional mappings only" bullet before changing - // this -- deleting it drops a true edge and reds 11 tests, and NO - // simulation-level test gates it. - let has_element_map = |a: &CanonicalDimensionName, b: &CanonicalDimensionName| -> bool { - self.find_mapping_info(a, b) - .is_some_and(|m| !m.element_map.is_empty()) - }; - if has_element_map(iterated_dim, source_dim) || has_element_map(source_dim, iterated_dim) { - return None; - } let iterated_named = match self.dimensions.get(iterated_dim)? { Dimension::Named(_, named) => named, Dimension::Indexed(_, _) => return None, }; + let source_named = match self.dimensions.get(source_dim)? { + Dimension::Named(_, named) => named, + Dimension::Indexed(_, _) => return None, + }; + if iterated_named.elements.len() != source_named.elements.len() { + return None; + } + // The positional diagonal: the target's i-th element reads the + // source's i-th, so the answer IS the source's element list. + Some(source_named.elements.clone()) + } + + /// Per element of `iterated_dim` in declared order, the `source_dim` + /// element a MAP-FOLLOWING reference reads -- [`Self::resolve_mapped_read`] + /// applied per element (GH #997). + /// + /// This is the describer for the two spellings that resolve name-first and + /// then through the declared element map: + /// + /// * a subscript naming a NON-ACTIVE dimension, typically the source's own + /// (`target[COP] = x[Aggregated Regions]`) -- C-LEARN's shape, and the + /// one Vensim's own reference page is written around; + /// * a stock's FLOW reference (`level[State] = INTEG(x, 0)` with `x` over + /// `Region`), which never passes through pass 0 and so keeps its + /// subscript-less form all the way to `get_implicit_subscript_off`. + /// + /// Unlike [`Self::positional_correspondence`] this works at every + /// cardinality, many-to-one included: C-LEARN maps a 3-element + /// `Aggregated Regions` onto a 7-element `COP`, and the checked-in + /// `Ref.vdf` identifies which source region each COP element read -- the + /// declared element map every time. `simulates_clearn` gates it. + /// + /// A mapping must be declared in one direction or the other, matching + /// execution: `compiler::subscript`'s `normalize_subscripts3` pairs a + /// non-active subscript dimension with an active one only through a + /// declared mapping, and refuses the reference otherwise + /// (`mapped_reference_semantics_tests::no_mapping_equal_cardinality` + /// measures the refusal). + /// + /// Returns `None` when no mapping is declared, when either dimension is + /// indexed, or when any iterated element fails to resolve. + pub fn executed_read_correspondence( + &self, + iterated_dim: &CanonicalDimensionName, + source_dim: &CanonicalDimensionName, + ) -> Option> { + if !self.has_mapping_to(iterated_dim, source_dim) + && !self.has_mapping_to(source_dim, iterated_dim) + { + return None; + } + let iterated = self.dimensions.get(iterated_dim)?; + let source = self.dimensions.get(source_dim)?; + let Dimension::Named(_, iterated_named) = iterated else { + return None; + }; iterated_named .elements .iter() - .map(|elem| self.translate_via_mapping(source_dim, iterated_dim, elem)) + .map(|elem| self.resolve_mapped_read(source, iterated, elem)) .collect() } + /// Which of the target equation's iterated dimensions execution pairs an + /// index naming the NON-ACTIVE dimension `index_dim` with (GH #997). + /// + /// This mirrors `compiler::subscript::normalize_subscripts3`'s + /// `IndexExpr3::Dimension` arm, which is where the pairing is decided: + /// having failed to match an active dimension by NAME, it takes the first + /// active dimension carrying a mapping to or from the index's dimension + /// and emits an `IndexOp::ActiveDimRef` for it. + /// [`Self::resolve_mapped_read`] then resolves the element. + /// + /// Two deliberate narrowings relative to that arm, both conservative for a + /// describer (they decline, and a declining caller keeps its + /// cross-product, which is a superset of the true reads): + /// + /// * an `index_dim` that IS one of `target_iterated_dims` is not this + /// rule's business -- it is the positional spelling, and the caller has + /// already handled it; + /// * AMBIGUITY declines. Execution breaks a tie between two viable active + /// dimensions by POSITION (`position()` returns the first), which is a + /// deterministic but arbitrary rule that a describer should not bake in: + /// a model whose index dimension maps to two of the target's iterated + /// dimensions is a modelling defect, and describing it by silently + /// picking one would attribute influence along edges chosen by + /// declaration order. + /// + /// `index_dim` is NOT required to be the source axis's own dimension. + /// Execution does not require it either -- the index dimension only picks + /// the active axis, and the element is then resolved against the source's + /// own axis -- so requiring it would under-describe a legal spelling at no + /// benefit. + pub fn mapped_read_partner_dim( + &self, + index_dim: &CanonicalDimensionName, + target_iterated_dims: &[String], + ) -> Option { + if !self.dimensions.contains_key(index_dim) { + return None; + } + let mut partner: Option = None; + for name in target_iterated_dims { + let candidate = CanonicalDimensionName::from_raw(name); + if &candidate == index_dim { + return None; + } + if self.has_mapping_to(index_dim, &candidate) + || self.has_mapping_to(&candidate, index_dim) + { + if partner.is_some() { + return None; + } + partner = Some(candidate); + } + } + partner + } + /// Check if child is a subdimension of parent. /// For named dimensions, checks element containment. For indexed dimensions, /// uses the declared `parent` field. @@ -1603,7 +1718,13 @@ mod tests { assert_eq!(result, Some(CanonicalElementName::from_raw("a1"))); } - // ========== Tests for mapped_element_correspondence (GH #527) ========== + // ===== Tests for the two spelling-keyed correspondences (GH #527/#997) ===== + // + // The rows below are the (mapping kind x spelling) product, and they are + // deliberately the SAME kinds `crate::mapped_reference_semantics_tests` + // measures against the VM -- that module is the authority for what + // execution does, and these assert that the two describers report it. A + // kind measured there and missing here is a describer with no gate. fn canon_elems(names: &[&str]) -> Vec { names @@ -1612,235 +1733,387 @@ mod tests { .collect() } - /// Positional mapping declared on the iterated dimension (the - /// classifier-accepted direction): one source element per iterated - /// element, by position. - #[test] - fn test_mapped_correspondence_positional_forward() { + fn dim(name: &str, elems: &[&str]) -> datamodel::Dimension { + datamodel::Dimension::named( + name.to_string(), + elems.iter().map(|e| e.to_string()).collect(), + ) + } + + fn with_element_map( + mut d: datamodel::Dimension, + target: &str, + pairs: &[(&str, &str)], + ) -> datamodel::Dimension { + d.mappings = vec![datamodel::DimensionMapping { + target: target.to_string(), + element_map: pairs + .iter() + .map(|(a, b)| (a.to_string(), b.to_string())) + .collect(), + }]; + d + } + + fn state_region( + ctx: &DimensionsContext, + ) -> ( + Option>, + Option>, + ) { use crate::common::CanonicalDimensionName; + let s = CanonicalDimensionName::from_raw("State"); + let r = CanonicalDimensionName::from_raw("Region"); + ( + ctx.positional_correspondence(&s, &r), + ctx.executed_read_correspondence(&s, &r), + ) + } - let mut state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string()], - ); + /// A positional (`maps_to`) mapping: BOTH spellings read the same + /// diagonal, which is why the fork went unnoticed for so long. Declared on + /// the iterated dimension. + #[test] + fn positional_mapping_makes_both_spellings_agree_forward() { + let mut state = dim("State", &["s1", "s2"]); state.set_maps_to("Region".to_string()); - let region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], - ); - let ctx = DimensionsContext::from(&[state, region]); + let ctx = DimensionsContext::from(&[state, dim("Region", &["a", "b"])]); - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("State"), - &CanonicalDimensionName::from_raw("Region"), - ); - assert_eq!(result, Some(canon_elems(&["a", "b"]))); + let (positional, executed) = state_region(&ctx); + assert_eq!(positional, Some(canon_elems(&["a", "b"]))); + assert_eq!(executed, positional); } - /// Positional mapping declared on the SOURCE dimension (the reverse - /// declaration direction, which arises for bare references): still a - /// positional correspondence. + /// The same pair with the mapping declared on the SOURCE dimension. + /// Declaration direction changes nothing -- measured across all 40 mapped + /// cells of `mapped_reference_semantics_tests`. #[test] - fn test_mapped_correspondence_positional_reverse_declared() { - use crate::common::CanonicalDimensionName; - - let state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string()], - ); - let mut region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], - ); + fn positional_mapping_makes_both_spellings_agree_reverse_declared() { + let mut region = dim("Region", &["a", "b"]); region.set_maps_to("State".to_string()); - let ctx = DimensionsContext::from(&[state, region]); + let ctx = DimensionsContext::from(&[dim("State", &["s1", "s2"]), region]); - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("State"), - &CanonicalDimensionName::from_raw("Region"), - ); - assert_eq!(result, Some(canon_elems(&["a", "b"]))); + let (positional, executed) = state_region(&ctx); + assert_eq!(positional, Some(canon_elems(&["a", "b"]))); + assert_eq!(executed, positional); } - /// An EXPLICIT (non-positional) element map -- here a many-to-one - /// `State{s1,s2,s3}→Region{a,b}` -- returns None: the executed A2A - /// lowering resolves mapped references positionally, ignoring the - /// element map (GH #753; element-map diagonals are gated on the engine - /// honoring them in execution), so callers must keep the conservative - /// broadcast. + /// A PERMUTED explicit element map is where the two spellings part. + /// + /// This test used to assert `None` for both, under the name + /// `test_mapped_correspondence_permuted_element_map_is_none_both_directions`. + /// The decline was not a claim about execution: one function served both + /// spellings and could not tell which it was being asked about, so it + /// answered neither. `mapped_reference_semantics_tests`' `Permuted` row + /// measures both answers against the VM -- the iterated and bare-in-equation + /// spellings read `[10, 20, 30]` (positional) while the source-own-dim and + /// stock-flow spellings read `[30, 10, 20]` (the map) -- so each describer + /// now returns its own spelling's answer. Both declaration directions are + /// covered, since the pair is queried both ways round. #[test] - fn test_mapped_correspondence_element_map_is_none() { + fn a_permuted_element_map_splits_the_two_spellings() { use crate::common::CanonicalDimensionName; + let state = with_element_map( + dim("State", &["s1", "s2"]), + "Region", + &[("s1", "b"), ("s2", "a")], + ); + let ctx = DimensionsContext::from(&[state, dim("Region", &["a", "b"])]); - let mut state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string(), "s3".to_string()], + let (positional, executed) = state_region(&ctx); + assert_eq!( + positional, + Some(canon_elems(&["a", "b"])), + "the iterated spelling folds the active dim to an ordinal and never \ + consults the map" ); - state.mappings = vec![datamodel::DimensionMapping { - target: "Region".to_string(), - element_map: vec![ - ("s1".to_string(), "a".to_string()), - ("s2".to_string(), "a".to_string()), - ("s3".to_string(), "b".to_string()), - ], - }]; - let region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], + assert_eq!( + executed, + Some(canon_elems(&["b", "a"])), + "the source-own-dim spelling follows the declared map" ); - let ctx = DimensionsContext::from(&[state, region]); - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("State"), - &CanonicalDimensionName::from_raw("Region"), + // The same pair with the roles swapped: the element map now sits on the + // SOURCE side, which `translate_via_mapping` reads in reverse. + let s = CanonicalDimensionName::from_raw("State"); + let r = CanonicalDimensionName::from_raw("Region"); + assert_eq!( + ctx.positional_correspondence(&r, &s), + Some(canon_elems(&["s1", "s2"])) + ); + assert_eq!( + ctx.executed_read_correspondence(&r, &s), + Some(canon_elems(&["s2", "s1"])) ); - assert_eq!(result, None); } - /// Even a same-size, well-formed (permuted) explicit element map is - /// declined -- the executed lowering would read positionally, so a - /// map-following diagonal would drop the true edges. Also pins the - /// REVERSE declaration direction (map declared on the source dim). + /// A MANY-TO-ONE element map (C-LEARN's shape): the executed rule resolves + /// every target element, while the positional one declines -- there is no + /// third source element for the third target ordinal to read, and the + /// iterated spelling does not compile at all on such a pair + /// (`mapped_reference_semantics_tests`' `(ManyToOne, IteratedDim)` cell is + /// a refusal). + /// + /// The executed half is what closes GH #997: it is the correspondence + /// C-LEARN's `FF stop growth year[COP] = FF stop growth year + /// Aggregated[Aggregated Regions]` needs, and the checked-in `Ref.vdf` + /// identifies the map as the rule Vensim applies there. #[test] - fn test_mapped_correspondence_permuted_element_map_is_none_both_directions() { - use crate::common::CanonicalDimensionName; - - let mut state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string()], - ); - state.mappings = vec![datamodel::DimensionMapping { - target: "Region".to_string(), - element_map: vec![ - ("s1".to_string(), "b".to_string()), - ("s2".to_string(), "a".to_string()), - ], - }]; - let region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], + fn a_many_to_one_element_map_resolves_only_on_the_executed_rule() { + let state = with_element_map( + dim("State", &["s1", "s2", "s3"]), + "Region", + &[("s1", "a"), ("s2", "a"), ("s3", "b")], ); - let ctx = DimensionsContext::from(&[state, region]); + let ctx = DimensionsContext::from(&[state, dim("Region", &["a", "b"])]); - let state_name = CanonicalDimensionName::from_raw("State"); - let region_name = CanonicalDimensionName::from_raw("Region"); - // Declared on the iterated dim (State -> Region). + let (positional, executed) = state_region(&ctx); + assert_eq!(positional, None); + assert_eq!(executed, Some(canon_elems(&["a", "a", "b"]))); + } + + /// Two dimensions declaring the SAME element names, related by a positional + /// mapping: the executed rule stops at NAME IDENTITY, the positional one + /// reads by ordinal, and the two disagree because the names are declared in + /// a different order. `mapped_reference_semantics_tests`' + /// `SharedElementNames` row is the VM oracle -- it is the row that shows + /// map-following is really name-FIRST. + /// + /// Vensim's own Example 3 (`PTASKS <-> TASKS`, a subrange copy) makes + /// shared element names an ordinary idiom rather than an oddity, so this is + /// not a corner the describers may leave to the other one's answer. + #[test] + fn shared_element_names_split_the_two_spellings() { + let mut state = dim("State", &["cal", "ann", "bob"]); + state.set_maps_to("Region".to_string()); + let ctx = DimensionsContext::from(&[state, dim("Region", &["ann", "bob", "cal"])]); + + let (positional, executed) = state_region(&ctx); assert_eq!( - ctx.mapped_element_correspondence(&state_name, ®ion_name), - None + positional, + Some(canon_elems(&["ann", "bob", "cal"])), + "by ordinal: State's first element reads Region's first" ); - // Same pair queried with State as the SOURCE dim: the element map - // is then on the source side (the reverse declaration direction) - // and must be declined the same way. assert_eq!( - ctx.mapped_element_correspondence(®ion_name, &state_name), - None + executed, + Some(canon_elems(&["cal", "ann", "bob"])), + "by name: each State element reads the Region element it shares a \ + name with" ); } - /// A positional mapping between different-size dimensions cannot - /// translate (no element-level map to disambiguate): None, so callers - /// keep the conservative broadcast. + /// A positional mapping between different-size dimensions has no + /// element-level map to disambiguate, so NEITHER rule can translate. #[test] - fn test_mapped_correspondence_positional_size_mismatch_is_none() { - use crate::common::CanonicalDimensionName; - - let mut state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string(), "s3".to_string()], - ); + fn a_positional_size_mismatch_declines_on_both_rules() { + let mut state = dim("State", &["s1", "s2", "s3"]); state.set_maps_to("Region".to_string()); - let region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], - ); - let ctx = DimensionsContext::from(&[state, region]); + let ctx = DimensionsContext::from(&[state, dim("Region", &["a", "b"])]); - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("State"), - &CanonicalDimensionName::from_raw("Region"), - ); - assert_eq!(result, None); + let (positional, executed) = state_region(&ctx); + assert_eq!(positional, None); + assert_eq!(executed, None); } - /// A partial element map (an iterated element with no pair) is None -- - /// it is an explicit element map (declined wholesale by the positional- - /// only gate), and even without the gate it could not give every - /// iterated element a source. + /// A PARTIAL element map (an iterated element with no pair) declines on the + /// executed rule -- there is no source element to name for `s2` -- while + /// the positional rule, which never reads the map, answers by ordinal. + #[test] + fn a_partial_element_map_declines_only_on_the_executed_rule() { + let state = with_element_map(dim("State", &["s1", "s2"]), "Region", &[("s1", "a")]); + let ctx = DimensionsContext::from(&[state, dim("Region", &["a", "b"])]); + + let (positional, executed) = state_region(&ctx); + assert_eq!(positional, Some(canon_elems(&["a", "b"]))); + assert_eq!(executed, None); + } + + /// With NO mapping declared in either direction both rules decline, so an + /// unrelated pair keeps the caller's conservative broadcast (GH #527's + /// rule: the diagonal follows a correspondence the MODEL declares). + /// + /// The iterated spelling does in fact compile and read positionally between + /// two such dimensions -- `mapped_reference_semantics_tests:: + /// no_mapping_equal_cardinality` measures it -- so the positional decline + /// here is a deliberate superset rather than a description of execution; + /// see `positional_correspondence`'s rustdoc. #[test] - fn test_mapped_correspondence_partial_element_map_is_none() { + fn an_undeclared_pair_declines_on_both_rules() { + let ctx = + DimensionsContext::from(&[dim("State", &["s1", "s2"]), dim("Region", &["a", "b"])]); + + let (positional, executed) = state_region(&ctx); + assert_eq!(positional, None); + assert_eq!(executed, None); + } + + /// Single-hop only, matching `has_mapping_to` (and the LTM classifier): a + /// chained `A→B→C` mapping yields None for `(A, C)` on both rules. + #[test] + fn a_transitive_chain_declines_on_both_rules() { use crate::common::CanonicalDimensionName; - let mut state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string()], + let mut dim_a = dim("DimA", &["a1", "a2"]); + dim_a.set_maps_to("DimB".to_string()); + let mut dim_b = dim("DimB", &["b1", "b2"]); + dim_b.set_maps_to("DimC".to_string()); + let ctx = DimensionsContext::from(&[dim_a, dim_b, dim("DimC", &["c1", "c2"])]); + + let a = CanonicalDimensionName::from_raw("DimA"); + let c = CanonicalDimensionName::from_raw("DimC"); + assert_eq!(ctx.positional_correspondence(&a, &c), None); + assert_eq!(ctx.executed_read_correspondence(&a, &c), None); + } + + // ===== resolve_mapped_read: the per-element executed rule (GH #997) ===== + + /// The three steps in order, on one context: NAME beats the declared + /// element map, the map is consulted only where the name misses, and an + /// element neither names nor the map reaches declines. + #[test] + fn resolve_mapped_read_tries_name_before_the_element_map() { + // `State` and `Region` share the element name `shared`, and the map + // sends `shared` somewhere else -- so the two rules give different + // answers for it and the assertion is not vacuous. + let state = with_element_map( + dim("State", &["shared", "s2"]), + "Region", + &[("shared", "a"), ("s2", "shared")], + ); + let ctx = DimensionsContext::from(&[state, dim("Region", &["a", "shared"])]); + let source = ctx + .get(&CanonicalDimensionName::from_raw("Region")) + .unwrap(); + let active = ctx.get(&CanonicalDimensionName::from_raw("State")).unwrap(); + + assert_eq!( + ctx.resolve_mapped_read(source, active, &CanonicalElementName::from_raw("shared")), + Some(CanonicalElementName::from_raw("shared")), + "the source axis declares `shared` itself, so the map is not consulted" ); - state.mappings = vec![datamodel::DimensionMapping { - target: "Region".to_string(), - element_map: vec![("s1".to_string(), "a".to_string())], - }]; - let region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], + assert_eq!( + ctx.resolve_mapped_read(source, active, &CanonicalElementName::from_raw("s2")), + Some(CanonicalElementName::from_raw("shared")), + "`s2` is not a Region element, so the declared map answers" ); - let ctx = DimensionsContext::from(&[state, region]); + assert_eq!( + ctx.resolve_mapped_read(source, active, &CanonicalElementName::from_raw("nope")), + None + ); + } - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("State"), - &CanonicalDimensionName::from_raw("Region"), + /// Step 3: the source dimension maps to a PARENT of the active + /// subdimension, so an element of the subdimension translates through the + /// parent-directed map. This is the arm `get_implicit_subscript_off` has + /// always had and `build_view_from_ops` gained when the two were unified. + #[test] + fn resolve_mapped_read_translates_through_a_mapped_parent() { + let mut source = dim("Source", &["x", "y", "z"]); + source.set_maps_to("Parent".to_string()); + let parent = dim("Parent", &["p1", "p2", "p3"]); + // A subdimension of `Parent`: its elements are a subset, in order. + let sub = dim("Sub", &["p2", "p3"]); + let ctx = DimensionsContext::from(&[source, parent, sub]); + + let source_dim = ctx + .get(&CanonicalDimensionName::from_raw("Source")) + .unwrap(); + let active = ctx.get(&CanonicalDimensionName::from_raw("Sub")).unwrap(); + assert_eq!( + ctx.resolve_mapped_read(source_dim, active, &CanonicalElementName::from_raw("p2")), + Some(CanonicalElementName::from_raw("y")), + "p2 is Parent's second element, so it reads Source's second" ); - assert_eq!(result, None); } - /// No declared mapping in either direction: None. + // ===== mapped_read_partner_dim: the pairing (GH #997) ===== + + /// The partner is the target-iterated dimension the index's dimension is + /// mapped to, in either declaration direction; a dimension the target does + /// not iterate is not a candidate. #[test] - fn test_mapped_correspondence_unmapped_is_none() { - use crate::common::CanonicalDimensionName; + fn mapped_read_partner_dim_finds_the_mapped_iterated_dimension() { + let mut region = dim("Region", &["a", "b"]); + region.set_maps_to("State".to_string()); + let ctx = DimensionsContext::from(&[ + region, + dim("State", &["s1", "s2"]), + dim("Age", &["young", "old"]), + ]); - let state = datamodel::Dimension::named( - "State".to_string(), - vec!["s1".to_string(), "s2".to_string()], + let region_name = CanonicalDimensionName::from_raw("Region"); + assert_eq!( + ctx.mapped_read_partner_dim(®ion_name, &["state".to_string(), "age".to_string()]), + Some(CanonicalDimensionName::from_raw("State")) ); - let region = datamodel::Dimension::named( - "Region".to_string(), - vec!["a".to_string(), "b".to_string()], + assert_eq!( + ctx.mapped_read_partner_dim(®ion_name, &["age".to_string()]), + None, + "no iterated dimension is mapped to Region" ); - let ctx = DimensionsContext::from(&[state, region]); + } - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("State"), - &CanonicalDimensionName::from_raw("Region"), + /// An index naming a dimension the target ITSELF iterates is the positional + /// spelling, not this rule's business: it declines so the caller's own + /// iterated-dim arm keeps it. + #[test] + fn mapped_read_partner_dim_declines_an_iterated_dimension_name() { + let mut region = dim("Region", &["a", "b"]); + region.set_maps_to("State".to_string()); + let ctx = DimensionsContext::from(&[region, dim("State", &["s1", "s2"])]); + + assert_eq!( + ctx.mapped_read_partner_dim( + &CanonicalDimensionName::from_raw("Region"), + &["region".to_string(), "state".to_string()] + ), + None ); - assert_eq!(result, None); } - /// Single-hop only, matching `has_mapping_to` (and the LTM - /// classifier): a chained `A→B→C` mapping yields None for `(A, C)`. + /// AMBIGUITY declines. Execution breaks the tie by position; a describer + /// that copied that would attribute influence along edges chosen by + /// declaration order, so both candidates disqualify the pairing. #[test] - fn test_mapped_correspondence_transitive_chain_is_none() { - use crate::common::CanonicalDimensionName; + fn mapped_read_partner_dim_declines_when_two_iterated_dims_are_viable() { + let mut region = dim("Region", &["a", "b"]); + region.mappings = vec![ + datamodel::DimensionMapping { + target: "State".to_string(), + element_map: vec![], + }, + datamodel::DimensionMapping { + target: "County".to_string(), + element_map: vec![], + }, + ]; + let ctx = DimensionsContext::from(&[ + region, + dim("State", &["s1", "s2"]), + dim("County", &["c1", "c2"]), + ]); - let mut dim_a = datamodel::Dimension::named( - "DimA".to_string(), - vec!["a1".to_string(), "a2".to_string()], - ); - dim_a.set_maps_to("DimB".to_string()); - let mut dim_b = datamodel::Dimension::named( - "DimB".to_string(), - vec!["b1".to_string(), "b2".to_string()], - ); - dim_b.set_maps_to("DimC".to_string()); - let dim_c = datamodel::Dimension::named( - "DimC".to_string(), - vec!["c1".to_string(), "c2".to_string()], + assert_eq!( + ctx.mapped_read_partner_dim( + &CanonicalDimensionName::from_raw("Region"), + &["state".to_string(), "county".to_string()] + ), + None ); - let ctx = DimensionsContext::from(&[dim_a, dim_b, dim_c]); + } - let result = ctx.mapped_element_correspondence( - &CanonicalDimensionName::from_raw("DimA"), - &CanonicalDimensionName::from_raw("DimC"), + /// A name that is no dimension at all declines (it is a variable read or a + /// literal, and neither is this rule's). + #[test] + fn mapped_read_partner_dim_declines_a_non_dimension_name() { + let ctx = DimensionsContext::from(&[dim("State", &["s1", "s2"])]); + assert_eq!( + ctx.mapped_read_partner_dim( + &CanonicalDimensionName::from_raw("not_a_dim"), + &["state".to_string()] + ), + None ); - assert_eq!(result, None); } // ========== Existing tests ========== diff --git a/src/simlin-engine/src/lib.rs b/src/simlin-engine/src/lib.rs index 13f811d4f..efadc3ab2 100644 --- a/src/simlin-engine/src/lib.rs +++ b/src/simlin-engine/src/lib.rs @@ -23,6 +23,8 @@ pub mod ai_info; mod alloc; pub mod analysis; #[cfg(test)] +mod array_operand_materialization_tests; +#[cfg(test)] mod array_tests; mod ast; pub mod builtins; @@ -63,6 +65,8 @@ pub mod ltm_finding; pub mod ltm_post; #[cfg(test)] mod macro_expansion_tests; +#[cfg(test)] +mod mapped_reference_semantics_tests; pub mod mdl; mod model; mod module_functions; diff --git a/src/simlin-engine/src/ltm_agg.rs b/src/simlin-engine/src/ltm_agg.rs index f1b9a27d1..a282506a0 100644 --- a/src/simlin-engine/src/ltm_agg.rs +++ b/src/simlin-engine/src/ltm_agg.rs @@ -73,9 +73,13 @@ //! is per-result-slot constant, the arrayed generalization of the GH #737 //! scalar feeder). The carve-outs: a reducer over a *dynamic index* //! (`SUM(pop[idx,*])`, `idx` non-literal) is not statically describable, a -//! mapped iterated axis whose mapping is element-mapped (GH #756) or -//! non-positional is declined (a positional mapping is accepted in EITHER -//! declaration direction since GH #757), a StarRange +//! mapped iterated axis with no DECLARED correspondence, or a cardinality +//! mismatch, is declined (a declared mapping is accepted in EITHER +//! declaration direction since GH #757, and since GH #997 an explicit +//! element map too -- this spelling folds to an ordinal, so the map is +//! honoured as a declaration but never read), a `MappedRead` axis (GH #997: +//! its executed rule admits a many-to-one correspondence the one-slot-per-row +//! remap cannot invert), a StarRange //! naming a NON-subdimension (a mid-edit inconsistency that must not //! silently widen to the full extent) is declined -- `compute_read_slice` //! returns `None`, the reducer is not hoisted, and its reference stays on @@ -403,6 +407,13 @@ pub(crate) fn builtin_routes_through_agg(builtin: &BuiltinFn) -> bool { /// every element of that axis feeds the agg result slot; with /// `subset: Some(elems)` (a StarRange over a PROPER subdimension, /// GH #766) only the subdimension's elements do. +/// - [`AxisRead::MappedRead`] -- the axis is likewise iterated over the +/// target's dimension space, but the subscript names a NON-ACTIVE dimension +/// (`x[Region]` under a `State`-iterating equation), which execution resolves +/// name-first and then through the declared element map rather than by +/// ordinal (GH #997). It is a DIRECT-reference verdict only: +/// [`compute_read_slice`] declines a slice containing one, so no aggregate +/// node ever carries it. /// /// `PartialOrd`/`Ord`/`Hash` ride along because `RefShape::PerElement` /// (GH #525, T6 of the shape-expressiveness design) embeds an @@ -440,6 +451,37 @@ pub enum AxisRead { /// the axis's elements -- a subdimension covering the whole axis /// normalizes to `None` so the full-extent representation is unique. Reduced { subset: Option> }, + /// This source axis is iterated over the target's dimension space, but the + /// subscript spells a dimension the equation does NOT iterate -- typically + /// the source's own (`ff_stop_growth_year_aggregated[Aggregated Regions]` + /// inside a `COP`-iterating equation, C-LEARN's shape and GH #997's). + /// + /// Structurally the same pair as [`AxisRead::Iterated`]; the difference is + /// the RESOLUTION RULE, which is why it is a separate variant rather than a + /// flag. The iterated spelling folds its index to an ordinal + /// (`ast::expr3`'s Pass 1) and never consults the declared element map; + /// this one survives to `IndexOp::ActiveDimRef` and + /// `compiler::subscript::build_view_from_ops` resolves it name-first, then + /// through the map -- so the two read DIFFERENT source elements wherever a + /// model declares an element map or the two dimensions share element + /// names. Every consumer must pick the matching correspondence + /// (`executed_read_correspondence` here, + /// `positional_correspondence` for `Iterated`), which is what a + /// separate variant makes a compile error rather than a silent + /// mis-attribution. + /// + /// Reachable only from the DIRECT-reference classifier: `compute_read_slice` + /// declines a reducer slice containing one, so the aggregate machinery's + /// slot remaps -- which are the preimage of a BIJECTION -- never meet the + /// many-to-one correspondence this variant admits. + MappedRead { + /// Canonical name of the TARGET equation's iterated dimension this axis + /// is paired with, as `DimensionsContext::mapped_read_partner_dim` + /// decides. + dim: String, + /// Canonical name of the SOURCE's declared dimension on this axis. + source_dim: String, + }, } /// The agg result-slot coordinate (an element of the `Iterated` axis's @@ -449,16 +491,24 @@ pub enum AxisRead { /// - Literal case (`target_dim == source_dim`): the identity -- slot /// coordinate == source element. /// - Mapped case (GH #534): the PREIMAGE inversion of -/// [`crate::dimensions::DimensionsContext::mapped_element_correspondence`] +/// [`crate::dimensions::DimensionsContext::positional_correspondence`] /// `(target_dim, source_dim)` -- that helper is indexed by TARGET element /// position and yields the source element read for it, so the slot for a /// given source element is the target element whose correspondence entry -/// names it. The helper's positional-only gate (an explicit element map -/// returns `None` -- GH #756) makes the correspondence a bijection -/// (index-identity, equal cardinality), so every source element has -/// exactly one preimage; the inversion is still written generally and -/// declines (returns `None`) if a source element has zero or multiple -/// preimages, mirroring `expand_same_element`'s general-shape inversion. +/// names it. +/// +/// The POSITIONAL correspondence is the right one here and not merely the +/// historical one: this helper serves an [`AxisRead::Iterated`] axis, whose +/// index spells a dimension the equation ITERATES, and `ast::expr3`'s Pass 1 +/// folds that to an ordinal without consulting any declared element map +/// (GH #997). Being positional it is also a bijection (index-identity, equal +/// cardinality), so every source element has exactly one preimage; the +/// inversion is still written generally and declines (returns `None`) if a +/// source element has zero or multiple preimages, mirroring +/// `expand_same_element`'s general-shape inversion. That generality is what +/// keeps the MANY-TO-ONE correspondence out: an `AxisRead::MappedRead` axis, +/// whose executed rule admits one, never reaches an agg read slice at all +/// (`compute_read_slice` declines it). /// /// `None` means "no usable slot remap": `compute_read_slice` then declines /// to hoist (classification), and the emitters fall back to their @@ -476,7 +526,7 @@ pub(crate) fn iterated_axis_slot_elements( } let t = CanonicalDimensionName::from_raw(target_dim); let s = CanonicalDimensionName::from_raw(source_dim); - let corr = dim_ctx.mapped_element_correspondence(&t, &s)?; + let corr = dim_ctx.positional_correspondence(&t, &s)?; let target_named = match dim_ctx.get(&t)? { crate::dimensions::Dimension::Named(_, named) => named, crate::dimensions::Dimension::Indexed(_, _) => return None, @@ -1374,6 +1424,11 @@ fn rank_result_dims_from_read_slice( ctx, )); } + // Unreachable: `compute_read_slice` declines a slice carrying a + // `MappedRead` axis, so no agg node holds one (GH #997). Declining + // rather than guessing keeps that a conservative fallback if the + // hoisting gate ever widens. + AxisRead::MappedRead { .. } => return None, } } Some(result_dims) @@ -1631,13 +1686,27 @@ fn compute_read_slice(arg_expr: &Expr2, ctx: &AggWalkCtx<'_>) -> Option = indices .iter() .zip(dims) .map(|(idx, axis_dim)| { classify_axis_access(idx, axis_dim, ctx.target_iterated_dims, ctx.dim_ctx) }) - .collect() + .collect::>()?; + // A `MappedRead` axis (GH #997) is a DIRECT-reference verdict only. + // Hoisting one would put a possibly MANY-TO-ONE correspondence into + // machinery whose slot remap is the preimage of a bijection + // (`iterated_axis_slot_elements`), so such a reducer keeps the + // conservative un-hoisted path it had before #997 -- unchanged + // behaviour, stated here rather than left to fall out of a missing + // arm somewhere downstream. + if slice + .iter() + .any(|ax| matches!(ax, AxisRead::MappedRead { .. })) + { + return None; + } + Some(slice) } _ => None, } @@ -1670,29 +1739,44 @@ fn compute_read_slice(arg_expr: &Expr2, ctx: &AggWalkCtx<'_>) -> Option None, + // The index names neither an element of this axis nor a + // dimension the equation iterates. It may still be a + // NON-ACTIVE dimension execution pairs with one of the + // target's iterated dims through a declared mapping + // (`x[Region]` under a `State`-iterating equation, GH #997) -- + // the spelling `compiler::subscript::normalize_subscripts3` + // turns into an `IndexOp::ActiveDimRef` and resolves + // name-first, then through the element map. `MappedRead` is + // that verdict, and it declines when the pairing is absent or + // ambiguous, or when the per-element correspondence is not + // usable -- in which case the reference keeps the conservative + // shape it had before. + crate::dimensions::AxisIndexName::Unresolved => { + let index_dim = crate::common::CanonicalDimensionName::from_raw(name_str); + let partner = + dim_ctx.mapped_read_partner_dim(&index_dim, target_iterated_dims)?; + dim_ctx.executed_read_correspondence(&partner, axis_dim.canonical_name())?; + Some(AxisRead::MappedRead { + dim: partner.as_str().to_string(), + source_dim: src_dim_name.to_string(), + }) + } } } IndexExpr2::Expr(Expr2::Const(..)) => { @@ -1987,6 +2093,9 @@ fn accept_source_slices(refs: Vec<(String, Vec)>) -> Option None, + // Unreachable in an agg slice (see `compute_read_slice`); the + // `Some(None)` declines the whole combination if it ever is. + AxisRead::MappedRead { .. } => Some(None), }) .collect() } @@ -2101,7 +2210,10 @@ fn result_dims_from_read_slice( // equation, the agg→target projection (GH #528), and the // element-graph slot naming all key on. AxisRead::Iterated { dim, .. } => Some(canonical_dim_to_datamodel(dim, dm_dims)), - AxisRead::Pinned(_) | AxisRead::Reduced { .. } => None, + // A `MappedRead` axis cannot reach an agg (`compute_read_slice` + // declines it, GH #997); it contributes no result axis, as this + // function's return type leaves no way to decline. + AxisRead::Pinned(_) | AxisRead::Reduced { .. } | AxisRead::MappedRead { .. } => None, }) .collect() } @@ -2209,6 +2321,8 @@ pub(crate) fn rank_output_slot_parts_for_row( AxisRead::Reduced { .. } => { per_output_axis.push(result_axis_elements.next()?.clone()); } + // Unreachable in an agg slice (see `compute_read_slice`). + AxisRead::MappedRead { .. } => return None, } } if result_axis_elements.next().is_some() { @@ -2468,7 +2582,12 @@ pub(crate) fn render_read_slice_for_diagnostic(slice: &[AxisRead]) -> String { .iter() .map(|ax| match ax { AxisRead::Pinned(e) => e.clone(), - AxisRead::Iterated { source_dim, .. } => source_dim.clone(), + // Both spellings render as the dimension NAME the index carries, + // which for a `MappedRead` is the source's own -- the index the + // equation actually spells. + AxisRead::Iterated { source_dim, .. } | AxisRead::MappedRead { source_dim, .. } => { + source_dim.clone() + } AxisRead::Reduced { subset: None } => "*".to_string(), AxisRead::Reduced { subset: Some(elems), diff --git a/src/simlin-engine/src/ltm_agg_tests.rs b/src/simlin-engine/src/ltm_agg_tests.rs index 9fbd0dfad..458f391f5 100644 --- a/src/simlin-engine/src/ltm_agg_tests.rs +++ b/src/simlin-engine/src/ltm_agg_tests.rs @@ -1075,13 +1075,18 @@ fn mapped_iterated_dim_sliced_reducer_is_hoisted_with_pair() { assert_eq!(synthetic[0].equation_text, "sum(matrix[state, *])"); } -/// GH #534 (conservative gate, element-mapped): a sliced reducer over an -/// EXPLICIT element-mapped pair stays un-hoisted -- the executed A2A -/// lowering resolves mapped references positionally and ignores the -/// element map (GH #756), so `mapped_element_correspondence` declines -/// and the reference keeps its conservative shape. -#[test] -fn element_mapped_sliced_reducer_is_not_hoisted() { +/// GH #997 (flipped from the GH #534-era conservative pin): a sliced reducer +/// over an EXPLICIT element-mapped pair IS hoisted, with a POSITIONAL slot +/// remap. +/// +/// `matrix[State, *]` spells the dimension the equation ITERATES, and +/// `mapped_reference_semantics_tests`' `(Permuted, IteratedDim)` cell measures +/// that spelling reading by ordinal against the VM -- the declared element map +/// is not consulted. The old decline came from one correspondence serving both +/// spellings and answering neither; `classify_axis_access` now asks +/// `positional_correspondence`, which describes this one exactly. +#[test] +fn element_mapped_sliced_reducer_is_hoisted_with_positional_slots() { let project = TestProject::new("element_mapped_slice") .named_dimension("Region", &["r1", "r2"]) .named_dimension("D2", &["x", "y"]) @@ -1100,19 +1105,26 @@ fn element_mapped_sliced_reducer_is_not_hoisted() { ); let result = agg_nodes(&project); - assert!( - result.aggs.iter().all(|a| !a.reads_var("matrix")), - "an element-mapped sliced reducer must not be hoisted; got: {:?}", - result.aggs + let synthetic: Vec<&AggNode> = result.aggs.iter().filter(|a| a.is_synthetic).collect(); + assert_eq!(synthetic.len(), 1, "got: {:?}", result.aggs); + assert_eq!( + synthetic[0].source_read_slice("matrix"), + &[ + AxisRead::Iterated { + dim: "state".to_string(), + source_dim: "region".to_string() + }, + AxisRead::Reduced { subset: None } + ] ); - assert!(result.synthetic_by_key.is_empty()); + assert_eq!(synthetic[0].result_dims, vec!["State".to_string()]); } /// GH #757 (flipped from the GH #534-era conservative pin): a sliced /// reducer whose POSITIONAL mapping is declared only in the REVERSE /// direction (on the source's `Region` toward `State`) is now hoisted -- /// `classify_axis_access`'s mapped arm gates on -/// `iterated_axis_slot_elements` / `mapped_element_correspondence`, +/// `iterated_axis_slot_elements` / `positional_correspondence`, /// which accepts both declaration directions (the compiler's /// `translate_via_mapping` resolves both, so declining one direction /// was pure over-conservatism). The slice and `result_dims` are @@ -1352,8 +1364,9 @@ fn whole_rhs_broadcast_pinned_mix_mints_synthetic_agg() { } /// GH #534: `iterated_axis_slot_elements` -- identity for the literal -/// case, the positional preimage for a mapped pair, `None` for an -/// element-mapped or unmapped pair. +/// case, and the positional preimage for any mapped pair -- element map +/// included, since this helper serves the ITERATED spelling and execution +/// resolves that by ordinal (GH #997). `None` for an unmapped pair. #[test] fn iterated_axis_slot_elements_cases() { use crate::datamodel::{Dimension as DmDimension, DimensionMapping}; @@ -1398,14 +1411,20 @@ fn iterated_axis_slot_elements_cases() { Some(vec!["s1".to_string(), "s2".to_string()]) ); - // Explicit element map: declined (GH #756 positional-only gate). + // Explicit element map: the POSITIONAL slots, not the map's. This asserted + // `None` until GH #997. `iterated_axis_slot_elements` serves the ITERATED + // spelling only (an `AxisRead::Iterated` axis, whose index names a + // dimension the equation iterates), which execution folds to an ordinal -- + // so the map is not consulted and the slots are the positional diagonal. + // The map here is the reverse permutation (s1↦r2), so an accidental + // map-following remap would give ["s2", "s1"] and fail this row. let ctx_elem = DimensionsContext::from(&[ named("Region", &["r1", "r2"], vec![]), named("State", &["s1", "s2"], vec![element_mapped]), ]); assert_eq!( iterated_axis_slot_elements("state", "region", ®ion_elems, &ctx_elem), - None + Some(vec!["s1".to_string(), "s2".to_string()]) ); // Unmapped pair: declined. @@ -2980,7 +2999,7 @@ fn classify_axis_access_resolves_a_colliding_name_element_first() { assert!(mapped_ctx.is_dimension_name("region")); assert!( mapped_ctx - .mapped_element_correspondence( + .positional_correspondence( &CanonicalDimensionName::from_raw("region"), &CanonicalDimensionName::from_raw("bucket"), ) diff --git a/src/simlin-engine/src/ltm_augment.rs b/src/simlin-engine/src/ltm_augment.rs index cc8f8b591..6e3bf2357 100644 --- a/src/simlin-engine/src/ltm_augment.rs +++ b/src/simlin-engine/src/ltm_augment.rs @@ -771,9 +771,13 @@ fn wrap_non_matching_in_previous( // `PREVIOUS(SUM(pop[*]))`, which is `PREVIOUS` of a scalar (the // reducer's result, even a partial reduce, is scalar in the // enclosing apply-to-all context) and evaluates fine -- rather - // than recursing into it and emitting `SUM(PREVIOUS(pop[*]))`, - // which is silently `0.0` at every step under an active A2A - // dimension because codegen has no LoadPrev-of-array-view path. + // than recursing into it and emitting `SUM(PREVIOUS(pop[*]))`. + // The wrap is the GH #517 semantics -- freeze the reducer's + // RESULT -- and is kept for that reason. Its original + // justification is stale: the inner form used to be a stubbed + // `0.0` at every step because codegen had no array-`PREVIOUS` + // path, and GH #995 phase C3 gave it one (a view over + // `prev_values`), so both forms compile now. // If the live reference *is* inside this reducer (the now // test-only `RefShape::Wildcard` path where `SUM(pop[*])` is the // live thing), recurse normally so the live `pop[*]` stays @@ -991,178 +995,10 @@ fn freeze_lookup_table_indices( Expr0::Subscript(ident, indices, loc) } -/// A parse failure in a ceteris-paribus partial-equation builder. -/// -/// The ceteris-paribus PREVIOUS-wrapping transform ([`wrap_non_matching_in_previous`]) -/// can only run on a successfully-parsed `Expr0`. If `Expr0::new` returns -/// `Err` (genuinely unparseable text) or `Ok(None)` (an empty/whitespace -/// equation), there is *no* AST to wrap, so the transform cannot be applied. -/// -/// Why this is an error rather than a silent fallback (GH #311): the prior -/// code returned the lowercased input text unchanged on parse failure. With -/// no PREVIOUS() wrapping, that "partial" is identical to the target's full -/// equation, so the link-score numerator `(partial - PREVIOUS(target))` -/// equals the denominator `(target - PREVIOUS(target))` and the score -/// magnitude collapses to a constant `|Δz/Δz| = 1` -- a hidden attribution -/// error that is *worse* than no score at all, and one that compiles cleanly -/// so no downstream diagnostic catches it. Returning a structured error lets -/// the (db-bearing) caller skip emitting the link-score variable and surface -/// a `Warning` naming the variable and the offending equation text, the -/// established "loud failure" pattern in this codebase -/// (cf. `emit_unscoreable_disjoint_edge_warning`). -/// -/// The text being parsed is itself produced by the engine (`print_eqn` / -/// `expr2_to_string` over a compiled AST), so `Err` is effectively -/// unreachable in production; `Ok(None)` is reachable for a target with an -/// empty equation. Either way the failure is rare and unexpected -- exactly -/// the case where a silent semantics-changing fallback is most dangerous. -/// -/// `UnfreezablePartial` (GH #743) is the second loud-failure class: the -/// equation parsed fine, but neither ceteris-paribus convention can be -/// rendered as a compilable equation -- the changed-first partial would -/// freeze an array slice (`PREVIOUS(matrix[d1,*])`, which has no -/// LoadPrev-of-array-view codegen path: a hard compile error in a user -/// equation, and a SILENTLY-stubbed-to-0 helper in an LTM fragment, which -/// poisoned the score into plausible-looking garbage like the constant -/// `-1/growth-rate`), and the changed-last fallback is unfreezable too (or -/// has no live occurrence to freeze). The caller skips the score and warns. -#[derive(Debug, Clone, PartialEq, Eq)] -pub(crate) enum PartialEquationErrorKind { - /// The equation text failed to parse (or was empty); there is no AST - /// to transform. - Parse, - /// Neither the changed-first nor the changed-last ceteris-paribus - /// convention can be rendered as a compilable equation (GH #743). - UnfreezablePartial, - /// The live source is a BARE reference to an arrayed variable inside an - /// array-reducer argument (GH #779): the changed-last partial cannot be - /// rendered faithfully for it, and the spelling's own execution - /// semantics carry a spurious factor (GH #789). Selects a diagnostic - /// that names the shape and the subscripted-spelling workaround. - BareReducerFeeder, - /// An arrayed dep of the target's equation cannot be projected onto the - /// target element this partial is for, so no correct element subscript - /// exists for it. `equation_text` carries `dep@element`. Emitting anyway - /// leaves the dep's dimension-name subscript in a scalar fragment, which - /// becomes a `PREVIOUS`-capture helper that cannot lower WHILE THE PARENT - /// STILL COMPILES -- a score that silently reads part of its own equation - /// as 0. The reachable cause is an explicit element map, which - /// `DimensionsContext::mapped_element_correspondence` declines. - UnprojectableDep, - /// The target's equation applies an ORDER-STATISTIC, array-producing - /// builtin (`VECTOR SORT ORDER`, `RANK`, `ALLOCATE AVAILABLE`, - /// `ALLOCATE BY PRIORITY`) and this partial is a per-element SCALAR one - /// (GH #995 option C): the scalarization pins the builtin's argument down - /// to a single element, and an order statistic of one element is - /// meaningless (`vm_vector_sort_order` on a 1-element view is rank 0 - /// always). Today such a fragment also fails codegen loudly - /// ("array-producing builtin outside AssignTemp context"); declining at - /// generation keeps the drop loud even if a future Pass-1 widening - /// (option A) makes the fragment compile -- which would otherwise convert - /// it into a silent constant-0 partial. The element pin belongs on the - /// RESULT (the A2A-shaped whole-array score, which stays emitted), never - /// on a rank-like builtin's argument. - RankLikePartial, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub(crate) struct PartialEquationError { - /// The original (pre-transform) equation text the failure is about. The - /// db-bearing caller embeds this in the diagnostic message so the failure - /// names the concrete offending equation. - pub equation_text: String, - /// Which loud-failure class this is; selects the diagnostic wording. - pub kind: PartialEquationErrorKind, -} - -impl PartialEquationError { - pub(crate) fn new(equation_text: &str) -> Self { - PartialEquationError { - equation_text: equation_text.to_string(), - kind: PartialEquationErrorKind::Parse, - } - } - - fn unfreezable(equation_text: &str) -> Self { - PartialEquationError { - equation_text: equation_text.to_string(), - kind: PartialEquationErrorKind::UnfreezablePartial, - } - } - - fn bare_reducer_feeder(equation_text: &str) -> Self { - PartialEquationError { - equation_text: equation_text.to_string(), - kind: PartialEquationErrorKind::BareReducerFeeder, - } - } - - /// `dep` cannot be projected onto target element `element`. - pub(crate) fn unprojectable_dep(dep: &str, element: &str) -> Self { - PartialEquationError { - equation_text: format!("{dep}@{element}"), - kind: PartialEquationErrorKind::UnprojectableDep, - } - } - - fn rank_like_partial(equation_text: &str) -> Self { - PartialEquationError { - equation_text: equation_text.to_string(), - kind: PartialEquationErrorKind::RankLikePartial, - } - } -} - -/// Does `expr` apply an ARRAY-PRODUCING builtin -- the set a per-element -/// SCALAR partial must decline over (GH #995 option C)? -/// -/// The set is exactly codegen's AssignTemp-required family -/// (`compiler::codegen`'s `TodoArrayBuiltin` arms): `VECTOR SORT ORDER`, -/// `RANK`, `ALLOCATE AVAILABLE`, `ALLOCATE BY PRIORITY`, and -/// `VECTOR ELM MAP` -- every builtin whose RESULT is an array, which a -/// scalar fragment cannot hold. The order-statistic subset (everything but -/// ELM MAP) is additionally a semantic trap: pinning its argument to one -/// element changes the ranking rather than selecting a slot, so those must -/// stay declined even if a future Pass-1 widening makes the fragment -/// compile. Deliberately NOT in the set: `VECTOR SELECT`, whose selection -/// reduces to a scalar (per-element pinning of the non-reduced axes is -/// exactly right). This is the same result-type distinction -/// `ltm_agg::reducer_collapses_to_scalar` draws for `RANK` (GH #771/#742), -/// applied at the partial-generation boundary. -fn contains_rank_like_builtin(expr: &Expr0) -> bool { - let is_rank_like = |name: &str| { - matches!( - name.to_ascii_lowercase().as_str(), - "vector_sort_order" - | "rank" - | "allocate_available" - | "allocate_by_priority" - | "vector_elm_map" - ) - }; - match expr { - Expr0::Const(..) | Expr0::Var(..) => false, - Expr0::Subscript(_, indices, _) => indices.iter().any(|idx| match idx { - IndexExpr0::Expr(e) => contains_rank_like_builtin(e), - IndexExpr0::Range(l, r, _) => { - contains_rank_like_builtin(l) || contains_rank_like_builtin(r) - } - IndexExpr0::Wildcard(_) - | IndexExpr0::StarRange(_, _) - | IndexExpr0::DimPosition(_, _) => false, - }), - Expr0::App(UntypedBuiltinFn(name, args), _) => { - is_rank_like(name) || args.iter().any(contains_rank_like_builtin) - } - Expr0::Op1(_, inner, _) => contains_rank_like_builtin(inner), - Expr0::Op2(_, l, r, _) => contains_rank_like_builtin(l) || contains_rank_like_builtin(r), - Expr0::If(c, t, e, _) => { - contains_rank_like_builtin(c) - || contains_rank_like_builtin(t) - || contains_rank_like_builtin(e) - } - } -} +#[path = "ltm_augment_partial_error.rs"] +mod partial_error; +use partial_error::contains_rank_like_builtin; +pub(crate) use partial_error::{PartialEquationError, PartialEquationErrorKind}; /// Build a partial equation for a per-shape link score. /// @@ -1382,12 +1218,21 @@ fn wrap_changed_first_ast( /// expression evaluates to an array view, not a scalar. /// /// Used by [`contains_unfreezable_previous`] to decide whether a `PREVIOUS` -/// argument can be frozen: `PREVIOUS` of an array view has no codegen path -/// (no LoadPrev-of-array-view), so `PREVIOUS(matrix[d1,*])` -- or any -/// expression embedding such a slice outside a reducer -- cannot compile. -/// A reducer application (`SUM(matrix[d1,*])`) collapses the slice to a -/// scalar, so a wildcard *inside* a reducer is fine (`PREVIOUS(SUM(arr[*]))` -/// is the deliberate GH #517 whole-reducer freeze). +/// argument is one this layer will spell INLINE. A reducer application +/// (`SUM(matrix[d1,*])`) collapses the slice to a scalar, so a wildcard +/// *inside* a reducer is fine (`PREVIOUS(SUM(arr[*]))` is the deliberate +/// GH #517 whole-reducer freeze); a slice that no reducer collapses is routed +/// to the materialized freeze helper ([`crate::ltm_augment_array_freeze`], +/// GH #1003) or declined. +/// +/// The original reason -- "`PREVIOUS` of an array view has no codegen path" -- +/// is stale as of GH #995 phase C3, which gave it one (a view over +/// `prev_values`). The routing is unchanged because the helper buys something +/// the inline spelling does not: its arms are qualified with the AXIS +/// dimension, so a named subdimension that is not a positional prefix of its +/// parent still reads the name-correct row (PR #1001). Retiring the helper in +/// favour of the inline form is an open simplification, and would have to +/// carry that guarantee. fn expr_is_array_slice_valued(expr: &Expr0) -> bool { match expr { Expr0::Const(..) | Expr0::Var(..) => false, @@ -1425,17 +1270,25 @@ fn expr_is_array_slice_valued(expr: &Expr0) -> bool { /// call whose argument is array-slice-valued (see /// [`expr_is_array_slice_valued`])? /// -/// Such a partial can never evaluate correctly (GH #743): `PREVIOUS` of an -/// array view has no codegen path. As a *user* equation it is a hard -/// `NotSimulatable` compile error; as an LTM link-score fragment the doomed -/// `PREVIOUS` is routed through a synthesized implicit helper -/// (`$⁚$⁚ltm⁚…⁚arg0`) whose fragment fails to compile SILENTLY -- it keeps -/// a layout slot with no bytecode and reads a constant 0 -- so the partial -/// silently loses the frozen term while the outer score still compiles, -/// producing plausible-looking garbage (the constant `-1/growth-rate` -/// scores of GH #743). The partial-equation builders therefore treat this -/// shape as a routing decision: fall back to the changed-last attribution, -/// or fail loudly. +/// GH #743's original reading was that such a partial can never evaluate +/// correctly, because `PREVIOUS` of an array view had no codegen path: as a +/// *user* equation it was a hard `NotSimulatable` compile error, and as an LTM +/// link-score fragment the doomed `PREVIOUS` was routed through a synthesized +/// implicit helper (`$⁚$⁚ltm⁚…⁚arg0`) whose fragment failed to compile SILENTLY +/// -- it kept a layout slot with no bytecode and read a constant 0 -- so the +/// partial silently lost the frozen term while the outer score still compiled, +/// producing plausible-looking garbage (the constant `-1/growth-rate` scores of +/// GH #743). +/// +/// BOTH halves of that premise have moved and the routing is kept on other +/// grounds. GH #1003 materializes most of these as a `$⁚ltm⁚freeze⁚…` helper +/// ([`crate::ltm_augment_array_freeze`]) whose arms are qualified against the +/// AXIS dimension, and GH #995 phase C3 gave the inline spelling a path of its +/// own (a view over `prev_values`). What this predicate still routes is the +/// residue neither reaches -- a slice this layer will not spell inline and no +/// helper can materialize. The partial-equation builders therefore treat this +/// shape as a routing decision: fall back to the changed-last attribution, or +/// fail loudly. fn contains_unfreezable_previous(expr: &Expr0) -> bool { match expr { Expr0::Const(..) | Expr0::Var(..) => false, @@ -2274,14 +2127,22 @@ fn pin_iterated_dim_indices(expr: Expr0, dims: &[String], parts: &[String]) -> O /// type is just how the answer travels to the rewrite. #[derive(Clone)] pub(crate) struct DepElementPin { - /// The resolved axes, in the dep's declaration order. + /// The resolved axes for an already-SUBSCRIPTED reference whose index names + /// one of the dep's own dimensions (`dep[Region]`), as + /// `(dimension name, element spelling)` in the dep's declaration order. An + /// axis that does not project is simply absent, which is all such a + /// reference needs -- it spells its other axes itself. pub(crate) axes: Vec<(String, String)>, - /// Whether `axes` covers EVERY dimension the dep declares. Only a complete - /// pin can subscript a BARE reference, which must be spelled at the dep's - /// full arity; an incomplete one still substitutes the dimension-name - /// indices of an already-subscripted reference, which is all that reference - /// needs. - pub(crate) complete: bool, + /// The full row a BARE reference (`dep`) is spelled with, in the dep's + /// declaration order, or `None` when some axis does not project (a bare + /// reference must be spelled at the dep's full arity or not at all). + /// + /// A separate row rather than a `complete` flag over `axes` because the two + /// spellings resolve by DIFFERENT rules (GH #997): a bare reference is + /// rewritten into the iterated spelling and read positionally, while a + /// dimension-name subscript follows the declared element map. See + /// `post_transform::dep_element_pins`. + pub(crate) bare_row: Option>, } /// Replace every reference to a pinned dep in `equation_text` with that dep's @@ -2424,12 +2285,12 @@ fn subscript_idents_in_expr0( Expr0::Const(..) => expr, Expr0::Var(ref ident, loc) => { let canonical = Ident::new(ident.as_str()); - // Only a COMPLETE pin can spell a bare reference: a subscript + // Only a COMPLETE row can spell a bare reference: a subscript // covering some of the dep's axes is not a legal reference at all. - match pins.get(&canonical).filter(|pin| pin.complete) { - Some(pin) => Expr0::Subscript( + match pins.get(&canonical).and_then(|pin| pin.bare_row.as_ref()) { + Some(row) => Expr0::Subscript( ident.clone(), - pin.axes.iter().map(|(_, elem)| pin_index(elem)).collect(), + row.iter().map(|elem| pin_index(elem)).collect(), loc, ), None => expr, @@ -2998,9 +2859,10 @@ pub(crate) fn quote_ident(ident: &str) -> String { /// shapes that reach `emit_per_shape_link_scores` are a *whole-RHS* /// variable-backed reducer's argument (`total = SUM(population[*])`), a /// bare dynamic index (`arr[i+1]`), the dynamic-index reducer carve-out -/// (`SUM(pop[idx, *])`), a mapped sliced reducer the correspondence -/// declines (element-mapped, the GH #756 positional-only rule; -/// reverse-declared positional pairs are accepted since GH #757), +/// (`SUM(pop[idx, *])`), a sliced reducer the correspondence declines (an +/// UNDECLARED pair, a cardinality mismatch, or a `MappedRead` axis -- +/// GH #997; a DECLARED mapping is accepted in either direction since +/// GH #757, an explicit element map included since #997), /// or a DE-HOISTED array-valued reducer's wildcard arg /// (`RANK(pop[*], 1)` -- GH #771: RANK is not `reducer_is_hoistable`, so /// its wildcard-subscripted argument stays a `Direct` `Wildcard` site and diff --git a/src/simlin-engine/src/ltm_augment_array_freeze.rs b/src/simlin-engine/src/ltm_augment_array_freeze.rs index d2d46c930..56cdb3216 100644 --- a/src/simlin-engine/src/ltm_augment_array_freeze.rs +++ b/src/simlin-engine/src/ltm_augment_array_freeze.rs @@ -7,13 +7,19 @@ //! //! A ceteris-paribus partial freezes every non-live reference at `PREVIOUS`. //! When the frozen reference is an ARRAY SLICE (`arr[pin, *]`, -//! `arr[pin, *:Sub]`), the inline spelling `PREVIOUS()` cannot compile: -//! codegen requires an array-valued operand to be a view over storage, and -//! there is no LoadPrev-of-a-view. The wrap used to either decline the score +//! `arr[pin, *:Sub]`), the inline spelling `PREVIOUS()` did not compile: +//! codegen required an array-valued operand to be a view over storage, and +//! there was no LoadPrev-of-a-view. The wrap used to either decline the score //! loudly (`UnfreezablePartial`, GH #743) or -- on the per-target-element //! emitter path, which never doom-checked -- emit a fragment that failed //! codegen and read a constant 0. //! +//! GH #995 phase C3 has since given the inline form a path of its own (a view +//! over `prev_values`), so "cannot compile" is no longer why this module +//! exists. It is retained for the name-correct row rule stated below -- the +//! arms are qualified against the AXIS dimension, which the inline spelling +//! does not do -- and collapsing the two is tracked as follow-on work. +//! //! [`materialize_array_freezes`] rewrites each such `PREVIOUS()` into a //! reference to a synthesized `$⁚ltm⁚freeze⁚…` helper: an `Equation::Arrayed` //! aux with one arm per slice row, each arm `PREVIOUS(arr[pin, axis·elem])` -- diff --git a/src/simlin-engine/src/ltm_augment_partial_error.rs b/src/simlin-engine/src/ltm_augment_partial_error.rs new file mode 100644 index 000000000..bd3e6a7d7 --- /dev/null +++ b/src/simlin-engine/src/ltm_augment_partial_error.rs @@ -0,0 +1,199 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! The ceteris-paribus partial-equation FAILURE vocabulary: the typed error a +//! partial-equation builder returns instead of emitting a wrong-but-compiling +//! score, plus the array-producing-builtin walk the `RankLikePartial` class is +//! decided by. A child of `ltm_augment` (mounted via `#[path]`, so `super::*` +//! resolves the parent's items and callers keep their +//! `crate::ltm_augment::*` paths); split out only for the per-file line cap. + +use crate::ast::{Expr0, IndexExpr0}; +use crate::builtins::UntypedBuiltinFn; + +/// A parse failure in a ceteris-paribus partial-equation builder. +/// +/// The ceteris-paribus PREVIOUS-wrapping transform ([`wrap_non_matching_in_previous`]) +/// can only run on a successfully-parsed `Expr0`. If `Expr0::new` returns +/// `Err` (genuinely unparseable text) or `Ok(None)` (an empty/whitespace +/// equation), there is *no* AST to wrap, so the transform cannot be applied. +/// +/// Why this is an error rather than a silent fallback (GH #311): the prior +/// code returned the lowercased input text unchanged on parse failure. With +/// no PREVIOUS() wrapping, that "partial" is identical to the target's full +/// equation, so the link-score numerator `(partial - PREVIOUS(target))` +/// equals the denominator `(target - PREVIOUS(target))` and the score +/// magnitude collapses to a constant `|Δz/Δz| = 1` -- a hidden attribution +/// error that is *worse* than no score at all, and one that compiles cleanly +/// so no downstream diagnostic catches it. Returning a structured error lets +/// the (db-bearing) caller skip emitting the link-score variable and surface +/// a `Warning` naming the variable and the offending equation text, the +/// established "loud failure" pattern in this codebase +/// (cf. `emit_unscoreable_disjoint_edge_warning`). +/// +/// The text being parsed is itself produced by the engine (`print_eqn` / +/// `expr2_to_string` over a compiled AST), so `Err` is effectively +/// unreachable in production; `Ok(None)` is reachable for a target with an +/// empty equation. Either way the failure is rare and unexpected -- exactly +/// the case where a silent semantics-changing fallback is most dangerous. +/// +/// `UnfreezablePartial` (GH #743) is the second loud-failure class: the +/// equation parsed fine, but neither ceteris-paribus convention can be +/// rendered as a compilable equation -- the changed-first partial would +/// freeze an array slice (`PREVIOUS(matrix[d1,*])`) and the changed-last +/// fallback is unfreezable too (or has no live occurrence to freeze). The +/// caller skips the score and warns. +/// +/// The compilability premise behind the array-slice half has MOVED TWICE and +/// the class is retained on neither of its original grounds. `PREVIOUS` of an +/// array slice was a hard compile error in a user equation and a +/// silently-stubbed-to-0 helper in an LTM fragment (poisoning a score into +/// plausible garbage like the constant `-1/growth-rate`); GH #1003 then +/// materialized the freeze as its own synthetic variable +/// ([`crate::ltm_augment_array_freeze`]), so most of these score instead of +/// declining; and GH #995 phase C3 gave the inline form a codegen path of its +/// own (a view over `prev_values`). What is left is the residue no helper can +/// materialize -- a dynamically pinned slice -- which is why the class stays. +/// Whether the freeze helper itself is still needed now that the inline form +/// compiles is an open simplification, deliberately not taken with C3. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum PartialEquationErrorKind { + /// The equation text failed to parse (or was empty); there is no AST + /// to transform. + Parse, + /// Neither the changed-first nor the changed-last ceteris-paribus + /// convention can be rendered as a compilable equation (GH #743). + UnfreezablePartial, + /// The live source is a BARE reference to an arrayed variable inside an + /// array-reducer argument (GH #779): the changed-last partial cannot be + /// rendered faithfully for it, and the spelling's own execution + /// semantics carry a spurious factor (GH #789). Selects a diagnostic + /// that names the shape and the subscripted-spelling workaround. + BareReducerFeeder, + /// An arrayed dep of the target's equation cannot be projected onto the + /// target element this partial is for, so no correct element subscript + /// exists for it. `equation_text` carries `dep@element`. Emitting anyway + /// leaves the dep's dimension-name subscript in a scalar fragment, which + /// becomes a `PREVIOUS`-capture helper that cannot lower WHILE THE PARENT + /// STILL COMPILES -- a score that silently reads part of its own equation + /// as 0. The reachable cause is a pair with no DECLARED correspondence at + /// all -- two dimensions sharing element names, which the simulation + /// resolves by name while `allocate_implicit_axes_partial` pairs axes only + /// by name or by a declared mapping. (An explicit element map was the + /// reachable cause until GH #997 made that spelling projectable.) + UnprojectableDep, + /// The target's equation applies an ORDER-STATISTIC, array-producing + /// builtin (`VECTOR SORT ORDER`, `RANK`, `ALLOCATE AVAILABLE`, + /// `ALLOCATE BY PRIORITY`) and this partial is a per-element SCALAR one + /// (GH #995 option C): the scalarization pins the builtin's argument down + /// to a single element, and an order statistic of one element is + /// meaningless (`vm_vector_sort_order` on a 1-element view is rank 0 + /// always). Today such a fragment also fails codegen loudly + /// ("array-producing builtin outside AssignTemp context"); declining at + /// generation keeps the drop loud even if a future Pass-1 widening + /// (option A) makes the fragment compile -- which would otherwise convert + /// it into a silent constant-0 partial. The element pin belongs on the + /// RESULT (the A2A-shaped whole-array score, which stays emitted), never + /// on a rank-like builtin's argument. + RankLikePartial, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct PartialEquationError { + /// The original (pre-transform) equation text the failure is about. The + /// db-bearing caller embeds this in the diagnostic message so the failure + /// names the concrete offending equation. + pub equation_text: String, + /// Which loud-failure class this is; selects the diagnostic wording. + pub kind: PartialEquationErrorKind, +} + +impl PartialEquationError { + pub(crate) fn new(equation_text: &str) -> Self { + PartialEquationError { + equation_text: equation_text.to_string(), + kind: PartialEquationErrorKind::Parse, + } + } + + pub(super) fn unfreezable(equation_text: &str) -> Self { + PartialEquationError { + equation_text: equation_text.to_string(), + kind: PartialEquationErrorKind::UnfreezablePartial, + } + } + + pub(super) fn bare_reducer_feeder(equation_text: &str) -> Self { + PartialEquationError { + equation_text: equation_text.to_string(), + kind: PartialEquationErrorKind::BareReducerFeeder, + } + } + + /// `dep` cannot be projected onto target element `element`. + pub(crate) fn unprojectable_dep(dep: &str, element: &str) -> Self { + PartialEquationError { + equation_text: format!("{dep}@{element}"), + kind: PartialEquationErrorKind::UnprojectableDep, + } + } + + pub(super) fn rank_like_partial(equation_text: &str) -> Self { + PartialEquationError { + equation_text: equation_text.to_string(), + kind: PartialEquationErrorKind::RankLikePartial, + } + } +} + +/// Does `expr` apply an ARRAY-PRODUCING builtin -- the set a per-element +/// SCALAR partial must decline over (GH #995 option C)? +/// +/// The set is exactly codegen's AssignTemp-required family +/// (`compiler::codegen`'s `TodoArrayBuiltin` arms): `VECTOR SORT ORDER`, +/// `RANK`, `ALLOCATE AVAILABLE`, `ALLOCATE BY PRIORITY`, and +/// `VECTOR ELM MAP` -- every builtin whose RESULT is an array, which a +/// scalar fragment cannot hold. The order-statistic subset (everything but +/// ELM MAP) is additionally a semantic trap: pinning its argument to one +/// element changes the ranking rather than selecting a slot, so those must +/// stay declined even if a future Pass-1 widening makes the fragment +/// compile. Deliberately NOT in the set: `VECTOR SELECT`, whose selection +/// reduces to a scalar (per-element pinning of the non-reduced axes is +/// exactly right). This is the same result-type distinction +/// `ltm_agg::reducer_collapses_to_scalar` draws for `RANK` (GH #771/#742), +/// applied at the partial-generation boundary. +pub(super) fn contains_rank_like_builtin(expr: &Expr0) -> bool { + let is_rank_like = |name: &str| { + matches!( + name.to_ascii_lowercase().as_str(), + "vector_sort_order" + | "rank" + | "allocate_available" + | "allocate_by_priority" + | "vector_elm_map" + ) + }; + match expr { + Expr0::Const(..) | Expr0::Var(..) => false, + Expr0::Subscript(_, indices, _) => indices.iter().any(|idx| match idx { + IndexExpr0::Expr(e) => contains_rank_like_builtin(e), + IndexExpr0::Range(l, r, _) => { + contains_rank_like_builtin(l) || contains_rank_like_builtin(r) + } + IndexExpr0::Wildcard(_) + | IndexExpr0::StarRange(_, _) + | IndexExpr0::DimPosition(_, _) => false, + }), + Expr0::App(UntypedBuiltinFn(name, args), _) => { + is_rank_like(name) || args.iter().any(contains_rank_like_builtin) + } + Expr0::Op1(_, inner, _) => contains_rank_like_builtin(inner), + Expr0::Op2(_, l, r, _) => contains_rank_like_builtin(l) || contains_rank_like_builtin(r), + Expr0::If(c, t, e, _) => { + contains_rank_like_builtin(c) + || contains_rank_like_builtin(t) + || contains_rank_like_builtin(e) + } + } +} diff --git a/src/simlin-engine/src/ltm_augment_pin_tests.rs b/src/simlin-engine/src/ltm_augment_pin_tests.rs index c12d9ce58..8720300d7 100644 --- a/src/simlin-engine/src/ltm_augment_pin_tests.rs +++ b/src/simlin-engine/src/ltm_augment_pin_tests.rs @@ -140,11 +140,9 @@ fn per_element_pin_descends_into_range_endpoints() { /// /// - per AXIS, [`dep_axis_elements`](super::post_transform) has three outcomes: /// the target ITERATES the dep's own dimension (identity), the target -/// iterates a dimension with a usable positional CORRESPONDENCE to it -/// (`mapped_element_correspondence`), or neither (declined). The declined arm -/// is reached two ways -- an unrelated dimension, and an explicit ELEMENT map, -/// which the correspondence refuses because execution resolves positionally -/// and ignores it (GH #756) -- and both are rows; +/// iterates a dimension with a usable CORRESPONDENCE to it (which of the two +/// the spelling picks -- GH #997), or neither (declined). The declined arm is +/// reached by an unrelated dimension, and is a row; /// - per DEP, the axis outcomes combine into three: every axis resolved /// (`complete`, the only kind that may subscript a bare reference), some /// resolved (present but incomplete -- it may still substitute a @@ -247,7 +245,7 @@ fn dep_element_pins_projection_enumeration() { name: &str| -> Option<(Vec<(String, String)>, bool)> { pins.get(&Ident::::new(name)) - .map(|p| (p.axes.clone(), p.complete)) + .map(|p| (p.axes.clone(), p.bare_row.is_some())) }; let axis = |dim: &str, elem: &str| (dim.to_string(), elem.to_string()); @@ -342,15 +340,20 @@ fn dep_element_pins_projection_enumeration() { assert_eq!( repeated_pins .get(&Ident::::new("w")) - .map(|p| (p.axes.clone(), p.complete)), + .map(|p| (p.axes.clone(), p.bare_row.is_some())), Some((vec![axis("region", "region\u{B7}nyc")], true)), "a subset dep under a repeated-dimension target reads the FIRST axis; a \ name-keyed map keeps only the last and would say `boston`" ); - // An EXPLICIT element map is declined by `mapped_element_correspondence` - // (execution resolves positionally and ignores it, GH #756), so the same - // `mapped[State,Age]` dep loses its State axis and becomes incomplete. + // An EXPLICIT element map is where the pin's TWO rows part (GH #997). This + // block asserted a single declining row until then, on the reasoning that + // execution "resolves positionally and ignores the map" -- true of a BARE + // reference and false of a `mapped[State, ...]` subscript, and one row could + // not say both. The map below sends `west` to `boston`, the REVERSE of the + // positional diagonal (`boston` is Region's second, so positionally it reads + // State's second, `east`), so the two rows disagree on every element and + // neither assertion can pass by accident. let element_mapped = build_ctx(vec![ ("west".to_string(), "boston".to_string()), ("east".to_string(), "nyc".to_string()), @@ -361,11 +364,51 @@ fn dep_element_pins_projection_enumeration() { &target_elements, &element_mapped, ); + let mapped_pin = pins + .get(&Ident::::new("mapped")) + .expect("the mapped dep projects on both rows"); assert_eq!( - axes_of(&pins, "mapped"), - Some((vec![axis("age", "age\u{B7}young")], false)), - "an element-mapped axis must decline: following the map would spell a \ - read the positionally-resolving simulation never performs" + mapped_pin.axes, + vec![ + axis("state", "state\u{B7}west"), + axis("age", "age\u{B7}young") + ], + "a `mapped[State, Age]` subscript FOLLOWS the declared element map: the \ + index survives to `IndexOp::ActiveDimRef` and `build_view_from_ops` \ + resolves it name-first, then through the map" + ); + assert_eq!( + mapped_pin.bare_row, + Some(vec![ + "state\u{B7}east".to_string(), + "age\u{B7}young".to_string() + ]), + "a BARE `mapped` reference is rewritten into the iterated spelling by \ + pass 0 and read by ORDINAL, so it reads State's second element -- the \ + other one" + ); + + // The same dep under a POSITIONAL mapping: the two rows coincide, which is + // why one row sufficed before GH #997 and why nothing about the shipped + // positional cases moves. + let positional_pins = super::post_transform::dep_element_pins( + &pinnable, + &target_dims, + &target_elements, + &positional, + ); + let positional_mapped = positional_pins + .get(&Ident::::new("mapped")) + .expect("the mapped dep projects"); + assert_eq!( + positional_mapped.bare_row, + Some( + positional_mapped + .axes + .iter() + .map(|(_, elem)| elem.clone()) + .collect::>() + ) ); } @@ -459,11 +502,11 @@ impl PinFixture { /// /// `declare_on_state` picks the DECLARATION DIRECTION: `true` declares the /// mapping on `State` toward `Region`, `false` on `Region` toward `State`. - /// `mapped_element_correspondence` honors both (GH #757), so both must pin. + /// Both correspondences honor both directions (GH #757), so both must pin. /// A non-empty `element_map` makes the mapping an EXPLICIT element map, which - /// the correspondence declines (GH #756: the executed A2A lowering resolves - /// positionally and ignores the map, so following it would spell a row the - /// simulation never reads). + /// changes nothing for the ITERATED spelling these fixtures use: execution + /// folds that index to an ordinal and never reads the map (GH #997), so the + /// pin is the positional element either way. fn mapped(declare_on_state: bool, element_map: Vec<(String, String)>) -> Self { use crate::ltm_agg::AxisRead; let mut state = datamodel::Dimension::named( @@ -1496,14 +1539,19 @@ fn per_element_pin_colliding_element_name_verdict_enumeration() { /// dropped the whole `pop -> growth` score edge. /// /// The verdicts are not this rule's opinion. Each row is whatever -/// `DimensionsContext::mapped_element_correspondence` says, reached through +/// `DimensionsContext::positional_correspondence` says, reached through /// [`super::post_transform::per_element_row_for_target`] -- the SAME derivation the /// occurrence-driven pin and the link-score NAME use, and the same one /// `ltm_agg::classify_axis_access` gates its `Iterated` arm on (through /// `iterated_axis_slot_elements`, the correspondence's preimage inversion). So the /// rows below double as an agreement statement: a mapped pair this rule pins is /// exactly a mapped pair the classifier calls `Iterated`, in both declaration -/// directions, and an element-mapped pair declines in both. +/// directions. +/// +/// The correspondence is the POSITIONAL one because every index here spells a +/// dimension the target ITERATES, which `ast::expr3`'s Pass 1 folds to an ordinal +/// (GH #997). An element-mapped pair therefore pins too -- and pins to the +/// ORDINAL's element, not the map's. /// /// Both columns are the SAME for every row here, and that is the point: these are /// `Pinned` and `Unspellable` verdicts, neither of which the freeze context can @@ -1530,26 +1578,33 @@ fn per_element_pin_mapped_axis_verdict_enumeration() { Some("pop[region\u{B7}boston, age\u{B7}old]"), ), ]; - let declined: [PinIndexCell<'_>; 1] = [( - "an element-mapped (non-positional) pair", + // GH #997: an element-mapped pair pins POSITIONALLY, so its expected + // spelling is the same as the positional rows'. The map below is the reverse + // permutation (ma -> nyc), so `region\u{B7}nyc` is what a map-following pin + // would produce and its absence is asserted separately below. + let element_mapped: [PinIndexCell<'_>; 1] = [( + "an element-mapped pair (the ordinal wins)", "pop[State, old]", - None, - None, + Some("pop[region\u{B7}boston, age\u{B7}old]"), + Some("pop[region\u{B7}boston, age\u{B7}old]"), )]; - // Declaration direction must not matter: `mapped_element_correspondence` - // honors a mapping declared on either dimension (GH #757), so a reverse-declared + // Declaration direction must not matter: `positional_correspondence` honors a + // mapping declared on either dimension (GH #757), so a reverse-declared // positional pair pins identically. A forward-only gate here would silently drop // half the mapped models. for declare_on_state in [true, false] { let fx = PinFixture::mapped(declare_on_state, vec![]); assert_pin_index_verdicts(&fx, "pop[State, young]", &positional); - // An EXPLICIT element map is declined even though it names a correspondence: - // the executed A2A lowering resolves mapped references POSITIONALLY and - // ignores the map (GH #756), so pinning the row the map names would spell a - // read the simulation never performs -- a compilable, confidently wrong - // score, which is the one outcome worse than none. + // An EXPLICIT element map does not change the verdict, because this + // spelling never reads the map: `pop[State, old]` names the dimension the + // equation ITERATES, and `mapped_reference_semantics_tests`' `Permuted` + // row measures that spelling folding to an ordinal against the VM. This + // block asserted a loud decline until GH #997, on the (correct, but + // spelling-specific) reasoning that FOLLOWING the map would spell a read + // the simulation never performs -- the fix is to describe the ordinal, + // not to describe nothing. let fx = PinFixture::mapped( declare_on_state, vec![ @@ -1557,6 +1612,38 @@ fn per_element_pin_mapped_axis_verdict_enumeration() { ("ma".to_string(), "nyc".to_string()), ], ); - assert_pin_index_verdicts(&fx, "pop[State, young]", &declined); + assert_pin_index_verdicts(&fx, "pop[State, young]", &element_mapped); + + // GH #997, the OTHER spelling on the SAME fixture: an index naming the + // source's own `Region` -- a dimension this target does NOT iterate -- + // pins through the ELEMENT MAP, so it reads `nyc` where the + // iterated-dimension spelling above reads `boston`. One fixture, two + // indices, two rules; that is the whole of #997 in one assertion. + assert_pin_index_verdicts( + &fx, + "pop[State, young]", + &[( + "the source's own dimension name (the map-following spelling)", + "pop[Region, old]", + Some("pop[region\u{B7}nyc, age\u{B7}old]"), + Some("pop[region\u{B7}nyc, age\u{B7}old]"), + )], + ); + + // The discriminator: the declared map sends `ma` to `nyc`, so a + // map-following pin would spell `region\u{B7}nyc`. It must not appear. + let (ast, deps, occurrences) = fx.parse( + "pop[State, young] + LOOKUP(pop[State, old], input)", + &["input"], + ); + let slot_occurrences = SlotOccurrences::new(&occurrences); + let text = fx + .generate(&ast, &deps, &slot_occurrences.for_slot(0)) + .expect("the element-mapped pair pins positionally"); + assert!( + !text.contains("region\u{B7}nyc"), + "the element map's own element must not be pinned for an \ + iterated-dimension index; got: {text}" + ); } } diff --git a/src/simlin-engine/src/ltm_augment_post_transform.rs b/src/simlin-engine/src/ltm_augment_post_transform.rs index 3a06cfc69..c69a8031d 100644 --- a/src/simlin-engine/src/ltm_augment_post_transform.rs +++ b/src/simlin-engine/src/ltm_augment_post_transform.rs @@ -100,16 +100,22 @@ pub(super) fn qualify_axis_element(elem: &str, dim: &crate::dimensions::Dimensio } /// The source row a per-axis access vector reads for one full target -/// element: project the target element onto the `Iterated` axes -/// (slot-remapped through `mapped_element_correspondence` for a -/// positionally-mapped pair -- the correspondence is indexed by TARGET -/// element position and yields the source element the executed simulation -/// reads) and fill `Pinned` axes with their literals. One bare element -/// name per axis, in source-axis order. `None` when an `Iterated` dim is -/// missing from the target projection or the mapped remap is unusable (a +/// element: project the target element onto the `Iterated` / `MappedRead` +/// axes and fill `Pinned` axes with their literals. One bare element name +/// per axis, in source-axis order. `None` when a projected axis's dim is +/// missing from the target projection or its correspondence is unusable (a /// mid-edit inconsistency; callers degrade conservatively) -- and for any /// `Reduced` axis, which the `PerElement` invariant excludes. /// +/// The two projected axes take DIFFERENT correspondences, which is the whole +/// of GH #997 at the per-axis level: an `Iterated` index spells a dimension +/// the equation iterates and is folded to an ordinal, so it reads the +/// POSITIONAL diagonal; a `MappedRead` index spells a non-active dimension, +/// survives to `IndexOp::ActiveDimRef`, and is resolved name-first then +/// through the declared element map. Both correspondences are indexed by +/// TARGET element position and yield the source element the executed +/// simulation reads for it. +/// /// This is the SINGLE row derivation for the `PerElement` family's /// emission: the link-score NAME's row (computed by /// `emit_per_element_link_scores`) and the equation's live-reference row @@ -130,13 +136,21 @@ pub(crate) fn per_element_row_for_target( if dim == source_dim { Some(elem.clone()) } else { - let corr = dim_ctx.mapped_element_correspondence( + let corr = dim_ctx.positional_correspondence( &CanonicalDimensionName::from_raw(dim), &CanonicalDimensionName::from_raw(source_dim), )?; corr.get(*idx).map(|e| e.as_str().to_string()) } } + AxisRead::MappedRead { dim, source_dim } => { + let (_, idx) = target_elem_by_dim.get(dim)?; + let corr = dim_ctx.executed_read_correspondence( + &CanonicalDimensionName::from_raw(dim), + &CanonicalDimensionName::from_raw(source_dim), + )?; + corr.get(*idx).map(|e| e.as_str().to_string()) + } AxisRead::Reduced { .. } => None, }) .collect() @@ -176,16 +190,15 @@ pub(crate) fn per_element_row_for_target( /// /// What remains here is the per-axis ELEMENT translation, which is LTM's own and /// deliberately narrower than the compiler's: an axis allocated to a target axis -/// of a different name resolves through -/// `DimensionsContext::mapped_element_correspondence`, which declines explicit -/// element maps (GH #756 -- the executed A2A lowering resolves positionally and -/// ignores them), so this accepts exactly the mapped pairs the occurrence-driven -/// pin and `ltm_agg::classify_axis_access` accept. +/// of a DIFFERENT name resolves through the spelling-keyed correspondence +/// `spelling` selects (GH #997), so this accepts exactly the mapped pairs +/// `ltm_agg::classify_axis_access` accepts for that same spelling. fn dep_axis_elements( dep_dims: &[crate::dimensions::Dimension], target_dims: &[crate::dimensions::Dimension], target_elements: &[String], dim_ctx: &crate::dimensions::DimensionsContext, + spelling: DepSpelling, ) -> Vec> { use crate::common::CanonicalElementName; if target_dims.len() != target_elements.len() { @@ -202,9 +215,8 @@ fn dep_axis_elements( if dep_dim.canonical_name() == target_dim.canonical_name() { return Some(target_elem.clone()); } - // A different-named axis: the element is whatever the executed A2A - // lowering reads there, i.e. the positional correspondence. - let corr = dim_ctx.mapped_element_correspondence( + let corr = spelling.correspondence( + dim_ctx, target_dim.canonical_name(), dep_dim.canonical_name(), )?; @@ -214,36 +226,87 @@ fn dep_axis_elements( .collect() } -/// The element a variable declared over `dep_dims` reads at ONE target element, -/// one bare element name per declared dimension -- [`dep_axis_elements`] with -/// every axis resolved. `None` when some declared dimension does not project, -/// because a BARE reference has to be spelled at the dep's full arity or not -/// at all. +/// Which resolution rule a dep reference gets, because the two forms +/// [`dep_element_pins`] serves are spelled differently and execution resolves +/// them differently (GH #997). +#[derive(Clone, Copy, PartialEq, Eq)] +pub(crate) enum DepSpelling { + /// A BARE reference (`dep`) in an equation body. `compiler::context`'s + /// `lower_pass0` rewrites it into the iterated-dimension spelling, whose + /// index `ast::expr3` Pass 1 folds to an ordinal -- so it reads + /// POSITIONALLY and never consults a declared element map. + Bare, + /// An already-subscripted reference whose index names one of the dep's OWN + /// declared dimensions (`dep[Region]` under a `State`-iterating target). + /// That index survives to `IndexOp::ActiveDimRef` and + /// `compiler::subscript`'s `build_view_from_ops` resolves it name-first, + /// then through the declared element map. + OwnDimSubscript, +} + +impl DepSpelling { + fn correspondence( + self, + dim_ctx: &crate::dimensions::DimensionsContext, + target_dim: &crate::common::CanonicalDimensionName, + dep_dim: &crate::common::CanonicalDimensionName, + ) -> Option> { + match self { + DepSpelling::Bare => dim_ctx.positional_correspondence(target_dim, dep_dim), + DepSpelling::OwnDimSubscript => { + dim_ctx.executed_read_correspondence(target_dim, dep_dim) + } + } + } +} + +/// The element a variable declared over `dep_dims` reads at ONE target element +/// when referenced BARE, one bare element name per declared dimension -- +/// [`dep_axis_elements`] with every axis resolved. `None` when some declared +/// dimension does not project, because a BARE reference has to be spelled at +/// the dep's full arity or not at all. pub(crate) fn dep_row_for_target( dep_dims: &[crate::dimensions::Dimension], target_dims: &[crate::dimensions::Dimension], target_elements: &[String], dim_ctx: &crate::dimensions::DimensionsContext, ) -> Option> { - dep_axis_elements(dep_dims, target_dims, target_elements, dim_ctx) - .into_iter() - .collect() + dep_axis_elements( + dep_dims, + target_dims, + target_elements, + dim_ctx, + DepSpelling::Bare, + ) + .into_iter() + .collect() } /// The element-pin table for ONE target element: each dep in `pinnable` mapped /// to the elements IT reads there ([`DepElementPin`]), qualified in its own /// dimensions' space so a frozen read compiles to a direct LoadPrev. /// -/// A dep no axis of which projects is ABSENT from the table entirely (nothing -/// to rewrite). A dep only SOME of whose axes project is present but not -/// [`complete`](DepElementPin::complete): its dimension-name indices are still -/// substituted -- that is the GH #654 helper-aux fix, and it only needs the -/// axes the reference actually spells as dimension names -- while a BARE -/// reference to it is left alone, since no correct full-arity subscript exists. -/// Leaving it bare is the loud direction: a bare multi-slot reference in a -/// scalar fragment fails to compile and surfaces an `Assembly` warning, where -/// the pre-GH #974 full-target-tuple pin silently mis-read the dep whenever the -/// arity happened to match. +/// A dep no axis of which projects under EITHER spelling is ABSENT from the +/// table entirely (nothing to rewrite). A dep only SOME of whose axes project +/// is present but has no [`bare_row`](DepElementPin::bare_row): its +/// dimension-name indices are still substituted -- that is the GH #654 +/// helper-aux fix, and it only needs the axes the reference actually spells as +/// dimension names -- while a BARE reference to it is left alone, since no +/// correct full-arity subscript exists. Leaving it bare is the loud direction: +/// a bare multi-slot reference in a scalar fragment fails to compile and +/// surfaces an `Assembly` warning, where the pre-GH #974 full-target-tuple pin +/// silently mis-read the dep whenever the arity happened to match. +/// +/// The two rows come from two DIFFERENT correspondences, and that is the whole +/// content of GH #997 in miniature: `axes` serves an already-subscripted +/// `dep[]`, which execution resolves name-first then through +/// the declared element map, while `bare_row` serves a bare `dep`, which pass 0 +/// rewrites into the iterated spelling and execution resolves by ordinal. They +/// coincide for every positional mapping between dimensions with disjoint +/// element names -- the only pairs that projected at all before GH #997 -- and +/// differ under an explicit element map or where the two dimensions share +/// element names. One table answering both spellings with one rule is what made +/// C-LEARN's element-mapped deps unpinnable. /// /// `pinnable` carries each dep's declared `Dimension`s, resolved ONCE per /// target equation by the caller; only the projection is per element. @@ -256,20 +319,39 @@ pub(crate) fn dep_element_pins( pinnable .iter() .filter_map(|(ident, dep_dims)| { - let elems = dep_axis_elements(dep_dims, target_dims, target_elements, dim_ctx); - let complete = elems.iter().all(Option::is_some); - let axes: Vec<(String, String)> = elems + let qualify = |elems: &[Option]| -> Vec<(String, String)> { + elems + .iter() + .zip(dep_dims) + .filter_map(|(elem, dim)| { + elem.as_ref() + .map(|e| (dim.name().to_string(), qualify_axis_element(e, dim))) + }) + .collect() + }; + let subscripted = dep_axis_elements( + dep_dims, + target_dims, + target_elements, + dim_ctx, + DepSpelling::OwnDimSubscript, + ); + let bare = dep_axis_elements( + dep_dims, + target_dims, + target_elements, + dim_ctx, + DepSpelling::Bare, + ); + let axes = qualify(&subscripted); + let bare_row = bare .iter() - .zip(dep_dims) - .filter_map(|(elem, dim)| { - elem.as_ref() - .map(|e| (dim.name().to_string(), qualify_axis_element(e, dim))) - }) - .collect(); - if axes.is_empty() { + .all(Option::is_some) + .then(|| qualify(&bare).into_iter().map(|(_, elem)| elem).collect()); + if axes.is_empty() && bare_row.is_none() { return None; } - Some((ident.clone(), DepElementPin { axes, complete })) + Some((ident.clone(), DepElementPin { axes, bare_row })) }) .collect() } @@ -305,6 +387,10 @@ fn axes_as_read_slice(occ: &OccurrenceSite, arity: usize) -> Option Some(AxisRead::MappedRead { + dim: dim.clone(), + source_dim: source_dim.clone(), + }), OccurrenceAxis::Reduced { .. } | OccurrenceAxis::MismatchedIterated { .. } | OccurrenceAxis::Dynamic => None, @@ -384,28 +470,39 @@ pub(super) fn pin_source_subscript_indices( } return indices; } - // Partially describable: substitute only the axes the IR classified - // `Iterated`, and hand every other index to the wrap's own index pass. + // Partially describable: substitute only the axes the IR classified as + // PROJECTED -- `Iterated` or (GH #997) `MappedRead`, each carrying its own + // resolution rule into the shared row derivation -- and hand every other + // index to the wrap's own index pass. let axes = node_occ.map(|o| o.axes.as_slice()).unwrap_or(&[]); indices .into_iter() .enumerate() .map(|(i, idx)| { - let substituted = match (axes.get(i), ctx.from_dims.get(i)) { - (Some(OccurrenceAxis::Iterated { dim, source_dim }), Some(from_dim)) => { - let ax = crate::ltm_agg::AxisRead::Iterated { + let projected_axis = match axes.get(i) { + Some(OccurrenceAxis::Iterated { dim, source_dim }) => { + Some(crate::ltm_agg::AxisRead::Iterated { + dim: dim.clone(), + source_dim: source_dim.clone(), + }) + } + Some(OccurrenceAxis::MappedRead { dim, source_dim }) => { + Some(crate::ltm_agg::AxisRead::MappedRead { dim: dim.clone(), source_dim: source_dim.clone(), - }; - per_element_row_for_target( - std::slice::from_ref(&ax), - ctx.target_elem_by_dim, - ctx.dim_ctx, - ) - .map(|row| qualify_axis_element(&row[0], from_dim)) + }) } _ => None, }; + let substituted = match (projected_axis, ctx.from_dims.get(i)) { + (Some(ax), Some(from_dim)) => per_element_row_for_target( + std::slice::from_ref(&ax), + ctx.target_elem_by_dim, + ctx.dim_ctx, + ) + .map(|row| qualify_axis_element(&row[0], from_dim)), + _ => None, + }; match substituted { Some(part) => IndexExpr0::Expr(Expr0::Var( RawIdent::new_from_str(&part), @@ -460,16 +557,45 @@ enum IndexVerdict { /// enclosing freeze on the descents the wrap does not enter -- a pre-existing /// `PREVIOUS`/`INIT`, a whole-frozen reducer (GH #984). Keep, - /// No pin can spell it, because the SHARED row derivation - /// ([`per_element_row_for_target`]) cannot resolve the axis: a dimension the - /// target does not iterate, an iterated dimension with no usable positional - /// correspondence to this source axis (unmapped, element-mapped, or a - /// transposition), an index no axis owns. Left alone it keeps a + /// No pin can spell it, because NEITHER shared row derivation + /// ([`per_element_row_for_target`] on an `Iterated` or a `MappedRead` axis) + /// can resolve it: a dimension the target does not iterate and no iterated + /// dimension is mapped to, an iterated dimension with no usable positional + /// correspondence to this source axis (unmapped, or a transposition), an + /// ambiguous mapped pairing, an index no axis owns. Left alone it keeps a /// DIMENSION-name subscript, which cannot resolve in a scalar fragment, so /// this one is LOUD -- a compilability verdict. Unspellable, } +/// The [`crate::ltm_agg::AxisRead::MappedRead`] axis for a subscript index that +/// names the non-active dimension `index_dim` against source axis `axis_dim`, or +/// `None` when execution pairs it with no single iterated dimension of this +/// target (GH #997). +/// +/// The pairing and its usability gate are `DimensionsContext`'s, so this asks +/// the same two questions `ltm_agg::classify_axis_access`'s `Unresolved` arm +/// asks and cannot accept a spelling the classifier rejects. It builds an +/// `AxisRead` only to hand to the shared row derivation; it decides no shape. +fn mapped_read_axis( + index_dim: &str, + axis_dim: &crate::dimensions::Dimension, + ctx: &PerElementRefCtx<'_>, +) -> Option { + use crate::common::CanonicalDimensionName; + let target_iterated: Vec = ctx.target_elem_by_dim.keys().cloned().collect(); + let index_canon = CanonicalDimensionName::from_raw(index_dim); + let partner = ctx + .dim_ctx + .mapped_read_partner_dim(&index_canon, &target_iterated)?; + ctx.dim_ctx + .executed_read_correspondence(&partner, axis_dim.canonical_name())?; + Some(crate::ltm_agg::AxisRead::MappedRead { + dim: partner.as_str().to_string(), + source_dim: axis_dim.name().to_string(), + }) +} + /// Row-pin a source subscript the occurrence IR deliberately records NOTHING /// for, by NAME alone. Returns the rewritten indices plus whether the rule /// DISCHARGED the subscript. @@ -509,18 +635,24 @@ enum IndexVerdict { /// dim)` pair to [`per_element_row_for_target`], the SAME single row derivation /// the occurrence-driven pin uses. That is what makes the identity axis /// (`pop[Region, ..]` over a `Region` axis, the structural substitution -/// [`pin_bare_source_ref`] performs for a bare `Var`) and a positionally-MAPPED -/// axis (`effect[State, ..]` over a `Region` axis with a `State`/`Region` -/// mapping, either declaration direction -- GH #527 / #757) ONE arm rather than -/// two: the derivation resolves both through -/// `DimensionsContext::mapped_element_correspondence`, so this rule accepts -/// EXACTLY the mapped pairs `ltm_agg::classify_axis_access` accepts (that -/// classifier's `Iterated` arm gates on `iterated_axis_slot_elements`, the -/// preimage inversion of the same correspondence). An axis the derivation -/// declines -- no mapping, an explicit element map (GH #756: execution resolves -/// positionally and ignores it), a transposition, a dimension this target does -/// not iterate -- is `IndexVerdict::Unspellable`, and it is unspellable because -/// the SHARED derivation says so, not because the name differs; +/// [`pin_bare_source_ref`] performs for a bare `Var`) and a MAPPED axis +/// (`effect[State, ..]` over a `Region` axis with a `State`/`Region` mapping, +/// either declaration direction -- GH #527 / #757) ONE arm rather than two: the +/// derivation resolves both through `positional_correspondence`, which is the +/// rule execution applies to an iterated-dimension index; +/// - otherwise, an index spelling a NON-ITERATED dimension that execution pairs +/// with one of the target's iterated dims (`effect[Aggregated Regions, ..]` +/// under a `COP`-iterating target -- C-LEARN's shape, GH #997) is replaced the +/// same way, through an [`crate::ltm_agg::AxisRead::MappedRead`] so the row +/// derivation applies the name-first-then-element-map rule THIS spelling gets. +/// The pairing and its gate are `DimensionsContext::mapped_read_partner_dim` / +/// `executed_read_correspondence`, the same two questions +/// `ltm_agg::classify_axis_access` asks, so this rule accepts exactly the +/// spellings the classifier accepts; +/// - an axis BOTH derivations decline -- no mapping either way, a transposition, +/// an ambiguous pairing, a dimension this target does not iterate -- is +/// `IndexVerdict::Unspellable`, and it is unspellable because the SHARED +/// derivations say so, not because the name differs; /// - anything that is not a bare identifier is kept verbatim, because this rule has /// nothing to say about it. A numeric literal, arithmetic over literals, and an /// `@N` POSITION index (which `compiler::context`'s subscript lowering resolves @@ -632,15 +764,34 @@ fn pin_dimension_name_indices( } } crate::dimensions::AxisIndexName::Unresolved => { - if ctx.dim_ctx.is_dimension_name(&name) { - // A dimension name no target coordinate projects onto - // this axis. - IndexVerdict::Unspellable - } else { + if !ctx.dim_ctx.is_dimension_name(&name) { // Neither an element of this axis nor a dimension: a // variable read selecting the element at runtime, // already frozen by the wrap. IndexVerdict::Keep + } else if let Some(row) = + mapped_read_axis(&name, dim, ctx).and_then(|axis| { + per_element_row_for_target( + std::slice::from_ref(&axis), + ctx.target_elem_by_dim, + ctx.dim_ctx, + ) + }) + { + // GH #997: the index names a NON-ITERATED dimension + // -- typically the source's own -- that execution + // pairs with one of the target's iterated dims and + // resolves through the declared element map. The + // element is again the SHARED row derivation's + // answer, reached with an `AxisRead::MappedRead` + // rather than an `Iterated` so it takes the + // map-following rule this spelling gets. This is + // where C-LEARN's `x[Aggregated Regions]` deps land. + IndexVerdict::Pinned(qualify_axis_element(&row[0], dim)) + } else { + // A dimension name no target coordinate projects onto + // this axis, on either rule. + IndexVerdict::Unspellable } } } diff --git a/src/simlin-engine/src/ltm_augment_tests.rs b/src/simlin-engine/src/ltm_augment_tests.rs index dca38d6f4..7612164b0 100644 --- a/src/simlin-engine/src/ltm_augment_tests.rs +++ b/src/simlin-engine/src/ltm_augment_tests.rs @@ -79,7 +79,11 @@ fn pin_table_with_completeness( .iter() .map(|(dim, elem)| ((*dim).to_string(), (*elem).to_string())) .collect(), - complete, + // The fixture's two spellings read the same row (its axes + // are same-named, so no correspondence is consulted); + // `complete` selects whether a BARE reference is spellable. + bare_row: complete + .then(|| axes.iter().map(|(_, elem)| (*elem).to_string()).collect()), }, ) }) @@ -1899,8 +1903,11 @@ fn test_partial_equation_share_bare_shape() { // is wrapped in PREVIOUS() *as a whole*: `PREVIOUS(sum(population[*]))`, // which is PREVIOUS of the scalar total and evaluates fine. The // earlier form `sum(PREVIOUS(population[*]))` was the GH #517 bug -- - // identically `0.0` at every step under an active A2A dimension - // because codegen has no LoadPrev-of-array-view path. + // identically `0.0` at every step under an active A2A dimension, because + // codegen had no array-`PREVIOUS` path and the fragment stubbed. GH #995 + // phase C3 gave it one, so that form now compiles; the wrap is kept as the + // GH #517 semantics (freeze the reducer's RESULT), which is what this row + // pins. let equation = "population / SUM(population[*])"; let deps = deps_set(&["population"]); let source = Ident::::new("population"); @@ -5166,10 +5173,12 @@ fn wrap_missing_live_source_occurrence_is_loud_not_silent_freeze() { // // The chooser builds the standard changed-first guard form, but when the // changed-first partial would embed `PREVIOUS` of an array slice (a -// wildcard/star-range-subscripted reference -- no LoadPrev-of-array-view -// codegen path exists, so the equation can only silently stub or hard-fail), +// wildcard/star-range-subscripted reference this layer will not spell inline), // it falls back to the changed-last attribution (only the live source -// frozen), and errors loudly when both conventions are unfreezable. +// frozen), and errors loudly when both conventions are unfreezable. The +// original reason -- no codegen path for an array `PREVIOUS` -- is stale since +// GH #995 phase C3; the routing is unchanged, and `ltm_augment_array_freeze`'s +// module doc carries what the materialized helper still buys. /// The GH #743 shape: live `frac` (Bare, iterated-dim feeder) inside a /// reducer whose co-source is a wildcard slice. Changed-first would freeze diff --git a/src/simlin-engine/src/ltm_augment_wrap_test_support.rs b/src/simlin-engine/src/ltm_augment_wrap_test_support.rs index 83a9855d9..74451ae9e 100644 --- a/src/simlin-engine/src/ltm_augment_wrap_test_support.rs +++ b/src/simlin-engine/src/ltm_augment_wrap_test_support.rs @@ -152,6 +152,32 @@ pub(crate) fn live_source_occurrence_axis( } return OccurrenceAxis::MismatchedIterated { dim: d }; } + // GH #997: a NON-iterated dimension name execution pairs with one of + // the target's iterated dims, resolved name-first then through the + // element map. Mirrors `classify_axis_access`'s `Unresolved` arm -- + // element-of-this-axis first (the literal pass below), then the + // partner pairing and the executed correspondence. + if let Some(dim_ctx) = dim_ctx + && resolve_literal_element_index(idx, i, source_dim_elements).is_none() + && i < ic.source_dim_names.len() + { + let index_dim = crate::common::CanonicalDimensionName::from_raw(&d); + let source_dim = + crate::common::CanonicalDimensionName::from_raw(ic.source_dim_names[i].as_str()); + if let Some(partner) = dim_ctx + .mapped_read_partner_dim(&index_dim, ic.target_iterated_dims) + .filter(|p| { + dim_ctx + .executed_read_correspondence(p, &source_dim) + .is_some() + }) + { + return OccurrenceAxis::MappedRead { + dim: partner.as_str().to_string(), + source_dim: ic.source_dim_names[i].clone(), + }; + } + } } match resolve_literal_element_index(idx, i, source_dim_elements) { Some(e) => OccurrenceAxis::Pinned(e), @@ -391,9 +417,9 @@ pub(crate) fn is_live_source_iterated_dim_subscript( /// source's `i`-th axis -- by name, or through a usable positional-mapping /// remap? The mapped arm consults the SAME /// [`crate::ltm_agg::iterated_axis_slot_elements`] / -/// `mapped_element_correspondence` gate the Expr2 classifier +/// `positional_correspondence` gate the Expr2 classifier /// (`ltm_agg::classify_axis_access`) uses -- BOTH declaration directions -/// (GH #757), positional mappings only (GH #756) -- so the partial +/// (GH #757) -- so the partial /// builder's live-shape match and the reference-site IR agree by /// construction. (No mapping context ⇒ no mapped recognition; the by-name /// check still applies.) @@ -468,6 +494,31 @@ pub(crate) fn classify_expr0_per_element_axes( }); continue; } + // GH #997: an index naming a NON-iterated dimension that execution + // pairs with one of the target's iterated dims. Mirrors + // `classify_axis_access`'s `Unresolved` arm gate for gate -- + // element-of-this-axis first (below), then the partner pairing and + // the executed correspondence. + if let Some(dim_ctx) = dim_ctx + && !source_dim_elements[i].iter().any(|e| e == &d) + { + let index_dim = crate::common::CanonicalDimensionName::from_raw(&d); + let source_dim = crate::common::CanonicalDimensionName::from_raw( + ctx.source_dim_names[i].as_str(), + ); + if let Some(partner) = + dim_ctx.mapped_read_partner_dim(&index_dim, ctx.target_iterated_dims) + && dim_ctx + .executed_read_correspondence(&partner, &source_dim) + .is_some() + { + axes.push(AxisRead::MappedRead { + dim: partner.as_str().to_string(), + source_dim: ctx.source_dim_names[i].clone(), + }); + continue; + } + } } // Position-strict literal resolution: the Expr2 classifier resolves // each index against ITS axis only, so the any-dimension fallback @@ -528,7 +579,29 @@ pub(crate) fn other_dep_occurrence_axes( }; let d = canonicalize(name.as_str()).into_owned(); if !ctx.target_iterated_dims.iter().any(|t| t == &d) { - return OccurrenceAxis::Dynamic; + // GH #997: a non-iterated dimension name execution pairs with + // one of the target's iterated dims is `MappedRead`, not a + // dynamic index. Mirrors `classify_axis_access`'s arm; anything + // it declines stays `Dynamic`. + return match (dep_dims.and_then(|dd| dd.get(i)), dim_ctx) { + (Some(dep_dim), Some(dim_ctx)) if dep_dim.canonical_element(&d).is_none() => { + let index_dim = crate::common::CanonicalDimensionName::from_raw(&d); + match dim_ctx + .mapped_read_partner_dim(&index_dim, ctx.target_iterated_dims) + .filter(|p| { + dim_ctx + .executed_read_correspondence(p, dep_dim.canonical_name()) + .is_some() + }) { + Some(partner) => OccurrenceAxis::MappedRead { + dim: partner.as_str().to_string(), + source_dim: canonicalize(dep_dim.name()).into_owned(), + }, + None => OccurrenceAxis::Dynamic, + } + } + _ => OccurrenceAxis::Dynamic, + }; } match dep_dims.and_then(|dd| dd.get(i)) { Some(dep_dim) if other_dep_axis_lines_up(&d, dep_dim, dim_ctx) => { @@ -552,9 +625,9 @@ pub(crate) fn other_dep_occurrence_axes( /// positional-mapping remap? The dep-side sibling of /// [`expr0_iterated_axis_lines_up`], consulting the SAME /// [`crate::ltm_agg::iterated_axis_slot_elements`] / -/// `mapped_element_correspondence` gate (both declaration directions, -/// positional mappings only) so the live-source and other-dep recognizers -/// can never disagree about which mapped pairs are usable. +/// `positional_correspondence` gate (both declaration directions) so the +/// live-source and other-dep recognizers can never disagree about which +/// mapped pairs are usable. #[cfg(test)] pub(crate) fn other_dep_axis_lines_up( d: &str, @@ -713,11 +786,17 @@ pub(crate) fn classify_expr0_subscript_shape( && let Some(axes) = classify_expr0_per_element_axes(indices, source_dim_elements, ctx, dim_ctx) { - let n_iterated = axes + use crate::ltm_agg::AxisRead; + let n_projected = axes .iter() - .filter(|a| matches!(a, crate::ltm_agg::AxisRead::Iterated { .. })) + .filter(|a| matches!(a, AxisRead::Iterated { .. } | AxisRead::MappedRead { .. })) .count(); - if n_iterated > 0 && n_iterated < axes.len() { + let all_iterated = axes.iter().all(|a| matches!(a, AxisRead::Iterated { .. })); + // Mirrors `classify_iterated_dim_shape`: all-`Iterated` is the `Bare` + // case handled above, all-`Pinned` falls through to the literal pass, + // and everything else -- including an all-`MappedRead` subscript + // (GH #997) -- is `PerElement`. + if n_projected > 0 && !all_iterated { return RefShape::PerElement { axes }; } } diff --git a/src/simlin-engine/src/ltm_classifier_agreement_tests.rs b/src/simlin-engine/src/ltm_classifier_agreement_tests.rs index ab30c92cd..018dcf2e2 100644 --- a/src/simlin-engine/src/ltm_classifier_agreement_tests.rs +++ b/src/simlin-engine/src/ltm_classifier_agreement_tests.rs @@ -1302,8 +1302,8 @@ fn agree_mapped_dim_forward_declaration_is_bare() { fn agree_mapped_dim_reverse_declaration_is_bare() { // GH #757 reverse direction: the mapping is declared `State -> Region`, // but the target iterates `Region` and reads a `State`-dimensioned source. - // `mapped_element_correspondence` accepts both declaration directions, so - // both families classify `speed[Region]` as Bare. + // `positional_correspondence` accepts both declaration directions, so both + // families classify `speed[Region]` as Bare. let tp = TestProject::new("main") .with_sim_time(0.0, 3.0, 1.0) .named_dimension("Region", &["r1", "r2"]) @@ -1314,10 +1314,12 @@ fn agree_mapped_dim_reverse_declaration_is_bare() { } #[test] -fn agree_element_mapped_dim_declines_to_dynamic() { - // GH #756 positional-only gate: an EXPLICIT element map (not positional) is - // declined by `mapped_element_correspondence`, so `pop[State]` keeps the - // conservative DynamicIndex on both families (not Bare). +fn agree_element_mapped_iterated_dim_is_bare() { + // An EXPLICIT element map on the ITERATED spelling: `pop[State]` names the + // dimension the equation iterates, which execution folds to an ordinal + // (GH #997), so `positional_correspondence` answers and both families + // classify Bare. This asserted DynamicIndex until GH #997, when one + // correspondence served both spellings and answered neither. let tp = TestProject::new("main") .with_sim_time(0.0, 3.0, 1.0) .named_dimension("Region", &["r1", "r2"]) @@ -1332,6 +1334,102 @@ fn agree_element_mapped_dim_declines_to_dynamic() { assert_classifier_families_agree(&tp); } +#[test] +fn agree_element_mapped_source_own_dim_is_per_element() { + // GH #997's class-D shape, the OTHER spelling of the fixture above: + // `pop[Region]` names the SOURCE's own dimension inside a `State`-iterating + // equation, which execution resolves name-first then through the element + // map. Both families must classify it `PerElement` with a `MappedRead` + // axis -- the Expr0 mirror gains the same arm, so the agreement gate is + // what keeps the test-support classifier from drifting. + let tp = TestProject::new("main") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("Region", &["r1", "r2"]) + .named_dimension_with_element_mapping( + "State", + &["s1", "s2"], + "Region", + &[("s1", "r2"), ("s2", "r1")], + ) + .array_aux("pop[Region]", "100") + .array_aux("mapped[State]", "pop[Region] * 2"); + let compared = assert_classifier_families_agree(&tp); + // The name promises a SHAPE, so pin it: agreement alone would be satisfied + // by both families calling it `DynamicIndex`, which is what they did before + // GH #997. + pin_edge( + &compared, + "pop", + "mapped", + &[RefShape::PerElement { + axes: vec![AxisRead::MappedRead { + dim: "state".to_string(), + source_dim: "region".to_string(), + }], + }], + ); +} + +#[test] +fn agree_many_to_one_mapped_source_own_dim_is_per_element() { + // The cardinality `positional_correspondence` cannot describe at all + // (three target elements, two source ones), so this row is reachable only + // through the executed rule -- C-LEARN's shape. + let tp = TestProject::new("main") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("Region", &["r1", "r2"]) + .named_dimension_with_element_mapping( + "State", + &["s1", "s2", "s3"], + "Region", + &[("s1", "r1"), ("s2", "r1"), ("s3", "r2")], + ) + .array_aux("pop[Region]", "100") + .array_aux("mapped[State]", "pop[Region] * 2"); + let compared = assert_classifier_families_agree(&tp); + pin_edge( + &compared, + "pop", + "mapped", + &[RefShape::PerElement { + axes: vec![AxisRead::MappedRead { + dim: "state".to_string(), + source_dim: "region".to_string(), + }], + }], + ); +} + +#[test] +fn agree_shared_element_names_mapped_source_own_dim_is_per_element() { + // Both dimensions declare the same element names in a different order, and + // the map is a third permutation: the executed rule stops at NAME identity. + // The families must agree on the axis, not merely on the shape. + let tp = TestProject::new("main") + .with_sim_time(0.0, 3.0, 1.0) + .named_dimension("Region", &["e2", "e1"]) + .named_dimension_with_element_mapping( + "State", + &["e1", "e2"], + "Region", + &[("e1", "e2"), ("e2", "e1")], + ) + .array_aux("pop[Region]", "100") + .array_aux("mapped[State]", "pop[Region] * 2"); + let compared = assert_classifier_families_agree(&tp); + pin_edge( + &compared, + "pop", + "mapped", + &[RefShape::PerElement { + axes: vec![AxisRead::MappedRead { + dim: "state".to_string(), + source_dim: "region".to_string(), + }], + }], + ); +} + #[test] fn agree_dynamic_index_expression() { // A non-literal dynamic index (`pop[idx]` with a scalar `idx`, and an diff --git a/src/simlin-engine/src/ltm_finding.rs b/src/simlin-engine/src/ltm_finding.rs index 0f39cefd1..fab1766da 100644 --- a/src/simlin-engine/src/ltm_finding.rs +++ b/src/simlin-engine/src/ltm_finding.rs @@ -768,13 +768,17 @@ fn parse_link_offsets( /// (`scale[a]`, `boost[r,a]`, `x[s]`) that named no real element node, so /// every loop through such a feeder dangled and was silently undiscoverable. /// -/// The MAPPED leg is covered only for POSITIONAL mappings: `expand_same_element` -/// declines element-mapped pairs (the GH #756 positional-only gate). Such a -/// pair never reaches here anyway -- `link_score_dimensions` declines to -/// retarget it to the target's dims (no Bare A2A score is emitted; the edge -/// takes the GH #758 loud skip instead), so there is no dimensioned score for -/// `parse_link_offsets` to expand. If that upstream gate is ever relaxed, the -/// projection here inherits the same positional-only correspondence in lockstep. +/// The MAPPED leg covers every pair whose two reference spellings AGREE, which +/// since GH #997 includes an explicit element map at unequal cardinality +/// (C-LEARN's many-to-one). It cannot cover a pair whose spellings DISAGREE, +/// and does not have to: `expand_same_element` emits the UNION of both +/// diagonals there, which would put two from-nodes on the one slot this +/// function assigns per target element -- so `link_score_dimensions` denies +/// such a pair the arrayed retarget (`db::analysis::mapped_pair_projects_uniquely`) +/// and it takes the GH #758 loud skip instead, leaving no dimensioned score for +/// `parse_link_offsets` to expand. That is the whole reason the gate is +/// STRICTER than the element graph's rule; the lockstep with +/// `expand_same_element` is what makes the from-node names match either way. fn expand_a2a_link_offsets( from_var: &str, to_var: &str, diff --git a/src/simlin-engine/src/mapped_reference_semantics_tests.rs b/src/simlin-engine/src/mapped_reference_semantics_tests.rs new file mode 100644 index 000000000..9497bbf41 --- /dev/null +++ b/src/simlin-engine/src/mapped_reference_semantics_tests.rs @@ -0,0 +1,1420 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! What the EXECUTED simulation reads for a cross-dimension arrayed +//! reference, pinned cell by cell against the VM (GH #997, and the +//! execution-side issues #756 / #753 it describes). +//! +//! `DimensionsContext`'s two spelling-keyed correspondences +//! (`positional_correspondence` and `executed_read_correspondence`, split out +//! by GH #997) rest on a fork in executed behavior: which of THREE resolution rules a +//! mapped reference gets -- by ordinal (POSITIONAL), by the element's own +//! NAME in the source dimension, or by the declared ELEMENT MAP -- depends on +//! how the reference is spelled, and the last two are a single name-first +//! path rather than two independent ones. That account was assembled from +//! reading the lowering plus one ground-truth comparison (C-LEARN's +//! `Ref.vdf`, gated by `simulates_clearn`). This module measures it instead, +//! so the rustdoc's rows are checkable and a change to any of them turns a +//! test red rather than silently invalidating the reasoning built on top. +//! +//! The two-rule framing (positional versus element map) is the natural one +//! and it is what an earlier revision of that rustdoc said. It is wrong; see +//! "Three resolution rules, not two" below. +//! +//! # The matrix +//! +//! Every cell is a whole model built through the production path +//! (`TestProject` -> salsa -> VM) with per-element source values that are +//! pairwise distinct, so which source element a target element read is +//! uniquely identified by the number that comes out. Three axes: +//! +//! **Reference spelling** ([`Spelling`], 4 variants). The first three are +//! GH #997's; the fourth ([`Spelling::StockFlow`]) was found by measuring +//! during this work, and the same change that added this module added its +//! bullet to the correspondence rustdocs. +//! +//! **Mapping kind** ([`MappingKind`], 5 variants) plus two no-mapping +//! controls (`no_mapping_*`), which are what distinguishes "resolves +//! positionally" from "there is no mapping machinery involved at all". +//! +//! **Declaration direction** ([`Direction`], 2 variants): which of the two +//! dimensions carries the mapping. Both directions of each fixture encode +//! the SAME element correspondence, so an expectation that holds for one +//! and not the other is a direction effect; [`expected`] is keyed on +//! (kind, spelling) only and `every_cell_of_the_matrix` asserts it for both +//! directions, which is where direction-insensitivity is asserted rather +//! than assumed. +//! +//! 4 x 5 x 2 = 40 mapped cells, all in `every_cell_of_the_matrix`; the two +//! controls contribute 4 cells each in `no_mapping_equal_cardinality` and +//! `no_mapping_unequal_cardinality`. The three axis enumerations are walked +//! through `Spelling::all` / `MappingKind::all` / `Direction::all`, each +//! built from a successor `match` so that adding a variant fails to compile +//! rather than silently escaping the matrix. +//! +//! No cell is collapsed away: the `MappingKind::Positional` row is not +//! discriminating (all three resolution rules agree there by construction) +//! and is kept precisely because that is why the fork went unnoticed -- +//! `fixture_discriminates` asserts which rows discriminate and which do not, +//! so "not discriminating" is recorded rather than implied by a missing test. +//! +//! # Three resolution rules, not two +//! +//! The obvious framing is positional-versus-element-map. It is wrong, and +//! [`MappingKind::SharedElementNames`] is the row that shows it: the two +//! map-following spellings actually resolve **name-first**, trying the active +//! element's own name in the source dimension and consulting the element map +//! only when that misses. So there are three candidate answers per fixture, +//! and [`assert_cell`] excludes every one that did not happen instead of +//! merely matching the one that did. +//! +//! # What the Vensim reference actually says +//! +//! Quoted from +//! (retrieved 2026-08-01): +//! +//! - the trigger: "Quite simply a mapping is an indication to Vensim that a +//! Subscript that appears on the right but not the left of an equation has +//! a valid interpretation." +//! - the shape it is written for: "Normally, an equation such as +//! `Quality[product] = work quality[worker type]` would generate an +//! error." -- the right-hand subscript names the SOURCE's own range, which +//! is [`Spelling::SourceOwnDim`] here. +//! - an explicit, order-bearing map: the general form +//! `Rhsub:rh1,rh2->(Lhsub:lh1,lh2), (Lhbigger:Lhbsubr1,Lhbsubr2), +//! (lhopposite:lho2,lho1)`, where "The map-to choices consist of the name +//! of the map-to subscript range, followed by a colon : and the elements +//! or subranges in the order the mapping should occur." The `lhopposite` +//! choice lists `lho2,lho1` against a range declared `lho1,lho2` -- a +//! PERMUTED map, which is [`MappingKind::Permuted`]. +//! - many-to-one: Example 5's `class: class1,class2->(metal:class1 +//! metal,class2 metal)` with `attractiveness[metal] = class +//! attractivness[class] * ...`, and "Note that while metal has 4 elements +//! class only has two. ... In this case two elements of metal belong to +//! each element of class, but it could also have been 3 and 1." That is +//! [`MappingKind::ManyToOne`] on [`Spelling::SourceOwnDim`], and it is the +//! shape C-LEARN ships. +//! - the iterated spelling IS legal Vensim, and the page credits a mapping +//! for it: Example 3 declares `PTASKS <-> TASKS` (stated to be the same as +//! `PTASKS : CLEAR,DIG,BUILD -> TASKS`), writes `prereq qual[task,ptask] = +//! quality factors[ptask]`, and says "This will work even though quality +//! factors is actually defined by task, not by ptask." The right-hand +//! subscript names a range the LEFT side iterates while the source is +//! declared over another -- structurally [`Spelling::IteratedDim`]. +//! - BOTH declaration directions appear. Examples 1, 4 and 5 declare the +//! mapping on the source variable's own range pointing at the left-hand one +//! (`Rhsub -> Lhsub`, the form the prose describes) = [`Direction::OnSourceDim`]; +//! Example 3 declares it on `PTASKS`, a range the left side iterates, +//! pointing at `TASKS`, the source's own range = [`Direction::OnIteratedDim`]. +//! Neither direction is a Simlin extension. (Example 2 does not bear on +//! this, PROVIDED `aging` is declared over `AGE`: both of its ranges are +//! then subranges of the source's own. The page never states `aging`'s +//! declaration; it is inferable only from its three subscripted uses, +//! which between them cover all five `AGE` cohorts.) +//! +//! What the page does NOT settle, recorded as UNVERIFIED rather than assumed: +//! +//! - which RULE Vensim applies to the iterated spelling. Example 3 cannot +//! say, because `PTASKS` is a full subrange copy of `TASKS` with identical +//! element names in the same order, so positional, name-identity and +//! map-following all coincide there. The residual doubt therefore runs +//! toward Vensim MAPPING on that spelling where this engine resolves +//! positionally -- not toward Vensim rejecting it. +//! - anything about [`MappingKind::ReverseCardinality`]. That fixture leaves +//! one source element with no correspondent, which the page's Example 4 +//! forbids ("the Subscript Ranges ... must not overlap and must completely +//! exhaust the second group"), so it is not expressible in Vensim's syntax +//! at all. The row exists to isolate this engine's range check -- it is why +//! [`MappingKind::ManyToOne`] is refused on the positional spellings -- and +//! no Vensim-parity inference should be drawn from it. +//! +//! Element maps themselves are a `simlin:mapping`/`simlin:elem` vendor +//! extension: XMILE 1.0's §2.5 `` block admits only +//! `` and `` and the spec has no dimension-mapping +//! construct, the sole prose occurrence of "mapping" being §3.6's discussion +//! of mapping unsupported FUNCTIONS. Checked by stripping the tags from +//! `docs/reference/xmile-v1.0.html` and searching the prose -- the method the +//! root `CLAUDE.md` prescribes, and the only one that works here, since the +//! file is non-UTF-8 and plain `grep` reports no match for ANY query. +//! +//! # The two routes a subscript-less reference can take +//! +//! The measured surprise, and the reason [`Spelling::StockFlow`] exists: a +//! reference spelled with NO subscript has two different lowerings with +//! different answers. +//! +//! - Inside an equation body (`target[State] = x`), `Context::lower_pass0` +//! rewrites the bare `Expr2::Var` into an `Expr2::Subscript` spelled with +//! the ACTIVE dimension's name -- so a bare in-equation reference IS the +//! [`Spelling::IteratedDim`] spelling by the time anything resolves it, +//! and is positional. When `match_dimensions_with_mapping` finds no +//! correspondence at all it emits a wildcard instead, and the reference +//! becomes a whole-array broadcast -- a third behavior, which +//! `no_mapping_unequal_cardinality` separates from the other two. +//! - As a stock's inflow/outflow (`level[State] = INTEG(feed, 0)` with +//! `feed` declared over `Region`), the reference never passes through +//! pass 0: `Context::fold_flows` calls `get_ref` directly, reaching +//! `get_implicit_subscript_off`, whose +//! `dim.get_offset(&element).or_else(...)` tries the active element's own +//! NAME in the source dimension first and consults +//! `DimensionsContext::translate_via_mapping` only when that misses. +//! +//! So the two subscript-less spellings disagree, and +//! `a_bare_equation_reference_and_a_flow_reference_disagree` pins that +//! disagreement directly, since it is the observable consequence of the +//! routing and the thing a future refactor is most likely to erase by +//! accident. + +use crate::common::ErrorCode; +use crate::datamodel; +use crate::test_common::TestProject; + +/// How the reference to the `Region`-declared source is written. +#[derive(Copy, Clone)] +enum Spelling { + /// `target[State] = x[State]` -- the subscript names the dimension the + /// equation ITERATES. Pass 1 (`ast::expr3::Pass1Context`) folds an + /// active dimension name to that dimension's ordinal, which then indexes + /// the source's storage raw. + IteratedDim, + /// `target[State] = x[Region]` -- the subscript names a dimension that is + /// NOT active, here the source's own. It survives pass 1 as an + /// `IndexExpr3::Dimension`, is normalized to an `IndexOp::ActiveDimRef` + /// by the free function `compiler::subscript::normalize_subscripts3`, and + /// is resolved in that module's `build_view_from_ops`, whose + /// `dim.get_offset(subscript).or_else(...)` tries the active element's own + /// NAME in the source dimension before falling back to + /// `DimensionsContext::translate_via_mapping`. + SourceOwnDim, + /// `target[State] = x` -- no subscript, inside an equation body. + BareInEquation, + /// `level[State] = INTEG(x, 0)` where `x` is a `Region`-declared flow -- + /// no subscript, and a different lowering route from `BareInEquation`. + StockFlow, +} + +/// The correspondence between `State` (the target's dimension) and `Region` +/// (the source's), and how it is declared. +#[derive(Copy, Clone)] +enum MappingKind { + /// `maps_to` with no element map: correspondence is by position, so + /// positional resolution and map-following cannot be told apart. + Positional, + /// An explicit element map over equal cardinalities that is NOT the + /// identity permutation. + Permuted, + /// An explicit element map from 3 target elements onto 2 source + /// elements (C-LEARN's shape, and Vensim's Example 5). + ManyToOne, + /// An explicit element map from 2 target elements onto 3 source + /// elements -- the many-to-one arrangement with the cardinalities + /// swapped, so a positional read stays in range where `ManyToOne`'s runs + /// off the end. Not expressible in Vensim (it leaves a source element + /// with no correspondent); see the module docs. + ReverseCardinality, + /// An explicit element map between two dimensions that declare the SAME + /// element names in a DIFFERENT order -- Vensim's Example 3 idiom, where + /// a subrange copy shares its parent's element names. + /// + /// This is the row that shows map-following is really NAME-first: all + /// three candidate answers are distinct here, and the two map-following + /// spellings return the name-identity one. + SharedElementNames, +} + +/// Every variant of the three axis enumerations, walked as a successor +/// chain rather than written as a literal array. +/// +/// The chain is what makes the matrix exhaustive. A literal array compiles +/// fine when a variant is added and is missing from it, so the new variant +/// would never be run; here the `match` has no arm for it and the module +/// fails to build. [`expected`] enforces the other half -- a variant with no +/// row is likewise a compile error there. +impl Spelling { + fn all() -> Vec { + successors(Spelling::IteratedDim, |s| match s { + Spelling::IteratedDim => Some(Spelling::SourceOwnDim), + Spelling::SourceOwnDim => Some(Spelling::BareInEquation), + Spelling::BareInEquation => Some(Spelling::StockFlow), + Spelling::StockFlow => None, + }) + } +} + +impl MappingKind { + fn all() -> Vec { + successors(MappingKind::Positional, |k| match k { + MappingKind::Positional => Some(MappingKind::Permuted), + MappingKind::Permuted => Some(MappingKind::ManyToOne), + MappingKind::ManyToOne => Some(MappingKind::ReverseCardinality), + MappingKind::ReverseCardinality => Some(MappingKind::SharedElementNames), + MappingKind::SharedElementNames => None, + }) + } +} + +fn successors(first: T, next: impl Fn(T) -> Option) -> Vec { + let mut out = vec![first]; + while let Some(n) = next(*out.last().expect("seeded above")) { + out.push(n); + } + out +} + +/// Which dimension carries the mapping declaration. +#[derive(Copy, Clone)] +enum Direction { + /// Declared on `State`, the dimension the target equation iterates. + /// Vensim's Example 3 (see module docs). + OnIteratedDim, + /// Declared on `Region`, the source variable's own dimension. Vensim's + /// Examples 1, 4 and 5 (see module docs). + OnSourceDim, +} + +impl Direction { + fn all() -> Vec { + successors(Direction::OnIteratedDim, |d| match d { + Direction::OnIteratedDim => Some(Direction::OnSourceDim), + Direction::OnSourceDim => None, + }) + } +} + +/// What a cell of the matrix does when it runs. +enum Expected { + /// The target's elements, in declared order, read these source values. + Reads(&'static [f64]), + /// The model does not compile, with this diagnostic code. + Refused(ErrorCode), +} + +impl Spelling { + fn label(self) -> &'static str { + match self { + Spelling::IteratedDim => "target[State] = x[State]", + Spelling::SourceOwnDim => "target[State] = x[Region]", + Spelling::BareInEquation => "target[State] = x", + Spelling::StockFlow => "target[State] = INTEG(x, 0)", + } + } +} + +impl MappingKind { + fn label(self) -> &'static str { + match self { + MappingKind::Positional => "positional (maps_to)", + MappingKind::Permuted => "permuted element map", + MappingKind::ManyToOne => "many-to-one element map (3 State onto 2 Region)", + MappingKind::ReverseCardinality => { + "reverse-cardinality element map (2 State onto 3 Region)" + } + MappingKind::SharedElementNames => { + "element map over dimensions sharing element names, reordered" + } + } + } + + /// The source elements and their (pairwise distinct) values. + fn source(self) -> &'static [(&'static str, &'static str)] { + match self { + MappingKind::Positional | MappingKind::Permuted | MappingKind::ReverseCardinality => { + &[("Ruby", "10"), ("Rose", "20"), ("Reed", "30")] + } + MappingKind::ManyToOne => &[("Ruby", "10"), ("Rose", "20")], + MappingKind::SharedElementNames => &[("Ann", "10"), ("Bob", "20"), ("Cal", "30")], + } + } + + /// The target's elements in declared order. + fn target_elements(self) -> &'static [&'static str] { + match self { + MappingKind::Positional | MappingKind::Permuted | MappingKind::ManyToOne => { + &["Steel", "Slate", "Stone"] + } + MappingKind::ReverseCardinality => &["Steel", "Slate"], + // The SOURCE's names, ROTATED: that is what pulls name identity + // apart from position, so the row has three distinct answers + // rather than two. + MappingKind::SharedElementNames => &["Cal", "Ann", "Bob"], + } + } + + /// The declared correspondence as (target element, source element) + /// pairs, in the target's declared order. + fn correspondence(self) -> &'static [(&'static str, &'static str)] { + match self { + // Identity by position -- `maps_to` carries no element list. + MappingKind::Positional => &[("Steel", "Ruby"), ("Slate", "Rose"), ("Stone", "Reed")], + MappingKind::Permuted => &[("Steel", "Reed"), ("Slate", "Ruby"), ("Stone", "Rose")], + MappingKind::ManyToOne => &[("Steel", "Ruby"), ("Slate", "Rose"), ("Stone", "Ruby")], + MappingKind::ReverseCardinality => &[("Steel", "Reed"), ("Slate", "Ruby")], + // Chosen to differ from BOTH the positional and the name-identity + // answer, so a cell that follows the map is unmistakable. + MappingKind::SharedElementNames => &[("Cal", "Bob"), ("Ann", "Cal"), ("Bob", "Ann")], + } + } + + /// What each target element reads if the declared element map is + /// followed. + fn map_reads(self) -> Vec { + self.correspondence() + .iter() + .map(|(_, src)| { + self.source() + .iter() + .find(|(name, _)| name == src) + .map(|(_, v)| v.parse::().unwrap()) + .unwrap_or_else(|| panic!("correspondence names an unknown source {src}")) + }) + .collect() + } + + /// What each target element reads if resolved POSITIONALLY (target + /// ordinal indexes the source's storage), or `None` when an ordinal runs + /// off the end of the source. + fn positional_reads(self) -> Option> { + let source = self.source(); + self.target_elements() + .iter() + .enumerate() + .map(|(i, _)| source.get(i).map(|(_, v)| v.parse::().unwrap())) + .collect() + } + + /// The element names this fixture's two dimensions have in common. + /// + /// [`Self::name_identity_reads`] cannot answer this: it collects into an + /// `Option`, so ONE missing element makes the whole thing `None` and a + /// partially-overlapping fixture would look disjoint. Since `None` is + /// also what switches off the name-identity exclusion in [`assert_cell`], + /// reading disjointness off it would let a row silently lose that + /// exclusion while a test asserted it had none to lose. + fn shared_element_names(self) -> Vec<&'static str> { + self.target_elements() + .iter() + .copied() + .filter(|elem| self.source().iter().any(|(name, _)| name == elem)) + .collect() + } + + /// What each target element reads if the source is indexed by the target + /// element's own NAME, or `None` unless EVERY target element is present + /// in the source (a partial overlap has no well-defined answer for this + /// rule, so there is nothing to exclude). + fn name_identity_reads(self) -> Option> { + let source = self.source(); + self.target_elements() + .iter() + .map(|elem| { + source + .iter() + .find(|(name, _)| name == elem) + .map(|(_, v)| v.parse::().unwrap()) + }) + .collect() + } + + /// Every resolution rule's answer, labelled. `assert_cell` asserts the + /// measured read equals the expected rule's answer and differs from each + /// other rule's, so a cell excludes what did not happen rather than only + /// matching what did. + fn candidate_answers(self) -> [(&'static str, Option>); 3] { + [ + ("positional", self.positional_reads()), + ("name identity", self.name_identity_reads()), + ("element map", Some(self.map_reads())), + ] + } +} + +/// The dimension pair for one (kind, direction), with the mapping declared +/// on exactly one of them. +fn dimensions(kind: MappingKind, direction: Direction) -> Vec { + let region_elems: Vec = kind + .source() + .iter() + .map(|(name, _)| name.to_string()) + .collect(); + let state_elems: Vec = kind + .target_elements() + .iter() + .map(|s| s.to_string()) + .collect(); + let mut region = datamodel::Dimension::named("Region".to_string(), region_elems); + let mut state = datamodel::Dimension::named("State".to_string(), state_elems); + + match (kind, direction) { + (MappingKind::Positional, Direction::OnIteratedDim) => { + state.set_maps_to("Region".to_string()) + } + (MappingKind::Positional, Direction::OnSourceDim) => { + region.set_maps_to("State".to_string()) + } + (_, Direction::OnIteratedDim) => { + state.mappings = vec![datamodel::DimensionMapping { + target: "Region".to_string(), + element_map: kind + .correspondence() + .iter() + .map(|(t, s)| (t.to_string(), s.to_string())) + .collect(), + }]; + } + (_, Direction::OnSourceDim) => { + region.mappings = vec![datamodel::DimensionMapping { + target: "State".to_string(), + element_map: kind + .correspondence() + .iter() + .map(|(t, s)| (s.to_string(), t.to_string())) + .collect(), + }]; + } + } + vec![region, state] +} + +/// Build the whole model for one cell. +fn model(kind: MappingKind, direction: Direction, spelling: Spelling) -> TestProject { + let mut project = TestProject::new("mapped_reference"); + project.dimensions = dimensions(kind, direction); + match spelling { + Spelling::StockFlow => project + .array_flow_with_ranges("x[Region]", kind.source().to_vec()) + .array_stock("target[State]", "0", &["x"], &[], None), + Spelling::IteratedDim => project + .array_with_ranges("x[Region]", kind.source().to_vec()) + .array_aux("target[State]", "x[State]"), + Spelling::SourceOwnDim => project + .array_with_ranges("x[Region]", kind.source().to_vec()) + .array_aux("target[State]", "x[Region]"), + Spelling::BareInEquation => project + .array_with_ranges("x[Region]", kind.source().to_vec()) + .array_aux("target[State]", "x"), + } +} + +/// Run one cell and report what the target's elements read, or the message +/// that stopped it -- kept verbatim so a VM RUN failure is distinguishable +/// from a compile refusal when a cell's expectation turns out to be wrong. +fn run_cell( + kind: MappingKind, + direction: Direction, + spelling: Spelling, +) -> Result, String> { + let project = model(kind, direction, spelling); + let results = project.run_vm()?; + Ok(kind + .target_elements() + .iter() + .map(|elem| { + let key = format!("target[{}]", crate::canonicalize(elem)); + *results + .get(&key) + .unwrap_or_else(|| panic!("no series for {key}")) + .last() + .expect("empty series") + }) + .collect()) +} + +/// The executed behavior of every (mapping kind, spelling) pair. +/// +/// Exhaustive over the product of the two enumerations: adding a variant to +/// either is a compile error here, not a silently uncovered cell. Direction +/// is deliberately NOT a parameter -- see the module docs. +fn expected(kind: MappingKind, spelling: Spelling) -> Expected { + use MappingKind::*; + use Spelling::*; + match (kind, spelling) { + // A positional mapping makes the two resolutions agree, so all four + // spellings return the same numbers. This row is why the fork below + // could go unnoticed for so long. + (Positional, IteratedDim) + | (Positional, SourceOwnDim) + | (Positional, BareInEquation) + | (Positional, StockFlow) => Expected::Reads(&[10.0, 20.0, 30.0]), + + // The permuted row is the fork, in its clearest form: the same + // three source values, read in two different orders depending only + // on how the reference is spelled. + (Permuted, IteratedDim) => Expected::Reads(&[10.0, 20.0, 30.0]), + (Permuted, BareInEquation) => Expected::Reads(&[10.0, 20.0, 30.0]), + (Permuted, SourceOwnDim) => Expected::Reads(&[30.0, 10.0, 20.0]), + (Permuted, StockFlow) => Expected::Reads(&[30.0, 10.0, 20.0]), + + // Many-to-one: the positional spellings have no third source + // element to index and are refused. `Generic` is what the static + // subscript resolution reports ("Index out of bounds for dimension + // 0", `compiler::subscript`) -- it is pinned as the code that ships, + // not endorsed; a mapping-aware diagnostic would be better and is + // GH #753's territory. + (ManyToOne, IteratedDim) => Expected::Refused(ErrorCode::Generic), + (ManyToOne, BareInEquation) => Expected::Refused(ErrorCode::Generic), + (ManyToOne, SourceOwnDim) => Expected::Reads(&[10.0, 20.0, 10.0]), + (ManyToOne, StockFlow) => Expected::Reads(&[10.0, 20.0, 10.0]), + + // Reverse cardinality isolates WHY many-to-one is refused: it is the + // positional index leaving the source's range, not unequal + // cardinality as such. Here every target ordinal is in range, so the + // positional spellings compile -- and read the wrong elements. + (ReverseCardinality, IteratedDim) => Expected::Reads(&[10.0, 20.0]), + (ReverseCardinality, BareInEquation) => Expected::Reads(&[10.0, 20.0]), + (ReverseCardinality, SourceOwnDim) => Expected::Reads(&[30.0, 10.0]), + (ReverseCardinality, StockFlow) => Expected::Reads(&[30.0, 10.0]), + + // Shared element names. The positional spellings are unmoved -- they + // never look at a name. The other two return NAME IDENTITY, not the + // element map: `Cal` reads `Cal` (30) though the map says `Bob` (20). + // Source values 10/20/30 over {Ann,Bob,Cal}; target {Cal,Ann,Bob}. + (SharedElementNames, IteratedDim) => Expected::Reads(&[10.0, 20.0, 30.0]), + (SharedElementNames, BareInEquation) => Expected::Reads(&[10.0, 20.0, 30.0]), + (SharedElementNames, SourceOwnDim) => Expected::Reads(&[30.0, 10.0, 20.0]), + (SharedElementNames, StockFlow) => Expected::Reads(&[30.0, 10.0, 20.0]), + } +} + +fn assert_cell(kind: MappingKind, direction: Direction, spelling: Spelling) { + let where_ = format!( + "{} / {} / declared on {}", + kind.label(), + spelling.label(), + match direction { + Direction::OnIteratedDim => "State (the iterated dim)", + Direction::OnSourceDim => "Region (the source's dim)", + } + ); + match expected(kind, spelling) { + Expected::Reads(want) => { + let got = run_cell(kind, direction, spelling).unwrap_or_else(|e| { + panic!("{where_}: expected {want:?}, but the model did not run: {e}") + }); + assert_eq!(got, want, "{where_}"); + + // Every OTHER resolution rule's answer must be excluded, not + // merely un-asserted. A rule that cannot apply to this fixture + // yields `None`, and one that agrees with `want` has nothing to + // exclude -- `fixture_discriminates` records which rows are in + // that position, so it is a stated property rather than a gap. + for (rule, answer) in kind.candidate_answers() { + if let Some(answer) = answer + && answer != want + { + assert_ne!(got, answer, "{where_}: read the {rule} answer {answer:?}"); + } + } + } + Expected::Refused(code) => { + assert_refused(&model(kind, direction, spelling), code, &where_); + } + } +} + +/// A compile refusal, pinned to the FAILING VARIABLE as well as the code. +/// +/// `TestProject::assert_compile_error_vm` accepts any Error-severity +/// diagnostic anywhere in the project carrying the code, which for a code as +/// broad as `Generic` is close to no constraint at all. `TestProject::compile` +/// reports `("model.variable", code)` pairs from the same +/// `collect_all_diagnostics` pass, so the pin can name the target. A message +/// substring would be stronger still, but the diagnostic these cells produce +/// is an `EquationError`, which carries a code and a span and no text. +fn assert_refused(project: &TestProject, code: ErrorCode, where_: &str) { + let errors = match project.compile() { + Ok(_) => panic!("{where_}: expected a compile failure, but it compiled"), + Err(errors) => errors, + }; + let want = ("main.target".to_string(), code); + assert!( + errors.contains(&want), + "{where_}: expected {want:?} among the diagnostics, got {errors:?}" + ); +} + +#[test] +fn every_cell_of_the_matrix() { + for kind in MappingKind::all() { + for direction in Direction::all() { + for spelling in Spelling::all() { + assert_cell(kind, direction, spelling); + } + } + } +} + +/// Which rows of the matrix can tell the resolution rules apart. +/// +/// Without this, a reader cannot distinguish "this cell pins map-following" +/// from "this cell would pass either way", and the `Positional` row -- which +/// is the second kind -- looks like coverage it is not. The `match` is +/// exhaustive, so a new mapping kind has to declare its discriminating power +/// here as well as its rows in [`expected`]. +#[test] +fn fixture_discriminates() { + for kind in MappingKind::all() { + let map = kind.map_reads(); + let positional = kind.positional_reads(); + let name_identity = kind.name_identity_reads(); + match kind { + MappingKind::Positional => assert_eq!( + Some(map), + positional, + "the positional row must NOT discriminate -- if it does, the \ + fixture no longer models a positional mapping" + ), + MappingKind::ManyToOne => assert!( + positional.is_none(), + "the many-to-one row's positional read must run off the end of \ + the source; that is what makes its two refused cells meaningful" + ), + MappingKind::Permuted | MappingKind::ReverseCardinality => { + assert_ne!( + Some(map), + positional, + "{}: map-following and positional must differ, or its cells \ + pass either way", + kind.label() + ); + assert_eq!( + kind.shared_element_names(), + Vec::<&str>::new(), + "{}: these rows must share NO element names, so that name \ + identity cannot apply and the row is a clean two-way test", + kind.label() + ); + } + // The only row where all THREE rules apply and disagree. That is + // the whole point of it: without three distinct answers it could + // not show that map-following is really name-first. + MappingKind::SharedElementNames => { + let map = Some(map); + assert_ne!(map, positional, "{}", kind.label()); + assert_ne!(map, name_identity, "{}", kind.label()); + assert_ne!(positional, name_identity, "{}", kind.label()); + assert_eq!( + kind.shared_element_names(), + kind.target_elements(), + "{}: EVERY target element must also be a source element, or \ + the name-identity rule is only partly applicable here", + kind.label() + ); + } + } + } +} + +/// Control: with NO mapping declared at all, at equal cardinality. +/// +/// The point is that two of the four spellings do not require a mapping to +/// exist. `IteratedDim` never consults one (pass 1 folds the active +/// dimension to an ordinal and indexes the source raw), and +/// `BareInEquation` falls back to a whole-array broadcast -- so a +/// cross-dimension read between two dimensions declared to have NOTHING to +/// do with each other compiles and silently produces numbers. The two +/// spellings that DO consult the mapping are refused. +#[test] +fn no_mapping_equal_cardinality() { + let dims = || { + vec![ + datamodel::Dimension::named( + "Region".to_string(), + vec!["Ruby".to_string(), "Rose".to_string(), "Reed".to_string()], + ), + datamodel::Dimension::named( + "State".to_string(), + vec![ + "Steel".to_string(), + "Slate".to_string(), + "Stone".to_string(), + ], + ), + ] + }; + let cases: [(Spelling, Expected); 4] = [ + (Spelling::IteratedDim, Expected::Reads(&[10.0, 20.0, 30.0])), + ( + Spelling::BareInEquation, + Expected::Reads(&[10.0, 20.0, 30.0]), + ), + ( + Spelling::SourceOwnDim, + Expected::Refused(ErrorCode::MismatchedDimensions), + ), + ( + Spelling::StockFlow, + Expected::Refused(ErrorCode::MismatchedDimensions), + ), + ]; + assert_no_mapping_cases(dims, &["Steel", "Slate", "Stone"], &cases); +} + +/// Control: NO mapping, and the target has FEWER elements than the source. +/// +/// This is what separates `BareInEquation` from `IteratedDim`. They agree in +/// every other no-mapping cell, which reads like one behavior; here the +/// broadcast fallback needs the two extents to match and is refused, while +/// the iterated spelling -- which only needs its ordinal to be in range -- +/// still compiles. Two routes, not one. +#[test] +fn no_mapping_unequal_cardinality() { + let dims = || { + vec![ + datamodel::Dimension::named( + "Region".to_string(), + vec!["Ruby".to_string(), "Rose".to_string(), "Reed".to_string()], + ), + datamodel::Dimension::named( + "State".to_string(), + vec!["Steel".to_string(), "Slate".to_string()], + ), + ] + }; + let cases: [(Spelling, Expected); 4] = [ + (Spelling::IteratedDim, Expected::Reads(&[10.0, 20.0])), + ( + Spelling::BareInEquation, + Expected::Refused(ErrorCode::MismatchedDimensions), + ), + ( + Spelling::SourceOwnDim, + Expected::Refused(ErrorCode::MismatchedDimensions), + ), + ( + Spelling::StockFlow, + Expected::Refused(ErrorCode::MismatchedDimensions), + ), + ]; + assert_no_mapping_cases(dims, &["Steel", "Slate"], &cases); +} + +fn assert_no_mapping_cases( + dims: impl Fn() -> Vec, + target_elements: &[&str], + cases: &[(Spelling, Expected)], +) { + let source = [("Ruby", "10"), ("Rose", "20"), ("Reed", "30")]; + for (spelling, want) in cases { + let mut project = TestProject::new("no_mapping"); + project.dimensions = dims(); + let project = match spelling { + Spelling::StockFlow => project + .array_flow_with_ranges("x[Region]", source.to_vec()) + .array_stock("target[State]", "0", &["x"], &[], None), + Spelling::IteratedDim => project + .array_with_ranges("x[Region]", source.to_vec()) + .array_aux("target[State]", "x[State]"), + Spelling::SourceOwnDim => project + .array_with_ranges("x[Region]", source.to_vec()) + .array_aux("target[State]", "x[Region]"), + Spelling::BareInEquation => project + .array_with_ranges("x[Region]", source.to_vec()) + .array_aux("target[State]", "x"), + }; + let label = spelling.label(); + match want { + Expected::Reads(want) => { + let results = project + .run_vm() + .unwrap_or_else(|e| panic!("no mapping / {label}: expected it to run: {e}")); + let got: Vec = target_elements + .iter() + .map(|elem| { + *results[&format!("target[{}]", crate::canonicalize(elem))] + .last() + .expect("empty series") + }) + .collect(); + assert_eq!(got, *want, "no mapping / {label}"); + } + Expected::Refused(code) => { + assert_refused(&project, *code, &format!("no mapping / {label}")); + } + } + } +} + +/// The two subscript-less spellings disagree, and this is the single +/// assertion that says so out loud. +/// +/// The two correspondences split "bare" between them for exactly this reason: +/// a bare reference in an EQUATION is positional (`positional_correspondence`), +/// while a stock's flow reference -- equally subscript-less -- resolves +/// name-first and, where the names differ, follows the element map. The +/// difference is entirely the lowering route (module docs), so any refactor +/// that unifies the two -- which is a natural thing to want -- changes +/// executed numbers on one side or the other, and this test is what makes +/// that loud. +/// +/// `Permuted` is the kind to run it on: its two dimensions share no element +/// names, so the flow reference reaches its element-map fallback rather than +/// stopping at name identity. +#[test] +fn a_bare_equation_reference_and_a_flow_reference_disagree() { + let kind = MappingKind::Permuted; + for direction in Direction::all() { + let bare = run_cell(kind, direction, Spelling::BareInEquation) + .expect("the bare equation reference compiles"); + let flow = + run_cell(kind, direction, Spelling::StockFlow).expect("the flow reference compiles"); + assert_eq!(bare, kind.positional_reads().unwrap()); + assert_eq!(flow, kind.map_reads()); + assert_ne!( + bare, flow, + "the two subscript-less spellings must still disagree under a \ + permuted element map" + ); + } +} + +/// GH #996, on the EXECUTED path: an earlier dependency axis must not claim +/// BY MAPPING the active slot a later axis matches BY NAME. +/// +/// This is the hazard shape as a whole model rather than a hand-built call +/// to `allocate_implicit_axes_partial`. Reaching the allocator from a real +/// model constrains the fixture: ordinary expression references never get +/// there, because `Context::lower_pass0` rewrites a bare arrayed reference +/// into an explicit subscript first (module docs). Tagging each call with its +/// caller and running the whole lib suite found EXACTLY TWO production +/// callers, both wiring rather than expressions -- `Context::fold_flows` (a +/// stock's flow references) and `compiler::Var::new` (the stock +/// self-reference and module input wiring), with zero arriving via +/// `lower_from_expr3` -- so a stock whose FLOW is declared over two +/// dimensions is the way in. The counts and their measurement condition are +/// on `compiler::context`'s `get_implicit_subscripts`. +/// +/// `feed` is declared `[Board Type, Line]` and `level` iterates +/// `[Line, Shift]`. `Board Type` maps to BOTH `Line` (positionally) and +/// `Shift` (through a non-identity element map), so it can claim either +/// slot; `Line` matches slot 0 by name and has nothing else. Name-first +/// allocation therefore gives `Line` its slot and leaves `Board Type` the +/// `Shift` slot, which it resolves through the element map: +/// `Day Shift -> Oak Board`, `Night Shift -> Pine Board`. +/// +/// The values are what makes this a pin rather than a smoke test. Reading +/// the element map gives `level[Line One, Day Shift] = feed[Oak Board, Line +/// One] = 104`; the swap gives 101, and every one of the four cells has a +/// distinct wrong answer. Under the pre-#996 order-greedy allocation the +/// model does not compile at all (`Board Type`, processed first, takes the +/// `Line` slot by mapping and `Line` finds it gone) -- verified by +/// temporarily restoring the per-dimension staging, which turns this test +/// into `MismatchedDimensions` on `level`. +/// +/// The dimension and element names carry capitals and spaces deliberately: +/// GH #996 records that a lowercase single-word name is already canonical, +/// so a fixture built from one passes vacuously. +#[test] +fn the_996_hazard_shape_compiles_and_reads_name_first() { + let project = TestProject::new("implicit_axis_precedence") + .named_dimension("Line", &["Line One", "Line Two"]) + .named_dimension("Shift", &["Day Shift", "Night Shift"]) + .named_dimension_with_mappings( + "Board Type", + &["Pine Board", "Oak Board"], + &[ + // Positional, and present only so the mapping pass can + // reach the `Line` slot -- the steal the fix prevents. + ("Line", &[]), + // Non-identity, so the slot it legitimately gets is + // resolved by the map rather than by position. + ( + "Shift", + &[("Pine Board", "Night Shift"), ("Oak Board", "Day Shift")], + ), + ], + ) + .array_with_ranges( + "boardweight[Board Type]", + vec![("Pine Board", "1"), ("Oak Board", "4")], + ) + .array_with_ranges( + "lineweight[Line]", + vec![("Line One", "100"), ("Line Two", "200")], + ) + .array_flow( + "feed[Board Type, Line]", + "boardweight[\"Board Type\"] + lineweight[Line]", + None, + ) + .array_stock("level[Line, Shift]", "0", &["feed"], &[], None); + + let results = project + .run_vm() + .expect("the hazard shape must compile and run"); + + for (cell, want, wrong) in [ + ("level[line_one,day_shift]", 104.0, 101.0), + ("level[line_one,night_shift]", 101.0, 104.0), + ("level[line_two,day_shift]", 204.0, 201.0), + ("level[line_two,night_shift]", 201.0, 204.0), + ] { + let got = *results[cell].last().expect("empty series"); + assert_eq!(got, want, "{cell}"); + assert_ne!(got, wrong, "{cell}: read the axis-swapped element instead"); + } +} + +// =========================================================================== +// Two axes driven by ONE target dimension, and what LTM describes about them +// =========================================================================== +// +// The matrix above varies ONE source axis. A reference can also spell two axes +// that resolve from the SAME active target element -- `matrix[Region1, Region2]` +// under a `State`-iterating equation with both regions mapped to `State`, or its +// iterated twin `matrix[State, State]`, or the degenerate `matrix[D, D]`. Each +// index goes through the same resolution the matrix pins for a single axis, and +// against the same active element, so the read is that dimension's DIAGONAL: N +// reads over an NxN source, not N^2. +// +// LTM has to describe exactly those reads. It derives them twice -- the element +// GRAPH from `db::ltm::read_slice_row_parts`, the link-score NAMES from +// `ltm_augment::per_element_row_for_target` -- and the two disagreed: the second +// projects one target element through each axis and so always produced the +// diagonal, while the first enumerated each axis independently and crossed them. +// The cross rows became element edges the simulation never traverses and loop +// candidates built on them (9 loops over a 3x3 source where there are 3). The +// assertions below are on one fixture per shape so the VM oracle and the +// description are the same model, not two models that happen to agree. + +/// `matrix[Ra,Rb]` values, 11..33, distinct per cell so the pair of source +/// elements a target element read is uniquely identified by the number. +fn diagonal_matrix_cells() -> Vec<(&'static str, &'static str)> { + vec![ + ("ra1,rb1", "11"), + ("ra1,rb2", "12"), + ("ra1,rb3", "13"), + ("ra2,rb1", "21"), + ("ra2,rb2", "22"), + ("ra2,rb3", "23"), + ("ra3,rb1", "31"), + ("ra3,rb2", "32"), + ("ra3,rb3", "33"), + ] +} + +/// `State` element-mapped to BOTH `Ra` and `Rb`, by two DIFFERENT rotations, so +/// the diagonal `matrix[map1(s), map2(s)]` is off the matrix's own diagonal and +/// no cell can be reached by two different rules. +/// +/// * `s1 -> ra2, rb3` (23) +/// * `s2 -> ra3, rb1` (31) +/// * `s3 -> ra1, rb2` (12) +fn two_mapped_axes_project(name: &str) -> TestProject { + TestProject::new(name) + .named_dimension_with_mappings( + "State", + &["s1", "s2", "s3"], + &[ + ("Ra", &[("s1", "ra2"), ("s2", "ra3"), ("s3", "ra1")]), + ("Rb", &[("s1", "rb3"), ("s2", "rb1"), ("s3", "rb2")]), + ], + ) + .named_dimension("Ra", &["ra1", "ra2", "ra3"]) + .named_dimension("Rb", &["rb1", "rb2", "rb3"]) +} + +/// Every element edge the model's LTM element graph carries, as +/// `"from[row] -> to[element]"` strings. +fn element_edge_pairs(project: &TestProject) -> Vec { + use crate::db::{SimlinDb, model_element_causal_edges, sync_from_datamodel}; + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + let edges = model_element_causal_edges(&db, sync.models["main"].source, sync.project).clone(); + let mut pairs: Vec = edges + .edges + .iter() + .flat_map(|(from, tos)| tos.iter().map(move |to| format!("{from} -> {to}"))) + .collect(); + pairs.sort(); + pairs +} + +/// The edges out of `from_prefix`, so an assertion can be exhaustive about one +/// reference without restating the rest of the model's graph. +fn edges_from(project: &TestProject, from_prefix: &str) -> Vec { + element_edge_pairs(project) + .into_iter() + .filter(|p| p.starts_with(from_prefix)) + .collect() +} + +/// Both indices of `matrix[Ra,Rb]` resolve against the one active `State` +/// element, so the read is the diagonal of the two mappings -- three reads over +/// a 3x3 source. +#[test] +fn two_axes_mapped_to_one_target_dimension_read_the_diagonal() { + let results = two_mapped_axes_project("two_mapped_exec") + .array_with_ranges("matrix[Ra,Rb]", diagonal_matrix_cells()) + .array_aux("target[State]", "matrix[Ra,Rb]") + .run_vm() + .expect("the two-mapped-axes shape must compile and run"); + for (cell, want) in [ + ("target[s1]", 23.0), + ("target[s2]", 31.0), + ("target[s3]", 12.0), + ] { + assert_eq!( + *results[cell].last().expect("empty series"), + want, + "{cell}: each index must resolve through its own map against the \ + SAME active State element" + ); + } +} + +/// The element graph describes exactly those three reads. +/// +/// Exhaustive over the reference's own edges rather than a spot check: the +/// defect was extra rows, and an assertion that only names the three real ones +/// passes just as well with six phantoms beside them. +#[test] +fn two_mapped_axes_emit_only_the_diagonal_element_edges() { + let project = two_mapped_axes_project("two_mapped_edges") + .array_with_ranges("matrix[Ra,Rb]", diagonal_matrix_cells()) + .array_aux("target[State]", "matrix[Ra,Rb]"); + assert_eq!( + edges_from(&project, "matrix["), + vec![ + "matrix[ra1,rb2] -> target[s3]".to_string(), + "matrix[ra2,rb3] -> target[s1]".to_string(), + "matrix[ra3,rb1] -> target[s2]".to_string(), + ], + "the cross rows (matrix[ra1,rb1] and the five others) are reads the \ + simulation never makes" + ); +} + +/// The two `Iterated` spellings of the same shape. Both are older than the +/// mapped one and both had the same defect, reached through a different +/// classification: an all-`Iterated` subscript used to be `RefShape::Bare` +/// unconditionally, and `expand_same_element` -- which sees only the two +/// variables' dimension lists, never the reference -- cannot express "these two +/// axes share a coordinate". It unioned the candidates for `matrix[D,D]` (15 +/// edges over a 3x3 source) and claimed the target position for the first axis +/// alone for `matrix[State,State]` (9 edges). +#[test] +fn two_iterated_axes_on_one_target_dimension_read_the_diagonal() { + // Positional mappings, so the iterated spelling folds to an ordinal: s_i + // reads ra_i and rb_i. + let mapped = TestProject::new("two_iterated") + .named_dimension_with_mappings("State", &["s1", "s2", "s3"], &[("Ra", &[]), ("Rb", &[])]) + .named_dimension("Ra", &["ra1", "ra2", "ra3"]) + .named_dimension("Rb", &["rb1", "rb2", "rb3"]) + .array_with_ranges("matrix[Ra,Rb]", diagonal_matrix_cells()) + .array_aux("target[State]", "matrix[State,State]"); + let results = mapped.run_vm().expect("must compile and run"); + for (cell, want) in [ + ("target[s1]", 11.0), + ("target[s2]", 22.0), + ("target[s3]", 33.0), + ] { + assert_eq!(*results[cell].last().expect("empty series"), want, "{cell}"); + } + assert_eq!( + edges_from(&mapped, "matrix["), + vec![ + "matrix[ra1,rb1] -> target[s1]".to_string(), + "matrix[ra2,rb2] -> target[s2]".to_string(), + "matrix[ra3,rb3] -> target[s3]".to_string(), + ], + "matrix[State,State] reads the diagonal" + ); + + // The degenerate spelling: one dimension, named twice. + let same = TestProject::new("same_dim_twice") + .named_dimension("D", &["d1", "d2", "d3"]) + .array_with_ranges( + "matrix[D,D]", + vec![ + ("d1,d1", "11"), + ("d1,d2", "12"), + ("d1,d3", "13"), + ("d2,d1", "21"), + ("d2,d2", "22"), + ("d2,d3", "23"), + ("d3,d1", "31"), + ("d3,d2", "32"), + ("d3,d3", "33"), + ], + ) + .array_aux("target[D]", "matrix[D,D]"); + let results = same.run_vm().expect("must compile and run"); + for (cell, want) in [ + ("target[d1]", 11.0), + ("target[d2]", 22.0), + ("target[d3]", 33.0), + ] { + assert_eq!(*results[cell].last().expect("empty series"), want, "{cell}"); + } + assert_eq!( + edges_from(&same, "matrix["), + vec![ + "matrix[d1,d1] -> target[d1]".to_string(), + "matrix[d2,d2] -> target[d2]".to_string(), + "matrix[d3,d3] -> target[d3]".to_string(), + ], + "matrix[D,D] reads the diagonal" + ); +} + +/// The boundary of the rule above: a TARGET that repeats the dimension. +/// +/// `cube[D1,D1] = pop[D1,D1]` resolves BOTH of the reference's indices to the +/// target's FIRST `D1` axis, so `cube[r1,r2]` reads `pop[r1,r1]` -- four reads, +/// which are NOT a diagonal in the target's own element tuple. Every +/// per-element derivation addresses a target axis by dimension NAME, and this +/// target has two coordinates for one name. +/// +/// **Why the retarget is narrowed to a singly-named target dimension, and what +/// that costs.** The reason is the SCORE surface, not the edges. `RefShape` +/// decides both, and `emit_per_element_link_scores` refuses a repeated-dimension +/// target outright, so retargeting this shape would convert an emitted `Bare` +/// link score into the loud per-element skip -- on every edge into or out of a +/// repeated-dimension variable, with the loops through them dropped. That is a +/// product decision about a shape this change is not about. On EDGES the +/// retarget would be better, and the assertions below say so rather than hiding +/// it: `Bare` emits 12 edges covering 2 of the 4 real reads, `PerElement` would +/// emit exactly those same 2 and no phantoms. Both miss the same two real reads, +/// so the retarget removes phantoms without fixing the missing half -- the half +/// that breaks loop discovery. +/// +/// **Pre-existing residual, pinned rather than fixed.** The missing half is +/// `db::analysis::expand_same_element` keying target positions by dimension NAME +/// (`HashMap<&str, usize>`), so a target repeating `D1` records only its LAST +/// axis and both source axes claim it. Fixing it means teaching that function +/// about repeated dimensions on either side -- the third instance of "a +/// dimension name is not an axis identity" here after GH #974 and GH #986 -- +/// which changes the arm every ordinary bare arrayed reference uses plus +/// `ltm_finding::expand_a2a_link_offsets`. Doing it there lets the edges and the +/// scores move together instead of trading one for the other. +/// +/// **Blast radius, measured.** Vensim REJECTS a repeated-dimension declaration +/// ("DimA appears more than once on LHS", `vensim-probes/repeated_dimension.mdl` +/// in Vensim DSS 2026-08-04), so no MDL-imported model reaches this residual and +/// it is confined to hand-authored XMILE/JSON/protobuf. The XMILE v1.0 spec does +/// exemplify the declaration, so the shape stays legitimate and the residual +/// stays worth fixing -- just not urgent, and not from an importer. +#[test] +fn a_repeated_target_dimension_reads_the_first_axis_on_both_sides() { + let project = TestProject::new("square_owner_reads") + .named_dimension("D1", &["r1", "r2"]) + .array_with_ranges( + "pop[D1,D1]", + vec![ + ("r1,r1", "11"), + ("r1,r2", "12"), + ("r2,r1", "21"), + ("r2,r2", "22"), + ], + ) + .array_aux("cube[D1,D1]", "pop[D1,D1]"); + let results = project.run_vm().expect("must compile and run"); + for (cell, want) in [ + ("cube[r1,r1]", 11.0), + ("cube[r1,r2]", 11.0), + ("cube[r2,r1]", 22.0), + ("cube[r2,r2]", 22.0), + ] { + assert_eq!( + *results[cell].last().expect("empty series"), + want, + "{cell}: both indices resolve to the target's FIRST D1 axis" + ); + } + + // The DISCRIMINATOR for the narrowing. Both of the above hold under the + // retarget too -- `PerElement` drops the same real edge and, being phantom- + // free, would satisfy the first line and fail the second only incidentally. + // What actually separates the two is that `Bare` keeps this edge SCOREABLE. + // Under the retarget `pop -> cube` gets no link-score variable and a loud + // per-element skip instead, so this assertion is what the narrowing is for. + let scored = square_owner_link_score_names(); + assert!( + scored + .iter() + .any(|n| n.ends_with("link_score\u{205A}pop\u{2192}cube")), + "the narrowing keeps this edge on the Bare emitter, which scores it; \ + retargeting it to PerElement replaces the score with a loud skip. \ + got: {scored:?}" + ); + + // The RESIDUAL's current behaviour, in both its directions. Neither line is + // a statement of what should be true: a fix to `expand_same_element` reds + // this test and has to restate what became true, which is the point of + // pinning a defect rather than leaving it silent. + let edges = edges_from(&project, "pop["); + assert!( + !edges.contains(&"pop[r1,r1] -> cube[r1,r2]".to_string()), + "residual (missing): a real read still has no edge; got {edges:?}" + ); + assert!( + edges.contains(&"pop[r1,r1] -> cube[r2,r1]".to_string()), + "residual (phantom): a read that never happens still has one; got {edges:?}" + ); +} + +/// The link scores a LOOP-carrying twin of the square-owner shape emits. +/// +/// Split out because a score only exists for an edge inside a feedback loop, so +/// the acyclic fixture above cannot ask the question. Same shape: `cube` and +/// `grow` both repeat `D1`, and the reference `pop[D1,D1]` inside `cube` is the +/// one whose `RefShape` the narrowing decides. +fn square_owner_link_score_names() -> Vec { + use crate::db::{ + SimlinDb, model_ltm_variables, set_project_ltm_enabled, sync_from_datamodel_incremental, + }; + let datamodel = TestProject::new("square_owner_scores") + .named_dimension("D1", &["r1", "r2"]) + .array_stock("pop[D1,D1]", "10", &["grow"], &[], None) + .array_flow("grow[D1,D1]", "cube[D1,D1] * 0.01", None) + .array_aux("cube[D1,D1]", "pop[D1,D1]") + .build_datamodel(); + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &datamodel, None); + set_project_ltm_enabled(&mut db, sync.project, true); + let mut names: Vec = + model_ltm_variables(&db, sync.models["main"].source_model, sync.project) + .vars + .iter() + .map(|v| v.name.clone()) + .filter(|n| n.contains("link_score")) + .collect(); + names.sort(); + names +} + +/// A canonical element name containing a COMMA does not derail the mapped +/// projection. +/// +/// The `PerElement` element-edge arm used to take the row derivation's +/// comma-JOINED slot string and re-split it on `,` to recover the per-axis +/// coordinates. A canonical element name can itself contain a comma -- a quoted +/// XMILE element `"a,b"` canonicalizes to `a,b` (measured: `canonicalize("a,b")` +/// is `a,b`, and a model declaring one compiles and simulates) -- so that +/// round-trip read one coordinate as two: the real edge was dropped and one to a +/// target element that does not exist was minted in its place. The arm now reads +/// `ReadSliceRowParts::slot_parts` directly and never serializes. +/// +/// `region` deliberately puts the comma element FIRST, so a mis-split shifts +/// every following coordinate rather than only the last. +/// +/// WHAT THIS CLOSES, AND WHAT IT DOES NOT. This fix covers the element-EDGE +/// surface. The link-SCORE surface still round-trips coordinate tuples through +/// comma-joined strings in several `db::ltm` emitters, and the defect is live +/// there (measured, pre-existing this branch): carrying the same `a,b` element +/// into an iterated-projection-feeder agg +/// (`x[state] = 1 + SUM(matrix[state,*] * frac[state])`) emits the agg->target +/// half as `$⁚ltm⁚link_score⁚$⁚ltm⁚agg⁚0[a]→x[a,b]` -- the agg's slot subscript +/// lost the `,b`, so the two halves of one agg name DIFFERENT variables and the +/// co-source row degrades to the delta-ratio fallback, while the comma-free +/// `s2` control carries the real partial. The arity guards in +/// `qualify_element_csv`/`target_elem_by_dim_for` are why the outcome is a +/// wrong/degraded score rather than a phantom edge. Sweeping the remaining +/// `split(',')` sites (link_scores.rs x8, loops.rs x1) onto structured parts is +/// its own change; until then the invariant to hold in NEW code is: never +/// serialize a coordinate tuple you will re-split. +#[test] +fn a_comma_bearing_element_name_survives_the_mapped_projection() { + let project = TestProject::new("comma_elem") + .named_dimension_with_element_mapping( + "state", + &["a,b", "s2"], + "region", + &[("a,b", "r2"), ("s2", "r1")], + ) + .named_dimension("region", &["r1", "r2"]) + .array_with_ranges("x[region]", vec![("r1", "10"), ("r2", "20")]) + .array_aux("target[state]", "x[region]"); + + // The executed read, so the edges below are checked against behaviour + // rather than against the derivation that produces them. + let results = project.run_vm().expect("must compile and run"); + assert_eq!(*results["target[a,b]"].last().expect("series"), 20.0); + assert_eq!(*results["target[s2]"].last().expect("series"), 10.0); + + assert_eq!( + edges_from(&project, "x["), + vec![ + "x[r1] -> target[s2]".to_string(), + "x[r2] -> target[a,b]".to_string(), + ], + "the comma element must stay ONE coordinate: splitting it drops \ + x[r2] -> target[a,b] and mints an edge to a target that does not exist" + ); +} + +/// The two derivations agree, on a model with a LOOP through the shape. +/// +/// The link-score NAMES already came out diagonal (they are projected from one +/// target element by `ltm_augment::per_element_row_for_target`), so a test that +/// only checked them would have been green throughout. The claim worth pinning +/// is that the element graph now names the same rows -- and the loop count is +/// how that becomes visible: an element loop is built out of element edges, so +/// six phantom edges through `matrix` minted six extra circuits, each carrying +/// a link score that does not exist. +#[test] +fn a_loop_through_two_mapped_axes_is_described_once_per_executed_read() { + use crate::db::{ + SimlinDb, model_ltm_variables, set_project_ltm_enabled, sync_from_datamodel_incremental, + }; + + // `lvl` breaks the algebraic loop: matrix -> target -> fb -> grow -> lvl -> + // matrix. At t=0 `lvl` is 0, so `target` still reads the fixture's own + // diagonal. + let project = two_mapped_axes_project("two_mapped_loop") + .array_with_ranges("base[Ra,Rb]", diagonal_matrix_cells()) + .array_stock("lvl[Ra,Rb]", "0", &["grow"], &[], None) + .array_flow("grow[Ra,Rb]", "fb * 0.01", None) + .array_aux("matrix[Ra,Rb]", "base[Ra,Rb] + lvl[Ra,Rb]") + .array_aux("target[State]", "matrix[Ra,Rb]") + .aux("fb", "SUM(target[*])", None); + + #[allow(deprecated)] + let circuits = { + use crate::db::{model_element_loop_circuits, sync_from_datamodel}; + let datamodel = project.build_datamodel(); + let db = SimlinDb::default(); + let sync = sync_from_datamodel(&db, &datamodel); + model_element_loop_circuits(&db, sync.models["main"].source, sync.project) + .circuits + .len() + }; + assert_eq!( + circuits, 3, + "one element loop per executed read; crossing the two axes minted nine" + ); + + let datamodel = project.build_datamodel(); + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &datamodel, None); + set_project_ltm_enabled(&mut db, sync.project, true); + let ltm = model_ltm_variables(&db, sync.models["main"].source_model, sync.project); + let mut scores: Vec<&str> = ltm + .vars + .iter() + .map(|v| v.name.as_str()) + .filter(|n| n.contains("link_score\u{205A}matrix[")) + .collect(); + scores.sort_unstable(); + assert_eq!( + scores, + [ + "$\u{205A}ltm\u{205A}link_score\u{205A}matrix[ra1,rb2]\u{2192}target[s3]", + "$\u{205A}ltm\u{205A}link_score\u{205A}matrix[ra2,rb3]\u{2192}target[s1]", + "$\u{205A}ltm\u{205A}link_score\u{205A}matrix[ra3,rb1]\u{2192}target[s2]", + ], + "the scores name the same three rows the edges do" + ); + + // And the loops are scored rather than dropped: every element loop through + // the shape has a score variable, which is what an edge with no matching + // score would have cost. + let loop_scores = ltm + .vars + .iter() + .filter(|v| v.name.contains("\u{205A}loop_score\u{205A}")) + .count(); + assert_eq!(loop_scores, 3, "every executed loop keeps a score"); +} diff --git a/src/simlin-engine/src/mdl/CLAUDE.md b/src/simlin-engine/src/mdl/CLAUDE.md index 1c4d39b36..f63050ac5 100644 --- a/src/simlin-engine/src/mdl/CLAUDE.md +++ b/src/simlin-engine/src/mdl/CLAUDE.md @@ -25,7 +25,7 @@ For design history and detailed implementation notes, see [docs/design/mdl-parse ### Conversion (`convert/`) - `mod.rs` -- Main conversion orchestration, group building, `DataProvider` threading -- `variables.rs` -- Variable type detection (stock/flow/aux) and building; EXCEPT default_equation handling, GET DIRECT resolution +- `variables.rs` -- Variable type detection (stock/flow/aux) and building; EXCEPT default_equation handling, GET DIRECT resolution. **A single apply-to-all MDL equation imports as one `Equation::ApplyToAll`**, not as N identical per-element slots: `build_variable_with_elements` expands every subscripted LHS to the cartesian product of its subscripts, so `y[DimA] = ` used to arrive as one slot per element all carrying the same ``. That is the same equation written N times, and it is the form in which a DIMENSION reference cannot resolve -- Vensim's `DimA` in an expression is the element's 1-based position, and a per-element slot has no active apply-to-all dimension for it -- so `y[DimA] = VECTOR ELM MAP(x[three], (DimA - 1))`, legal Vensim and correct through the XMILE reader, failed to compile through this one. The collapse is gated by `slots_are_one_apply_to_all` plus a `single_apply_to_all` precondition: the source must be ONE equation with no `:EXCEPT:` (overrides and defaults are genuinely per-element); it must not be EXTERNAL-DATA-backed -- the whole opaque `{GET ...}` family, not just the `GET DIRECT` calls this module can resolve (`external_data::is_external_data_placeholder`), because a resolvable one's slots agree only as a property of the spreadsheet while an UNRESOLVABLE one leaves an empty equation in every slot, and collapsing that yields `ApplyToAll(dims, "")` -- an `EmptyEquation` error where the `Arrayed` form imported cleanly, taking its readers with it since a variable with no parseable equation has no dimensions for a consumer's `SUM(v[Dim!])`; no slot may carry an INITIAL equation or a graphical function (`ApplyToAll` is `(dims, equation)` and has nowhere to put either -- collapsing an arrayed `ACTIVE INITIAL` silently drops its initial, and collapsing an arrayed `WITH LOOKUP` or a bare arrayed lookup table DELETES the table outright, since it lives in the slots and not on the variable); and the slots must cover the dimensions' full cartesian product (a partial cover is not an apply-to-all). Not every clause is independently load-bearing, and the mutation matrix says which: dropping the INITIAL, graphical-function or coverage clause each reds its own tests, while the `:EXCEPT:`/default/equation-text clauses are belt-and-braces -- `needs_substitution` is exactly `expanded_eqs.len() > 1 || has_except_eq`, so when `single_apply_to_all` holds no per-element substitution runs and slot text agreement is guaranteed by construction. The rule is a STRICTER form of the one `src/simlin-engine/tests/integration/mdl_equivalence.rs`'s `normalize_equation` applies to compare us against xmutil (which emits apply-to-all here): that normalizer collapses on slot agreement alone, ignoring source-equation count, coverage and external-data backing, so anything the importer collapses the harness would too and the comparison stays consistent. Tests: `convert/apply_to_all_tests.rs` (collapse rows and must-not-collapse rows, each named for why its slots differ) and the corpus gate `simulate::simulates_vector_mdl_genuine`, which runs `vector.mdl` against real-Vensim `vector.dat` -- a twin that did not exist, which is why the defect survived as long as the fixture. Corpus-wide effect measured with `src/simlin-engine/examples/mdl_compile_census.rs`: across 262 MDL files, one model moved fail -> ok and none the other way. - `stocks.rs` -- Stock/flow linking via is_all_plus_minus algorithm - `dimensions.rs` -- Dimension/subscript building with range expansion and `DimensionMapping` construction - `external_data.rs` -- GET DIRECT DATA/CONSTANTS/LOOKUPS/SUBSCRIPT resolution via `DataProvider` trait diff --git a/src/simlin-engine/src/mdl/convert/apply_to_all_tests.rs b/src/simlin-engine/src/mdl/convert/apply_to_all_tests.rs new file mode 100644 index 000000000..03aa3c4a5 --- /dev/null +++ b/src/simlin-engine/src/mdl/convert/apply_to_all_tests.rs @@ -0,0 +1,358 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! A single apply-to-all MDL equation must import as an `Equation::ApplyToAll`, +//! not as N identical per-element slots. +//! +//! `build_variable_with_elements` expands every subscripted LHS to the cartesian +//! product of its subscripts' elements, so `y[DimA] = ` became three +//! `Arrayed` slots all carrying the SAME `` text. Where the right-hand side +//! mentions the dimension itself -- Vensim's `DimA` is the element's 1-based +//! position -- that text has no active apply-to-all dimension to resolve against +//! once it sits in a per-element slot, so the variable failed to compile. Any +//! dimension-position arithmetic hits it, not just the `VECTOR ELM MAP` offset +//! that surfaced it. +//! +//! The rule implemented is the one the MDL equivalence harness already applies +//! when comparing against xmutil (`mdl_equivalence::normalize_equation`): slots +//! that are identical in equation text, initial text and graphical function ARE +//! an apply-to-all, so collapsing them loses nothing. The tests below are in two +//! halves -- shapes that must collapse, and shapes that must NOT, each named for +//! the reason it is per-element. + +use crate::datamodel::Equation; + +/// Import `mdl` and return the named variable's equation. +fn equation_of(mdl: &str, ident: &str) -> Equation { + let project = super::convert_mdl(mdl).expect("conversion should succeed"); + project.models[0] + .variables + .iter() + .find(|v| v.get_ident() == ident) + .unwrap_or_else(|| panic!("no variable {ident}")) + .get_equation() + .cloned() + .unwrap_or_else(|| panic!("{ident} has no equation")) +} + +/// Import `mdl`, simulate, and return the named element series' final values in +/// the order given. +fn finals(mdl: &str, keys: &[&str]) -> Vec { + let project = crate::open_vensim(mdl).expect("import"); + let tp = crate::test_common::TestProject::from_datamodel(project); + let all = tp.run_vm().expect("model must compile and run"); + keys.iter() + .map(|k| { + *all.get(*k) + .unwrap_or_else(|| panic!("no series {k}")) + .last() + .expect("empty series") + }) + .collect() +} + +const PREAMBLE: &str = "{UTF-8}\nDimA: A1, A2, A3 ~~|\nDimB: B1, B2 ~~|\n\ + DimX: one, two, three, four, five ~~|\nSubX: two, three, four ~~|\n\ + x[DimX] = 1, 2, 3, 4, 5 ~~|\n"; +const EPILOGUE: &str = "INITIAL TIME = 0 ~~|\nFINAL TIME = 1 ~~|\nTIME STEP = 1 ~~|\n\ + SAVEPER = TIME STEP ~~|\n"; + +fn model(body: &str) -> String { + format!("{PREAMBLE}{body}{EPILOGUE}") +} + +// =========================================================================== +// Must collapse to ApplyToAll +// =========================================================================== + +/// The shape that surfaced this: `vector.mdl`'s `y`, spelled byte-identically. +/// +/// The XMILE twin of this equation has always compiled and produced `3,4,5` +/// (real-Vensim ground truth, `test/sdeverywhere/models/vector/vector.dat`); +/// only the MDL import path failed, which is why the corpus never caught it -- +/// the `vector` fixture runs the XMILE file. +#[test] +fn a_dimension_position_offset_imports_as_apply_to_all() { + let mdl = model("y[DimA] = VECTOR ELM MAP(x[three], (DimA - 1)) ~~|\n"); + assert!( + matches!(equation_of(&mdl, "y"), Equation::ApplyToAll(_, _)), + "a single apply-to-all equation must not be exploded per element" + ); + assert_eq!( + finals(&mdl, &["y[a1]", "y[a2]", "y[a3]"]), + vec![3.0, 4.0, 5.0] + ); +} + +/// The same defect with no builtin involved: a dimension position used as plain +/// arithmetic. This is the row that shows the bug is about dimension references, +/// not about `VECTOR ELM MAP`. +#[test] +fn a_dimension_position_in_plain_arithmetic_imports_as_apply_to_all() { + let mdl = model("z[DimA] = 10 * (DimA - 1) ~~|\n"); + assert!(matches!(equation_of(&mdl, "z"), Equation::ApplyToAll(_, _))); + assert_eq!( + finals(&mdl, &["z[a1]", "z[a2]", "z[a3]"]), + vec![0.0, 10.0, 20.0] + ); +} + +/// Two dimensions at once: each position must resolve against its own axis. +#[test] +fn dimension_positions_on_two_axes_import_as_apply_to_all() { + let mdl = model("m[DimA,DimB] = 10 * (DimA - 1) + (DimB - 1) ~~|\n"); + assert!(matches!(equation_of(&mdl, "m"), Equation::ApplyToAll(_, _))); + assert_eq!( + finals( + &mdl, + &[ + "m[a1,b1]", "m[a1,b2]", "m[a2,b1]", "m[a2,b2]", "m[a3,b1]", "m[a3,b2]" + ] + ), + vec![0.0, 1.0, 10.0, 11.0, 20.0, 21.0] + ); +} + +/// A plain apply-to-all with no dimension reference at all. It compiled before +/// (the repeated text resolves fine per element), so this row is about the +/// STRUCTURE: the faithful translation of one MDL equation is one equation, and +/// exploding it bloats every imported arrayed model. +#[test] +fn an_ordinary_apply_to_all_equation_stays_one_equation() { + let mdl = model("c[DimA] = x[three] * 2 ~~|\n"); + match equation_of(&mdl, "c") { + Equation::ApplyToAll(dims, eqn) => { + assert_eq!(dims, vec!["DimA".to_string()]); + assert!(eqn.contains("three"), "unexpected equation text: {eqn}"); + } + other => panic!("expected ApplyToAll, got {other:?}"), + } + assert_eq!( + finals(&mdl, &["c[a1]", "c[a2]", "c[a3]"]), + vec![6.0, 6.0, 6.0] + ); +} + +/// A subrange LHS covering its own dimension collapses too -- the elements are +/// the full extent of `SubX`, so nothing is lost. +#[test] +fn a_subrange_apply_to_all_collapses_over_its_own_dimension() { + let mdl = model("s[SubX] = 10 * (SubX - 1) ~~|\n"); + assert!(matches!(equation_of(&mdl, "s"), Equation::ApplyToAll(_, _))); +} + +// =========================================================================== +// Must NOT collapse -- each row names why its slots genuinely differ +// =========================================================================== + +/// Element-specific equations: three different right-hand sides. +#[test] +fn per_element_equations_stay_arrayed() { + let mdl = model("p[A1] = 1 ~~|\np[A2] = 2 ~~|\np[A3] = 3 ~~|\n"); + match equation_of(&mdl, "p") { + Equation::Arrayed(_, elements, _, _) => assert_eq!(elements.len(), 3), + other => panic!("expected Arrayed, got {other:?}"), + } +} + +/// An `:EXCEPT:` equation: the excepted element differs from the default, and +/// the `Arrayed` form is what carries the default text. +#[test] +fn an_except_equation_stays_arrayed() { + let mdl = model("e[DimA] :EXCEPT: [A2] = 7 ~~|\ne[A2] = 99 ~~|\n"); + match equation_of(&mdl, "e") { + Equation::Arrayed(_, elements, _, _) => { + assert!(!elements.is_empty(), "EXCEPT must keep its element slots") + } + other => panic!("expected Arrayed, got {other:?}"), + } +} + +/// A subscripted numeric list: each element has its own value, so the slots +/// differ by construction. This is the commonest arrayed shape in the corpus and +/// the one a careless collapse would flatten to its first value. +#[test] +fn a_numeric_element_list_stays_arrayed() { + let mdl = model("n[DimA] = 4, 5, 6 ~~|\n"); + match equation_of(&mdl, "n") { + Equation::Arrayed(_, elements, _, _) => assert_eq!(elements.len(), 3), + other => panic!("expected Arrayed, got {other:?}"), + } + assert_eq!( + finals(&mdl, &["n[a1]", "n[a2]", "n[a3]"]), + vec![4.0, 5.0, 6.0] + ); +} + +/// A single apply-to-all equation carrying a GRAPHICAL FUNCTION stays arrayed. +/// +/// This is the one clause of `slots_are_one_apply_to_all` that blocks a shape +/// the source-shape clauses let through: `z[DimA] = WITH LOOKUP(...)` is ONE +/// equation, has no `:EXCEPT:`, no default, no INITIAL and no `GET DIRECT`, and +/// covers `DimA` fully -- so `single_apply_to_all` is true and only `gf.is_none()` +/// stands between it and a collapse. `Equation::ApplyToAll` is `(dims, equation)` +/// and has nowhere to put a table, and the table does NOT ride the variable here +/// (`Aux::gf` is `None`; it lives in the slots), so collapsing DELETES it: +/// measured, dropping that one condition turns this into +/// `ApplyToAll(["DimA"], "TIME")` -- a variable that was a lookup becomes plain +/// `= TIME`, with the whole suite green. +/// +/// The bare arrayed lookup table is the same exposure through a different MDL +/// form, and it degrades further: its slots hold an EMPTY equation plus the +/// table, so a collapse yields `ApplyToAll(["DimA"], "")` -- an empty equation +/// with no data at all. +#[test] +fn an_arrayed_graphical_function_stays_arrayed() { + let with_lookup = model("z[DimA] = WITH LOOKUP(Time, ((0,0.5),(1,1.36),(2,0.8))) ~~|\n"); + match equation_of(&with_lookup, "z") { + Equation::Arrayed(_, elements, _, _) => { + assert_eq!(elements.len(), 3); + assert!( + elements + .iter() + .all(|(_, eq, _, gf)| eq == "TIME" && gf.is_some()), + "every slot keeps the WITH LOOKUP table" + ); + } + other => panic!("expected Arrayed, got {other:?}"), + } + + let bare_table = model( + "t[DimA]( [(0,0)-(10,10)],(0,0),(5,5),(10,10) ) ~~|\nw[DimA] = LOOKUP(t[DimA], 5) ~~|\n", + ); + match equation_of(&bare_table, "t") { + Equation::Arrayed(_, elements, _, _) => { + assert_eq!(elements.len(), 3); + assert!( + elements + .iter() + .all(|(_, eq, _, gf)| eq.is_empty() && gf.is_some()), + "a lookup-only holder keeps its table in every slot" + ); + } + other => panic!("expected Arrayed, got {other:?}"), + } + // Its consumer is an ordinary apply-to-all and DOES collapse, which is what + // makes the pair a discriminator rather than a blanket "tables stay arrayed". + match equation_of(&bare_table, "w") { + Equation::ApplyToAll(_, eq) => assert_eq!(eq, "LOOKUP(t[DimA], 5)"), + other => panic!("expected ApplyToAll, got {other:?}"), + } +} + +/// Two apply-to-all equations over DIFFERENT subranges of one dimension -- +/// legal MDL, and the shape that says the collapse keys on the SOURCE equation +/// count rather than on the expanded slots. +/// +/// `q[SubLo] = 7` + `q[SubHi] = 7` jointly cover `DimX` and every expanded slot +/// agrees, so slot agreement and full coverage BOTH hold; only +/// `expanded_eqs.len() == 1` refuses it. Two equations are two equations, and +/// merging them would rewrite the model's structure -- the sibling row with +/// different right-hand sides is the same shape where merging would also be +/// numerically wrong, and it must land per element. +#[test] +fn apply_to_all_equations_over_different_subranges_stay_arrayed() { + let same = model( + "SubLo: one, two ~~|\nSubHi: three, four, five ~~|\nq[SubLo] = 7 ~~|\nq[SubHi] = 7 ~~|\n", + ); + match equation_of(&same, "q") { + Equation::Arrayed(dims, elements, _, _) => { + assert_eq!( + dims, + ["DimX"], + "the two subranges normalize to their parent" + ); + assert_eq!(elements.len(), 5); + } + other => panic!("expected Arrayed, got {other:?}"), + } + + let differing = model( + "SubLo: one, two ~~|\nSubHi: three, four, five ~~|\nr[SubLo] = 7 ~~|\nr[SubHi] = 9 ~~|\n", + ); + match equation_of(&differing, "r") { + Equation::Arrayed(_, elements, _, _) => assert_eq!(elements.len(), 5), + other => panic!("expected Arrayed, got {other:?}"), + } + assert_eq!( + finals( + &differing, + &["r[one]", "r[two]", "r[three]", "r[four]", "r[five]"] + ), + vec![7.0, 7.0, 9.0, 9.0, 9.0] + ); +} + +// =========================================================================== +// Apply-to-all intent is read off the SOURCE SPELLING, not off coverage +// =========================================================================== + +/// A SINGLETON dimension makes coverage arithmetic vacuous: with `DimA: a1`, +/// an element-specific `x[a1] = 5` produces one slot, which IS the whole +/// cartesian product. Coverage alone therefore cannot tell it from the +/// apply-to-all `x[DimA] = 5`, and the collapse silently rewrote the first into +/// the second. +/// +/// Two things are lost by that. The writer re-renders it as `x[DimA] = 5`, so +/// the as-written intent does not survive a round trip; and a later dimension +/// edit in Simlin (adding `a2`) silently applies the equation to an element the +/// MDL source never defined. +/// +/// The gate therefore asks the SOURCE: every LHS subscript must name a +/// dimension. That is the property "this equation is apply-to-all" actually +/// means, and unlike coverage it does not depend on how many elements the +/// dimension happens to have. +#[test] +fn an_element_specific_equation_over_a_singleton_dimension_stays_arrayed() { + let mdl = "{UTF-8}\nDimA: a1 ~~|\nx[a1] = 5 ~~|\n\ + INITIAL TIME = 0 ~~|\nFINAL TIME = 1 ~~|\nTIME STEP = 1 ~~|\nSAVEPER = TIME STEP ~~|\n"; + match equation_of(mdl, "x") { + Equation::Arrayed(dims, elements, _, _) => { + assert_eq!(dims, vec!["DimA".to_string()]); + assert_eq!( + elements.len(), + 1, + "the a1 slot must survive: the source named the ELEMENT" + ); + assert_eq!(elements[0].0, "a1"); + } + other => panic!("expected Arrayed (element-specific source), got {other:?}"), + } +} + +/// The control that keeps the rule honest: over the SAME one-element dimension, +/// the apply-to-all spelling must still collapse. If it did not, the fix would +/// just be "never collapse a singleton", which is a different (and wrong) rule. +#[test] +fn an_apply_to_all_equation_over_a_singleton_dimension_still_collapses() { + let mdl = "{UTF-8}\nDimA: a1 ~~|\nx[DimA] = 5 ~~|\n\ + INITIAL TIME = 0 ~~|\nFINAL TIME = 1 ~~|\nTIME STEP = 1 ~~|\nSAVEPER = TIME STEP ~~|\n"; + match equation_of(mdl, "x") { + Equation::ApplyToAll(dims, eq) => { + assert_eq!(dims, vec!["DimA".to_string()]); + assert_eq!(eq, "5"); + } + other => panic!("expected ApplyToAll (range source), got {other:?}"), + } +} + +/// A MIXED spelling: one axis names a dimension, the other pins an element. +/// The element subscript blocks the collapse BY SPELLING. +/// +/// Coverage would block this one too (two slots against a 3x2 product), so it +/// is not on its own evidence for the spelling clause -- the singleton row above +/// is. It is here because it is the shape a reader expects to find, and because +/// it pins that the two axes are judged independently rather than by whether +/// ANY axis is a dimension. +#[test] +fn an_element_pinned_axis_blocks_the_collapse_by_spelling() { + let mdl = model("y[DimA,B2] = 7 ~~|\n"); + match equation_of(&mdl, "y") { + Equation::Arrayed(_, elements, _, _) => { + assert_eq!(elements.len(), 3, "one slot per DimA element, pinned at B2"); + assert!(elements.iter().all(|(key, _, _, _)| key.ends_with(",B2"))); + } + other => panic!("expected Arrayed (element-pinned axis), got {other:?}"), + } +} diff --git a/src/simlin-engine/src/mdl/convert/dimensions.rs b/src/simlin-engine/src/mdl/convert/dimensions.rs index 604d23a7f..12f7c0a0f 100644 --- a/src/simlin-engine/src/mdl/convert/dimensions.rs +++ b/src/simlin-engine/src/mdl/convert/dimensions.rs @@ -675,12 +675,17 @@ x[DimA] = 1 // Should be recognized and processed correctly if let Variable::Aux(a) = x { + // What this test is about is that the ALIAS resolved to `DimA` -- + // `expand_subscript` recognized it -- which the dimension list + // carries. A single apply-to-all equation is one equation, so the + // expansion is no longer visible as element slots (see + // `convert::apply_to_all_tests`). match &a.equation { - crate::datamodel::Equation::Arrayed(dims, elements, _default_eq, _) => { + crate::datamodel::Equation::ApplyToAll(dims, eq) => { assert_eq!(dims, &["DimA"]); - assert_eq!(elements.len(), 2); + assert_eq!(eq, "1"); } - other => panic!("Expected Arrayed (dimension expansion), got {:?}", other), + other => panic!("Expected ApplyToAll over the alias, got {:?}", other), } } else { panic!("Expected Aux variable"); diff --git a/src/simlin-engine/src/mdl/convert/external_data.rs b/src/simlin-engine/src/mdl/convert/external_data.rs index c6e2ff9ef..d8f2947c6 100644 --- a/src/simlin-engine/src/mdl/convert/external_data.rs +++ b/src/simlin-engine/src/mdl/convert/external_data.rs @@ -450,6 +450,43 @@ pub(super) fn is_get_direct_ref(expr_str: &str) -> bool { trimmed.starts_with("{GET DIRECT") } +/// Check if an expression string is any EXTERNAL-DATA placeholder the normalizer +/// wrapped in braces -- the `GET DIRECT` family plus `GET XLS` / `GET VDF` / +/// `GET DATA` / `GET 123`. +/// +/// Deliberately broader than [`is_get_direct_ref`], which means "this is a call +/// this module can RESOLVE through a `DataProvider`" and must stay narrow (its +/// own test asserts `{GET XLS(...)}` is not one). This one means "this is an +/// opaque placeholder, not an equation", which is the question the apply-to-all +/// collapse has to ask: such a placeholder carries per-element data whether or +/// not we can read it today, and one that cannot be resolved leaves an EMPTY +/// equation behind. Collapsing that to `ApplyToAll(dims, "")` turns a variable +/// that imported cleanly into an `EmptyEquation` error -- and takes its readers +/// with it, since a variable with no parseable equation has no dimensions to +/// offer and `SUM(v[Dim!])` then fails as `CantSubscriptScalar`. Measured on +/// `v[DimA] := GET XLS DATA(...)` with no provider configured: clean import +/// before, two errors after. +/// +/// The family list mirrors `mdl::writer::is_data_equation`, which makes the same +/// opaque-placeholder distinction on the way out. +pub(super) fn is_external_data_placeholder(expr_str: &str) -> bool { + let s = expr_str.trim().trim_start_matches('{'); + [ + "GET DIRECT", + "GET XLS", + "GET VDF", + "GET DATA", + "GET 123", + "GET_DIRECT", + "GET_XLS", + "GET_VDF", + "GET_DATA", + "GET_123", + ] + .iter() + .any(|p| s.starts_with(p)) +} + /// Try to resolve a GET DIRECT reference from an expression string. /// Returns None if the string isn't a GET DIRECT reference or if no DataProvider /// is configured. diff --git a/src/simlin-engine/src/mdl/convert/mod.rs b/src/simlin-engine/src/mdl/convert/mod.rs index 4bcc7daf3..b057d9f37 100644 --- a/src/simlin-engine/src/mdl/convert/mod.rs +++ b/src/simlin-engine/src/mdl/convert/mod.rs @@ -7,6 +7,8 @@ //! This module converts parsed MDL AST items directly to `datamodel::Project`, //! bypassing the XMILE intermediate format. +#[cfg(test)] +mod apply_to_all_tests; mod dimensions; mod external_data; mod helpers; diff --git a/src/simlin-engine/src/mdl/convert/variables.rs b/src/simlin-engine/src/mdl/convert/variables.rs index 01249d006..c644e25aa 100644 --- a/src/simlin-engine/src/mdl/convert/variables.rs +++ b/src/simlin-engine/src/mdl/convert/variables.rs @@ -302,6 +302,66 @@ impl<'input> ConversionContext<'input> { /// /// NumberList and TabbedArray equations are excluded - they have special handling /// in build_equation that handles their multi-value RHS correctly. + /// Does this raw LHS subscript name a subscript RANGE (a dimension or a + /// subrange) rather than a single element? + /// + /// The one place that question is answered, because two answers would + /// diverge: the per-element `element_offsets` computation needs it to skip + /// pinned axes, and [`Self::build_variable_with_elements`]'s collapse gate + /// needs it to read apply-to-all intent off the source spelling. A subrange + /// is registered in `dimension_elements` like any other dimension, so both + /// qualify. + fn subscript_names_a_dimension(&self, subscript: &str) -> bool { + self.dimension_elements + .contains_key(&canonical_name(subscript)) + } + + /// True when these per-element slots ARE a single apply-to-all equation: + /// they cover the dimensions' full cartesian product and agree on equation + /// text, initial text and graphical function. + /// + /// Both halves are load-bearing. Agreement is what makes the collapse + /// lossless -- slots that differ carry per-element information (a numeric + /// list, `:EXCEPT:` overrides, or the per-element row/col offsets + /// `external_data::adjust_call_for_element` bakes into an arrayed + /// `GET DIRECT`, which is why those keep their slots). Full coverage is what + /// makes it faithful: a partial cover (some elements of the dimension having + /// no equation at all) is not an apply-to-all, and turning it into one would + /// invent equations for the missing elements. + fn slots_are_one_apply_to_all( + &self, + dims: &[String], + elements: &[(String, String, Option, Option)], + ) -> bool { + let Some((_, first_eq, first_initial, _)) = elements.first() else { + return false; + }; + // `Equation::ApplyToAll` is `(dims, equation)` and has nowhere to put an + // INITIAL equation or a graphical function -- an `Arrayed` slot carries + // both. So agreement is not enough: a variable whose slots share a + // non-`None` initial (an arrayed `ACTIVE INITIAL`) must keep its slots, + // or the collapse silently drops the initial and the variable starts the + // run from its active equation instead. + if first_initial.is_some() { + return false; + } + let agree = elements + .iter() + .all(|(_, eq, initial, gf)| eq == first_eq && initial.is_none() && gf.is_none()); + if !agree { + return false; + } + let expected: usize = dims + .iter() + .map(|d| { + self.dimension_elements + .get(&canonical_name(d)) + .map_or(0, |elems| elems.len()) + }) + .product(); + expected > 0 && elements.len() == expected + } + fn build_variable_with_elements( &self, name: &str, @@ -517,10 +577,7 @@ impl<'input> ConversionContext<'input> { let element_offsets: Vec = element_parts .iter() .zip(exp_eq.lhs_subscripts.iter()) - .filter(|(_, sub)| { - let canonical = canonical_name(sub); - self.dimension_elements.contains_key(&canonical) - }) + .filter(|(_, sub)| self.subscript_names_a_dimension(sub)) .map(|(elem, sub)| self.element_index_in_dimension(elem, sub).unwrap_or(0)) .collect(); @@ -560,12 +617,84 @@ impl<'input> ConversionContext<'input> { // When EXCEPT is the sole source of elements, excepted elements should // remain at 0 (undefined) rather than receiving the default. let has_except_default = has_except_eq && has_non_except_eq; - let equation = Equation::Arrayed( - formatted_dims.clone(), - elements, - default_equation, - has_except_default, - ); + // Was the MDL source a SINGLE apply-to-all equation? That -- not + // whether the expanded slots happen to agree -- is what licenses + // collapsing, and the difference is not academic: an arrayed + // `GET DIRECT CONSTANTS` reads its per-element values from a file, so + // its slots agreeing is a property of the DATA, and collapsing on that + // basis would make the imported structure change when the spreadsheet + // does. Those keep their slots, as do EXCEPT equations and + // element-specific overrides. + // + // The external-data test is the WHOLE opaque-placeholder family + // (`is_external_data_placeholder`), not just the `GET DIRECT` calls this + // module can resolve. An UNRESOLVABLE one -- `GET XLS DATA` with no + // `DataProvider`, the commonest shape in the checked-in corpus -- leaves + // an empty equation in every slot, and collapsing that yields + // `ApplyToAll(dims, "")`, which is an `EmptyEquation` error where the + // `Arrayed` form imported cleanly. It costs the readers too: a variable + // with no parseable equation has no dimensions, so a consumer's + // `SUM(v[Dim!])` then fails as `CantSubscriptScalar`. Measured on four + // corpus models (`groupon 1-3`, `get_with_missing_values_xlsx`), which + // gained failing variables until this test was widened. + // + // The gate also asks the SOURCE SPELLING, not just the expanded slots: + // every LHS subscript must name a subscript RANGE. Coverage arithmetic + // cannot substitute for that, because a SINGLETON dimension makes it + // vacuous -- with `DimA: a1`, the element-specific `x[a1] = 5` produces + // one slot, which is the whole cartesian product, and collapsed to + // `ApplyToAll([DimA], "5")`. That loses the source's meaning twice over: + // the writer re-renders it as `x[DimA] = 5`, and a later dimension edit + // adding `a2` silently extends an equation the MDL never wrote for it. + // Asking the spelling is also simply what "apply-to-all" means, so the + // other blocks (a pinned axis, mixed spellings) become principled rather + // than incidental consequences of counting elements. + let single_apply_to_all = expanded_eqs.len() == 1 + && !has_except_eq + && default_equation.is_none() + && expanded_eqs.iter().all(|e| { + e.lhs_subscripts + .iter() + .all(|sub| self.subscript_names_a_dimension(sub)) + }) + && !expanded_eqs.iter().any(|e| { + let eq_str = match &e.eq.equation { + MdlEquation::Regular(_, expr) | MdlEquation::Data(_, Some(expr)) => { + self.formatter.format_expr(expr) + } + // A `Data` equation with no expression is Vensim's + // implicit-data form: opaque for the same reason. + MdlEquation::Data(_, None) => return true, + _ => return false, + }; + super::external_data::is_external_data_placeholder(&eq_str) + }); + let equation = + if single_apply_to_all && self.slots_are_one_apply_to_all(&formatted_dims, &elements) { + // One MDL apply-to-all equation is ONE equation. Every subscripted + // LHS is expanded to the cartesian product of its subscripts above, + // so `y[DimA] = ` arrived here as N slots all carrying the same + // ``; that is the same equation written N times, and writing it + // once is both the faithful translation and the only form a + // dimension reference survives. Vensim's `DimA` in an expression is + // the element's 1-based POSITION, and a per-element slot has no + // active apply-to-all dimension for it to resolve against -- so + // `y[DimA] = VECTOR ELM MAP(x[three], (DimA - 1))`, legal Vensim and + // correct through our XMILE reader, failed to compile through this + // one. Collapsing loses nothing by construction (see + // `slots_are_one_apply_to_all` for the two things `ApplyToAll` + // cannot carry), and it is the rule the MDL equivalence harness + // already applies to compare us with xmutil, which emits + // apply-to-all here. + Equation::ApplyToAll(formatted_dims.clone(), elements[0].1.clone()) + } else { + Equation::Arrayed( + formatted_dims.clone(), + elements, + default_equation, + has_except_default, + ) + }; // Build the variable let ident = quoted_space_to_underbar(name); @@ -1604,7 +1733,7 @@ V300\n\ } #[test] - fn test_subscripted_equation_expands_to_arrayed() { + fn test_subscripted_apply_to_all_equation_stays_one_equation() { // Subscripted equations with dimension subscripts are expanded to Arrayed // so that element-specific overrides can be properly merged. let mdl = "DimA: a1, a2, a3 @@ -1626,17 +1755,17 @@ x[DimA] = 5 assert!(x.is_some(), "Should have x variable"); if let Some(Variable::Aux(a)) = x { + // One apply-to-all MDL equation is ONE equation: `x[DimA] = 5` + // used to arrive here as three identical `"5"` slots, which is the + // same equation written three times and is the form in which a + // dimension reference in the RHS cannot resolve (see + // `apply_to_all_tests`). match &a.equation { - Equation::Arrayed(dims, elements, _default_eq, _) => { + Equation::ApplyToAll(dims, eq) => { assert_eq!(dims, &["DimA"]); - assert_eq!(elements.len(), 3); - // All elements have the same equation "5" - for (key, eq, _, _) in elements { - assert!(["a1", "a2", "a3"].contains(&key.as_str())); - assert_eq!(eq, "5"); - } + assert_eq!(eq, "5"); } - other => panic!("Expected Arrayed equation, got {:?}", other), + other => panic!("Expected ApplyToAll equation, got {:?}", other), } } else { panic!("Expected Aux variable"); @@ -2588,17 +2717,14 @@ x[DimA] = y[DimA] * 2 if let Some(Variable::Aux(a)) = x { match &a.equation { - Equation::Arrayed(dims, elements, _default_eq, _) => { + Equation::ApplyToAll(dims, eq) => { assert_eq!(dims, &["DimA"]); - assert_eq!(elements.len(), 2); - for (_, eq, _, _) in elements { - assert_eq!( - eq, "y[DimA] * 2", - "Apply-to-all should preserve dimension names" - ); - } + assert_eq!( + eq, "y[DimA] * 2", + "Apply-to-all should preserve dimension names" + ); } - other => panic!("Expected Arrayed equation, got {:?}", other), + other => panic!("Expected ApplyToAll equation, got {:?}", other), } } else { panic!("Expected Aux variable"); diff --git a/src/simlin-engine/src/mdl/writer.rs b/src/simlin-engine/src/mdl/writer.rs index 07f7f12e1..2b1da4a2e 100644 --- a/src/simlin-engine/src/mdl/writer.rs +++ b/src/simlin-engine/src/mdl/writer.rs @@ -4609,6 +4609,12 @@ mod tests; #[path = "writer_lossiness_tests.rs"] mod lossiness_tests; +// Split out of writer_tests.rs for the same per-file line cap; the sketch +// element/connector/section serialization block. +#[cfg(test)] +#[path = "writer_sketch_tests.rs"] +mod sketch_tests; + // Property-based tests (own file per the per-file line cap; see the module's // header for the generator design and fixpoint conventions). #[cfg(test)] diff --git a/src/simlin-engine/src/mdl/writer_sketch_tests.rs b/src/simlin-engine/src/mdl/writer_sketch_tests.rs new file mode 100644 index 000000000..860dbde05 --- /dev/null +++ b/src/simlin-engine/src/mdl/writer_sketch_tests.rs @@ -0,0 +1,798 @@ +// Copyright 2026 The Simlin Authors. All rights reserved. +// Use of this source code is governed by the Apache License, +// Version 2.0, that can be found in the LICENSE file. + +//! MDL writer sketch-section tests (element, connector, and whole-section +//! serialization -- the former Phase 5 Tasks 1-3 block). Split out of +//! `writer_tests.rs` to stay under the per-file line cap (GH #645); shares the +//! `make_*` fixture helpers from the sibling `tests` module. + +use super::tests::{make_aux, make_project}; +use super::*; +use crate::datamodel::{self, ViewElement, view_element}; + +// ---- Phase 5 Task 1: Sketch element serialization (types 10, 11, 12) ---- + +#[test] +fn sketch_aux_element() { + let aux = view_element::Aux { + name: "Growth_Rate".to_string(), + uid: 1, + x: 100.0, + y: 200.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + }; + let mut buf = String::new(); + write_aux_element(&mut buf, &aux); + assert_eq!(buf, "10,1,Growth Rate,100,200,40,20,8,3,0,0,-1,0,0,0"); +} + +#[test] +fn sketch_stock_element() { + let stock = view_element::Stock { + name: "Population".to_string(), + uid: 2, + x: 300.0, + y: 150.0, + label_side: view_element::LabelSide::Top, + compat: None, + }; + let mut buf = String::new(); + write_stock_element(&mut buf, &stock); + assert_eq!(buf, "10,2,Population,300,150,40,20,3,3,0,0,0,0,0,0"); +} + +#[test] +fn sketch_flow_element_produces_valve_and_variable() { + let flow = view_element::Flow { + name: "Infection_Rate".to_string(), + uid: 6, + x: 295.0, + y: 191.0, + label_side: view_element::LabelSide::Bottom, + points: vec![], + compat: None, + label_compat: None, + }; + let mut buf = String::new(); + let valve_uids = HashMap::from([(6, 100)]); + let mut next_connector_uid = 200; + write_flow_element( + &mut buf, + &flow, + &valve_uids, + &HashSet::new(), + &mut next_connector_uid, + ); + // No flow points, so no pipe connectors; valve and label follow + assert!(buf.contains("11,100,0,295,191,6,8,34,3,0,0,1,0,0,0")); + // Label sits 20px below the valve (y 191 -> 211); its box is sized to the + // text ("Infection Rate" -> 14 chars * 6px = 84 wide, single-line height 11). + assert!(buf.contains("10,6,Infection Rate,295,211,84,11,40,3,0,0,-1,0,0,0")); +} + +#[test] +fn sketch_flow_element_emits_pipe_connectors_from_flow_points() { + let flow = view_element::Flow { + name: "Infection_Rate".to_string(), + uid: 6, + x: 150.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + points: vec![ + view_element::FlowPoint { + x: 100.0, + y: 100.0, + attached_to_uid: Some(1), + }, + view_element::FlowPoint { + x: 200.0, + y: 100.0, + attached_to_uid: Some(2), + }, + ], + compat: None, + label_compat: None, + }; + let mut buf = String::new(); + let valve_uids = HashMap::from([(6, 100)]); + let mut next_connector_uid = 200; + write_flow_element( + &mut buf, + &flow, + &valve_uids, + &HashSet::new(), + &mut next_connector_uid, + ); + + let connector_lines: Vec<&str> = buf.lines().filter(|line| line.starts_with("1,")).collect(); + assert_eq!( + connector_lines.len(), + 2, + "Expected two type-1 connector lines for flow endpoints: {}", + buf + ); + assert!( + connector_lines.iter().any(|line| line.contains(",100,1,")), + "Expected connector from valve uid 100 to endpoint uid 1: {}", + buf + ); + assert!( + connector_lines.iter().any(|line| line.contains(",100,2,")), + "Expected connector from valve uid 100 to endpoint uid 2: {}", + buf + ); +} + +#[test] +fn sketch_flow_element_derives_stock_connector_points_from_takeoffs() { + let flow = view_element::Flow { + name: "Infection_Rate".to_string(), + uid: 6, + x: 150.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + points: vec![ + view_element::FlowPoint { + x: 122.5, + y: 100.0, + attached_to_uid: Some(1), + }, + view_element::FlowPoint { + x: 177.5, + y: 100.0, + attached_to_uid: Some(2), + }, + ], + compat: None, + label_compat: None, + }; + let mut buf = String::new(); + let valve_uids = HashMap::from([(6, 100)]); + let elem_positions = HashMap::from([(1, (100, 100)), (2, (200, 100))]); + let stock_uids = HashSet::from([1, 2]); + let mut next_connector_uid = 200; + write_flow_element_with_context( + &mut buf, + &flow, + &valve_uids, + &HashSet::new(), + &mut next_connector_uid, + SketchTransform::identity(), + &elem_positions, + &stock_uids, + None, + ); + + // Sink pipe (last point) carries direction 4; source pipe (first point) + // carries direction 100 -- the endpoint *role*, not stock-vs-cloud. + assert!( + buf.contains("1,200,100,2,4,0,0,22,0,0,0,-1--1--1,,1|(200,100)|"), + "sink pipe connector should be reconstructed from the stock center: {buf}" + ); + assert!( + buf.contains("1,201,100,1,100,0,0,22,0,0,0,-1--1--1,,1|(100,100)|"), + "source pipe connector should be reconstructed from the stock center: {buf}" + ); + // Canonical bottom-label fallback: 20px below the valve (y 100 -> 120), + // box sized to the text ("Infection Rate" -> 14 chars * 6px = 84, h 11). + assert!( + buf.contains("10,6,Infection Rate,150,120,84,11,40,3,0,0,-1,0,0,0"), + "flow label should fall back to the canonical bottom label position: {buf}" + ); +} + +/// An outflow into a sink cloud: the cloud-side pipe (the sink, last point) +/// must carry direction 4 and the stock-side pipe (the source, first point) +/// direction 100 -- the reverse of "stock => 4, cloud => 100", which is what +/// Vensim's own outflow-to-cloud sketches do. +#[test] +fn sketch_flow_outflow_to_cloud_uses_role_based_direction_flags() { + let flow = view_element::Flow { + name: "Drain".to_string(), + uid: 6, + x: 150.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + points: vec![ + // Source: a stock at the left. + view_element::FlowPoint { + x: 122.5, + y: 100.0, + attached_to_uid: Some(1), + }, + // Sink: a cloud at the right. + view_element::FlowPoint { + x: 200.0, + y: 100.0, + attached_to_uid: Some(2), + }, + ], + compat: None, + label_compat: None, + }; + let mut buf = String::new(); + let valve_uids = HashMap::from([(6, 100)]); + let elem_positions = HashMap::from([(1, (100, 100)), (2, (200, 100))]); + let stock_uids = HashSet::from([1]); // only uid 1 is a stock; uid 2 is the cloud + let mut next_connector_uid = 200; + write_flow_element_with_context( + &mut buf, + &flow, + &valve_uids, + &HashSet::new(), + &mut next_connector_uid, + SketchTransform::identity(), + &elem_positions, + &stock_uids, + None, + ); + + assert!( + buf.contains("1,200,100,2,4,0,0,22,0,0,0,-1--1--1,,1|(200,100)|"), + "sink cloud pipe should carry direction 4: {buf}" + ); + assert!( + buf.contains("1,201,100,1,100,0,0,22,0,0,0,-1--1--1,,1|(100,100)|"), + "source stock pipe should carry direction 100: {buf}" + ); +} + +#[test] +fn valve_uids_do_not_collide_with_existing_elements() { + // stock uid=1, flow uid=2 -> valve must NOT get uid=1 + let elements = vec![ + ViewElement::Stock(view_element::Stock { + name: "Population".to_string(), + uid: 1, + x: 100.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + }), + ViewElement::Flow(view_element::Flow { + name: "Birth_Rate".to_string(), + uid: 2, + x: 200.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + points: vec![], + compat: None, + label_compat: None, + }), + ]; + + let valve_uids = allocate_valve_uids(&elements); + // The valve for flow uid=2 must not equal 1 (stock's uid) + let valve_uid = valve_uids[&2]; + assert_ne!(valve_uid, 1, "Valve UID collides with stock UID"); + assert_ne!(valve_uid, 2, "Valve UID collides with flow UID"); +} + +#[test] +fn sketch_cloud_element() { + let cloud = view_element::Cloud { + uid: 7, + flow_uid: 6, + x: 479.0, + y: 235.0, + compat: None, + }; + let mut buf = String::new(); + write_cloud_element(&mut buf, &cloud); + assert_eq!(buf, "12,7,48,479,235,10,8,0,3,0,0,-1,0,0,0"); +} + +#[test] +fn sketch_alias_element() { + let alias = view_element::Alias { + uid: 10, + alias_of_uid: 1, + x: 200.0, + y: 300.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + }; + let mut name_map = HashMap::new(); + name_map.insert(1, "Growth_Rate"); + let mut buf = String::new(); + write_alias_element(&mut buf, &alias, &name_map); + assert!(buf.starts_with("10,10,Growth Rate,200,300,40,20,8,2,0,3,-1,0,0,0,")); + assert!(buf.contains("128-128-128")); +} + +#[test] +fn sketch_alias_element_offsets_stock_ghost_coordinates() { + let alias = view_element::Alias { + uid: 10, + alias_of_uid: 1, + x: 200.0, + y: 300.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + }; + let mut name_map = HashMap::new(); + name_map.insert(1, "Population"); + let mut buf = String::new(); + write_alias_element_with_context( + &mut buf, + &alias, + &name_map, + &HashSet::from([1]), + SketchTransform::identity(), + None, + ); + assert!( + buf.starts_with("10,10,Population,222,317,40,20,8,2,0,3,-1,0,0,0,"), + "stock ghosts should serialize using Vensim's stock-alias offset: {buf}" + ); +} + +// ---- Phase 5 Task 2: Connector serialization (type 1) ---- + +#[test] +fn sketch_link_straight() { + let link = view_element::Link { + uid: 3, + from_uid: 1, + to_uid: 2, + shape: LinkShape::Straight, + polarity: None, + }; + let mut positions = HashMap::new(); + positions.insert(1, (100, 100)); + positions.insert(2, (200, 200)); + let mut buf = String::new(); + write_link_element(&mut buf, &link, &positions, false); + // Straight => control point (0,0), field 9 = 64 (influence connector) + assert_eq!(buf, "1,3,1,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)|"); +} + +#[test] +fn sketch_link_with_polarity_symbol() { + let link = view_element::Link { + uid: 5, + from_uid: 1, + to_uid: 2, + shape: LinkShape::Straight, + polarity: Some(LinkPolarity::Positive), + }; + let positions = HashMap::new(); + let mut buf = String::new(); + write_link_element(&mut buf, &link, &positions, false); + // polarity=43 ('+'), field 9 = 64 + assert!(buf.contains(",0,0,43,0,0,64,0,")); +} + +#[test] +fn sketch_link_with_polarity_letter() { + let link = view_element::Link { + uid: 5, + from_uid: 1, + to_uid: 2, + shape: LinkShape::Straight, + polarity: Some(LinkPolarity::Positive), + }; + let positions = HashMap::new(); + let mut buf = String::new(); + write_link_element(&mut buf, &link, &positions, true); + // polarity=83 ('S' for lettered positive), field 9 = 64 + assert!(buf.contains(",0,0,83,0,0,64,0,")); +} + +#[test] +fn sketch_link_arc_produces_nonzero_control_point() { + let link = view_element::Link { + uid: 3, + from_uid: 1, + to_uid: 2, + shape: LinkShape::Arc(45.0), + polarity: None, + }; + let mut positions = HashMap::new(); + positions.insert(1, (100, 100)); + positions.insert(2, (200, 100)); + let mut buf = String::new(); + write_link_element(&mut buf, &link, &positions, false); + // Arc should produce a non-(0,0) control point + assert!( + !buf.contains("|(0,0)|"), + "arc should not produce (0,0) control point" + ); +} + +#[test] +fn sketch_link_with_field_hints_preserves_nonsemantic_flags() { + let link = view_element::Link { + uid: 3, + from_uid: 1, + to_uid: 2, + shape: LinkShape::Straight, + polarity: None, + }; + let positions = HashMap::from([(1, (100, 100)), (2, (200, 116)), (100, (200, 100))]); + let compat = view_element::LinkSketchCompat { + uid: 3, + field4: 1, + field10: 7, + }; + let mut buf = String::new(); + write_link_element_with_context( + &mut buf, + &link, + &positions, + false, + Some(&compat), + SketchTransform::identity(), + None, + ); + assert_eq!(buf, "1,3,1,2,1,0,0,0,0,64,7,-1--1--1,,1|(0,0)|"); +} + +#[test] +fn sketch_link_with_field_hints_still_uses_link_geometry() { + let link = view_element::Link { + uid: 3, + from_uid: 1, + to_uid: 2, + shape: LinkShape::Arc(45.0), + polarity: None, + }; + let positions = HashMap::from([(1, (110, 100)), (2, (210, 100))]); + // A recorded compat carries only field4/field10; the control point is always + // recomputed from the link's Arc angle and the current endpoint positions. + let compat = view_element::LinkSketchCompat { + uid: 3, + field4: 0, + field10: 0, + }; + let mut buf = String::new(); + write_link_element_with_context( + &mut buf, + &link, + &positions, + false, + Some(&compat), + SketchTransform::identity(), + None, + ); + let (ctrl_x, ctrl_y) = compute_control_point((110, 100), (210, 100), 45.0); + assert_eq!( + buf, + format!("1,3,1,2,0,0,0,0,0,64,0,-1--1--1,,1|({ctrl_x},{ctrl_y})|") + ); +} + +#[test] +fn sketch_link_multipoint_emits_all_points() { + let points = vec![ + view_element::FlowPoint { + x: 150.0, + y: 120.0, + attached_to_uid: None, + }, + view_element::FlowPoint { + x: 170.0, + y: 140.0, + attached_to_uid: None, + }, + view_element::FlowPoint { + x: 190.0, + y: 160.0, + attached_to_uid: None, + }, + ]; + let link = view_element::Link { + uid: 4, + from_uid: 1, + to_uid: 2, + shape: LinkShape::MultiPoint(points), + polarity: None, + }; + let mut positions = HashMap::new(); + positions.insert(1, (100, 100)); + positions.insert(2, (200, 200)); + let mut buf = String::new(); + write_link_element(&mut buf, &link, &positions, false); + assert!( + buf.contains("3|(150,120)|(170,140)|(190,160)|"), + "multipoint should emit all three points: {buf}" + ); +} + +// ---- Phase 5 Task 3: Complete sketch section assembly ---- + +#[test] +fn sketch_section_structure() { + let elements = vec![ + ViewElement::Stock(view_element::Stock { + name: "Population".to_string(), + uid: 1, + x: 100.0, + y: 100.0, + label_side: view_element::LabelSide::Top, + compat: None, + }), + ViewElement::Aux(view_element::Aux { + name: "Growth_Rate".to_string(), + uid: 2, + x: 200.0, + y: 200.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + }), + ViewElement::Link(view_element::Link { + uid: 3, + from_uid: 2, + to_uid: 1, + shape: LinkShape::Straight, + polarity: None, + }), + ]; + let sf = datamodel::StockFlow { + name: None, + elements, + view_box: Default::default(), + zoom: 1.0, + use_lettered_polarity: false, + font: None, + sketch_compat: None, + }; + let views = vec![View::StockFlow(sf)]; + + let mut writer = MdlWriter::new(); + writer.write_sketch_section(&views); + let output = writer.buf; + + // Header + assert!( + output.starts_with("V300 Do not put anything below this section"), + "should start with V300 header" + ); + // View title + assert!(output.contains("*View 1\n"), "should have view title"); + // Font line + assert!( + output.contains("$192-192-192"), + "should have font settings line" + ); + // Elements + assert!( + output.contains("10,1,Population,"), + "should have stock element" + ); + assert!( + output.contains("10,2,Growth Rate,"), + "should have aux element" + ); + assert!(output.contains("1,3,2,1,"), "should have link element"); + // Terminator + assert!( + output.ends_with("///---\\\\\\\n"), + "should end with sketch terminator" + ); +} + +#[test] +fn sketch_section_in_full_project() { + let var = make_aux("x", "1", None, ""); + let elements = vec![ViewElement::Aux(view_element::Aux { + name: "x".to_string(), + uid: 1, + x: 100.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + })]; + let model = datamodel::Model { + name: "default".to_owned(), + sim_specs: None, + variables: vec![var], + views: vec![View::StockFlow(datamodel::StockFlow { + name: None, + elements, + view_box: Default::default(), + zoom: 1.0, + use_lettered_polarity: false, + font: None, + sketch_compat: None, + })], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }; + let project = make_project(vec![model]); + + let result = crate::mdl::project_to_mdl(&project); + assert!(result.is_ok()); + let mdl = result.unwrap(); + + // The sketch section should appear after the equations terminator + let terminator_pos = mdl + .find("\\\\\\---/// Sketch information") + .expect("should have equations terminator"); + let v300_pos = mdl.find("V300").expect("should have V300 header"); + assert!( + terminator_pos < v300_pos, + "V300 should come after equations terminator" + ); + + // The sketch terminator should be at the end + assert!( + mdl.contains("///---\\\\\\"), + "should have sketch terminator" + ); +} + +#[test] +fn sketch_roundtrip_teacup() { + // Read teacup.mdl, parse to Project, write sketch section, verify structure + let mdl_contents = include_str!("../../../../test/test-models/samples/teacup/teacup.mdl"); + let project = + crate::mdl::parse_mdl(mdl_contents).expect("teacup.mdl should parse successfully"); + + let model = &project.models[0]; + assert!( + !model.views.is_empty(), + "teacup model should have at least one view" + ); + + // Write the sketch section + let mut writer = MdlWriter::new(); + writer.write_sketch_section(&model.views); + let output = writer.buf; + + // Verify structural elements: the teacup model should have stocks, auxes, + // flows (valve + attached variable), links, and clouds. + assert!(output.contains("V300"), "output should contain V300 header"); + assert!( + output.contains("*View 1"), + "output should contain view title" + ); + assert!( + output.contains("///---\\\\\\"), + "output should end with sketch terminator" + ); + + // The teacup model elements (after roundtrip through datamodel): + // Stock: Teacup_Temperature -> type 10 with shape=3 + // Aux: Heat_Loss_to_Room flow -> type 11 valve + type 10 attached + // Aux: Room_Temperature, Characteristic_Time -> type 10 + // Links -> type 1 + // Clouds -> type 12 + + // Count element types in output + let lines: Vec<&str> = output.lines().collect(); + let type10_count = lines.iter().filter(|l| l.starts_with("10,")).count(); + let type11_count = lines.iter().filter(|l| l.starts_with("11,")).count(); + let type12_count = lines.iter().filter(|l| l.starts_with("12,")).count(); + let type1_count = lines.iter().filter(|l| l.starts_with("1,")).count(); + + // Teacup has: 1 stock (Teacup_Temperature), 3 auxes (Heat_Loss_to_Room, + // Room_Temperature, Characteristic_Time), 1 flow (Heat_Loss_to_Room) + // which produces valve+variable, plus 1 cloud. + // The exact numbers depend on the MDL->datamodel conversion, but + // we should have a reasonable set of elements. + assert!( + type10_count >= 2, + "should have at least 2 type-10 elements (variables/stocks), got {type10_count}" + ); + assert!( + type11_count >= 1, + "should have at least 1 type-11 element (valve), got {type11_count}" + ); + assert!( + type12_count >= 1, + "should have at least 1 type-12 element (cloud/comment), got {type12_count}" + ); + assert!( + type1_count >= 1, + "should have at least 1 type-1 element (connector), got {type1_count}" + ); + // Verify no empty lines were introduced between elements + let element_lines: Vec<&&str> = lines + .iter() + .filter(|l| { + l.starts_with("10,") + || l.starts_with("11,") + || l.starts_with("12,") + || l.starts_with("1,") + }) + .collect(); + assert!( + !element_lines.is_empty(), + "should have sketch elements in output" + ); + + // Verify the output can be re-parsed as a valid sketch section + let reparsed = crate::mdl::view::parse_views(&output); + assert!( + reparsed.is_ok(), + "re-serialized sketch should parse: {:?}", + reparsed.err() + ); + let views = reparsed.unwrap(); + assert!( + !views.is_empty(), + "re-parsed sketch should have at least one view" + ); + + // Verify all expected element types are present after re-parse + let view = &views[0]; + let has_variable = view + .iter() + .any(|e| matches!(e, crate::mdl::view::VensimElement::Variable(_))); + let has_connector = view + .iter() + .any(|e| matches!(e, crate::mdl::view::VensimElement::Connector(_))); + assert!(has_variable, "re-parsed view should have variables"); + assert!(has_connector, "re-parsed view should have connectors"); +} + +#[test] +fn sketch_roundtrip_preserves_view_title() { + let mdl_contents = r#"x = 5 +~ ~| +\\\---/// Sketch information +V300 Do not put anything below this section - it will be ignored +*Overview +$192-192-192,0,Times New Roman|12||0-0-0|0-0-0|0-0-255|-1--1--1|-1--1--1|96,96,100,0 +10,1,x,100,100,40,20,8,3,0,0,-1,0,0,0 +///---\\\ +"#; + + let project = + crate::mdl::parse_mdl(mdl_contents).expect("source MDL should parse successfully"); + let mdl = crate::mdl::project_to_mdl(&project).expect("roundtrip MDL write should work"); + + assert!( + mdl.contains("*Overview\r\n"), + "Roundtrip should preserve original view title: {}", + mdl + ); +} + +#[test] +fn sketch_roundtrip_sanitizes_multiline_view_title() { + let var = make_aux("x", "5", Some("Units"), "A constant"); + let model = datamodel::Model { + name: "default".to_owned(), + sim_specs: None, + variables: vec![var], + views: vec![View::StockFlow(datamodel::StockFlow { + name: Some("Overview\r\nMain".to_owned()), + elements: vec![ViewElement::Aux(view_element::Aux { + name: "x".to_owned(), + uid: 1, + x: 100.0, + y: 100.0, + label_side: view_element::LabelSide::Bottom, + compat: None, + })], + view_box: Default::default(), + zoom: 1.0, + use_lettered_polarity: false, + font: None, + sketch_compat: None, + })], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }; + let project = make_project(vec![model]); + + let mdl = crate::mdl::project_to_mdl(&project).expect("MDL write should succeed"); + assert!( + mdl.contains("*Overview Main\r\n"), + "view title should be serialized as a single line: {mdl}", + ); + + let reparsed = crate::mdl::parse_mdl(&mdl).expect("written MDL should parse"); + let View::StockFlow(sf) = &reparsed.models[0].views[0]; + assert_eq!( + sf.name.as_deref(), + Some("Overview Main"), + "sanitized title should roundtrip through MDL", + ); +} diff --git a/src/simlin-engine/src/mdl/writer_tests.rs b/src/simlin-engine/src/mdl/writer_tests.rs index 053d86db5..63d8e45c9 100644 --- a/src/simlin-engine/src/mdl/writer_tests.rs +++ b/src/simlin-engine/src/mdl/writer_tests.rs @@ -2271,792 +2271,6 @@ fn equations_section_with_dimensions() { assert!(dim_pos < var_pos, "dimensions should come before variables"); } -// ---- Phase 5 Task 1: Sketch element serialization (types 10, 11, 12) ---- - -#[test] -fn sketch_aux_element() { - let aux = view_element::Aux { - name: "Growth_Rate".to_string(), - uid: 1, - x: 100.0, - y: 200.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - }; - let mut buf = String::new(); - write_aux_element(&mut buf, &aux); - assert_eq!(buf, "10,1,Growth Rate,100,200,40,20,8,3,0,0,-1,0,0,0"); -} - -#[test] -fn sketch_stock_element() { - let stock = view_element::Stock { - name: "Population".to_string(), - uid: 2, - x: 300.0, - y: 150.0, - label_side: view_element::LabelSide::Top, - compat: None, - }; - let mut buf = String::new(); - write_stock_element(&mut buf, &stock); - assert_eq!(buf, "10,2,Population,300,150,40,20,3,3,0,0,0,0,0,0"); -} - -#[test] -fn sketch_flow_element_produces_valve_and_variable() { - let flow = view_element::Flow { - name: "Infection_Rate".to_string(), - uid: 6, - x: 295.0, - y: 191.0, - label_side: view_element::LabelSide::Bottom, - points: vec![], - compat: None, - label_compat: None, - }; - let mut buf = String::new(); - let valve_uids = HashMap::from([(6, 100)]); - let mut next_connector_uid = 200; - write_flow_element( - &mut buf, - &flow, - &valve_uids, - &HashSet::new(), - &mut next_connector_uid, - ); - // No flow points, so no pipe connectors; valve and label follow - assert!(buf.contains("11,100,0,295,191,6,8,34,3,0,0,1,0,0,0")); - // Label sits 20px below the valve (y 191 -> 211); its box is sized to the - // text ("Infection Rate" -> 14 chars * 6px = 84 wide, single-line height 11). - assert!(buf.contains("10,6,Infection Rate,295,211,84,11,40,3,0,0,-1,0,0,0")); -} - -#[test] -fn sketch_flow_element_emits_pipe_connectors_from_flow_points() { - let flow = view_element::Flow { - name: "Infection_Rate".to_string(), - uid: 6, - x: 150.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - points: vec![ - view_element::FlowPoint { - x: 100.0, - y: 100.0, - attached_to_uid: Some(1), - }, - view_element::FlowPoint { - x: 200.0, - y: 100.0, - attached_to_uid: Some(2), - }, - ], - compat: None, - label_compat: None, - }; - let mut buf = String::new(); - let valve_uids = HashMap::from([(6, 100)]); - let mut next_connector_uid = 200; - write_flow_element( - &mut buf, - &flow, - &valve_uids, - &HashSet::new(), - &mut next_connector_uid, - ); - - let connector_lines: Vec<&str> = buf.lines().filter(|line| line.starts_with("1,")).collect(); - assert_eq!( - connector_lines.len(), - 2, - "Expected two type-1 connector lines for flow endpoints: {}", - buf - ); - assert!( - connector_lines.iter().any(|line| line.contains(",100,1,")), - "Expected connector from valve uid 100 to endpoint uid 1: {}", - buf - ); - assert!( - connector_lines.iter().any(|line| line.contains(",100,2,")), - "Expected connector from valve uid 100 to endpoint uid 2: {}", - buf - ); -} - -#[test] -fn sketch_flow_element_derives_stock_connector_points_from_takeoffs() { - let flow = view_element::Flow { - name: "Infection_Rate".to_string(), - uid: 6, - x: 150.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - points: vec![ - view_element::FlowPoint { - x: 122.5, - y: 100.0, - attached_to_uid: Some(1), - }, - view_element::FlowPoint { - x: 177.5, - y: 100.0, - attached_to_uid: Some(2), - }, - ], - compat: None, - label_compat: None, - }; - let mut buf = String::new(); - let valve_uids = HashMap::from([(6, 100)]); - let elem_positions = HashMap::from([(1, (100, 100)), (2, (200, 100))]); - let stock_uids = HashSet::from([1, 2]); - let mut next_connector_uid = 200; - write_flow_element_with_context( - &mut buf, - &flow, - &valve_uids, - &HashSet::new(), - &mut next_connector_uid, - SketchTransform::identity(), - &elem_positions, - &stock_uids, - None, - ); - - // Sink pipe (last point) carries direction 4; source pipe (first point) - // carries direction 100 -- the endpoint *role*, not stock-vs-cloud. - assert!( - buf.contains("1,200,100,2,4,0,0,22,0,0,0,-1--1--1,,1|(200,100)|"), - "sink pipe connector should be reconstructed from the stock center: {buf}" - ); - assert!( - buf.contains("1,201,100,1,100,0,0,22,0,0,0,-1--1--1,,1|(100,100)|"), - "source pipe connector should be reconstructed from the stock center: {buf}" - ); - // Canonical bottom-label fallback: 20px below the valve (y 100 -> 120), - // box sized to the text ("Infection Rate" -> 14 chars * 6px = 84, h 11). - assert!( - buf.contains("10,6,Infection Rate,150,120,84,11,40,3,0,0,-1,0,0,0"), - "flow label should fall back to the canonical bottom label position: {buf}" - ); -} - -/// An outflow into a sink cloud: the cloud-side pipe (the sink, last point) -/// must carry direction 4 and the stock-side pipe (the source, first point) -/// direction 100 -- the reverse of "stock => 4, cloud => 100", which is what -/// Vensim's own outflow-to-cloud sketches do. -#[test] -fn sketch_flow_outflow_to_cloud_uses_role_based_direction_flags() { - let flow = view_element::Flow { - name: "Drain".to_string(), - uid: 6, - x: 150.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - points: vec![ - // Source: a stock at the left. - view_element::FlowPoint { - x: 122.5, - y: 100.0, - attached_to_uid: Some(1), - }, - // Sink: a cloud at the right. - view_element::FlowPoint { - x: 200.0, - y: 100.0, - attached_to_uid: Some(2), - }, - ], - compat: None, - label_compat: None, - }; - let mut buf = String::new(); - let valve_uids = HashMap::from([(6, 100)]); - let elem_positions = HashMap::from([(1, (100, 100)), (2, (200, 100))]); - let stock_uids = HashSet::from([1]); // only uid 1 is a stock; uid 2 is the cloud - let mut next_connector_uid = 200; - write_flow_element_with_context( - &mut buf, - &flow, - &valve_uids, - &HashSet::new(), - &mut next_connector_uid, - SketchTransform::identity(), - &elem_positions, - &stock_uids, - None, - ); - - assert!( - buf.contains("1,200,100,2,4,0,0,22,0,0,0,-1--1--1,,1|(200,100)|"), - "sink cloud pipe should carry direction 4: {buf}" - ); - assert!( - buf.contains("1,201,100,1,100,0,0,22,0,0,0,-1--1--1,,1|(100,100)|"), - "source stock pipe should carry direction 100: {buf}" - ); -} - -#[test] -fn valve_uids_do_not_collide_with_existing_elements() { - // stock uid=1, flow uid=2 -> valve must NOT get uid=1 - let elements = vec![ - ViewElement::Stock(view_element::Stock { - name: "Population".to_string(), - uid: 1, - x: 100.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - }), - ViewElement::Flow(view_element::Flow { - name: "Birth_Rate".to_string(), - uid: 2, - x: 200.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - points: vec![], - compat: None, - label_compat: None, - }), - ]; - - let valve_uids = allocate_valve_uids(&elements); - // The valve for flow uid=2 must not equal 1 (stock's uid) - let valve_uid = valve_uids[&2]; - assert_ne!(valve_uid, 1, "Valve UID collides with stock UID"); - assert_ne!(valve_uid, 2, "Valve UID collides with flow UID"); -} - -#[test] -fn sketch_cloud_element() { - let cloud = view_element::Cloud { - uid: 7, - flow_uid: 6, - x: 479.0, - y: 235.0, - compat: None, - }; - let mut buf = String::new(); - write_cloud_element(&mut buf, &cloud); - assert_eq!(buf, "12,7,48,479,235,10,8,0,3,0,0,-1,0,0,0"); -} - -#[test] -fn sketch_alias_element() { - let alias = view_element::Alias { - uid: 10, - alias_of_uid: 1, - x: 200.0, - y: 300.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - }; - let mut name_map = HashMap::new(); - name_map.insert(1, "Growth_Rate"); - let mut buf = String::new(); - write_alias_element(&mut buf, &alias, &name_map); - assert!(buf.starts_with("10,10,Growth Rate,200,300,40,20,8,2,0,3,-1,0,0,0,")); - assert!(buf.contains("128-128-128")); -} - -#[test] -fn sketch_alias_element_offsets_stock_ghost_coordinates() { - let alias = view_element::Alias { - uid: 10, - alias_of_uid: 1, - x: 200.0, - y: 300.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - }; - let mut name_map = HashMap::new(); - name_map.insert(1, "Population"); - let mut buf = String::new(); - write_alias_element_with_context( - &mut buf, - &alias, - &name_map, - &HashSet::from([1]), - SketchTransform::identity(), - None, - ); - assert!( - buf.starts_with("10,10,Population,222,317,40,20,8,2,0,3,-1,0,0,0,"), - "stock ghosts should serialize using Vensim's stock-alias offset: {buf}" - ); -} - -// ---- Phase 5 Task 2: Connector serialization (type 1) ---- - -#[test] -fn sketch_link_straight() { - let link = view_element::Link { - uid: 3, - from_uid: 1, - to_uid: 2, - shape: LinkShape::Straight, - polarity: None, - }; - let mut positions = HashMap::new(); - positions.insert(1, (100, 100)); - positions.insert(2, (200, 200)); - let mut buf = String::new(); - write_link_element(&mut buf, &link, &positions, false); - // Straight => control point (0,0), field 9 = 64 (influence connector) - assert_eq!(buf, "1,3,1,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)|"); -} - -#[test] -fn sketch_link_with_polarity_symbol() { - let link = view_element::Link { - uid: 5, - from_uid: 1, - to_uid: 2, - shape: LinkShape::Straight, - polarity: Some(LinkPolarity::Positive), - }; - let positions = HashMap::new(); - let mut buf = String::new(); - write_link_element(&mut buf, &link, &positions, false); - // polarity=43 ('+'), field 9 = 64 - assert!(buf.contains(",0,0,43,0,0,64,0,")); -} - -#[test] -fn sketch_link_with_polarity_letter() { - let link = view_element::Link { - uid: 5, - from_uid: 1, - to_uid: 2, - shape: LinkShape::Straight, - polarity: Some(LinkPolarity::Positive), - }; - let positions = HashMap::new(); - let mut buf = String::new(); - write_link_element(&mut buf, &link, &positions, true); - // polarity=83 ('S' for lettered positive), field 9 = 64 - assert!(buf.contains(",0,0,83,0,0,64,0,")); -} - -#[test] -fn sketch_link_arc_produces_nonzero_control_point() { - let link = view_element::Link { - uid: 3, - from_uid: 1, - to_uid: 2, - shape: LinkShape::Arc(45.0), - polarity: None, - }; - let mut positions = HashMap::new(); - positions.insert(1, (100, 100)); - positions.insert(2, (200, 100)); - let mut buf = String::new(); - write_link_element(&mut buf, &link, &positions, false); - // Arc should produce a non-(0,0) control point - assert!( - !buf.contains("|(0,0)|"), - "arc should not produce (0,0) control point" - ); -} - -#[test] -fn sketch_link_with_field_hints_preserves_nonsemantic_flags() { - let link = view_element::Link { - uid: 3, - from_uid: 1, - to_uid: 2, - shape: LinkShape::Straight, - polarity: None, - }; - let positions = HashMap::from([(1, (100, 100)), (2, (200, 116)), (100, (200, 100))]); - let compat = view_element::LinkSketchCompat { - uid: 3, - field4: 1, - field10: 7, - }; - let mut buf = String::new(); - write_link_element_with_context( - &mut buf, - &link, - &positions, - false, - Some(&compat), - SketchTransform::identity(), - None, - ); - assert_eq!(buf, "1,3,1,2,1,0,0,0,0,64,7,-1--1--1,,1|(0,0)|"); -} - -#[test] -fn sketch_link_with_field_hints_still_uses_link_geometry() { - let link = view_element::Link { - uid: 3, - from_uid: 1, - to_uid: 2, - shape: LinkShape::Arc(45.0), - polarity: None, - }; - let positions = HashMap::from([(1, (110, 100)), (2, (210, 100))]); - // A recorded compat carries only field4/field10; the control point is always - // recomputed from the link's Arc angle and the current endpoint positions. - let compat = view_element::LinkSketchCompat { - uid: 3, - field4: 0, - field10: 0, - }; - let mut buf = String::new(); - write_link_element_with_context( - &mut buf, - &link, - &positions, - false, - Some(&compat), - SketchTransform::identity(), - None, - ); - let (ctrl_x, ctrl_y) = compute_control_point((110, 100), (210, 100), 45.0); - assert_eq!( - buf, - format!("1,3,1,2,0,0,0,0,0,64,0,-1--1--1,,1|({ctrl_x},{ctrl_y})|") - ); -} - -#[test] -fn sketch_link_multipoint_emits_all_points() { - let points = vec![ - view_element::FlowPoint { - x: 150.0, - y: 120.0, - attached_to_uid: None, - }, - view_element::FlowPoint { - x: 170.0, - y: 140.0, - attached_to_uid: None, - }, - view_element::FlowPoint { - x: 190.0, - y: 160.0, - attached_to_uid: None, - }, - ]; - let link = view_element::Link { - uid: 4, - from_uid: 1, - to_uid: 2, - shape: LinkShape::MultiPoint(points), - polarity: None, - }; - let mut positions = HashMap::new(); - positions.insert(1, (100, 100)); - positions.insert(2, (200, 200)); - let mut buf = String::new(); - write_link_element(&mut buf, &link, &positions, false); - assert!( - buf.contains("3|(150,120)|(170,140)|(190,160)|"), - "multipoint should emit all three points: {buf}" - ); -} - -// ---- Phase 5 Task 3: Complete sketch section assembly ---- - -#[test] -fn sketch_section_structure() { - let elements = vec![ - ViewElement::Stock(view_element::Stock { - name: "Population".to_string(), - uid: 1, - x: 100.0, - y: 100.0, - label_side: view_element::LabelSide::Top, - compat: None, - }), - ViewElement::Aux(view_element::Aux { - name: "Growth_Rate".to_string(), - uid: 2, - x: 200.0, - y: 200.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - }), - ViewElement::Link(view_element::Link { - uid: 3, - from_uid: 2, - to_uid: 1, - shape: LinkShape::Straight, - polarity: None, - }), - ]; - let sf = datamodel::StockFlow { - name: None, - elements, - view_box: Default::default(), - zoom: 1.0, - use_lettered_polarity: false, - font: None, - sketch_compat: None, - }; - let views = vec![View::StockFlow(sf)]; - - let mut writer = MdlWriter::new(); - writer.write_sketch_section(&views); - let output = writer.buf; - - // Header - assert!( - output.starts_with("V300 Do not put anything below this section"), - "should start with V300 header" - ); - // View title - assert!(output.contains("*View 1\n"), "should have view title"); - // Font line - assert!( - output.contains("$192-192-192"), - "should have font settings line" - ); - // Elements - assert!( - output.contains("10,1,Population,"), - "should have stock element" - ); - assert!( - output.contains("10,2,Growth Rate,"), - "should have aux element" - ); - assert!(output.contains("1,3,2,1,"), "should have link element"); - // Terminator - assert!( - output.ends_with("///---\\\\\\\n"), - "should end with sketch terminator" - ); -} - -#[test] -fn sketch_section_in_full_project() { - let var = make_aux("x", "1", None, ""); - let elements = vec![ViewElement::Aux(view_element::Aux { - name: "x".to_string(), - uid: 1, - x: 100.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - })]; - let model = datamodel::Model { - name: "default".to_owned(), - sim_specs: None, - variables: vec![var], - views: vec![View::StockFlow(datamodel::StockFlow { - name: None, - elements, - view_box: Default::default(), - zoom: 1.0, - use_lettered_polarity: false, - font: None, - sketch_compat: None, - })], - loop_metadata: vec![], - groups: vec![], - macro_spec: None, - }; - let project = make_project(vec![model]); - - let result = crate::mdl::project_to_mdl(&project); - assert!(result.is_ok()); - let mdl = result.unwrap(); - - // The sketch section should appear after the equations terminator - let terminator_pos = mdl - .find("\\\\\\---/// Sketch information") - .expect("should have equations terminator"); - let v300_pos = mdl.find("V300").expect("should have V300 header"); - assert!( - terminator_pos < v300_pos, - "V300 should come after equations terminator" - ); - - // The sketch terminator should be at the end - assert!( - mdl.contains("///---\\\\\\"), - "should have sketch terminator" - ); -} - -#[test] -fn sketch_roundtrip_teacup() { - // Read teacup.mdl, parse to Project, write sketch section, verify structure - let mdl_contents = include_str!("../../../../test/test-models/samples/teacup/teacup.mdl"); - let project = - crate::mdl::parse_mdl(mdl_contents).expect("teacup.mdl should parse successfully"); - - let model = &project.models[0]; - assert!( - !model.views.is_empty(), - "teacup model should have at least one view" - ); - - // Write the sketch section - let mut writer = MdlWriter::new(); - writer.write_sketch_section(&model.views); - let output = writer.buf; - - // Verify structural elements: the teacup model should have stocks, auxes, - // flows (valve + attached variable), links, and clouds. - assert!(output.contains("V300"), "output should contain V300 header"); - assert!( - output.contains("*View 1"), - "output should contain view title" - ); - assert!( - output.contains("///---\\\\\\"), - "output should end with sketch terminator" - ); - - // The teacup model elements (after roundtrip through datamodel): - // Stock: Teacup_Temperature -> type 10 with shape=3 - // Aux: Heat_Loss_to_Room flow -> type 11 valve + type 10 attached - // Aux: Room_Temperature, Characteristic_Time -> type 10 - // Links -> type 1 - // Clouds -> type 12 - - // Count element types in output - let lines: Vec<&str> = output.lines().collect(); - let type10_count = lines.iter().filter(|l| l.starts_with("10,")).count(); - let type11_count = lines.iter().filter(|l| l.starts_with("11,")).count(); - let type12_count = lines.iter().filter(|l| l.starts_with("12,")).count(); - let type1_count = lines.iter().filter(|l| l.starts_with("1,")).count(); - - // Teacup has: 1 stock (Teacup_Temperature), 3 auxes (Heat_Loss_to_Room, - // Room_Temperature, Characteristic_Time), 1 flow (Heat_Loss_to_Room) - // which produces valve+variable, plus 1 cloud. - // The exact numbers depend on the MDL->datamodel conversion, but - // we should have a reasonable set of elements. - assert!( - type10_count >= 2, - "should have at least 2 type-10 elements (variables/stocks), got {type10_count}" - ); - assert!( - type11_count >= 1, - "should have at least 1 type-11 element (valve), got {type11_count}" - ); - assert!( - type12_count >= 1, - "should have at least 1 type-12 element (cloud/comment), got {type12_count}" - ); - assert!( - type1_count >= 1, - "should have at least 1 type-1 element (connector), got {type1_count}" - ); - // Verify no empty lines were introduced between elements - let element_lines: Vec<&&str> = lines - .iter() - .filter(|l| { - l.starts_with("10,") - || l.starts_with("11,") - || l.starts_with("12,") - || l.starts_with("1,") - }) - .collect(); - assert!( - !element_lines.is_empty(), - "should have sketch elements in output" - ); - - // Verify the output can be re-parsed as a valid sketch section - let reparsed = crate::mdl::view::parse_views(&output); - assert!( - reparsed.is_ok(), - "re-serialized sketch should parse: {:?}", - reparsed.err() - ); - let views = reparsed.unwrap(); - assert!( - !views.is_empty(), - "re-parsed sketch should have at least one view" - ); - - // Verify all expected element types are present after re-parse - let view = &views[0]; - let has_variable = view - .iter() - .any(|e| matches!(e, crate::mdl::view::VensimElement::Variable(_))); - let has_connector = view - .iter() - .any(|e| matches!(e, crate::mdl::view::VensimElement::Connector(_))); - assert!(has_variable, "re-parsed view should have variables"); - assert!(has_connector, "re-parsed view should have connectors"); -} - -#[test] -fn sketch_roundtrip_preserves_view_title() { - let mdl_contents = r#"x = 5 -~ ~| -\\\---/// Sketch information -V300 Do not put anything below this section - it will be ignored -*Overview -$192-192-192,0,Times New Roman|12||0-0-0|0-0-0|0-0-255|-1--1--1|-1--1--1|96,96,100,0 -10,1,x,100,100,40,20,8,3,0,0,-1,0,0,0 -///---\\\ -"#; - - let project = - crate::mdl::parse_mdl(mdl_contents).expect("source MDL should parse successfully"); - let mdl = crate::mdl::project_to_mdl(&project).expect("roundtrip MDL write should work"); - - assert!( - mdl.contains("*Overview\r\n"), - "Roundtrip should preserve original view title: {}", - mdl - ); -} - -#[test] -fn sketch_roundtrip_sanitizes_multiline_view_title() { - let var = make_aux("x", "5", Some("Units"), "A constant"); - let model = datamodel::Model { - name: "default".to_owned(), - sim_specs: None, - variables: vec![var], - views: vec![View::StockFlow(datamodel::StockFlow { - name: Some("Overview\r\nMain".to_owned()), - elements: vec![ViewElement::Aux(view_element::Aux { - name: "x".to_owned(), - uid: 1, - x: 100.0, - y: 100.0, - label_side: view_element::LabelSide::Bottom, - compat: None, - })], - view_box: Default::default(), - zoom: 1.0, - use_lettered_polarity: false, - font: None, - sketch_compat: None, - })], - loop_metadata: vec![], - groups: vec![], - macro_spec: None, - }; - let project = make_project(vec![model]); - - let mdl = crate::mdl::project_to_mdl(&project).expect("MDL write should succeed"); - assert!( - mdl.contains("*Overview Main\r\n"), - "view title should be serialized as a single line: {mdl}", - ); - - let reparsed = crate::mdl::parse_mdl(&mdl).expect("written MDL should parse"); - let View::StockFlow(sf) = &reparsed.models[0].views[0]; - assert_eq!( - sf.name.as_deref(), - Some("Overview Main"), - "sanitized title should roundtrip through MDL", - ); -} - // ---- Free-text sanitization choke point (GH #849) ---- // // Free-text fields -- a variable's units/documentation, a group's name/doc, @@ -5866,8 +5080,29 @@ fn wildcard_reduce_over_apply_to_all_survives_re_rendering() { uid: None, compat: Compat::default(), }); + // A genuinely per-element arrayed variable, so the second render also + // exercises the `Arrayed` branch of the wildcard recovery. + let per_element = Variable::Aux(datamodel::Aux { + ident: "per_element".to_owned(), + equation: Equation::Arrayed( + vec!["DimA".to_owned()], + vec![ + ("A1".to_owned(), "1".to_owned(), None, None), + ("A2".to_owned(), "2".to_owned(), None, None), + ], + None, + false, + ), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: Compat::default(), + }); let target = make_aux("target", "-((-0) ^ sum(arr[*]))", None, ""); - let mut project = make_project(vec![make_model(vec![arr, target])]); + let target2 = make_aux("target2", "sum(per_element[*])", None, ""); + let mut project = make_project(vec![make_model(vec![arr, per_element, target, target2])]); project.dimensions = vec![datamodel::Dimension { name: "DimA".to_owned(), elements: datamodel::DimensionElements::Named(vec!["A1".to_owned(), "A2".to_owned()]), @@ -5888,12 +5123,27 @@ fn wildcard_reduce_over_apply_to_all_survives_re_rendering() { .iter() .find(|v| v.get_ident() == "arr") .expect("arr survives the round trip"); - assert!( - matches!(arr2.get_equation(), Some(Equation::Arrayed(..))), - "the apply-to-all must re-import as per-element sub-equations, else this \ - test is not exercising the re-render shape: {:?}", + // The re-import keeps the apply-to-all as ONE equation (the importer no + // longer explodes a single A2A equation into per-element slots). What the + // wildcard recovery actually needs is the variable's declared DIMENSIONS, + // which both equation shapes carry -- `per_element` below is the same + // round trip over a genuinely per-element variable, so the `Arrayed` path + // through `WriterContext` stays covered. + assert!( + matches!(arr2.get_equation(), Some(Equation::ApplyToAll(..))), + "a single apply-to-all equation must re-import as one equation: {:?}", arr2.get_equation(), ); + let per_element = p2.models[0] + .variables + .iter() + .find(|v| v.get_ident() == "per_element") + .expect("per_element survives the round trip"); + assert!( + matches!(per_element.get_equation(), Some(Equation::Arrayed(..))), + "genuinely per-element equations must stay per-element: {:?}", + per_element.get_equation(), + ); let (mdl2, warnings2) = crate::mdl::project_to_mdl_with_warnings(&p2).unwrap(); assert!( @@ -5904,6 +5154,12 @@ fn wildcard_reduce_over_apply_to_all_survives_re_rendering() { mdl2.contains("SUM(arr[DimA!])"), "second render emitted a reader-rejected bare wildcard:\n{mdl2}" ); + assert!( + // The writer spells the MDL name with a space (`quoted_space_to_underbar` + // reversed), so match the emitted spelling rather than the ident. + mdl2.contains("SUM(per element[DimA!])"), + "second render lost the bang form over a per-element array:\n{mdl2}" + ); let p3 = crate::mdl::parse_mdl(&mdl2).expect("second render must re-parse"); let (mdl3, warnings3) = crate::mdl::project_to_mdl_with_warnings(&p3).unwrap(); diff --git a/src/simlin-engine/src/test_common.rs b/src/simlin-engine/src/test_common.rs index 9b8451af7..b892686fb 100644 --- a/src/simlin-engine/src/test_common.rs +++ b/src/simlin-engine/src/test_common.rs @@ -154,6 +154,15 @@ impl TestProject { self } + /// Add an already-built dimension, for a shape the named constructors do + /// not cover -- a test that varies ONE dimension's mapping across fixtures + /// while keeping the rest of the model identical builds it directly and + /// hands it in here. + pub fn with_dimension(mut self, dim: Dimension) -> Self { + self.dimensions.push(dim); + self + } + /// Add a named dimension with a dimension mapping (e.g., DimA -> DimB) pub fn named_dimension_with_mapping( mut self, @@ -196,6 +205,39 @@ impl TestProject { self } + /// Add a named dimension carrying SEVERAL mappings at once, each an + /// optional element map (`&[]` means positional correspondence). + /// + /// A dimension with two mapping targets is the shape the implicit-axis + /// allocator's precedence rule is about (GH #996): one target can be + /// claimed by an earlier dependency axis while a later axis needs the + /// other. `named_dimension_with_mapping` and + /// `named_dimension_with_element_mapping` each declare exactly one, so + /// neither can express it. + pub fn named_dimension_with_mappings( + mut self, + name: &str, + elements: &[&str], + mappings: &[(&str, &[(&str, &str)])], + ) -> Self { + let mut dim = Dimension::named( + name.to_string(), + elements.iter().map(|s| s.to_string()).collect(), + ); + dim.mappings = mappings + .iter() + .map(|(target, element_map)| datamodel::DimensionMapping { + target: target.to_string(), + element_map: element_map + .iter() + .map(|(s, t)| (s.to_string(), t.to_string())) + .collect(), + }) + .collect(); + self.dimensions.push(dim); + self + } + /// Add an auxiliary variable pub fn aux(mut self, name: &str, equation: &str, units: Option<&str>) -> Self { self.variables.push(Variable::Aux(datamodel::Aux { @@ -890,6 +932,343 @@ pub fn parse_array_declaration(decl: &str) -> (String, Vec) { } } +/// A `main` model that instantiates ONE arrayed sub-model TWICE, with a +/// different input wired into each instance. +/// +/// The sub-model reduces an array three ways -- `SUM(arr[*])`, an array-valued +/// `SUM(PREVIOUS(arr[*]))` and `SUM(INIT(arr[*]))` (GH #995) -- so all three +/// chunk-shaped view regions (`Curr`, `Prev`, `Initial`) are exercised on the +/// same fixture. Every reduction is pushed as a STATIC VIEW, whose `base_off` +/// comes from the sub-model's own layout and is therefore module-relative; a +/// backend that fails to add the executing instance's `module_off` reads the +/// ROOT's slots instead, and both instances then return the same wrong series. +/// +/// Three separate slips are distinguishable in the numbers: +/// +/// * `arr[D] = in * w[D]` with `w = [1, 2, 4]`, so `SUM(arr[*]) = 7 * in` and a +/// base offset that is wrong WITHIN the instance lands on a different weight. +/// * the two instances' inputs differ by 100x, so cross-instance aliasing shows +/// up as one instance's series appearing in the other. +/// * both inputs vary with TIME, so `PREVIOUS` and `INIT` are distinguishable +/// from `curr` and from each other. +/// +/// Note what was NOT broken, so the fixture's shape reads as necessary rather +/// than belt-and-braces: a cross-module read taken FROM THE ROOT was always +/// correct, because the root's `module_off` is 0 and the dropped addend is +/// invisible there (`array_tests::cross_module_array_reference_tests` passed +/// throughout). Only a view pushed while EXECUTING INSIDE an instance was wrong, +/// which is why this drives two instances rather than reading into one. The +/// two-HOP twin ([`nested_instance_arrayed_submodel_project`]) covers the other +/// axis a single hop cannot separate. +/// +/// Shared by the VM pin +/// (`array_operand_materialization_tests::an_array_view_inside_a_module_instance_reads_that_instance`) +/// and the wasm pin +/// (`wasmgen::module::tests::compile_simulation_arrayed_submodel_views_address_their_instance`), +/// because the two backends agreeing proves nothing here: the wasm view emitter +/// mirrors the VM opcode for opcode, so it mirrored this defect too. Both assert +/// the absolute series. +pub fn two_instance_arrayed_submodel_project() -> Project { + let aux = |ident: &str, eqn: &str, compat: datamodel::Compat| { + Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: Equation::Scalar(eqn.to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat, + }) + }; + let instance = |ident: &str, src: &str| { + Variable::Module(datamodel::Module { + references: vec![datamodel::ModuleReference { + src: src.to_string(), + dst: format!("{ident}.in"), + }], + ident: ident.to_string(), + model_name: "submodel".to_string(), + documentation: String::new(), + units: None, + compat: datamodel::Compat::default(), + ai_state: None, + uid: None, + }) + }; + let arrayed = |ident: &str, eqn: Equation| { + Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: eqn, + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }) + }; + + Project { + name: "two_instance_arrayed_submodel".to_string(), + sim_specs: SimSpecs { + start: 0.0, + stop: 3.0, + dt: datamodel::Dt::Dt(1.0), + save_step: None, + sim_method: datamodel::SimMethod::Euler, + time_units: None, + }, + dimensions: vec![Dimension::named( + "D".to_string(), + vec!["e1".to_string(), "e2".to_string(), "e3".to_string()], + )], + units: vec![], + models: vec![ + datamodel::Model { + name: "main".to_string(), + sim_specs: None, + variables: vec![ + aux("a_in", "10 * (1 + TIME)", datamodel::Compat::default()), + aux("b_in", "1000 * (1 + TIME)", datamodel::Compat::default()), + instance("sub_a", "a_in"), + instance("sub_b", "b_in"), + ], + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + datamodel::Model { + name: "submodel".to_string(), + sim_specs: None, + variables: vec![ + aux( + "in", + "0", + datamodel::Compat { + can_be_module_input: true, + ..datamodel::Compat::default() + }, + ), + arrayed( + "w", + Equation::Arrayed( + vec!["D".to_string()], + vec![ + ("e1".to_string(), "1".to_string(), None, None), + ("e2".to_string(), "2".to_string(), None, None), + ("e3".to_string(), "4".to_string(), None, None), + ], + None, + false, + ), + ), + arrayed( + "arr", + Equation::ApplyToAll(vec!["D".to_string()], "in * w[D]".to_string()), + ), + aux("out_curr", "SUM(arr[*])", datamodel::Compat::default()), + aux( + "out_prev", + "SUM(PREVIOUS(arr[*]))", + datamodel::Compat::default(), + ), + aux( + "out_init", + "SUM(INIT(arr[*]))", + datamodel::Compat::default(), + ), + ], + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }, + ], + source: Default::default(), + ai_information: None, + } +} + +/// The TWO-HOP twin of [`two_instance_arrayed_submodel_project`]: `main` +/// instantiates `mid` twice, and each `mid` instantiates `inner` once. +/// +/// A one-hop fixture cannot separate two different addressing rules. The VM +/// reaches a nested instance by ACCUMULATING (`module_off + decl.off` at each +/// `EvalModule`), so a backend that applied only the LAST hop's offset -- or that +/// re-based from the root at each hop -- still gets a one-hop model right and +/// this one wrong. `mid` therefore carries a scalar of its own AHEAD of the +/// module declaration, so `inner`'s block does not start at its parent's base and +/// the two hops' offsets are distinct non-zero numbers that must sum. +/// +/// Same arithmetic as the one-hop fixture (`SUM(arr[*]) = 7 * in`, inputs 100x +/// apart, both time-varying), so +/// [`two_instance_arrayed_submodel_expected`]'s reasoning carries over; only the +/// variable prefixes differ. +pub fn nested_instance_arrayed_submodel_project() -> Project { + let aux = |ident: &str, eqn: &str, compat: datamodel::Compat| { + Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: Equation::Scalar(eqn.to_string()), + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat, + }) + }; + let instance = |ident: &str, model: &str, src: &str| { + Variable::Module(datamodel::Module { + references: vec![datamodel::ModuleReference { + src: src.to_string(), + dst: format!("{ident}.in"), + }], + ident: ident.to_string(), + model_name: model.to_string(), + documentation: String::new(), + units: None, + compat: datamodel::Compat::default(), + ai_state: None, + uid: None, + }) + }; + let input = || datamodel::Compat { + can_be_module_input: true, + ..datamodel::Compat::default() + }; + let arrayed = |ident: &str, eqn: Equation| { + Variable::Aux(datamodel::Aux { + ident: ident.to_string(), + equation: eqn, + documentation: String::new(), + units: None, + gf: None, + ai_state: None, + uid: None, + compat: datamodel::Compat::default(), + }) + }; + let model = |name: &str, variables: Vec| datamodel::Model { + name: name.to_string(), + sim_specs: None, + variables, + views: vec![], + loop_metadata: vec![], + groups: vec![], + macro_spec: None, + }; + + Project { + name: "nested_instance_arrayed_submodel".to_string(), + sim_specs: SimSpecs { + start: 0.0, + stop: 3.0, + dt: datamodel::Dt::Dt(1.0), + save_step: None, + sim_method: datamodel::SimMethod::Euler, + time_units: None, + }, + dimensions: vec![Dimension::named( + "D".to_string(), + vec!["e1".to_string(), "e2".to_string(), "e3".to_string()], + )], + units: vec![], + models: vec![ + model( + "main", + vec![ + aux("a_in", "10 * (1 + TIME)", datamodel::Compat::default()), + aux("b_in", "1000 * (1 + TIME)", datamodel::Compat::default()), + instance("m_a", "mid", "a_in"), + instance("m_b", "mid", "b_in"), + ], + ), + model( + "mid", + vec![ + aux("in", "0", input()), + // Occupies mid's slot 0, so `inr`'s block starts past its + // parent's base and the two hops' offsets are both non-zero. + aux("pad", "in * 0", datamodel::Compat::default()), + instance("inr", "inner", "in"), + ], + ), + model( + "inner", + vec![ + aux("in", "0", input()), + arrayed( + "w", + Equation::Arrayed( + vec!["D".to_string()], + vec![ + ("e1".to_string(), "1".to_string(), None, None), + ("e2".to_string(), "2".to_string(), None, None), + ("e3".to_string(), "4".to_string(), None, None), + ], + None, + false, + ), + ), + arrayed( + "arr", + Equation::ApplyToAll(vec!["D".to_string()], "in * w[D]".to_string()), + ), + aux("out_curr", "SUM(arr[*])", datamodel::Compat::default()), + aux( + "out_prev", + "SUM(PREVIOUS(arr[*]))", + datamodel::Compat::default(), + ), + aux( + "out_init", + "SUM(INIT(arr[*]))", + datamodel::Compat::default(), + ), + ], + ), + ], + source: Default::default(), + ai_information: None, + } +} + +/// The series [`nested_instance_arrayed_submodel_project`] must produce. +pub fn nested_instance_arrayed_submodel_expected() -> Vec<(&'static str, Vec)> { + let p = |instance: &str, var: &str| -> &'static str { + Box::leak(format!("m_{instance}\u{b7}inr\u{b7}{var}").into_boxed_str()) + }; + vec![ + (p("a", "out_curr"), vec![70.0, 140.0, 210.0, 280.0]), + (p("b", "out_curr"), vec![7000.0, 14000.0, 21000.0, 28000.0]), + (p("a", "out_prev"), vec![0.0, 70.0, 140.0, 210.0]), + (p("b", "out_prev"), vec![0.0, 7000.0, 14000.0, 21000.0]), + (p("a", "out_init"), vec![70.0; 4]), + (p("b", "out_init"), vec![7000.0; 4]), + ] +} + +/// The series [`two_instance_arrayed_submodel_project`] must produce, as +/// `(variable, values)` pairs. `in` is `10*(1+TIME)` for `sub_a` and 100x that +/// for `sub_b`, and `SUM(arr[*]) = 7 * in`. +pub fn two_instance_arrayed_submodel_expected() -> Vec<(&'static str, Vec)> { + vec![ + ("sub_a\u{b7}out_curr", vec![70.0, 140.0, 210.0, 280.0]), + ( + "sub_b\u{b7}out_curr", + vec![7000.0, 14000.0, 21000.0, 28000.0], + ), + // The first step has no snapshot yet, so an array PREVIOUS reads its + // only permitted fallback, 0. + ("sub_a\u{b7}out_prev", vec![0.0, 70.0, 140.0, 210.0]), + ("sub_b\u{b7}out_prev", vec![0.0, 7000.0, 14000.0, 21000.0]), + ("sub_a\u{b7}out_init", vec![70.0; 4]), + ("sub_b\u{b7}out_init", vec![7000.0; 4]), + ] +} + #[cfg(test)] mod tests { use super::*; diff --git a/src/simlin-engine/src/vm.rs b/src/simlin-engine/src/vm.rs index 9a267ed91..0014dd4c3 100644 --- a/src/simlin-engine/src/vm.rs +++ b/src/simlin-engine/src/vm.rs @@ -10,7 +10,7 @@ use smallvec::SmallVec; use crate::alloc::allocate_available; use crate::bytecode::{ BuiltinId, ByteCode, ByteCodeContext, CompiledInitial, CompiledModule, DimId, LookupMode, Op2, - Opcode, RuntimeView, STACK_CAPACITY, TempId, + Opcode, RuntimeView, STACK_CAPACITY, TempId, ViewStorage, }; use crate::common::{Canonical, Error, ErrorCode, ErrorKind, Ident, Result}; use crate::dimensions::match_dimensions_two_pass; @@ -459,6 +459,97 @@ impl Stack { } } +/// The three chunk-shaped f64 regions a static view can be read from, plus the +/// two pieces of run state that say what a snapshot read means before its +/// snapshot exists. +/// +/// `temp_storage` is deliberately NOT a field: it is the one region an opcode +/// can also WRITE through while reading views (every array-producing opcode +/// does), so it stays a separate `&mut` parameter and only the read side is +/// bundled here. Every field is `Copy`, so a caller mints one for the length of +/// a read loop and drops it before touching `curr`/`temp_storage` mutably. +#[derive(Clone, Copy)] +pub(crate) struct ChunkRegions<'a> { + curr: &'a [f64], + /// The snapshot taken after the previous step's stocks (`PREVIOUS`). + prev: &'a [f64], + /// The snapshot taken after the initials phase (`INIT`). + initial: &'a [f64], + /// Mirrors `EvalState::use_prev_fallback`: true until the first + /// `prev_values` snapshot exists. + use_prev_fallback: bool, + /// Which phase is being evaluated, so an `Initial` view resolves its + /// "during initials the snapshot is not taken yet" branch exactly as + /// `Opcode::LoadInitial` does. + part: StepPart, +} + +impl<'a> ChunkRegions<'a> { + /// A bundle over `curr` alone, for test harnesses that build a + /// `ViewStorage::Curr` view by hand and address no snapshot region. The + /// snapshot slices are deliberately EMPTY rather than aliases of `curr`, so + /// a view that mis-routes to one panics instead of quietly returning + /// plausible values. + #[cfg(test)] + pub(crate) fn curr_only(curr: &'a [f64]) -> Self { + ChunkRegions { + curr, + prev: &[], + initial: &[], + use_prev_fallback: false, + part: StepPart::Flows, + } + } + + /// The slice a view's elements live in and the flat base to add its + /// `offset`/`flat_offset` to, or `None` when every element of the view reads + /// the `PREVIOUS` fallback instead of a buffer. + /// + /// The `None` case is the array route's half of the first-step semantics, + /// and it is deliberately a BRANCH rather than a reliance on `prev_values` + /// being zero-filled: `Opcode::LoadPrev` returns its caller-supplied + /// fallback while `use_prev_fallback` is set, and an array-valued + /// `PREVIOUS` can only carry the default fallback of 0 + /// (`codegen::is_default_previous_fallback` rejects any other), so element + /// for element the two routes agree by construction. Making it a branch is + /// also what keeps the wasm backend -- whose `reset` does not clear the + /// snapshot regions -- able to mirror this with the same `select` its + /// scalar `LoadPrev` already emits. + #[inline] + fn backing<'s>( + &self, + view: &RuntimeView, + temp_storage: &'s [f64], + context: &ByteCodeContext, + ) -> Option<(&'s [f64], usize)> + where + 'a: 's, + { + match view.storage { + ViewStorage::Curr => Some((self.curr, view.base_off as usize)), + ViewStorage::Temp => Some((temp_storage, context.temp_offsets[view.base_off as usize])), + ViewStorage::Prev => { + if self.use_prev_fallback { + None + } else { + Some((self.prev, view.base_off as usize)) + } + } + // During initials the snapshot has not been captured yet, so read + // `curr` -- which IS the initial value being computed. Mirrors + // `Opcode::LoadInitial`. + ViewStorage::Initial => { + let data = if self.part == StepPart::Initials { + self.curr + } else { + self.initial + }; + Some((data, view.base_off as usize)) + } + } + } +} + /// Mutable evaluation state grouped into a single struct to reduce argument /// count in eval functions (was 11-14 args, now 6-10). In `eval_bytecode`, /// the fields are destructured into local reborrows for ergonomic access; @@ -1871,6 +1962,22 @@ impl Vm { let mut prev_values = &mut *state.prev_values; let use_prev_fallback = state.use_prev_fallback; + // The read-only chunk regions a static view can address. Minted fresh + // at each use rather than bound once: `curr` is `&mut` here and several + // arms write it (and `temp_storage`) in the same breath as reading a + // view, so a long-lived shared reborrow would not typecheck. + macro_rules! regions { + () => { + ChunkRegions { + curr: &*curr, + prev: &*prev_values, + initial: initial_values, + use_prev_fallback, + part, + } + }; + } + let mut condition = false; let mut subscript_index: SmallVec<[(u16, u16); 4]> = SmallVec::new(); let mut subscript_index_valid = true; @@ -2336,7 +2443,7 @@ impl Vm { Opcode::PushStaticView { view_id } => { let static_view = &context.static_views[*view_id as usize]; - view_stack.push(static_view.to_runtime_view()); + view_stack.push(static_view.to_runtime_view(module_off as u32)); } Opcode::PushVarViewDirect { @@ -2496,12 +2603,13 @@ impl Vm { view.offset as usize + iter_state.current }; - let value = if view.is_temp { - let temp_off = context.temp_offsets[view.base_off as usize]; - temp_storage[temp_off + flat_off] - } else { - curr[view.base_off as usize + flat_off] - }; + let value = Self::read_view_element( + view, + flat_off, + regions!(), + temp_storage, + context, + ); stack.push(value); } } @@ -2619,12 +2727,13 @@ impl Vm { }; if let Some(flat_off) = result { - let value = if source_view.is_temp { - let temp_off = context.temp_offsets[source_view.base_off as usize]; - temp_storage[temp_off + flat_off] - } else { - curr[source_view.base_off as usize + flat_off] - }; + let value = Self::read_view_element( + source_view, + flat_off, + regions!(), + temp_storage, + context, + ); stack.push(value); } else { // Out of bounds or no matching dimension - return NaN @@ -2733,12 +2842,13 @@ impl Vm { }; if let Some(flat_off) = result { - let value = if source_view.is_temp { - let temp_off = context.temp_offsets[source_view.base_off as usize]; - temp_storage[temp_off + flat_off] - } else { - curr[source_view.base_off as usize + flat_off] - }; + let value = Self::read_view_element( + source_view, + flat_off, + regions!(), + temp_storage, + context, + ); stack.push(value); } else { // Out of bounds or no matching dimension - return NaN @@ -2781,8 +2891,14 @@ impl Vm { // Empty views return 0.0 for SUM (the additive identity) Opcode::ArraySum {} => { let view = view_stack.last().unwrap(); - let sum = - Self::reduce_view(temp_storage, view, curr, context, |acc, v| acc + v, 0.0); + let sum = Self::reduce_view( + temp_storage, + view, + regions!(), + context, + |acc, v| acc + v, + 0.0, + ); stack.push(sum); } @@ -2794,7 +2910,7 @@ impl Vm { let max = Self::reduce_view( temp_storage, view, - curr, + regions!(), context, |acc, v| if v > acc { v } else { acc }, f64::NEG_INFINITY, @@ -2811,7 +2927,7 @@ impl Vm { let min = Self::reduce_view( temp_storage, view, - curr, + regions!(), context, |acc, v| if v < acc { v } else { acc }, f64::INFINITY, @@ -2828,7 +2944,7 @@ impl Vm { let sum = Self::reduce_view( temp_storage, view, - curr, + regions!(), context, |acc, v| acc + v, 0.0, @@ -2847,7 +2963,7 @@ impl Vm { let sum = Self::reduce_view( temp_storage, view, - curr, + regions!(), context, |acc, v| acc + v, 0.0, @@ -2859,7 +2975,7 @@ impl Vm { let variance_sum = Self::reduce_view( temp_storage, view, - curr, + regions!(), context, |acc, v| acc + (v - mean).powf(2.0), 0.0, @@ -2969,12 +3085,13 @@ impl Vm { let flat_off = view.flat_offset(&ordered_source_indices); - let value = if view.is_temp { - let temp_off = context.temp_offsets[view.base_off as usize]; - temp_storage[temp_off + flat_off] - } else { - curr[view.base_off as usize + flat_off] - }; + let value = Self::read_view_element( + view, + flat_off, + regions!(), + temp_storage, + context, + ); stack.push(value); } } @@ -3031,7 +3148,7 @@ impl Vm { let sel_val = Self::read_view_element( sel_view, sel_off, - curr, + regions!(), temp_storage, context, ); @@ -3041,7 +3158,7 @@ impl Vm { let expr_val = Self::read_view_element( expr_view, expr_off, - curr, + regions!(), temp_storage, context, ); @@ -3081,7 +3198,7 @@ impl Vm { offset_view, *write_temp_id, *full_source_len, - curr, + regions!(), temp_storage, context, ); @@ -3097,7 +3214,7 @@ impl Vm { input_view, direction, *write_temp_id, - curr, + regions!(), temp_storage, context, ); @@ -3123,7 +3240,7 @@ impl Vm { let val = Self::read_view_element( input_view, flat_off, - curr, + regions!(), temp_storage, context, ); @@ -3214,7 +3331,7 @@ impl Vm { let val = Self::read_view_element( requests_view, flat_off, - curr, + regions!(), temp_storage, context, ); @@ -3234,7 +3351,7 @@ impl Vm { let val = Self::read_view_element( profile_view, flat_off, - curr, + regions!(), temp_storage, context, ); @@ -3308,7 +3425,7 @@ impl Vm { let val = Self::read_view_element( requests_view, flat_off, - curr, + regions!(), temp_storage, context, ); @@ -3329,7 +3446,7 @@ impl Vm { let val = Self::read_view_element( priority_view, flat_off, - curr, + regions!(), temp_storage, context, ); @@ -3368,7 +3485,7 @@ impl Vm { fn reduce_view( temp_storage: &[f64], view: &RuntimeView, - curr: &[f64], + regions: ChunkRegions<'_>, context: &ByteCodeContext, f: Fold, init: f64, @@ -3383,19 +3500,25 @@ impl Vm { let size = view.size(); + let Some((data, base)) = regions.backing(view, temp_storage, context) else { + // A PREVIOUS view before the first snapshot: every element is the + // fallback 0, so fold that many zeros rather than reading a buffer. + // Same iteration count and same order, so the FP result matches a + // zero-filled region exactly. + let mut acc = init; + for _ in 0..size { + acc = f(acc, 0.0); + } + return acc; + }; + // Dense linear run (the overwhelmingly common case: whole arrays and // leading-dimension slices): fold over the backing slice directly, - // skipping the per-element index decompose + stride dot product and - // the per-element temp/curr branch. Iteration order is identical to - // the general path (row-major == ascending flat offset for a linear - // run), so FP reduction results are bit-identical. + // skipping the per-element index decompose + stride dot product. + // Iteration order is identical to the general path (row-major == + // ascending flat offset for a linear run), so FP reduction results are + // bit-identical. if let Some(start) = view.dense_linear_start() { - let base = if view.is_temp { - context.temp_offsets[view.base_off as usize] - } else { - view.base_off as usize - }; - let data: &[f64] = if view.is_temp { temp_storage } else { curr }; let mut acc = init; for &value in &data[base + start..base + start + size] { acc = f(acc, value); @@ -3411,15 +3534,7 @@ impl Vm { for _ in 0..size { let flat_off = view.flat_offset(&indices); - - let value = if view.is_temp { - let temp_off = context.temp_offsets[view.base_off as usize]; - temp_storage[temp_off + flat_off] - } else { - curr[view.base_off as usize + flat_off] - }; - - acc = f(acc, value); + acc = f(acc, data[base + flat_off]); increment_indices(&mut indices, dims); } @@ -3427,7 +3542,8 @@ impl Vm { } /// Read a single element from a RuntimeView at a pre-computed memory offset. - /// Handles both variable views (from curr[]) and temp views (from temp_storage[]). + /// Routes through the view's [`ViewStorage`], so a temp view reads + /// `temp_storage` and a snapshot view reads `prev_values`/`initial_values`. /// The `flat_off` parameter is the actual memory offset within the view's storage, /// NOT a sequential iteration index. For contiguous views, flat_off equals the /// iteration index. For non-contiguous or sparse views, the caller must compute @@ -3436,15 +3552,13 @@ impl Vm { pub(crate) fn read_view_element( view: &RuntimeView, flat_off: usize, - curr: &[f64], + regions: ChunkRegions<'_>, temp_storage: &[f64], context: &ByteCodeContext, ) -> f64 { - if view.is_temp { - let temp_off = context.temp_offsets[view.base_off as usize]; - temp_storage[temp_off + flat_off] - } else { - curr[view.base_off as usize + flat_off] + match regions.backing(view, temp_storage, context) { + Some((data, base)) => data[base + flat_off], + None => 0.0, } } @@ -4983,6 +5097,115 @@ mod superinstruction_tests { #[path = "vm_reset_run_to_and_constants_tests.rs"] mod vm_reset_run_to_and_constants_tests; +/// `ChunkRegions::backing` is where a view's storage region is resolved, and it +/// is the ONE place the VM reproduces the two snapshot semantics the scalar +/// opcodes carry. It is tested directly because it cannot be reached +/// end-to-end: `Vm::new` and `Vm::reset` zero-fill `prev_values`, and the only +/// moments `use_prev_fallback` is set are exactly the moments the buffer is +/// still zeroed -- so the fallback BRANCH and the buffer AGREE on every run, and +/// deleting the branch changes no simulation result. (The wasm backend is not so +/// lucky: its `reset` does not clear the snapshot regions, which is why the +/// `select` there IS observable and is pinned by +/// `wasmgen::module_tests::compile_simulation_repeated_run_resets_previous_fallback_for_an_array_view`.) +/// Keeping the branch anyway is what makes the two backends state the same rule +/// rather than one relying on an initialization the other does not perform. +/// +/// Rows are the full cross of the enumeration -- 4 `ViewStorage` arms x +/// `use_prev_fallback` x `StepPart::Initials`-or-not -- rather than only the +/// cells that vary, so an arm that starts reading a flag it should ignore is +/// caught too. `Curr` and `Temp` are inert on both factors; `Prev` varies with +/// the flag alone; `Initial` with the phase alone. +#[cfg(test)] +mod chunk_regions_tests { + use super::*; + use smallvec::smallvec; + + /// Each region carries a distinct value at slot 0, so the value read + /// identifies WHICH region was selected -- a resolver that returned the + /// right slice for the wrong reason cannot pass. + const CURR: f64 = 1.0; + const PREV: f64 = 2.0; + const INITIAL: f64 = 3.0; + const TEMP: f64 = 4.0; + + fn context() -> ByteCodeContext { + let mut ctx = ByteCodeContext::default(); + ctx.set_temp_info(vec![0], 1); + ctx + } + + fn view(storage: ViewStorage) -> RuntimeView { + let mut v = RuntimeView::for_var(0, smallvec![1], smallvec![0]); + v.storage = storage; + v + } + + /// The value `backing` selects for `storage` under the given run state, or + /// `None` when it reports "this view reads the PREVIOUS fallback". + fn resolve(storage: ViewStorage, use_prev_fallback: bool, part: StepPart) -> Option { + let curr = [CURR]; + let prev = [PREV]; + let initial = [INITIAL]; + let temp = [TEMP]; + let regions = ChunkRegions { + curr: &curr, + prev: &prev, + initial: &initial, + use_prev_fallback, + part, + }; + regions + .backing(&view(storage), &temp, &context()) + .map(|(data, base)| data[base]) + } + + #[test] + fn backing_routes_every_storage_arm_under_every_run_state() { + for &part in &[StepPart::Initials, StepPart::Flows, StepPart::Stocks] { + for &fallback in &[true, false] { + let ctx = format!("part={part:?} use_prev_fallback={fallback}"); + + // Curr and Temp read their own region unconditionally. + assert_eq!( + resolve(ViewStorage::Curr, fallback, part), + Some(CURR), + "Curr must be inert on both factors ({ctx})" + ); + assert_eq!( + resolve(ViewStorage::Temp, fallback, part), + Some(TEMP), + "Temp must be inert on both factors ({ctx})" + ); + + // Prev: the fallback while no snapshot exists, the snapshot + // after. This mirrors `Opcode::LoadPrev` exactly, and the + // fallback an ARRAY-valued PREVIOUS may carry is always 0 + // (`codegen::is_default_previous_fallback`), which is what + // `None` means to every caller. + assert_eq!( + resolve(ViewStorage::Prev, fallback, part), + if fallback { None } else { Some(PREV) }, + "Prev must follow use_prev_fallback and nothing else ({ctx})" + ); + + // Initial: `curr` during the initials phase (the snapshot has + // not been captured yet -- `curr` IS the initial value being + // computed), the snapshot otherwise. Mirrors + // `Opcode::LoadInitial`. + assert_eq!( + resolve(ViewStorage::Initial, fallback, part), + Some(if part == StepPart::Initials { + CURR + } else { + INITIAL + }), + "Initial must follow the phase and nothing else ({ctx})" + ); + } + } + } +} + /// Tests for empty-view behavior in VM array reducer opcodes (AC2). /// /// Zero-element dimensions cannot currently arise through the model compilation @@ -5019,7 +5242,14 @@ mod empty_view_reduce_tests { let curr: [f64; 0] = []; let temp: [f64; 0] = []; let ctx = empty_context(); - let result = Vm::reduce_view(&temp, &view, &curr, &ctx, |acc, v| acc + v, 0.0); + let result = Vm::reduce_view( + &temp, + &view, + ChunkRegions::curr_only(&curr), + &ctx, + |acc, v| acc + v, + 0.0, + ); assert!(result.is_nan()); } @@ -5040,12 +5270,19 @@ mod empty_view_reduce_tests { assert!(!view.is_contiguous(), "offset slice must not be contiguous"); // elements are curr[2 + 4 .. 2 + 8] = [6, 7, 8, 9] - let sum = Vm::reduce_view(&temp, &view, &curr, &ctx, |acc, v| acc + v, 0.0); + let sum = Vm::reduce_view( + &temp, + &view, + ChunkRegions::curr_only(&curr), + &ctx, + |acc, v| acc + v, + 0.0, + ); assert_eq!(sum, 30.0); let max = Vm::reduce_view( &temp, &view, - &curr, + ChunkRegions::curr_only(&curr), &ctx, |acc, v| if v > acc { v } else { acc }, f64::NEG_INFINITY, @@ -5068,7 +5305,14 @@ mod empty_view_reduce_tests { assert_eq!(view.dense_linear_start(), None); // elements: 1,2, 5,6, 9,10 - let sum = Vm::reduce_view(&temp, &view, &curr, &ctx, |acc, v| acc + v, 0.0); + let sum = Vm::reduce_view( + &temp, + &view, + ChunkRegions::curr_only(&curr), + &ctx, + |acc, v| acc + v, + 0.0, + ); assert_eq!(sum, 33.0); } @@ -5079,7 +5323,14 @@ mod empty_view_reduce_tests { let curr: [f64; 0] = []; let temp: [f64; 0] = []; let ctx = empty_context(); - let result = Vm::reduce_view(&temp, &view, &curr, &ctx, |acc, v| acc + v, 0.0); + let result = Vm::reduce_view( + &temp, + &view, + ChunkRegions::curr_only(&curr), + &ctx, + |acc, v| acc + v, + 0.0, + ); assert_eq!(result, 0.0); } @@ -5101,7 +5352,7 @@ mod empty_view_reduce_tests { let result = Vm::reduce_view( &temp, &view, - &curr, + ChunkRegions::curr_only(&curr), &ctx, |acc, v| if v > acc { v } else { acc }, f64::NEG_INFINITY, @@ -5122,7 +5373,7 @@ mod empty_view_reduce_tests { let result = Vm::reduce_view( &temp, &view, - &curr, + ChunkRegions::curr_only(&curr), &ctx, |acc, v| if v < acc { v } else { acc }, f64::INFINITY, @@ -5146,7 +5397,14 @@ mod empty_view_reduce_tests { let ctx = empty_context(); // reduce_view returns the sum init value (0.0) for empty views; // the ArrayMean opcode guards view.size()==0 before dividing by count - let sum = Vm::reduce_view(&temp, &view, &curr, &ctx, |acc, v| acc + v, 0.0); + let sum = Vm::reduce_view( + &temp, + &view, + ChunkRegions::curr_only(&curr), + &ctx, + |acc, v| acc + v, + 0.0, + ); assert_eq!(sum, 0.0); assert_eq!(view.size(), 0); // Without the guard: sum / count = 0.0 / 0.0 = NaN (IEEE); guard makes it explicit @@ -5169,7 +5427,14 @@ mod empty_view_reduce_tests { // reduce_view returns the sum init value (0.0) for empty views; // the ArrayStddev opcode guards size==0 before dividing by the // population-variance divisor `size`. - let sum = Vm::reduce_view(&temp, &view, &curr, &ctx, |acc, v| acc + v, 0.0); + let sum = Vm::reduce_view( + &temp, + &view, + ChunkRegions::curr_only(&curr), + &ctx, + |acc, v| acc + v, + 0.0, + ); assert_eq!(sum, 0.0); assert_eq!(view.size(), 0); // Without the guard: `variance_sum / (size as f64)` = `0.0 / 0.0` = NaN. diff --git a/src/simlin-engine/src/vm_vector_elm_map.rs b/src/simlin-engine/src/vm_vector_elm_map.rs index d7e5840e1..e1f4ee629 100644 --- a/src/simlin-engine/src/vm_vector_elm_map.rs +++ b/src/simlin-engine/src/vm_vector_elm_map.rs @@ -11,7 +11,7 @@ use smallvec::SmallVec; use crate::bytecode::{ByteCodeContext, RuntimeView, TempId}; -use crate::vm::{Vm, increment_indices}; +use crate::vm::{ChunkRegions, Vm, increment_indices}; /// Genuine-Vensim VECTOR ELM MAP: result element `i` = /// `source[base_i + round(offset[i])]` over the source variable's FULL @@ -35,7 +35,7 @@ pub(crate) fn vector_elm_map( offset_view: &RuntimeView, write_temp_id: TempId, full_source_len: u32, - curr: &[f64], + regions: ChunkRegions<'_>, temp_storage: &mut [f64], context: &ByteCodeContext, ) { @@ -93,7 +93,8 @@ pub(crate) fn vector_elm_map( Some(start) => start + i, None => offset_view.flat_offset(&off_indices), }; - let offset_val = Vm::read_view_element(offset_view, off_flat, curr, temp_storage, context); + let offset_val = + Vm::read_view_element(offset_view, off_flat, regions, temp_storage, context); // base_i: 0 for a full-array source; else the sliced view's flat // offset at this element's carried-dim projection. @@ -119,7 +120,7 @@ pub(crate) fn vector_elm_map( if flat_i < 0 || flat_i >= full_len as i64 { f64::NAN } else { - Vm::read_view_element(source_view, flat_i as usize, curr, temp_storage, context) + Vm::read_view_element(source_view, flat_i as usize, regions, temp_storage, context) } }; temp_storage[temp_off + i] = elem; diff --git a/src/simlin-engine/src/vm_vector_sort_order.rs b/src/simlin-engine/src/vm_vector_sort_order.rs index c0a20a3f7..1bdbb0ea9 100644 --- a/src/simlin-engine/src/vm_vector_sort_order.rs +++ b/src/simlin-engine/src/vm_vector_sort_order.rs @@ -11,7 +11,7 @@ use smallvec::SmallVec; use crate::bytecode::{ByteCodeContext, RuntimeView, TempId}; -use crate::vm::{Vm, increment_indices}; +use crate::vm::{ChunkRegions, Vm, increment_indices}; /// Genuine-Vensim VECTOR SORT ORDER. /// @@ -50,7 +50,7 @@ pub(crate) fn vector_sort_order( input_view: &RuntimeView, direction: i32, write_temp_id: TempId, - curr: &[f64], + regions: ChunkRegions<'_>, temp_storage: &mut [f64], context: &ByteCodeContext, ) { @@ -83,7 +83,7 @@ pub(crate) fn vector_sort_order( row.clear(); for local_idx in 0..inner { let flat_off = input_view.flat_offset(&indices); - let val = Vm::read_view_element(input_view, flat_off, curr, temp_storage, context); + let val = Vm::read_view_element(input_view, flat_off, regions, temp_storage, context); row.push((val, local_idx)); increment_indices(&mut indices, &input_view.dims); } diff --git a/src/simlin-engine/src/wasmgen/lower.rs b/src/simlin-engine/src/wasmgen/lower.rs index 64c458150..41f74cf81 100644 --- a/src/simlin-engine/src/wasmgen/lower.rs +++ b/src/simlin-engine/src/wasmgen/lower.rs @@ -76,7 +76,7 @@ use crate::bytecode::{ use crate::vm::{StepPart, make_module_key}; use super::WasmGenError; -use super::views::ElementAddr; +use super::views::{ElementAddr, RegionBases}; use super::views::{ViewBase, ViewDesc}; /// Bytes per f64 slot. @@ -264,6 +264,29 @@ pub(crate) struct EmitCtx<'a> { pub ctx: &'a ByteCodeContext, } +impl EmitCtx<'_> { + /// The region base addresses a view's element addressing needs. + /// + /// `initial` resolves the "during initials the snapshot is not taken yet" + /// branch HERE, at compile time, exactly as `emit_load_initial` does for the + /// scalar opcode -- the emitter knows which program it is lowering, so the + /// blob needs no runtime test. The VM makes the same decision at run time + /// (`ChunkRegions::backing`) because its one interpreter loop serves every + /// phase. + pub(crate) fn region_bases(&self) -> RegionBases { + RegionBases { + curr: self.curr_base, + temp_storage: self.temp_storage_base, + prev: self.prev_values_base, + initial: if self.step_part == StepPart::Initials { + self.curr_base + } else { + self.initial_values_base + }, + } + } +} + // Reserved global slots (absolute, module-independent), mirroring `crate::vm`. // `Apply` reads `curr[TIME_OFF]` / `curr[DT_OFF]` for the time-driven builtins. const TIME_OFF: u16 = 0; @@ -1435,11 +1458,12 @@ fn emit_ops( )); } // `PushVarViewDirect` builds a contiguous view from raw dim sizes - // (dim_ids all 0), the base for a dynamic subscript. It is the only - // `CurrModuleRelative` view opcode: the VM folds the runtime `module_off` - // into the base, where `PushStaticView` bakes an absolute slot in - // and `PushTempView` addresses `temp_storage` with no `module_off` - // at all. + // (dim_ids all 0), the base for a dynamic subscript. Its base is + // module-relative -- the VM folds the runtime `module_off` in -- and + // so is `PushStaticView`'s, whose `base_off` comes from the + // fragment's own model layout (`StaticArrayView::to_runtime_view` + // adds the same addend). `PushTempView` is the only one that + // addresses `temp_storage` with no `module_off` at all. Opcode::PushVarViewDirect { base_off, dim_list_id, @@ -1448,7 +1472,7 @@ fn emit_ops( let n = dims.len(); state.view_stack.push(ViewDesc::contiguous( u32::from(*base_off), - ViewBase::CurrModuleRelative, + ViewBase::Curr, dims, vec![0u16; n], )); @@ -2865,7 +2889,7 @@ pub(crate) fn emit_view_element_load( f: &mut Function, ) -> Result<(), WasmGenError> { let addr = desc - .element_addr(iter_idx, ctx.curr_base, ctx.temp_storage_base, ctx.ctx) + .element_addr(iter_idx, ctx.region_bases(), ctx.ctx) .ok_or_else(bad_temp_view)?; emit_addr_load(addr, ctx, f); Ok(()) @@ -2881,7 +2905,7 @@ fn emit_view_offset_load( f: &mut Function, ) -> Result<(), WasmGenError> { let addr = desc - .element_addr_for_flat(flat, ctx.curr_base, ctx.temp_storage_base, ctx.ctx) + .element_addr_for_flat(flat, ctx.region_bases(), ctx.ctx) .ok_or_else(bad_temp_view)?; emit_addr_load(addr, ctx, f); Ok(()) @@ -2919,6 +2943,14 @@ fn emit_addr_load(addr: ElementAddr, ctx: &EmitCtx, f: &mut Function) { /// neither is present it is a bare `0`. fn emit_addr_load_unguarded(addr: ElementAddr, ctx: &EmitCtx, f: &mut Function) { use Instruction as Ins; + // A PREVIOUS-region element reads the fallback 0 until the first snapshot + // exists. The fallback operand is pushed FIRST because `select` yields its + // deeper operand when the condition is non-zero -- the same shape + // `emit_load_prev` uses for the scalar opcode, and the value the VM's + // `ChunkRegions::backing` `None` arm produces. + if addr.prev_fallback_gated { + f.instruction(&Ins::F64Const(0.0.into())); + } let mut pushed = false; if addr.module_relative { push_module_relative_base(ctx, f); @@ -2938,6 +2970,10 @@ fn emit_addr_load_unguarded(addr: ElementAddr, ctx: &EmitCtx, f: &mut Function) f.instruction(&Ins::I32Const(0)); } f.instruction(&Ins::F64Load(memarg(addr.const_byte_offset))); + if addr.prev_fallback_gated { + f.instruction(&Ins::GlobalGet(ctx.use_prev_fallback_global)); + f.instruction(&Ins::Select); + } } /// The `Unsupported` error for a temp-backed view whose `base_off` is not a diff --git a/src/simlin-engine/src/wasmgen/lower_tests.rs b/src/simlin-engine/src/wasmgen/lower_tests.rs index bdf1bb4d4..28bdf1945 100644 --- a/src/simlin-engine/src/wasmgen/lower_tests.rs +++ b/src/simlin-engine/src/wasmgen/lower_tests.rs @@ -2215,6 +2215,7 @@ fn load_temp_dynamic_floors_fractional_index() { use crate::bytecode::{ DimensionInfo, RuntimeSparseMapping, RuntimeView, StaticArrayView, SubdimensionRelation, + ViewStorage, }; use smallvec::SmallVec; @@ -2227,10 +2228,10 @@ fn seed_run(base_byte: u64, values: &[f64]) -> Vec<(u64, f64)> { } /// Read element `iter_idx` of `view` from a flat slab `data` indexed by slot, -/// using the VM's own addressing (`to_runtime_view().flat_offset`). The +/// using the VM's own addressing (`to_runtime_view(0).flat_offset`). The /// addressing oracle for every reducer parity check. fn vm_view_element(view: &StaticArrayView, data: &[f64], iter_idx: usize) -> f64 { - let rv = view.to_runtime_view(); + let rv = view.to_runtime_view(0); let n = rv.dims.len(); let mut indices: SmallVec<[u16; 4]> = smallvec::smallvec![0; n]; let mut remaining = iter_idx; @@ -2245,7 +2246,7 @@ fn vm_view_element(view: &StaticArrayView, data: &[f64], iter_idx: usize) -> f64 /// The VM's expected `ArraySum` over `view`'s elements drawn from `data`. fn vm_sum(view: &StaticArrayView, data: &[f64]) -> f64 { - (0..view.to_runtime_view().size()) + (0..view.to_runtime_view(0).size()) .map(|i| vm_view_element(view, data, i)) .sum() } @@ -2261,7 +2262,7 @@ fn dense_view(base_off: u32, dims: &[u16]) -> StaticArrayView { strides.reverse(); StaticArrayView { base_off, - is_temp: false, + storage: ViewStorage::Curr, dims: dims.iter().copied().collect(), strides, offset: 0, @@ -2315,14 +2316,14 @@ fn static_view_sum_transposed_strides_matches_vm() { let data = [11.0, 12.0, 13.0, 21.0, 22.0, 23.0]; let view = StaticArrayView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::from_slice(&[3, 2]), strides: SmallVec::from_slice(&[1, 3]), offset: 0, sparse: SmallVec::new(), dim_ids: SmallVec::from_slice(&[0, 0]), }; - assert!(!view.to_runtime_view().is_contiguous()); + assert!(!view.to_runtime_view(0).is_contiguous()); let got = run_static_reduce(view.clone(), Opcode::ArraySum {}, &data); // Sum is order-independent and covers all six cells regardless. assert_eq!(got, vm_sum(&view, &data)); @@ -2336,7 +2337,7 @@ fn static_view_max_transposed_picks_right_cells() { let data = [11.0, 12.0, 99.0, 21.0, 22.0, 23.0]; let view = StaticArrayView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::from_slice(&[3, 2]), strides: SmallVec::from_slice(&[1, 3]), offset: 0, @@ -2355,7 +2356,7 @@ fn static_view_sum_sparse_matches_vm() { let data = [5.0, 6.0, 7.0, 8.0]; let view = StaticArrayView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::from_slice(&[2]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -2372,13 +2373,13 @@ fn static_view_sum_sparse_matches_vm() { #[test] fn static_temp_view_sum_reads_temp_storage() { - // A contiguous temp view (is_temp) reads temp_storage, not curr. temp_id + // A contiguous temp view reads temp_storage, not curr. temp_id // 0 lives at temp_offsets[0]=0, so its slot 0 is byte TEMP_BASE. let mut context = ByteCodeContext::default(); context.set_temp_info(vec![0], 3); let view = StaticArrayView { base_off: 0, // temp_id 0 - is_temp: true, + storage: ViewStorage::Temp, dims: SmallVec::from_slice(&[3]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -2408,7 +2409,7 @@ fn static_temp_view_honors_temp_offset() { context.set_temp_info(vec![0, 4], 6); let view = StaticArrayView { base_off: 1, // temp_id 1 - is_temp: true, + storage: ViewStorage::Temp, dims: SmallVec::from_slice(&[2]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -2741,7 +2742,7 @@ fn reducer_size_multidim_is_product() { fn empty_static_view() -> StaticArrayView { StaticArrayView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::from_slice(&[0]), strides: SmallVec::from_slice(&[1]), offset: 0, @@ -2943,7 +2944,7 @@ fn invalid_view_size_is_still_the_size() { /// A contiguous temp `StaticArrayView` over `dims` at `temp_id`. fn temp_view(temp_id: u32, dims: &[u16]) -> StaticArrayView { let mut v = dense_view(temp_id, dims); - v.is_temp = true; + v.storage = ViewStorage::Temp; v } @@ -3621,7 +3622,7 @@ fn reducer_over_view_exceeding_cap_is_unsupported() { // than emit a multi-megabyte function. let mut context = ByteCodeContext::default(); let view_id = context.add_static_view(dense_view(0, &[300, 300])); - assert!(dense_view(0, &[300, 300]).to_runtime_view().size() > MAX_UNROLL_UNITS); + assert!(dense_view(0, &[300, 300]).to_runtime_view(0).size() > MAX_UNROLL_UNITS); let ctx = ctx_with_arrays(&context); let code = vec![ Opcode::PushStaticView { view_id }, @@ -3698,7 +3699,7 @@ fn reducer_just_under_cap_compiles_and_matches_vm() { // this pins the boundary intent.) let data: Vec = (0..64).map(|i| (i as f64) * 0.5).collect(); let view = dense_view(0, &[64]); - assert!(view.to_runtime_view().size() <= MAX_UNROLL_UNITS); + assert!(view.to_runtime_view(0).size() <= MAX_UNROLL_UNITS); let got = run_static_reduce(view.clone(), Opcode::ArraySum {}, &data); assert_eq!(got, vm_sum(&view, &data)); } @@ -3739,8 +3740,8 @@ fn vm_vector_select_oracle( max_value: f64, action: i32, ) -> f64 { - let sel_rv = sel_view.to_runtime_view(); - let expr_rv = expr_view.to_runtime_view(); + let sel_rv = sel_view.to_runtime_view(0); + let expr_rv = expr_view.to_runtime_view(0); let size = sel_rv.size().min(expr_rv.size()); let mut selected: Vec = Vec::new(); let mut sel_idx: SmallVec<[u16; 4]> = smallvec::smallvec![0; sel_rv.dims.len()]; @@ -3981,11 +3982,11 @@ fn vm_elm_map_oracle( context.set_temp_info(vec![0], temp_slots); let mut temp_storage = vec![0.0f64; temp_slots]; crate::vm_vector_elm_map::vector_elm_map( - &source.to_runtime_view(), - &offset.to_runtime_view(), + &source.to_runtime_view(0), + &offset.to_runtime_view(0), 0, full_source_len, - data, + crate::vm::ChunkRegions::curr_only(data), &mut temp_storage, &context, ); @@ -4324,10 +4325,10 @@ fn vm_sort_order_oracle( context.set_temp_info(vec![0], temp_slots); let mut temp_storage = vec![0.0f64; temp_slots]; crate::vm_vector_sort_order::vector_sort_order( - &input.to_runtime_view(), + &input.to_runtime_view(0), direction, 0, - data, + crate::vm::ChunkRegions::curr_only(data), &mut temp_storage, &context, ); @@ -4343,7 +4344,7 @@ fn vm_rank_oracle( data: &[f64], temp_slots: usize, ) -> Vec { - let rv = input.to_runtime_view(); + let rv = input.to_runtime_view(0); let size = rv.size(); let mut indexed: Vec<(f64, usize)> = Vec::with_capacity(size); let mut idx: SmallVec<[u16; 4]> = smallvec::smallvec![0; rv.dims.len()]; @@ -4482,14 +4483,14 @@ fn vector_sort_order_transposed_view_matches_vm() { // the gather. Cross-checked vs the sibling over every element. let view = StaticArrayView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::from_slice(&[3, 2]), strides: SmallVec::from_slice(&[1, 3]), offset: 0, sparse: SmallVec::new(), dim_ids: SmallVec::from_slice(&[0, 0]), }; - assert!(!view.to_runtime_view().is_contiguous()); + assert!(!view.to_runtime_view(0).is_contiguous()); let data = [11.0, 12.0, 13.0, 21.0, 22.0, 23.0]; assert_sort_order_matches(&view, 1.0, &data, 6); assert_sort_order_matches(&view, 0.0, &data, 6); @@ -4737,7 +4738,7 @@ fn vm_lookup_array_oracle( tables: &[&[(f64, f64)]], temp_slots: usize, ) -> Vec { - let rv = input.to_runtime_view(); + let rv = input.to_runtime_view(0); let size = rv.size(); let mut idx: SmallVec<[u16; 4]> = smallvec::smallvec![0; rv.dims.len()]; let mut temp = vec![0.0f64; temp_slots]; @@ -4931,7 +4932,7 @@ fn lookup_array_strided_view_offsets_match_vm() { // are [0, 2, 1, 3]. let input = StaticArrayView { base_off: 0, - is_temp: false, + storage: ViewStorage::Curr, dims: SmallVec::from_slice(&[2, 2]), strides: SmallVec::from_slice(&[1, 2]), offset: 0, @@ -5104,11 +5105,11 @@ fn vm_allocate_available_oracle( avail: f64, data: &[f64], ) -> Vec { - let requests: Vec = (0..requests_view.to_runtime_view().size()) + let requests: Vec = (0..requests_view.to_runtime_view(0).size()) .map(|i| vm_view_element(requests_view, data, i)) .collect(); let n = requests.len(); - let pp_size = profile_view.to_runtime_view().size(); + let pp_size = profile_view.to_runtime_view(0).size(); let pp_values: Vec = (0..pp_size) .map(|i| vm_view_element(profile_view, data, i)) .collect(); @@ -5137,11 +5138,11 @@ fn vm_allocate_by_priority_oracle( supply: f64, data: &[f64], ) -> Vec { - let requests: Vec = (0..requests_view.to_runtime_view().size()) + let requests: Vec = (0..requests_view.to_runtime_view(0).size()) .map(|i| vm_view_element(requests_view, data, i)) .collect(); let n = requests.len(); - let priorities: Vec = (0..priority_view.to_runtime_view().size()) + let priorities: Vec = (0..priority_view.to_runtime_view(0).size()) .map(|i| vm_view_element(priority_view, data, i)) .collect(); let profiles: Vec<(f64, f64, f64, f64)> = (0..n) diff --git a/src/simlin-engine/src/wasmgen/module_tests.rs b/src/simlin-engine/src/wasmgen/module_tests.rs index d0c6b35be..8c6b5203d 100644 --- a/src/simlin-engine/src/wasmgen/module_tests.rs +++ b/src/simlin-engine/src/wasmgen/module_tests.rs @@ -737,6 +737,65 @@ fn compile_simulation_repeated_run_resets_previous_fallback() { ); } +/// The ARRAY twin of the test above, and the one thing the corpus gate cannot +/// reach (GH #995): an array-valued `PREVIOUS` is a VIEW over `prev_values`, and +/// a view read is a plain `f64.load` at a constant address -- nothing about it +/// consults `use_prev_fallback` unless the emitter puts a `select` there. +/// +/// On a FIRST run that omission is invisible: wasm linear memory starts zeroed, +/// so the snapshot region reads 0 anyway, which is exactly the fallback. It only +/// shows up on a second run, because the blob's `reset` deliberately does NOT +/// clear the snapshot regions (it sets the flag instead, which is all the scalar +/// `LoadPrev` needs). So the second run's step 0 would read the FIRST run's +/// final `prev_values` -- a plausible array of stale numbers, no diagnostic. +/// +/// `SUM(PREVIOUS(x[*]))` is 0 at t=0 and the previous step's total afterwards; +/// the stale reading is the first run's last total (42), which is what this pins +/// against. +#[test] +fn compile_simulation_repeated_run_resets_previous_fallback_for_an_array_view() { + let datamodel = crate::test_common::TestProject::new("prev_array_repeat") + .with_sim_time(0.0, 5.0, 1.0) + .indexed_dimension("d", 3) + .array_stock("x[d]", "10", &["grow"], &[], None) + .array_flow("grow[d]", "1", None) + .aux("x_prev_sum", "SUM(PREVIOUS(x[*]))", None) + .build_datamodel(); + + let sim = compile_sim(&datamodel, "main"); + let artifact = compile_simulation(&sim).expect("wasm codegen"); + + let runs = run_artifact_results_repeated(&artifact, 2); + let (first, second) = (&runs[0], &runs[1]); + assert_eq!( + first, second, + "second run() diverged from the first -- a PREVIOUS VIEW read the stale \ + snapshot region instead of the fallback" + ); + + let off = artifact + .layout + .var_offsets + .iter() + .find(|(name, _)| name == "x_prev_sum") + .map(|(_, off)| *off) + .expect("x_prev_sum in layout"); + assert_eq!( + second[off], 0.0, + "SUM(PREVIOUS(x[*])) at t0 on the second run must be the fallback 0, not \ + the first run's final total (42); got {}", + second[off] + ); + // ... and the step after t0 must be the real previous total (3 * 10), so the + // fallback is not being returned forever. + let n_slots = artifact.layout.n_slots; + assert_eq!( + second[n_slots + off], + 30.0, + "the step after the fallback must read the real snapshot" + ); +} + /// Regression (PR #620 review): a stock at an absolute slot offset >= 65536 /// must address its real slot under RK integration, not `off & 0xFFFF`. Such /// offsets are reachable in a large nested model (each submodel/SMOOTH/DELAY @@ -3785,3 +3844,125 @@ fn set_value_nonconstant_returns_error() { "VM must accept the overridable constant" ); } + +/// The wasm backend must broadcast a MIXED-SHAPE computed array operand the +/// way the VM does (GH #995). +/// +/// The lowering pass that materializes such an operand +/// (`compiler::array_operand`) shapes its temp by the JOIN of the arrays in it, +/// so `vals[d] + matrix[e,d]` iterates over the `[e,d]` shape and reads `vals` +/// broadcast down the rows. Both backends then have to place that narrower +/// source themselves -- the VM through `Opcode::LoadIterViewAt`'s dimension +/// matching, wasm through its own unrolled iteration -- and this is the only +/// row that exercises the disagreement, because the corpus fixture's operands +/// are all single-shaped. Both operand orders run, since the join is the thing +/// making them the same program. +#[test] +fn compile_simulation_mixed_shape_array_operand_matches_vm() { + for (name, eqn) in [ + ( + "mix_narrow_first", + "VECTOR SORT ORDER(vals[d] + matrix[e,d], 1)", + ), + ( + "mix_wide_first", + "VECTOR SORT ORDER(matrix[e,d] + vals[d], 1)", + ), + ] { + let datamodel = crate::test_common::TestProject::new(name) + .with_sim_time(0.0, 2.0, 1.0) + .indexed_dimension("d", 3) + .indexed_dimension("e", 2) + .array_with_ranges("vals[d]", vec![("1", "30"), ("2", "10"), ("3", "20")]) + .array_with_ranges( + "matrix[e,d]", + vec![ + ("1,1", "1"), + ("1,2", "2"), + ("1,3", "3"), + ("2,1", "10"), + ("2,2", "20"), + ("2,3", "30"), + ], + ) + .array_aux("out[e,d]", eqn) + .build_datamodel(); + + let sim = compile_sim(&datamodel, "main"); + let artifact = compile_simulation(&sim).expect("wasm codegen"); + let checked = assert_matches_vm(sim, &artifact); + assert!(checked > 0, "{name}: no variables compared"); + } +} + +/// The wasm twin of +/// `array_operand_materialization_tests::an_array_view_inside_a_module_instance_reads_that_instance`. +/// +/// Asserted against ABSOLUTE series rather than through `assert_matches_vm` +/// alone, because parity is exactly what this defect had: `views::ViewDesc` +/// mirrors the VM's addressing arm for arm, so when `PushStaticView` dropped the +/// instance's `module_off` the wasm emitter dropped it too and the two backends +/// agreed on the same wrong numbers. The parity check runs as well -- it is what +/// keeps the two addressing implementations from drifting once both are right. +#[test] +fn compile_simulation_arrayed_submodel_views_address_their_instance() { + let datamodel = crate::test_common::two_instance_arrayed_submodel_project(); + let sim = compile_sim(&datamodel, "main"); + let artifact = compile_simulation(&sim).expect("wasm codegen"); + let data = run_artifact_results(&artifact); + let n_slots = artifact.layout.n_slots; + + for (name, expected) in crate::test_common::two_instance_arrayed_submodel_expected() { + let off = artifact + .layout + .var_offsets + .iter() + .find(|(n, _)| n == name) + .map(|(_, o)| *o) + .unwrap_or_else(|| panic!("{name} missing from the wasm layout")); + for (c, want) in expected.iter().enumerate() { + let got = data[c * n_slots + off]; + assert!( + (got - want).abs() < 1e-9, + "{name} at chunk {c}: expected {want}, got {got}" + ); + } + } + + assert_matches_vm(sim, &artifact); +} + +/// The wasm twin of +/// `array_operand_materialization_tests::an_array_view_inside_a_nested_module_instance_reads_that_instance`. +/// +/// wasm reaches a nested instance by passing `module_off + decl.off` as the +/// child function's param 0, so the two hops must sum there exactly as they do +/// in the VM's recursive `eval`. Asserted against absolute series for the same +/// reason the one-hop wasm pin is. +#[test] +fn compile_simulation_nested_arrayed_submodel_views_address_their_instance() { + let datamodel = crate::test_common::nested_instance_arrayed_submodel_project(); + let sim = compile_sim(&datamodel, "main"); + let artifact = compile_simulation(&sim).expect("wasm codegen"); + let data = run_artifact_results(&artifact); + let n_slots = artifact.layout.n_slots; + + for (name, expected) in crate::test_common::nested_instance_arrayed_submodel_expected() { + let off = artifact + .layout + .var_offsets + .iter() + .find(|(n, _)| n == name) + .map(|(_, o)| *o) + .unwrap_or_else(|| panic!("{name} missing from the wasm layout")); + for (c, want) in expected.iter().enumerate() { + let got = data[c * n_slots + off]; + assert!( + (got - want).abs() < 1e-9, + "{name} at chunk {c}: expected {want}, got {got}" + ); + } + } + + assert_matches_vm(sim, &artifact); +} diff --git a/src/simlin-engine/src/wasmgen/vector.rs b/src/simlin-engine/src/wasmgen/vector.rs index c2f405f29..2e01359e7 100644 --- a/src/simlin-engine/src/wasmgen/vector.rs +++ b/src/simlin-engine/src/wasmgen/vector.rs @@ -301,16 +301,28 @@ fn push_all_valid(views: &[&ViewDesc], f: &mut Function) { /// view's `base_off`, NOT folding in its `offset`, which the caller already folds /// into the flat index). For a module-relative var view the runtime `module_off` /// addend is signalled via the returned `bool`. -fn view_storage_base(view: &ViewDesc, ctx: &EmitCtx) -> Result<(u64, bool), WasmGenError> { +fn view_storage_base(view: &ViewDesc, ctx: &EmitCtx) -> Result { + let bases = ctx.region_bases(); + let region = |base: u32| u64::from(base) + u64::from(view.base_off) * u64::from(SLOT_SIZE); match view.base { - ViewBase::CurrAbsolute => Ok(( - u64::from(ctx.curr_base) + u64::from(view.base_off) * u64::from(SLOT_SIZE), - false, - )), - ViewBase::CurrModuleRelative => Ok(( - u64::from(ctx.curr_base) + u64::from(view.base_off) * u64::from(SLOT_SIZE), - true, - )), + ViewBase::Curr => Ok(StorageBase { + base_byte: region(bases.curr), + module_relative: true, + prev_fallback_gated: false, + }), + // The snapshot regions share `curr`'s slot numbering; `prev` carries the + // `use_prev_fallback` gate, and `initial`'s phase branch is already + // folded into `bases.initial` (see `views::ViewBase`). + ViewBase::Prev => Ok(StorageBase { + base_byte: region(bases.prev), + module_relative: true, + prev_fallback_gated: true, + }), + ViewBase::Initial => Ok(StorageBase { + base_byte: region(bases.initial), + module_relative: true, + prev_fallback_gated: false, + }), ViewBase::Temp => { let temp_off = *ctx .ctx @@ -321,14 +333,24 @@ fn view_storage_base(view: &ViewDesc, ctx: &EmitCtx) -> Result<(u64, bool), Wasm "wasmgen: vector-op source references an out-of-range temp id".to_string(), ) })? as u64; - Ok(( - u64::from(ctx.temp_storage_base) + temp_off * u64::from(SLOT_SIZE), - false, - )) + Ok(StorageBase { + base_byte: u64::from(bases.temp_storage) + temp_off * u64::from(SLOT_SIZE), + module_relative: false, + prev_fallback_gated: false, + }) } } } +/// The result of [`view_storage_base`]: where a view's storage element 0 lives +/// and which runtime addends/gates a read of it needs. +#[derive(Clone, Copy)] +struct StorageBase { + base_byte: u64, + module_relative: bool, + prev_fallback_gated: bool, +} + // ── VectorSelect (vm.rs:2444-2502) ────────────────────────────────────────── /// Lower `VectorSelect`, mirroring `vm.rs:2444-2502`. The two operands are on the @@ -663,7 +685,7 @@ fn emit_vector_elm_map_body( .map(|sd| offset_view.dim_ids.iter().position(|od| od == sd)) .collect(); - let (src_base_byte, src_module_relative) = view_storage_base(source_view, ctx)?; + let src_storage = view_storage_base(source_view, ctx)?; let offset_val = ctx.vector_f64_locals[0]; let flat_i = ctx.vector_i32_locals[0]; @@ -728,7 +750,7 @@ fn emit_vector_elm_map_body( f.instruction(&f64_const(f64::NAN)); f.instruction(&Ins::Else); // source[flat_i]: base byte + flat_i*8 (+ module_off*8 if module-relative) - emit_storage_indexed_load(src_base_byte, src_module_relative, flat_i, ctx, f); + emit_storage_indexed_load(src_storage, flat_i, ctx, f); f.instruction(&Ins::End); f.instruction(&Ins::F64Store(memarg(temp_addr))); @@ -740,14 +762,22 @@ fn emit_vector_elm_map_body( /// constant `base_byte` and `flat_i` (an i32 local) is the runtime slot index: /// `f64.load[base_byte + (module_off? )*8 + flat_i*8]`. The constant `base_byte` /// rides in the `memarg.offset`; the runtime part is `(module_off + flat_i) * 8` -/// for a module-relative view, else `flat_i * 8`. -fn emit_storage_indexed_load( - base_byte: u64, - module_relative: bool, - flat_i: u32, - ctx: &EmitCtx, - f: &mut Function, -) { +/// for a module-relative view, else `flat_i * 8`. A PREVIOUS-region source +/// additionally gets the `use_prev_fallback` select. +fn emit_storage_indexed_load(storage: StorageBase, flat_i: u32, ctx: &EmitCtx, f: &mut Function) { + let StorageBase { + base_byte, + module_relative, + prev_fallback_gated, + } = storage; + // A PREVIOUS source reads the fallback 0 for every element until the first + // snapshot exists; the fallback operand goes on first so `select` (which + // yields its DEEPER operand when the condition is non-zero) picks it while + // the flag is set -- the same shape `emit_load_prev` uses for the scalar + // opcode, and the VM's `ChunkRegions::backing` `None` arm. + if prev_fallback_gated { + f.instruction(&Ins::F64Const(0.0.into())); + } if module_relative { push_module_relative_base(ctx, f); // module_off * 8 f.instruction(&Ins::LocalGet(flat_i)); @@ -760,6 +790,10 @@ fn emit_storage_indexed_load( f.instruction(&Ins::I32Mul); } f.instruction(&Ins::F64Load(memarg(base_byte))); + if prev_fallback_gated { + f.instruction(&Ins::GlobalGet(ctx.use_prev_fallback_global)); + f.instruction(&Ins::Select); + } } // ── VectorSortOrder (vm_vector_sort_order.rs:49-101) ───────────────────────── diff --git a/src/simlin-engine/src/wasmgen/views.rs b/src/simlin-engine/src/wasmgen/views.rs index e55f8c4a2..166f08d77 100644 --- a/src/simlin-engine/src/wasmgen/views.rs +++ b/src/simlin-engine/src/wasmgen/views.rs @@ -12,7 +12,7 @@ //! The VM resolves every array access through a runtime stack of [`RuntimeView`]s //! built and transformed by the `Push*View` / `View*` opcodes. Because every //! static view's geometry (base offset, dims, strides, offset, sparsity, -//! is_temp) is known at compile time, the wasm emitter maintains a *compile-time* +//! storage region) is known at compile time, the wasm emitter maintains a *compile-time* //! stack of [`ViewDesc`]s instead, mirroring the static parts of `RuntimeView` //! field-for-field and reproducing the `RuntimeView::apply_*` transforms in //! `apply_*` here. Element addressing then routes through a single source of @@ -21,26 +21,53 @@ //! //! [`RuntimeView`]: crate::bytecode::RuntimeView -use crate::bytecode::{ByteCodeContext, StaticArrayView}; +use crate::bytecode::{ByteCodeContext, StaticArrayView, ViewStorage}; /// Where a view's base address lives, mirroring how the VM resolves the base of /// a `RuntimeView` element read (`reduce_view` in `vm.rs`). #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub(crate) enum ViewBase { - /// `curr[base_off + ..]` at an *absolute* slot base. This is what - /// `PushStaticView` produces: `StaticArrayView::to_runtime_view` copies - /// `base_off` verbatim (no `module_off` added), so the byte address is - /// `curr_base + (base_off + flat) * 8` with no runtime addend. - CurrAbsolute, - /// `curr[module_off + base_off + ..]`. `PushVarViewDirect` folds the - /// runtime `module_off` into the base (`vm.rs`'s `PushVarViewDirect` arm), - /// so a read adds `module_off * 8` to the constant address. In the current - /// single-root scope `module_off == 0`, but the distinction is preserved so - /// Phase 7 can thread a real `module_off` without changing addressing. - CurrModuleRelative, - /// `temp_storage[temp_offsets[base_off] + ..]` (`is_temp`): the base is a - /// temp id, resolved against the `temp_storage` region via `temp_offsets`. + /// `curr[module_off + base_off + ..]`. Both view opcodes land here: + /// `PushVarViewDirect` builds the base from the variable's own slot offset, + /// and `PushStaticView` carries one baked by `resolve_static_view` out of the + /// FRAGMENT'S OWN model layout. Either way the offset is module-relative -- + /// the same offset `LoadVar` reads as `curr[module_off + off]` -- so the read + /// adds `module_off * 8` to the constant address. `module_off` is 0 for the + /// root instance and each sub-model instance's own slot base otherwise. + Curr, + /// `temp_storage[temp_offsets[base_off] + ..]`: the base is a temp id, + /// resolved against the `temp_storage` region via `temp_offsets`. The ONE + /// base `module_off` must not touch: temp storage is per-evaluation, not a + /// slab region, and every instance shares it while it runs. Temp, + /// `prev_values[module_off + base_off + ..]`: the array form of `PREVIOUS` + /// (GH #995), reading the same snapshot region the scalar `LoadPrev` does -- + /// at the same instance-relative address, since the region is an `n_slots` + /// copy of `curr`. Like `LoadPrev`, the read is gated on `use_prev_fallback`: + /// while it is set the element is the fallback `0`, which is the only + /// fallback an array-valued `PREVIOUS` may carry + /// (`codegen::is_default_previous_fallback`). + Prev, + /// `initial_values[module_off + base_off + ..]`: the array form of `INIT`. + /// Its "during the initials phase the snapshot does not exist yet, read + /// `curr`" branch is resolved at COMPILE time from `EmitCtx::step_part`, + /// exactly as `emit_load_initial` does, so it needs no runtime gate. + Initial, +} + +/// The byte offsets of the four regions a view's elements can live in, as one +/// argument so a new region is added in one place rather than at every +/// addressing call site. +/// +/// `initial` is already resolved for the program being emitted (`curr_base` +/// during the initials phase, the snapshot region otherwise) -- see +/// [`ViewBase::Initial`]. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub(crate) struct RegionBases { + pub curr: u32, + pub temp_storage: u32, + pub prev: u32, + pub initial: u32, } /// A single sparse-dimension mapping, mirroring @@ -89,16 +116,19 @@ pub(crate) struct ViewDesc { impl ViewDesc { /// Build a `ViewDesc` from a baked [`StaticArrayView`] (`PushStaticView`). /// - /// `StaticArrayView::to_runtime_view` copies `base_off` verbatim with no - /// `module_off`, so the base is [`ViewBase::CurrAbsolute`] for a variable - /// view and [`ViewBase::Temp`] when `is_temp`. + /// A static view's `base_off` is module-relative (`resolve_static_view` + /// reads it out of the fragment's own model layout), and + /// `StaticArrayView::to_runtime_view` adds the executing instance's + /// `module_off` to it for the three chunk-shaped regions. The wasm bases + /// mirror that, region for region. pub fn from_static(view: &StaticArrayView) -> Self { ViewDesc { base_off: view.base_off, - base: if view.is_temp { - ViewBase::Temp - } else { - ViewBase::CurrAbsolute + base: match view.storage { + ViewStorage::Curr => ViewBase::Curr, + ViewStorage::Temp => ViewBase::Temp, + ViewStorage::Prev => ViewBase::Prev, + ViewStorage::Initial => ViewBase::Initial, }, dims: view.dims.to_vec(), strides: view.strides.to_vec(), @@ -415,15 +445,12 @@ impl ViewDesc { /// addressing -- the unrolled reducer (Task 2), the iteration loop (Task 3), /// and Phase 6 all route through it. /// - /// - `CurrAbsolute`: `const = curr_base + (base_off + flat) * 8`, - /// `module_relative = false` (static views bake `module_off` in already). - /// - `Temp`: `const = temp_storage_base + (temp_offsets[base_off] + flat)*8`, - /// `module_relative = false`. - /// - `CurrModuleRelative`: `const = curr_base + (base_off + flat) * 8`, + /// - `Curr`/`Prev`/`Initial`: `const = + (base_off + flat) * 8`, /// `module_relative = true` (the caller adds `module_off * 8`). The VM - /// folds `module_off` into the base at `PushVarViewDirect` time; - /// in the single-root scope `module_off == 0`, so the read is the same as - /// `CurrAbsolute` today, but the flag keeps Phase 7 correct. + /// folds the same addend into the base at push time + /// (`StaticArrayView::to_runtime_view`, `PushVarViewDirect`). + /// - `Temp`: `const = temp_storage_base + (temp_offsets[base_off] + flat)*8`, + /// `module_relative = false` -- a temp id is not a slab slot. /// /// A dynamically-subscripted view (`runtime_off_local` set, Task 4) carries /// the runtime addend + validity flag in the returned [`ElementAddr`]; static @@ -431,12 +458,11 @@ impl ViewDesc { pub fn element_addr( &self, iter_idx: usize, - curr_base: u32, - temp_storage_base: u32, + bases: RegionBases, ctx: &ByteCodeContext, ) -> Option { let flat = self.flat_element_offset(iter_idx); - self.element_addr_for_flat(flat, curr_base, temp_storage_base, ctx) + self.element_addr_for_flat(flat, bases, ctx) } /// Like [`element_addr`](Self::element_addr) but for an *already-computed* @@ -448,28 +474,43 @@ impl ViewDesc { pub fn element_addr_for_flat( &self, flat: usize, - curr_base: u32, - temp_storage_base: u32, + bases: RegionBases, ctx: &ByteCodeContext, ) -> Option { let flat = flat as u64; - let (const_byte_offset, module_relative) = match self.base { - ViewBase::CurrAbsolute => ( - u64::from(curr_base) + (u64::from(self.base_off) + flat) * 8, - false, - ), - ViewBase::CurrModuleRelative => ( - u64::from(curr_base) + (u64::from(self.base_off) + flat) * 8, + let (const_byte_offset, module_relative, prev_fallback_gated) = match self.base { + ViewBase::Curr => ( + u64::from(bases.curr) + (u64::from(self.base_off) + flat) * 8, true, + false, ), ViewBase::Temp => { let temp_off = *ctx.temp_offsets.get(self.base_off as usize)? as u64; - (u64::from(temp_storage_base) + (temp_off + flat) * 8, false) + ( + u64::from(bases.temp_storage) + (temp_off + flat) * 8, + false, + false, + ) } + // The snapshot regions are `n_slots` wide and share `curr`'s slot + // numbering, so only the base changes. `prev` additionally carries + // the runtime fallback gate; `initial`'s phase branch was already + // resolved into `bases.initial`. + ViewBase::Prev => ( + u64::from(bases.prev) + (u64::from(self.base_off) + flat) * 8, + true, + true, + ), + ViewBase::Initial => ( + u64::from(bases.initial) + (u64::from(self.base_off) + flat) * 8, + true, + false, + ), }; Some(ElementAddr { const_byte_offset, module_relative, + prev_fallback_gated, runtime_off_local: self.runtime_off_local, valid_local: self.valid_local, }) @@ -490,6 +531,11 @@ impl ViewDesc { pub(crate) struct ElementAddr { pub const_byte_offset: u64, pub module_relative: bool, + /// True for a [`ViewBase::Prev`] element: the load is wrapped in the + /// same `select` on `use_prev_fallback` that the scalar `LoadPrev` emits, so + /// the element reads `0` until the first snapshot exists. The VM reaches the + /// identical value through `ChunkRegions::backing`'s `None` arm. + pub prev_fallback_gated: bool, pub runtime_off_local: Option, pub valid_local: Option, } @@ -506,7 +552,12 @@ mod tests { fn to_runtime_view(d: &ViewDesc) -> RuntimeView { RuntimeView { base_off: d.base_off, - is_temp: matches!(d.base, ViewBase::Temp), + storage: match d.base { + ViewBase::Temp => ViewStorage::Temp, + ViewBase::Prev => ViewStorage::Prev, + ViewBase::Initial => ViewStorage::Initial, + ViewBase::Curr => ViewStorage::Curr, + }, dims: SmallVec::from_slice(&d.dims), strides: SmallVec::from_slice(&d.strides), offset: d.offset, @@ -539,10 +590,22 @@ mod tests { } } + /// Region bases for the addressing tests: only `curr` and `temp_storage` + /// vary here, and the snapshot bases are given distinct sentinel values so a + /// misrouted base shows up as an unexpected address rather than as 0. + fn bases(curr: u32, temp_storage: u32) -> RegionBases { + RegionBases { + curr, + temp_storage, + prev: 900_000, + initial: 800_000, + } + } + fn dense(base_off: u32, dims: &[u16]) -> ViewDesc { ViewDesc::contiguous( base_off, - ViewBase::CurrAbsolute, + ViewBase::Curr, dims.to_vec(), vec![0u16; dims.len()], ) @@ -626,29 +689,48 @@ mod tests { assert_flat_matches_vm(&d); } + /// Every [`ViewBase`] arm's region, its `module_off` verdict, and its + /// `PREVIOUS`-fallback gate -- one row per variant, derived from the enum + /// rather than from the arms that happened to have tests. + /// + /// The `module_relative` column is the whole of the GH #995 module-instance + /// fix on this side: the three chunk-shaped regions are addressed by the + /// executing instance's slot base (the VM adds the same addend in + /// `StaticArrayView::to_runtime_view`), and `Temp` is not, because a temp id + /// is not a slab slot. #[test] - fn element_addr_curr_absolute_const() { - let d = dense(2, &[3]); - let ctx = ByteCodeContext::default(); - // element 1 at curr_base=0: (base_off 2 + flat 1) * 8 = 24. - let a = d.element_addr(1, 0, 0, &ctx).unwrap(); - assert_eq!(a.const_byte_offset, 24); - assert!(!a.module_relative); - // A static view carries no runtime addend or validity gate. - assert_eq!(a.runtime_off_local, None); - assert_eq!(a.valid_local, None); - } - - #[test] - fn element_addr_curr_module_relative_flag() { - let d = ViewDesc::contiguous(2, ViewBase::CurrModuleRelative, vec![3], vec![0]); - let ctx = ByteCodeContext::default(); - let a = d.element_addr(1, 0, 0, &ctx).unwrap(); - assert_eq!(a.const_byte_offset, 24); - assert!( - a.module_relative, - "var views carry a runtime module_off addend" - ); + fn element_addr_covers_every_view_base() { + let mut ctx = ByteCodeContext::default(); + ctx.set_temp_info(vec![0, 4], 8); + // element 1 of a 3-element view based at slot 2: flat 3 slots = 24 bytes + // past the region base. + for (base, region_base, module_relative, gated) in [ + (ViewBase::Curr, 0u64, true, false), + (ViewBase::Prev, 900_000, true, true), + (ViewBase::Initial, 800_000, true, false), + // A temp base is a temp ID, so its region offset comes from + // `temp_offsets[2]`... which only has two entries here; use id 1 + // (offset 4) and expect `(4 + 1) * 8` past the temp region. + (ViewBase::Temp, 0, false, false), + ] { + let base_off = if base == ViewBase::Temp { 1 } else { 2 }; + let d = ViewDesc::contiguous(base_off, base, vec![3], vec![0]); + let a = d.element_addr(1, bases(0, 500_000), &ctx).unwrap(); + let expect = if base == ViewBase::Temp { + 500_000 + (4 + 1) * 8 + } else { + region_base + 24 + }; + assert_eq!(a.const_byte_offset, expect, "{base:?}: region + offset"); + assert_eq!(a.module_relative, module_relative, "{base:?}: module_off"); + assert_eq!( + a.prev_fallback_gated, gated, + "{base:?}: PREVIOUS fallback gate" + ); + // A static view carries no runtime addend or validity gate. + assert_eq!(a.runtime_off_local, None, "{base:?}"); + assert_eq!(a.valid_local, None, "{base:?}"); + } } #[test] @@ -657,7 +739,7 @@ mod tests { ctx.set_temp_info(vec![0, 4], 8); let d = ViewDesc::contiguous(1, ViewBase::Temp, vec![2], vec![0]); // temp_storage_base = 1000; temp 1 offset = 4; element 1 -> (4+1)*8 = 40. - let a = d.element_addr(1, 0, 1000, &ctx).unwrap(); + let a = d.element_addr(1, bases(0, 1000), &ctx).unwrap(); assert_eq!(a.const_byte_offset, 1000 + 40); assert!(!a.module_relative); } @@ -670,7 +752,7 @@ mod tests { d.runtime_off_local = Some(9); d.valid_local = Some(7); let ctx = ByteCodeContext::default(); - let a = d.element_addr(0, 0, 0, &ctx).unwrap(); + let a = d.element_addr(0, bases(0, 0), &ctx).unwrap(); // Element 0: const base is just curr_base + base_off*8 = 0; the runtime // index offset rides in local 9, the validity in local 7. assert_eq!(a.const_byte_offset, 0); @@ -694,7 +776,7 @@ mod tests { /// Build a `ViewDesc` with explicit dims/dim_ids (row-major contiguous). fn view_with_dim_ids(dims: &[u16], dim_ids: &[u16]) -> ViewDesc { - ViewDesc::contiguous(0, ViewBase::CurrAbsolute, dims.to_vec(), dim_ids.to_vec()) + ViewDesc::contiguous(0, ViewBase::Curr, dims.to_vec(), dim_ids.to_vec()) } #[test] diff --git a/src/simlin-engine/tests/integration/ltm_array_agg.rs b/src/simlin-engine/tests/integration/ltm_array_agg.rs index 07f862b7f..ae05734ed 100644 --- a/src/simlin-engine/tests/integration/ltm_array_agg.rs +++ b/src/simlin-engine/tests/integration/ltm_array_agg.rs @@ -4532,33 +4532,47 @@ fn whole_rhs_bare_reducer_stays_scored() { } // --------------------------------------------------------------------------- -// GH #758: declined element-mapped sliced reducer -> loud unscoreable edge +// GH #758: a declined sliced reducer -> loud unscoreable edge // --------------------------------------------------------------------------- -/// The GH #758 fixture: an inline sliced reducer over an ELEMENT-mapped -/// dimension pair -- `growth[State] = 1 + SUM(matrix[State,*])` where -/// `matrix` is declared over `Region` and `State` carries an explicit -/// element map to `Region` (not a positional correspondence). -/// `mapped_element_correspondence` declines it (the GH #756 -/// positional-only gate), so the reducer is NOT hoisted and the -/// `matrix → growth` reference stays on the conservative path. The -/// feedback loops close through `pop → matrix → growth → SUM(growth[*]) -/// → inflow → pop`, so every enumerated loop traverses the declined edge. +/// The GH #758 fixture: an inline sliced reducer over a dimension pair with NO +/// declared correspondence -- `growth[State] = 1 + SUM(matrix[State,*])` where +/// `matrix` is declared over `Region`, and `State` has DISJOINT element names +/// and no mapping to it. +/// +/// The model COMPILES because `State` is the dimension the equation ITERATES, +/// so `ast::expr3`'s Pass 1 folds the index to that dimension's ordinal and it +/// indexes `Region`'s storage raw -- a POSITIONAL read consulting neither names +/// nor mappings, and the shape +/// `mapped_reference_semantics_tests::no_mapping_equal_cardinality` measures. +/// `build_view_from_ops` is never reached. `positional_correspondence` +/// nonetheless declines, because GH #527's rule is that the DESCRIBED diagonal +/// follows a correspondence the MODEL declares; the reducer is therefore NOT +/// hoisted and the `matrix → growth` reference stays on the conservative path. +/// The feedback loops close through +/// `pop → matrix → growth → SUM(growth[*]) → inflow → pop`, so every enumerated +/// loop traverses the declined edge. +/// +/// The element names are deliberately disjoint from `Region`'s. An earlier +/// revision reused `Region`'s names, which made the read look name-matched when +/// it is positional -- true only by the coincidence that both lists were +/// declared in the same order. +/// +/// This fixture used an explicit ELEMENT MAP until GH #997. That pair is no +/// longer declined -- the same ordinal fold applies, so it hoists with +/// positional slots (`element_mapped_sliced_reducer_hoists_and_scores_its_loops` +/// asserts the recovery). The undeclared pair keeps the decline, so the GH #758 +/// contract is pinned by a shape that still reaches it. /// /// With `with_drain`, a second, independent loop `pop → drain → pop` /// (A2A over Region, not traversing the declined edge) is added so tests /// can pin that the degradation is surgical. -fn gh758_element_mapped_fixture(with_drain: bool) -> datamodel::Project { - let mut p = TestProject::new("gh758_element_mapped") +fn gh758_declined_slice_fixture(with_drain: bool) -> datamodel::Project { + let mut p = TestProject::new("gh758_declined_slice") .with_sim_time(0.0, 8.0, 1.0) .named_dimension("Region", &["west", "east"]) .named_dimension("D2", &["x", "y"]) - .named_dimension_with_element_mapping( - "State", - &["CA", "NY"], - "Region", - &[("CA", "east"), ("NY", "west")], - ) + .named_dimension("State", &["ca", "ny"]) .array_aux_direct( "matrix", vec!["Region".into(), "D2".into()], @@ -4591,7 +4605,7 @@ fn assembly_warnings( .collect() } -/// GH #758: the declined element-mapped sliced-reducer edge must degrade +/// GH #758: the declined sliced-reducer edge must degrade /// LOUDLY -- one Warning naming the edge, NO link-score variable, and NO /// loop scores through it -- instead of emitting a broken-by-construction /// scalar link score (a scalar equation referencing the arrayed `matrix` / @@ -4599,8 +4613,8 @@ fn assembly_warnings( /// score through the edge into a warned 0-stub (17 Assembly warnings on /// this fixture before the fix). #[test] -fn declined_element_mapped_reducer_edge_skips_loudly() { - let project = gh758_element_mapped_fixture(false); +fn declined_sliced_reducer_edge_skips_loudly() { + let project = gh758_declined_slice_fixture(false); let mut db = SimlinDb::default(); let sync = sync_from_datamodel_incremental(&mut db, &project, None); @@ -4677,13 +4691,249 @@ fn declined_element_mapped_reducer_edge_skips_loudly() { } } +/// GH #997: the shape the GH #758 fixture USED to be -- the same sliced reducer +/// over an EXPLICIT element-mapped pair -- now hoists, and every loop through it +/// scores. +/// +/// `SUM(matrix[State,*])` names the dimension the equation ITERATES, which +/// `ast::expr3`'s Pass 1 folds to an ordinal +/// (`mapped_reference_semantics_tests`' `(Permuted, IteratedDim)` cell, measured +/// against the VM), so the slots are POSITIONAL and the declared map is not +/// consulted. The map here is the reverse permutation (CA -> east), so the +/// element-graph assertions below distinguish the two rules rather than passing +/// on either. +#[test] +fn element_mapped_sliced_reducer_hoists_and_scores_its_loops() { + let project = TestProject::new("gh997_element_mapped") + .with_sim_time(0.0, 8.0, 1.0) + .named_dimension("Region", &["west", "east"]) + .named_dimension("D2", &["x", "y"]) + .named_dimension_with_element_mapping( + "State", + &["CA", "NY"], + "Region", + &[("CA", "east"), ("NY", "west")], + ) + .array_aux_direct( + "matrix", + vec!["Region".into(), "D2".into()], + "pop[Region] * 0.05", + None, + ) + .array_aux("growth[State]", "1 + SUM(matrix[State, *])") + .array_flow("inflow[Region]", "SUM(growth[*]) * 0.01", None) + .array_stock("pop[Region]", "100", &["inflow"], &[], None) + .build_datamodel(); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + set_project_ltm_enabled(&mut db, sync.project, true); + let ltm = model_ltm_variables(&db, sync.models["main"].source_model, sync.project); + let names: Vec<&str> = ltm.vars.iter().map(|v| v.name.as_str()).collect(); + + // The reducer is hoisted, and the source rows land on the POSITIONAL slot: + // `west` is Region's first element, `ca` is State's first. The declared map + // says the opposite, so these two names are the discriminator. + for (row, slot) in [("west", "ca"), ("east", "ny")] { + for d2 in ["x", "y"] { + let want = format!( + "{LINK_SCORE_PREFIX}matrix[{row},{d2}]\u{2192}$\u{205A}ltm\u{205A}agg\u{205A}0[{slot}]" + ); + assert!( + names.contains(&want.as_str()), + "expected {want:?}; got: {names:?}" + ); + } + } + + // No decline: the GH #758 warning is gone and the loops score. + let warnings = assembly_warnings(&db, sync.project); + assert!(warnings.is_empty(), "got: {warnings:?}"); + assert!( + names.iter().any(|n| n.starts_with(LOOP_SCORE_PREFIX)), + "loops through the hoisted reducer must score; got: {names:?}" + ); + + // And the scores are real numbers, not stubs. + let compiled = compile_project_incremental(&db, sync.project, "main") + .expect("LTM-enabled compilation should succeed"); + let mut vm = Vm::new(compiled).expect("VM construction should succeed"); + vm.run_to_end().expect("VM simulation should run"); + let results = vm.into_results(); + for name in ltm_score_var_names(&results) { + let var = ltm_var(<m.vars, &name); + let base = offset_of(&results, &name); + for slot in 0..slot_count(var, &project.dimensions) { + let series = series_at(&results, base + slot); + assert!( + series.iter().all(|v| v.is_finite()), + "score {name} slot {slot} must stay finite; got {series:?}" + ); + } + } +} + +// --------------------------------------------------------------------------- +// GH #997 blocker 1: EDGES may be a union, SCORES may not +// --------------------------------------------------------------------------- + +/// The `x[State] -> target[State]` loop fixture used by the two tests below, +/// with the `State`/`Region` correspondence supplied by the caller. +/// +/// `target[State] = x[State] * 1.02` reads the ITERATED spelling (positional); +/// `x[Region] = level[Region] * 0.5` and a stock close the loop, so every +/// element edge of the mapped pair sits on a cycle and would carry a loop score +/// if the edge were scored. +fn gh997_mapped_loop_fixture(state: datamodel::Dimension) -> datamodel::Project { + TestProject::new("gh997_mapped_loop") + .with_sim_time(0.0, 6.0, 1.0) + .named_dimension("Region", &["a", "b"]) + .with_dimension(state) + .array_aux_direct("x", vec!["Region".into()], "level[Region] * 0.5", None) + .array_aux_direct("target", vec!["State".into()], "x[State] * 1.02", None) + .array_flow("inflow[Region]", "SUM(target[*]) * 0.01", None) + .array_stock("level[Region]", "100", &["inflow"], &[], None) + .build_datamodel() +} + +/// A mapped pair whose two spellings DISAGREE must NOT get an arrayed link +/// score, even though its element edges are the union of both diagonals. +/// +/// The permuted element map (s1↦b, s2↦a) is one-to-one, so the union carries +/// TWO source elements per target element: the positional diagonal (s1→a) and +/// the map's (s1→b). A Bare A2A score has ONE slot per target element and +/// `ltm_finding::expand_a2a_link_offsets` maps both union edges onto it, so the +/// phantom from-node would read the real edge's non-zero score -- a compilable, +/// confidently wrong number, which GH #758 treats as worse than none. +/// +/// `db::analysis::mapped_pair_projects_uniquely` is the gate. Before it, this +/// fixture emitted one arrayed score and ZERO warnings where the pre-GH #997 +/// tree emitted none and two; it now takes the same loud skip the pre-#997 tree +/// did, for a reason stated in terms of the score's slot rather than of the +/// correspondence's existence. +#[test] +fn a_disagreeing_mapped_pair_is_denied_the_arrayed_score() { + let mut state = datamodel::Dimension::named( + "State".to_string(), + vec!["s1".to_string(), "s2".to_string()], + ); + state.mappings = vec![datamodel::DimensionMapping { + target: "Region".to_string(), + element_map: vec![ + ("s1".to_string(), "b".to_string()), + ("s2".to_string(), "a".to_string()), + ], + }]; + let project = gh997_mapped_loop_fixture(state); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + set_project_ltm_enabled(&mut db, sync.project, true); + let ltm = model_ltm_variables(&db, sync.models["main"].source_model, sync.project); + let names: Vec<&str> = ltm.vars.iter().map(|v| v.name.as_str()).collect(); + + assert!( + !names + .iter() + .any(|n| n.contains("link_score\u{205A}x\u{2192}target")), + "a disagreeing mapped pair must emit NO x->target link score; got: {names:?}" + ); + // The decline must be LOUD, and the assertion must name the EDGE rather + // than two substrings a dozen unrelated messages contain. + let warnings = assembly_warnings(&db, sync.project); + assert!( + warnings.iter().any(|w| match &w.error { + DiagnosticError::Assembly(m) => + m.contains("LTM link score for edge x -> target could not be computed"), + _ => false, + }), + "the decline must be LOUD -- one warning naming the edge; got: {warnings:?}" + ); + + // And the consequence the gate exists for: the loop through the denied edge + // must drop, not score. Without the gate this fixture emits six loop scores + // whose attribution runs through a phantom from-node reading the real + // edge's series out of the shared slot. + assert!( + !names + .iter() + .any(|n| n.starts_with("$\u{205A}ltm\u{205A}loop_score\u{205A}")), + "no loop score may survive through the denied edge; got: {names:?}" + ); + + // The never-fewer-edges direction is untouched: the element graph still + // emits BOTH diagonals. The gate withholds the score, not the edges. + let edges = model_element_causal_edges(&db, sync.models["main"].source_model, sync.project); + for (src, tgt) in [("a", "s1"), ("b", "s1"), ("a", "s2"), ("b", "s2")] { + assert!( + edges + .edges + .get(&format!("x[{src}]")) + .is_some_and(|t| t.contains(&format!("target[{tgt}]"))), + "element edge x[{src}] -> target[{tgt}] must survive the score gate; \ + got: {:?}", + edges.edges.get(&format!("x[{src}]")) + ); + } +} + +/// The companion: a pair whose two spellings AGREE keeps the arrayed score. +/// +/// A plain positional `maps_to` is the shape every pre-GH #997 mapped edge had, +/// so this is what the singleton gate must not cost. The union is a singleton +/// per target element, the retarget fires, and the score is arrayed over +/// `State` with the diagonal element edges. +#[test] +fn an_agreeing_mapped_pair_keeps_the_arrayed_score() { + let mut state = datamodel::Dimension::named( + "State".to_string(), + vec!["s1".to_string(), "s2".to_string()], + ); + state.set_maps_to("Region".to_string()); + let project = gh997_mapped_loop_fixture(state); + + let mut db = SimlinDb::default(); + let sync = sync_from_datamodel_incremental(&mut db, &project, None); + set_project_ltm_enabled(&mut db, sync.project, true); + let ltm = model_ltm_variables(&db, sync.models["main"].source_model, sync.project); + + let score = ltm + .vars + .iter() + .find(|v| v.name.contains("link_score\u{205A}x\u{2192}target")) + .unwrap_or_else(|| { + panic!( + "an agreeing mapped pair must keep its arrayed score; got: {:?}", + ltm.vars.iter().map(|v| &v.name).collect::>() + ) + }); + assert_eq!( + score.dimensions, + vec!["State".to_string()], + "the score is retargeted to the TARGET's dimensions" + ); + let warnings = assembly_warnings(&db, sync.project); + assert!(warnings.is_empty(), "got: {warnings:?}"); + + // And the edges are the single diagonal, not a union. + let edges = model_element_causal_edges(&db, sync.models["main"].source_model, sync.project); + assert_eq!( + edges.edges.get("x[a]").map(|t| t.contains("target[s1]")), + Some(true) + ); + assert_eq!( + edges.edges.get("x[a]").map(|t| t.contains("target[s2]")), + Some(false) + ); +} + /// GH #758 (surgical degradation): a second feedback loop that does NOT /// traverse the unscoreable edge keeps its real loop score while the /// doomed loops are dropped -- the skip is per-loop, not a blanket /// collapse of LTM output. #[test] -fn declined_element_mapped_reducer_keeps_unaffected_loops() { - let project = gh758_element_mapped_fixture(true); +fn declined_sliced_reducer_keeps_unaffected_loops() { + let project = gh758_declined_slice_fixture(true); let mut db = SimlinDb::default(); let sync = sync_from_datamodel_incremental(&mut db, &project, None); @@ -5974,9 +6224,9 @@ fn per_element_source_read_as_a_lookup_table_by_constant_expr_keeps_its_scores() /// dropped: exactly the same silent-zero outcome the un-mapped twin regressed into, /// reached by a different route. /// -/// The fix is that the pin asks `per_element_row_for_target` -- hence -/// `DimensionsContext::mapped_element_correspondence` -- which source element an -/// axis reads, instead of comparing names. That is the same derivation the score's +/// The fix is that the pin asks `per_element_row_for_target` -- hence the +/// spelling-keyed correspondence on `DimensionsContext` -- which source element +/// an axis reads, instead of comparing names. That is the same derivation the score's /// NAME comes from, which is why the assertions below can pair `s1` with `a` and /// `s2` with `b` and expect the name and the pinned row to agree. #[test] @@ -6849,7 +7099,8 @@ fn per_element_body_with_iterated_other_dep_scores() { /// GH #525 (T6 review corner pin): a MAPPED `PerElement` reference -- /// `mid[State] = pop[State, young] * 0.05` over `pop[Region, Age]` with a /// positional `State→Region` mapping -- exercises -/// `per_element_row_for_target`'s `mapped_element_correspondence` arm: the +/// `per_element_row_for_target`'s `AxisRead::Iterated` arm, hence +/// `positional_correspondence`: the /// row's Region element is the positional preimage of the target's State /// element (s1↔r1, s2↔r2), so the emitted names carry the SOURCE-dim row /// and the diagonal only. @@ -11234,9 +11485,12 @@ fn gh754_lower_dim_feeder_loop_discoverable_in_discovery_mode() { /// `pop[]` (e.g. `pop[r1]`) in lockstep with the element /// graph, so the mapped loop is discoverable. /// -/// (Only POSITIONAL mappings reach here: an element-mapped pair is declined -/// upstream by `link_score_dimensions` -- no Bare A2A score is emitted, the -/// GH #756 positional-only gate -- so no phantom can be minted for it.) +/// (Only pairs whose two reference spellings AGREE reach here -- every +/// positional mapping, and since GH #997 a many-to-one element map too. A pair +/// whose spellings DISAGREE is declined upstream by `link_score_dimensions` +/// via `db::analysis::mapped_pair_projects_uniquely`, precisely because +/// `expand_same_element`'s union would put two from-nodes on one score slot, +/// so no phantom can be minted for it either.) #[test] fn gh754_mapped_feeder_loop_discoverable_in_discovery_mode() { let project = TestProject::new("gh754_mapped_feeder") diff --git a/src/simlin-engine/tests/integration/simulate.rs b/src/simlin-engine/tests/integration/simulate.rs index 340fae5cd..4a128751d 100644 --- a/src/simlin-engine/tests/integration/simulate.rs +++ b/src/simlin-engine/tests/integration/simulate.rs @@ -1701,6 +1701,31 @@ fn simulates_array_broadcast() { simulate_path("../../test/array_broadcast/array_broadcast.xmile"); } +/// GH #995: an array-producing builtin whose array operand is COMPUTED rather +/// than a plain reference. Codegen consumes such an operand as a view over +/// storage, so the lowering must evaluate it into a temp first; before #995 +/// these equations did not compile at all, in an ordinary apply-to-all model +/// with LTM disabled. Gates VECTOR SORT ORDER, RANK and VECTOR ELM MAP against +/// hand-computed values, plus a control (`order_ref`) that sorts the same +/// array through a named variable -- the operand shape that always worked. +#[test] +fn simulates_vector_computed_operand() { + simulate_path("../../test/vector_computed_operand/vector_computed_operand.xmile"); +} + +/// GH #995 phase C3: an array-valued `PREVIOUS()` / `INIT()` operand. Both +/// compile to a VIEW over one of the VM's snapshot buffers rather than to a +/// per-element opcode, so this gates the view arithmetic on the snapshot regions +/// end to end -- VM, protobuf and XMILE round-trips, and wasm-backend parity, +/// where an `Unsupported` outcome is a hard failure. Every source array moves +/// with time, so reading `curr` where `prev` was meant is a different answer at +/// every step but the first; `order_ref` is the control, sorting the same freeze +/// captured PER ELEMENT (the route that always worked). +#[test] +fn simulates_vector_snapshot_operand() { + simulate_path("../../test/vector_snapshot_operand/vector_snapshot_operand.xmile"); +} + #[test] fn simulates_modules() { simulate_path("../../test/modules_hares_and_foxes/modules_hares_and_foxes.stmx"); @@ -2159,6 +2184,23 @@ fn simulates_vector_xmile_genuine() { simulate_path("../../test/sdeverywhere/models/vector/vector.xmile"); } +/// The MDL twin of `simulates_vector_xmile_genuine`: the SAME model, through +/// the MDL importer instead of the XMILE reader, against the same real-Vensim +/// `vector.dat`. +/// +/// This gate did not exist, and its absence hid a defect for as long as the +/// fixture has. The importer expanded every subscripted LHS into per-element +/// slots, so `y[DimA] = VECTOR ELM MAP(x[three], (DimA - 1))` -- legal Vensim, +/// and correct through the XMILE reader -- became three slots each carrying an +/// unresolvable `DimA - 1` and failed to compile. Running only the XMILE file +/// meant the two readers were never compared on a model that uses dimension +/// arithmetic. `mdl::convert::apply_to_all_tests` pins the mechanism; this pins +/// it against genuine Vensim output. +#[test] +fn simulates_vector_mdl_genuine() { + simulate_mdl_path("../../test/sdeverywhere/models/vector/vector.mdl"); +} + #[test] fn simulates_lookup_arrayed() { simulate_path("../../test/lookup_arrayed/lookup_arrayed.xmile"); @@ -4267,6 +4309,8 @@ static ALL_INCREMENTALLY_COMPILABLE_MODELS: &[&str] = &[ "../../test/array_sum_expr/array_sum_expr.xmile", "../../test/array_multi_source/array_multi_source.xmile", "../../test/array_broadcast/array_broadcast.xmile", + "../../test/vector_computed_operand/vector_computed_operand.xmile", + "../../test/vector_snapshot_operand/vector_snapshot_operand.xmile", "../../test/modules_hares_and_foxes/modules_hares_and_foxes.stmx", "../../test/modules2/modules2.xmile", "../../test/circular-dep-1/model.stmx", @@ -6031,8 +6075,13 @@ fn corpus_clearn_macros_import() { /// /// Layout impact (the resource this gate protects -- #654's VM limit of 65,536 /// u16 result slots, NOT `wasmgen::lower`'s unrelated `MAX_UNROLL_UNITS`): the -/// per-step result-row width is **29,717 slots**, 45% of the ceiling, with -/// 35,819 free. +/// per-step result-row width is **30,123 slots**, 46% of the ceiling, with +/// 35,413 free. Both numbers come from +/// `examples/ltm_slot_width.rs`, so re-deriving them is a command rather than a +/// reconstruction -- and they are the CURRENT totals: the transition records +/// below quote earlier values as the left-hand side of a move, which is what +/// they are for. This header is the budget statement against the #654 ceiling, +/// so a stale figure here is the one that misleads. /// /// It last moved twice in the GH #995 burndown. The array-freeze /// materializer took the count 6,800 -> 6,858 (+58 content-named @@ -6053,7 +6102,26 @@ fn corpus_clearn_macros_import() { /// MAP's view-position source argument) compile via 4 whole-dep freeze /// helpers, leaving ZERO failing LTM fragments on C-LEARN. /// -/// It last moved UPWARD, 30,416 -> 30,947 (+531), when GH #996 stopped the axis +/// It last moved with GH #997's spelling-keyed correspondence, 6,757 -> 6,848 +/// (+91) / width 29,717 -> 29,808 (+91, one scalar slot per new variable). +/// +/// The +91 is arithmetic, not a bare observation: 13 previously-declined edges +/// x 7 `COP` elements = 91 per-element scalar link scores, one per (edge, +/// target element). Each is scalar, so the width moves by the same 91. All 91 +/// are the class-D shape #997 recovers: C-LEARN reads five +/// `X Aggregated[Aggregated Regions]` variables inside `COP`-iterating +/// equations, a subscript naming the SOURCE's own dimension across a +/// many-to-one element map. Before #997 one correspondence answered for both +/// reference spellings and declined that one, so the per-element completeness +/// guard dropped 13 edges outright (`aggregate_switch -> ff_stop_growth_year` +/// and siblings) and the five source edges themselves collapsed onto the +/// conservative cross-product. Now they pin through the declared map and score +/// per (source row, target element); the element graph LOST 210 phantom edges +/// (15,826 -> 15,616) in the same change, which is the other half of the same +/// fix and costs no slots. Measured with `examples/ltm_declined_edges.rs` +/// (13 unprojectable-dep declines -> 0) and `examples/ltm_edge_coverage.rs`. +/// +/// It moved UPWARD before that, 30,416 -> 30,947 (+531), when GH #996 stopped the axis /// allocator stealing a name-matched slot: 135 link scores that the per-element /// completeness guard had been declining now pin their indices and are emitted /// again. An earlier note here argued that DOWNWARD was the safe direction, and @@ -6076,6 +6144,20 @@ fn corpus_clearn_macros_import() { /// terms, which is why the emission is accepted; if the margin were tight, the /// right move would be to sequence #996 behind #995 instead. /// +/// It moved UPWARD again, 6,848 -> 7,163 (+315) and 29,808 -> 30,123 slots, when +/// the MDL importer stopped exploding a single apply-to-all equation into N +/// identical per-element slots (`mdl::convert::apply_to_all_tests`). C-LEARN is +/// imported from MDL, so its element-mapped aggregation variables +/// (`annual_reduction_aggregated`, `annual_reduction_semi_agg`, ...) now arrive +/// as `Ast::ApplyToAll` and reach the per-element mapped-row emitter this branch +/// fixed, instead of sitting in per-slot equations whose sites contribute only +/// to their own element. The change is STRICTLY ADDITIVE -- measured by diffing +/// the emitted name sets with `examples/ltm_var_dump.rs`: 315 added, **0 +/// removed** -- and every added name is a `[] -> []` +/// mapped score, i.e. exactly the GH #997 shape. Declines are unchanged (5, +/// same names, all rank-like-partial) and the margin is still wide: 35,413 free +/// against the 65,536-slot ceiling. +/// /// The pin below catches emission changes in EITHER direction, and re-deriving /// it means re-measuring BOTH numbers, not just the count. #[test] @@ -6101,7 +6183,7 @@ fn clearn_ltm_var_count_guardrail() { }) .sum(); assert_eq!( - total, 6757, + total, 7163, "C-LEARN's emitted LTM var count moved; if this is an intentional \ emission change, re-derive the layout-slot impact (the #654 \ ceiling) and update this pin with the new numbers" diff --git a/test/vector_computed_operand/output.csv b/test/vector_computed_operand/output.csv new file mode 100644 index 000000000..0360dc8f0 --- /dev/null +++ b/test/vector_computed_operand/output.csv @@ -0,0 +1,4 @@ +Time,adjusted[North],adjusted[South],adjusted[West],order[North],order[South],order[West],order_ref[North],order_ref[South],order_ref[West],ranks[North],ranks[South],ranks[West],mapped[North],mapped[South],mapped[West] +0,30,10,20,1,2,0,1,2,0,3,1,2,20,30,10 +1,30,110,20,2,0,1,2,0,1,2,3,1,20,30,110 +2,30,210,20,2,0,1,2,0,1,2,3,1,20,30,210 diff --git a/test/vector_computed_operand/vector_computed_operand.xmile b/test/vector_computed_operand/vector_computed_operand.xmile new file mode 100644 index 000000000..2e6725bc8 --- /dev/null +++ b/test/vector_computed_operand/vector_computed_operand.xmile @@ -0,0 +1,125 @@ + + +
+ Test + Simlin +
+ + 0 + 2 +
1
+ + + + + + + + + + + + + + 30 + + + 10 + + + 20 + + + + + + + + 0 + + + 100 + + + 0 + + + + + + + + 2 + + + 0 + + + 1 + + + + + + + [0, 0, 0] at t=0, [0, 100, 0] at t=1, [0, 200, 0] at t=2 + bump[Region] * TIME + + + + + + The same array the three builtins below compute inline, as + an ordinary variable. Referencing it is the operand shape + that always compiled, so `order_ref` is the control for + `order`. + base[Region] + boost[Region] + + + + + + VECTOR SORT ORDER over a computed operand. 0-based + genuine-Vensim permutation: position j holds the source + index of the j-th smallest element. + VECTOR SORT ORDER(base[Region] + boost[Region], 1) + + + + + + Control: the same sort over the pre-materialized array. + VECTOR SORT ORDER(adjusted[Region], 1) + + + + + + RANK over a computed operand. 1-based. + RANK(base[Region] + boost[Region], 1) + + + + + + VECTOR ELM MAP whose SOURCE is computed: element i is + source[offs[i]] over the computed array's full storage. + VECTOR ELM MAP(base[Region] + boost[Region], offs[Region]) + + + + + + + diff --git a/test/vector_snapshot_operand/output.csv b/test/vector_snapshot_operand/output.csv new file mode 100644 index 000000000..e58560e34 --- /dev/null +++ b/test/vector_snapshot_operand/output.csv @@ -0,0 +1,4 @@ +Time,vals[North],vals[South],vals[West],captured[North],captured[South],captured[West],order[North],order[South],order[West],order_ref[North],order_ref[South],order_ref[West],init_order[North],init_order[South],init_order[West],mapped[North],mapped[South],mapped[West],nested[North],nested[South],nested[West],row_sum[R1],row_sum[R2],picked[R1],picked[R2] +0,30,10,20,0,0,0,0,1,2,0,1,2,1,2,0,0,0,0,1,2,0,0,0,0,0 +1,5,20,20,30,10,20,1,2,0,1,2,0,1,2,0,10,30,10,1,2,0,6,60,4,20 +2,-20,30,20,5,20,20,0,1,2,0,1,2,1,2,0,5,5,20,1,0,2,7,70,5,20 diff --git a/test/vector_snapshot_operand/vector_snapshot_operand.xmile b/test/vector_snapshot_operand/vector_snapshot_operand.xmile new file mode 100644 index 000000000..b0e4eeb16 --- /dev/null +++ b/test/vector_snapshot_operand/vector_snapshot_operand.xmile @@ -0,0 +1,207 @@ + + +
+ Test + Simlin +
+ + 0 + 2 +
1
+ + + + + + + + + + + + + + + + + [30, 10, 20] at t=0, [5, 20, 20] at t=1, [-20, 30, 20] at t=2 + + 30 - 25 * TIME + + + 10 + 10 * TIME + + + 20 + + + + + + + [2, 0, 1] at t=0, [1, 0, 1] at t=1, [0, 0, 1] at t=2 + + 2 - TIME + + + 0 + + + 1 + + + + + + + Constant, so the nested-operand row below is the previous + array plus a fixed offset. + + 30 + + + 10 + + + 20 + + + + + + + Row R1 is [1+TIME, 2, 3]; row R2 is [10, 20, 30+10*TIME]. + + 1 + TIME + + + 2 + + + 3 + + + 10 + + + 20 + + + 30 + 10 * TIME + + + + + + + + Selects columns 1 and 3 of row R1, column 2 of row R2. + + 1 + + + 0 + + + 1 + + + 0 + + + 1 + + + 0 + + + + + + + + The CONTROL, and the oracle this whole feature has to agree + with: the same freeze captured per element, which compiles + to one LoadPrev per slot -- the route that always worked. + `order_ref` below must equal `order`. + PREVIOUS(vals[Region]) + + + + + + VECTOR SORT ORDER over an array-valued PREVIOUS. 0-based + genuine-Vensim permutation. + VECTOR SORT ORDER(PREVIOUS(vals[Region]), 1) + + + + + + Control: the same sort over the per-element capture. + VECTOR SORT ORDER(captured[Region], 1) + + + + + + The INIT twin: `initial_values` is the post-initials + snapshot, so this is the t=0 array at every step -- + including t=0, where PREVIOUS reads its fallback instead. + VECTOR SORT ORDER(INIT(vals[Region]), 1) + + + + + + VECTOR ELM MAP whose SOURCE is the previous array: element + i is prev_vals[offs[i]] over the source variable's full + storage. + VECTOR ELM MAP(PREVIOUS(vals[Region]), offs[Region]) + + + + + + The array-valued PREVIOUS nested inside a COMPUTED operand: + the operand materializes into a temp whose loop body reads + the snapshot view per element. + VECTOR SORT ORDER(PREVIOUS(vals[Region]) + base[Region], 1) + + + + + + A reducer over a strict SLICE of the snapshot: a prev view + of `matrix[Row,*]` is that row of the snapshot, exactly as + the curr view of the same slice is that row of curr. + SUM(PREVIOUS(matrix[Row,*])) + + + + + + The shape GH #1001 was written against: a per-row VECTOR + SELECT over the previous step's matrix rows. + VECTOR SELECT(sel[Row,*], PREVIOUS(matrix[Row,*]), 0, 0, 0) + + + + + + + diff --git a/vensim-probes/README.md b/vensim-probes/README.md new file mode 100644 index 000000000..fb703c031 --- /dev/null +++ b/vensim-probes/README.md @@ -0,0 +1,154 @@ +# External-tool probe models + +Questions this branch could not settle from documentation or from the +checked-in ground truth. Each model is small, self-contained, and uses values +chosen so that **every candidate rule produces a different output**. + +Nothing here is committed to `test/`. Each `.mdl` carries a generated sketch +(`cargo run -p simlin-engine --example layout_probe_models`) so it opens with a +visible diagram; that harness splices only the sketch block and leaves the +equation text byte-identical. + +| model | tool | status | +|---|---|---| +| `elm_map_computed_source.mdl` | Vensim DSS | **answered 2026-08-04** | +| `repeated_dimension.mdl` | Vensim DSS | **answered 2026-08-04** | +| `elm_map_variable_sources.mdl` | Vensim DSS | awaiting a run | +| `stella_repeated_dimension.stmx` | Stella | awaiting a run | + +--- + +## 1. `elm_map_computed_source.mdl` — ANSWERED 2026-08-04 + +**Question:** is an inline expression legal as argument 1 of `VECTOR ELM MAP`, +and if so what storage does the mapping range over? + +**Result: Rule R.** Vensim refuses to simulate the model: + +> `Argument 1 to function VECTOR ELM MAP must be a normal variable` + +(raised for `probe elem expr`). Inline expressions are rejected outright. The +model aborted before producing values, so the `probe helper *` rows went +unanswered — model 3 re-asks them. + +| variable | equation | **R: rejected** (measured) | V: transparent, full storage | T: confined to the temp (Simlin) | +|---|---|---|---|---| +| `ctl slice` | `VECTOR ELM MAP(d[DimA,B1], off[DimA])` | — (aborted) | `1,1,5,5,6,6` | `1,1,5,5,6,6` | +| `probe slice expr` | `VECTOR ELM MAP(d[DimA,B1] * 1, off[DimA])` | **error** ✅ | `1,1,5,5,6,6` | `1,1,2,2,2,2` | +| `ctl elem` | `VECTOR ELM MAP(x[three], DimA - 1)` | — (aborted) | `3,4,5` | `3,4,5` | +| `probe elem expr` | `VECTOR ELM MAP(x[three] * 1, DimA - 1)` | **error** ✅ | `3,4,5` | `3,:NA:,:NA:` | + +**What this settles.** There is no Vensim behaviour to match, because Vensim has +none: the shape is a syntax error. Simlin accepting it is an **extension**, and +the extension is *defined* by helper-equivalence — an inline expression means +exactly what the same values pre-assigned to a named helper variable mean, which +is the spelling that IS legal Vensim. The temp-confined semantics implement that +definition and are no longer provisional. + +## 2. `repeated_dimension.mdl` — ANSWERED 2026-08-04 + +**Question:** does Vensim accept a variable declared over the same subscript +range twice (`sq[DimA,DimA]`), and if so what does reading it mean? + +**Result: Rule A.** Vensim refuses to simulate the model: + +> `DimA appears more than once on LHS` + +(raised for `probe copy`). The declaration is illegal Vensim. + +| variable | equation | **A: declaration rejected** (measured) | D: diagonal (Simlin) | F: true 2-D | +|---|---|---|---|---| +| `probe copy` | `sq[DimA,DimA]` | **error** ✅ | `11,11,11,22,22,22,33,33,33` | `11,12,13,21,22,23,31,32,33` | +| `probe sort` | `VECTOR SORT ORDER(sq[DimA,DimA], 1)` | — (aborted) | `0,0,0,1,1,1,2,2,2` | `0,1,2,0,1,2,0,1,2` | +| `probe sum` | `SUM(sq[DimA!,DimA!])` | — (aborted) | `66` | `198` | + +**What this settles, and what it does not.** The shape is **unreachable from MDL +import**: no Vensim model can declare it, so no imported model can contain it. +That bounds Simlin's repeated-dimension residual family to hand-authored +XMILE / JSON / protobuf. + +It does **not** make the shape illegitimate. The XMILE v1.0 spec exemplifies the +declaration directly — `docs/reference/xmile-v1.0.html` shows "A 2D +non-apply-to-all array with dimensions X by X, where X is size 2" with +`` (verified in-repo). So a conformant XMILE file +may contain it and Simlin must keep reading it. Note the spec exemplifies only +the **declaration**, with per-element equations; it says nothing about what a +*reference* such as `sq[X,X]` on a right-hand side means, which is exactly the +part Simlin gets wrong. Model 4 asks Stella. + +--- + +## 3. `elm_map_variable_sources.mdl` — AWAITING A VENSIM RUN + +The follow-up model 1 could not answer, with **no expression arguments +anywhere**, so it simulates. Every argument is a variable reference. + +**The one live question.** For the whole-variable spelling +`VECTOR ELM MAP(helper[DimA], off[DimA])`, Simlin collapses the base to 0 — +making it identical to `VECTOR ELM MAP(helper[A1], off[DimA])`. Taking the base +from the reference instead (what the documented rule says, and what `vector.dat` +shows for the *strict slice* `d[DimA,B1]`) predicts something different, +including two `:NA:`s. `vector.dat` covers the strict-slice spelling but **not** +this one. + +Fixture: `d` flat storage `[1,4,2,5,3,6]`; `off = [0,1,1]`; `off2 = [0,1,2]`; +`helper[DimA] = d[DimA,B1]` = `[1,2,3]`; `x = [1,2,3,4,5]`, `three` its third +element. The Simlin column is **measured**, not predicted. + +| variable | equation | Simlin (measured) | base-from-reference predicts | R2: Vensim rejects the spelling | +|---|---|---|---|---| +| `ctl slice` | `VECTOR ELM MAP(d[DimA,B1], off[DimA])` | `1,1,5,5,6,6` ✅ matches `vector.dat` | `1,1,5,5,6,6` | — | +| `ctl elem` | `VECTOR ELM MAP(x[three], (DimA - 1))` | `3,4,5` ✅ | `3,4,5` | — | +| `ctl elem off` | `VECTOR ELM MAP(x[three], off2[DimA])` | `3,4,5` ✅ | `3,4,5` | — | +| `probe helper elem` | `VECTOR ELM MAP(helper[A1], off[DimA])` | `1,1,2,2,2,2` | `1,1,2,2,2,2` (base is 0 here by construction) | — | +| **`probe helper slice`** | **`VECTOR ELM MAP(helper[DimA], off[DimA])`** | **`1,1,2,2,2,2`** | **`1,1,3,3,:NA:,:NA:`** | error | + +**How to read the result.** + +- Vensim prints `1,1,3,3,:NA:,:NA:` → Simlin has a real **base bug** for the + whole-variable source spelling: it should take the base from the reference (as + it already does for a strict slice) and does not. +- Vensim prints `1,1,2,2,2,2` → Simlin matches; the two helper spellings are + genuinely one thing. +- Vensim errors on `helper[DimA]` as argument 1 → Rule R2, also an answer: the + legal spelling is element-pinned only, and Simlin accepting the whole-array + spelling is another extension to define rather than match. + +**Note on `ctl elem`, now fixed.** Building this probe surfaced a real MDL +importer defect: it is spelled byte-identically to `vector.mdl`'s `y`, and Simlin +failed it **through MDL import only** (the importer exploded `y[DimA] = ...` into +per-element `Arrayed` slots, where `DimA - 1` has no active apply-to-all +dimension to resolve against), while the same equation via XMILE gave `3,4,5`. +The corpus never caught it because it ran `vector.xmile` and had no MDL twin. + +Fixed on this branch: a single apply-to-all MDL equation now imports as one +`Equation::ApplyToAll` instead of N identical slots +(`mdl::convert::apply_to_all_tests`), and `simulates_vector_mdl_genuine` is the +new corpus gate running `vector.mdl` against real-Vensim `vector.dat`. The row +above is re-measured: `ctl elem` prints `3,4,5`. `ctl elem off` is kept as a +second control that never depended on the fix. + +## 4. `stella_repeated_dimension.stmx` — AWAITING A STELLA RUN + +The tiebreaker for model 2: the XMILE spec exemplifies an `X by X` declaration, +Vensim rejects the equivalent, so Stella decides whether any shipping tool reads +the shape and how. + +`sq[i,j] = 10*i + j`, so every cell is distinct. Simlin column **measured**: + +| variable | equation | Simlin (measured) | true 2-D | diagonal | +|---|---|---|---|---| +| `probe_copy` | `sq[X, X]` | **`11,11,11, 22,22,22, 33,33,33`** | `11,12,13, 21,22,23, 31,32,33` | same as Simlin | +| `probe_row` | `SUM(sq[X, *])` | `36, 66, 96` ✅ | `36, 66, 96` | `11, 22, 33` | +| `probe_sum` | `SUM(sq[*, *])` | `198` ✅ | `198` | `66` | + +This sharpens what Simlin's defect actually is. The **storage is a correct 2-D +array** — the reducers read all nine distinct cells and agree with the true 2-D +column. Only the subscripted **reference** `sq[X,X]` collapses, because both +subscripts resolve to the first axis. So the residual is confined to +reference resolution, not to array construction or reduction, which is a much +smaller thing to fix than the earlier framing suggested. + +If Stella prints the true 2-D row for `probe_copy`, Simlin has a plain bug with +an external referent. If Stella rejects the declaration, the spec example is a +dead letter in both major tools and the shape is Simlin-and-spec-only. diff --git a/vensim-probes/elm_map_computed_source.mdl b/vensim-probes/elm_map_computed_source.mdl new file mode 100644 index 000000000..4c8038671 --- /dev/null +++ b/vensim-probes/elm_map_computed_source.mdl @@ -0,0 +1,77 @@ +{UTF-8} +DimA: A1, A2, A3 ~~| +DimB: B1, B2 ~~| +DimX: one, two, three, four, five ~~| + +d[A1,B1] = 1 ~~| +d[A1,B2] = 4 ~~| +d[A2,B1] = 2 ~~| +d[A2,B2] = 5 ~~| +d[A3,B1] = 3 ~~| +d[A3,B2] = 6 ~~| + +off[DimA] = 0, 1, 1 ~~| + +x[DimX] = 1, 2, 3, 4, 5 ~~| + +ctl slice[DimA,DimB] = VECTOR ELM MAP(d[DimA,B1], off[DimA]) + ~ ~ CONTROL. Reproduces the checked-in ground truth (vector.mdl's f): + expected 1,1,5,5,6,6. If this is not what Vensim prints, the whole + sheet is mis-set-up and nothing below can be read. | + +probe slice expr[DimA,DimB] = VECTOR ELM MAP(d[DimA,B1] * 1, off[DimA]) + ~ ~ QUESTION 1. Same as the control except arg 1 is an EXPRESSION + rather than a variable element reference. Does Vensim accept it at + all, and if so does the mapping still range over d's full storage? | + +helper[DimA] = d[DimA,B1] + ~ ~ The same slice, pre-assigned to a variable of its own. | + +probe helper elem[DimA,DimB] = VECTOR ELM MAP(helper[A1], off[DimA]) + ~ ~ QUESTION 2. The pre-materialized spelling our implementation + claims a computed source is equivalent to: a fresh variable holding + exactly the computed values, addressed from its FIRST element. | + +probe helper slice[DimA,DimB] = VECTOR ELM MAP(helper[DimA], off[DimA]) + ~ ~ QUESTION 3. The same helper addressed per element instead. | + +ctl elem[DimA] = VECTOR ELM MAP(x[three], DimA - 1) + ~ ~ CONTROL. Reproduces vector.mdl's y: expected 3,4,5. | + +probe elem expr[DimA] = VECTOR ELM MAP(x[three] * 1, DimA - 1) + ~ ~ QUESTION 4. The single-element base, computed. | + +INITIAL TIME = 0 ~~| +FINAL TIME = 1 ~~| +TIME STEP = 1 ~~| +SAVEPER = TIME STEP ~~| + +\\\---/// Sketch information - do not modify anything except names +V300 Do not put anything below this section - it will be ignored +*View 1 +$192-192-192,0,Times New Roman|12||0-0-0|0-0-0|0-0-255|-1--1--1|-1--1--1|96,96,100,0 +10,1,ctl elem,401,140,40,22,8,3,0,0,-1,0,0,0 +10,2,ctl slice,342,188,40,22,8,3,0,0,-1,0,0,0 +10,3,d,127,284,40,20,8,3,0,0,-1,0,0,0 +10,4,helper,180,233,40,20,8,3,0,0,-1,0,0,0 +10,5,off,144,188,40,20,8,3,0,0,-1,0,0,0 +10,6,probe elem expr,428,246,40,22,8,3,0,0,-1,0,0,0 +10,7,probe helper elem,121,241,44,22,8,3,0,0,-1,0,0,0 +10,8,probe helper slice,228,124,48,22,8,3,0,0,-1,0,0,0 +10,9,probe slice expr,50,268,40,22,8,3,0,0,-1,0,0,0 +10,10,x,512,50,40,20,8,3,0,0,-1,0,0,0 +10,11,d,279,216,40,20,8,2,0,3,-1,0,0,0,128-128-128,0-0-0,|12||128-128-128 +10,12,off,273,188,40,20,8,2,0,3,-1,0,0,0,128-128-128,0-0-0,|12||128-128-128 +10,13,x,455,183,40,20,8,2,0,3,-1,0,0,0,128-128-128,0-0-0,|12||128-128-128 +1,14,10,1,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,15,11,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,16,12,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,17,3,4,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,18,13,6,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,19,4,7,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,20,5,7,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,21,4,8,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,22,5,8,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,23,3,9,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,24,5,9,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +///---\\\ diff --git a/vensim-probes/elm_map_variable_sources.mdl b/vensim-probes/elm_map_variable_sources.mdl new file mode 100644 index 000000000..1d05250b6 --- /dev/null +++ b/vensim-probes/elm_map_variable_sources.mdl @@ -0,0 +1,82 @@ +{UTF-8} +DimA: A1, A2, A3 ~~| +DimB: B1, B2 ~~| +DimX: one, two, three, four, five ~~| + +d[A1,B1] = 1 ~~| +d[A1,B2] = 4 ~~| +d[A2,B1] = 2 ~~| +d[A2,B2] = 5 ~~| +d[A3,B1] = 3 ~~| +d[A3,B2] = 6 ~~| + +off[DimA] = 0, 1, 1 ~~| + +x[DimX] = 1, 2, 3, 4, 5 ~~| + +helper[DimA] = d[DimA,B1] + ~ ~ The B1 column of d, pre-assigned to a variable of its own, so its + own storage is exactly three contiguous cells (1,2,3). | + +ctl slice[DimA,DimB] = VECTOR ELM MAP(d[DimA,B1], off[DimA]) + ~ ~ CONTROL. Reproduces the checked-in ground truth (vector.mdl's f): + expected 1,1,5,5,6,6. If this is not what Vensim prints, the sheet + is mis-set-up and nothing below can be read. | + +off2[DimA] = 0, 1, 2 ~~| + +ctl elem[DimA] = VECTOR ELM MAP(x[three], (DimA - 1)) + ~ ~ CONTROL, spelled byte-identically to vector.mdl's y: expected + 3,4,5. Simlin cannot compile this one THROUGH MDL IMPORT -- the + importer turns an apply-to-all arrayed equation into per-element + slots, where `DimA - 1` has no active dimension to resolve against. + The same equation via XMILE gives 3,4,5. Vensim is unaffected; the + row is kept because it is the literal ground-truth spelling. | + +ctl elem off[DimA] = VECTOR ELM MAP(x[three], off2[DimA]) + ~ ~ CONTROL, same expected 3,4,5, with the offset supplied by a + variable so it compiles on both sides. This is the control to read + if `ctl elem` errors. | + +probe helper elem[DimA,DimB] = VECTOR ELM MAP(helper[A1], off[DimA]) + ~ ~ The whole helper addressed from its FIRST element. Base 0 for + every result element, offsets 0,1,1 into helper's storage (1,2,3). | + +probe helper slice[DimA,DimB] = VECTOR ELM MAP(helper[DimA], off[DimA]) + ~ ~ THE DISCRIMINATING CELL. The same helper addressed PER ELEMENT. + If the base comes from the reference, result element (Ai,*) reads + helper[i + off[Ai]] and A3 runs off the end -> 1,1,3,3,:NA:,:NA:. + If the whole-variable spelling collapses to base 0 it is the same as + probe helper elem -> 1,1,2,2,2,2. Simlin prints the latter; nothing + in vector.dat covers this spelling. | + +INITIAL TIME = 0 ~~| +FINAL TIME = 1 ~~| +TIME STEP = 1 ~~| +SAVEPER = TIME STEP ~~| + +\\\---/// Sketch information - do not modify anything except names +V300 Do not put anything below this section - it will be ignored +*View 1 +$192-192-192,0,Times New Roman|12||0-0-0|0-0-0|0-0-255|-1--1--1|-1--1--1|96,96,100,0 +10,1,ctl elem,307,241,40,22,8,3,0,0,-1,0,0,0 +10,2,ctl elem off,144,294,40,22,8,3,0,0,-1,0,0,0 +10,3,ctl slice,195,50,40,22,8,3,0,0,-1,0,0,0 +10,4,d,230,89,40,20,8,3,0,0,-1,0,0,0 +10,5,helper,140,206,40,20,8,3,0,0,-1,0,0,0 +10,6,off,126,122,40,20,8,3,0,0,-1,0,0,0 +10,7,off2,92,319,40,20,8,3,0,0,-1,0,0,0 +10,8,probe helper elem,50,152,44,22,8,3,0,0,-1,0,0,0 +10,9,probe helper slice,128,146,48,22,8,3,0,0,-1,0,0,0 +10,10,x,256,253,40,20,8,3,0,0,-1,0,0,0 +1,11,10,1,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,12,7,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,13,10,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,14,4,3,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,15,6,3,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,16,4,5,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,17,5,8,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,18,6,8,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,19,5,9,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,20,6,9,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +///---\\\ diff --git a/vensim-probes/repeated_dimension.mdl b/vensim-probes/repeated_dimension.mdl new file mode 100644 index 000000000..ec0e4a17e --- /dev/null +++ b/vensim-probes/repeated_dimension.mdl @@ -0,0 +1,43 @@ +{UTF-8} +DimA: A1, A2, A3 ~~| + +sq[A1,A1] = 11 ~~| +sq[A1,A2] = 12 ~~| +sq[A1,A3] = 13 ~~| +sq[A2,A1] = 21 ~~| +sq[A2,A2] = 22 ~~| +sq[A2,A3] = 23 ~~| +sq[A3,A1] = 31 ~~| +sq[A3,A2] = 32 ~~| +sq[A3,A3] = 33 ~~| + +probe copy[DimA,DimA] = sq[DimA,DimA] + ~ ~ QUESTION 1. Does Vensim ACCEPT a variable declared over the same + subscript range twice, and does a plain copy read all nine cells or + only a diagonal? | + +probe sort[DimA,DimA] = VECTOR SORT ORDER(sq[DimA,DimA], 1) + ~ ~ QUESTION 2. If the declaration is accepted, what does a vector + operation over it sort -- each row independently, or something else? | + +probe sum = SUM(sq[DimA!,DimA!]) + ~ ~ QUESTION 3. Does a reducer over the repeated range sum all nine + cells or only the three diagonal ones? | + +INITIAL TIME = 0 ~~| +FINAL TIME = 1 ~~| +TIME STEP = 1 ~~| +SAVEPER = TIME STEP ~~| + +\\\---/// Sketch information - do not modify anything except names +V300 Do not put anything below this section - it will be ignored +*View 1 +$192-192-192,0,Times New Roman|12||0-0-0|0-0-0|0-0-255|-1--1--1|-1--1--1|96,96,100,0 +10,1,probe copy,104,50,40,22,8,3,0,0,-1,0,0,0 +10,2,probe sort,126,150,40,22,8,3,0,0,-1,0,0,0 +10,3,probe sum,50,60,40,22,8,3,0,0,-1,0,0,0 +10,4,sq,86,106,40,20,8,3,0,0,-1,0,0,0 +1,5,4,1,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,6,4,2,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +1,7,4,3,0,0,0,0,0,64,0,-1--1--1,,1|(0,0)| +///---\\\ diff --git a/vensim-probes/stella_repeated_dimension.stmx b/vensim-probes/stella_repeated_dimension.stmx new file mode 100644 index 000000000..e34d08424 --- /dev/null +++ b/vensim-probes/stella_repeated_dimension.stmx @@ -0,0 +1,87 @@ + + + +
+ Simlin + stella repeated dimension probe + Simlin +
+ + 0 + 1 +
1
+ + + + + + + + + + + + 11 + 12 + 13 + 21 + 22 + 23 + 31 + 32 + 33 + + + + + + + sq[X, X] + + + + + + SUM(sq[X, *]) + + + SUM(sq[*, *]) + + + + + + + + + + + +