Stateful encoders + documentation refresh

.gitignore

@@ -5,3 +5,9 @@ data/
 literature/
 Manifest.toml
 docs/build/
+
+# Literate output: regenerated from docs/src/examples/*.jl on every docs build
+docs/src/examples/*.md
+
+# local dev config
+.claude/

CLAUDE.md

@@ -26,7 +26,9 @@ test/
   types.jl, operations.jl, encoding.jl, inference.jl, representations.jl
   ext_display.jl                # rich-display tests, run in a separate process (extension loading is irreversible)
 docs/
-  src/examples/                 # Literate.jl tutorials (intro-to-hdc, dollar-of-mexico)
+  src/examples/                 # Literate.jl tutorials (intro-to-hdc, encoding-data,
+                                #   colours, dollar-of-mexico, iris); *.md is Literate
+                                #   output, gitignored and regenerated on every build
 ```
 
 ## Supported vector types (all subtypes of `AbstractHV{T} <: AbstractVector{T}`)
@@ -50,22 +52,26 @@ Constructor convention (settled; each form has exactly one meaning, documented o
 - `HV(x)` — token shorthand for `encode(HV, x)`; `HV(n::Number)` **throws an ArgumentError** naming both alternatives (`D = n` / `encode(HV, n)`) — this replaced the old one-time `@warn`
 - Tuples of reals read as data, like vectors
 
-`encode` is the canonical interface: `encode(HV, x; D)` is the deterministic token path (hash → seed), `encode(HV, x, strategy)` dispatches on `AbstractEncoding` strategies: `KMer(k)` (windows as atomic hashed tokens — resolves issue #53), `NGram(n)` (symbol-level shift-binding via `ngrams`), `Sequence()` (bundlesequence), `BagOfSymbols()` (multiset). New strategies = one struct + one `encode` method. Planned follow-up: stateful `AbstractEncoder{HV}` structs (RandomProjection, LevelEncoder) with `encode`/`decode`.
+`encode` is the canonical interface: `encode(HV, x; D)` is the deterministic token path (hash → seed), `encode(HV, x, strategy)` dispatches on `AbstractEncoding` strategies: `KMer(k)` (windows as atomic hashed tokens — resolves issue #53), `NGram(n)` (symbol-level shift-binding via `ngrams`), `Sequence()` (bundlesequence), `BagOfSymbols()` (multiset). New strategies = one struct + one `encode` method.
+
+Stateful encoders live in the `AbstractEncoder{HV}` hierarchy (encode.jl): instances hold hypervector state built once at construction and slot in as the first argument of `encode`, with `decode` as the inverse. Two members: `LevelEncoder` encodes scalars against a shared level set (see below); `RandomProjection` encodes fixed-length real feature vectors (RGB triples, embeddings) through a shared `D × d` projection matrix: `z = R·x`, then a per-type nonlinearity (sign-like for binary/bipolar/ternary with a scalar-or-vector threshold `θ`, `β`-scaled identity/logistic/tanh for real/graded, `exp(im·β·z)` for FHRR). **Standardize features when they are on incommensurable scales — not as a reflex**: measured on iris, raw features beat standardized ones 0.98 vs 0.83, because there the largest-spread features are also the most discriminative (see the iris tutorial). Matrix flavours: `:gaussian` (default), `:bipolar`, `:sparse_ternary`. Constructors: `RandomProjection(HV, d; D, matrix, θ, β, seed, rng)`, `RandomProjection(HV, R::AbstractMatrix; θ, β)` (supplied matrix), and a data-driven ternary form `RandomProjection(TernaryHV, X; target_sparsity)` that solves θ from data at construction (immutable — construction-from-data, not fitting; `rethreshold(rp, θ)` returns a new encoder sharing `R`). RP's `decode(rp, hv, references)` is nearest-neighbour clean-up only (no analytic inverse exists — the nonlinearity is lossy); it errors without a reference set. FHRR + `:gaussian` is exactly random Fourier features: similarity approximates a Gaussian kernel with bandwidth `β`. Both encoders' FHRR paths route through the single internal `phase_encode(z, β)` helper — never duplicate it.
 
 Scalar `getindex` returns the element type `T`; non-scalar indexing returns a plain `Vector` of values, never a hypervector. Hypervectors are immutable (no `setindex!`).
 
 ## Core operations
 
 | Operation | Function | Operator | Effect |
 |-----------|----------|----------|--------|
-| Bundle | `bundle(hvs)` | `+` | Superposition; result similar to all inputs |
+| Bundle | `bundle(hvs)` | `+`, `sum` | Superposition; result similar to all inputs |
 | Bind | `bind(hv1, hv2)` | `*` | Association; result dissimilar to inputs |
 | Unbind | `unbind(hv1, hv2)` | `/` | Inverse of bind (throws for `RealHV`: real MAP binding is not exactly invertible) |
 | Shift | `shift(hv, k)` / `ρ(hv, k)` | — | Circular shift by k positions |
 | Perturbate | `perturbate(hv, n_or_p)` | — | Flip n positions or fraction p |
 
 In-place variants: `shift!`, `ρ!`, `perturbate!`.
 
+**Bundling is m-way, not pairwise.** The rule depends on how many inputs there are, so bundling is *not* associative for BinaryHV/BipolarHV/RealHV/FHRR. `bundle(hvs)`, `sum(hvs)` and chained `x + y + z` (Julia parses a `+` chain as one variadic call) all reach `bundle` with the whole collection and are equivalent; `(x + y) + z`, `reduce(+, hvs)` and `foldl(+, hvs)` bundle a bundle, which biases the result towards the last inputs and destroys the property that a bundle is equally similar to each part.
+
 ## Encoding strategies (compose primitives into structured representations)
 
 - `multiset(hvs)` — bundle a set of vectors
@@ -75,11 +81,14 @@ In-place variants: `shift!`, `ρ!`, `perturbate!`.
 - `crossproduct(U, V)` — cross product of two sets
 - `ngrams(hvs, n)` — n-gram statistics for text/sequence encoding
 - `graph(sources, targets)` — directed/undirected graph encoding
-- `level(hv, n)` / `encodelevel` / `decodelevel` / `convertlevel` — numeric level encoding
+
+Numeric values are encoded with `LevelEncoder` (encode.jl), which replaced the old `level`/`encodelevel`/`decodelevel`/`convertlevel` function family. It builds its level set **once** at construction (the old family rebuilt it per call, producing incomparable encodings). Two mechanisms, selected by dispatch: a perturbation **ladder** for all types (`LevelEncoder(HV, range, n; bandwidth = 2/n)`, flip fraction per step) and **fractional power encoding** for FHRR (`LevelEncoder(FHRR, values; β)`, continuous via `base^(β·x)`; pass an explicit level count `n` to get a ladder instead). `encode(lvl, x)` maps a number to a hypervector; `decode(lvl, hv)` maps back via nearest-neighbour similarity (FHRR also supports `method = :analytic` for continuous, clean-vector decoding). `LevelEncoder(levels, values)` wraps precomputed hypervectors — this labelled-codebook form is the seam for a future shared item-memory abstraction.
 
 ## Similarity and inference
 
-- `similarity(u, v)` / `δ(u, v)` — type-dispatched similarity (cosine for bipolar/real, Jaccard for binary/graded, complex dot for FHRR)
+- `similarity(u, v)` / `δ(u, v)` — type-dispatched similarity (cosine for bipolar/ternary/real/graded-bipolar/FHRR, Jaccard for binary/graded)
+- `similaritymetric(HV)` — which metric a type uses (`:cosine` / `:jaccard`)
+- `chancesimilarity(HV)` — what an *unrelated* pair scores: `0.0` under cosine, `1/3` under Jaccard. Needed to interpret any similarity number; assumes the type's default `distr`
 - `nearest_neighbor(u, collection)` — find closest match
 - `nearest_neighbor(u, collection, k)` — k-nearest neighbors
 
@@ -152,10 +161,10 @@ julia --project -e 'using Pkg; Pkg.test()'
 - [x] ~~`bind`: no docstring (only `unbind` has one)~~ (added)
 - [ ] `shift` / `ρ`: no docstrings
 - [ ] `perturbate` / `perturbate!`: no docstrings
-- [ ] `level`: docstring is minimal; does not explain the perturbation-based correlation mechanism
+- [x] ~~`level`: docstring is minimal; does not explain the perturbation-based correlation mechanism~~ (obsolete: the family was replaced by `LevelEncoder`, whose docstring covers both mechanisms)
 - [ ] `three_pi` and `fuzzy_xor` helper functions: no docstrings
 - [ ] Internal functions (`aggfun`, `bindfun`, `neutralbind`, `noisy_and`, `elementreduce!`, `offsetcombine`, `empty_vector`, `eldist`, `vectype`): none documented
-- [ ] `docs/make.jl`: Contents block references `examples.md` but actual pages are in `examples/` subdirectory
+- [x] ~~`docs/make.jl`: Contents block references `examples.md`~~ (index.md rewritten as a real landing page)
 - [ ] No docstring for `^` on FHRR — exponentiation is an important FHRR feature
 - [ ] No developer docs on how to add a new HV type (what methods to implement, traits, etc.)
 

README.md

@@ -147,7 +147,26 @@ Additionally, we provide common encoder strategies for different data structures
 - `crossproduct`
 - `ngrams`
 - `graph`
-- `level`
+
+Numeric values are encoded with the stateful `LevelEncoder`, which builds a set of
+level-correlated hypervectors once and shares it across every `encode`/`decode` call:
+
+```julia
+lvl = LevelEncoder(BipolarHV, (0, 2π), 100)  # 100 levels over the interval
+hv = encode(lvl, π / 3)                      # hypervector for a numeric value
+decode(lvl, hv)                              # ≈ π/3, via similarity matching
+```
+
+Fixed-length feature vectors (an RGB triple, an embedding — standardize them first!)
+are encoded with `RandomProjection`, which projects through a matrix drawn once and
+shared by every call; for `FHRR` this is exactly a random Fourier feature map, whose
+similarity approximates a Gaussian kernel:
+
+```julia
+rp = RandomProjection(BipolarHV, 3)          # ℝ³ → 10,000-dimensional hypervectors
+hv = encode(rp, [0.9, -0.2, 0.4])            # nearby inputs ↦ similar hypervectors
+decode(rp, hv, references)                   # nearest-neighbour clean-up (needs a codebook)
+```
 
 Finally, the `similarity` function can be used to compare two hypervectors, by default using
 the best similarity metric for the hypervector type:

TODO.md

@@ -22,10 +22,54 @@ The package now has an explicit layer taxonomy: **primitives** (`operations.jl`)
   #53** (windows as atomic hashed tokens — genuinely different from `NGram(n)`,
   which shift-binds symbol encodings via `ngrams`); also `Sequence()` and
   `BagOfSymbols()`. Extension point: one struct + one `encode` method.
-- [ ] Follow-up PR: stateful encoders as an `AbstractEncoder{HV}` hierarchy with
-  `encode`/`decode` — `RandomProjection` (fixed projection matrix) and
-  `LevelEncoder` (fixed level set; subsumes the §1.4b ladder problem). `encode`
-  deliberately keeps its first argument free for encoder instances.
+- [x] Stateful encoders as an `AbstractEncoder{HV}` hierarchy with
+  `encode`/`decode` — **`LevelEncoder` landed (2026-07-15)**: builds its level
+  set once at construction (fixes §1.4b by construction), replaces and deletes
+  the `level`/`encodelevel`/`decodelevel`/`convertlevel` family. Ladder
+  (perturbation, all types, `bandwidth` keyword) + fractional power encoding
+  (FHRR, `β`), selected by dispatch. `encode` deliberately keeps its first
+  argument free for encoder instances.
+- [x] `RandomProjection` as the next `AbstractEncoder{HV}` — **landed
+  (2026-07-15)**: fixed `D × d` projection matrix (`:gaussian`/`:bipolar`/
+  `:sparse_ternary`), per-type nonlinearities, parametric scalar-or-vector
+  ternary threshold `θ` (+ data-driven `target_sparsity` constructor,
+  `rethreshold` sharing `R`), supplied-matrix constructor, FHRR = random
+  Fourier features via the shared internal `phase_encode(z, β)` helper (also
+  now used by `LevelEncoder`'s fractional power path — single source of
+  truth). Closes the colour/embedding (RGB / ESM-2-style feature-vector)
+  random-projection gap.
+- [ ] Shared item-memory/codebook abstraction (labelled HV collection +
+  nearest-neighbour lookup): now seeded by `LevelEncoder(levels, values)` — its
+  `decode` is built on that stored collection. `RandomProjection`'s
+  `decode(rp, hv, references)` clean-up is the **second consumer** of the same
+  shape (currently via raw `nearest_neighbor` over a user-supplied reference
+  set); the tutorial's colour reverse-lookup and a future `train`/`predict`
+  need it too. Factor it out.
+
+---
+
+## 0b. Documentation refresh (2026-07-27)
+
+Branch `docs-encoders-refresh` = `38-integration` + the encoders work, with the
+tutorials brought up to the current API and the encoder layer documented:
+
+- `iris.jl` ported to `LevelEncoder` (it used all four removed level functions, and
+  defined local `encode`/`decode` helpers colliding with the new exports).
+- `api.md` restructured into Combinators / Encoders; the stateful encoders reach the
+  API page for the first time.
+- New tutorials: **Encoding data** (token path, KMer vs NGram, language recognition
+  from character k-mers — the issue #53 payoff) and **Colours** (RandomProjection end
+  to end, incl. decode-as-clean-up, the tuning knobs, and FHRR-as-random-Fourier-
+  features verified numerically in the page).
+- Iris gained a second act encoding the same data with one `RandomProjection`.
+- Landing page rewritten; developer guide added to the nav; `Remotes.GitHub` silences
+  the navbar warning; generated `docs/src/examples/*.md` untracked and gitignored
+  (they are Literate output, regenerated every build — they were a third of the diff).
+- New API from review questions: `sum(hvs)` now bundles the whole collection, and
+  `similaritymetric` / `chancesimilarity` make similarity scores interpretable.
+
+Still unwritten from the plan: the amino-acids page and a "Why hyperdimensional?"
+page (low- vs high-dimensional similarity histograms). Neither blocks anything.
 
 ---
 
@@ -140,15 +184,21 @@ Remaining inconsistencies found by verification (not yet fixed):
   (`doctest(fix = true)`) — element signs flipped, so **the doctest CI job fails
   until this is done**.
 
-### 1.4b Instance-path `convertlevel` builds encoder and decoder over DIFFERENT ladders
+### 1.4b Instance-path `convertlevel` builds encoder and decoder over DIFFERENT ladders — FIXED (2026-07-15)
+Fixed structurally by the `LevelEncoder` refactor (§0): the level set is built
+once in the constructor and shared by every `encode`/`decode` call, and `seed`
+makes the whole ladder deterministic. The old function family (including the
+broken instance path) is deleted, no deprecation shims. Locked by the
+"one shared level set" testset in `test/encoding.jl` (exact grid round-trips +
+`!isdefined` checks for the removed names). Original notes:
 Found (by execution) while fixing §1.4, deliberately NOT fixed in that PR.
 `convertlevel(hv::AbstractHV, numvals)` calls `encodelevel(hv, ...)` and
 `decodelevel(hv, ...)`, each of which builds its own `level(hv, m)` ladder — and
 `level` perturbation is unseeded, so the two ladders share only the base vector.
 Measured: `decode(encode(x))` errors up to 1.0 (mean 0.41) on the instance path vs
 exactly 0.0 when both are built from one shared `level(...)` ladder.
-- [ ] Fix: build the ladder once in `convertlevel` (and/or make `level`
-  deterministic given the base vector), then assert the roundtrip in tests.
+- [x] Fix: build the ladder once (now: in the `LevelEncoder` constructor), then
+  assert the roundtrip in tests.
 
 ### 1.5c `perturbate` resamples from the TYPE-default distribution, not `hv.distr` — FIXED (2026-07-14)
 Instance-level `eldist(hv) = hv.distr` methods added for RealHV/GradedHV/
@@ -270,6 +320,33 @@ users an export conflict.
   `HV(this; D)` — no `rng` keyword; always `Xoshiro(hash(this))`. Breaking in
   output: every seeded/token vector changed; README + example outputs regenerated.
 
+### 2.6 `RandomProjection(TernaryHV, ::AbstractMatrix)` positional collision (found 2026-07-15)
+
+The merged ternary constructor reads the positional matrix as **training data**
+when `target_sparsity` is given and as a **supplied projection matrix**
+otherwise — same positional shape, opposite meanings, disambiguated only by
+keyword presence. The "ternary constructor: positional collision" testset in
+`test/encoding.jl` pins what tests *can* pin: each path's documented reading,
+and that a data-shaped (non-square) matrix misread as a projection matrix
+cannot encode the intended features (immediate `DimensionMismatch` at first
+use). What tests cannot make safe, deliberately not patched in that test-only
+pass:
+
+- A **square** matrix is genuinely ambiguous: both readings yield a working
+  encoder, so a forgotten `target_sparsity` silently produces a projection
+  encoder built from data (and vice versa is undetectable).
+- The misread's error is a downstream feature-length `DimensionMismatch`; it
+  never names the actual mistake (matrix misread as R instead of X).
+- The ternary method's signature carries the data-path keywords, so the
+  supplied-matrix path **silently accepts and ignores** `D`, `matrix` and
+  `seed` (`RandomProjection(TernaryHV, R; D = 500)` returns a
+  `size(R, 1)`-dimensional encoder); the generic-type equivalent throws a
+  `MethodError`. Inconsistent and unlockable without blessing it.
+- [ ] Fix by renaming one path rather than patching: a keyword-only or
+  distinctly named data-driven constructor (e.g. `fit_sparsity(TernaryHV, X;
+  target_sparsity, ...)` or `RandomProjection(TernaryHV, d; from_data = X)`),
+  and reject inapplicable kwargs on the supplied-matrix path.
+
 ---
 
 ## 3. README (the first thing JuliaCon attendees will open)
@@ -324,7 +401,7 @@ users an export conflict.
   is used" but `method` is a mandatory kwarg for plain vectors (`src/inference.jl:38`).
   Also `:hamming` returns a match *count*, not a similarity in [0,1] — document or normalize.
 - [ ] Typo `src/inference.jl:71`: "and `u`` " (stray backtick).
-- [ ] `level` docstring: document the perturbation-based correlation mechanism and the FHRR `^`-based variant.
+- [x] ~~`level` docstring: document the perturbation-based correlation mechanism and the FHRR `^`-based variant~~ (obsolete: the family was replaced by `LevelEncoder`, whose docstring documents both mechanisms with doctests).
 - [ ] Issue #36: the intro tutorial's ngrams example yields the wrong result — re-derive it.
 - [ ] Developer docs: how to add a new HV type (required methods: constructor,
   `eldist`, `empty_vector`, `bundle`, `bind`, `similarity`, traits…).
@@ -339,7 +416,7 @@ Coverage gaps (each hid a §1 bug):
   approximate fuzzy recovery for the graded types, exact FHRR division, and the
   explicit `ArgumentError` for RealHV.
 - [ ] `perturbate` on FHRR (currently broken, §1.3).
-- [ ] Instance-path `encodelevel`/`decodelevel`/`convertlevel` (currently broken, §1.4).
+- [x] ~~Instance-path `encodelevel`/`decodelevel`/`convertlevel` (currently broken, §1.4)~~ (family deleted; `LevelEncoder` has its own testset: all 7 types, round-trips, bandwidth, FPE, precomputed constructor, bounds).
 - [ ] `bundle`/`bind` preserving custom `distr` (§1.5).
 - [ ] `similarity(...; method = :jaccard / :hamming)` — only `:cosine` is tested.
 - [ ] `isapprox` bootstrap path — the whole similarity testset is skipped for

docs/Project.toml

@@ -1,10 +1,14 @@
 [deps]
+AlgebraOfGraphics = "cbdf2221-f076-402e-a563-3d30da359d67"
 CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
 Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"
+DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
 Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 Glob = "c27321d9-0574-5035-807b-f59d2c89b15c"
-Handcalcs = "e8a07092-c156-4455-ab8e-ed8bc81edefb"
 HyperdimensionalComputing = "ebb7690e-f605-4e7a-86a0-727dbed6b293"
 Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
-Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+LiveServer = "16fef848-5104-11e9-1b77-fb7a48bbb589"
+MLDatasets = "eb30cadb-4394-5ae3-aed4-317e484a6458"
+Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
+UnicodePlots = "b8865327-cd53-5732-bb35-84acbb429228"