Skip to content

Spec 005: per-page provenance (v0.5.0) - #6

Merged
aucontraire merged 8 commits into
masterfrom
005-page-provenance
Jun 29, 2026
Merged

aucontraire merged 8 commits into
masterfrom
005-page-provenance

Conversation

@aucontraire

@aucontraire aucontraire commented Jun 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Every wiki page now carries a cumulative provenance: frontmatter block (latest checkpoint, contributing conversations, action, confidence, total edits, exchange count) backed by a permanent per-checkpoint JSON record at <vault>/InsightMesh/.history/checkpoints/<conv-id>/cp-<NNN>.json and a shadow git repository for git log -p style diff history of every page's evolution across edits.
  • Designed as a stable read contract (additive forward-compatibility within schema_version=1) for the planned in-Obsidian viewer plugin in a separate repo: aucontraire/insightmesh-obsidian.
  • All bookkeeping is orchestrator-side; Editor's contract is unchanged; provenance failures never fail the run (FR-019).

What's new

src/history.py (NEW, ~660 lines): 8 Pydantic v2 models (CheckpointRecord + supporting + ProvenanceFrontmatter), CheckpointRecordRead permissive read-side subclass per FR-002, 4 custom exceptions, 7 pure helpers (compute_checkpoint_payload, write_checkpoint_metadata atomic + immutable per FR-001a, merge_page_provenance cumulative + atomic per FR-011, snapshot_page, init_shadow_repo idempotent three-state per FR-012, commit_checkpoint FR-014 format, is_git_available cached). No new runtime dependencies.

src/orchestrator.py (modified): new _sanitize_conversation_subdir, _normalize_vault_relative_path, _editor_decision_to_record, _build_exchange_records, _write_provenance helpers. _write_provenance is wired into the existing Spec 004 checkpoint loop after Editor success and before the cursor save, per FR-017.

src/exports.py (modified): refactored _to_role_content to share a _message_payload helper; added _to_role_content_with_ids + _provider_from_adapter. extract_conversation now populates ChatTranscript.metadata with provider, models_used, exchange_message_ids.

.claude/agents/editor.md (doc-only): note that orchestrator owns the provenance: frontmatter block per FR-017.

Tests: tests/test_history.py NEW (37 tests) + tests/test_orchestrator.py extended (+25 tests: 14 US1 + 10 US2 + 1 absolute-path regression). tests/fixtures/provenance_cp_001.json NEW for forward-compat fixture.

Docs: docs/index.md admonition refreshed to feature Spec 005; docs/reference/cli.md new "Per-checkpoint provenance record (Spec 005)" section under On-disk artifacts; docs/known-limitations.md gains a "RESOLVED in Spec 005" entry; README.md updated; status tables refreshed across index.md + README.md.

On-disk artifacts (per checkpoint)

<vault>/InsightMesh/
├── <page>.md                                    # now carries `provenance:` frontmatter block
└── .history/                                    # NEW
    ├── .git/                                    # shadow git repo, machine-greppable commits per FR-014
    ├── checkpoints/
    │   └── <conv-id-or-_flat>/
    │       └── cp-<NNN>.json                    # CheckpointRecord (permanent, immutable per FR-001a)
    └── pages/
        └── <sanitized-slug>.md                  # page snapshots, git-tracked

Test plan

  • uv run pytest: 302 passing
  • uv run mypy --strict src/: clean across 9 source files
  • uv run ruff check src/ tests/: all checks passed
  • uv run black --check src/ tests/: 19 files clean
  • uv run ruff check --select TID: no banned-API violations (constitution v1.1.2 §Project Standards)
  • uv run mkdocs build --strict: clean
  • insightmesh --version → 0.5.0
  • Real-data smoke (T020) on all 3 transcript code paths: Claude.ai conversation (provider="anthropic", models_used=[], real echomine UUIDs), ChatGPT conversation (provider="openai", models_used=["gpt-4o"], real ids), Spec 001 flat-array (provider=null, conv_id=null, _flat/ sentinel subdir, null message ids). Findings memory: project_spec005_findings.md.

Bugs found + fixed during real-data smoke

S1: WikiPageResult.file_path from MCPVault is an absolute filesystem path, not vault-relative. _normalize_vault_relative_path originally only handled bare-filename + already-prefixed inputs, producing garbage like InsightMesh//Users/.../InsightMesh/Page.md in results.pages_created / pages_updated. Fix in commit bdea9df with regression test TestProvenanceAbsolutePathNormalization. Same lesson as Spec 004 findings: mocked tests with synthetic defaults silently mask the failures real Editor output surfaces.

Spec process (followed full speckit cycle)

/speckit-specify → /speckit-clarify (1 question, locked per-conversation subdir naming) → /speckit-plan → /speckit-tasks (21 tasks) → /speckit-checklist (added second durability-and-resilience checklist of 38 items) → /spec-gaps all (15 remediation groups applied) → /speckit-analyze (1 MEDIUM + 4 LOW found) → /reconcile all (all 5 closed) → /speckit-implement → real-data smoke (T020). 21/21 tasks done, both checklists 100% (16+38 items checked).

Constitution compliance

  • Anti-Slop: 1 new src/ file (src/history.py) justified in plan.md Complexity Tracking; modifications to src/orchestrator.py and src/exports.py surgical; no new dependencies
  • Project Standards: all data shapes are pydantic.BaseModel(ConfigDict(strict=True)) — the one spec-mandated extra="allow" deviation on CheckpointRecordRead is documented inline per FR-002
  • Incremental Delivery: US1 ships as an MVP shippable without git on PATH; US2 layers shadow-repo on top

Out of scope (deferred to future specs)

  • In-Obsidian viewer plugin — separate repo: aucontraire/insightmesh-obsidian
  • CLI insightmesh history <page> subcommand
  • Obsidian Dataview / Bases templates (Properties panel doesn't render nested YAML natively — design decision; covered in spec Key Entities)
  • Phase B Postgres migration of the checkpoint JSON
  • Pruning / compaction of .history/

Full list in specs/005-page-provenance/spec.md § Out of Scope.

Drafts spec/plan/tasks/contracts/data-model for per-page provenance:
structured per-checkpoint JSON sidecar files, cumulative frontmatter
provenance blocks on each touched wiki page, and a shadow git
repository at <vault>/InsightMesh/.history/ for diff history.

25 FRs (including FR-001a/b for immutability + deterministic ordering,
FR-016a for stderr format, FR-021/022 for crash resilience), 7 SCs,
2 user stories: US1 P1 (structured JSON + cumulative frontmatter)
ships as MVP without requiring git; US2 P2 (shadow-git diff history)
layers on top. 21 tasks across 5 phases, 100% FR-to-task coverage
verified via /speckit-analyze + /reconcile.

Designed as a stable read contract for the planned in-Obsidian viewer
plugin (separate repo: aucontraire/insightmesh-obsidian); no Spec 005
changes are required to enable that plugin.

Also updates CLAUDE.md plan reference and .specify/feature.json to
point at this spec.
Foundational layer for per-page provenance:

- NEW src/history.py (659 lines): 8 Pydantic v2 models (CheckpointRecord,
  ConversationRecord, ExchangeRecord, EditorDecisionRecord, EditorBlock,
  ResultsRecord, LinksRecord, ProvenanceFrontmatter, ExchangeMessageIds)
  with strict=True + extra=forbid for write-side; CheckpointRecordRead
  subclass with extra=allow per FR-002 additive-evolution rule. Custom
  exceptions (HistoryError + ShadowRepoUnavailable +
  ShadowRepoCommitFailed + FrontmatterParseFailed). Pure helpers:
  compute_checkpoint_payload, write_checkpoint_metadata (atomic per FR-001
  + immutability per FR-001a), merge_page_provenance (cumulative math per
  FR-009 + atomic per FR-011), snapshot_page, init_shadow_repo
  (three-state per FR-012), commit_checkpoint (FR-014 format),
  is_git_available (cached subprocess probe).

- MODIFIED src/exports.py: refactored _to_role_content to share a
  _message_payload helper (no behavior change; all 43 existing tests pass).
  Added _to_role_content_with_ids and _provider_from_adapter. Extended
  extract_conversation to populate ChatTranscript.metadata with provider,
  models_used, and per-exchange message ids (FR-003 + FR-004). No new
  dependencies.

- NEW tests/test_history.py: 37 tests covering all 11 subtests (a)-(k)
  enumerated in T005. CheckpointRecordRead forward-compat verified
  end-to-end including the FR-020 negative-invariant (no SHA fields).

Gate (uv run pytest + mypy --strict + ruff + black): all green.
276 passed, 1 skipped (T012 fixture file bridge waiting).

Marks T001-T005 done in tasks.md. Implementation surface for US1/US2
helpers (T006-T008, T014-T016) also landed in this commit since the
whole src/history.py module was written as a coherent unit; those
tasks will be marked done in the next commit alongside the orchestrator
wiring (T009-T010).
Wires the history module into the per-checkpoint loop in run_batch
so the FR-017 steps 1+2 of the provenance bookkeeping land after
every successful Editor return and before the Spec 004 cursor save.

src/orchestrator.py:
- New private helpers: _sanitize_conversation_subdir (conv-id or _flat
  sentinel), _editor_decision_to_record (EditorDecision → history
  EditorDecisionRecord with file-path derivation + signals.model_dump()),
  _build_exchange_records (resolves per-exchange message ids from
  transcript.metadata).
- New _write_provenance() helper: builds CheckpointRecord via
  compute_checkpoint_payload, atomically writes
  .history/checkpoints/<conv-subdir>/cp-<NNN>.json, then per-page
  cumulative frontmatter merge with prior-pointer fallback. All errors
  caught at function boundary per FR-019; never propagates to fail
  the run.
- run_batch captures session_log_path return from write_session_log
  and calls _write_provenance after Historian digest merge, wrapped
  in belt-and-suspenders try/except for FR-019 contract clarity.
- Adds src.history + src.wiki imports; no new dependencies.

.claude/agents/editor.md: documentation-only note under "What You Do
NOT Do" stating the orchestrator owns the provenance: frontmatter
block (FR-017). No behavior change in the agent prompt.

tests/fixtures/provenance_cp_001.json: hand-authored fixture with
realistic conversation + exchanges + editor decision plus injected
unknown top-level + sub-block extras to exercise the
CheckpointRecordRead forward-compatibility path (FR-002).

Gate: ruff + black + mypy --strict + pytest all green.
276 passed (1 skip turned to PASS now that T012 fixture exists).
T013: 14 integration tests for _write_provenance covering all US1
acceptance scenarios and edge cases (a-n):
  (a) end-to-end JSON + frontmatter write
  (b) cumulative merge across two checkpoints (total_edits, exchange_count union)
  (c) FR-001a immutability collision (refuses overwrite, page frontmatter NOT touched)
  (d) FR-009 prior-pointer fallback warning (missing prior checkpoint → upper-bound)
  (e) FR-010 malformed YAML (page unchanged, JSON still writes)
  (f) FR-011 frontmatter atomicity (no leftover .tmp files)
  (g) recoverable Editor parse failure (empty decisions → valid JSON)
  (h) empty checkpoint (R10: JSON written, no frontmatter writes)
  (i) page disappeared edge case G12 (logs, decision still recorded)
  (j) _flat sentinel + flat-array transcript (null conv.id, _flat subdir)
  (k) FR-019: OSError in write_checkpoint_metadata caught + logged, run continues
  (l) FR-005 signals coercion via repr() for non-JSON-serializable values
  (m) FR-018 session log byte-identical before/after provenance
  (n) FR-021 process kill via KeyboardInterrupt mid-write, clean re-run

Two source-side fixes surfaced during test development:

1. src/orchestrator.py: added _normalize_vault_relative_path helper
   so EditorDecisionRecord.file and ResultsRecord.pages_* are stored
   in the canonical InsightMesh/-prefixed vault-root-relative shape
   per data-model.md. Editor may emit either bare filename or
   prefixed form; the orchestrator now normalizes both into one
   consistent contract.

2. src/history.py: changed write_checkpoint_metadata and
   merge_page_provenance from try/except Exception to try/finally
   for the .tmp cleanup. KeyboardInterrupt is a BaseException, not
   Exception, and the old except-clause didn't catch it — leaving
   orphan .tmp files in the kill scenario. try/finally cleans up
   unconditionally and the os.path.exists guard keeps the happy
   path a safe no-op.

Gate: ruff + black + mypy --strict (zero new errors above baseline) +
pytest all green. 291 passed (was 276 + 14 new + 1 fixture-bridge
turned passing). Marks T013 done in tasks.md.

US1 (P1) is now feature-complete and independently shippable as MVP.
US2 (T017 + T018) layers shadow-repo diff history on top.
US2 layers the shadow git repo + page snapshots + machine-greppable
commits on top of US1's structured JSON + frontmatter.

src/orchestrator.py:
- Imports the US2 helpers from src.history (is_git_available,
  init_shadow_repo, snapshot_page, commit_checkpoint, plus exceptions).
- Extends _write_provenance in-place per F1 resolution from /reconcile:
  same function now performs FR-017 steps 1+2 (US1) AND steps 3+4 (US2)
  in order. Step 3: snapshot each touched page into .history/pages/
  with slug = decision.file.removeprefix("InsightMesh/") (defensive
  per-page FileNotFoundError + OSError isolation). Step 4: skip cleanly
  when git not on PATH (FR-015); call init_shadow_repo (idempotent per
  FR-012's three-state contract); call commit_checkpoint; catch
  ShadowRepoUnavailable / ShadowRepoCommitFailed per FR-016 with
  [provenance] prefix + run still exits 0.

tests/test_orchestrator.py: 10 new US2 integration tests (T018 a-j):
- (a) Single-commit checkpoint w/ FR-014 subject + body
- (b) Two-checkpoint diff history (git log -p shows added line)
- (c+d) FR-012 init idempotency: pre-existing .git/ untouched;
  .history/ with no .git/ gets re-initialized non-destructively
- (e) No-git fallback (patched is_git_available=False) lands JSON +
  frontmatter, no .git/, stderr [provenance] git not on PATH
- (f) Commit-failure fallback: artifact-on-disk + sweep on next commit
- (g) Empty-result checkpoint: JSON writes, no .git/ no pages/
- (h) User manual commits in .history/ coexist non-destructively
- (i) Pre-populated .history/ advances monotonically (cp-001 → cp-002)
- (j) Standalone commit_checkpoint unit test verifies FR-014 message
  format byte-precisely from real git output

Tests requiring git use shutil.which guard + skip_if_no_git; the
no-git fallback test patches src.orchestrator.is_git_available so
it runs in any environment. Module-scope _GIT_AVAILABLE_CACHE reset
between tests via autouse fixture.

T019: full constitution gate green (uv run pytest + mypy --strict +
ruff + black + ruff TID251). 301 tests pass, 0 mypy errors above the
pre-existing 14-error baseline in test_orchestrator.py (none from
Spec 005 additions).

T021 docs: docs/known-limitations.md gains a "No per-page provenance
— RESOLVED in Spec 005" section covering JSON sidecar + frontmatter +
shadow-git architecture + the no-git fallback + cross-link to the
quickstart + the planned insightmesh-obsidian viewer plugin.
docs/index.md status table gains a Spec 005 row. mkdocs build --strict
passes.

US1 + US2 + Polish (minus T020 real-data smoke) are now feature-complete.
T020 remains open as the user-driven validation step against real exports.
Real-data smoke (Spec 005 T020) surfaced that MCPVault's
WikiPageResult.file_path is an absolute filesystem path
(/Users/.../Vault/InsightMesh/Page.md), not a vault-relative one.
_normalize_vault_relative_path only handled the "bare filename" and
"already prefixed" inputs, so it blindly prepended InsightMesh/ to
the absolute path producing visible-garbage like:

  InsightMesh//Users/me/Vault/InsightMesh/Page.md

in cp-001.json's results.pages_created / pages_updated.

editor.decisions[].file was correctly normalized in the smoke (because
candidate_existing_page was None and sanitize_filename(draft_title) gave
a bare filename), but the same bug would fire on updates where Editor
sets candidate_existing_page to an absolute path from a prior MCPVault
read.

Fix: _normalize_vault_relative_path takes an optional vault_root and
strips it from absolute inputs before prefixing InsightMesh/. Both
_editor_decision_to_record and _write_provenance now pass vault_root
on every call. Absolute paths outside the vault leave the path
verbatim so the malformation is loud enough to spot in downstream
diagnostics (defensive: visibly-wrong beats silently-wrong).

Added regression test TestProvenanceAbsolutePathNormalization that
constructs a WikiPageResult with an absolute path AND an
EditorDecision with an absolute candidate_existing_page, asserting
both decisions[].file and results.pages_created normalize to
"InsightMesh/<page>.md" with no double-prefix or absolute-path leak.

Gate: ruff + black + mypy --strict + pytest all green.
302 passed (was 301; +1 regression test).
docs/index.md: replace the "Latest: Spec 004" admonition with a Spec
005 summary covering the provenance frontmatter block, the per-
checkpoint JSON record, the shadow git diff history, and the planned
insightmesh-obsidian viewer plugin. Spec 004 demoted to "prior
milestones still apply" alongside Spec 002 + 003.

docs/reference/cli.md: new "Per-checkpoint provenance record (Spec
005)" section under On-disk artifacts. Documents the three artifact
shapes (cp-<NNN>.json under per-conversation subdir, frontmatter
provenance: block on touched pages, shadow git repo at .history/.git/),
includes jq + git log recipes for inspection, notes the schema_version
=1 + additive forward-compat rule, lists the fallback behavior (no git
on PATH, commit failure, malformed YAML — all non-fatal per FR-019),
and cross-references the companion insightmesh-obsidian viewer plugin
repo. Plus a one-line cross-reference from the Vault output section to
the new provenance block.

tasks.md: T020 (real-data smoke) marked complete after the third smoke
on tests/fixtures/long_chat_export.json validated the flat-array code
path (_flat sentinel subdir, null fields end-to-end). All 21 tasks in
Spec 005 now done.

mkdocs build --strict passes. Findings memory project_spec005_findings
already updated with full 3-smoke matrix.
Bump pyproject.toml version 0.4.0 → 0.5.0 and uv.lock entry to match.
Matches the Spec 004 → v0.4.0 release pattern from PR #4.

README.md: replace the "Latest (v0.4.0): Spec 004" admonition with a
v0.5.0 / Spec 005 summary (provenance frontmatter block + per-checkpoint
JSON + shadow git diff history + all three transcript shapes validated
on real data + pointer to the planned insightmesh-obsidian viewer
plugin). Status table gains a Spec 005 row. Spec 004 stays in the
status table as the immediately-prior milestone.

docs/reference/cli.md: bump the `insightmesh --version` example output
from 0.4.0 → 0.5.0.

Gate green: insightmesh --version reports 0.5.0; 302 tests pass; mypy
--strict clean across 9 source files; ruff + black + mkdocs --strict
all pass.

Historical attribution references to v0.4.0 in how-to/long-conversations.md
and how-to/troubleshooting.md are intentionally left alone (those
describe when a feature was INTRODUCED, not the current version).
@aucontraire aucontraire self-assigned this Jun 29, 2026
@aucontraire
aucontraire merged commit fd9eaf4 into master Jun 29, 2026
2 checks passed
@aucontraire
aucontraire deleted the 005-page-provenance branch June 29, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant