Repository navigation
Spec 005: per-page provenance (v0.5.0) - #6
Merged
Merged
Conversation
Drafts spec/plan/tasks/contracts/data-model for per-page provenance: structured per-checkpoint JSON sidecar files, cumulative frontmatter provenance blocks on each touched wiki page, and a shadow git repository at <vault>/InsightMesh/.history/ for diff history. 25 FRs (including FR-001a/b for immutability + deterministic ordering, FR-016a for stderr format, FR-021/022 for crash resilience), 7 SCs, 2 user stories: US1 P1 (structured JSON + cumulative frontmatter) ships as MVP without requiring git; US2 P2 (shadow-git diff history) layers on top. 21 tasks across 5 phases, 100% FR-to-task coverage verified via /speckit-analyze + /reconcile. Designed as a stable read contract for the planned in-Obsidian viewer plugin (separate repo: aucontraire/insightmesh-obsidian); no Spec 005 changes are required to enable that plugin. Also updates CLAUDE.md plan reference and .specify/feature.json to point at this spec.
Foundational layer for per-page provenance: - NEW src/history.py (659 lines): 8 Pydantic v2 models (CheckpointRecord, ConversationRecord, ExchangeRecord, EditorDecisionRecord, EditorBlock, ResultsRecord, LinksRecord, ProvenanceFrontmatter, ExchangeMessageIds) with strict=True + extra=forbid for write-side; CheckpointRecordRead subclass with extra=allow per FR-002 additive-evolution rule. Custom exceptions (HistoryError + ShadowRepoUnavailable + ShadowRepoCommitFailed + FrontmatterParseFailed). Pure helpers: compute_checkpoint_payload, write_checkpoint_metadata (atomic per FR-001 + immutability per FR-001a), merge_page_provenance (cumulative math per FR-009 + atomic per FR-011), snapshot_page, init_shadow_repo (three-state per FR-012), commit_checkpoint (FR-014 format), is_git_available (cached subprocess probe). - MODIFIED src/exports.py: refactored _to_role_content to share a _message_payload helper (no behavior change; all 43 existing tests pass). Added _to_role_content_with_ids and _provider_from_adapter. Extended extract_conversation to populate ChatTranscript.metadata with provider, models_used, and per-exchange message ids (FR-003 + FR-004). No new dependencies. - NEW tests/test_history.py: 37 tests covering all 11 subtests (a)-(k) enumerated in T005. CheckpointRecordRead forward-compat verified end-to-end including the FR-020 negative-invariant (no SHA fields). Gate (uv run pytest + mypy --strict + ruff + black): all green. 276 passed, 1 skipped (T012 fixture file bridge waiting). Marks T001-T005 done in tasks.md. Implementation surface for US1/US2 helpers (T006-T008, T014-T016) also landed in this commit since the whole src/history.py module was written as a coherent unit; those tasks will be marked done in the next commit alongside the orchestrator wiring (T009-T010).
Wires the history module into the per-checkpoint loop in run_batch so the FR-017 steps 1+2 of the provenance bookkeeping land after every successful Editor return and before the Spec 004 cursor save. src/orchestrator.py: - New private helpers: _sanitize_conversation_subdir (conv-id or _flat sentinel), _editor_decision_to_record (EditorDecision → history EditorDecisionRecord with file-path derivation + signals.model_dump()), _build_exchange_records (resolves per-exchange message ids from transcript.metadata). - New _write_provenance() helper: builds CheckpointRecord via compute_checkpoint_payload, atomically writes .history/checkpoints/<conv-subdir>/cp-<NNN>.json, then per-page cumulative frontmatter merge with prior-pointer fallback. All errors caught at function boundary per FR-019; never propagates to fail the run. - run_batch captures session_log_path return from write_session_log and calls _write_provenance after Historian digest merge, wrapped in belt-and-suspenders try/except for FR-019 contract clarity. - Adds src.history + src.wiki imports; no new dependencies. .claude/agents/editor.md: documentation-only note under "What You Do NOT Do" stating the orchestrator owns the provenance: frontmatter block (FR-017). No behavior change in the agent prompt. tests/fixtures/provenance_cp_001.json: hand-authored fixture with realistic conversation + exchanges + editor decision plus injected unknown top-level + sub-block extras to exercise the CheckpointRecordRead forward-compatibility path (FR-002). Gate: ruff + black + mypy --strict + pytest all green. 276 passed (1 skip turned to PASS now that T012 fixture exists).
T013: 14 integration tests for _write_provenance covering all US1 acceptance scenarios and edge cases (a-n): (a) end-to-end JSON + frontmatter write (b) cumulative merge across two checkpoints (total_edits, exchange_count union) (c) FR-001a immutability collision (refuses overwrite, page frontmatter NOT touched) (d) FR-009 prior-pointer fallback warning (missing prior checkpoint → upper-bound) (e) FR-010 malformed YAML (page unchanged, JSON still writes) (f) FR-011 frontmatter atomicity (no leftover .tmp files) (g) recoverable Editor parse failure (empty decisions → valid JSON) (h) empty checkpoint (R10: JSON written, no frontmatter writes) (i) page disappeared edge case G12 (logs, decision still recorded) (j) _flat sentinel + flat-array transcript (null conv.id, _flat subdir) (k) FR-019: OSError in write_checkpoint_metadata caught + logged, run continues (l) FR-005 signals coercion via repr() for non-JSON-serializable values (m) FR-018 session log byte-identical before/after provenance (n) FR-021 process kill via KeyboardInterrupt mid-write, clean re-run Two source-side fixes surfaced during test development: 1. src/orchestrator.py: added _normalize_vault_relative_path helper so EditorDecisionRecord.file and ResultsRecord.pages_* are stored in the canonical InsightMesh/-prefixed vault-root-relative shape per data-model.md. Editor may emit either bare filename or prefixed form; the orchestrator now normalizes both into one consistent contract. 2. src/history.py: changed write_checkpoint_metadata and merge_page_provenance from try/except Exception to try/finally for the .tmp cleanup. KeyboardInterrupt is a BaseException, not Exception, and the old except-clause didn't catch it — leaving orphan .tmp files in the kill scenario. try/finally cleans up unconditionally and the os.path.exists guard keeps the happy path a safe no-op. Gate: ruff + black + mypy --strict (zero new errors above baseline) + pytest all green. 291 passed (was 276 + 14 new + 1 fixture-bridge turned passing). Marks T013 done in tasks.md. US1 (P1) is now feature-complete and independently shippable as MVP. US2 (T017 + T018) layers shadow-repo diff history on top.
US2 layers the shadow git repo + page snapshots + machine-greppable
commits on top of US1's structured JSON + frontmatter.
src/orchestrator.py:
- Imports the US2 helpers from src.history (is_git_available,
init_shadow_repo, snapshot_page, commit_checkpoint, plus exceptions).
- Extends _write_provenance in-place per F1 resolution from /reconcile:
same function now performs FR-017 steps 1+2 (US1) AND steps 3+4 (US2)
in order. Step 3: snapshot each touched page into .history/pages/
with slug = decision.file.removeprefix("InsightMesh/") (defensive
per-page FileNotFoundError + OSError isolation). Step 4: skip cleanly
when git not on PATH (FR-015); call init_shadow_repo (idempotent per
FR-012's three-state contract); call commit_checkpoint; catch
ShadowRepoUnavailable / ShadowRepoCommitFailed per FR-016 with
[provenance] prefix + run still exits 0.
tests/test_orchestrator.py: 10 new US2 integration tests (T018 a-j):
- (a) Single-commit checkpoint w/ FR-014 subject + body
- (b) Two-checkpoint diff history (git log -p shows added line)
- (c+d) FR-012 init idempotency: pre-existing .git/ untouched;
.history/ with no .git/ gets re-initialized non-destructively
- (e) No-git fallback (patched is_git_available=False) lands JSON +
frontmatter, no .git/, stderr [provenance] git not on PATH
- (f) Commit-failure fallback: artifact-on-disk + sweep on next commit
- (g) Empty-result checkpoint: JSON writes, no .git/ no pages/
- (h) User manual commits in .history/ coexist non-destructively
- (i) Pre-populated .history/ advances monotonically (cp-001 → cp-002)
- (j) Standalone commit_checkpoint unit test verifies FR-014 message
format byte-precisely from real git output
Tests requiring git use shutil.which guard + skip_if_no_git; the
no-git fallback test patches src.orchestrator.is_git_available so
it runs in any environment. Module-scope _GIT_AVAILABLE_CACHE reset
between tests via autouse fixture.
T019: full constitution gate green (uv run pytest + mypy --strict +
ruff + black + ruff TID251). 301 tests pass, 0 mypy errors above the
pre-existing 14-error baseline in test_orchestrator.py (none from
Spec 005 additions).
T021 docs: docs/known-limitations.md gains a "No per-page provenance
— RESOLVED in Spec 005" section covering JSON sidecar + frontmatter +
shadow-git architecture + the no-git fallback + cross-link to the
quickstart + the planned insightmesh-obsidian viewer plugin.
docs/index.md status table gains a Spec 005 row. mkdocs build --strict
passes.
US1 + US2 + Polish (minus T020 real-data smoke) are now feature-complete.
T020 remains open as the user-driven validation step against real exports.
Real-data smoke (Spec 005 T020) surfaced that MCPVault's WikiPageResult.file_path is an absolute filesystem path (/Users/.../Vault/InsightMesh/Page.md), not a vault-relative one. _normalize_vault_relative_path only handled the "bare filename" and "already prefixed" inputs, so it blindly prepended InsightMesh/ to the absolute path producing visible-garbage like: InsightMesh//Users/me/Vault/InsightMesh/Page.md in cp-001.json's results.pages_created / pages_updated. editor.decisions[].file was correctly normalized in the smoke (because candidate_existing_page was None and sanitize_filename(draft_title) gave a bare filename), but the same bug would fire on updates where Editor sets candidate_existing_page to an absolute path from a prior MCPVault read. Fix: _normalize_vault_relative_path takes an optional vault_root and strips it from absolute inputs before prefixing InsightMesh/. Both _editor_decision_to_record and _write_provenance now pass vault_root on every call. Absolute paths outside the vault leave the path verbatim so the malformation is loud enough to spot in downstream diagnostics (defensive: visibly-wrong beats silently-wrong). Added regression test TestProvenanceAbsolutePathNormalization that constructs a WikiPageResult with an absolute path AND an EditorDecision with an absolute candidate_existing_page, asserting both decisions[].file and results.pages_created normalize to "InsightMesh/<page>.md" with no double-prefix or absolute-path leak. Gate: ruff + black + mypy --strict + pytest all green. 302 passed (was 301; +1 regression test).
docs/index.md: replace the "Latest: Spec 004" admonition with a Spec 005 summary covering the provenance frontmatter block, the per- checkpoint JSON record, the shadow git diff history, and the planned insightmesh-obsidian viewer plugin. Spec 004 demoted to "prior milestones still apply" alongside Spec 002 + 003. docs/reference/cli.md: new "Per-checkpoint provenance record (Spec 005)" section under On-disk artifacts. Documents the three artifact shapes (cp-<NNN>.json under per-conversation subdir, frontmatter provenance: block on touched pages, shadow git repo at .history/.git/), includes jq + git log recipes for inspection, notes the schema_version =1 + additive forward-compat rule, lists the fallback behavior (no git on PATH, commit failure, malformed YAML — all non-fatal per FR-019), and cross-references the companion insightmesh-obsidian viewer plugin repo. Plus a one-line cross-reference from the Vault output section to the new provenance block. tasks.md: T020 (real-data smoke) marked complete after the third smoke on tests/fixtures/long_chat_export.json validated the flat-array code path (_flat sentinel subdir, null fields end-to-end). All 21 tasks in Spec 005 now done. mkdocs build --strict passes. Findings memory project_spec005_findings already updated with full 3-smoke matrix.
Bump pyproject.toml version 0.4.0 → 0.5.0 and uv.lock entry to match. Matches the Spec 004 → v0.4.0 release pattern from PR #4. README.md: replace the "Latest (v0.4.0): Spec 004" admonition with a v0.5.0 / Spec 005 summary (provenance frontmatter block + per-checkpoint JSON + shadow git diff history + all three transcript shapes validated on real data + pointer to the planned insightmesh-obsidian viewer plugin). Status table gains a Spec 005 row. Spec 004 stays in the status table as the immediately-prior milestone. docs/reference/cli.md: bump the `insightmesh --version` example output from 0.4.0 → 0.5.0. Gate green: insightmesh --version reports 0.5.0; 302 tests pass; mypy --strict clean across 9 source files; ruff + black + mkdocs --strict all pass. Historical attribution references to v0.4.0 in how-to/long-conversations.md and how-to/troubleshooting.md are intentionally left alone (those describe when a feature was INTRODUCED, not the current version).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
provenance:frontmatter block (latest checkpoint, contributing conversations, action, confidence, total edits, exchange count) backed by a permanent per-checkpoint JSON record at<vault>/InsightMesh/.history/checkpoints/<conv-id>/cp-<NNN>.jsonand a shadow git repository forgit log -pstyle diff history of every page's evolution across edits.schema_version=1) for the planned in-Obsidian viewer plugin in a separate repo: aucontraire/insightmesh-obsidian.What's new
src/history.py(NEW, ~660 lines): 8 Pydantic v2 models (CheckpointRecord+ supporting +ProvenanceFrontmatter),CheckpointRecordReadpermissive read-side subclass per FR-002, 4 custom exceptions, 7 pure helpers (compute_checkpoint_payload,write_checkpoint_metadataatomic + immutable per FR-001a,merge_page_provenancecumulative + atomic per FR-011,snapshot_page,init_shadow_repoidempotent three-state per FR-012,commit_checkpointFR-014 format,is_git_availablecached). No new runtime dependencies.src/orchestrator.py(modified): new_sanitize_conversation_subdir,_normalize_vault_relative_path,_editor_decision_to_record,_build_exchange_records,_write_provenancehelpers._write_provenanceis wired into the existing Spec 004 checkpoint loop after Editor success and before the cursor save, per FR-017.src/exports.py(modified): refactored_to_role_contentto share a_message_payloadhelper; added_to_role_content_with_ids+_provider_from_adapter.extract_conversationnow populatesChatTranscript.metadatawithprovider,models_used,exchange_message_ids..claude/agents/editor.md(doc-only): note that orchestrator owns theprovenance:frontmatter block per FR-017.Tests:
tests/test_history.pyNEW (37 tests) +tests/test_orchestrator.pyextended (+25 tests: 14 US1 + 10 US2 + 1 absolute-path regression).tests/fixtures/provenance_cp_001.jsonNEW for forward-compat fixture.Docs:
docs/index.mdadmonition refreshed to feature Spec 005;docs/reference/cli.mdnew "Per-checkpoint provenance record (Spec 005)" section under On-disk artifacts;docs/known-limitations.mdgains a "RESOLVED in Spec 005" entry;README.mdupdated; status tables refreshed acrossindex.md+README.md.On-disk artifacts (per checkpoint)
Test plan
uv run pytest: 302 passinguv run mypy --strict src/: clean across 9 source filesuv run ruff check src/ tests/: all checks passeduv run black --check src/ tests/: 19 files cleanuv run ruff check --select TID: no banned-API violations (constitution v1.1.2 §Project Standards)uv run mkdocs build --strict: cleaninsightmesh --version→0.5.0provider="anthropic",models_used=[], real echomine UUIDs), ChatGPT conversation (provider="openai",models_used=["gpt-4o"], real ids), Spec 001 flat-array (provider=null,conv_id=null,_flat/sentinel subdir, null message ids). Findings memory:project_spec005_findings.md.Bugs found + fixed during real-data smoke
S1:
WikiPageResult.file_pathfrom MCPVault is an absolute filesystem path, not vault-relative._normalize_vault_relative_pathoriginally only handled bare-filename + already-prefixed inputs, producing garbage likeInsightMesh//Users/.../InsightMesh/Page.mdinresults.pages_created/pages_updated. Fix in commitbdea9dfwith regression testTestProvenanceAbsolutePathNormalization. Same lesson as Spec 004 findings: mocked tests with synthetic defaults silently mask the failures real Editor output surfaces.Spec process (followed full speckit cycle)
/speckit-specify→/speckit-clarify(1 question, locked per-conversation subdir naming) →/speckit-plan→/speckit-tasks(21 tasks) →/speckit-checklist(added seconddurability-and-resiliencechecklist of 38 items) →/spec-gaps all(15 remediation groups applied) →/speckit-analyze(1 MEDIUM + 4 LOW found) →/reconcile all(all 5 closed) →/speckit-implement→ real-data smoke (T020). 21/21 tasks done, both checklists 100% (16+38 items checked).Constitution compliance
src/file (src/history.py) justified in plan.md Complexity Tracking; modifications tosrc/orchestrator.pyandsrc/exports.pysurgical; no new dependenciespydantic.BaseModel(ConfigDict(strict=True))— the one spec-mandatedextra="allow"deviation onCheckpointRecordReadis documented inline per FR-002Out of scope (deferred to future specs)
insightmesh history <page>subcommand.history/Full list in
specs/005-page-provenance/spec.md§ Out of Scope.