Skip to content

feat: deterministic reference verification for origin archives - #4

Open
sdgsfh wants to merge 2 commits into
DeepMathLLM:mainfrom
sdgsfh:feat/reference-verification-tool
Open

feat: deterministic reference verification for origin archives#4
sdgsfh wants to merge 2 commits into
DeepMathLLM:mainfrom
sdgsfh:feat/reference-verification-tool

Conversation

@sdgsfh

@sdgsfh sdgsfh commented Sep 5, 2026

Copy link
Copy Markdown

Problem

The acceptance gate in run_archive.py (verify_math_object_origin_archive) combines one deterministic layer (format checks) with a model review across three dimensions. The historical dimension currently has no deterministic support: whether a cited source actually exists — and whether its journal, volume, year, and pages are copied correctly — is judged by the model alone. That is precisely where archives fail silently, because a hallucinated or miscopied citation still looks plausible to a reviewer model.

Design

Add tools/verify_references.py, a deterministic reference checker that runs before/around any model review:

  • Structure (offline): contiguous [n] numbering in the Sources section; every in-text citation [n] resolves to a listed source; uncited entries are warnings, not failures.
  • Existence: each source is resolved against external indexes — Crossref (DOI lookup or scored bibliographic query), OpenAlex, zbMATH Open, the Internet Archive, and Open Library. A confident match requires title similarity and surname overlap and a year within ±2, so a real title with fabricated authors (or vice versa) does not pass. zbMATH Open and the Internet Archive extend coverage to the 19th-century literature (Puiseux 1850, Jordan 1870, …) that origin archives routinely cite.
  • Link liveness: DOIs/URLs must resolve (warning by default; --strict escalates to failure).
  • Consistency: year/volume/pages claimed by an entry are compared against the matched records. Claims contradicted by every matched record are failures; disagreements between the indexes themselves (e.g., first-page-only records) are reported as warnings, not silently resolved.

The tool is stdlib-only and never judges whether a source supports a claim — claim-level support stays with the model review. Exit codes: 0 pass / 1 fail / 2 usage error. --json emits a machine-readable report; --offline restricts all lookups to a JSON cache so CI runs without network.

Implementation

  • tools/verify_references.py — CLI and library (verify_archive_text()); fetchers are injectable for testing.
  • tests/test_verify_references.py — 22 offline unittest cases whose fixtures are trimmed real API responses from the five indexes (Keller 1939 via Crossref DOI + OpenAlex + zbMATH, Alon 1999 via bibliographic query, Jordan 1870 via the Internet Archive, Dudeney 1917 via Open Library, …).
  • Negative controls: fabricated citation rejected; wrong DOI attached to an unrelated title rejected; pages contradicted by both indexes rejected; non-contiguous numbering and dangling in-text citations rejected.
  • README section documenting scope, usage, and the integration boundary.
  • No changes to run_archive.py or any production logic.

Verification

  • python -m unittest discover -s tests -p "test_*.py" — 39/39 pass (22 new + the existing 17 runner regressions), Python 3.11.
  • Live run against 9 real origin archives produced with this pipeline (34 references total): 34/35 sources verified. The single failure is the honest boundary — Riemann's 1857 Göttingen academy paper pre-dates zbMATH's 1868 coverage and is absent from the book indexes — and the tool reports it as unverifiable rather than passing it.
  • The live run surfaced three real-world data quirks, each now a regression test: Elsevier DOIs containing parentheses (10.1016/0040-9383(87)90009-7), Crossref family fields that carry initials ("H. Kauffman"), and Crossref titles with footnote suffixes ("…cell complexes††Partially supported by…").

Limitations

  • Claim-level evidence support remains with the model reviewer; this tool only guarantees that cited sources exist and are copied correctly.
  • Title extraction from free-form citations is heuristic (italic spans preferred; comma-segment analysis otherwise); exotic citation styles may need the documented conventions.
  • Book/article title collisions are handled with publisher-hint routing; an entry that matches only a journal article while claiming book-series data will still be flagged.
  • Link checks are warning-level by default because some publishers rate-limit HEAD/GET (observed: intermittent 503s from doi.org for AMS journals); --strict is available for hard enforcement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant