docs: add section-detection plan and reorganize documentation#9
Merged
Conversation
Plan for a detect_sections stage that auto-drafts sections.txt for multi-section city directories (alphabetical / street / classified / frontmatter), unlocking per-section NER prompt routing. The consumer side (section_utils + extract_entries) is already wired; this plans the missing producer. Phased, heuristic-first with an optional model backstop, with verification ground rules and a session log for a local implementation pass. https://claude.ai/code/session_019WuZdFZZXu5wb7CFeEP5dE
Organize docs/ by lifecycle: point-in-time roadmaps move under docs/plans/ (huggingface-uv-scripts, section-detection-plan, aws-migration-plan), each gaining a Status: line. Evergreen reference + background docs stay flat. Add docs/README.md as a categorized index (Reference / Background / Plans) and link it from README.md. Repoint the moved-file references in CLAUDE.md and the section-detection plan's internal link. Document the dangling CLAUDE.md reference to refactor-plan.md (a local-only working doc never committed) as a known gap in CLAUDE.md and the index rather than leaving it as a silent broken link. The iiif-content-search.md reference in comparison-htr-alto-pipeline.md is an upstream-project citation, left as-is. https://claude.ai/code/session_019WuZdFZZXu5wb7CFeEP5dE
The multi-session simplification refactor is done. Remove the now-stale 'Active work: simplification plan' section from CLAUDE.md (it pointed at a local-only refactor-plan.md that was never committed) and drop the matching index entry. No remaining references to refactor-plan remain in the repo. https://claude.ai/code/session_019WuZdFZZXu5wb7CFeEP5dE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a comprehensive implementation plan for automatic section detection in multi-section city directories, plus reorganizes documentation with a new index.
Key changes
New plan:
docs/plans/section-detection-plan.md— a detailed 6-phase roadmap for building adetect_sectionspipeline stage that auto-draftssections.txtby analyzing per-page OCR features (geometry, text patterns, density). The plan is scoped to produce a draft artifact only (never overwriting curated files), mirrors the existingdetect_columnsstage pattern, and defers implementation to a local session with actualoutput/data on disk.Documentation index:
docs/README.md— new entry point organizing reference docs (pipeline-stages, usage-examples, key-design-decisions, costs), background (prior-work, comparison-htr-alto-pipeline), and plans (section-detection, huggingface-uv-scripts, aws-migration) with status indicators.Plan status headers: Added
Status:lines todocs/plans/huggingface-uv-scripts.mdanddocs/plans/aws-migration-plan.mdto clarify that these are point-in-time roadmaps, not active work.Updated CLAUDE.md: Corrected reference to the new
docs/plans/huggingface-uv-scripts.mdlocation and clarified that the local-models strategy is deferred to a future session.Updated README.md: Added links to the new docs index and plans.
Implementation notes
The section-detection plan is not code yet — it is a detailed specification for a future implementer. Key design decisions:
sections.txtis fully wired inextract_entries.py,run_gemini_ocr.py,generate_prompt.py, andselect_pages.pyviautils/section_utils.py. The plan only needs to build the producer.output/tulsa_1921/andoutput/hearnes_brooklyn_city_directory_for_hearnesbrooklync1852/to confirm sections separate cleanly on these features and calibrate thresholds.sections_draft.txt+sections_report.csv; user reviews and promotes the draft tosections.txt.detect_columns, it is declarative instages.pyand only runs when--detect-sectionsis passed.The plan includes a full file/function map, test strategy, risk caveats, and a session log template for tracking progress.
https://claude.ai/code/session_019WuZdFZZXu5wb7CFeEP5dE