feat(markdown): strip page-edge furniture on short documents - #427
Conversation
Running headers, footer credits, and page indicators are proven by cross-page repetition, which needs three-plus pages; on one- and two-page documents they survived untouched. Add a positional and typographic classifier for that gap, dispatched explicitly by document length (strip_header_footer_lines), and consolidate all header/footer policy — repetition classifier, edge classifier, shared predicates — into a new markdown::furniture module. The edge classifier only strips where evidence is strong: the outermost block (<= 2 lines) per page edge, vertically isolated from the body by 1.8x the median leading, set strictly smaller than body text, short, and non-structural. Guards, each earned by a measured false positive: - same-size blocks stay (section headings, affiliations, continuation paragraphs merely sit at the margin) - digit- or (digit)-led blocks stay (footnotes, legal enumerations) - three-plus item clusters on a baseline stay (figure label rows, numeric rows of undetected borderless tables) - number-only blocks set above body size stay (cover years); at or below body size they strip on isolation alone (page indicators) - pages under 8 lines and blocks over 90 chars are never touched
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/markdown/furniture.rs">
<violation number="1" location="src/markdown/furniture.rs:152">
P1: An isolated one- or two-cell numeric row is stripped even though it can be substantive table or form content. Preserve numeric blocks unless they match explicit folio syntax or have stronger page-indicator evidence, rather than treating every small, isolated numeric block as furniture.
(Based on your team's feedback about requiring text context in numeric runs.)</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
| if block.iter().any(|&i| line_font(i) > body_size * 1.2) { | ||
| return None; | ||
| } | ||
| return Some(block.to_vec()); |
There was a problem hiding this comment.
P1: An isolated one- or two-cell numeric row is stripped even though it can be substantive table or form content. Preserve numeric blocks unless they match explicit folio syntax or have stronger page-indicator evidence, rather than treating every small, isolated numeric block as furniture.
(Based on your team's feedback about requiring text context in numeric runs.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/markdown/furniture.rs, line 152:
<comment>An isolated one- or two-cell numeric row is stripped even though it can be substantive table or form content. Preserve numeric blocks unless they match explicit folio syntax or have stronger page-indicator evidence, rather than treating every small, isolated numeric block as furniture.
(Based on your team's feedback about requiring text context in numeric runs.) </comment>
<file context>
@@ -0,0 +1,799 @@
+ if block.iter().any(|&i| line_font(i) > body_size * 1.2) {
+ return None;
+ }
+ return Some(block.to_vec());
+ }
+ // Textual furniture must be set strictly smaller than body text.
</file context>
There was a problem hiding this comment.
Valid — fixed in 9e8947b, one step stricter than suggested: number-only blocks now strip only on the full page-indicator shape (a single text run, ≤20 chars, set no larger than body text). Multi-cell numeric rows keep on cluster evidence and long numeric strings keep on length, with a two-cell row test ("42.50 38.20" at the bottom edge survives). Measured cost: 3 fewer strips on the header/footer bench (420→417, still well above the pre-change 328), zero movement elsewhere.
- Number-only blocks strip only on the full page-indicator shape: one
text run, <= 20 chars, set no larger than body text. Multi-cell
numeric rows are form or table data and stay.
- Marker-led guard extends to symbol footnotes ('† ...') and single
lowercase-letter markers ('a See ...'); capitalized one-letter words
('A Publication ...') remain strippable prose.
- The repetition classifier now checks structural shape on the original
text: normalization strips leading digits, so numbered headings like
'1. Introduction' were losing their protection along with the number.
- Restore the repetition classifier's doc comment to strip_repeated_lines;
the module move had left it fused onto the dispatcher.
|
@cubic-dev-ai review the latest changes — all four findings addressed in 9e8947b (three fixes with tests, doc restoration). |
@abimaelmartell I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
There was a problem hiding this comment.
1 existing issue remains and no new issues found across 3 files
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
…t (letter) markers
- The repetition classifier's structural check ran only at candidate
collection; a structural line sharing a normalized key with a
non-structural candidate (normalization strips the heading's leading
digit) was still stripped by the removal passes. All three passes —
individual, Y-band, and sibling propagation — now recheck the
original text.
- The marker-led guard accepts parenthesized-letter enumerations
('(a) See ...'), common in legal and scholarly documents.
|
@cubic-dev-ai review the latest changes — both round-2 findings fixed in c93728e with regression tests. |
@abimaelmartell I have started the AI code review. It will take a few minutes to complete. |
…git) enumerations
- The ten-character candidate minimum counted UTF-8 bytes, so short
non-ASCII headings (a six-character CJK section head is 18 bytes)
slipped past the floor and were stripped as repeated furniture. Count
characters in both frequency loops; a Japanese corpus doc recovers a
repeated section heading.
- is_structural_line recognizes parenthesized numbered markers
('(1) Sign and date ...'), protecting repeated enumerations in the
repetition classifier the same way bare numbered lines already were.
|
@cubic-dev-ai review the latest changes — both remaining findings fixed in 666ff0f (character-count floor with a CJK regression recovered on the corpus, (digit) markers structural). |
@abimaelmartell A review is already in progress. Try again after it finishes. |
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
- The Y-band removal pass inserted whole bands after checking only the
coalesced text, so a structural sibling banded with a plain fragment
was stripped on every page after the first. Each member is now checked
individually, matching the sibling-propagation pass.
- A parenthesized marker must be a short digit/lowercase token closed by
')' — '(1)', '(a)', '(iv)'. Parenthetical prose footers
('(all amounts in thousands)') strip again.
|
@cubic-dev-ai review the latest changes — both findings fixed in 0cf7907 with regression tests. |
@abimaelmartell A review is already in progress. Try again after it finishes. |
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
'(α)'-style footnote markers are lowercase too; the paren-marker check was ASCII-only while the bare single-letter guard already accepted Unicode. Token extraction and validation now use Unicode alphanumeric and lowercase classes, keeping the three-character and closing-paren bounds.
|
@cubic-dev-ai review the latest changes — Unicode paren markers fixed in 95af535 with a test. |
@abimaelmartell I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 existing issue remains and no new issues found across 3 files
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
… floor - A Y-band holding a structural member is one physical row of content: a precomputed protected-band set now guards ALL removal paths (individual, band, propagation), so a heading's row-mates are never half-removed. Replaces the per-member filter, which mangled rows. - Every digit check in the marker and structural guards uses Unicode is_numeric(): '(١)' earns the same protection as '(1)'. - The edge classifier requires strictly more than 8 lines per page — at the floor, the two edge blocks could remove half the content.
|
@cubic-dev-ai review the latest changes — all four findings fixed in ac5b1a3 with tests (whole-row band protection, Unicode digits throughout, stricter page floor). |
@abimaelmartell I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
…icate, boundary-safe row protection - normalize_for_comparison trims Unicode numerics, so headers whose page numbers use non-ASCII digits normalize to one key across pages and the repetition classifier can see them. - The parenthesized-marker rule is one shared predicate (starts_with_paren_marker) used by both the edge classifier and is_structural_line: repeated '(a) See ...' annotations survive on three-plus-page documents too. - Whole-row structural protection extends to neighboring quantization buckets: same-row fragments 0.02pt apart can straddle a rounding boundary, and the boundary must never split a row's protection.
|
@cubic-dev-ai review the latest changes — all three findings from the last review fixed in 64c5b6d with regression tests:
Validation flat: header/footer bench 416/753, order 451/884, corpus 193/193 with zero snapshot changes; 1,154 tests. |
@abimaelmartell I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 existing issue remains and no new issues found across 3 files
Shadow auto-approve: would not auto-approve because issues were found.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
The sort-join-normalize sequence was triplicated across the band frequency build and both band passes, with each copy free to drift. One helper (coalesced_band) now produces the sorted members, the coalesced row text, and its normalized comparison key; each site keeps only its own guards. Behavior-neutral: full suite unchanged.
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.
Re-trigger cubic
1 participant
What
Running headers, footer credits, and page indicators are stripped by proving cross-page repetition — which needs three-plus pages. On one- and two-page documents that evidence doesn't exist, so furniture survived untouched: journal running heads, "downloaded from … by guest" credits, "1 / 1" page indicators, footer attributions.
How
A new
markdown::furnituremodule now owns all header/footer policy: the existing repetition classifier, a new positional/typographic edge classifier for short documents, and an explicit dispatcher (strip_header_footer_lines) that picks the policy by document length.The edge classifier strips only where evidence is strong — the outermost block (≤2 lines) per page edge, vertically isolated from the body by 1.8× the median leading, set strictly smaller than body text, short (≤90 chars), and non-structural. Every guard was earned by a measured false positive, not designed upfront:
A footnote block also protects itself structurally: its outermost line sits at normal leading from its own block, so the run exceeds the block cap.
Results
Notes
The stripping remains behind the existing
strip_headers_footersoption (default on). The known residual trade-off: a short isolated caption set smaller than body text at the extreme page edge of a 1–2 page document can strip; the corpus shows no instance, and the cluster/digit/size guards cover the observed shapes.🤖 Generated with Claude Code
Summary by cubic
Strips page-edge furniture on 1–2‑page documents and hardens repetition-based removal. Previously we only stripped headers/footers proven by cross‑page repetition (≥3 pages), so short docs kept running heads, credits, and page indicators.
markdown::furnitureowns policy and exposesstrip_header_footer_lines;markdown/mod.rsnow calls it.<3pages use the edge classifier;≥3use repetition; the first repeated header/footer occurrence is kept. No API change (strip_headers_footersstays default‑on).coalesced_bandhelper now coalesces/normalizes band text consistently across frequency and removal passes (behavior‑neutral).Written for commit 9bb1349. Summary will update on new commits.