Skip to content

fix: avoid false analysis limits in documentation - #516

Merged
mohgupta-ship-it merged 16 commits into
mainfrom
codex/fix-documentation-analysis-limits
Sep 12, 2026
Merged

fix: avoid false analysis limits in documentation#516
mohgupta-ship-it merged 16 commits into
mainfrom
codex/fix-documentation-analysis-limits

Conversation

@mohgupta-ship-it

@mohgupta-ship-it mohgupta-ship-it commented Sep 10, 2026

Copy link
Copy Markdown
Member

Valid Markdown code spans and JSON request placeholders could produce false incomplete-analysis/AE1 results. Delimiters could also pair across unrelated Markdown blocks or table cells and hide unresolved runtime commands. This change establishes delimiter ownership from complete document context, preserving code bodies and original source coordinates.

  • Recognize all strings in bounded, validated JSON values: object keys/values, array elements and scalar strings, including explicit JSON fences with attributes, list/blockquote containers and standalone JSON after frontmatter. Container indentation follows visual tab columns without expanding payload bytes. Invalid, partial or oversized containers grant no ownership; unknown fragment context stays conservative.
  • Keep inline delimiter pairing within Markdown blocks and proven GFM table cells. Preserve ordinary multiline paragraphs and list continuations, distinguish table rows/headers from Setext headings and real block interruptions, and keep empty list items from creating phantom paragraphs. Retain fenced, indented and raw-HTML code handling, inline Make expressions and repeated documentation references.
  • Recover runtime commands in disjoint validated JSON strings when an earlier parse would otherwise skip them. Retain the whole-content result, raw escapes and source evidence without reparsing overlapping suffixes. Forward JSON traversal and cancellation handling from fix: make JSON quote candidate scanning linear #521 remain intact.
  • Preserve genuine removal-instruction findings and incomplete coverage for unresolved runtime commands. Proven JSON delimiters do not exempt their contents from analysis.

Completeness and risk remain separate: existing findings and thresholds are unchanged. Benign documentation can complete, while unresolved static inspection remains incomplete and fails strict CLI/MCP installation gates even when semantic analysis succeeds.

Validation on frozen source ca0ceb77e252c0ad5dc84eb435323edb80d28c57:

  • Added 295 permanent tests across five modules, including exact reported inputs, paired benign controls, malformed containers, raw quote/source locations, table/JSON composition, parser-state transitions and counted-work/cancellation checks. Tests-first red checkpoints preceded the corresponding corrections. The focused suite passes 722 tests (295 new + 427 existing); an independent clean-source replay passes all 295 new tests.
  • Local make test-ci: 4,793 passed, 14 skipped, 38 provider/integration tests deselected and four expected failures; 89% coverage.
  • 80 permanent public test cases execute 160 CLI/MCP invocations with successful deterministic semantic transports. Enabled mode checks all three semantic analyzers and completed work; disabled mode checks zero calls. Incomplete inputs are rejected, while benign complete controls retain their expected installation decisions.
  • Actual codex_cli replay: 40 fixtures in both modes, 80 CLI scans passing their expected completeness, evidence and exact strict/risk-threshold exit contracts. Enabled scans recorded 150 successful analyzer-level attempts out of 150, with all three semantic analyzers completed and no degraded scans. Disabled scans recorded zero attempts.
  • Supplemental actual-provider MCP check: four exact reported fixtures in both modes, 8/8 passing, with 14/14 analyzer-level attempts successful and zero disabled-mode attempts. This exercises mcp_server.run_scan directly; it does not certify HTTP/stdio wire transport.
  • Fresh Docker build and the unmodified local-skill/GitHub-URL smoke tests pass with --no-llm. All 97 installed Python source hashes match the frozen revision. Live-provider replay ran separately on the host using that same verified source.
  • Repository lint/format, diff checks, secret scan, DCO and independent implementation review pass. Three pre-existing pytest timeout annotations were unregistered in the local environment; the new work/cancellation tests assert counters and callbacks directly.

The targeted and independent runs overlap the full suite; their counts are not additional unique tests. Provider telemetry counts analyzer-level attempts, not individual underlying model requests. These results supersede the earlier validation summary for b5c5d8e. A separate pre-existing legacy-backtick interpretation gap is not claimed fixed.

All five hosted CI checks pass on published ca0ceb77e252c0ad5dc84eb435323edb80d28c57: changes, lint, test-unit, DCO and docker-smoke (CI run). The Linux full suite independently matches the local result: 4,793 passed, 14 skipped, 38 deselected, four expected failures and 89% coverage.

Fixes #515.

Prepared by Codex on behalf of Mohit Gupta.

Distinguish Markdown code delimiters from shell substitutions during coverage checks, and reject JSON value closers as fallback removal-marker openers. Add documentation regressions and preserve shell reconstruction coverage across scripts and Markdown code blocks.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Keep list-relative indentation and HTML block terminators when identifying Markdown delimiters. Exercise incomplete-result gates with and without semantic analyzers. Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Recognize raw HTML tags at line endings and close fenced list code using its container indentation. Add positive and negative boundary regressions. Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Preserve candidate spans while caching quote ends and checking the artifact deadline throughout scanning. Add deterministic deadline and reconstruction compatibility regressions. Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Verify bounded work for shared quote suffixes and exercise real graph/MCP findings and completeness with semantic analysis enabled and disabled. Structured model transports are mocked for repeatable CI.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
fix: make JSON quote candidate scanning linear

lgtm
Build a fresh graph after mocking LLM availability so credential-free runs exercise all five failing analyzer stages. Preserve every telemetry, report and SARIF assertion.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Recognize structural quotes in bounded, validated JSON arrays, objects,
scalar strings and explicit JSON fences, including container prefixes and
frontmatter bodies. Preserve source offsets and linear, cancellable scanning.

Keep inline delimiter pairing within Markdown blocks so separate list items
and headings cannot hide unresolved runtime commands. Retain genuine command
findings, incomplete coverage and existing advisory risk policies.

Add tests-first reviewer reproductions, neighboring JSON and Markdown
controls, linear-work/deadline checks and inline Make reference regressions.
Exercise CLI and MCP with deterministic enabled/disabled LLM transports.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
SanHsien added a commit to SanHsien/SkillSpector that referenced this pull request Sep 11, 2026
…VIDIA#524

Raise reviewed_pr_through to 527 and reviewed_issue_through to 524 in
tools/upstream_baseline.json (commit axis unchanged at 69dcdfb). Every
item gets a verdict in docs/DECISIONS.md: NVIDIA#493/NVIDIA#507/NVIDIA#508/NVIDIA#511 verified
via git merge-base --is-ancestor as already included through the
2.11.1/2.11.2 sync (including NVIDIA#521, which merged only into the still-
open NVIDIA#516 stack, not main); the remaining 27 items stay "wait for
upstream merge", none adopted now.

Two items get dedicated comparison notes per docs/DIVERGENCE.md's
static_runner.py and scripts/compare_scan_accuracy.py rows: NVIDIA#522 uses a
different env var name and different default/semantics than this
fork's SKILLSPECTOR_MAX_STATIC_SECONDS, so merging it cannot simply
delete the divergence row and needs a downstream env var migration
first; NVIDIA#490 extends this fork's own upstream PR NVIDIA#486 with a Python
3.14/POSIX edge case the fork's Windows environment does not hit, so
NVIDIA#486 is left untouched pending upstream's own resolution. NVIDIA#501-NVIDIA#505 and
NVIDIA#518 are also flagged as near-verbatim matches to this fork's existing
Windows test divergence rows, worth revisiting for row deletion once
merged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: SanHsien <34234698+SanHsien@users.noreply.github.com>
mohgupta-ship-it and others added 5 commits September 12, 2026 01:19
Add tests-first reproductions for table row and cell boundaries, tab-column JSON containers, hidden runtime commands and bounded owned-string recovery. Preserve benign documentation, raw findings, strict CLI and MCP coverage gates in enabled and disabled LLM transport modes.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Add tests-first cases for Setext-like table rows and headers, real block interruptions and strict CLI/MCP gates in enabled and disabled LLM transport modes.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Retain complete and installable benign table exits in both semantic modes, and reproduce empty-list items incorrectly opening a paragraph before the next table header. Preserve red evidence before correcting that state transition.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Keep inline delimiters within proven table cells and respect contextual header, row and block boundaries. Interpret JSON container indentation in visual columns while preserving raw source offsets. Recover skipped runtime JSON strings through disjoint projected spans without overlapping suffix parses or changing raw finding evidence.

Validated against tests-first regressions, documentation and JSON overlap tests, GFM reference cases, bounded-work checks and enabled/disabled CLI/MCP transport modes.

Prepared by Codex on behalf of Mohit Gupta.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Reviewed current head bce6afadb2819d2235510e183eaa3a197ac02463 — APPROVE.

I reviewed the post-merge source diff, including bounded JSON ownership, Markdown block/table delimiter ownership, source-offset restoration, runtime checks, and the public completeness gates. The implementation remains conservative when context is incomplete, keeps command bodies analyzable, bounds validation and projection work, and has extensive success/failure/boundary coverage. I found no required changes.

The head changed during review from ca0ceb77e252c0ad5dc84eb435323edb80d28c57 to this merge commit, so the earlier assessment was discarded and the current diff was re-reviewed. Merge gate: GitHub reports mergeStateStatus=BLOCKED and no checks yet on the new head; required lint, unit, and DCO checks must pass before merging.

@mohgupta-ship-it

Copy link
Copy Markdown
Member Author

Thanks everyone for code reviews. As I have got two approvals on this change. I wll merge it post the pipeline passes.

@mohgupta-ship-it
mohgupta-ship-it merged commit 66e7983 into main Sep 12, 2026
5 checks passed
@mohgupta-ship-it
mohgupta-ship-it deleted the codex/fix-documentation-analysis-limits branch September 12, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation code spans and JSON placeholders trigger false incomplete-analysis findings

2 participants