Skip to content

Daily Test Coverage Improver Research and Plan #43

Open

Description

@hnipps

Repository Summary

claude-agent-kit is a pure-markdown Claude Code plugin — a CLI plugin for Anthropic's Claude Code. There is no build step, no compilation, and no package manager. All content is plain .md or .json files. Python scripts (scripts/*.py) are stdlib-only.

What "coverage" means here: This repo uses structural validation coverage, not line/branch coverage. Coverage = what percentage of files, fields, and invariants are validated by tests.


Current Test Coverage

Existing test suites

Suite Command Tests Status
Structural validation pytest tests/test_validate.py -v 200 (160 pass, 40 skip) ✅ All passing
Build sections unit tests pytest tests/test_build_sections.py -v 40 ✅ All passing
Hook behavioral evals python3 evals/run_evals.py 44 ✅ All passing
Prompt regression python3 evals/prompt_regression.py N/A ✅ Passing

What's already validated in tests/test_validate.py

  • agents/*.md (9 files): frontmatter delimiters, required fields (name, description, model, tools, maxTurns), model validity, maxTurns format, non-empty body, category, costTier
  • skills/*/SKILL.md (12 files): required fields, agent ref if present, context: fork + agent pair, model validity if present
  • hooks/hooks.json: file exists, valid JSON, structure schema, valid event names, command scripts exist on disk
  • scripts/*.py (9 files): Python syntax (via py_compile)
  • docs/agent-mapping.md: bidirectional cross-reference with agents/ directory
  • scripts/detect.py: detect_persona_switch() function (5 tests)

What's validated in evals/ (but NOT in tests/)

  • detect_ulw() (13 parametrized tests in evals/hook_evals.py)
  • hook_router.py behavioral contracts (PreToolUse blocking, persona injection, stop continuation)
  • sanitize.py and state.py (tested implicitly via hook_evals.py)

Identified Coverage Gaps

Gap 1: .claude-plugin/plugin.json — 0% coverage

The plugin manifest is never validated. It contains name, version, description fields. If malformed, the plugin won't load. Zero tests today.

Gap 2: Python script conventions — 0% coverage

CLAUDE.md mandates: all Python scripts (except import-only modules) must have:

  • #!/usr/bin/env python3 shebang line
  • Executable bit (chmod +x)
  • if __name__ == "__main__": entry point

Currently only syntax is checked. No tests verify these conventions, which are critical for the hook system to work.

Scripts requiring these conventions: build_sections.py, detect.py, hook_router.py, prompt_version.py, state.py, telemetry.py

Import-only exceptions (no shebang/main needed): __init__.py, _debug.py, sanitize.py

Gap 3: Skill body non-empty — 0% coverage

Agent files are validated to have non-empty prompt bodies. Skill files are not. 12 skill files have no body content check.

Gap 4: detect_ulw() not in tests/ — only in evals/

detect_ulw() is covered in evals/hook_evals.py but absent from tests/test_validate.py. Having it in the main test suite improves discoverability and CI clarity.

Gap 5: sanitize.py, state.py, prompt_version.py unit tests — only implicit coverage

These stdlib scripts are tested indirectly via evals/hook_evals.py but have no dedicated unit tests in tests/. Their behavioral contracts (parse_hook_input, read_json, write_json, compute_hashes) are well-defined and easy to test in isolation.

Gap 6: test_build_sections.py not in CI

validate.yml runs pytest tests/test_validate.py -v but not pytest tests/test_build_sections.py -v. Those 40 unit tests for the dynamic prompt generator are not executed in CI.


CI Commands

These are the commands needed to fully run the test suite:

# Step 1: Setup
- name: Set up Python
  uses: actions/setup-python@v5
  with:
    python-version: '3.13'

# Step 2: Install dependencies
- name: Install pytest
  run: pip install pytest

# Step 3: Structural validation (primary)
- name: Run validation tests
  run: python3 -m pytest tests/test_validate.py -v

# Step 4: Unit tests for prompt generator (NOT currently in CI)
- name: Run build_sections unit tests
  run: python3 -m pytest tests/test_build_sections.py -v

# Step 5: Hook behavioral evals
- name: Run evaluation suite
  run: python3 evals/run_evals.py

# Step 6: Prompt regression detection (separate job in CI)
- name: Run prompt regression detection
  run: python3 evals/prompt_regression.py

Steps 3–5 are all currently passing locally. Step 4 works but is not in the validate.yml workflow.


Test Organization

  • New tests for structural invariants → add to tests/test_validate.py
  • New unit tests for individual Python scripts → new file tests/test_scripts.py or tests/test_utils.py
  • All test files go in tests/; pyproject.toml sets testpaths = ["tests"]
  • Both pytest and unittest.TestCase styles work
  • Import scripts via: sys.path.insert(0, os.path.join(ROOT_DIR, "scripts"))

Improvement Plan

Priority 1 — Plugin manifest validation (new file coverage)

Add to tests/test_validate.py: validate .claude-plugin/plugin.json has required fields (name, version, description) and valid JSON.

Priority 2 — Python script conventions (structural invariant)

Add to tests/test_validate.py: validate that non-import-only scripts have shebang line, executable bit, and __main__ entry point. This catches regressions in the hook system's execution contract.

Priority 3 — Skill body non-empty (parity with agent checks)

Add to tests/test_validate.py: TestSkillFrontmatter.test_body_is_non_empty — parallel to the existing TestAgentFrontmatter.test_body_is_non_empty.

Priority 4 — detect_ulw() in tests/

Add TestULWDetection class to tests/test_validate.py with key positive/negative cases for detect_ulw().

Priority 5 — Script utility unit tests

Create tests/test_scripts.py with unit tests for sanitize.parse_hook_input, state.read_json/write_json, and prompt_version.compute_hashes.

Priority 6 — Add test_build_sections.py to CI

Update .github/workflows/validate.yml to include python3 -m pytest tests/test_build_sections.py -v.


How to Control this Workflow

Add comments to this discussion to provide feedback, adjust the plan, or redirect priorities.

You can also use these commands:

gh aw disable daily-test-improver --repo nicholls-inc/claude-agent-kit
gh aw enable daily-test-improver --repo nicholls-inc/claude-agent-kit
gh aw run daily-test-improver --repo nicholls-inc/claude-agent-kit --repeat (number-of-repeats)
gh aw logs daily-test-improver --repo nicholls-inc/claude-agent-kit

What Happens Next

On the next run, this workflow will move to Phase 2: it will select one specific coverage gap from the plan above, implement new tests for it, verify they pass against the current codebase, and open a draft pull request with the improvements. It will then comment on this discussion with a summary of what was done.

Each subsequent run will pick the next gap and continue until coverage is substantially improved.


Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

Tip: Discussion creation may fail if the specified category is not announcement-capable. Consider using the "Announcements" category or another announcement-capable category in your workflow configuration.

Generated by Daily Test Coverage Improver

To install this workflow, run gh aw add githubnext/agentics/workflows/daily-test-improver.md@ee50a3b7d1d3eb4a8c409ac9409fd61c9a66b0f5. View source at https://github.com/githubnext/agentics/tree/ee50a3b7d1d3eb4a8c409ac9409fd61c9a66b0f5/workflows/daily-test-improver.md.

  • expires on Mar 5, 2026, 2:55 PM UTC

Activity

  1. hnipps commented on Feb 26, 2026

    @hnipps
    MemberAuthor

    🏷️ Issue Triaged

    Hi @hnipps! I've categorized this issue as enhancement based on the following analysis:

    Reasoning: This issue documents a research and improvement plan for expanding test coverage of an existing system — identifying structural gaps and proposing phased additions to tests/test_validate.py and CI. That fits the enhancement label: improving existing infrastructure without being a bug fix or entirely new feature.

    View Triage Details

    Analysis

    • Keywords detected: "test coverage", "improvement plan", "gaps identified", "CI", "validate.yml", "priority", "phases"
    • Issue type indicators: Structured plan with numbered phases, gap analysis of current test state, incremental improvements to existing test suite and workflow — all indicators of an enhancement rather than a bug or new feature
    • Confidence: High

    Recommended Next Steps

    • Review the 6 identified coverage gaps and confirm the priority ordering matches your current focus
    • Phase 6 (adding test_build_sections.py to CI) is the lowest-friction improvement — a single-line change to validate.yml
    • Consider linking this to issue Daily Test Coverage Improver Research and Plan #38, which covers a similar scope from an earlier run

    References: Triage run §22451862224

    Generated by Issue Triage Agent

    To install this workflow, run gh aw add github/gh-aw/.github/workflows/issue-triage-agent.md@852cb06ad52958b402ed982b69957ffc57ca0619. View source at https://github.com/github/gh-aw/tree/852cb06ad52958b402ed982b69957ffc57ca0619/.github/workflows/issue-triage-agent.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions