Repository navigation
Description
Repository Summary
claude-agent-kit is a pure-markdown Claude Code plugin — a CLI plugin for Anthropic's Claude Code. There is no build step, no compilation, and no package manager. All content is plain .md or .json files. Python scripts (scripts/*.py) are stdlib-only.
What "coverage" means here: This repo uses structural validation coverage, not line/branch coverage. Coverage = what percentage of files, fields, and invariants are validated by tests.
Current Test Coverage
Existing test suites
| Suite | Command | Tests | Status |
|---|---|---|---|
| Structural validation | pytest tests/test_validate.py -v |
200 (160 pass, 40 skip) | ✅ All passing |
| Build sections unit tests | pytest tests/test_build_sections.py -v |
40 | ✅ All passing |
| Hook behavioral evals | python3 evals/run_evals.py |
44 | ✅ All passing |
| Prompt regression | python3 evals/prompt_regression.py |
N/A | ✅ Passing |
What's already validated in tests/test_validate.py
- agents/*.md (9 files): frontmatter delimiters, required fields (
name,description,model,tools,maxTurns), model validity, maxTurns format, non-empty body,category,costTier - skills/*/SKILL.md (12 files): required fields, agent ref if present,
context: fork+agentpair, model validity if present - hooks/hooks.json: file exists, valid JSON, structure schema, valid event names, command scripts exist on disk
- scripts/*.py (9 files): Python syntax (via
py_compile) - docs/agent-mapping.md: bidirectional cross-reference with
agents/directory - scripts/detect.py:
detect_persona_switch()function (5 tests)
What's validated in evals/ (but NOT in tests/)
detect_ulw()(13 parametrized tests inevals/hook_evals.py)hook_router.pybehavioral contracts (PreToolUse blocking, persona injection, stop continuation)sanitize.pyandstate.py(tested implicitly viahook_evals.py)
Identified Coverage Gaps
Gap 1: .claude-plugin/plugin.json — 0% coverage
The plugin manifest is never validated. It contains name, version, description fields. If malformed, the plugin won't load. Zero tests today.
Gap 2: Python script conventions — 0% coverage
CLAUDE.md mandates: all Python scripts (except import-only modules) must have:
#!/usr/bin/env python3shebang line- Executable bit (
chmod +x) if __name__ == "__main__":entry point
Currently only syntax is checked. No tests verify these conventions, which are critical for the hook system to work.
Scripts requiring these conventions: build_sections.py, detect.py, hook_router.py, prompt_version.py, state.py, telemetry.py
Import-only exceptions (no shebang/main needed): __init__.py, _debug.py, sanitize.py
Gap 3: Skill body non-empty — 0% coverage
Agent files are validated to have non-empty prompt bodies. Skill files are not. 12 skill files have no body content check.
Gap 4: detect_ulw() not in tests/ — only in evals/
detect_ulw() is covered in evals/hook_evals.py but absent from tests/test_validate.py. Having it in the main test suite improves discoverability and CI clarity.
Gap 5: sanitize.py, state.py, prompt_version.py unit tests — only implicit coverage
These stdlib scripts are tested indirectly via evals/hook_evals.py but have no dedicated unit tests in tests/. Their behavioral contracts (parse_hook_input, read_json, write_json, compute_hashes) are well-defined and easy to test in isolation.
Gap 6: test_build_sections.py not in CI
validate.yml runs pytest tests/test_validate.py -v but not pytest tests/test_build_sections.py -v. Those 40 unit tests for the dynamic prompt generator are not executed in CI.
CI Commands
These are the commands needed to fully run the test suite:
# Step 1: Setup
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.13'
# Step 2: Install dependencies
- name: Install pytest
run: pip install pytest
# Step 3: Structural validation (primary)
- name: Run validation tests
run: python3 -m pytest tests/test_validate.py -v
# Step 4: Unit tests for prompt generator (NOT currently in CI)
- name: Run build_sections unit tests
run: python3 -m pytest tests/test_build_sections.py -v
# Step 5: Hook behavioral evals
- name: Run evaluation suite
run: python3 evals/run_evals.py
# Step 6: Prompt regression detection (separate job in CI)
- name: Run prompt regression detection
run: python3 evals/prompt_regression.pySteps 3–5 are all currently passing locally. Step 4 works but is not in the validate.yml workflow.
Test Organization
- New tests for structural invariants → add to
tests/test_validate.py - New unit tests for individual Python scripts → new file
tests/test_scripts.pyortests/test_utils.py - All test files go in
tests/;pyproject.tomlsetstestpaths = ["tests"] - Both
pytestandunittest.TestCasestyles work - Import scripts via:
sys.path.insert(0, os.path.join(ROOT_DIR, "scripts"))
Improvement Plan
Priority 1 — Plugin manifest validation (new file coverage)
Add to tests/test_validate.py: validate .claude-plugin/plugin.json has required fields (name, version, description) and valid JSON.
Priority 2 — Python script conventions (structural invariant)
Add to tests/test_validate.py: validate that non-import-only scripts have shebang line, executable bit, and __main__ entry point. This catches regressions in the hook system's execution contract.
Priority 3 — Skill body non-empty (parity with agent checks)
Add to tests/test_validate.py: TestSkillFrontmatter.test_body_is_non_empty — parallel to the existing TestAgentFrontmatter.test_body_is_non_empty.
Priority 4 — detect_ulw() in tests/
Add TestULWDetection class to tests/test_validate.py with key positive/negative cases for detect_ulw().
Priority 5 — Script utility unit tests
Create tests/test_scripts.py with unit tests for sanitize.parse_hook_input, state.read_json/write_json, and prompt_version.compute_hashes.
Priority 6 — Add test_build_sections.py to CI
Update .github/workflows/validate.yml to include python3 -m pytest tests/test_build_sections.py -v.
How to Control this Workflow
Add comments to this discussion to provide feedback, adjust the plan, or redirect priorities.
You can also use these commands:
gh aw disable daily-test-improver --repo nicholls-inc/claude-agent-kit
gh aw enable daily-test-improver --repo nicholls-inc/claude-agent-kit
gh aw run daily-test-improver --repo nicholls-inc/claude-agent-kit --repeat (number-of-repeats)
gh aw logs daily-test-improver --repo nicholls-inc/claude-agent-kitWhat Happens Next
On the next run, this workflow will move to Phase 2: it will select one specific coverage gap from the plan above, implement new tests for it, verify they pass against the current codebase, and open a draft pull request with the improvements. It will then comment on this discussion with a summary of what was done.
Each subsequent run will pick the next gap and continue until coverage is substantially improved.
Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.
Tip: Discussion creation may fail if the specified category is not announcement-capable. Consider using the "Announcements" category or another announcement-capable category in your workflow configuration.
Generated by Daily Test Coverage Improver
To install this workflow, run
gh aw add githubnext/agentics/workflows/daily-test-improver.md@ee50a3b7d1d3eb4a8c409ac9409fd61c9a66b0f5. View source at https://github.com/githubnext/agentics/tree/ee50a3b7d1d3eb4a8c409ac9409fd61c9a66b0f5/workflows/daily-test-improver.md.
- expires on Mar 5, 2026, 2:55 PM UTC
Activity
hnipps commentedon Feb 26, 2026
🏷️ Issue Triaged
Hi @hnipps! I've categorized this issue as enhancement based on the following analysis:
Reasoning: This issue documents a research and improvement plan for expanding test coverage of an existing system — identifying structural gaps and proposing phased additions to
tests/test_validate.pyand CI. That fits the enhancement label: improving existing infrastructure without being a bug fix or entirely new feature.View Triage Details
Analysis
Recommended Next Steps
test_build_sections.pyto CI) is the lowest-friction improvement — a single-line change tovalidate.ymlReferences: Triage run §22451862224