Skip to content

README accuracy: AVE coverage gap, telemetry, sync tooling, doc drift #79

Description

@chaksaray

Tracking issue for follow-ups surfaced during a README accuracy pass (see linked PR).

1. Detection rule coverage gap

This scanner has detection rules for AVE-2026-00001 through AVE-2026-00051 (51 records).
The published AVE corpus (https://raw.githubusercontent.com/aveproject/ave/main/dist/ave-records-latest.json)
currently has 80 records, AVE-2026-00001 through AVE-2026-00080. That leaves
AVE-2026-00052 through AVE-2026-00080 (29 records) with no rule in this repo.

Suggested batching (cheapest/most detectable first): config-literal detectable records
(TLS verify off, unpinned dependency, DOCKER_HOST redirect, bind-all-interfaces, endpoint
redirect, auto-run config, approval-gate bypass), then injection/traversal patterns, then
records needing Semgrep/YARA structural context, then multi-component records that likely
need new scanner/core/toxic_flows/flows.py chain definitions rather than single rules
(fragment reassembly, consensus poisoning, agent substitution have no single-line
signature).

2. Per-scan LLM/inference usage telemetry

The scanner does not currently report per-scan LLM token usage or cost. At least one
competitor (NVIDIA SkillSpector) exposes this via metadata.inference_usage. Worth
scoping for the LLM engine and any future multi-engine telemetry work.

3. scripts/sync_record_counts.py

The README badge and AVE record counts are currently hand-maintained against the live
AVE corpus. A script that regenerates them from
https://raw.githubusercontent.com/aveproject/ave/main/dist/ave-records-latest.json
would remove that drift surface. Does not exist yet.

4. Stale schema_version comment

scanner/ave_meta.py docstring says "Source: Piranha DB AVE records, schema_version
1.0.0, last synced 2026-06-21." The live AVE schema is 1.1.0. This is a comment, not an
assertion the code enforces, so it is not a functional bug, but it should be corrected
or removed so it stops citing a superseded version.

5. confidence_baseline possibly sourceable from the AVE corpus

The scanner states finding confidence "starts from the AVE-class baseline." Worth
checking whether scanner/ave_meta.py's hardcoded baseline table could instead be
sourced from the AVE corpus's own confidence_baseline field per record, removing a
drift surface. See scanner/core/fp_pipeline.py for where this is consumed.

Not tracked here (explicitly out of scope)

  • The PiranhaDB (api.piranha.bawbel.io) vs api.aveproject.org question. The scanner
    code hardcodes api.piranha.bawbel.io as the piranha_url value in multiple engine
    files; the README was intentionally left matching that actual runtime output rather
    than swapping to an unverified host. Whether PiranhaDB is retired/replaced or split
    from the neutral AVE reference API needs a maintainer decision before any host swap.
  • PIRANHA_API_URL / BAWBEL_PIRANHA_URL naming: PROJECT_STRUCTURE.md documents
    PIRANHA_API_URL in .env.example, but that var is not actually present in
    .env.example, and the code (scanner/suppression/justified.py) reads
    BAWBEL_PIRANHA_URL instead. Config/docs drift, not a README wording issue.
  • Sandbox image pin bawbel/sandbox:1.2.3 in the README: no code reference confirms
    whether this is an intentionally independent version or stale. Left unchanged,
    flagged VERIFY in the PR.
  • Cross-repo status contradictions between bawbel-gate and ARCHITECTURE.md (Phase 8
    of the originating task spec). Separate, higher-severity issue; not a README fix.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions