Skip to content

Bug hunting is a hand procedure: no baseline record, no reusable fuzz, no finding record #54

Description

@wolfgang-aura

Why

init-run --defect-report accepts the output of a bug hunt, but nothing in the
harness produces one. On 2026-09-06 the target openai/openai-agents-python had
17 of its 22 unassigned issues already claimed by an open pull request
(#53), so taking an
existing issue was not an option and the hunt was done by hand in the session.
That makes it a procedure that exists only in a transcript.

The procedure that worked, and should be a command

  1. Clone at the current default-branch head and record the base commit.
  2. Build the environment the target's own CI builds. uv sync --all-extras
    failed here on Python 3.14 because aiohttp has no cp314 wheel and wants
    MSVC; pinning to 3.13 fixed it. The harness should try the CI interpreter,
    then step down until the environment installs, and record which one it used
    and why it differs from CI.
  3. Run the full suite and record the baseline. 8398 passed, 17 failed, and all
    17 failures were one already-reported Windows symlink-privilege issue. A
    baseline failure that maps to an existing issue is not a finding, and the
    harness should say so rather than leaving an agent to notice.
  4. Pick a pure module reachable from the public API and fuzz it differentially
    against a model of what it should do. agents.apply_diff survived 4000
    generated cases across LF and CRLF, blank lines, trailing whitespace, and
    lines beginning with the format's own markers.
  5. Read the layer above the pure core, where the I/O and the ordering live.
    The defect found was there: WorkspaceEditor.apply_operation writes the moved
    file and then removes the source, guarded only by moved_destination != destination, so a rename differing only in case deletes the file it just
    wrote.

What the harness should own

  • A hunt command that takes a screened target, does steps 1 to 3, and writes a
    baseline record: base commit, interpreter, pass and fail counts, and each
    failure matched against the target's open issues.
  • The fuzz harness as a reusable piece, given a module and a model function.
    The generator's first version emitted overlapping hunk context and produced
    260 false findings out of 800; a shared one that is known-clean on a passing
    target is worth more than a per-session rewrite.
  • A finding record that carries the reproducer, the conditions it needs, and
    which of those conditions this host can actually satisfy. This defect needs a
    case-insensitive filesystem and a case-sensitive path comparison; this Windows
    box supplies the first and not the second, so the reproducer models the second.
    That distinction has to survive into the upstream report, and today nothing
    carries it.

Next action

Specify the baseline record first, since reproduce and orchestrate both
already want a known-good starting state and neither has one for a target whose
suite has never been run here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions