Skip to content

[Bug] Evidence Pack with update_raw=True Blocks Thread Pool for Up to 30 Minutes — DoS #323

Description

@ionfwsrijan

Description

The evidence_pack endpoint is a sync handler that, when called with update_raw=True, runs the full scanner suite (semgrep, osv-scanner, gitleaks) synchronously without threadpool delegation or timeout. Each scanner has a 600-second timeout, meaning a single request can block a thread pool worker for up to 1800 seconds (30 minutes). With ~40 concurrent requests, the entire thread pool is exhausted and no requests can be served.

Actual Behavior

Sync handler blocks worker thread:
backend/app/main.py (lines 1056-1107):

def evidence_pack(
    job_id: str = Form(...),
    update_raw: bool = Form(False),
):
    # ...
    if update_raw:
        verify_dir = job_dir / "raw_verify"
        if verify_dir.exists():
            _scan_repo_dir(repo_dir, job_dir=job_dir, raw_dir_name="raw_verify")  # ← BLOCKS
        else:
            _scan_repo_dir(repo_dir, job_dir=job_dir, raw_dir_name="raw")  # ← BLOCKS

_scan_repo_dir spawns 3 long-running processes:
backend/app/main.py (lines 256-319) → calls:

  • semgrep.py line 20: run_cmd(cmd, cwd=repo_dir, timeout_s=600) — up to 10 minutes
  • osv.py line 40: run_cmd(cmd, cwd=repo_dir, timeout_s=600) — up to 10 minutes
  • gitleaks.py line 24: run_cmd(cmd, cwd=repo_dir, timeout_s=600) — up to 10 minutes

No auth on this endpoint:
The evidence_pack endpoint has no Depends(verify_api_key) — any anonymous user can call it.

No concurrency limit or rate limiting:
No per-request or global rate limiting exists on this endpoint.

Attack Vector

for i in $(seq 1 50); do
  curl -X POST http://target:8000/evidence-pack -d "job_id=X&update_raw=true" &
done

~40 concurrent requests exhaust the entire thread pool. All HTTP requests stop being served, including health checks. The application appears completely down.

Impact

  • Complete server unavailability from ~40 concurrent unauthenticated requests
  • Cascading failure: SSE streams stall, health checks fail, other users cannot use the app
  • No visibility: no SSE stream, no progress indicator, no way for an operator to know it's being abused
  • Worker must be killed and restarted

Expected Background

The /verify endpoint (issue #276) has similar blocking behavior but is tracked separately. The evidence pack endpoint is worse because it is unauthenticated and has no visibility.

Expected Behavior

Long-running scanner operations must not block the thread pool. The endpoint must have rate limiting and authentication.

Proposed Fix

  1. main.py: Make the evidence_pack handler async and run _scan_repo_dir in a background task (BackgroundTasks) or thread pool executor with a timeout
  2. main.py: Add Depends(verify_api_key) to the evidence_pack endpoint
  3. main.py: Add rate limiting (e.g., 1 concurrent evidence pack per user)
  4. main.py: Add a maximum repository size check before spawning scanners
  5. api.ts: Update the frontend to send auth headers with this request

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions