Skip to content

feat: add GitHub issue queues with manual and opt-in automatic dispatch - #1254

Closed
filipenevola wants to merge 3 commits into
jayminwest:mainfrom
filipenevola:feat/github-project-tracker
Closed

filipenevola wants to merge 3 commits into
jayminwest:mainfrom
filipenevola:feat/github-project-tracker

Conversation

@filipenevola

Copy link
Copy Markdown
Contributor

Summary

Add a standalone GitHub Issues / Projects v2 tracker with an operator queue and optional automatic pickup. Teams can keep their existing GitHub backlog instead of maintaining a second Seeds queue. Manual dispatch is the default; connecting a tracker alone never starts paid work.

This is built directly on current upstream main (524b702a), without downstream fork changes or organization-specific configuration.

Changes

GitHub adapter (extensions/tracker-github)

  • One warren-tracker/v1 adapter supports repository-only issues and user- or organization-owned Projects v2, including cross-repository Projects.
  • Select repository and/or Project membership, the selected Project's own single-select status options, and optional labels (all or any). Status/label names are configuration, not hard-coded workflow assumptions.
  • Paginate issues, Project items/fields, and labels; fail closed on invalid options, incomplete responses, or pagination exhaustion. Exclude PRs, drafts, archived items, and inaccessible content.
  • Keep readiness separate from open/closed lifecycle. GitHub credentials stay in the extension; issue closing is separately opt-in and does not move Project status or labels.

Shared Warren integration

  • Activate the existing per-project tracker configuration through a resolver, retaining Seeds as the fallback. Repository-controlled endpoints and credential variable names require operator allowlists; redirects are refused.
  • Add an optional issue-listing capability without requiring new endpoints from existing adapters.
  • Add operator-only issue listing and a shared issue-dispatch endpoint, plus a Ready issues panel with explicit agent/cost confirmation. Issue title/body become the task; readiness and repository identity are rechecked before execution.
  • Manual and automatic queue dispatch share a persisted run receipt and the existing project lock, preventing duplicate queue launches across concurrent requests and restarts.

Opt-in automatic pickup

Requires an explicit repository-to-Warren-project map, credentials, agent, persistent SQLite journal, per-run cap, and daily budget. Concurrency is bounded. Unknown dispatch results remain journaled and block further automatic pickup rather than being blindly retried. Failed/cancelled runs are not automatically retried. No merge or deployment is requested.

The core changes are needed to make the adapter usable through the existing UI and runtime; the GitHub transport and unattended controller remain entirely out-of-process. The extension README documents setup, permissions, failure recovery, and protocol additions.

Test plan

  • bun run check:all — 12/12 gates passed, including lint, typecheck, tests/coverage, generated docs, dependency/layer checks, and bundle budget.
  • bun run build:ui.
  • Standalone extension lint/typecheck and 18 tests; the same 18 tests also pass inside the built Docker image.
  • Published tracker conformance CLI against a disposable fake-GitHub adapter — 9 case groups passed.
  • Real SQLite/filesystem and cross-process HTTP integration: manual dispatch followed by automatic pickup/controller restart retains one run; stale eligibility, wrong repository, and spectator access are refused.
  • Local browser verification through the fake runtime: review/confirm dispatch, navigate to the run, return to its queue receipt; no horizontal overflow at 375px.
  • Read-only smoke test against an existing GitHub organization Project, including pagination and Project-specific status filtering. No live issue mutations or paid runs; no private fixture data included.

Boundaries / limitations

  • Projects v2 only; GitHub Enterprise Server endpoints are configurable but not live-qualified.
  • Queue idempotency assumes Warren's supported single control-plane process, not multiple independent servers sharing its database.
  • GitHub eligibility can change after the final read; there is no cross-system transaction. Moving a status does not cancel a running agent.
  • Budget accounting is deliberately conservative and relies on reported/estimated usage, not a guaranteed billing ceiling.
  • Filter selection is configuration-driven, not a graphical Project/status setup wizard.

Submitted for maintainer review; no auto-merge requested.

@filipenevola

Copy link
Copy Markdown
Contributor Author

CI follow-up: fc432d2 regenerates the bundle budget using the same Bun 1.3.14 runtime as CI. The official measurement now matches CI exactly (JS gzip 200141 bytes); no application code or guard thresholds were changed by hand. All 12 local gates pass again. The first docker-build failure happened while downloading moby/buildkit:buildx-stable-1 (HTTP 502), before the project image build; the new commit triggers a fresh run.

@filipenevola

Copy link
Copy Markdown
Contributor Author

Added an equivalent generic startup-readiness fix after staging reproduced an ingress race: automatic pickup now verifies the read-only Warren queue path before reserving budget or attempting the dispatch mutation. Readiness failures safely retry; post-mutation uncertainty remains fail-closed and durable. Includes regression tests for unavailable/unsupported/malformed queues and exactly-once pickup after recovery. Full source gates passed (12/12), and the extension has 24 passing tests.

@luantaraschi luantaraschi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this rather than only reading it, since it is the biggest PR in the queue and Jaymin is out until the 14th. On a clean oven/bun:1.3.14 container at a36726e, bun run check:all passes 12 of 12. The contract change is additive and every new field is optional (supportsIssueListing, ready, url, repositoryUrl, resolveForProject), so the jira and ado extensions need no edits.

I also pointed the adapter at one of my repos with GITHUB_LABELS set. It returned the same 19 ready issues the GitHub API returns for that label, 24 issues in /issue-statuses with the pull requests left out (a PR number comes back 404), 404 for an issue outside the configured scope, 403 on close without GITHUB_ALLOW_CLOSE, and 401 without the bearer. With AUTO_DISPATCH_ENABLED unset, /dispatches reports automatic false and no scheduler starts, which is what the description promises.

Four things in the line comments, none of them blocking in my view, and one question: readiness never looks at assignees, so an issue a person has claimed stays in the queue and the automatic path can pick it up. That may be intended for a label-driven queue, but it is the kind of surprise an operator hits once. Whether a feature this size lands is Jaymin's call, so this is a comment review rather than an approval.

"raw": {
"js": 690019,
"css": 54756
"js": 694824,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every earlier raise in this file carries a $comment entry saying what grew and by how much, and the script header says the ratchet only goes down. This one raises all four numbers without a note. A one-line entry for the new project-issues page would keep the file readable the way the previous five raises did.

if (!repository) throw new TrackerFailure("invalid_scope", "Repository scope is required", 422);
const result = new Map<number, IssueRecord>();
for (let page = 1; page <= this.config.maxPages; page++) {
const url = `${this.config.apiUrl}/repos/${repository}/issues?state=all&sort=created&direction=asc&per_page=100&page=${page}`;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lists state=all on every poll, and /issue-statuses lists it again. A closed issue can never be ready, so the ready path could ask for state=open and leave state=all to /issue-statuses. On a repo with a few thousand issues the difference is tens of REST calls per tick at the default 60 second interval.

maxPages: integer(env, "GITHUB_MAX_PAGES", 100),
timeoutMs: integer(env, "GITHUB_TIMEOUT_MS", 15000, 120000),
port: integer(env, "TRACKER_PORT", 8080, 65535),
bearer: env.TRACKER_BEARER,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With TRACKER_BEARER unset and GITHUB_ALLOW_CLOSE=true, POST /issues/{id}/close is reachable by anyone who can reach the port, using the extension's GitHub token. The jira adapter has the same optional bearer, so this matches the house pattern, but refusing that pair at load time would close the one combination that mutates GitHub without auth.

for (const row of this.store.list()) {
if (row.state !== "running" || row.runId === null) continue;
// A failed poll isn't evidence that a run ended. Keep the reservation and stop this tick.
const run = await this.warren.getRun(row.runId);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a run gets deleted on the Warren side, getRun throws on the 404 and this tick stops before pickup. Nothing settles the row, so the next tick throws at the same place, and pickup never runs again until someone edits the sqlite file. Failing closed is the right default here, but a 404 could settle the row as uncertain, which already parks the controller and shows up on /dispatches with a message instead of a console line per tick.

@jayminwest

Copy link
Copy Markdown
Owner

Thanks for putting this together. GitHub Issues/Projects support is a useful integration for Warren, and I appreciate the care around explicit dispatch, duplicate prevention, and keeping GitHub credentials in the extension.

I’m closing this PR because its scope goes beyond the tracker integration: it adds an issue-queue UI, a new dispatch flow, automatic pickup with its own SQLite-backed DispatchStore and budget accounting, and shared core changes. I want to discuss those product and architecture decisions before bringing them upstream.

Could you open an issue so we can agree on the scope of a smaller GitHub tracker integration before reworking this? We can discuss the UI and automatic pickup separately.

Thanks again for contributing and using Warren!

@jayminwest jayminwest closed this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants