SoW: Autonomous GitHub agents (issue-crusher / PR reviewer / docs) on a shared codegraph engine

[sanil-23]

Summary

Build autonomous GitHub agents inside OpenHuman — a GitHub issue-crusher, a PR reviewer, and a docs (.md) writer — as user-enableable skills sitting on top of a shared codegraph retrieval engine plus a shared orchestration layer. Separately, build a merged-PR "source tree" memory using the existing memory_tree pipeline.

This SoW captures a design that was validated empirically (see Evidence), not just proposed.

---

Architecture

SKILLS (agent.toml defs · enable in Intelligence/Skills UI)
  • github-issue-crusher   • pr-reviewer   • docs-writer (.md)
  each = prompt + tool allowlist + autonomy tier + iteration cap + trigger
        |
SHARED TOOLS the skills call
  • codegraph_search   • Composio GitHub   • #2631 grep/glob/lsp/read/edit/shell
        |
codegraph MODULE  (src/openhuman/codegraph/)        ← the retrieval engine
  index:    checkout → BM25 + structural-aug vectors + repo-map (incremental)
  retrieve: seed(query) = BM25 ∪ dense → RRF → candidate block + repo-map
  reuses the embeddings provider (local Ollama); NOT memory_tree
        |
ORCHESTRATION (shared services)
  trigger (cron poll / GitHub webhook) → clone/checkout (~/OpenHuman/projects,
  #2631 trusted root) → branch + push + open PR → issue-claim state

---

Skills (each an agent.toml in the agent-definition registry)

Skill	Trigger	Uses	Output
github-issue-crusher	issue opened/labeled	codegraph_search (seed) → grep/lsp/edit/run_tests → Composio	branch + PR (Closes #N)
pr-reviewer	PR opened/synced	codegraph_search (touched code in context) → read/grep → Composio	inline review comments
docs-writer (.md)	manual / on-merge	codegraph_search (read code) → edit	README / docs / AGENTS.md PR

docs-writer doubles as the maintainer of a per-repo AGENTS.md "learnings" file the crusher reads for context.

---

codegraph engine

Two-stage retrieval: zero-build seed → capped agentic.

• Index (per checkout, zero LLM, incremental via blob-sha diff):
  1. BM25 over raw file text (lexical).
  2. Structural-aug vectors — per file, tree-sitter extracts signatures + docstrings + imports + called-symbol names → embed that (local model). Embed structure, never raw code.
  3. Repo-map — tree-sitter defs/refs → graph → PageRank → ranked signature outline.
• Retrieve (per issue, zero LLM): BM25.rank ∪ Dense(struct).rank → RRF fuse → top-k candidate block; render repo-map for orientation.
• Then capped agentic: the agent starts warm from the candidate block + repo-map, uses feat(agent): agentic coding runtime — gated OS capabilities (filesystem, shell, install) via deterministic permission tiers + chat approvals tinyhumansai/openhuman#2631 tools to pick the exact file and fix it, bounded by the agent.toml iteration cap.

Exposed as a codegraph_search tool + RPC. Reuses only the embeddings provider config (Ollama) — not memory_tree.

---

Evidence (A/B harness — feat/codebase-memory-ab, bench/codebase-memory-ab/)

SWE-bench_Lite retrieval eval, file-level recall vs the files each gold patch edits, n=18 across requests/flask/pytest/pylint/sphinx/xarray, bge-small held constant:

recall@10:  Dense(raw) 0.556  <  BM25 0.667  <  Dense(structural-aug) 0.778

• Raw-code embeddings lose to BM25 (confirmed across small and 1000–1900-file repos).
• Structural-augmentation dense (free, ast-based) beats BM25 at recall@5/10/20.
• BM25 and structural-aug are complementary → union of their top-10 = recall@10 ≈ 1.0 on every winnable instance.

→ The seed = BM25 ∪ structural-aug + repo-map is the empirically-backed design.

---

Decisions locked

• ✅ Seed (BM25 ∪ structural-aug + repo-map) → capped agentic.
• ❌ No raw-code vector index (loses to lexical).
• ❌ No memory_tree for code reading — its recall is over a lossy summary tree; admission drops chunks, source-id idempotency blocks re-index, and it pollutes personal memory.
• ❌ No LLM-gloss embeddings / hosted vector DB for v1 (build-token cost, unneeded).
• Reuse only the embeddings provider (local Ollama).

---

Merged-PR memory tree (separate workstream — where memory_tree does fit)

Build a source tree of merged PRs via the existing memory_tree pipeline (scope = "pr:<owner>/<repo>", ingest each merged PR as a document). PR history is prose + episodic + summarizable + recency-relevant — exactly memory_tree's home turf (unlike code). Enables semantic recall like "have we fixed something like this before?" to inform the issue-crusher.

---

Orchestration & open gaps

• Trigger glue (cron poll / GitHub webhook → spawn skill agent) — net-new.
• Clone/checkout into ~/OpenHuman/projects (feat(agent): agentic coding runtime — gated OS capabilities (filesystem, shell, install) via deterministic permission tiers + chat approvals tinyhumansai/openhuman#2631 trusted root) — easy half (anonymous for public repos via shell git clone; git_operations has no clone/push).
• git push credentials for opening the PR (and cloning private repos) — the real gap.
• Headless approval model — feat(agent): agentic coding runtime — gated OS capabilities (filesystem, shell, install) via deterministic permission tiers + chat approvals tinyhumansai/openhuman#2631's ApprovalGate is interactive-chat-only by design; background/cron turns aren't gated. Need a PR-review-as-approval model for autonomy.
• Issue-claim state — avoid double-work, one-branch-per-issue.
• .md skill scope — confirm docs-writer vs Q&A-over-docs.

---

Build order

1. codegraph module — index + codegraph_search tool (shared dependency, build first).
2. github-issue-crusher agent.toml — first consumer, proves the engine end-to-end.
3. Orchestration glue — trigger → checkout → PR + creds.
4. pr-reviewer + docs-writer skills — reuse the same engine + glue.
5. Merged-PR memory tree — parallel workstream on memory_tree.

---

References

• A/B harness + results: branch feat/codebase-memory-ab, bench/codebase-memory-ab/ (README.md, phase1_report.json, issue-crusher-flow.excalidraw).
• Execution substrate: PR feat(agent): agentic coding runtime — gated OS capabilities (filesystem, shell, install) via deterministic permission tiers + chat approvals tinyhumansai/openhuman#2631 (agentic-runtime — shell/lsp/grep tools + [autonomy] SecurityPolicy + ApprovalGate).