Skip to content

[Draft] Storage: embedded SQLite instead of Neo4j+Qdrant — follow-up to #32 #57

Description

@lexasub

Status: stream of thoughts, might be nonsense. Draft for discussion, not a commitment.

Follow-up to #32. I asked "which graph DB to replace Neo4j" — but after studying the market (codegraph 57k⭐, CGC 4k⭐, Graphify 100k⭐, Bitloops, CCE) the conclusion is different: a graph DB isn't needed at all.

Why

raged's actual load profile (measured from the code):

  • find_callers/callees: [*1..5], LIMIT 200
  • expand_neighbourhood: [*1..4], LIMIT 500
  • 45 methods in Neo4jRepository, ~40 of them get/put/delete by id
  • Zero analytics: no pagerank, no shortestPath — nothing

A codebase graph = 10⁴–10⁵ nodes. Recursive CTE in SQLite / in-memory BFS over 100k edges — milliseconds. Neo4j is for 10⁶+ and multi-step analytics. Overkill ×100 here.

The market confirms this:

  • codegraph: tree-sitter WASM → SQLite+FTS5, one file = one project, zero config
  • CGC: embedded KuzuDB/FalkorDB/LadybugDB (also not a server), Neo4j only as an option
  • Bitloops: SQLite + sqlite-vec + DuckDB, with a typed graph (17 node kinds, 18 edge kinds) + assertion layer — all without a graph DB

Proposal

  • SQLite (or DuckDB) as the core; valid_from/valid_to already exist — temporal out of the box
  • Qdrant → sqlite-vec (10⁵ vectors is plenty; numpy cosine also fine)
  • schema_manager (935 lines of Neo4j DDL ceremony) → ~40 lines CREATE TABLE + FTS5
  • repositories/ is already an adapter seam — migration is cheap

Tension with #35 (cross-repo support) — open question

#35 has a full design for cross-project CALLS edges: CrossProjectResolver (in-memory symbol index → project_id), CROSS_PROJECT_CALL edge with resolution_method: import/type/name, confidence import=0.9 / type=0.6, dependency config with version constraints. Notably, #56 (merged global symbol table) is essentially the intra-project version of the same mechanism — a hash table name→id, not a graph feature.

My "one file = one project, no joins" contradicts that. But both can be true if we separate two things:

  • Cross-project resolution (resolve import com.example.Service to the right project) — a symbol lookup across separate files, cheap in SQLite: one global name→(project, id) index table. No graph DB needed.
  • Multi-project analytics (query across projects as one graph) — that's what needs a server DB, and agents don't actually need it; they work on one project at a time.

So the SQLite pivot doesn't kill #35 — it changes its shape: CrossProjectResolver becomes a table + lookup, not a Neo4j query. Worth deciding explicitly: keep cross-project edges at all, or resolve-on-demand?

Open question: is there anything we lose from a graph DB that I'm not seeing? E.g. the dream of impact analytics on a 10⁶-node megamonolith — there a graph DB would come back. But raged is agent-facing; an agent needs ms-latency answers about a single project.


Relates to: #35 (cross-repo support), #24 (multi-index for third-party libs), #20 (advanced semantic search), #31 (multi-project configs)

Metadata

Metadata

Assignees

No one assigned

    Labels

    draftDraft / stream of thoughts — not a commitmentresearch

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions