Maintainer-only — do not assign to GSoC / first-time contributors
This work is not suitable for non-maintainers. It touches local developer UX, CI, deploy glue, and every path that still assumes a file-backed SQLite cache. Schedule it after the GSoC Librarian / Module C work is merged, or fold it into that merge if maintainers are already in the middle of the Postgres/pgvector cutover — do not open it as a parallel community-facing “good first issue.”
Why
OpenCRE now has a vec-only embedding store (embeddings.embedding_vec). Real similarity uses Postgres pgvector (<=>). SQLite remains a second dialect in Makefiles, default DEV_DATABASE_URL fallbacks, standards_cache.sqlite, CI, import/export, and Module C’s in_memory backend.
That dual path is increasingly expensive and error-prone (legacy CSV→vec rewrites, “fail on pgvector-on-SQLite” guards, sync scripts, different Alembic behavior). The long-term shape should be: Postgres (+ pgvector) everywhere we run code that matters.
Goal
Drop SQLite as a supported application database for OpenCRE. Migrate Makefiles, scripts, CI, docs, and defaults so local/dev/test/prod all assume Postgres (pgvector image locally).
In scope (checklist)
Out of scope
- Re-embedding the corpus on Heroku (still forbidden — OOM risk)
- HNSW index work (can stay a separate follow-up on Essential plan capacity)
- Community drive-by PRs
Acceptance
- Fresh clone + documented Makefile path brings up Postgres and runs tests without creating/using SQLite
- No remaining “supported” SQLite cache workflow in maintainer docs
- CI green on Postgres
- Clear
SystemExit / docs if someone still points CRE_CACHE_FILE at a .sqlite path
Related
Maintainer-only — do not assign to GSoC / first-time contributors
This work is not suitable for non-maintainers. It touches local developer UX, CI, deploy glue, and every path that still assumes a file-backed SQLite cache. Schedule it after the GSoC Librarian / Module C work is merged, or fold it into that merge if maintainers are already in the middle of the Postgres/pgvector cutover — do not open it as a parallel community-facing “good first issue.”
Why
OpenCRE now has a vec-only embedding store (
embeddings.embedding_vec). Real similarity uses Postgres pgvector (<=>). SQLite remains a second dialect in Makefiles, defaultDEV_DATABASE_URLfallbacks,standards_cache.sqlite, CI, import/export, and Module C’sin_memorybackend.That dual path is increasingly expensive and error-prone (legacy CSV→vec rewrites, “fail on pgvector-on-SQLite” guards, sync scripts, different Alembic behavior). The long-term shape should be: Postgres (+ pgvector) everywhere we run code that matters.
Goal
Drop SQLite as a supported application database for OpenCRE. Migrate Makefiles, scripts, CI, docs, and defaults so local/dev/test/prod all assume Postgres (pgvector image locally).
In scope (checklist)
AGENTS.md/ scripts: nostandards_cache.sqliteas default cache;make docker-postgres(or equivalent) is the documented local DBDEV_DATABASE_URL/PROD_DATABASE_URL/CMDConfigpaths (or make them hard-fail with a clear message)CRE_LIBRARIAN_RETRIEVER_BACKEND=in_memoryonce CI is on Postgres/pgvectorrewrite_sqlite_embeddings_to_vec.py-style escape hatches once caches are gonescripts/db/*remain the only production DB storyOut of scope
Acceptance
SystemExit/ docs if someone still pointsCRE_CACHE_FILEat a.sqlitepathRelated