Parallelize File Parsing

[lexasub]

Better implementations - rewrite parsing to rust
Difficulty: ⭐⭐⭐ Hard
Files: ast_rag/cli.py, ast_rag/ast_parser.py
Description: Parse multiple files in parallel using concurrent.futures.
Steps:

1. Use ThreadPoolExecutor or ProcessPoolExecutor
2. Ensure thread-safety of ParserManager
3. Add progress bar for parallel processing

[r0h1tb]

I implemented this and then benchmarked it, and I don't think it's worth doing in Python — posting the numbers so the decision is based on data rather than my say-so.

Measured on 10 cores, CPython 3.12, tree-sitter 0.24, 800 real source files. Reproducible via benchmarks/parallel_parsing_benchmark.py in #48:

workload	8 workers
parse only (tree-sitter C)	0.99x
extract only (Python)	1.17x
end to end	0.96x
end to end, ProcessPoolExecutor	0.22x

Threads: py-tree-sitter doesn't release the GIL around Parser.parse, so the C parsing that should have been the entire win is fully serialised — 0.99x on the pure-parse microbenchmark makes that unambiguous. Extraction is Python (walking query matches, allocating ASTNode/ASTEdge) and is GIL-bound by construction; it gets 1.17x, presumably from whatever QueryCursor does release. End to end nets out slightly below 1.0 once lock and scheduling overhead is paid.

Processes: worse, and structurally so. Tree isn't picklable, so workers can't share the parse cache and each has to re-parse from source; then every task ships its extracted nodes and edges back over a pipe. Against ~2ms of per-file work, that overhead dominates — roughly 4.5x slower at 8 processes.

I had a working ThreadPoolExecutor implementation with a --jobs flag and deterministic input-order reassembly, and dropped it: it added a config surface and real complexity in exchange for a measurable slowdown.

What the exercise did surface is that the shared-state assumptions here are already violated. WorkspaceWatcher parses from threading.Timer callbacks — a new thread per debounce cycle — while sharing one ParserManager, so the single tree_sitter.Parser is already crossing threads today. #48 fixes that plus a racy LazyTree._ensure (documented as load-once, wasn't) and an unbounded _hashes leak in BoundedParseCache where the LRU evicted trees but never their hashes.

So I'd suggest leaving this issue open and treating the note already in the description — moving parsing to Rust, or otherwise getting the work out from under the GIL — as the actual path. Happy to take a run at that if it's useful.