feat(knowledge): read every format, show what the index holds, and let an agent search it by KevinHuSh · Pull Request #489 · EverMind-AI/Raven

KevinHuSh · 2026-09-18T05:01:39Z

Summary

Everything a knowledge base could not read, and the pieces it cut them into
saying what they are.

Reading a PDF. A PDF records glyphs at coordinates; whether they are a
heading, a table cell or a running footer is inferred from how they look. There
are now two readers. The geometric one infers structure from type size and
position and needs nothing but the file. The other asks deepdoc's models, ported
into raven/knowledge/parser/deepdoc: a layout model that says what each region
is, a detector and recognizer that read a page carrying no text at all, and a
table-structure model that rebuilds a grid from a picture of one.
Detecting boxes around text the file already spells
out exactly costs an inference pass per page to learn less, and recognising it
costs accuracy.

The layout graph answers with a shape its own postprocessing
raises on, and its classes are DocLayout-YOLO's order rather than the list
Both were established by running the model, and the label order is pinned by a test.

Tables go in as tables. <table> is the only shape that keeps what a grid
states -- which column a cell stood under, which cells are merged, which row is
the header -- and it is the shape a model reads a table in, which is what these
chunks are for. PDF, Word and PowerPoint all state their merges exactly, and
they now survive as colspan and rowspan. The flat form this replaces had to
repeat a header standing over two columns across both of them, because there was
nowhere to record a span; it had no callers left afterwards and is gone. Two
deviations from upstream, both about what the string is for: every cell's text
is escaped, where upstream interpolates it raw and a cell holding a < produces
markup that is no longer a table, and the span attributes are written only where
there is a span, where upstream emits <td > for every ordinary cell.

A picture of where each chunk came from. A chunk is a few hundred words with
a page number on it, which is enough to cite and not enough to answer whether
the parser read the page correctly. So every PDF chunk that knows where it sits
carries one image: the regions it covers, cropped out of the rendered pages and
stacked. here they are already structured in the parser's element spans, so
the markers would only put text nobody wrote into the thing being embedded.
The panel draws a chunk's tables as tables, parsed and
rebuilt as elements rather than injected: a person can rewrite any piece from
that same panel, so what comes back from the store is user input however it got
there.

The page that reads them. Upstream retired the knowledge page while this
branch was out; it is rebuilt against the new architecture, and the half that
never existed is the one a reader needs -- what a document actually became once
it was cut and embedded. Documents sit in one-level folders and arrive by four
routes: a file, a folder, a URL, a written note. Adding and indexing are two
calls on the contract and nothing drains that queue on its own, so a page that
stopped after the first left every upload saying Queued for good.

Opening a file takes the whole page: the document on the left, the pieces the
index holds of it on the right. The left half is the file drawn as its own
pages rather than framed by the browser's PDF viewer, which reads where to open
once and answers to nothing afterwards -- and which nothing can be drawn on.
Pages are rendered once and kept, and the region a piece was cut from is washed
over the page it came from, placed in fractions of the page so the picture's
own scale never enters the arithmetic. Clicking a piece takes the file there;
double-clicking opens it for rewriting.

The pieces can be searched, filtered by state, turned on and off, deleted,
rewritten and added to, one at a time or in a batch. Every one of those was
already on the contract with no reader for it. A base's settings are back and
fuller: how many pieces a search returns, how documents are cut, and how much
prose comes with a table or a figure. The embedding model is a picker while the
base holds no pieces and a stated fact once it does -- moving it then is the
collection made again at the new width and every document re-cut, which is not
a thing a panel does by being saved.

An agent can read them. The retrieval engine was there and nothing could
reach it: no agent had a tool that opened a library. A reader now points a
conversation at bases from the composer, the way they pin a working directory,
and the selection rides in the same place -- the session's metadata, bound
around the turn.

The tool takes a query and never a base. The scope is the reader's choice, so a
model cannot reach material nobody offered it, and it is withheld where nothing
was picked rather than offered and refused. A hit says which document and page
it came from, because a hit a reader cannot place is a hit they cannot check --
and the passages are pressable in the transcript, walking into the base, the
document, the folder it is filed under and then the piece. A citation a reader
cannot follow is a claim they have to take on trust, and the whole point of
searching their own material is that they can go and look.

The word segmenter and the resources. raven.core.Tokenizer is Infinity's
rag_tokenizer, ported: a forward and backward maximum match over a dictionary
trie, with stemming and lemmatisation on the latin path. XGBoost is not carried
over. None of the data lives in git -- the dictionary, the NLTK corpora and the
ONNX graphs run to about a hundred megabytes, so make fetch-resources on an
install and an optional Docker layer behind RAVEN_RESOURCES both point at one
script. A checkout that could not reach the network still works, and every
feature that reads this data says so when asked.

Type

Verification

Run on macOS against Python 3.12, with the deepdoc weights fetched (the model
tests skip without them):

uv run pytest                        27450 passed, 111 skipped, 3 failed
npm run test --prefix ui-web          3276 passed
npm run test --prefix ui-tui          2106 passed
npm run type-check --prefix ui-web    clean (tsc)
npm run lint --prefix ui-web          clean (4 pre-existing hook warnings)
make lint-python                      clean (ruff check + format)
make lint-types                       clean (ty)
make lint-imports                     10 contracts kept, 0 broken
make check-source-language            clean
make check-large-files                clean
make build-ui                         built, both boot snapshots match
node ui-web/scripts/check-css.mjs     clean
node ui-web/scripts/check-class-namespace.mjs  clean

The three Python failures are not this branch's and were each confirmed on the
base alone: one asks a host without LibreOffice to refuse a render and this host
has LibreOffice installed, and two simulation cases fail on an unmodified
checkout of the base commit.

The RPC contract was regenerated for both consumers and both match it
(npm run gen --prefix ui-web, npm run gen:rpc --prefix ui-tui, then
gen:check). The page's boot snapshots and region goldens were refreshed where
the composer bar gained a button, and each records that button and nothing else.

Relevant tests pass locally
Relevant lint / type checks pass locally
User-facing docs or screenshots are updated when needed

Risk

A PDF's tables, and a Word or PowerPoint file's, are now indexed as HTML rather
than as header-prefixed lines. Chunk text changes shape for those documents, so
a base indexed before this and searched after it holds both forms until its
documents are reindexed; retrieval keeps working either way, because both forms
are text. The same applies to how much prose a table or a figure carries with
it: the setting now reaches as far as it says, where before it stopped at the
nearest paragraph, so a chunk cut before this keeps the narrower context until
its document is rebuilt. Rolling back is reverting the branch and reindexing.

The deepdoc models are optional. An install that never ran make fetch-resources reads PDFs geometrically, exactly as before, and says so when
the models are asked for by name. Nothing here reaches the network at request
time.

Chunk crops and rendered pages are written at index time and on first view, both
under the state directory rather than in the repository, and both are deleted
with their document. A page is immutable for the life of its document id, so
the route serving them caches; the document route does not. All three routes
take ids and no path, and the two image routes answer under the strictest
sandbox in the tree.

What an agent may search is the reader's choice and nothing else. The tool takes
no base argument, so a model cannot name one, and it is absent from the schema
entirely for a conversation that named none -- a lane nobody pointed at a base
is unchanged by this branch. The bases a session carries are stored beside its
working directory and travel no further.

Security impact considered
Backward compatibility considered
Rollback path is clear for risky changes

Related Issues

N/A

…s own package ParserBase and TextParser move out of the single `_parser.py` module into `raven/knowledge/parser/`, one module per format family, the way RAGFlow lays out `deepdoc.parser`. The contract is the package `__init__`; a concrete parser is imported from its own module, which keeps the dependency one-directional (re-exporting them from the package put three modules in an import cycle and the cycle budget test caught it). Every section now carries a reading order, its place in the source file. Nothing downstream can recover that: search returns hits by score, and a section's text does not say where it came from. Formats laid out on a page add a page number, a box and a layout type, and `metadata["elements"]` records each paragraph, table and figure against the character range it occupies in the section, so a hit in the middle of a long section still resolves to a place in the file. DocxParser reads the OOXML package with the standard library. Walking `word/document.xml` is the only way to keep the order -- a document object model hands back paragraphs and tables as separate collections, and which table fell between which paragraphs cannot be rebuilt from those -- and it means Word support adds no dependency, the way the HTML parser reads markup without one. Sections are cut at headings, so a Word document and its exported Markdown index alike. Tables are composed as RAGFlow composes them, header rows lifted onto every body row, because a table is the one structure a chunk boundary renders meaningless. Boxes are horizontal only for flowing text. Word stores page geometry and indentation, so the band a paragraph occupies is exact; where it lands down the page is decided by a layout engine this parser does not run, and a fabricated top would be indistinguishable from a measured one to anything drawing a highlight. Anchored drawings state their own offsets and get a complete box. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

…s it accepts Two faults on the same path, both of which make a base answer nothing. A base was searched with whatever embedding model is configured today. Its vectors answer to the model it was built with, so a query embedded with another one lands somewhere unrelated in the same space -- and the guard against that refused the base outright, which took the whole search down as soon as the configured pin moved. A base is now searched and indexed with its own model: the requested bases are grouped by model, one embedding call per distinct one, and each base is searched with the vector its own space uses. Bases sharing a model share a call, so the ordinary case still costs one round trip. Indexing follows the same rule, because a document added to an older base has to land in the space that base's collection already holds. A model id does not name a credential, so a base now records the provider that served it. For the bases written before that field existed, the endpoint the memory backend configured is tried first, and only when it still names that base's model -- a base older than raven's own pin came through that file, and it is the only record of where its vectors came from. The provider is also a base setting, which is the way back for a base whose endpoint has moved on, short of a rebuild. The model and the width stay fixed: those are what the collection was built to. One base that cannot be reached is skipped and named in the outcome rather than failing the search for the others, and when every base drops out the reason is raised -- an empty result and an empty result for cause are not the same answer, and only the second says what to fix. Separately, every chunk of a document went up in one request. Endpoints cap that (DashScope at 20) and reject the whole request rather than the excess, so a document past the cap indexed nothing. Inputs are now sent in batches: the size the config states (the EverOS file already carried one and nobody read it), a default of 16 otherwise, and halved from there whenever an endpoint refuses a batch by name -- remembered per endpoint and model, so one document pays for the lesson. A refusal that is not about the batch is not retried, because retrying a rejected input in smaller batches only rejects it again. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

Clicking a file used to fill the panel beside the base rail. It now takes the whole page: the widest thing on screen should be the thing being read, and picking another base is not a move anyone makes mid-document. Beside the file are the pieces the index actually holds of it, which until now had no reader at all -- "View Chunks" was a disabled stub, and nothing behind it existed at any layer. The two halves answer one question together: a chunk list alone says nothing about where a cut landed, and a document alone says nothing about what a search can find in it. The chunks are read back from the store rather than re-cut from the file. What a reader wants to see is what the search matches against, and a fresh parse shows what a rebuild would produce instead -- a different thing the moment a chunking setting has moved. They are listed in reading order, which is the chunker's own numbering as it walks the sections a parser produced. Ordered twice on purpose: a scan of the store has no order to promise, and the panel that claims reading order establishes it rather than trusting the wire. Each piece carries what the parser found and nothing it did not -- which page, what kind of region, which heading it sits under. A text file shows none of those, because absent means the format does not know, never that the value is zero. This is the first reader for the metadata the parsers record. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

Two pieces of one thing: the chunks a file was cut into can now be read and changed, and the cutting itself follows the document instead of a byte count. A chunk has an identity. Derived from its own text, so the same passage keeps its name across a rebuild and a reader who turned a paragraph off last week is still looking at that paragraph; a document that repeats itself still gets one id per copy, because hashing content alone would merge two pieces into one and acting on either would act on both. With a name, a chunk can be acted on. Enable and disable, delete, write one by hand, rewrite one in place. Disabled means out of retrieval entirely rather than ranked lower, and the filter sits in the store's own search, where every caller present and future passes through it -- the rpc layer is the only door into this engine, so that is the whole of "not retrieved and not fed to the agent". An edit replaces rather than patches: a piece whose text changed and whose vector did not would be found by the old words and read as the new ones. Reindexing a document still deletes every piece of it, written ones included. Keywords answer when the vectors cannot. A base whose embedding model has gone is searched by BM25 rather than skipped, and the result says which of its answers came from words. The index is tokenized by n-grams rather than by whitespace, which is what makes it work at all on a language that writes without spaces. Vector scores and BM25 scores are not comparable numbers, so mixed results are merged by rank. The panel is the other half. A file opens beside its chunks, twenty to a page, with a search box that runs the same retrieval the base does, selection with enable, disable and delete, a full-text or cut-down view, and a double click to rewrite a piece. It is the first reader for the positional metadata the parsers have been recording all along. Underneath, chunking stopped cutting through words. The parser records where every paragraph, table and figure begins and ends, and nothing read it: chunks were sliced on a byte budget that landed mid-word, so both halves embedded as the nonsense they had become. Whole elements are now packed until the next one will not fit, and where a single element is larger than any chunk and has to be split, the cut falls between words. NaiveChunker is RAGFlow's naive_merge_docx, ported: cut on the delimiters, let a table or a figure stand alone, merge neighbours until one is full, and never split a unit -- the size is read before a unit is added, so a chunk overshoots rather than cutting. Sizes are counted with a tokenizer rather than with bytes over four, which read a Chinese character as three quarters of a token. The delimiter field takes several delimiters, one per character, or a longer one in backticks. Overlap repeats a neighbour at each end and is off by default. Those settings now reach a chunker at all. The strategy, the size, the separator and the overlap were saved, shown in the panel, and read by nothing: one chunker was built at startup and used for every base. They decide here -- except the strategy, which is not consulted while the structural chunker is reworked, and which the panel says is not in force rather than offering a switch that decides nothing. A base still carrying the overlap default from when the setting did nothing is read as having none, so making the setting work does not switch overlap on across every base unasked. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

The CI gate reads added lines for CJK and admits it only inside the zones AGENTS.md section 1.3 names; a chunker test is not one of them. The two fixtures are the full-width full stop the default delimiter carries and a short Chinese string whose token count has to exceed what a bytes-over-four estimate would give, so both have to stay Chinese to test anything -- written as escapes the way the docx parser and the gate's own source write theirs. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

Three conflicts, all in the embedding path this branch also reworked. `_embedding.py` was taken from upstream and this branch's additions re-applied on top, rather than the other way round: upstream moved the pin from a knowledge-only block to the one `embedding` block every subsystem reads, and stopped inheriting the endpoint from the memory backend at all -- a knowledge base that needs the memory plugin installed in order to answer is one that stops working when it is not, and the legacy file is now adopted once by `raven doctor --fix` instead of being read on every call. Re-applied on that base: the provider recorded on a resolved endpoint, batching with a learned cap, `asking_for`, and `embedding_config_for` -- the last now built on upstream's own `wire_model` rather than on the prefix-stripping this branch had hand-rolled for the same purpose. The manager's fallback for a base with no recorded provider loses its first step with that: there is no inherited endpoint to try any more, so a base whose model the configured endpoint does not serve falls back to keyword search, and the way to fix it is the provider setting on the base. The tests for the removed behaviour went with it, and the two tests for `embedding_config_for` and the five for batching were re-applied. The generated rpc client was regenerated from the merged contract rather than merged as text. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

…t all A `.doc` had no parser, so every reindex of a base holding one failed on that file and always would have -- while the viewer beside it rendered the same file happily, by converting it with LibreOffice. It is converted now too, to `.docx` rather than to the PDF the viewer wants: a PDF has lost the headings, the table cells and the paragraph boundaries the docx parser reports positions from, and this package has no PDF parser to read one with anyway. The shared LibreOffice helper grew a target rather than a second copy of the profile and teardown rules, and the conversion runs off the event loop, which is also serving the page watching that document's status. No LibreOffice gives the install hint; a conversion that exits 0 having written nothing -- a corrupt or password-protected file does -- fails the document rather than indexing an empty one and calling it ready. A base whose model cannot be reached now says so in its own header rather than in a line of the gateway log. The reason is answered from what is recorded rather than by calling the endpoint: a base list is drawn on every visit, and probing each endpoint to draw it would spend a request per base per render. So it catches the case that happens -- a base whose model is not the configured one and which names no provider of its own -- and an endpoint that is merely down still reads as reachable until the request says otherwise. The repair is where the warning points: the provider a base reaches its model through, which unlike the model and the width can move without invalidating a vector. It sits with the model as one endpoint rather than as a field of its own, because a model id does not name a credential and neither half is worth reading without the other. The file list gains the chunk count. A file that has not been through the chunker shows a dash rather than a zero: a document that failed, one still queued and one in a base with no model have no count to report, and a zero reads as a file that was cut into nothing. Table and figure context default to about a sentence either side rather than to nothing, and the add-a-source menu carries the icons the rest of this page's menus already do. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

…ny python3 Every other python line in the Makefile runs through uv against PYTHON_VERSION; this one called whatever `python3` the shell resolved. That worked by luck until the page assembler started writing its output with `Path.write_text(newline="")`, which is 3.10 and later -- a stock macOS resolves `python3` to 3.9, so `make build-ui` now fails there with an unexpected keyword argument, which reads as a defect in the script rather than as the wrong interpreter. The two `python3` lines in `fetch-templates` have the same shape and are left alone: they are a maintainer path behind a token this change cannot exercise, and an untested edit there would be a guess wearing a fix's clothes. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

@r

… model Three things a knowledge base could not do, and one gap behind all of them. A vision pin. Raven could draw a picture long before it could look at one: `tools.media` holds the painting tool, and nothing named the other direction. `vision.model` / `vision.provider` is that direction, written as a pair like every other pin and offered on the Defaults page under the painting row. The picker cannot filter by kind, because a model that reads a picture is a text model, so the row asks a different question of the same facts: whether it takes an image as input. Unset means off rather than "follow the conversation" -- an indexing run has no conversation, and an image posted to a model that cannot see one fails at the endpoint. Images. An uploaded png, jpeg, gif, bmp, tiff or webp now indexes: the vision model describes it, and the description is what gets embedded. One image is one chunk with no chunker change -- the section carries a single element span marked as a figure, which the chunker already reads to mean "never split, never merge", so a long description stays one piece. A picture with no vision model configured fails with a sentence naming where to set one, because a picture nobody described indexes as nothing at all. Figures inside a Word file. The parser already found them and recorded their page and box; what it lacked was the picture. It now resolves `a:blip/@r:embed` through the package relationships and describes each one, four at a time, with the paragraph either side carried into the prompt. Optional here, unlike an uploaded image: a document is worth indexing whether or not its figures were read. Warnings. A parse has two kinds of bad news and had one way to report either. A file it cannot read at all is an exception that fails the row; a part it could not read is neither, and used to reach only the log -- so a document with half its pictures missing from the index looked exactly like one with none, and the only symptom was a search that answered worse. Parsers now leave notes, the manager writes them onto the document beside its status, and the page marks the row while leaving it ready and searchable. The prompt, and what a figure is asked for, are ported from RAGFlow (Apache-2.0, recorded in NOTICES.md). The OCR ladder in front of it is not: there is no OCR engine in this tree, so there is one path and it says so. Also on this branch, from the same run of work: - A knowledge base is created on a model picked from every embedding model the install can reach, grouped by provider, with the configured default preselected. The same picker in the settings panel moves an existing base onto another model, which drops the collection, rebuilds it at the new width and requeues every document -- asked for once, naming how many. - `embedding_provider` reaches the page, which it never did: the settings picker drew as "as configured" whatever a base had recorded. - A spreadsheet parser (xlsx, xls, csv) after RAGFlow's, where a row labelled by its header is the unit, with the html and markdown renderers kept for the surface that will want them. - RTF reads as text, and `prepare_image` moves from `raven/agent/tools/media.py` to `raven/utils/images.py` so the knowledge package can prepare an image without drawing a new import cycle. Verified: 23386 python tests, 1872 ui-web, 2074 ui-tui, ruff, ty, the import cycle budget, and the page build. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: preserve section and chunk invariants and correct the embedding, fallback-search, CSV, and settings paths before merging.

I reviewed the full github/main...HEAD diff and relevant surrounding code, callers, branch history, and compatibility paths. I also checked AGENTS.md/CLAUDE.md, CONTEXT-MAP.md, and the Knowledge domain contract. The eight concrete failures are marked inline. I found no evidence that tests were removed or weakened merely to get green; however, the new naive-chunker test explicitly asserts behavior that contradicts the existing Chunk contract, as noted inline.

Verification:

uv run pytest on all touched knowledge/parser/RPC tests: 495 passed, 2 skipped because LibreOffice is unavailable.
npm test --prefix ui-web: 108 files and 1,872 tests passed.
npm run --prefix ui-web type-check: passed.
npm run --prefix ui-web gen:check: passed; generated RPC client matches 183 methods.
uv run python scripts/check_source_language.py github/main...HEAD: passed. (make itself is unavailable in this environment.)
Full uv run pytest -x: stopped during collection because the environment lacks the optional websockets package; it also reported the pre-existing optional dingtalk_stream skip.

…to be Eight findings from review, each reachable in production. **A merged chunk described only its first section.** NaiveChunker crosses section boundaries on purpose -- refusing to would turn a document of short sections into a chunk per section, each too small to answer anything -- but the chunk then carried the first section's page, box and heading path as if they were facts about the whole of it. They are now merged to match: the page is a range (`page_number` and `page_end`, which the shape already had room for), a box that would span pages is dropped because a rectangle across two sheets of paper is not a location, and every part that went in is recorded in `elements` with its own page, box, layout and heading path, addressed by where it sits in the chunk's final text. A chunk built from one section keeps that section's metadata byte for byte, which is the common case and is pinned by a test. The overlap and the prose lifted in front of a table both prepend text after the merge has decided those offsets, so `_overlapped` now reports how much it put in front and the spans move with it. Unshifted they pointed at the neighbour's words. `ChunkerBase` is restated to match: whether a chunk may span two sections is a strategy's choice, and what the base demands instead is metadata that describes the chunk it is on. **A base's own provider stopped working when the global pin was cleared.** `_client_for` resolved the configured endpoint first and raised when there was none, so a base whose recorded provider still had valid credentials failed to index and silently dropped to keyword search -- while `embedding_reach` went on reporting that same base as reachable. The configured endpoint is now resolved optionally, and only the case that actually needs it raises. **Chunk edits could reach another document.** A chunk id names its text, two documents in one base can hold the same sentence, and the store's predicates carried only the id. A caller holding an id from document B could switch or delete B's row through document A, and the delete path then rewrote A's count. Both predicates now name the document. **Hand edits left the numbering inconsistent.** Deleting the middle of three left indexes 0 and 2 with every survivor still claiming a total of 3, and an append wrote N+1 on the new row while the existing N said N -- so a hit reported a position its own document disagreed with. A new store operation renumbers the document after both. **Keyword fallback was indistinguishable from a similarity.** An unreachable embedding endpoint turns a search into an ordinary-looking result set scored by BM25 on an unbounded scale, and the RPC forwarded those numbers under a field documented as a similarity with the provenance discarded. Hits now carry `retrieval`, the answer carries `by_keyword` with the reason, and the recall panel says both: the score is drawn differently and labelled, and one line above the results names what changed. **The Windows CSV fallback decoded cp1252 as UTF-8 with replacement**, so "Cafe" with an acute accent was permanently indexed as a replacement character. It now decodes cp1252 as cp1252, and refuses a file that is neither rather than mangling it. **Restore Defaults wrote zero** for the two context sizes, turning the feature off rather than restoring it -- the engine, the record and the page all default them to 64. It restores from the same DEFAULTS as every other field, and the existing test covers them. **The heading prefix was not paid out of the chunk budget.** `_cut` received a budget narrowed by what the prefix costs and then split at the full chunk size in two branches, so every later chunk came back over the limit once the prefix was on it. `_split_text` takes the budget, both branches pass it, and the overlap is clamped below it so a small budget cannot fail to terminate. Verified: 23548 python tests, 1874 ui-web, 2074 ui-tui, ruff, ty, the page build and the css and class gates. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

…nges The TUI keeps its own generated client beside the page's, and `lint:rpc` checks it against `rpc-schema/openrpc.json` by hash. Both commits before this one changed that schema -- `embedding_provider` and `warning` on the knowledge shapes, then `retrieval`, `by_keyword` and `KnowledgeFallback` for the search answer -- and only the page's client was regenerated, so CI failed on a gate that had nothing to do with the change and everything to do with the half of the contract that was not rebuilt. No behaviour here: the file is generated, and this is the output of `npm run gen:rpc --prefix ui-tui` against the schema as it already stands. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: preserve section provenance, renumber migrated chunks, and honor a base's recorded provider when model names collide.

Six earlier findings are fixed and their threads are resolved. Two existing blockers remain: cross-section chunks are still flattened to the first section by the RPC/UI while the canonical and parser contracts still forbid the merge, and renumbering cannot update migrated rows whose chunk_id is empty. One new provider-selection blocker is marked inline.

Coverage: I checked the repository rules and canonical knowledge terms; the full PR diff plus the changes since d11723b02f5a; affected callers across manager, LanceDB, RPC/schema/generated clients, and web UI; relevant history; backward compatibility with pre-ID collections; test changes for weakening; and the chunk/provenance architecture. I found no weakened tests.

Verification on this head: 447 focused Python tests passed and 2 LibreOffice-dependent tests skipped; 144 knowledge-page tests passed (with happy-dom's existing aborted/refused iframe-fetch diagnostics), web type-check and generated-client check passed; after installing the declared TUI dependencies, its RPC drift check and type-check passed; the source-language gate passed; and a direct cp1252 CSV check preserved Café.

gloryfromca

Blocking: the naive chunker must preserve section provenance before merge.

This head only merges main's bilingual README update; the affected knowledge, storage, RPC, and UI paths are byte-for-byte unchanged from the reviewed revision. One blocker remains. It meets all three fourth-round criteria: this PR introduces cross-Section merging, it is reached in ordinary indexing because _chunker_for() currently returns NaiveChunker for every base regardless of smart_chunking, and an operator has no exposed setting or reindex path that preserves the later Section's displayed page and heading.

Two nonblocking follow-ups remain named here: migrated empty-ID rows need reindexing before manual mutations can renumber them consistently, and same-named models on different providers require clearing or aligning the global pin so the base's provider wins. I withdrew those blockers and resolved their threads because each has a working operator escape.

Coverage this round: the main merge delta, the full PR diff's standing findings, the affected callers and history, backward compatibility, test changes, repository rules/canonical terms, and the chunking architecture. No tests were rerun because this revision changes only target-branch README content and none of the reviewed code, tests, schemas, generated clients, or dependencies.

…st the pin Reproduced as reported. With a pin naming provider A and a base recorded against provider B, both serving the same model id, `_client_for` returned A's client and never tried B: the base's queries went out on another account's key, against vectors that account did not make, and the base fell back to keywords whenever A was the one that was down. `embedding_reach` had the same hole in the other direction -- it called such a base reachable because the *pin* was usable, while the account it actually queries through had no credential left. Two providers reselling one model id is ordinary, and the contract around this code already says a model id does not name a credential. That is the whole reason a base records the provider beside it. The three places that ask this question now ask it the same way, through one `_serves(endpoint, model, provider)`: the id must match, and so must the provider whenever the base recorded one. An empty provider still means "wherever this is configured", which is every base written before the field existed and the one case where the id alone decides. Writing the predicate three times is how the three answers drifted apart, which is why this is a function rather than a third copy of the condition. Both new tests fail against the previous revision and pass against this one. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: the naive chunker must preserve section provenance before merge.

This revision correctly fixes the provider-aware matching follow-up when two providers expose the same model id, including the legacy empty-provider path. It does not touch the sole open blocker: the naive chunker, parser/CONTEXT contract, and RPC projection are unchanged. I reproduced the ordinary two-section path again; it still emits one chunk spanning pages 2 and 7, while knowledge.documents.chunks exposes only page 2 and the first heading. The existing thread therefore still stands.

At this sixth review round, I am explicitly claiming all three blocker conditions: this PR introduces the cross-section merge, every normally configured base currently uses NaiveChunker, and the document UI/RPC has no way to recover or display the second section's provenance.

Verification: uv run pytest tests/test_knowledge_manager.py tests/test_knowledge_naive_chunker.py tests/test_rpc_knowledge.py -x (205 passed); uv run python scripts/check_source_language.py github/main...HEAD (passed); focused runtime reproduction (failure remains). I rechecked the repository rules and canonical CONTEXT.md, the current delta in the full PR diff, callers and RPC/UI consumers, commit history, backward compatibility, test changes/no weakening, and the Section-to-Chunk architecture contract.

gloryfromca

Blocking: the naive chunker must preserve section provenance before merge.

This revision only removes trailing whitespace from the web type declaration. None of the chunker, canonical contract, RPC projection, or document UI paths behind the sole open blocker changed, so the existing thread still stands and there are no new findings. The previously established late-round conditions remain unchanged: the PR introduces the behavior, every normally configured base takes this path, and the operator has no way to recover the hidden section provenance.

Verification: uv run pytest tests/test_knowledge_manager.py tests/test_knowledge_naive_chunker.py tests/test_rpc_knowledge.py -x (205 passed); uv run python scripts/check_source_language.py github/main...HEAD (passed). I checked the repository rules and canonical CONTEXT.md, the one-line revision delta in the full PR diff, callers and RPC/UI consumers, history, backward compatibility, test changes/no weakening, and the Section-to-Chunk architecture contract.

…laration One character, on the `retrieval` field added a few commits back. The pre-commit gate rewrites it and then fails the run for having had to, which is how CI reported it. Nothing else on the branch carries one: the whole diff against upstream/main scans clean, and the hook set now runs clean over the same range. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: the naive chunker must preserve section provenance before merge.

This head rewrites the trailing-space cleanup commit, but its tree is byte-for-byte identical to the last reviewed revision (git diff --quiet 404d11c0dc71 HEAD). The sole open provenance blocker is therefore unchanged; there are no new findings and no settled thread to resolve.

Verification: git diff --check github/main...HEAD (passed); uv run python scripts/check_source_language.py github/main...HEAD (passed); uv run pytest tests/test_knowledge_manager.py tests/test_knowledge_naive_chunker.py tests/test_rpc_knowledge.py -x (205 passed). I checked the repository rules and canonical CONTEXT.md, the rewritten commit and full PR diff, callers and RPC/UI consumers, history, backward compatibility, test changes/no weakening, and the Section-to-Chunk architecture contract.

NaiveChunker crosses section boundaries, which is what it is for. What it lost crossing them was which section each part came from: the chunk carried the first part's `section_ordinal`, and every later part was silently attributed to a section it never came from. That key is defined as *the* section identity, and its own docstring says why a heading path cannot stand in for it -- two same-named children of one parent share a path. So there was nothing else to recover it from: a hit in the second half of a merged chunk, widened back to "its" section, would be handed the text of a different one, with nothing anywhere saying the attribution was a guess. Every span in `elements` now carries the section it was cut from, and the chunk-level key is dropped when the parts disagree -- a chunk built from several sections does not have one, and keeping the first is the mis-attribution the key exists to prevent. The heading path stays at chunk level even then, because it is a label rather than an identity, and each part's own travels beside its section. The origin is read from the section before `_rows` overlays an element's reading order on top of it, which is why it is held apart from the row metadata: after that overlay a row cannot say which section it belongs to. A parser that writes no ordinal falls back to the section's own reading order -- one section per row for a spreadsheet, one per file for plain text, the same fact under the only name that parser records it by -- and a section that states neither gets no origin rather than an invented one. Three of the six new tests fail against the previous revision. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: per-part section provenance must reach the RPC/UI consumer.

The new chunker metadata and its regressions are internally consistent, but they do not close the sole open blocker: section_ordinal remains nested in metadata.elements, while the RPC projection/schema and web row still discard those elements and label the merged chunk only with its first page and heading. The original reproduction still fails at that consumer boundary, so the thread remains open. There are no additional findings.

This continues to meet the late-round blocker bar: this PR introduces the cross-section merge, every normally configured base uses it, and the operator-facing consumer has no route to the hidden provenance.

Verification: git diff --check github/main...HEAD (passed); uv run python scripts/check_source_language.py github/main...HEAD (passed); uv run pytest tests/test_knowledge_manager.py tests/test_knowledge_naive_chunker.py tests/test_rpc_knowledge.py -x (211 passed); focused RPC projection reproduction (failure remains). I checked the repository rules and canonical CONTEXT.md, the current delta in the full PR diff, chunking callers and RPC/UI consumers, history, backward compatibility, test changes/no weakening, and the Section-to-Chunk architecture contract.

…on to a chunk A deck could not be uploaded at all: no parser claimed the two PowerPoint media types, so the picker refused the file and the engine had nothing to route it to. It reads now, with no new dependency -- a `.pptx` is OOXML, the same zip of XML the Word and spreadsheet parsers already walk with the standard library. `python-pptx` is in the lockfile only because the deck engine declares it, and nothing under `raven/` may assume a plugin's dependency is installed. A legacy `.ppt` goes through LibreOffice first, the path `.doc` and `.xls` already take, chosen by the bytes rather than the extension so a mislabelled file still reads. What it reads follows RAGFlow's `RAGFlowPptParser`: shapes in the order they sit on the slide rather than the order the file lists them, bulleted paragraphs marked by their indent, tables composed header-onto-row, group shapes walked through. Two things about the format are worth stating, because both read as a working parser returning nothing: - `sldId` carries two attributes whose local name is `id` -- the slide's own number and the relationship pointing at its part. Matching by local name is correct everywhere else in OOXML and picks the wrong one here, which resolves to no relationship and so to no slides at all. - An ElementTree element with no children is falsy, so `found or default` takes the default for a placeholder that was found. That is how no deck had a title. Slide order comes from the presentation's own list, never the part names: `slide10.xml` sorts before `slide2.xml`, and a reordered deck keeps the numbering it was first saved with. Placeholders that state no geometry inherit it from the slide layout, which is what PowerPoint does -- without it a title sorts after every shape that does state a position. One slide is one chunk. That is the parser's point and not an accident of sizing, so it is a contract rather than a side effect: a section may be marked `ATOMIC`, and every chunker honours it -- the section is not split however long it runs, nothing is merged into it, and the overlap does not reach into it. Honoured in all three chunkers because which one a base runs is a setting, and a guarantee a parser relies on cannot depend on a setting. Checked before the element spans are read, because on a slide those spans say where each shape sits, not where the text may be cut. The table composer moves out of the Word parser into `parser/_tables.py`. Two formats want the same composition and the same reason for it, and a third will. Verified against the deck engine's own bundled templates: 15 slides parse to 15 sections and 15 chunks, at chunk sizes of 16, 512 and 8192 and with overlap on, with pages running 1 to 15. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

…ot use `_chunker_for` returns a naive chunker for every base and says in as many words that `smart_chunking` "is not consulted yet" and that "the panel says it is not in force". The panel said the opposite. The switch was on by default, its help said the document's own headings and sections were what cut -- and while it was on it greyed out the separator, which is the setting that actually decides. So a reader could not edit the one field in force without flipping a switch that changes nothing, and believed their documents were being cut on structure while every base was cut at the separator. The engine is left alone: which chunker runs is a decision about the structural pass, not about this panel. What changes is that the panel now says what the engine does. The switch is shown and not operable -- it is the shape the answer will take, which is why the setting is kept at all -- and the separator is editable, because it is what cuts. The case that pinned the old behaviour pinned the bug: it asserted the separator was disabled while the switch was on. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: the unresolved before-merge findings R10-R16 still need fixes or maintainer scope decisions.

I reviewed the two-commit delta from ecaac0b42b73 through eab523b00aab. It substantively implements the requested R7-R9 remedies: model switches drain the web base's pending rows, derived base counts are refreshed and the RPC refuses an unconfirmed destructive switch, and the unimplemented structure control is now inert while the effective separator remains editable. I found no new defect in this delta; I am leaving those three threads for their author to verify and close.

R10-R16 are still open, and this delta does not touch their underlying behavior or licensing/dependency decisions, so the revision remains blocked on that existing record.

Coverage for this delta: repository rules and relevant knowledge contracts, changed code and its RPC/web callers, commit history, backward-compatibility implications, architecture boundaries, and whether tests were weakened. Verification: 197 passed in the focused Python knowledge/RPC suite; 131 passed in the focused web store/page suite; generated-client check and TypeScript check passed; source-language and large-file gates passed; git diff --check passed; and the merge-tree against current github/main is clean.

…the core R10. "Restore defaults" wrote `chunk_overlap: 215`, which is the number every base carried while nothing read the setting -- and the records layer reads a base still holding exactly it as having no overlap. So restoring defaults set the one value that means "never set": the chunker used a 215-token overlap for the rest of the process and the next load read it back as 0. The default is 0, which is what the engine's is. R11. The composer's base list returned early whenever it was not null and stored `[]` on failure, so a failed first read was never retried -- while the note above it promised that opening the menu again is the retry. It stays null now, and a base made or dropped on the page invalidates it, so the menu offers what exists rather than what existed when it first opened. R12. `search_document` keyed its lookup by chunk id. Rows written before ids existed all carry the empty one, so the dictionary kept the last of them and answered every one of those hits with that same piece: a search of such a document came back as one chunk repeated, and the branch written for them could never run. Only ids that name a row are keyed on. R13. EverMind-AI#426 added `embedding: false` so a base can be made without reaching the embedding endpoint, and the page never sent it -- so with the endpoint down there was no way to make a base at all. The create form offers it. R14. `pypdf`, `pdfplumber` and `scikit-learn` are imported nowhere; the comments beside them described uses that do not exist. Removed. `datrie`, `hanziconv` and `nltk` are reached only through `resources.warm_dictionary`, and datrie is the one package that must compile: PyPI has no wheel for any Python this project supports, so every install builds a C extension and `uv tool install raven` fails outright without a toolchain -- the whole install lost, for a module almost nothing calls. They move to a `segmenter` extra, and both steps that reach them skip with a sentence rather than raising. Verified against a core wheel in a clean venv: none of the six installs, and `python -m raven.resources` still fetches the dictionary and exits 0. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

Blocking: the remaining R11 rename-cache defect and R15-R16 licensing decisions still need resolution or explicit maintainer deferral.

I reviewed the delta from eab523b00aab through f1fbd835ad4f. R10, R12, R13, and R14 are substantively addressed by the corrected overlap default, legacy empty-id lookup, embedding-free create path, and dependency split. R11's failed-read retry and create/delete invalidation are fixed, but renaming a base still goes through write() without invalidating the composer's cached mentions list, so that part of the original thread remains. R15 and R16 are untouched. I found no separate new blocker in this delta and am leaving the existing threads to their author or a maintainer.

Coverage: repository rules and relevant knowledge contracts, the changed diff and its resource/RPC/web callers, commit history, backward compatibility, dependency and architecture boundaries, and test changes. Verification: the focused web page/store/mentions suites passed (148 tests); generated-client and TypeScript checks passed; lockfile, source-language, large-file, and git diff --check gates passed; and the merge-tree against current github/main is clean. The Python manager/resource run produced 134 passes and 12 failures because this host intentionally lacks the new segmenter extra and also lacks gcc, so datrie cannot be installed here; the manager plus the new no-segmenter cases pass, while CI's declared setup uses uv sync --all-extras on a compiler-equipped runner.

The deepdoc pipeline here was ported from RAGFlow, which reads its PDFs with pdfplumber and rasterises through pypdfium2. The port swapped that for PyMuPDF, which does all of it in one library -- and which is AGPL-3.0. A knowledge base is part of the core wheel, so that put an AGPL obligation on every install, including the ones that never open a PDF. The plugins that need PyMuPDF's writing side pin it themselves and keep the obligation on their own wheels. Everything above a PDF asked the same four questions of it -- how big is this page, what text is on it, what tables are on it, what does it look like -- in one library's vocabulary, four times over. They are answered once now, in parser/_pdfdoc.py, in this package's own terms: a page has a box, blocks, tables and a rendering, and swapping what reads them is an edit to that one file. Two differences between the libraries had to be closed rather than renamed: * a PDF's content stream is not bounded by its page. A line set too long simply carries on past the edge, and PyMuPDF clips it the way a viewer does while pdfplumber reports every glyph the stream draws. A line running off the page then came back as a box wider than the page, which is a box no region of that page can contain -- and since the layout model places text by asking which region it fell in, that text was read and then dropped. Cropping each page to its own box is the same cut a viewer makes: on the fixture that caught this, both libraries keep 113 of the 240 characters. * pdfplumber hands out words, not lines or paragraphs. Words are gathered by where they sit vertically, then cut wherever the gap between two is too wide to be a space -- without that second pass a two-column page reads straight across the gutter, because both columns share a baseline. Lines are gathered into blocks against every open run rather than only the newest, because on that same page the newest one is the other column. Pages are also released as the reader passes them: pdfplumber holds every page's objects once read, and the reader walks forward and never looks back. Verification: uv run pytest tests/ -k knowledge -- 1096 passed, 1 xfailed. The deepdoc weights are installed locally, so the layout, OCR and table-structure paths all ran rather than skipping. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

The second of the core's four PDF readers. A page is rendered and the regions of each chunk are cut out of it, which the shared document reader already does -- so this is the render call and the one helper around it, and nothing about the cutting or the stacking below it. Verification: uv run pytest tests/ -k "crop or knowledge" -- 1125 passed, 1 xfailed. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

The third of the four. Page sizes and the rendered page the reader's side places its highlights on, both through the shared document reader. PIL goes with it on the render path: the reader already hands back an image, so there is nothing left to build one out of a pixel buffer. Verification: uv run pytest tests/ -k "knowledge or pages" -- 1234 passed, 1 xfailed. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

The last of the four, and the one that changes shape rather than just calls. This module probed for PyMuPDF and fell back to poppler's pdftoppm, because the only rasteriser the gateway had rode in with a deck engine: an install without one could draw no page at all, and said so. There is no such install now -- the knowledge reader is a hard dependency of this package -- so the probe, the fallback and the "no PDF rasteriser on the gateway host" answer all go. What goes with them is the scaling the fallback needed. pdftoppm takes the size it draws in flags, and the page's own size had to be read back out of pdfinfo first, because a width alone leaves the height to a number inside the file. The ceiling that bounds it is in _thumb_zoom, which the drawing path has always used and which test_a_tall_page_is_ drawn_small_rather_than_expensively still holds: a 72 x 14400 point page draws under five megapixels rather than 328. A PDF with fewer pages than were asked for is now covered too; it had been written and never reached, because the test monkeypatched a reader whose page_count was always one. Verification: uv run pytest tests/test_rpc_files.py -- 85 passed, 1 failed. The failure is test_a_host_without_libreoffice_says_so, which fails the same way on this branch's base: macOS resolves soffice under /Applications, so blanking PATH does not hide it. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

The last step of the four above it, and the point of them. Nothing in `raven/` imports PyMuPDF now except the deck template gallery, which probes for it with find_spec and only exists when the ppt-engine plugin -- which pins it -- is installed. So the core wheel no longer carries an AGPL-3.0 dependency for a feature that is absent without the plugin, which is what the deptry note beside this list has always said it must not do. pdfplumber and pdfminer.six arrive in its place. The second is declared rather than left transitive because one name is imported from it directly: pdfplumber wraps every cause it can hit in one exception type, so a truncated download and a password-protected file arrive identically, and only what is underneath tells them apart. PyMuPDF moves to the dev group, as a PDF *writer*. The reader this wheel now ships cannot draw a PDF, and a fixture has to be drawn before it can be parsed. A dev group is not distributed, so no obligation rides with it. experimental/simulation/render.py still imports it and is left alone: `packages = ["raven"]`, so it is not in the wheel. Verification: `uv run --frozen --all-extras deptry raven` -- no issues, where it reported one before pdfminer.six was declared. `uv run pytest tests/` -- 29414 passed, 121 skipped, 2 xfailed, 3 failed. The three are the macOS-environmental ones this branch started with and they fail the same way on its base: soffice resolved under /Applications, a case-insensitive filesystem, and process-descendant enumeration. `make check-source-language` and `make check-large-files` both clean. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

0xKT

One new before-merge item at 3501fd7, R18: the unit 4/4 job is red because of EVEROS_* environment leakage between tests that this PR's test count re-dealt into one shard; the inline comment carries the cause, the reproduction and a one-move fix tried at the head. R1-R10 and R12-R15 are verified and resolved; R11 (rename invalidation), R16 and R17 stay open as noted on their threads.

gloryfromca

Blocking: the open R11 rename-cache defect and R16 model-licensing decision still need resolution.

I reviewed the revision delta from f1fbd835ad4f through 6c7050c53947: the PyMuPDF removal, the new pdfplumber/PDFium document abstraction, every parser/crop/page/preview caller, dependency and lock changes, relevant history and compatibility paths, test changes, and both main merges. I also checked the applicable AGENTS/CONTEXT contracts and the RPC/UI architecture boundary. I found no new defect in that delta, and R15 is fixed by keeping PyMuPDF out of the distributed core wheel. R11 and R16 are unchanged and remain open, so this revision is still blocked.

R18's concrete CI failure is no longer reproducible on this head: the current GitHub run has all four unit shards and the coverage gate green after the latest main merge. I left that thread open for its author to disposition.

Verification: the focused PDF/parser/crop/RPC suite passed 201 tests with the in-repo EverOS plugin on PYTHONPATH; the first invocation without that repository plugin path produced 201 setup errors and no test failures. The knowledge page/store/mentions suites passed 148 tests (Happy DOM emitted its existing aborted-fetch diagnostics). uv lock --check, generated-client check, TypeScript, source-language, large-file, diff, and merge-tree checks passed; GitHub reports every required check green.

The `@` menu reads the list of bases once and keeps it, on purpose: a menu that refetched every time it opened would spend a round trip to draw three names that have not changed. Two things dropped that list -- making a base and deleting one -- and renaming was not one of them. A rename reaches the store through `write`, the one path every card control writes through. So the menu went on offering the base under the name it used to have, and went on doing it for the life of the tab, because the list is never read a second time. The reader renames a base on the knowledge page, opens the composer to attach it, and the name they just chose is not there. Keyed on the name actually changing rather than on the field being passed: a star and a pin come through the same call, and the menu shows neither, so neither is worth dropping a list over. Verification: npx vitest run -- 238 files, 3701 passed. The new rename test fails with the one line removed; the star test is the control that it does not fire on every write. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

NOTICES.md gave these four models one licence, Apache-2.0, taken from the Hugging Face repository that publishes them. The repository is Apache-2.0; the weights in it are not uniformly so, and `layout.onnx` -- the page-layout model, the largest of the four and the one the whole model-driven read path turns on -- stamps `AGPL-3.0 License` into its own ONNX metadata, under `author: Ultralytics`. Its metadata contradicts itself: the URL beside that licence field points at RAGFlow's own LICENSE, which is Apache-2.0. AGPL-3.0 is also what Ultralytics' export tooling stamps by default on anything built with it, so the field may be recording the tool rather than the publisher's intent. This repository cannot settle which was meant and does not try to. It records the fact, and says where the decision actually lands. Where it lands is narrow, because no Raven artifact carries a weight. Verified rather than assumed: `uv build --wheel` produces 1181 entries with no `.onnx`, no `res/` and no `nltk_data` among them -- `raven/knowledge/res/` is gitignored and the wheel's include list names neither. `python -m raven.resources` fetches them into an install that already exists, so a `pip install` distributes nothing and an install that never runs `make fetch-resources` reads PDFs geometrically and loads none of them. Building an image with the weights already fetched into it is redistribution, and that is the case this note is for. NOTICES.md ships in the wheel (dist-info/licenses), so it reaches every install that could do so. The same flat Apache-2.0 claim was in resources.py beside the repo constant; it now points at this section instead of restating it. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

…e_parsers

gloryfromca

Blocking: the deepdoc licensing notice and default Docker distribution must agree before this merges.

R11 is fixed: a successful rename now invalidates the composer's cached base list, unrelated settings writes do not, and the focused store/mentions suites pass (73 tests). I reviewed all four changed files, their callers and history, the applicable AGENTS/CONTEXT constraints, compatibility, and whether the new tests weaken prior coverage; I found no issue with that fix.

R16 is not settled. The new table correctly records layout.onnx as AGPL-3.0-stamped, but the same notice says no Raven artifact contains the weights. The unchanged ordinary Docker/Compose build defaults RAVEN_RESOURCES to 1 and fetches those files into the resulting Raven image, so that artifact does redistribute them. I added the concrete mismatch to the existing R16 thread rather than duplicating the finding.

Verification: npm test -- --run src/features/knowledge/store.test.ts src/state/mentions.test.ts passed 73 tests. uv lock --check, source-language, large-file, diff, and merge-tree checks passed. The new GitHub run was still in progress when this review was posted.

The notice said no Raven artifact carries these weights. That was true of the wheel, which is what it was checked against, and false of the image: `Dockerfile` defaults `RAVEN_RESOURCES=1` and runs the same fetch at build time, so the container `docker/docker-compose.yml` builds -- the one the top-level README's install instructions produce -- has all four baked in, `layout.onnx` among them. A notice that overclaims on the one artifact most people will actually run is worse than no notice. Corrected rather than resolved by changing the build. The default is deliberate and the reason is sound where it is set: a container that first reached Hugging Face when somebody uploaded a PDF would fail in a deployment with no egress, long after the build could have said so. And this project publishes no image -- there is no registry push in .github/workflows -- so Compose building one on the operator's machine is their own fetch happening inside a container rather than beside one. What was missing was the warning at the moment it matters, so it is now in the three places someone deciding to publish an image passes through: NOTICES.md states which artifact carries what, `docker/README.md` says it in the RAVEN_RESOURCES bullet, and the Dockerfile says it beside the ARG that sets the default. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

gloryfromca

No blockers; suggestions only, and they are marked inline.

The R16 distribution decision is now explicit and internally consistent: the wheel excludes the weights; the default locally-built container includes them; the notice and Docker documentation identify pushing that image or shipping an appliance as redistribution; and RAVEN_RESOURCES=0 is the documented build without them. That is the maintainer's in-scope choice, so the issue that held my previous stance is resolved. I found no new defect in the three-file delta.

I reviewed the diff, its Docker/Compose callers and history, the applicable AGENTS/CONTEXT constraints, compatibility of the existing build argument, and the unchanged test surface. Existing nonblocking follow-ups remain, including the three LivXue threads and the previously named compatibility/documentation items.

Verification: uv lock --check, source-language, large-file, full pre-commit checks for the three changed files, diff, and merge-tree checks passed. The documentation build was green; the new general CI run was still in progress when this review was posted.

0xKT · 2026-10-12T09:24:52Z

Status from the maintainers' review at 03bea96: all sixteen inline items (R1-R16) are verified and resolved, and R18 was downgraded to a follow-up. What still stands between this PR and a merge:

R17, the PR description: it is the squash commit body on main, and it still describes the head as it was at 19a8009 (see the review body for the fourteen points; the chunker change from R9 and the #397 reference from R13 belong there too). A rewrite, or a reply saying which points you decline and why, closes it either way.
The three threads LivXue opened on 2026-10-10 are unresolved; the ruleset requires every thread resolved. A reply on each (fixed, deferred, or declined) is enough.
The CI run for 03bea96 is still in progress.

Once those are in, the review verdict gets refreshed against the head. No new code is being asked for.

0xKT · 2026-10-12T11:28:28Z

Merged at 82c76a8; thanks for turning the sixteen items around in a day. The follow-up list (F1-F26 above, plus the EVEROS_* test-environment leak noted on the R18 thread) stays with you to track and take on as you see fit; no tracking issues will be opened from the maintainers' side.

## Summary `bind_roles_here` writes the four EverOS roles into `os.environ` directly, records them in `raven_everos.config._BOUND_HERE` and publishes that record as `RAVEN_EVEROS_BOUND`. `monkeypatch` undoes none of it: it records only what it changed itself, and `delenv(..., raising=False)` on a name that is not there records nothing. The everos backend tests have put the `EVEROS_*` variables back since #489, but not the record or the set, and the binding is not theirs alone: every runtime assembled with the memory plugin on binds through the plugin's start. A run of the whole suite with a watcher on these three things (2026-10-15) found fourteen files binding -- the design and ppt launchers, eval hooks, plugin tools, remote file tools, direct chat, onboarding, the TUI command tests. So one test's binding was the next test's environment, and which test paid depended on how the suite was dealt into shards: ten `tests/test_rpc_settings.py` save cases refused with "decided by the `EVEROS_LLM__*` environment variables" in shard 4/4 of #920 (whose tree predated #489), and nowhere else. What this does, tests only: - `tests/_everos_env.py`: `confined_everos_binding()`, a block that puts the variables (`EVEROS_ROOT` included), the provenance variable and the contents of the provenance set back when it ends, exception or not; and `forget_everos_binding(monkeypatch)`, which starts a reader with none of them and swaps the set rather than clearing it. - `tests/conftest.py`: one autouse fixture, `_no_leaked_everos_binding`, around every test, next to the other process-global restores (`_unbind_the_permission_turn`, `_no_leaked_usage_sink`). It replaces the everos backend file's own fixture, which covered the variables alone. - `tests/test_rpc_settings.py` and `tests/test_config_update_everos.py`: the two readers of `role_is_env_managed` start with no role of any kind exported, not only no embedding, through the helper. - `tests/test_everos_env_confinement.py`: the helper's contract (a binding made inside the block is undone; what was there before comes back; so on an exception; the forget half), and one slow test per representative file that, in a child interpreter started clean and running under the suite's own conftest, checks that the file leaves no variable, no record and an empty set behind. Red on the tree before this change for all three targets (the backend file left the record; the launcher case left twelve empty role variables, `EVEROS_ROOT` under a `tmp_path` that is gone, and the record). The board's own reproduction (the named everos case, then `tests/test_rpc_settings.py`, one process) does not fail on main any more because of #489's fixture; it was kept as a check and passes. Third commit, after review: the block restored the contents of whichever set the attribute held when it ended. That is the wrong object for a test that binds and then swaps the attribute with `monkeypatch`, because `monkeypatch`'s undo runs after an autouse fixture's teardown here (`_isolate_tracing_state_dir` asks for it and is set up earlier; autouse fixtures of one conftest are set up in name order): the swapped-in set got cleaned and the module's own came back with everything bound into it -- a probe handed the next test twelve keys. No test in the suite binds before it swaps, so nothing failed, but the helper's docstring claimed the reverse order. The block now keeps the object it began with and restores that object's contents and the attribute's binding to it, whatever the attribute holds by then; the docstring says why, and a test that binds, swaps with a MonkeyPatch undone only after the block, and reads a clean set is red before the change. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [x] CI / tooling - [ ] Refactor - [ ] Other ## Verification - A throwaway pytest plugin recording, per test, every change to `EVEROS_*`, `RAVEN_EVEROS_BOUND` and `_BOUND_HERE`, over the whole suite (`-n 6`): before this change 63 tests in 16 files changed them (14 binding, 2 clearing); after it, the watcher wrote nothing: no test changed them (31719 passed, same run). - `pytest tests/test_everos_env_confinement.py -n 0`: 8 passed, with and without `CI=1`; the three child-interpreter cases fail on the tree before this change with the leftovers named above. (The first push had them print the report after `pytest.main`, which on CI never returns -- the suite's conftest hard-exits the interpreter in `pytest_unconfigure` when `CI` is set -- so shard 1/4 failed on "the child did not report" while the targets had passed; `CI=1` reproduces that locally. The child now writes the report from a `pytest_sessionfinish` plugin into a file the parent reads.) - `tests/test_everos_backend.py` 191 passed, `tests/test_agents_ppt_launcher.py` 61 passed, `tests/test_rpc_settings.py` 153 passed, `tests/test_config_update_everos.py` 50 passed, each on its own; the board's reproduction order 154 passed; the fourteen binding files followed by every reader in one process, 1261 passed. - Full suite on the merged tree (`-n 6`): after the third commit, 31720 passed, 173 skipped, 2 xfailed, 3 failed -- the same three that fail on main on this machine (a LibreOffice case and two simulation cases); the related group (`test_everos_backend`, `test_rpc_settings`, `test_config_update_everos`, `test_everos_config`, `test_everos_server`, the two launcher/eval-hook files) 651 passed. - `ruff check`/`ruff format`, `scripts/check_source_language.py`, `scripts/check_large_files.py`, `scripts/check_commit_messages.py`: pass. No production code changes, so the diff coverage gate has no changed executable lines to measure. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk - [ ] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Tests only. A test that asserted on a binding left by an earlier test would now see it gone; the full-suite run says none does. Rollback is reverting the commit. ## Related Issues N/A --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>

…els (#1026) ## Summary The release notes template had drifted from what main ships, so the draft CI builds for the next tag would carry four stale statements. This brings the template, and RELEASING.md which describes it, in line with main. - Install: `install.sh` now downloads the word dictionary and PDF models the knowledge reader uses (#489). `RAVEN_MINIMAL=1` skips them and `install.ps1` does not download them, so the section names `raven resources` for both cases. - Install: bare `raven` opens the page where a browser can be opened and the TUI only otherwise; the terminal entry is `raven tui`. The v0.2.4 notes corrected this by hand, but the template kept the old sentence. - Upgrade: the update row at the bottom of the sidebar upgrades a page served by 0.2.3 or later. This too was written into the v0.2.4 notes by hand. - Assets: a release attaches four plugin wheels since #883 added `everos_cloud_memory`; the plugin list still installs the other three. This should merge before the 0.2.5 version bump (#1025), so the drafts its tags build start from the corrected template. Each statement was checked against main: - `install.sh` runs `raven.resources --optional` unless `RAVEN_MINIMAL` is set; `install.ps1` has no resource step. - With no subcommand, `raven` opens the page when a browser can be opened and the TUI otherwise (`raven/cli/commands.py`). - `release.yml` builds the `everos-cloud-memory` wheel beside the other three plugin wheels and uploads every wheel, while `raven-plugins.txt` lists only `everos_memory`, `design_engine` and `ppt_engine`. ## Type - [ ] Fix - [ ] Feature - [x] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `sed "s/__VERSION__/9.9.9/g; s/__TAG__/v9.9.9/g" .github/release-notes-template.md`, the substitution `release.yml` runs: no placeholder is left, and the three TODO markers the draft keeps for hand-filling are unchanged. - `PYTHONPATH=. python3 scripts/check_commit_messages.py origin/main..HEAD`: exit 0 - `npx commitlint --from origin/main --to HEAD --config commitlint.config.cjs`: exit 0 - `PYTHONPATH=. python3 scripts/check_large_files.py origin/main..HEAD` and `PYTHONPATH=. python3 scripts/check_source_language.py origin/main..HEAD`: exit 0 - No test run: no test reads either file. - [ ] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk Docs only. What changes is the Install, Upgrade and Release Status text of the next release drafts. Rollback is reverting this commit. - [ ] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>

## Summary Bump the package version from 0.2.4 to 0.2.5 for the next release. Only pyproject.toml and uv.lock change. Since v0.2.4, main merged 115 PRs: 30 features, 74 fixes, three perf, three docs, three test, one refactor and one build. The larger features are work on registered machines (#920, #922, #934, #947, #983, #1005), the host as an agent manager (#1010), the knowledge reader (#489), Personas (#709), the everos-cloud-memory plugin (#883) with everos 1.4.2 (#950), and the trajectory view with bug report export and upload (#877, #926, #948). None carries a BREAKING CHANGE footer. The maintainer chose a patch bump, the way v0.2.1 through v0.2.4 were for their features. The release workflow now also builds the everos-cloud-memory wheel (#883), a step no release has run yet, so a v0.2.5-rc1 tag goes first to dry-run the pipeline. The stable tag and release notes follow once this lands. uv.lock is regenerated with uv 0.12.12. uv 0.9.x rewrites the dependency markers of five unrelated packages, and uv 0.13.x raises the lock revision from 3 to 5. uv 0.12.12 leaves only the raven version line changed. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [x] Other ## Verification - `uvx --from uv==0.12.12 uv lock` on a clean worktree cut from origin/main: Updated raven v0.2.4 -> v0.2.5 - `git diff --numstat origin/main..HEAD`: one line each in pyproject.toml and uv.lock - `uvx --from uv==0.13.3 uv lock --check`: exit 0, so the `uv export --locked` step in release.yml accepts the lock - `PYTHONPATH=. python3 scripts/check_commit_messages.py origin/main..HEAD`: exit 0 - `npx commitlint --from origin/main --to HEAD --config commitlint.config.cjs`: exit 0 - No test run: the change is version metadata only. - [ ] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk No code change. Rollback is reverting this commit before any v0.2.5 tag is pushed. - [ ] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>

KevinHuSh and others added 12 commits September 16, 2026 18:13

Merge remote-tracking branch 'upstream/main' into feat/file_parsers

a3b7351

Merge remote-tracking branch 'upstream/main' into feat/file_parsers

df7fe4a

Merge remote-tracking branch 'upstream/main' into feat/file_parsers

d11723b

gloryfromca reviewed Sep 18, 2026

View reviewed changes

KevinHuSh changed the title ~~feat(*): read images with a vision model, and pick a base's embedding model~~ feat(knowledge): read images with a vision model, and pick a base's embedding model Sep 18, 2026

KevinHuSh and others added 3 commits September 18, 2026 13:45

Merge remote-tracking branch 'upstream/main' into feat/file_parsers

7de8fbc

gloryfromca reviewed Sep 18, 2026

View reviewed changes

Comment thread raven/knowledge/_manager.py Outdated

Merge branch 'main' into feat/file_parsers

94def10

gloryfromca reviewed Sep 18, 2026

View reviewed changes

KevinHuSh force-pushed the feat/file_parsers branch from 404d11c to 81ea4bb Compare September 18, 2026 06:24

gloryfromca reviewed Sep 18, 2026

View reviewed changes

KevinHuSh and others added 2 commits September 18, 2026 15:03

Merge branch 'main' into feat/file_parsers

351d187

gloryfromca reviewed Oct 12, 2026

View reviewed changes

KevinHuSh and others added 7 commits October 12, 2026 15:40

merge: bring feat/file_parsers up to date with upstream main

3501fd7

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>

Merge branch 'main' into feat/file_parsers

6c7050c

0xKT reviewed Oct 12, 2026

View reviewed changes

Comment thread tests/conftest.py

gloryfromca reviewed Oct 12, 2026

View reviewed changes

KevinHuSh and others added 3 commits October 12, 2026 16:57

Merge remote-tracking branch 'origin/feat/file_parsers' into feat/fil…

6a94e0b

…e_parsers

gloryfromca reviewed Oct 12, 2026

View reviewed changes

0xKT approved these changes Oct 12, 2026

View reviewed changes

0xKT merged commit 82c76a8 into EverMind-AI:main Oct 12, 2026
31 checks passed

Handsome-wzw mentioned this pull request Oct 15, 2026

test(tests): put the EverOS role binding back after every test #964

Merged

5 of 12 tasks

This was referenced Oct 20, 2026

chore: bump version to 0.2.5 #1025

Merged

docs: refresh the release notes template for resources and plugin wheels #1026

Merged

0xKT mentioned this pull request Oct 20, 2026

fix(*): fetch the knowledge reader's models on upgrade and on native Windows installs #1027

Open

Conversation

KevinHuSh commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type

Verification

Risk

Related Issues

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

0xKT left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

gloryfromca left a comment

Choose a reason for hiding this comment

Uh oh!

0xKT commented Oct 12, 2026

Uh oh!

Uh oh!

0xKT commented Oct 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

KevinHuSh commented Sep 18, 2026 •

edited

Loading