Repository navigation
fix(*): fetch the knowledge reader's models on upgrade and on native Windows installs #1027
Description
Summary
raven resources (added in #489) downloads the deepdoc models and a word dictionary, about 110 MB, into the installed package. Only install.sh, the Makefile and the Dockerfile run it. install.ps1, raven upgrade and the update row at the bottom of the page's sidebar never do, so a native Windows install made with install.ps1, and an install upgraded from 0.2.4 or earlier with raven upgrade or the update row, starts without them until someone runs raven resources by hand. Rerunning install.sh, which also upgrades, does fetch them.
Nothing at run time tells the user. The PDF reader only checks whether the weights are present and, when they are not, reads the file geometrically: no layout model and no table-structure model (a table is found only where its lines are drawn), and a page with no text layer goes to the configured vision model, or is reported unread when there is none.
@KevinHuSh, this came in with #489. Could you take it?
Steps to reproduce
- Once 0.2.5 is published, upgrade a 0.2.4 install with
raven upgrade(or install 0.2.5 on native Windows withinstall.ps1). - With Raven's own interpreter, run
python -c "from raven.knowledge.parser.deepdoc import _pipeline; print(_pipeline.usable(), _pipeline.readable())": both areFalse. - With no vision model configured, add a scanned PDF to a knowledge base: the document's warning says its pages have no text layer and were not read because no vision model is configured. Nothing says the recognizer is missing or names
raven resources.
Expected behavior
Every supported install path ends with the models in place unless the user opted out (RAVEN_MINIMAL=1 for install.sh, RAVEN_RESOURCES=0 for the Docker build), and a Raven running without them says so and names raven resources.
Actual behavior
install.shruns the tool environment's python with-m raven.resources --optionalthroughinstall_resources, skipped underRAVEN_MINIMAL.install.ps1has no such step, and nothing underraven/updates/callsraven.resources, so neitherraven upgradenor the update row (both go throughraven/updates/upgrade.py) fetches them.raven/knowledge/parser/deepdoc/_pipeline.py:usable()andreadable()only test that the weight files exist. The knowledge manager buildsPdfParser()with its defaults (raven/knowledge/_manager.py), and on that path (layout=None)raven/knowledge/parser/pdf_parser.pyfalls back to the geometric reader without a note.- The two notes
pdf_parser.pyrecords when the layout model or the recognizer is asked for by name (layout=True/ocr=True) and missing, and its module docstring, still saymake fetch-resources, which a wheel install cannot run. The product does not reach those notes today, but the dictionary and model loaders already nameraven resources(raven/core/tokenizer.py,raven/knowledge/parser/deepdoc/_onnx.py). - Nothing at run time reads the word dictionary yet (only
raven/resources.pyimportsraven.core.tokenizer, to warm it), so the deepdoc models are the half whose absence changes behavior.
Suggested fix:
install.ps1: runraven\Scripts\python.exefrom the tool directory with-m raven.resources --optionalafter the install, skipped underRAVEN_MINIMAL, asinstall.shdoes.raven/updates/upgrade.py: in the embedded helper (_UPGRADE_HELPER_SOURCE), afterinstall()succeeds and beforerestart(), run the new tool environment's python with-m raven.resources --optional. Bothraven upgradeand the update row hand the install to the helper of the Raven being replaced, so this only covers upgrades started from the release that carries it; an install upgrading into that release from 0.2.4 or 0.2.5 still arrives without the files.- That is why a run-time path matters too: the new Raven should notice the missing models (at start, in
raven doctor, or when a PDF is indexed) and either fetch them in the background with--optionalor say so and nameraven resources. pdf_parser.py: change the twomake fetch-resourcesnotes and the docstring toraven resources.
The fetch skips files already present at their published size, so repeating it is cheap. The 0.2.5 release notes tell upgraders to run raven resources once; that line can go once this ships.
Environment
Raven 0.2.5 (tag v0.2.5). Found while writing the 0.2.5 release notes; not reproduced on a Windows machine.
Activity