Skip to content

fix(*): fetch the knowledge reader's models on upgrade and on native Windows installs #1027

Open

Description

@0xKT

Summary

raven resources (added in #489) downloads the deepdoc models and a word dictionary, about 110 MB, into the installed package. Only install.sh, the Makefile and the Dockerfile run it. install.ps1, raven upgrade and the update row at the bottom of the page's sidebar never do, so a native Windows install made with install.ps1, and an install upgraded from 0.2.4 or earlier with raven upgrade or the update row, starts without them until someone runs raven resources by hand. Rerunning install.sh, which also upgrades, does fetch them.

Nothing at run time tells the user. The PDF reader only checks whether the weights are present and, when they are not, reads the file geometrically: no layout model and no table-structure model (a table is found only where its lines are drawn), and a page with no text layer goes to the configured vision model, or is reported unread when there is none.

@KevinHuSh, this came in with #489. Could you take it?

Steps to reproduce

  1. Once 0.2.5 is published, upgrade a 0.2.4 install with raven upgrade (or install 0.2.5 on native Windows with install.ps1).
  2. With Raven's own interpreter, run python -c "from raven.knowledge.parser.deepdoc import _pipeline; print(_pipeline.usable(), _pipeline.readable())": both are False.
  3. With no vision model configured, add a scanned PDF to a knowledge base: the document's warning says its pages have no text layer and were not read because no vision model is configured. Nothing says the recognizer is missing or names raven resources.

Expected behavior

Every supported install path ends with the models in place unless the user opted out (RAVEN_MINIMAL=1 for install.sh, RAVEN_RESOURCES=0 for the Docker build), and a Raven running without them says so and names raven resources.

Actual behavior

  • install.sh runs the tool environment's python with -m raven.resources --optional through install_resources, skipped under RAVEN_MINIMAL. install.ps1 has no such step, and nothing under raven/updates/ calls raven.resources, so neither raven upgrade nor the update row (both go through raven/updates/upgrade.py) fetches them.
  • raven/knowledge/parser/deepdoc/_pipeline.py: usable() and readable() only test that the weight files exist. The knowledge manager builds PdfParser() with its defaults (raven/knowledge/_manager.py), and on that path (layout=None) raven/knowledge/parser/pdf_parser.py falls back to the geometric reader without a note.
  • The two notes pdf_parser.py records when the layout model or the recognizer is asked for by name (layout=True / ocr=True) and missing, and its module docstring, still say make fetch-resources, which a wheel install cannot run. The product does not reach those notes today, but the dictionary and model loaders already name raven resources (raven/core/tokenizer.py, raven/knowledge/parser/deepdoc/_onnx.py).
  • Nothing at run time reads the word dictionary yet (only raven/resources.py imports raven.core.tokenizer, to warm it), so the deepdoc models are the half whose absence changes behavior.

Suggested fix:

  • install.ps1: run raven\Scripts\python.exe from the tool directory with -m raven.resources --optional after the install, skipped under RAVEN_MINIMAL, as install.sh does.
  • raven/updates/upgrade.py: in the embedded helper (_UPGRADE_HELPER_SOURCE), after install() succeeds and before restart(), run the new tool environment's python with -m raven.resources --optional. Both raven upgrade and the update row hand the install to the helper of the Raven being replaced, so this only covers upgrades started from the release that carries it; an install upgrading into that release from 0.2.4 or 0.2.5 still arrives without the files.
  • That is why a run-time path matters too: the new Raven should notice the missing models (at start, in raven doctor, or when a PDF is indexed) and either fetch them in the background with --optional or say so and name raven resources.
  • pdf_parser.py: change the two make fetch-resources notes and the docstring to raven resources.

The fetch skips files already present at their published size, so repeating it is cheap. The 0.2.5 release notes tell upgraders to run raven resources once; that line can go once this ships.

Environment

Raven 0.2.5 (tag v0.2.5). Found while writing the 0.2.5 release notes; not reproduced on a Windows machine.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions