Tübingen Search Engine

A custom search engine with link crawling and an indexing pipeline and client frontend.

Components

This repo is a uv workspace with separate components:

• crawl/ — link crawler that fetches and stores pages
• search/ — BM25 index, CLI, and FastAPI search API
• verdict-ml/ — PageVerdict and LinkVerdict runtime; training and model releases live in labeling-lab
• client/ — React frontend for the search API

Quickstart

Install dependencies:

uv sync

Choose a data source:

Latest snapshot

uv run data-fetch

Crawl locally

uv run crawl
uv run index
uv run embed

Search:

uv run search -q "tübingen attractions"

Web UI

Start the API and client in separate terminals:

uv run uvicorn tuebingen_search.api:app

cd client
npm install  # once
npm run dev

Vite prints the local URL for the client.

For the crawl report, HTTP API, web client, and all options, see the component READMEs linked above (e.g. search/).