A layout-aware pipeline for extracting structured geological data from PDF publications. Converts unstructured geoscience reports into validated JSON using Docling for PDF parsing, Instructor + Azure OpenAI for structured LLM extraction, and optional normalization against the Mindat mineral database and GeoSciML controlled vocabularies.
- Layout-aware PDF parsing — Docling preserves headers, tables, and document structure as semantic Markdown, replacing brittle OCR approaches
- Multi-candidate LLM extraction — generates 3 extraction candidates per document, then uses a second LLM call to select the best result
- Pluggable schemas — built-in USGS Bulletin 1693 deposit model schema; add new schemas by dropping a single Python file in
schemas/custom/ - Mindat normalization — maps extracted mineral and rock names to canonical Mindat entries
- GeoSciML vocabulary matching — aligns geological terms to IUGS/CGI controlled vocabularies (age, lithology, tectonic setting, etc.)
- MCP server architecture — tools exposed via FastMCP for integration with AI assistants or other MCP clients
- Observable — full LLM call tracing via Arize Phoenix
- Configurable — all temperatures, retries, timeouts, and directory paths are env-overridable via
.env
PDF
└─► Docling (ingestion) → Markdown
└─► Instructor (extraction) → Pydantic JSON (3 candidates → best selected)
├─► Mindat API → normalized mineral/rock names
└─► GeoSciML vocab → controlled-vocabulary term alignment
└─► data/outputs/<doc>/
The pipeline runs as a sequential async workflow (client/workflow.py). Tools are also exposed through a single FastMCP server (servers/geo_server.py) for MCP client integration.
- Python ≥ 3.12
- Azure OpenAI deployment (GPT-4o recommended)
- Mindat API key (optional, for mineral normalization)
- Arize Phoenix endpoint (optional, for tracing)
git clone https://github.com/<your-org>/ageoke_v2.git
cd ageoke_v2
uv venv && uv sync
source .venv/bin/activateCopy .env.example to .env and fill in your credentials:
AZURE_DEPLOYMENT_NAME=gpt-4o
AZURE_OPENAI_API_VERSION=2024-08-01-preview
AZURE_OPENAI_API_ENDPOINT=https://<your-resource>.openai.azure.com/
AZURE_OPENAI_API_KEY=<your-key>
# Optional
MINDAT_API_KEY=<your-key>
PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006/v1/tracesEdit the parameters at the top of main.py, then run:
# main.py — edit these before running
INPUT_DIR = "data/inputs/my_papers" # folder containing PDFs
SCHEMA = None # None = use default (usgs_deposit_model)
SKIP_MINDAT = False
SKIP_GEOSCIML = False
START_PAGE = None # 0-based, or None for full document
END_PAGE = Nonepython main.pyOutputs are written to data/outputs/<input_folder_name>_output/ — one JSON file per PDF.
# Full pipeline with default schema (USGS deposit model)
python main.py extract data/inputs/usgs_bulletin.pdf
# Skip Mindat and GeoSciML steps
python main.py extract data/inputs/report.pdf --no-mindat --no-geosciml
# Use a custom schema, process specific pages only
python main.py extract data/inputs/bgs_log.pdf --schema bgs_borehole --start-page 0 --end-page 5python main.py extract-dir data/inputs/my_papers/ --schema usgs_deposit_modelOutputs are written to data/outputs/my_papers_output/.
python main.py ingest data/inputs/report.pdfpython main.py list-schemasFinal results are written to data/outputs/<folder_name>_output/ — one JSON file per PDF. Intermediate files from each pipeline stage are stored under data/outputs/intermediate/:
| Location | Contents |
|---|---|
data/outputs/intermediate/ingested/ |
Docling-parsed Markdown |
data/outputs/intermediate/extracted/<stem>/ |
3 LLM candidates + best-selected JSON |
data/outputs/intermediate/mindat_matched/ |
Extraction with normalized mineral/rock names |
data/outputs/<folder>_output/ |
Final output with GeoSciML vocabulary alignments |
- Create
schemas/custom/my_source.pywith aSchemaModelclass extendingBaseExtractionSchema - Implement
schema_metadata()— declare which fields to send to Mindat and GeoSciML matching - Run
python main.py list-schemasto confirm it is auto-discovered
See schemas/usgs_deposit_model.py for a complete example and schemas/custom/README.md for field conventions.
| Schema name | Description |
|---|---|
usgs_deposit_model |
USGS Bulletin 1693 descriptive mineral deposit models (20 fields) |
generic_geology |
Minimal fallback schema for unstructured geological text |
All settings live in utils/config.py (Pydantic BaseSettings) and can be overridden via environment variables or .env:
| Variable | Default | Description |
|---|---|---|
DEFAULT_SCHEMA_NAME |
usgs_deposit_model |
Schema used when --schema is not specified |
NUM_EXTRACTION_CANDIDATES |
3 |
LLM candidates generated per document |
EXTRACTION_TEMPERATURE |
0.2 |
Temperature for extraction calls |
MAX_RETRIES |
3 |
Retries per LLM call |
LLM_TIMEOUT_SECONDS |
120 |
Per-call timeout |
FILE_PROCESSING_TIMEOUT_SECONDS |
600 |
Per-file pipeline timeout |
The FastMCP server exposes pipeline tools for integration with MCP-compatible AI assistants:
python servers/geo_server.py# Verify schema registry
.venv/bin/python -c "from utils.schema_registry import list_available_schemas; print(list_available_schemas())"
# Run tests
pytestSee LICENSE.