Pull spatial data from Overpass, STAC, ArcGIS REST, OGC API - Features and bulk file downloads into one normalized staging layout — resumably, with checksums and provenance for every part.
Every spatial data source speaks a different dialect. Overpass returns OSM elements
that are not geometries. ArcGIS paginates with resultOffset and silently caps your
page size. STAC hands you items whose real payload is somewhere else entirely. Open
data portals just publish a 400 MB zipped shapefile behind a redirect.
So every ingestion project grows the same pile of one-off scripts, and each one reinvents the same four things badly:
- Resuming. A harvest dies at page 400 of 500. Without a manifest, the only safe move is starting over.
- Retrying. Retrying everything hammers the server; retrying nothing means one blip loses an hour of work.
- Rate limiting. Public endpoints ban clients that ignore fair-use limits, and Overpass is stricter than most.
- Knowing what you got. Six months later: which endpoint answered, what exactly was asked, and is this file still the one that arrived?
geo-ingest solves those four problems once, in the engine, so a connector is just an
adapter that yields pages. Everything lands as newline-delimited GeoJSON with an
inferred schema sidecar, so downstream tools see one shape regardless of where the
data came from.
This project is not published to a package index — install it from a clone.
git clone https://github.com/geospatial-etl/geo-ingest.git
cd geo-ingest
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e . # or: pip install -r requirements.txtPython 3.11 or newer is required. To run the test suite, install the dev extras
instead: pip install -e ".[dev]".
Verify the install:
geo-ingest version
python -m geo_ingest version # equivalent, no PATH neededDefine a source entirely on the command line:
geo-ingest run berlin-cafes \
--connector overpass \
-o query='node[amenity=cafe]({{bbox}});' \
--bbox 13.35,52.48,13.45,52.54 \
--rate-limit 0.5 \
-vOr put it in a config file and run it by name:
# sources.yml
staging_dir: staging
sources:
berlin-cafes:
connector: overpass
description: Cafes in central Berlin
bbox: [13.35, 52.48, 13.45, 52.54]
rate_limit:
requests_per_second: 0.5
options:
query: 'node[amenity=cafe]({{bbox}});'geo-ingest run berlin-cafes --config sources.yml$ geo-ingest run berlin-cafes --config examples/sources.yml -v
╭────────────── geo-ingest ──────────────╮
│ berlin-cafes via overpass │
│ staging: staging │
╰────────────────────────────────────────╯
[14:25:31] INFO submitting Overpass query to https://overpass-api.de/api/interpreter
[14:25:36] INFO staged part-00000.ndjson (487 features, 214803 bytes)
⠹ done ━━━━━━━━━━━━━━━━━━━━━━━━━━ 487 features 0:00:05
run id 20260719T142531Z-berlin-cafes
features 487
parts 1
duration 5.2s
directory staging/berlin-cafes/20260719T142531Z-berlin-cafesWhat landed on disk:
$ ls staging/berlin-cafes/20260719T142531Z-berlin-cafes/
manifest.json part-00000.ndjson provenance.json schema.json
$ head -c 220 staging/berlin-cafes/20260719T142531Z-berlin-cafes/part-00000.ndjson
{"type":"Feature","geometry":{"type":"Point","coordinates":[13.3891,52.5163]},"properties":{"amenity":"cafe","name":"Cafe Einstein","osm_type":"node","osm_id":123456789},"id":"node/123456789"}The inferred schema tells you what the properties actually are, without a second pass:
$ jq '.properties | to_entries | .[:3]' .../schema.json
[
{ "key": "amenity", "value": { "type": "string", "nullable": false, "count": 487, "null_count": 0 } },
{ "key": "cuisine", "value": { "type": "string", "nullable": true, "count": 231, "null_count": 256 } },
{ "key": "osm_id", "value": { "type": "integer","nullable": false, "count": 487, "null_count": 0 } }
]Verify it later, then convert for analysis:
$ geo-ingest verify 20260719T142531Z-berlin-cafes
ok 1 part(s), 487 feature(s) verified
$ geo-ingest to-parquet 20260719T142531Z-berlin-cafes
wrote staging/berlin-cafes/.../features.parquet (0.09 MiB)All commands accept --config/-c, --staging-dir/-s, --verbose/-v (repeatable) and
--quiet/-q.
Ingest a source. The source comes from the config file, or is defined inline with
--connector plus --option.
| Option | Description |
|---|---|
--connector |
Connector to use when defining a source via flags. |
-o, --option key=value |
Connector option; repeatable. Values are parsed as JSON when possible. |
--bbox min_lon,min_lat,max_lon,max_lat |
Spatial filter in EPSG:4326. |
--datetime |
RFC 3339 instant or interval. |
--max-features |
Stop after staging this many features. |
--page-size |
Preferred features per page; connectors clamp to server limits. |
--rate-limit |
Maximum requests per second. |
--max-attempts |
Retry attempts per request. |
--timeout |
Per-request timeout in seconds. |
--run-id |
Explicit run identifier instead of a generated one. |
--resume |
Continue an existing run, skipping completed parts. |
--to-parquet |
Convert to GeoParquet when the run finishes. |
List configured sources and every available connector. --json for machine output.
Show a source's resolved settings, or a connector's summary. --schema includes the
connector's full options schema; --json emits it raw.
Show a run's progress, part count, feature count and errors. --parts lists every
part with its checksum; --source narrows the search; --json for machine output.
List staged runs, newest first. --source, --limit, --json.
Continue an interrupted run. Requires --config pointing at the file that defined the
source, since the run needs its original settings.
Re-check every staged part against its recorded SHA-256. Exits non-zero on any
mismatch, so it works as a pipeline gate. --quick checks sizes only.
Convert a completed run to a single GeoParquet file. --output/-O, --compression
(snappy, gzip, zstd, brotli), --crs, --overwrite.
staging_dir: staging # root for staged runs
defaults: # merged underneath every source
timeout: 60
retry:
max_attempts: 5
initial_backoff: 1.0
max_backoff: 60.0
multiplier: 2.0
jitter: 0.2
respect_retry_after: true
sources:
my-source:
connector: ogc_features # required
description: Free text carried into provenance
bbox: [min_lon, min_lat, max_lon, max_lat]
datetime: "2026-01-01/2026-06-30"
crs: EPSG:4326
max_features: 50000
page_size: 1000
timeout: 120
headers:
Authorization: "Bearer ${MY_API_TOKEN}"
rate_limit:
requests_per_second: 2.0
burst: 5
retry:
max_attempts: 8
options: # connector-specific; validated per connector
url: https://example.org/ogcapi
collection: buildings${VAR} and ${VAR:-default} are expanded from the environment throughout the file,
which keeps tokens out of version control. Unknown keys are rejected rather than
ignored, so a typo fails loudly instead of silently doing nothing.
Run geo-ingest describe <connector> --schema for the full validated option set.
POSTs an Overpass QL query. {{bbox}} expands to the source bbox in Overpass's
south,west,north,east order; {{area}} expands to an area id resolved from
area_name. The [out:json] header and trailing out statement are added
automatically. Elements are converted to geometries: nodes to Points, closed ways to
Polygons when their tags imply an area, and multipolygon relations stitched into rings.
Key options: query (required), url, area_name, timeout, out_mode,
include_metadata. Overpass answers a query in one response, so this connector has no
pagination and cannot resume mid-query. Set a low requests_per_second.
Searches a STAC API with pystac-client, filtered by collection, bbox and datetime.
Items are already GeoJSON, so normalization lifts assets, collection and bbox
into properties where tabular tools can reach them. Asset downloading is opt-in via
download_assets, since catalogs routinely reference gigabyte-scale rasters.
Key options: url (required), collections, query, filter, sortby, ids,
limit, max_items, download_assets, asset_keys, flatten_assets.
Paginates a FeatureServer or MapServer layer's query endpoint with resultOffset /
resultRecordCount. Reads the layer metadata first to honour maxRecordCount, sorts
by the object-id field so offset paging is stable, and trusts exceededTransferLimit
where the server reports it. ESRI JSON geometries are converted to GeoJSON, including
ring grouping for multipart polygons.
Key options: url (required, ending in the layer index), where, out_fields,
page_size, order_by_fields, out_sr, return_geometry, token, extra_params.
Downloads GeoJSON, zipped shapefiles, GeoPackages, NDJSON or CSV. Verifies
Content-Length against bytes written, and optionally an expected SHA-256 or MD5.
Shapefiles and GeoPackages are read with geopandas and reprojected to the source CRS;
CSV coordinate columns are auto-detected when not named explicitly.
Key options: url or files[], format, sha256, md5, size, layer,
x_field, y_field, keep_downloads. Each file is one page, so multi-file sources
resume at file granularity.
Fetches /collections/{id}/items and follows the server's own rel="next" link
rather than synthesising offsets, which is both what the spec requires and more robust
against token-based paging. The next-link URL is stored as the resume cursor.
Key options: url (required, the API landing page), collection (required), limit,
max_pages, extra_params, fetch_collection_metadata.
staging/<source>/<run_id>/
manifest.json progress and per-part checksums
provenance.json what was requested, when, and what came back
schema.json inferred property names and types
part-00000.ndjson newline-delimited GeoJSON features
part-00001.ndjson
downloads/ raw files (http connector)
assets/ downloaded assets (stac connector)
Every part is newline-delimited GeoJSON: one Feature per line, always with type,
geometry and properties. NDJSON is the staging format because it is append-friendly
and interrupt-safe — you can add a line without rewriting the file. It is deliberately
not the analysis format; that is what to-parquet is for.
Each page is written to part-NNNNN.ndjson.part, fsynced, then atomically renamed onto
its final name with os.replace. Only then is the part recorded in the manifest, which
is itself saved atomically.
This ordering is what makes a kill -9 safe. The manifest never claims a part that is
not fully on disk. If the process dies after the rename but before the manifest save,
the part exists but is unrecorded — and --resume simply refetches that one page,
overwriting it identically. The worst case is one redundant page, never a truncated
file mistaken for a complete one. Stray .part files are swept at the start of every
run.
Resumption uses opaque cursors. Each page carries a cursor meaning "the position
after this page" — an ArcGIS offset, an OGC next-link URL, a STAC item count. On
resume the engine hands the last completed part's cursor back to the connector, so a
run interrupted at page 400 refetches nothing. Connectors that genuinely cannot resume
mid-stream (Overpass) declare resumable = False.
Retries are scoped to conditions a later attempt could plausibly fix: 429, 5xx (except 501 and 505, which are permanent statements about the server), timeouts, and connection errors. A 4xx other than 429 means the request itself is wrong, and is never retried.
Delay for attempt n is initial_backoff * multiplier^(n-1), capped at max_backoff,
then jittered by ±jitter to avoid synchronised retry storms. A Retry-After header
wins over the computed delay — but is still capped, so a hostile or buggy server cannot
stall a run indefinitely.
Rate limiting is a token bucket, shared per source, so concurrent runs against the same endpoint honour one combined budget rather than each getting a full allowance.
provenance.json records the source config (with credentials redacted), every request
URL and query issued — including POST bodies for Overpass, so a query can be replayed
verbatim — UTC ISO 8601 timestamps, the tool version, per-part SHA-256 and feature
counts, and the CRS as declared by the source. On resume, earlier requests are
preserved rather than overwritten, so the document reflects everything the run ever did.
Property types are inferred as parts are staged, in one pass and bounded memory. Type
resolution is widening rather than first-wins: a field seen as integer then number
resolves to number, while genuinely mixed types resolve to string with every
observed type retained. Fields absent from some features count as nulls, so
null_count reflects true sparsity.
from geo_ingest import IngestEngine, SourceConfig
config = SourceConfig(
name="buildings",
connector="ogc_features",
options={"url": "https://demo.pygeoapi.io/master", "collection": "lakes"},
max_features=500,
)
result = IngestEngine(staging_dir="staging").run(config)
print(result.feature_count, result.directory)
for path in result.part_paths:
print(path)Library code logs through the standard logging module and never prints, so embedding
geo-ingest produces no stray output.
The examples/ directory contains a config covering several real public endpoints,
plus runnable walkthrough scripts:
examples/sources.yml— annotated source definitions for all five connectors.examples/quickstart.py— run a source and inspect what landed.examples/resume_demo.py— interrupt a run and resume it, showing the manifest.examples/library_usage.py— the engine as a library, including custom progress.
- CRS. Staged output is always GeoJSON's EPSG:4326. The
httpconnector reprojects shapefiles and GeoPackages; other connectors request WGS 84 from the server and trust the answer. A file with no declared CRS is passed through and flagged, not guessed at. - Overpass cannot resume. A query is one response, so an interrupted Overpass run restarts it. Split large areas into several bbox-tiled sources.
- STAC resume replays the search. Results are ordered but not offset-addressable, so resuming re-runs the search and skips already-seen items. Downloaded assets are genuinely skipped, which is the expensive part.
- Single-threaded. Runs are sequential by design; the shared rate limiter exists so that concurrent processes stay within budget, not to enable parallel fetching.
- Memory. A page is held in memory before being written. For very wide features,
lower
page_size. - GeoParquet conversion loads the full run into a GeoDataFrame. Runs much larger than available RAM should be converted per-part with your own script.
pip install -e ".[dev]"
pytestTests mock all HTTP with respx and httpx's MockTransport — the suite never touches
the network.
- Mastering geospatial data ingestion in Python — the broader patterns this tool implements.
- Fetching OSM data via the Overpass API — query design and fair-use practice.
- Syncing STAC catalogs with pystac-client — search, paging and asset handling.
MIT — see LICENSE. Copyright (c) 2026 Geospatial ETL.
Maintained by Geospatial ETL.