Skip to content

Repair Dutch data.overheid CKAN profile - #17

Merged
GirelliG-it merged 1 commit into
mainfrom
refactor/06-repair-dutch-profile
Sep 2, 2026
Merged

GirelliG-it merged 1 commit into
mainfrom
refactor/06-repair-dutch-profile

Conversation

@GirelliG-it

Copy link
Copy Markdown
Owner

Summary

  • Add closed CKAN search modes and preserve their enum types through profile contract, configuration, resolution, and tool construction.
  • Retain CBS StatLine OData v3 and add the official data.overheid.nl CKAN v3 route at /data/api/3; neither catalog requires an API key.
  • Use bounded local resource-metadata filtering for data.overheid.nl instead of its incompatible literal CKAN CSV filter.
  • Reject unsupported OpenDataSoft configuration and keep the Dutch profile manual_only.

Safety and behavior

  • Local-metadata search inspects at most twice the requested result count and admits only explicit, internally consistent plain-CSV metadata with a nonblank URL.
  • The existing loader policy remains authoritative before guarded retrieval or DuckDB inspection.
  • Query-driven URLs without .csv path evidence remain report-only.
  • Existing server-literal CKAN selection behavior and shared run-deadline accounting are preserved.
  • Persistent loading still requires the existing explicit authorization boundary.

Validation

  • ruff check .
  • ruff format --check .
  • pytest -p no:cacheprovider -q -m "not integration" — 1,107 passed
  • All pre-commit hooks passed.

A bounded compatibility audit made three metadata requests and examined 18 package observations, 17 unique packages, and 57 resources. It found five packages with loader-compatible CSV resources and no ordering traps within the audited sample.

A controlled RDW CSV retrieval completed within a 32 MiB ceiling with zero redirects and temporary-file cleanup. Deterministic inspection verified 6,650 non-empty rows across seven columns.

Remaining limitations

  • The Dutch profile remains manual_only.
  • Query-only CSV URLs remain report-only even when catalog metadata declares CSV.
  • The compatibility audit is bounded rather than catalog-wide.
  • Persistent DuckDB loading was not performed.

Add closed CKAN search modes and bounded local resource-metadata filtering for data.overheid.nl. Keep query-only resources report-only and preserve existing server-literal behavior.

Add contract, routing, authorization-boundary, and compatibility coverage while retaining manual-only status.
@GirelliG-it
GirelliG-it merged commit 65291ad into main Sep 2, 2026
2 checks passed
@GirelliG-it
GirelliG-it deleted the refactor/06-repair-dutch-profile branch September 2, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant