opencourts-ckan

The metadata, pipelines, extensions for the CKAN portion of open courts

Documentation

• Architecture — catalogue APIs (CKAN Action API, DCAT), system design, and local verification (including bin/test)
• Troubleshooting — common local dev problems (ports, WSL2, macOS, volumes)

Table of contents

• Documentation
• Development
  • Prerequisites
  • Setup
  • Common commands
  • Seeding Data
  • Running tests
  • Teardown
  • Remote debugging
  • Running with HTTPS
• Tracking ckan-docker upstream

Development

This project is forked from ckan/ckan-docker. The upstream development mode guide covers Docker internals, bin/ helper scripts, extensions, HTTPS, and debugging — use it when you need depth beyond this quickstart.

Problems starting the stack? See Troubleshooting.

Prerequisites

You need Docker and uv. All bin/ scripts require a Unix-like shell (bash). Run commands from a terminal on Linux, on macOS, or inside WSL2 on Windows.

Linux

• Install Docker Engine (Compose plugin included on current installs).
• Install uv.

macOS

• Install Docker Desktop for Mac.
• Install uv.
• Apple Silicon Macs (M1 and later): CKAN images are amd64 only — set export DOCKER_DEFAULT_PLATFORM=linux/amd64 before building. Intel Macs run the images natively and do not need this. See Troubleshooting → macOS.

Windows

Windows requires WSL2 and Docker Desktop. Follow the steps below to get your environment set up with the repo using both.

NOTE: If you run into issues, see Troubleshooting → Windows (WSL2).

1. Install WSL2 with Ubuntu:

   wsl --install -d Ubuntu

   Restart the machine if prompted.

2. Install Docker Desktop for Windows.

3. In Docker Desktop Settings:

   • General: enable Use the WSL 2 based engine
   • Resources → WSL Integration: enable integration for Ubuntu
   • Keep Docker Desktop running before continuing.

4. (Recommended) Install the Visual Studio Code WSL extension in VS Code (or a VS Code–based editor).

5. In PowerShell, clone the repo and cd into it:

   git clone https://github.com/opencourtsfyi/opencourts-ckan.git
   cd opencourts-ckan

6. From that same PowerShell window, open the project in WSL (for example, if using VS Code):

   wsl code .

7. In the editor, open Terminal → New Terminal and verify Docker:

   docker version

8. Install uv and ensure it is on your PATH. From the repo root:

   uv venv --python ">=3.12"

Notes

• Run all common commands from the WSL terminal only. PowerShell and CMD are not supported for this repo’s bin/ scripts.
• See Troubleshooting → Windows (WSL2) for slow Docker builds on /mnt/c/... and other WSL2 issues.

Setup

NOTE: this setup guide assumes you are running the solution in ckan-docker development mode. (See updating the environment file for development mode for more information).

Steps

Step	Description	Command/Action
1	Copy the environment template	cp .env.example .env
2	Build the Docker containers	bin/compose build
3	Start Docker Compose	bin/compose up
4	Wait until CKAN is healthy	curl -s http://localhost:5000/api/action/status_show returns "success": true
5	Visit the local CKAN site	Open http://localhost:5000/ in a browser
6	Log in with the default admin credentials	ckan_admin / test1234
7	(Optional) Explore catalogue APIs	See docs/ARCHITECTURE.md — seed test data first (Seeding Data)

NOTE: to run locally using HTTPS, see Running with HTTPS. Keep CKAN_SITE_URL in .env aligned with your HTTP/HTTPS choice; it affects catalogue and download URLs (see docs/ARCHITECTURE.md).

Common commands

Task	Command
Build containers	bin/compose build
Start stack	bin/compose up
Stop stack	bin/compose down
Stop and remove volumes (reset local data)	bin/compose down -v — see Troubleshooting
CKAN CLI	./bin/ckan …
Run tests	bin/test unit-setup, bin/test unit, bin/test smoke, bin/test catalogue, bin/test integration, bin/test all — see Running tests
Seed test data	uv run scripts/seed.py "$CKAN_API_TOKEN"
Stack smoke test	bin/test smoke
Catalogue smoke test	bin/test catalogue (also run by bin/test integration)
Check CKAN is up	curl -s http://localhost:5000/api/action/status_show

Notes

If you change CKAN_PORT_HOST in .env, update CKAN_SITE_URL to match and recreate containers. Port conflicts: Troubleshooting → Port already in use. The bin/compose and bin/ckan wrappers are described in ckan-docker development mode.

Seeding Data

The development environment can be seeded with test data. A small sample of the Datasets - Measures for Justice for NC has been added to this project. To set this up:

# create an API token
export CKAN_API_TOKEN="$(./bin/ckan user token add ckan_admin my-test-token | tail -1 | tr -d '[:space:]')"

# seed your database
uv run scripts/seed.py "$CKAN_API_TOKEN"

# Desired console output:
# Seeding CKAN at http://localhost:5000...
# Action 'organization_create' reported conflict for 'test-org' (already exists).
# Action 'package_create' reported conflict for 'test-package' (already exists).
# Action 'resource_create' succeeded for 'Test Data Measures For Justice: NC 2013'
# Action 'resource_create' succeeded for 'Test Data Measures For Justice: NC locations'
# Action 'resource_create' succeeded for 'Test Data Measures For Justice: NC measures'
# Action 'resource_create' succeeded for 'Test Data Measures For Justice: NC filters'

Once you've executed this script you should see a test organization, and datasets on your local development site. The seeded test-org and test-package datasets are used in the catalogue API examples in docs/ARCHITECTURE.md.

Running tests

bin/test is the documented entry point for local unit, integration, and smoke tests. Run commands from the repo root on Linux, macOS, or WSL2.

Prerequisites: dev stack running and healthy (bin/compose up -d; wait for status_show to succeed).

On Apple Silicon Macs, set export DOCKER_DEFAULT_PLATFORM=linux/amd64 before building or starting the stack (see Prerequisites → macOS).

# Unit tests (extensions in src/; no seed required)
bin/test unit-setup   # once after adding or updating an extension
bin/test unit

# Smoke — stack only: portal, DB, storage (no seed required)
bin/test smoke

# Integration — stack + catalogue (catalogue requires seed)
export CKAN_API_TOKEN="$(./bin/ckan user token add ckan_admin my-test-token | tail -1 | tr -d '[:space:]')"
uv run scripts/seed.py "$CKAN_API_TOKEN"
bin/test integration

# Unit tests, then integration
bin/test all

Command	What it runs	Seed required?
bin/test unit-setup	install extension test deps with uv inside ckan-dev	No
bin/test unit	pytest for each src/ckanext-* extension inside ckan-dev; no-op when src/ is empty	No
bin/test smoke	Portal, PostgreSQL, and CKAN storage	No
bin/test catalogue	Anonymous catalogue read access (FR-5)	Yes
bin/test integration	Stack checks, then catalogue checks	Yes
bin/test all	Unit tests, then integration	Yes (for integration)

bin/test all runs unit then integration. Integration already includes stack checks (smoke), so you do not need a separate bin/test smoke when running all.

Stack checks and catalogue checks live in separate scripts because stack health requires no seed data, while catalogue checks require seeded test-org / test-package data (FR-5). See docs/ARCHITECTURE.md § Local verification workflow.

bin/test unit is a no-op when there are no extensions in src/. When extensions exist, run bin/test unit-setup once (or after dependency changes), then bin/test unit. Setup and tests run as the ckan user inside ckan-dev using a dedicated virtualenv (/srv/app/.venv-unit) with uv; extensions do not need to be listed in CKAN__PLUGINS for tests to pass. To use an extension in the running CKAN instance, run bin/install_src, add its plugin name to CKAN__PLUGINS in .env, and restart CKAN (bin/restart).

Wait until CKAN is healthy (curl http://localhost:5000/api/action/status_show) before running tests, especially right after bin/compose up. If checks fail, see Troubleshooting → CKAN not ready yet.

CI: Non-draft pull requests to main, pushes to main, and manual workflow_dispatch runs execute the Catalog smoke test workflow. Draft PRs skip the smoke test to save CI time; mark ready for review or run the workflow manually to trigger it.

Teardown

To remove all Docker resources you have created (for example: containers, volumes):

bin/compose down -v

When to reset volumes and what gets removed: see Troubleshooting → Resetting volumes.

Remote debugging

1. Set USE_DEBUGPY_FOR_DEV=true in .env (see ckan-docker — Remote debugging).
2. Install extensions and start the stack:

bin/install_src
bin/compose up

3. Attach your editor — follow the ckan-docker VS Code attach steps.

NOTE: the containers also support debugging with pdb.

Running with HTTPS

For dev HTTPS (USE_HTTPS_FOR_DEV, self-signed cert on port 5000), follow ckan-docker — Running HTTPS on development mode, then recreate containers:

bin/compose down && bin/compose up

Keep CKAN_SITE_URL aligned with how you reach CKAN — it affects catalogue and download URLs.

To test the production-style nginx stack (HTTPS on port 8443) instead, use docker-compose.yml rather than bin/compose. See ckan-docker § NGINX and set in .env:

USE_HTTPS_FOR_DEV=false
CKAN_SITE_URL=https://localhost:8443
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000

docker compose -f docker-compose.yml up

Then visit https://localhost:8443.

Tracking ckan-docker upstream

This repo uses a subset of ckan/ckan-docker. To pull in upstream changes when upgrading CKAN versions:

# Set up a remote to track changes:
git remote add ckan-docker https://github.com/ckan/ckan-docker.git
git fetch ckan-docker --no-tags

# Review upstream changes:
git log HEAD..ckan-docker/master --oneline

# To selectively apply upstream changes
git merge --squash ckan-docker/master
# undo any changes we are definitely managing ourselves (currently just the readme):
git checkout HEAD -- README.md

# commit the changes, and track the SHA of the upstream commit we merged:
SHA=$(git rev-parse --short ckan-docker/master)
git commit -m "Merged upstream ckan-docker up to SHA: <$SHA>"
git tag -a ckan-docker-$SHA -m "Merged upstream ckan-docker up to SHA: <$SHA>"