Repository navigation
AutoGPT Platform - Preview Seed Fixture Bake #5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: AutoGPT Platform - Preview Seed Fixture Bake | |
| # Bakes a fully SYNTHETIC preview-database seed fixture and publishes it as a | |
| # rolling GitHub release asset (tag: preview-seed-fixture). Preview environments | |
| # restore this fixture BEFORE a PR's own migrations run, so schema changes are | |
| # exercised against populated tables (catching NOT-NULL / unique / backfill | |
| # failures that an empty DB would silently pass). | |
| # | |
| # SECURITY INVARIANT: this workflow holds NO cloud credentials and has NO read | |
| # path to any real database. It runs entirely against a throwaway Postgres | |
| # service container. The fixture contains ONLY Faker-generated synthetic rows | |
| # plus the already-public marketplace agent exports checked into | |
| # autogpt_platform/backend/agents/. No production data is ever touched. | |
| # | |
| # RESTORE PROCEDURE (consumers): apply restore-preamble.sql to the target | |
| # database FIRST — it creates the platform schema and installs the vector + | |
| # pg_trgm extensions into it, which pg_dump --schema=platform omits because | |
| # CREATE EXTENSION is not emitted for extensions living inside the dumped | |
| # schema. Then run: | |
| # gunzip -c fixture.dump.gz | pg_restore -d <db> --no-owner | |
| # Do NOT pass --exit-on-error: the dump's own CREATE SCHEMA platform collides | |
| # benignly with the preamble's. | |
| on: | |
| push: | |
| branches: [dev] | |
| paths: | |
| - "autogpt_platform/backend/migrations/**" | |
| - "autogpt_platform/backend/test/**" | |
| - ".github/workflows/platform-preview-seed-fixture.yml" | |
| schedule: | |
| # Weekly backstop (Mondays 06:00 UTC) so the fixture never goes stale even | |
| # if no migration/test change lands for a while. | |
| - cron: "0 6 * * 1" | |
| workflow_dispatch: | |
| # Coalesce overlapping bakes: a newer commit cancels an in-flight bake since | |
| # they would publish the same rolling asset anyway. | |
| concurrency: | |
| group: preview-seed-fixture-bake | |
| cancel-in-progress: true | |
| permissions: | |
| contents: write # required to create/update the rolling release + its assets | |
| defaults: | |
| run: | |
| shell: bash | |
| working-directory: autogpt_platform/backend | |
| jobs: | |
| bake: | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 45 | |
| services: | |
| # Throwaway Postgres. pgvector/pgvector:pg15 matches the platform's real | |
| # Postgres major version (supabase/postgres:15.8.x used in dev/prod) and | |
| # ships the `vector` extension (required by the docs-embedding migrations) | |
| # plus the `pg_trgm` contrib module (creator-search index migration). No | |
| # credentials of any kind — a disposable localhost container. | |
| postgres: | |
| image: pgvector/pgvector:pg15 | |
| env: | |
| POSTGRES_USER: postgres | |
| POSTGRES_PASSWORD: postgres | |
| POSTGRES_DB: postgres | |
| ports: | |
| - 5432:5432 | |
| options: >- | |
| --health-cmd "pg_isready -U postgres" | |
| --health-interval 10s | |
| --health-timeout 5s | |
| --health-retries 10 | |
| env: | |
| CI: "true" | |
| PLAIN_OUTPUT: "true" | |
| # Prisma connection to the throwaway container, using the platform schema | |
| # exactly like the backend tests / local dev (.env.default). | |
| DATABASE_URL: "postgresql://postgres:postgres@localhost:5432/postgres?schema=platform" | |
| DIRECT_URL: "postgresql://postgres:postgres@localhost:5432/postgres?schema=platform" | |
| # Dummy, UNREACHABLE Supabase config. e2e_test_data.py no longer talks to | |
| # Supabase at all (users are created through Better Auth via raw Prisma | |
| # inserts), but backend settings still read these values at import time, | |
| # so they only need to exist. Nothing in this job ever connects to them. | |
| SUPABASE_URL: "http://localhost:54321" | |
| SUPABASE_SERVICE_ROLE_KEY: "synthetic-bake-no-real-key" # pragma: allowlist secret | |
| # Non-production test key (identical to the one used in backend CI) so | |
| # backend module imports that read settings.secrets don't error. | |
| ENCRYPTION_KEY: "dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw=" # pragma: allowlist secret # DO NOT USE IN PRODUCTION | |
| steps: | |
| - name: Checkout repository | |
| uses: actions/checkout@v6 | |
| with: | |
| # Always bake dev: schedule/dispatch runs execute from the DEFAULT | |
| # branch (master), which would otherwise silently bake master's | |
| # schema into a fixture that previews (built from dev) restore. | |
| ref: dev | |
| # Only the release-publish step needs a token, via env — don't | |
| # persist credentials into the workspace git config. | |
| persist-credentials: false | |
| - name: Set up Python 3.12 | |
| uses: actions/setup-python@v5 | |
| with: | |
| python-version: "3.12" | |
| - name: Set up Python dependency cache | |
| uses: actions/cache@v5 | |
| with: | |
| path: ~/.cache/pypoetry | |
| key: poetry-${{ runner.os }}-py3.12-${{ hashFiles('autogpt_platform/backend/poetry.lock') }} | |
| - name: Install Poetry | |
| run: | | |
| HEAD_POETRY_VERSION=$(python ../../.github/workflows/scripts/get_package_version_from_lockfile.py poetry) | |
| echo "Using Poetry version ${HEAD_POETRY_VERSION}" | |
| curl -sSL https://install.python-poetry.org | POETRY_VERSION=$HEAD_POETRY_VERSION python3 - | |
| - name: Install Python dependencies | |
| run: poetry install | |
| - name: Generate Prisma Client | |
| run: poetry run prisma generate | |
| - name: Apply database migrations | |
| run: poetry run prisma migrate deploy | |
| # --- Seeders -------------------------------------------------------- | |
| # Order mirrors each script's documented invocation. test_data_creator is | |
| # REQUIRED (a failure fails the bake). The other three are best-effort: | |
| # a failure emits a LOUD ::warning:: but does not abort the bake. | |
| # | |
| # The two Python seeders below are additionally capped with GNU timeout: | |
| # this job runs NO RabbitMQ/Redis service containers, and the backend's | |
| # conn_retry would otherwise spin ~48 min acquiring them — blowing the | |
| # job-level timeout before dump/publish. `|| { ...; exit 0; }` catches | |
| # nonzero exits, not hangs; timeout converts a hang into exit 124. | |
| - name: "Seed: test_data_creator (REQUIRED)" | |
| run: poetry run python test/test_data_creator.py | |
| - name: "Seed: load-store-agents (public marketplace exports)" | |
| run: | | |
| poetry run load-store-agents || { | |
| echo "::warning title=seed-soft-failure::load-store-agents seeder failed; continuing without it." | |
| exit 0 | |
| } | |
| - name: "Seed: e2e_test_data (best-effort)" | |
| run: | | |
| timeout 300 poetry run python test/e2e_test_data.py || { | |
| echo "::warning title=seed-soft-failure::e2e_test_data seeding skipped/timed out (no queue/cache services in this job); continuing without it." | |
| exit 0 | |
| } | |
| - name: "Seed: test_data_updater (mutates existing rows)" | |
| run: | | |
| timeout 300 poetry run python test/test_data_updater.py || { | |
| echo "::warning title=seed-soft-failure::test_data_updater seeding skipped/timed out (no queue/cache services in this job); continuing without it." | |
| exit 0 | |
| } | |
| # --- Dump + manifest ------------------------------------------------ | |
| # pg_dump / psql run INSIDE the Postgres service container via docker exec | |
| # so the client version always matches the server exactly (the runner's | |
| # bundled client can lag the server major and refuse the dump). This is | |
| # the same pattern platform-fullstack-ci.yml uses. | |
| - name: Resolve Postgres service container | |
| run: | | |
| set -euo pipefail | |
| PG_CID=$(docker ps --filter "ancestor=pgvector/pgvector:pg15" --format '{{.ID}}' | head -n1) | |
| if [ -z "${PG_CID}" ]; then | |
| echo "::error title=no-pg-container::Could not locate the Postgres service container." | |
| docker ps | |
| exit 1 | |
| fi | |
| echo "PG_CID=${PG_CID}" >> "${GITHUB_ENV}" | |
| docker exec "${PG_CID}" pg_dump --version | |
| - name: Dump platform schema fixture (custom format, gzipped) | |
| run: | | |
| set -euo pipefail | |
| # Loud sanity check: the required seeder must have populated users. | |
| USER_COUNT=$(docker exec "${PG_CID}" psql -U postgres -d postgres -tAc "SELECT count(*) FROM platform.\"User\";" | tr -d '[:space:]') | |
| echo "Seeded platform.\"User\" rows: ${USER_COUNT}" | |
| if [ "${USER_COUNT:-0}" -lt 1 ]; then | |
| echo "::error title=empty-fixture::Seeded fixture is empty (platform.User has 0 rows); refusing to publish." | |
| exit 1 | |
| fi | |
| # Custom-format dump of the platform schema ONLY (schema + data, | |
| # including platform._prisma_migrations so the restore side can run | |
| # only the PR's delta migrations). -Z0 disables pg_dump's internal | |
| # compression so the outer gzip is the single compressor; restore is | |
| # therefore `gunzip -c fixture.dump.gz | pg_restore ...`. | |
| docker exec "${PG_CID}" pg_dump -U postgres -d postgres \ | |
| --format=custom -Z0 --schema=platform \ | |
| | gzip -9 > "${GITHUB_WORKSPACE}/fixture.dump.gz" | |
| ls -lh "${GITHUB_WORKSPACE}/fixture.dump.gz" | |
| - name: Write restore-preamble.sql | |
| run: | | |
| set -euo pipefail | |
| # pg_dump --schema=platform does NOT emit CREATE EXTENSION for | |
| # extensions installed INTO the dumped schema (the vector extension | |
| # lives in platform since 20260605120000_fix_vector_extension_search_path, | |
| # and pg_trgm lands there too), so restoring into a fresh database | |
| # fails with 'type "platform.vector" does not exist'. Consumers run | |
| # this preamble BEFORE pg_restore, and must NOT use --exit-on-error | |
| # (the dump's own CREATE SCHEMA collides benignly with the | |
| # preamble's). | |
| cat > "${GITHUB_WORKSPACE}/restore-preamble.sql" <<'SQL' | |
| CREATE SCHEMA IF NOT EXISTS platform; | |
| CREATE EXTENSION IF NOT EXISTS vector SCHEMA platform; | |
| CREATE EXTENSION IF NOT EXISTS pg_trgm SCHEMA platform; | |
| SQL | |
| cat "${GITHUB_WORKSPACE}/restore-preamble.sql" | |
| - name: Write manifest.json | |
| run: | | |
| set -euo pipefail | |
| # baked_at comes from the checked-out commit (NOT wall-clock), so the | |
| # manifest is reproducible for a given dev SHA. | |
| BAKED_AT="$(git log -1 --format=%cI)" | |
| # NOT GITHUB_SHA: on schedule/dispatch runs that is the default | |
| # branch's commit; the checkout above is pinned to dev. | |
| DEV_SHA="$(git rev-parse HEAD)" | |
| MIGRATION_HEAD="$(docker exec "${PG_CID}" psql -U postgres -d postgres -tAc \ | |
| "SELECT migration_name FROM platform._prisma_migrations WHERE finished_at IS NOT NULL ORDER BY finished_at DESC LIMIT 1;" \ | |
| | tr -d '[:space:]')" | |
| echo "baked_at=${BAKED_AT} dev_sha=${DEV_SHA} migration_head=${MIGRATION_HEAD}" | |
| if [ -z "${MIGRATION_HEAD}" ]; then | |
| echo "::error title=no-migration-head::Could not read a migration head from platform._prisma_migrations." | |
| exit 1 | |
| fi | |
| jq -n \ | |
| --arg baked_at "${BAKED_AT}" \ | |
| --arg dev_sha "${DEV_SHA}" \ | |
| --arg migration_head "${MIGRATION_HEAD}" \ | |
| --arg restore_preamble "Apply restore-preamble.sql to the target database BEFORE pg_restore: it creates the platform schema and installs the vector + pg_trgm extensions into it (pg_dump --schema=platform omits CREATE EXTENSION for extensions inside the dumped schema). Do NOT pass --exit-on-error to pg_restore — the dump's own CREATE SCHEMA collides benignly with the preamble's." \ | |
| '{baked_at: $baked_at, dev_sha: $dev_sha, migration_head: $migration_head, restore_preamble: $restore_preamble}' \ | |
| > "${GITHUB_WORKSPACE}/manifest.json" | |
| cat "${GITHUB_WORKSPACE}/manifest.json" | |
| # --- Publish rolling release --------------------------------------- | |
| - name: Publish rolling preview-seed-fixture release | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| set -euo pipefail | |
| TAG="preview-seed-fixture" | |
| FIXTURE="${GITHUB_WORKSPACE}/fixture.dump.gz" | |
| MANIFEST="${GITHUB_WORKSPACE}/manifest.json" | |
| PREAMBLE="${GITHUB_WORKSPACE}/restore-preamble.sql" | |
| MIGRATION_HEAD="$(jq -r .migration_head "${MANIFEST}")" | |
| DEV_SHA="$(jq -r .dev_sha "${MANIFEST}")" | |
| NOTES="Rolling, fully SYNTHETIC preview-database seed fixture baked from dev@${DEV_SHA:0:12} (migration head: ${MIGRATION_HEAD}). Preview environments restore fixture.dump.gz (custom-format, gzipped, platform schema only, including _prisma_migrations) BEFORE running a PR's own migrations, so schema changes run against populated tables. RESTORE: apply restore-preamble.sql to the target database first (it creates the platform schema and installs the vector + pg_trgm extensions into it, which pg_dump --schema=platform omits), then run 'gunzip -c fixture.dump.gz | pg_restore -d <db> --no-owner' WITHOUT --exit-on-error (the dump's own CREATE SCHEMA collides benignly with the preamble's). SECURITY INVARIANT: this bake holds no cloud credentials and has no read path to any real database — it runs entirely against a throwaway Postgres container and contains only Faker-generated synthetic rows plus the already-public marketplace agent exports checked into autogpt_platform/backend/agents/. This asset is regenerated on every eligible dev push and weekly; treat it as disposable." | |
| if gh release view "${TAG}" >/dev/null 2>&1; then | |
| echo "Updating existing rolling release ${TAG}" | |
| gh release edit "${TAG}" --title "Preview seed fixture (rolling)" --notes "${NOTES}" --prerelease | |
| gh release upload "${TAG}" "${FIXTURE}" "${MANIFEST}" "${PREAMBLE}" --clobber | |
| # Retarget the rolling tag at the baked dev commit so release | |
| # metadata matches the published assets (manifest.dev_sha is the | |
| # authoritative record either way). | |
| gh api -X PATCH "repos/${GITHUB_REPOSITORY}/git/refs/tags/${TAG}" \ | |
| -f sha="${DEV_SHA}" -F force=true >/dev/null | |
| else | |
| echo "Creating rolling release ${TAG}" | |
| gh release create "${TAG}" "${FIXTURE}" "${MANIFEST}" "${PREAMBLE}" \ | |
| --title "Preview seed fixture (rolling)" \ | |
| --notes "${NOTES}" \ | |
| --prerelease \ | |
| --target "${DEV_SHA}" | |
| fi |