Skip to content

S3-compatible object storage for uploads #7

Description

@sebprunier

Goal

Uploaded files (GeoJSON, CSV, Shapefile, …) are written to a local UPLOAD_DIR. This forces the API and worker to share a filesystem, which is awkward as soon as they run on different hosts or in different containers. Switching to an S3-compatible backend removes that constraint and enables horizontal scaling of workers.

Scope

  • A FileStorage interface with two implementations:
    • LocalFileStorage (current behaviour, default)
    • S3FileStorage (any S3-compatible: AWS S3, MinIO, Cloudflare R2, Scaleway, OVH Object Storage)
  • Backend selected via STORAGE_BACKEND=local|s3 env var
  • S3 config via env: S3_BUCKET, S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY, S3_SECRET_KEY, S3_FORCE_PATH_STYLE
  • API stores the file via the FileStorage, queues a job carrying the storage URI; worker reads it back through the same interface

Acceptance criteria

  • FileStorage interface and LocalFileStorage refactor (no behavioural change)
  • S3FileStorage using @aws-sdk/client-s3 v3
  • Selection via STORAGE_BACKEND env var
  • Vitest integration tests for the S3 backend, using a Testcontainers MinIO container (no AWS account required)
  • Documentation: docs/guide/configuration.md and a deployment example (compose with MinIO)

Out of scope

  • Automatic migration of files already on disk (one-time operator task)
  • Pre-signed URLs for direct browser uploads (could come later)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions