Feat/tusd service resumable uploads

[whlongg]

Description

Implement resumable uploads via the tus protocol (tusd) for large test data files, mitigating reverse-proxy request body limitations.

Type of change: New feature

What

• Introduces a standalone Django app, resumable_upload, responsible for generating JWT upload intents (/upload/intent/) and handling incoming tusd webhooks (/upload/hooks/).
• Incorporates tus-js-client into templates/problem/data.html. This enables chunked, resumable uploads for ZIP files exceeding the defined threshold (default: 100MB).
• Refactors ProblemDataView.post() in judge/views/problem_data.py. The view now correctly handles files that have already been relocated to DMOJ_PROBLEM_DATA_ROOT by the pre-finish blocking hook from tusd.
• Exposes new variables in dmoj/settings.py for TUSD configurations (endpoints, JWT secrets, hook secrets, and thresholds). Adds PyJWT>=2.0 to requirements.txt for secure token signing and verification.
• Handles Docker-to-Host path translation (TUSD_LOCAL_DATA_DIR) in the pre-finish webhook to resolve file system path mismatches between the tusd container and the Django host environment.
• Triggers metadata recalculation (file size and hash updates) within ProblemDataView.post() to ensure the database stays perfectly synced after the file is stealthily moved into the system by the webhook.

Why

Currently, uploading test data (ZIP files) for problems on VNOJ is processed directly via standard Django forms. However, reverse proxies like Cloudflare impose strict request body limits (e.g., 200MB) which cause direct uploads of large datasets (sometimes spanning several gigabytes) to fail.

By utilizing tusd as a dedicated upload server and the tus protocol for chunked, resumable uploads, we bypass these proxy constraints, resulting in a stable and reliable upload pipeline. Crucially, the implementation utilizes blocking pre-finish hooks, ensuring file processing and relocation are fully synchronized with Django, thereby completely eliminating race conditions before the client receives a success response.

sequenceDiagram
    participant Browser
    participant Django
    participant tusd

    Note over Browser,Django: Phase 1: Request Upload Intent
    Browser->>Django: POST /upload/intent/ (problem_code, file_type)
    Django->>Django: Check permissions (session/cookie)<br>Check if problem exists & editable<br>Check quota
    Django-->>Browser: JWT token (contains intent metadata)

    Note over Browser,tusd: Phase 2: Chunked Upload
    Browser->>tusd: POST /files/ (tus protocol, JWT in metadata)
    tusd->>Django: pre-create hook → POST /upload/hooks/
    Django->>Django: Decode JWT, verify expiration
    Django-->>tusd: 200 OK (allow upload creation)
    loop Each chunk
        Browser->>tusd: PATCH /files/{id} (chunk data)
    end

    Note over tusd,Django: Phase 3: Pre-finish (BLOCKING)
    tusd->>Django: pre-finish hook → POST /upload/hooks/
    Django->>Django: Decode JWT from metadata<br>Move file from tusd → DMOJ_PROBLEM_DATA_ROOT/{problem_code}/
    Django-->>tusd: 200 OK
    Note over tusd: Now it returns 204 to Browser
    tusd-->>Browser: HTTP 204 (upload complete)
    Note over Browser: onSuccess fires

Loading

Fixes #484

Deployment & Testing Notes

Local Testing with Docker

To test this setup locally, spin up a tusd container with the appropriate hooks pointing to your local Django server:

docker run -p 8080:8080 -v /home/whlong/Documents/VNOI-Admin/tusd-data:/srv/tusd-data \
  --add-host=host.docker.internal:host-gateway \
  tusproject/tusd \
  -hooks-http http://host.docker.internal:8000/upload/hooks/ \
  -hooks-enabled-events pre-create,pre-finish \
  -cors-allow-origin ".*"

Production Setup

• Nginx Configuration: In production, Nginx will need to be configured to correctly route traffic to both the Django application and the tusd server, ensuring CORS and max body size settings are correctly handled for the /files/ endpoint.
• Environment Variables: The following settings must be configured in dmoj/local_settings.py for production:
  • TUSD_ENDPOINT: The public URL of the tusd server (e.g., https://judge.example.com/files/).
  • TUSD_DATA_DIR: Path to the directory where tusd stores chunks (needs to be accessible by Django).
  • TUSD_HOOK_SECRET: A shared secret string used to authenticate webhooks sent from tusd to Django.
  • TUSD_JWT_SECRET: The secret key used to sign the upload intent JWTs (can fallback to Django's SECRET_KEY).
  • TUSD_JWT_EXPIRY_SECONDS: TTL for the intent token (default: 7200s).
  • TUSD_UPLOAD_THRESHOLD_BYTES: File size threshold to trigger tus uploads (default: 100MB).
  • TUSD_ALLOWED_FILE_TYPES: Tuple of permitted file types for the upload hooks.

How Has This Been Tested?

• Wrote and executed unit tests (resumable_upload/tests/test_views.py) for the resumable_upload app, testing JWT token logic, views, intent generation, and webhook validation.
• Spun up a local tusd container via Docker with hooks configured (-hooks-enabled-events pre-create,pre-finish) pointing to the local Django instance.
• Manually tested uploading a file > 100MB on the problem data edit page to verify the progress bar, chunking, and that the file is correctly moved to DMOJ_PROBLEM_DATA_ROOT/{problem_code}/ upon completion.
• Verified edge cases: Files < 100MB still use the traditional form upload flow, network interruptions successfully resume, and pre-finish failures correctly return errors to the frontend via onError.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• I have explained the purpose of this PR.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the README/documentation
• Any dependent changes have been merged and published in downstream modules
• Informed of breaking changes, testing and migrations (if applicable).
• Attached screenshots (if applicable).

By submitting this pull request, I confirm that my contribution is made under the terms of the AGPL-3.0 License.

[magnified103]

Can you verify whether these attributes are actually used in other places?

Also this module is intended to be 'context-agnostic'. Therefore, I think problem_code should be renamed to something else.

[whlongg]

The sub field should be kept for standard JWT compliance (RFC 7519), future-proof authorization checks at the hook level, and robust audit trailing/logging.