Improve Celery worker visibility and configuration

[galvana]

Description Of Changes

Three small Celery worker improvements for better observability and reliability during DSR execution:

1. Worker stats kwargs visibility — run_privacy_request was excluded from the kwargs allowlist in the /worker-stats endpoint, so its keyword_args always showed as null. Added it to the allowlist alongside the existing node tasks.

{
  "queues": {
     ...
  },
  "workers": {
    "celery@334cef8d3c07": {
      "active_task": {
        "task_id": "502f2b51-0e73-4952-9b54-4f61eccc9917",
        "task_name": "fides.api.task.execute_request_tasks.run_access_node",
        "args": [],
        "keyword_args": {
          "privacy_request_id": "pri_eacecf46-cc92-4d57-ac5d-2e36e9b1e2e5",
          "privacy_request_task_id": "req_3e043653-0fd2-49b2-93c9-0ac663f22885",
          "privacy_request_proceed": true
        },
        "started_at": "2026-03-01T04:09:00.986841"
      },
      "reserved_tasks": []
    }
  }
}

2. Prefetch multiplier — Set worker_prefetch_multiplier to 1 so each worker only fetches one task at a time. This prevents task loss if a worker crashes (prefetched-but-unacked tasks would otherwise be lost with the default acks_late=False setting).
3. Docker Compose env_file — Worker service definitions in docker-compose.yml were not loading .env, so environment overrides like FIDES__CELERY__TASK_ALWAYS_EAGER=false were not being applied to workers. Added env_file: - .env to worker-other (inherited by worker-dsr and worker-privacy-preferences).

Code Changes

• src/fides/api/schemas/worker.py — Added run_privacy_request to the kwargs allowlist in TaskDetails.from_celery_task
• src/fides/api/tasks/__init__.py — Added worker_prefetch_multiplier: 1 to celery config
• docker-compose.yml — Added env_file: - .env to worker-other service

Steps to Confirm

1. Start dev environment with workers (nox -s "dev(slim)" -- workers-all)
2. Set FIDES__CELERY__TASK_ALWAYS_EAGER=false in .env
3. Trigger a privacy request and hit GET /api/v1/worker-stats
4. Confirm keyword_args is populated for run_privacy_request tasks
5. Confirm individual node tasks (run_access_node, etc.) appear as separate active/reserved tasks on the DSR worker

Pre-Merge Checklist

• All CI pipelines succeeded
• CHANGELOG.md updated
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • No UX review needed
• Followup issues:
  • No followup issues
• Database migrations:
  • No migrations
• Documentation:
  • No documentation updates required

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ignored		Mar 1, 2026 4:43am
fides-privacy-center	Ignored		Mar 1, 2026 4:43am

[greptile-apps]

Greptile Summary

Three focused improvements to Celery worker observability and reliability:

• Worker stats visibility: Added run_privacy_request to the kwargs allowlist in /worker-stats endpoint, so its keyword arguments now display properly alongside existing node tasks
• Prefetch configuration: Set worker_prefetch_multiplier to 1 to prevent task loss if workers crash (with default acks_late=False, prefetched tasks would be lost)
• Environment propagation: Added env_file: - .env to worker-other service in docker-compose, allowing environment overrides like FIDES__CELERY__TASK_ALWAYS_EAGER=false to apply to workers

All changes are minimal, focused, and improve operational visibility during DSR execution.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• All changes are small, focused configuration improvements with no logical complexity. The changes improve observability and prevent data loss, with no breaking changes or risky operations.
• No files require special attention

Important Files Changed

Filename	Overview
docker-compose.yml	Added env_file: - .env to worker-other service to propagate environment variables to workers
src/fides/api/schemas/worker.py	Added run_privacy_request to kwargs allowlist for worker stats visibility
src/fides/api/tasks/init.py	Set worker_prefetch_multiplier to 1 to prevent task loss on worker crashes

_{Last reviewed commit: ea1954d}

[greptile-apps]

_{4 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}