Add celery_id DB tracking for privacy requests and request tasks

[galvana]

Description Of Changes

Stores pre-generated Celery task IDs directly on PrivacyRequest and RequestTask models instead of in Redis cache. This makes task tracking durable across Redis restarts and enables reliable task cancellation via batch revoke().

Key changes:

• Add celery_id column to privacyrequest and requesttask tables via Alembic migration
• Pre-generate UUIDs and persist to DB before dispatching Celery tasks
• Add cancel_celery_tasks() method on PrivacyRequest for batch revocation
• Add get_request_task_celery_task_ids() helper method
• Replace Redis-based cache_task_tracking_key with durable DB persistence
• Add privacy request retry count tracking with configurable limits via privacy_request_requeue_retry_count config
• Add _cancel_interrupted_tasks_and_error_privacy_request() for interrupted task cleanup
• Clean up unnecessary inline imports across source and test files

Code Changes

• src/fides/api/alembic/migrations/versions/xx_2026_02_28_1200_ce488ce97a31_add_celery_id_to_privacy_request_and_request_task.py - New migration adding celery_id column to both tables
• src/fides/api/models/privacy_request/privacy_request.py - Add celery_id column, cancel_celery_tasks(), get_request_task_celery_task_ids(), celery task cancellation in cancel_processing()
• src/fides/api/models/privacy_request/request_task.py - Add celery_id column
• src/fides/api/task/execute_request_tasks.py - Pre-generate and persist celery_id before dispatching; move uuid to top-level import
• src/fides/service/privacy_request/privacy_request_service.py - Pre-generate and persist celery_id for privacy request tasks; move uuid to top-level import
• src/fides/api/service/privacy_request/request_service.py - Add _cancel_interrupted_tasks_and_error_privacy_request(), retry count tracking in requeue flow
• tests/ops/models/privacy_request/test_privacy_request.py - Tests for cancel_celery_tasks, get_request_task_celery_task_ids, integration with cancel_processing; clean up inline imports
• tests/ops/models/privacy_request/test_request_task.py - Tests for celery_id column on RequestTask
• tests/ops/service/privacy_request/test_request_service.py - Tests for _cancel_interrupted_tasks_and_error_privacy_request, retry count requeue handling; clean up inline imports
• tests/ops/util/test_cache.py - Tests for retry count cache functions; consolidate inline imports to top-level
• tests/task/test_requeue_interrupted_tasks.py - Tests for enhanced requeue with retry limits; clean up inline imports

Steps to Confirm

1. Run the migration: verify celery_id column exists on both privacyrequest and requesttask tables
2. Submit a privacy request and verify celery_id is populated on the PrivacyRequest record before the Celery task starts
3. Verify request tasks also have celery_id populated
4. Cancel a running privacy request and verify cancel_celery_tasks() issues batch revoke
5. Run test suite: pytest --no-cov tests/ops/models/privacy_request/test_privacy_request.py::TestCancelCeleryTasks tests/ops/service/privacy_request/test_request_service.py::TestCancelInterruptedTasksAndErrorPrivacyRequest tests/task/test_requeue_interrupted_tasks.py::TestEnhancedRequeueInterruptedTasks tests/ops/util/test_cache.py::TestPrivacyRequestRetryCache -v

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • No UX review needed
• Followup issues:
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
• Documentation:
  • No documentation updates required

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ignored	Preview	Feb 28, 2026 11:17pm
fides-privacy-center	Ignored		Feb 28, 2026 11:17pm