Skip to content

feat(ai-service): Replace task_results global dict with Redis-backed task store#363

Open
ugoocreates-pixel wants to merge 2 commits into
ChainForgee:mainfrom
ugoocreates-pixel:feat/redis-task-store
Open

feat(ai-service): Replace task_results global dict with Redis-backed task store#363
ugoocreates-pixel wants to merge 2 commits into
ChainForgee:mainfrom
ugoocreates-pixel:feat/redis-task-store

Conversation

@ugoocreates-pixel

@ugoocreates-pixel ugoocreates-pixel commented Jul 16, 2026

Copy link
Copy Markdown

Closes #271

This PR replaces the module-level in-memory dictionary task_results in the AI Service (app/ai-service/tasks.py) with a persistent, Redis-backed task store.

Why this is necessary:

  • Resilience: A single-process in-memory dict gets destroyed whenever a pod/container restarts, losing active status tracking.
  • Scalability: In multi-process deployments, workers run in forked/different processes and cannot share status updates via a local in-memory dict, causing fork-bomb state mismatches.

Changes Made

  • Removed task_results: Deleted the task_results global dictionary.
  • Lazy Redis Client: Added lazy initialization of the Redis client (get_redis_client) targeting the configured REDIS_URL.
  • Status Persistence: Implemented tasks.set_status to write task payload states directly to Redis with a TTL of 24 hours (86400 seconds) matching Celery's result_expires.
  • Robust Retrieval: Updated tasks.get_task_status to check Celery's AsyncResult status first, and fallback to the Redis-backed store on failure or missing keys.
  • Unit & Integration Tests: Created app/ai-service/tests/test_tasks_redis.py covering:
    • Setting status to Redis with precise TTL.
    • Graceful fallback reading from Redis when Celery returns unfinished states or throws connection errors.
    • Correct E2E cross-process communication simulation (Process A writing to Redis, Process B observing status).

Copy link
Copy Markdown
Contributor

Hi 👋 Appreciate this PR! The CI is flagging a failing check — could you take a peek at the failed job and push a fix? Happy to help if you need a hand 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace task_results global dict with Redis-backed task store

2 participants