[Backend]: [Architecture/Performance]: Decouple Document Processing via Message Queue #75

🧭 Project Area

Backend / API

📝 Issue Summary and Problem Description

Issue summary:
uploadDocument in document.controller.js processes file uploads synchronously. It computes the SHA-256 checksum in-memory and blocks the HTTP request until the file is fully uploaded to S3.

Problem description:
For large files, computing a SHA-256 hash ties up the Node.js single-threaded event loop. Furthermore, waiting on S3 network I/O before returning a response makes the API feel sluggish and holds HTTP connections open unnecessarily, limiting concurrency.

Why this is important:
Synchronous heavy processing is a massive scalability bottleneck. Decoupling this into an asynchronous worker queue ensures the main API thread remains responsive to other users, drastically increasing overall throughput and preventing request timeouts.

Who is affected:
System scalability, and users uploading large identity/financial documents.

🔍 Current and Expected Behaviour

Current behaviour:
The client waits for the entire Hash computation + S3 upload process to finish before receiving a 201 Created.

Expected behaviour:
The API should accept the file, immediately place it in a processing queue, and return a 202 Accepted with a tracking ID or document placeholder. A background worker should then handle the heavy lifting (hashing and S3 uploading).

🧩 Reproduction and Evidence

Step 1: Upload a 50MB file to `/api/documents`.
Step 2: Observe the long TTFB (Time To First Byte) while the server computes the hash and uploads to S3, during which other requests on that Node thread are delayed.

🎯 Affected Scope

Affected files:

• backend/src/modules/documents/document.controller.js
• package.json (Adding queue dependencies)

Affected APIs:
POST /api/documents

Affected components:
Document upload logic, introduction of a background worker service (e.g. BullMQ).

Database changes required?: Yes (Adding a processing_status field to the document model to indicate it is queued vs ready).

🖥 Environment

Operating system: All
Browser and version: N/A
Node.js version: >= 20
Package manager and version: npm
Docker version: N/A
Deployment environment: Production, Staging

💡 Solution, Alternatives, and Estimate

Priority (P0 Critical / P1 High / P2 Medium / P3 Low): P2 Medium
Proposed solution:

1. Integrate BullMQ (using Redis) or native Node Worker Threads.
2. Move the crypto hashing and S3 upload logic into a background job processor.
3. Update the endpoint to return 202 Accepted immediately, updating the document state to "processing".
4. The worker updates the document status to "verified" or "ready" upon completion.

Alternative solutions:
Offloading the hashing to the frontend before upload, but this cannot be fully trusted for security purposes and S3 upload still blocks the server if not using presigned POSTs directly from the client.

Difficulty (Beginner / Intermediate / Advanced): Advanced
Estimated time: 4 to 6 hours
Dependencies or blockers: Requires Redis to be available if using BullMQ.

✅ Acceptance Criteria and Additional Notes

• Endpoint returns 202 Accepted immediately upon receiving the file buffer.
• Checksum and S3 upload are handled asynchronously off the main event loop via a queue.
• Document model correctly reflects a "processing" state until the worker completes.
• Integration tests verify the asynchronous job flow.

Additional notes:
I would like to work on this issue as part of ECSoC 2026 after receiving maintainer assignment. This is an L3 (Core Architectural Restructuring) issue.

🤝 Contributor Checklist

• I have searched existing issues and pull requests for duplicates.
• I can reproduce the issue, or I have clearly explained why reproduction is not applicable.
• I will follow the contribution guidelines and target ecsoc-2026.
• I agree to follow the Code of Conduct.
• I have removed secrets, private documents, and personal data.