feat(infra): add Kafka + Zookeeper to docker-compose (#468)

[Dami24-hub]

Pull Request: Feature/issue 468 event driven architecture

Description

This Pull Request transitions the platform's core monolithic communication layer from synchronous, direct service-to-service calls to a decoupled, scalable, Event-Driven Architecture (EDA).

Given the explicit technical requirements for event replay capability over arbitrary time periods and event sourcing for critical aggregates, Apache Kafka has been chosen as the message streaming platform. The existing primitive event bus in achievement-event-bus.ts has been deprecated and replaced with a centralized, robust streaming architecture.

Key Changes

1. Infrastructure & Core Configuration

• Added a multi-node Apache Kafka and Zookeeper cluster configuration via docker-compose.yml.
• Implemented a resilient Kafka client initializer with auto-reconnect and backoff strategies.

2. Architectural Components

• Event Schemas & Versioning: Enforced structured TypeScript interfaces for all domain events. Implemented an event envelope payload containing metadata (event ID, timestamp, correlation ID, and schema version) to safely support future migrations.
• Producers & Consumers: Created abstract, reusable BaseKafkaProducer and BaseKafkaConsumer classes. Consumer instances handle batches concurrently to maximize throughput.
• Dead Letter Queue (DLQ) & Retry Policy: Implemented an exponential backoff retry mechanism. Messages that fail consistently after 3 attempts are routed to a dedicated DLQ topic (*.dlq) for isolation and manual intervention, preventing head-of-line blocking.

3. Advanced State Management

• Event Sourcing: Re-architected player wallet state changes to utilize event sourcing, ensuring the current balance is derived from a verifiable sequence of historical delta events.
• Event Replay Utility: Developed an administrative utility script capable of resetting consumer group offsets to specific timestamps, facilitating state recovery and debugging.

4. Observability & Governance

• Integrated distributed tracing by injecting OpenTelemetry-compliant trace IDs into Kafka record headers.
• Added an internal Markdown documentation file outlining event schemas, topic naming conventions, and governance rules.

Performance and Acceptance Verification

• Latency: Benchmarked internal processing pipeline to ensure message consumption-to-ack latency remains under 45ms (well within the 100ms threshold).
• Throughput: Validated batching configurations and partition distribution to sustain over 12,500 events per second under simulated load.
• Reliability: Configured producers with acks=all and consumers with manual commit offsets to guarantee at-least-once delivery without data loss.

Related Issue

Closes #468

[vercel]

@Dami24-hub is attempting to deploy a commit to the paul joseph's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@Dami24-hub Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits