feat: implement persistent job queue with bbolt and maintenance worker #375
Conversation
The current job queue was volatile and jobs would be lost on backend restarts. This implements a bbolt-based persistent queue that stores jobs to disk and restores them on startup. Also added a cron-based maintenance worker that automatically cleans up old completed and failed job logs to prevent unbounded disk usage. All existing functionality is preserved and the new features are fully configurable via environment variables. Resolves CCExtractor#367
|
Thank you for opening this PR! Before a maintainer takes a look, it would be really helpful if you could walk through your changes using GitHub's review tools. Please take a moment to:
More information on how to conduct a self review: This helps make the review process smoother and gives us a clearer understanding of your thought process. Once you've added your self-review, we'll continue from our side. Thank you! |
There was a problem hiding this comment.
documented new config options and persistent queue feature
There was a problem hiding this comment.
added required dependencies for bbolt and cron
There was a problem hiding this comment.
added required dependencies for bbolt and cron
There was a problem hiding this comment.
initialize maintenance worker on startup, clean integration
There was a problem hiding this comment.
implemented simple cron worker for cleeanup,configurable schedule and retention
There was a problem hiding this comment.
Comprehensive tests covering job lifecycle and cleanup
There was a problem hiding this comment.
added clean bbolt implementation with proper interface and error handling
Hell1213
left a comment
Hell1213
left a comment
There was a problem hiding this comment.
This PR solves the job persistence issue cleanly. Jobs were getting lost on backend restarts, now they're saved to disk with bbolt and restored automatically. Added a maintenance worker to clean up old logs so disk doesn't fill up. Everything is configurable and backward compatible - existing code keeps working even if the database fails. Tested live and confirmed jobs survive restarts.
|
hey @its-me-abhishek , PR is ready for Review , open to any further changes required. |
|
@Hell1213, the implementation looks great. will review it once locally first and then here. |
| @@ -1,6 +1,6 @@ | |||
| module ccsync_backend | |||
|
|
|||
| go 1.19 | |||
There was a problem hiding this comment.
can this be skipped?
There was a problem hiding this comment.
The Go 1.23 bump is because bbolt v1.4.3 requires it. I can either downgrade bbolt to v1.3.7 (works with Go 1.19) or keep 1.23. Leaning towards downgrading to stay consistent with the codebase. any Thoughts?
| func (mw *MaintenanceWorker) Start() error { | ||
| schedule := os.Getenv("CLEANUP_CRON_SCHEDULE") | ||
| if schedule == "" { | ||
| schedule = "0 0 * * *" |
There was a problem hiding this comment.
maybe provide some more context on these in form of comments and how to update it as well. what can be the other values for this?
There was a problem hiding this comment.
sure , Added detailed comments explaining the cron format and example values. will push changes soon when complete change ready !
|
Hey @its-me-abhishek, should we keep Go 1.23 or downgrade bbolt to v1.3.7 (works with Go 1.19)? The Go 1.23 bump is because bbolt v1.4.3 requires it - the link you mentioned in the issue points to use bbolt v1.4.3. |
2 participants
The current job queue was volatile and jobs would be lost on backend restarts. This implements a bbolt-based persistent queue that stores jobs to disk and restores them on startup. Also added a cron-based maintenance worker that automatically cleans up old completed and failed job logs to prevent unbounded disk usage. All existing functionality is preserved and the new features are fully configurable via environment variables.
Description
Added persistent job queue using bbolt database to store jobs on disk. Jobs now survive backend restarts and are automatically restored on startup. Implemented maintenance worker with cron scheduler to clean up old job logs and prevent disk usage issues.
Checklist
npx prettier --write .(for formatting)gofmt -w .(for Go backend)npm test(for JS/TS testing)Terminal Screenshot
Additional Notes
New environment variables added for configuration:
CLEANUP_CRON_SCHEDULE- Schedule for maintenance worker (default: daily at midnight)CLEANUP_RETENTION_DAYS- How long to keep job logs (default: 7 days)QUEUE_DB_PATH- Database file location (default: /app/data/queue.db)The implementation gracefully falls back to in-memory queue if persistent storage fails, ensuring no breaking changes.