Skip to content

feat: add scheduled crawls (closes #14) - #55

Open
barunTayenjam wants to merge 1 commit into
StJudeWasHere:mainfrom
barunTayenjam:feat/scheduled-crawls
Open

barunTayenjam wants to merge 1 commit into
StJudeWasHere:mainfrom
barunTayenjam:feat/scheduled-crawls

Conversation

@barunTayenjam

Copy link
Copy Markdown

Closes #14.

Projects can now have a crawl schedule (hourly, daily or weekly), selected when the project is created or edited. While the server is running, a scheduler ticks every minute and starts a new crawl for every scheduled project whose next run time has passed.

Changes

  • migration 0076: projects.schedule_interval + projects.next_run
  • models/repository: schedule fields, FindScheduledProjects, UpdateProjectNextRun
  • SchedulerService (stdlib time.Ticker only, injected clock for tests); next_run always advances (even on error) so a failing project never blocks the schedule; BasicAuth projects are skipped as their credentials are only provided interactively and are not stored
  • SaveProject sets next_run; UpdateProject recomputes it only when the interval changes
  • [scheduler] enabled config (SEONAUT_SCHEDULER_ENABLED), default on
  • project add/edit forms: schedule select + validation error
  • translations: schedule strings in en, es, nl, uk, fa
  • docs: README + INSTALL.md (scheduled crawls section)

Design decisions

  • preset intervals (hourly/daily/weekly), not cron expressions: no new dependency, matches the "given time intervals" ask in Scheduling crawling job #14
  • no catch-up storm on restart: a project whose time passed while down runs at most once on the next tick, then resumes its interval
  • notification/report delivery is deliberately out of scope here; tracked in Notification feature. #15

Verification

  • go test -race -parallel 1 -short ./internal/... pass, go vet clean, go build -race clean, gofmt clean on touched files
  • new tests: TestRunOnce* (due/not-due/bogus-interval, error advances, BasicAuth skip), TestSchedule (interval validation + next_run), config scheduler assertion
  • end-to-end against MySQL 8.4 + local test site: manual crawl produces report/CSV; forcing next_run past due triggers a scheduled crawl within one tick and advances next_run +1h; invalid interval rejected with form error; edit form preselects interval; restart causes no extra crawls

Projects can now have a crawl schedule (hourly, daily or weekly),
selected when the project is created or edited. While the server is
running, a scheduler ticks every minute and starts a new crawl for
every scheduled project whose next run time has passed.

- migration 0076: projects.schedule_interval + projects.next_run
- models/repository: schedule fields, FindScheduledProjects,
  UpdateProjectNextRun
- SchedulerService with injected clock; next_run always advances
  (even on error) so a failing project never blocks the schedule;
  BasicAuth projects are skipped as their credentials are not stored
- SaveProject sets next_run; UpdateProject recomputes it only when
  the interval changes
- [scheduler] enabled config (SEONAUT_SCHEDULER_ENABLED), default on
- project add/edit forms: schedule select + validation error
- translations: schedule strings in en, es, nl, uk, fa
- docs: README + INSTALL.md (scheduled crawls section)

Verified end-to-end (MySQL 8.4 + local test site): manual crawl
produces report/CSV, forcing next_run past due triggers a scheduled
crawl within one tick and advances next_run, invalid interval is
rejected, restart causes no catch-up burst.
@barunTayenjam

Copy link
Copy Markdown
Author

CI note (for maintainers): both PR workflows are gated on first-time contributor approval (action_required). I verified the equivalent steps locally on this branch:

  • go vet ./internal/... — clean
  • go test -race -parallel 1 -short ./internal/... — all packages pass (incl. new TestRunOnce*, TestSchedule)
  • backend build replicated (CGO_ENABLED=0 go build ./cmd/server/main.go) — OK
  • frontend step replicated in node:18-alpine3.18 (same esbuild command as Dockerfile) — OK
  • manual e2e (MySQL 8.4 + local test site): scheduled crawl fires within one tick when next_run is past due, next_run advances, BasicAuth projects skipped, restart causes no catch-up crawls.

Happy to address any review feedback or rebase as needed.

@barunTayenjam

Copy link
Copy Markdown
Author

Closing for now while I finish local verification. Will reopen once tested. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduling crawling job

1 participant