feat: replace WebSocket server with HTTP/SSE service#129
Merged
Conversation
Replace the WebSocket-oriented server with a simpler HTTP service using direct SSE streaming for search and REST endpoints for non-streaming commands. MVP endpoints: - GET /health — container health check - GET /api/v1/system/info — server metadata and endpoint list - POST /api/v1/search/stream — SSE streaming BGP search with progress, element batches, cancellation on disconnect, max-results limit, and bounded channel backpressure Key design decisions: - Sequential file processing (not Rayon) for clean cancellation and ordered batches; parallelism deferred to future optimization - Server calls existing SearchFilters utility methods (to_broker_items, to_parser) directly — no new lens method needed - Cancellation via Arc<AtomicBool> set when SSE response is dropped - Independent wire DTOs (SearchStreamFilters) isolate API contract from internal type refactoring - Terminal event invariant: exactly one of completed/cancelled/error - Bounded mpsc channel (capacity 32): elements never dropped, progress events may be coalesced under backpressure Server config added to MonocleConfig: server_address, server_port, server_max_search_batch_size, server_max_search_results, server_search_timeout_secs. All configurable via monocle.toml and MONOCLE_* env vars. Removed all WebSocket modules (protocol, handler, sink, op_sink, router, operations, handlers/*) and ws_client_all example. Updated axum to 0.8, tower-http to 0.6; removed uuid and async-trait dependencies. See SSE_SERVICE_OVERHAUL_DESIGN.md for full design rationale.
Add 15 non-streaming REST endpoints organized in four tiers: Tier 1 — Stateless (no database): - POST /api/v1/time/parse - POST /api/v1/country/lookup - POST /api/v1/ip/lookup - GET /api/v1/ip/public Tier 2 — Database read-only (cache-only, return NOT_INITIALIZED if missing): - GET /api/v1/database/status - GET /api/v1/rpki/roa/lookup - GET /api/v1/rpki/aspa/lookup - GET /api/v1/pfx2as/lookup - GET /api/v1/as2rel/relationship - POST /api/v1/as2rel/search Tier 3 — Database refresh (explicit, no progress streaming): - POST /api/v1/database/refresh - POST /api/v1/inspect/refresh - POST /api/v1/as2rel/refresh Tier 4 — Composite query (cache-only): - POST /api/v1/rpki/roa/validate - POST /api/v1/rpki/aspa/validate - POST /api/v1/inspect/query All DB-backed handlers use spawn_blocking since MonocleDatabase is !Send. DB is opened per-request from config.data_dir. All DB-backed endpoints return HTTP 503 NOT_INITIALIZED if required data is missing — no auto_refresh/force_refresh knobs (deferred per design).
ASPA validation against a single customer-provider pair is insufficient. Proper ASPA validation requires checking full AS paths against ASPA records combined with AS relationship inference data (as2rel). The simple membership check has been removed and the endpoint is not exposed. The handler code is replaced with a TODO comment referencing the future design. ASPA lookup (/rpki/aspa/lookup) remains available.
When server_auth_enabled is true, requests to /api/v1/* require Authorization: Bearer <token>. /health stays open for container health checks. Server refuses to start if auth is enabled but token is empty. Config: server_auth_enabled, server_auth_token (monocle.toml or MONOCLE_SERVER_AUTH_ENABLED / MONOCLE_SERVER_AUTH_TOKEN env vars). CLI: --auth-enabled, --auth-token flags.
When --remote-url is passed to 'monocle search', the CLI POSTs the search filters to a remote Monocle HTTP service and consumes the SSE response stream. Results are formatted using the same output formatters as local search (PSV, JSON, JSON-line, table, markdown). --remote-token provides the Bearer token for auth-enabled servers. Added reqwest as a CLI dependency for the HTTP client. Added tokio-util 'io' feature for StreamReader to convert the response byte stream into an async line reader for SSE parsing.
Update Dockerfile with volume mounts for /data/monocle (SQLite database) and /cache/monocle (MRT file cache). Set MONOCLE_DATA_DIR to /data/monocle by default. Add docker-compose.yml with: - Persistent named volumes for data and cache - Health check via /health endpoint - Env-based configuration for all service settings - Auth config commented out for easy enablement Add monocle.toml.example showing all available configuration options including HTTP service, auth, cache TTLs, and RPKI RTR endpoint.
Replace WebSocket server documentation with HTTP/SSE service docs covering: - Updated 'monocle server' CLI help text and endpoint table - Remote search mode (--remote-url) documentation in search section - Docker deployment instructions updated for HTTP server - Server README expanded with all REST endpoint examples (curl commands), auth configuration, Docker deployment, CLI remote search, and architecture diagram showing the new src/server/rest/ module layout
Update README.md and AGENTS.md to replace all WebSocket references with HTTP/SSE. Update server module description in AGENTS.md project structure to show the new src/server/ layout (http.rs, search.rs, auth.rs, rest/). Fix server README link description from 'WebSocket API specification' to 'HTTP/SSE API specification' in both files.
Update all remaining WebSocket references across the codebase to reflect the HTTP/SSE architecture: - src/lens/mod.rs, as2rel/args.rs, country/mod.rs: doc comments - src/lens/README.md: server description, handler paths, related docs - src/database/README.md: refresh example, related docs link - ARCHITECTURE.md: project structure tree, server module description, HTTP/SSE flow, feature descriptions, start_server example - DEVELOPMENT.md: project structure tree, file location table, feature descriptions, new lens checklist - examples/README.md: remove ws_client_all, add sse_browser_test.html - examples/ws_client_all.js: deleted (stale WebSocket client demo)
There was a problem hiding this comment.
Pull request overview
This PR migrates Monocle’s server feature from a WebSocket-based JSON-RPC style API to an HTTP service that uses REST for non-streaming operations and SSE (text/event-stream) for streaming search results. It aligns the server surface area with the documented “DB-first/cache-only” behavior and adds a CLI client mode for remote streaming search.
Changes:
- Replaced the WebSocket server stack (protocol/router/handlers/op tracking) with an Axum 0.8 HTTP router, REST endpoints, and an SSE search streaming handler.
- Added server configuration for bind/auth/search limits and wired optional Bearer-token middleware for
/api/v1/*. - Added a CLI “remote search” client that consumes the SSE stream and formats results with existing output formatters; updated Docker + docs accordingly.
Reviewed changes
Copilot reviewed 57 out of 58 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/server/sink.rs | Removed WebSocket sink abstraction (no longer needed). |
| src/server/router.rs | Removed WS method registry + dispatcher. |
| src/server/query.rs | Removed WS query helper types. |
| src/server/protocol.rs | Removed WS request/response envelope and error/progress protocol types. |
| src/server/operations.rs | Removed WS operation registry/cancellation tracking. |
| src/server/op_sink.rs | Removed WS operation-scoped sink and terminal-guard logic. |
| src/server/handler.rs | Removed WS method trait/context and handler infrastructure. |
| src/server/handlers/mod.rs | Removed WS handler module registry/exports. |
| src/server/handlers/system.rs | Removed WS system handlers. |
| src/server/handlers/time.rs | Removed WS time handler. |
| src/server/handlers/country.rs | Removed WS country handler. |
| src/server/handlers/ip.rs | Removed WS IP handlers. |
| src/server/handlers/pfx2as.rs | Removed WS pfx2as handler. |
| src/server/handlers/database.rs | Removed WS database handlers. |
| src/server/mod.rs | Replaced WS server startup with HTTP server startup + /health + auth wiring. |
| src/server/http.rs | Added shared HTTP API error types and /api/v1 router (REST + SSE route wiring). |
| src/server/auth.rs | Added Bearer token middleware for /api/v1/*. |
| src/server/rest/mod.rs | Added REST handler module organization. |
| src/server/rest/time.rs | Added POST /api/v1/time/parse. |
| src/server/rest/country.rs | Added POST /api/v1/country/lookup. |
| src/server/rest/ip.rs | Added POST /api/v1/ip/lookup and GET /api/v1/ip/public. |
| src/server/rest/rpki.rs | Added RPKI REST endpoints for ROA/ASPA lookup + ROA validate. |
| src/server/rest/pfx2as.rs | Added GET /api/v1/pfx2as/lookup. |
| src/server/rest/as2rel.rs | Added AS2REL REST endpoints (search/relationship/refresh). |
| src/server/rest/inspect.rs | Added POST /api/v1/inspect/query. |
| src/server/rest/database.rs | Added database status + refresh endpoints and inspect refresh endpoint. |
| src/server/search.rs | Added SSE streaming search handler, wire DTOs, backpressure, cancellation, terminal events. |
| src/lib.rs | Updated public exports to reflect the new HTTP/SSE server API surface. |
| src/config.rs | Added server bind/auth/search-limit config fields + defaults + parsing. |
| src/lens/mod.rs | Updated docs to reflect HTTP/SSE (instead of WS) as a consumer. |
| src/lens/README.md | Updated lens-layer docs for the new server architecture. |
| src/lens/country/mod.rs | Updated docs to refer to HTTP/SSE usage. |
| src/lens/as2rel/args.rs | Updated docs to refer to HTTP/SSE usage. |
| src/database/README.md | Updated docs to reference HTTP API in server mode. |
| src/bin/monocle.rs | Updated server subcommand flags and startup to use new server config fields. |
| src/bin/commands/mod.rs | Registered the new remote-search CLI module. |
| src/bin/commands/search.rs | Added --remote-url/--remote-token path and a wrapper to run remote search client. |
| src/bin/commands/search_remote.rs | New remote SSE client that formats streamed BgpElem results. |
| src/bin/commands/config.rs | Updated “server defaults” display to reflect new server config fields. |
| README.md | Updated user-facing docs for HTTP/SSE server + remote search. |
| monocle.toml.example | Added example config for HTTP/SSE service settings. |
| examples/ws_client_all.rs | Removed WS example client. |
| examples/ws_client_all.js | Removed WS example client. |
| examples/sse_browser_test.html | Added browser-based SSE streaming test page. |
| examples/README.md | Updated examples list to reflect SSE example. |
| Dockerfile | Updated container entrypoint to run monocle server and added server env defaults. |
| docker-compose.yml | Added service-oriented compose config with /health check + persistent volumes. |
| ARCHITECTURE.md | Updated architecture docs to describe HTTP/SSE server design. |
| DEVELOPMENT.md | Updated contributor docs for REST/SSE handlers instead of WS handlers. |
| AGENTS.md | Updated agent/developer docs for HTTP/SSE server layout. |
| CHANGELOG.md | Added breaking change entry for server migration + new features summary. |
| Cargo.toml | Updated dependencies/features: axum 0.8, tower-http 0.6, added reqwest/tokio-stream, removed ws deps. |
| Cargo.lock | Locked updated dependency graph for the HTTP/SSE migration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Resolve merge conflicts between SSE service overhaul and db-refresh-optimization: Cargo.toml: keep database example from sse, add db_refresh_bench from main, remove stale ws_client_all reference (file deleted in sse overhaul). CHANGELOG.md: merge Breaking Changes (SSE) + Performance/Code Improvements (refresh) + New Features (SSE) under Unreleased changes. Review fix: remove unnecessary As2relSortOrder re-sort in AS2REL search endpoint that overrode the user's sort_by_asn flag. Review fix: remove misleading /rpki/aspa/validate endpoint from rpki module docs (deferred feature). Review fix: handle pfx2as refresh early in database_refresh with a clear message instead of returning a 500 InternalError from spawn_blocking. Review fix: install curl in Docker runtime image for healthcheck. Review fix: use constant-time byte comparison for bearer token auth to reduce timing side-channels. Review fix: update database README to note pfx2as HTTP API refresh is not yet implemented.
- Parse elem_type string into ParseElemType in SSE search DTO instead of silently discarding it (fixes wire contract mismatch). - Count files processed with zero matches as successful_files (fixes success/failure accounting skew). - Trim auth token before is_empty() check in server startup (prevents whitespace-only tokens from passing validation). - Pass elem_type from local filters through to remote search request (fixes remote/local behavioral difference). - Move pfx2as mode validation before spawn_blocking so invalid input returns 400 (client error) not 500 (server error).
- Validate prefix string in pfx2as_lookup and roa_lookup before spawn_blocking so invalid prefixes return 400 (client error) not 500. - Remote search client now exits with error on / terminal events or unexpected stream end, instead of returning Ok. - Docker runtime image runs as dedicated non-root user.
- Move remote-url dispatch after filter-file/prefix-file merging and filters.validate() so remote search respects file-based filters and rejects invalid input like local search does. - Accept case-insensitive Bearer scheme in auth middleware per RFC 7235. - Remove unnecessary mut on reqwest client builder in remote search. - Return HTTP 501 for pfx2as refresh via API so automation can detect the operation is not implemented.
- Return 400 on invalid peer_ip in SSE search instead of silently dropping the filter value. - ROA lookup now applies AND semantics when both prefix and asn are provided (intersect covering ROAs with origin ASN).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the WebSocket server with an HTTP service using SSE streaming for search and REST endpoints for all other capabilities.
Resolves #126, resolves #118.
Changes
Server architecture
GET /health— health check (always open)GET /api/v1/system/info— server metadata and endpoint listPOST /api/v1/search/stream— SSE streaming BGP search with progress, element batches, cancellation on disconnect, and max-results limitArc<AtomicBool>cancellation — no new lens method neededcompleted,cancelled, orerroruuidandasync-traitdepsREST API endpoints
/time/parse,/country/lookup,/ip/lookup,/ip/public/database/status,/rpki/roa/lookup,/rpki/aspa/lookup,/pfx2as/lookup,/as2rel/relationship,/as2rel/search/database/refresh,/inspect/refresh,/as2rel/refresh/rpki/roa/validate,/inspect/query503 NOT_INITIALIZEDif data is missing (no auto-refresh; use explicit/refreshendpoints)Auth
Authorization: Bearer <token>server_auth_enabled/server_auth_token(config file, env vars, or CLI flags)/healthstays open for container health checksCLI remote search
monocle search --remote-url <server>runs search against a remote Monocle service instead of locally--remote-tokenfor auth-enabled serversDocker deployment
Dockerfilewith volume mounts for/data/monocleand/cache/monocledocker-compose.ymlwith health check, persistent volumes, and env-based configmonocle.toml.exampleshowing all configuration optionsASPA validation
Documentation
examples/ws_client_all.js; addedexamples/sse_browser_test.htmlTest plan
cargo fmt --checkcleancargo clippy --all-features -- -D warningscleancargo test --all-features— 267 tests pass/health,/api/v1/system/info,/api/v1/search/stream(SSE with batching, max_results, cancellation)--remote-url