Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ pub struct MyResult { ... }

Feature flags (3 simple options):
- `lib`: Complete library (database + all lenses + display)
- `server`: WebSocket server (implies lib)
- `server`: HTTP/SSE server (implies lib)
- `cli`: Full CLI binary (implies lib and server)

## Project Structure
Expand All @@ -118,8 +118,11 @@ src/
│ ├── search/ # BGP search
│ ├── rpki/ # RPKI validation
│ └── inspect/ # Unified inspection
├── server/ # WebSocket server (requires `server` feature)
│ └── handlers/ # Method handlers
├── server/ # HTTP/SSE server (requires `server` feature)
│ ├── http.rs # REST router, API error types
│ ├── search.rs # SSE search streaming handler
│ ├── auth.rs # Token auth middleware
│ └── rest/ # REST endpoint handlers (time, ip, rpki, etc.)
└── bin/
├── monocle.rs # CLI entry point
└── commands/ # CLI command handlers
Expand Down Expand Up @@ -209,5 +212,5 @@ impl<'a> MyLens<'a> {

- `ARCHITECTURE.md` - Overall project structure
- `DEVELOPMENT.md` - Adding new lenses and fixing bugs
- `src/server/README.md` - WebSocket API specification
- `src/server/README.md` - HTTP/SSE API specification
- `examples/README.md` - Usage examples by feature tier
104 changes: 55 additions & 49 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Monocle Architecture

This document describes the architecture of the `monocle` project: a BGP information toolkit that can be used as both a Rust library, a command-line application, and a WebSocket server.
This document describes the architecture of the `monocle` project: a BGP information toolkit that can be used as both a Rust library, a command-line application, and an HTTP/SSE server.

## Goals and Design Principles

Expand Down Expand Up @@ -119,23 +119,21 @@ src/
│ └── time/ # Time parsing and formatting
│ └── mod.rs
├── server/ # WebSocket server (requires `server` feature)
│ ├── mod.rs # Server startup, handle_socket
│ ├── protocol.rs # Core protocol types (RequestEnvelope, ResponseEnvelope)
│ ├── router.rs # Router + Dispatcher
│ ├── handler.rs # WsMethod trait, WsContext
│ ├── sink.rs # WsSink (transport primitive)
│ ├── op_sink.rs # WsOpSink (terminal-guarded)
│ ├── operations.rs # Operation registry for cancellation
│ └── handlers/ # Method handlers
├── server/ # HTTP/SSE server (requires `server` feature)
│ ├── mod.rs # Server state, startup, /health, auth wiring
│ ├── http.rs # REST router, API error types, system/info
│ ├── search.rs # SSE search streaming handler
│ ├── auth.rs # Token auth middleware
│ └── rest/ # REST endpoint handlers
│ ├── mod.rs
│ ├── inspect.rs # inspect.query, inspect.refresh
│ ├── rpki.rs # rpki.validate, rpki.roas, rpki.aspas
│ ├── as2rel.rs # as2rel.search, as2rel.relationship
│ ├── database.rs # database.status, database.refresh
│ ├── parse.rs # parse.start, parse.cancel (streaming)
│ ├── search.rs # search.start, search.cancel (streaming)
│ └── ...
│ ├── time.rs # time/parse
│ ├── country.rs # country/lookup
│ ├── ip.rs # ip/lookup, ip/public
│ ├── rpki.rs # rpki/roa/lookup, rpki/aspa/lookup, rpki/roa/validate
│ ├── pfx2as.rs # pfx2as/lookup
│ ├── as2rel.rs # as2rel/search, as2rel/relationship, as2rel/refresh
│ ├── inspect.rs # inspect/query
│ └── database.rs # database/status, database/refresh, inspect/refresh
└── bin/
├── monocle.rs # CLI entry point
Expand Down Expand Up @@ -190,23 +188,28 @@ The Pfx2as repository (`Pfx2asRepository`) provides:

Note: The file-based cache has been removed; all pfx2as data now uses SQLite.

### `server` - WebSocket API

The WebSocket server (`monocle server`) provides programmatic access to monocle functionality:

- **Protocol**: JSON-RPC style with request/response envelopes
- **Streaming**: Progress updates for long-running operations (parse, search)
- **Terminal guard**: `WsOpSink` ensures exactly one terminal response per operation
- **Operation tracking**: `OperationRegistry` for cancellation support via `op_id`
- **DB-first policy**: Queries read from local SQLite cache

Available method namespaces:
- `system.*`: Server introspection (info, methods)
- `time.*`, `ip.*`, `country.*`: Utility lookups
- `rpki.*`, `as2rel.*`, `pfx2as.*`: BGP data queries
- `inspect.*`: Unified AS/prefix inspection
- `parse.*`, `search.*`: Streaming MRT operations
- `database.*`: Database management
### `server` - HTTP/SSE API

The HTTP/SSE server (`monocle server`) provides programmatic access to monocle functionality:

- **Protocol**: HTTP REST with JSON request/response bodies
- **Streaming**: SSE (`text/event-stream`) for search with progress, element batches, and terminal events
- **Cancellation**: Client closes HTTP connection; server detects drop via `Arc<AtomicBool>`
- **Backpressure**: Bounded mpsc channel (capacity 32); element batches never dropped
- **Terminal invariant**: Exactly one terminal event (`completed`, `cancelled`, or `error`)
- **Auth**: Optional token-based middleware (`Authorization: Bearer <token>`)
- **DB access**: Per-request `MonocleDatabase` in `spawn_blocking` (connection is `!Send`)

Available endpoint groups:
- `/health`: Health check (always open)
- `/api/v1/system/*`: Server introspection
- `/api/v1/search/*`: SSE streaming BGP search
- `/api/v1/time/*`, `/api/v1/ip/*`, `/api/v1/country/*`: Utility lookups
- `/api/v1/rpki/*`: RPKI ROA/ASPA lookup and validation
- `/api/v1/as2rel/*`: AS relationship queries
- `/api/v1/pfx2as/*`: Prefix-to-ASN mapping
- `/api/v1/inspect/*`: Unified AS/prefix inspection
- `/api/v1/database/*`: Database status and refresh

## Module Architecture

Expand Down Expand Up @@ -309,13 +312,14 @@ The CLI should not duplicate core logic. It should:
3. Application calls lens methods
4. Application consumes typed results directly, or uses `OutputFormat` to format for display

### WebSocket flow (conceptual)
### HTTP/SSE flow (conceptual)

1. Client connects and sends JSON request envelope
2. Router dispatches to appropriate handler
3. Handler creates lens with database reference
4. Lens executes operation (may send progress via `WsOpSink`)
5. Handler sends terminal result/error via `WsOpSink`
1. Client sends HTTP request (POST with JSON body, or GET)
2. Axum router dispatches to handler
3. Handler opens `MonocleDatabase` in `spawn_blocking` (for DB-backed endpoints)
4. Handler calls lens methods, formats result as JSON response
5. For search: handler returns `Sse` stream; worker sends events via bounded channel
6. Client disconnect cancels the search worker via `Arc<AtomicBool>`

## Feature Flags

Expand All @@ -332,15 +336,15 @@ cli (default)

**Quick Guide:**
- **Need the CLI binary?** Use `cli` (includes everything)
- **Need WebSocket server without CLI?** Use `server` (includes lib)
- **Need HTTP/SSE server without CLI?** Use `server` (includes lib)
- **Need only library/data access?** Use `lib` (database + all lenses + display)

### Feature Descriptions

| Feature | Description | Key Dependencies |
|---------|-------------|------------------|
| `lib` | Complete library: database + all lenses + display | `rusqlite`, `bgpkit-parser`, `bgpkit-broker`, `tabled`, etc. |
| `server` | WebSocket server (implies `lib`) | `axum`, `tokio`, `serde_json` |
| `server` | HTTP/SSE server (implies `lib`) | `axum`, `tokio`, `serde_json` |
| `cli` | Full CLI binary with progress bars (implies `lib` and `server`) | `clap`, `indicatif` |

### Use Case Scenarios
Expand All @@ -366,28 +370,30 @@ let lens = InspectLens::new(&db);
let result = lens.query("AS13335", &InspectQueryOptions::default())?;
```

#### Scenario 2: Library with WebSocket Server
#### Scenario 2: Library with HTTP/SSE Server
**Features**: `server`

Use when building applications that need:
- Everything in `lib`
- WebSocket server for remote API access
- HTTP/SSE server for remote API access

```toml
monocle = { version = "1.1", default-features = false, features = ["server"] }
```

```rust
use monocle::config::MonocleConfig;
use monocle::server::start_server;

// Start WebSocket server on default port
start_server("127.0.0.1:3000").await?;
let config = MonocleConfig::new(&None)?;
// Start HTTP/SSE server on default port
start_server(config).await?;
```

#### Scenario 3: CLI Binary (Default)
**Features**: `cli` (default)

The full CLI binary with all features, WebSocket server, and terminal UI:
The full CLI binary with all features, HTTP/SSE server, and terminal UI:

```toml
monocle = "1.1"
Expand All @@ -407,7 +413,7 @@ All of these combinations compile successfully:
|-------------|----------|
| (none) | Config types only, no functionality |
| `lib` | Full library functionality |
| `server` | Library + WebSocket server |
| `server` | Library + HTTP/SSE server |
| `cli` | Full CLI (includes everything) |

### Feature Dependencies
Expand All @@ -422,7 +428,7 @@ When you enable a higher-tier feature, lower-tier features are automatically inc
- `README.md` — user-facing CLI and library overview
- `CHANGELOG.md` — version history and breaking changes
- `DEVELOPMENT.md` — contributor guide for adding lenses and fixing bugs
- `src/server/README.md` — WebSocket API protocol specification
- `src/server/README.md` — HTTP/SSE API specification
- `src/database/README.md` — database module notes
- `src/lens/README.md` — lens module patterns and conventions
- `examples/README.md` — example code organized by feature tier
83 changes: 82 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@ All notable changes to this project will be documented in this file.

## Unreleased changes

### Breaking Changes

* Replaced WebSocket server with HTTP/SSE service. The `server` subcommand now
starts an HTTP server with REST endpoints and SSE search streaming instead of
a WebSocket server. All WebSocket modules (`protocol`, `handler`, `sink`,
`op_sink`, `router`, `operations`, `handlers/*`) have been removed.
* Updated `axum` to 0.8 and `tower-http` to 0.6.
* Removed `uuid` and `async-trait` dependencies (no longer needed without
WebSocket operation tracking).
* Updated `ServerArgs` CLI flags: removed `--data-dir`, `--max-concurrent-ops`,
`--max-message-size`, `--connection-timeout-secs`, `--ping-interval-secs`;
added `--max-search-batch-size`, `--max-search-results`, `--search-timeout-secs`.
`--address` and `--port` are now optional (default to config values).

### Performance Improvements

* Optimized database refresh (bulk insert) performance by 21–49% across all
Expand Down Expand Up @@ -33,7 +47,59 @@ All notable changes to this project will be documented in this file.

### New Features

* Added `--filter-file` (JSON) and `--prefix-file` (newline text) flags to `monocle parse`
* Added HTTP/SSE server with three MVP endpoints:
- `GET /health` — health check for container orchestration
- `GET /api/v1/system/info` — server metadata and endpoint list
- `POST /api/v1/search/stream` — SSE streaming BGP search with progress,
element batches, cancellation on disconnect, and max-results limit
* Added server configuration fields to `MonocleConfig`: `server_address`,
`server_port`, `server_max_search_batch_size`, `server_max_search_results`,
`server_search_timeout_secs`. All configurable via `monocle.toml` and
`MONOCLE_*` environment variables.
* SSE search streaming uses sequential file processing with `Arc<AtomicBool>`
cancellation — no new lens method needed. The server calls existing
`SearchFilters` utility methods (`to_broker_items`, `to_parser`) directly.
* Bounded mpsc channel (capacity 32) with backpressure: element batches are
never dropped; progress events may be coalesced under backpressure.
* Terminal event invariant: exactly one of `completed`, `cancelled`, or `error`.
* Added Phase 2 REST API endpoints:
- Tier 1 (stateless): `POST /time/parse`, `POST /country/lookup`,
`POST /ip/lookup`, `GET /ip/public`
- Tier 2 (DB read-only, cache-only): `GET /database/status`,
`GET /rpki/roa/lookup`, `GET /rpki/aspa/lookup`, `GET /pfx2as/lookup`,
`GET /as2rel/relationship`, `POST /as2rel/search`
- Tier 3 (DB refresh): `POST /database/refresh`, `POST /inspect/refresh`,
`POST /as2rel/refresh`
- Tier 4 (composite, cache-only): `POST /rpki/roa/validate`,
`POST /inspect/query`
ASPA validation (`/rpki/aspa/validate`) is deferred — it requires full
AS path validation combined with AS relationship inference data (as2rel),
not a simple membership check.
* Added token-based auth middleware (Phase 3):
- Config fields `server_auth_enabled` (default: false) and
`server_auth_token` in `MonocleConfig`
- CLI flags `--auth-enabled` and `--auth-token`
- Env vars `MONOCLE_SERVER_AUTH_ENABLED` and `MONOCLE_SERVER_AUTH_TOKEN`
- When enabled, `/api/v1/*` requires `Authorization: Bearer <token>`;
`/health` stays open for container health checks
- Server refuses to start if auth is enabled but token is empty
* Added CLI remote search mode (Phase 4):
- `--remote-url` flag on `monocle search` sends the query to a remote
Monocle HTTP service instead of running locally
- `--remote-token` provides the Bearer token for auth
- Consumes SSE stream and formats results with existing CLI output
formatters (PSV, JSON, table, markdown)
* Added Docker Compose deployment config (Phase 5):
- Updated `Dockerfile` with volume mounts for `/data/monocle` and
`/cache/monocle`
- `docker-compose.yml` with health check, persistent volumes, and
env-based configuration
- `monocle.toml.example` showing all service configuration options
All DB-backed endpoints return `NOT_INITIALIZED` (HTTP 503) if required
data is missing. No refresh policy knobs — users refresh via explicit
`/refresh` endpoints.

### Added `--filter-file` (JSON) and `--prefix-file` (newline text) flags to `monocle parse`
and `monocle search` for loading large filter sets from files. File filters merge with
CLI flags — union within each dimension (OR), AND across dimensions. Supports the
RIB-extract → filter-updates workflow at scale (#117).
Expand All @@ -43,6 +109,21 @@ All notable changes to this project will be documented in this file.
* Fixed `--time-format rfc3339` being ignored for `json`, `json-line`, and `json-pretty`
output formats. JSON output now honors `--time-format`: `unix` (default) emits a numeric
`timestamp` field (backward compatible); `rfc3339` emits an RFC 3339 string (#123).
* Validated `prefix` input in `pfx2as_lookup` and `roa_lookup` REST endpoints before
spawning blocking tasks, so invalid prefixes return 400 instead of 500.
* Remote search client now exits with a non-zero status when the SSE stream ends
with an `error` or `cancelled` event, or when the connection drops without a
`completed` event.
* Docker runtime image now runs as a dedicated non-root `monocle` user.
* Remote search now applies `--filter-file` / `--prefix-file` merging and filter
validation before dispatching, matching local search behavior.
* Auth middleware now accepts case-insensitive `Bearer` scheme tokens per RFC 7235.
* `POST /api/v1/database/refresh` with `source=pfx2as` now returns HTTP 501
instead of 200, so automation can detect the operation is not implemented.
* SSE search now returns 400 on invalid `peer_ip` values instead of silently
dropping them.
* `GET /api/v1/rpki/roa/lookup` now applies AND semantics when both `prefix`
and `asn` are provided (filters covering ROAs by origin ASN).

## v1.3.0 - 2026-05-27

Expand Down
Loading
Loading