Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -362,3 +362,7 @@ MigrationBackup/
# Fody - auto-generated XML schema
FodyWeavers.xsd
/CoverageReport

# Local environment variables
.env
.env.local
9 changes: 8 additions & 1 deletion Milvaion.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@
<File Path="docs/portaldocs/20-workflows.md" />
<File Path="docs/portaldocs/21-reporter-worker.md" />
<File Path="docs/portaldocs/22-reports.md" />
<File Path="docs/portaldocs/23-local-job-testing.md" />
</Folder>
<Folder Name="/src/">
<Project Path="src/MilvaionUI/" Type="Website" DisplayName="MilvaionUI">
Expand Down Expand Up @@ -165,7 +166,13 @@
<BuildType Solution="TemplateTest|*" Project="Debug" />
</Project>
</Folder>
<Folder Name="/tests/" />
<Folder Name="/tests/">
<Project Path="tests/SampleWorker.Tests/SampleWorker.Tests.csproj">
<BuildType Solution="TemplateTest|*" Project="Debug" />
<Build Solution="Debug|*" Project="false" />
<Build Solution="TemplateTest|*" Project="false" />
</Project>
</Folder>
<Folder Name="/tests/IntegrationTests/">
<Project Path="tests/Milvaion.IntegrationTests/Milvaion.IntegrationTests.csproj">
<BuildType Solution="TemplateTest|*" Project="Release" />
Expand Down
1 change: 1 addition & 0 deletions docs/portaldocs/00-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Welcome to the official Milvaion documentation. This documentation will help you
| [14-built-in-workers.md](14-built-in-workers.md) | Pre-built workers (HTTP Worker, etc.) |
| [20-workflows.md](20-workflows.md) | Build multi-step job pipelines with DAG-based orchestration |
| [21-reporter-worker.md](21-reporter-worker.md) | Automated metric report generation worker |
| [23-local-job-testing.md](23-local-job-testing.md) | Unit test jobs locally without RabbitMQ, Redis, or a live environment |

### Operations Guide

Expand Down
61 changes: 58 additions & 3 deletions docs/portaldocs/06-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ This page documents all configuration options for Milvaion API and Workers.
"AutomaticRecoveryEnabled": true,
"NetworkRecoveryInterval": 10,
"QueueDepthWarningThreshold": 100,
"QueueDepthCriticalThreshold": 500
"QueueDepthCriticalThreshold": 500,
"ManagementEnabled": false,
"ManagementPort": 15672
},
"JobDispatcher": {
"Enabled": true,
Expand Down Expand Up @@ -141,8 +143,10 @@ Open telemetry configurations. Set null or empty via environment variables if yo
| `Heartbeat` | `/` | Heartbeat interval in seconds (0 = disabled). |
| `AutomaticRecoveryEnabled` | `/` | Automatic connection recovery enabled. |
| `NetworkRecoveryInterval` | `/` | Network recovery interval in seconds. |
| `QueueDepthWarningThreshold` | `/` | Queue depth warning threshold. |
| `QueueDepthCriticalThreshold` | `/` | Queue depth critical threshold. |
| `QueueDepthWarningThreshold` | `5000` | Message count that triggers a Warning health status. |
| `QueueDepthCriticalThreshold` | `10000` | Message count that triggers a Critical health status. |
| `ManagementEnabled` | `false` | Enable RabbitMQ Management HTTP API integration for richer queue stats and dynamic queue discovery. |
| `ManagementPort` | `15672` | RabbitMQ Management plugin HTTP port. Used when `ManagementEnabled` is `true`. |

### Job Dispatcher Configuration

Expand Down Expand Up @@ -208,6 +212,57 @@ Open telemetry configurations. Set null or empty via environment variables if yo
| `ConsecutiveFailureThreshold` | `5` | Number of consecutive failures before a job is automatically disabled. Individual jobs can override this with their own AutoDisableThreshold setting. Default: 5 consecutive failures |
| `FailureWindowMinutes` | `60` | Time window in minutes for counting consecutive failures. Failures older than this window don't count towards the threshold. This prevents jobs from being disabled due to old historical failures. Default: 60 minutes (1 hour) |

### BasePath Configuration

Milvaion API supports hosting under a configurable sub-path (e.g. `/milvaion`) so that both the UI and the backend API can be scoped to a URL prefix. This is useful when deploying behind a reverse proxy alongside other services.

| Setting | Default | Description |
|---------|---------|-------------|
| `BasePath` | `""` | URL prefix the application is mounted under. Leave empty to host at the root. Example: `/milvaion` |

When `BasePath` is set:

- The REST API is available at `{BasePath}/api/v1/...` (e.g. `/milvaion/api/v1/jobs`)
- The SignalR hub is available at `{BasePath}/hubs/jobs`
- The Prometheus metrics endpoint is available at `{BasePath}/metrics`
- The Scalar/OpenAPI documentation is available at `{BasePath}/scalar/v1`
- The SPA (UI) is served at `{BasePath}` and all sub-routes fall back to the SPA index

When `BasePath` is empty or not set, the application is hosted at the root (`/`).

#### Example Configuration

```json
{
"MilvaionConfig": {
"BasePath": "/milvaion"
}
}
```

#### Environment Variable Override

```bash
MilvaionConfig__BasePath=/milvaion
```

#### Docker / docker-compose

```yaml
services:
milvaion-api:
image: milvasoft/milvaion:latest # pull from Docker Hub, no rebuild needed
environment:
- MilvaionConfig__BasePath=/milvaion
```

To revert to root hosting, remove the override or set it to empty:

```yaml
environment:
- MilvaionConfig__BasePath=
```

---

## Worker Configuration
Expand Down
76 changes: 76 additions & 0 deletions docs/portaldocs/10-monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -741,6 +741,82 @@ docker exec milvaion-rabbitmq rabbitmqctl list_connections

---

### Queue Depth Monitoring (`IQueueDepthMonitor`)

Milvaion includes a built-in `IQueueDepthMonitor` service that tracks queue depths and health status. When the **RabbitMQ Management HTTP API** is enabled, it provides richer statistics — including unacknowledged message counts — and automatically discovers **dynamic consumer queues** (e.g. `scheduled_jobs_queue.SampleWorker`) that are created at runtime when workers bind to the exchange.

#### Configuration

Enable Management API integration in `appsettings.json`:

```json
{
"MilvaionConfig": {
"RabbitMQ": {
"ManagementEnabled": true,
"ManagementPort": 15672
}
}
}
```

| Setting | Default | Description |
|---------|---------|-------------|
| `ManagementEnabled` | `false` | Enable RabbitMQ Management HTTP API integration |
| `ManagementPort` | `15672` | Management plugin HTTP port |
| `QueueDepthWarningThreshold` | `5000` | Message count that triggers a Warning health status |
| `QueueDepthCriticalThreshold` | `10000` | Message count that triggers a Critical health status |

> **Note:** The Management API uses the same `Username` and `Password` credentials configured under `MilvaionConfig:RabbitMQ`.

#### How it Works

| Scenario | Behavior |
|----------|----------|
| `ManagementEnabled: true` | Uses `GET /api/queues/{vhost}/{queue}` for individual queue stats |
| `ManagementEnabled: true` | Uses `GET /api/queues/{vhost}` to **discover all queues**, including dynamic routing-key queues |
| `ManagementEnabled: false` or API unavailable | Falls back to AMQP passive queue declare (core queues only, `MessagesUnacknowledged` will be `0`) |

#### Queue Health Statuses

| Status | Condition |
|--------|-----------|
| `Healthy` | Message count below warning threshold |
| `Warning` | Message count ≥ `QueueDepthWarningThreshold` |
| `Critical` | Message count ≥ `QueueDepthCriticalThreshold` |
| `Unavailable` | Queue or broker unreachable |

#### API Endpoints

Retrieve queue stats via the Admin API:

```bash
# All queues (includes dynamic consumer queues when Management API is enabled)
curl http://localhost:5000/api/v1/admin/queues

# Single queue depth
curl http://localhost:5000/api/v1/admin/queues/{queueName}
```

**Example response for a single queue:**

```json
{
"queueName": "scheduled_jobs_queue",
"messageCount": 42,
"consumerCount": 3,
"messagesReady": 40,
"messagesUnacknowledged": 2,
"healthStatus": "Healthy",
"healthMessage": "Queue operating normally",
"timestamp": "2026-01-14T18:10:00.000Z"
}
```

> **Dynamic queues:** Without Management API enabled, only the 8 core queues (`jobs`, `worker_logs`, `worker_heartbeat`, `worker_registration`, `status_updates`, `failed_occurrences`, `external_job_registration`, `external_job_occurrence`) are monitored. With it enabled, all queues in the vhost — including per-worker routing-key queues like `scheduled_jobs_queue.SampleWorker` — are returned automatically.

---

## Redis Monitoring

### Key Metrics
Expand Down
Loading