Skip to content

Commit b5dbb23

Browse files
committed
docs: publish NetOpsBench v0.2 validation
1 parent cefbf61 commit b5dbb23

16 files changed

Lines changed: 1636 additions & 22 deletions

‎README.md‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ It is built for researchers and engineers who want to compare LLM-backed, symbol
4747

4848
## News
4949

50+
- **2026-08**: 🚀 **NetOpsBench v0.2.0** - Large-topology benchmark release.
51+
- Add Xlarge CLOS and Fat-tree K=8/K=12 profiles, with 70 generated cases per large topology.
52+
- Replace per-client Python Pingmesh and iperf processes with the native Rust client agent for Pingmesh and background traffic.
53+
- Harden large-topology fault injection, recovery, observability, and exact runtime teardown.
54+
- Publish a versioned DeepSeek validation snapshot across all seven supported scales. See the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx).
5055
- **2026-05**: 🎉 **Initial Release** - NetOpsBench is now available as an open arena for agentic network troubleshooting.
5156
- Provide public SDK with `run_scenario()` and `run_suite()` APIs to launch live network environments from Python.
5257
- Equip native MCP tools of complete observability utilities and pre-configured SONiC-VS network covering XS, Small, Medium and Large scales.
@@ -96,11 +101,21 @@ Scenario YAML files define the benchmark case: topology scale, traffic profile,
96101

97102
NetOpsBench reports detection, fault type, device/interface localization, runtime, tool calls, and token usage so troubleshooting quality and operational cost can be compared together.
98103

99-
![Composite benchmark score](docs/public/assets/benchmark/fig_avg_score.png)
104+
| Scale | Topology | Switches | Clients | Cases |
105+
|---|---|---:|---:|---:|
106+
| XS | CLOS | 4 | 2 | 14 |
107+
| Small | CLOS | 6 | 8 | 15 |
108+
| Medium | CLOS | 12 | 16 | 28 |
109+
| Large | CLOS | 20 | 64 | 52 |
110+
| Xlarge | CLOS | 144 | 128 | 70 |
111+
| Fat-tree K=8 | Fat-tree | 80 | 128 | 70 |
112+
| Fat-tree K=12 | Fat-tree | 180 | 144 | 70 |
100113

101-
Read [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx) for scoring definitions and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for an example completed suite.
114+
![NetOpsBench v0.2.0 DeepSeek validation](docs/public/assets/benchmark/fig_deepseek_v02_quality.png)
102115

103-
Public agent trajectory artifacts are available in the [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/netopsbench-trace), including Harbor/ATIF traces, run reports, and summary CSVs for reproducible analysis.
116+
Read the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx), [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx), and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for the full validation snapshot and scoring definitions.
117+
118+
The public [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/netopsbench-trace) contains both the earlier cross-model snapshot and the [v0.2 seven-scale release](https://huggingface.co/datasets/yyyyyt/netopsbench-trace/tree/main/releases/netopsbench-0.2): 319 validated DeepSeek Harbor/ATIF trajectories across XS through Fat-tree K=12. Aggregate metrics and immutable publication provenance are recorded in the [v0.2 result snapshot](docs/public/assets/benchmark/deepseek_v02_release.json).
104119

105120
## Learn More
106121

‎docs/content/docs/debug-operate/observability.mdx‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,8 @@ or deleted.
120120

121121
## Cleanup
122122

123+
Xlarge and Fat-tree labs can consume substantial CPU, memory, container, and telemetry capacity. Start with one worker, inspect `docker stats`, host memory and pressure, and confirm one complete baseline before increasing concurrency. Keeping a topology warm is useful for repeated experiments, but do not leave unused large workers running alongside a new campaign.
124+
123125
SDK-visible runtimes:
124126

125127
```bash
@@ -129,6 +131,8 @@ netopsbench runtime teardown <runtime-name>
129131
netopsbench runtime teardown --all
130132
```
131133

134+
Prefer the exact runtime name. `teardown --all` is intended only when every runtime registered in the current workspace is owned by the caller; it is not a substitute for host-wide Docker cleanup. Do not use global `docker system prune`, broad container-name filters, or shared-network deletion on multi-user hosts.
135+
132136
Manual deployment teardown:
133137

134138
```bash

‎docs/content/docs/index.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ description: Documentation map for implementing, running, and evaluating NetOpsB
55

66
NetOpsBench evaluates troubleshooting agents on generated data-center network fault scenarios. A run provisions a SONiC-VS / Containerlab topology, injects a controlled fault or healthy episode, exposes runtime evidence, calls an agent, and scores the returned `DiagnosisResult` against scenario ground truth.
77

8+
<Callout title="NetOpsBench v0.2.0 · 2026-08" type="info">
9+
The current release adds Xlarge CLOS, Fat-tree K=8/K=12, and the native Rust Pingmesh/traffic client agent. Read the [v0.2.0 release notes](/docs/releases/v0.2.0).
10+
</Callout>
11+
812
These docs are organized around the agent-development workflow.
913

1014
## Main path
@@ -72,3 +76,4 @@ These docs are organized around the agent-development workflow.
7276
- [System Overview](/docs/architecture/system-overview) explains the runtime loop, evidence path, worker isolation, and report aggregation.
7377
- [Benchmark Methodology](/docs/run-benchmarks/methodology) defines scenario coverage, scoring, negative samples, and optional semantic fault-type matching.
7478
- [Benchmark Results](/docs/run-benchmarks/results) records one completed cross-model run for comparison context.
79+
- [NetOpsBench v0.2.0](/docs/releases/v0.2.0) summarizes the Xlarge, Fat-tree, native client-agent, and lifecycle release.

‎docs/content/docs/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"quickstart",
66
"build-your-agent",
77
"run-benchmarks",
8+
"releases",
89
"debug-operate",
910
"extend-netopsbench",
1011
"architecture"

‎docs/content/docs/quickstart.mdx‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: The shortest path from a fresh clone to one completed NetOpsBench r
55

66
This page verifies that the host can run one benchmark case. It stops after the first `BenchmarkReport`; agent implementation and larger benchmark runs are covered separately.
77

8+
Use XS for installation verification even when the eventual target is Xlarge or Fat-tree. It exercises the same SDK, native client-agent, observability, fault, scoring, and cleanup path with a much smaller resource footprint.
9+
810
## Requirements
911

1012
<Callout title="Linux host required" type="warn">
@@ -83,6 +85,8 @@ PYTHONPATH=. python examples/01_run_scenario.py --vendor openai
8385

8486
The run provisions an XS topology, starts observability, injects one generated fault, calls the reference agent, scores the returned diagnosis, and writes a `BenchmarkReport`.
8587

88+
`netopsbench benchmark prepare` only generates topology and scenario assets. It does not deploy a lab, start telemetry, or reserve resources; deployment begins when a session or runtime command is executed.
89+
8690
## After success
8791

8892
- Open saved agent trajectories with `netopsbench trace view`; it syncs trace-enabled runs into the local Harbor viewer cache automatically.
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"title": "Releases",
3+
"pages": ["v0.2.0"]
4+
}
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
title: NetOpsBench v0.2.0
3+
description: Xlarge CLOS, Fat-tree, and native client-agent release notes.
4+
---
5+
6+
**Released: 2026-08**
7+
8+
NetOpsBench v0.2.0 extends the live benchmark from the original CLOS profiles to Xlarge CLOS and two multi-tier Fat-tree profiles. It also replaces the client-side Python Pingmesh and iperf process fleet with a native Rust client agent while preserving the public observation, tool, evaluator, and SDK contracts.
9+
10+
## Large topology profiles
11+
12+
| Profile | Family | Switches | Clients | Cases | Operational | Agent-scored |
13+
|---|---|---:|---:|---:|---:|---:|
14+
| Xlarge | CLOS | 144 | 128 | 70 | 70 / 70 | 70 / 70 |
15+
| Fat-tree K=8 | Fat-tree | 80 | 128 | 70 | 70 / 70 | 70 / 70 |
16+
| Fat-tree K=12 | Fat-tree | 180 | 144 | 70 | 70 / 70 | 70 / 70 |
17+
18+
Each profile contains four healthy cases and fault cases spanning physical links, devices, impairments, routing, BGP, policy, and ACL behavior. Fault placement covers access and upper-fabric tiers; endpoint Pingmesh can remain healthy when ECMP successfully preserves forwarding, while interface and control-plane telemetry still expose the degraded redundancy.
19+
20+
## Runtime changes
21+
22+
- A single Rust binary runs independent Pingmesh and background-traffic processes in every client.
23+
- Pingmesh uses compact topology configuration, kernel receive timestamps, topology-local Telegraf ingestion, and full-window anomaly semantics.
24+
- Background traffic is runtime-owned and health-checked as a complete flow matrix instead of being recreated for every case.
25+
- Fault injection and recovery use transactional state tracking, readback, convergence checks, quarantine, and exact resource cleanup.
26+
- Large BGP collections preserve fast transition/index data while writing stable full snapshots at a lower cadence.
27+
28+
## Compatibility and results
29+
30+
The stable `netopsbench.sdk` agent, scenario, report, and tool entrypoints remain available. The v0.2 result snapshot is a release-validation reference, not a permanent model ranking; provider behavior and agent strategies can change independently of the benchmark contract.
31+
32+
See [Benchmark Results](/docs/run-benchmarks/results) for accuracy, localization, observability, cost, and trace inventory, the [public v0.2 trace release](https://huggingface.co/datasets/yyyyyt/netopsbench-trace/tree/main/releases/netopsbench-0.2) for all 319 ATIF trajectories, and [Benchmark Methodology](/docs/run-benchmarks/methodology) for the exact scoring and detector semantics.

‎docs/content/docs/run-benchmarks/methodology.mdx‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@ Pingmesh endpoint density.
2929
| Small | CLOS | 2 spine + 4 leaf | 8 |
3030
| Medium | CLOS | 4 spine + 8 leaf | 16 |
3131
| Large | CLOS | 4 spine + 16 leaf | 64 |
32-
| XLarge | CLOS | 16 spine + 128 leaf | 128 |
33-
| Fat-tree K8 | Fat-tree | 16 core + 32 aggregation + 32 edge | 128 |
34-
| Fat-tree K12 | Fat-tree | 36 core + 72 aggregation + 72 edge | 144 |
32+
| Xlarge | CLOS | 16 spine + 128 leaf | 128 |
33+
| Fat-tree K=8 | Fat-tree | 16 core + 32 aggregation + 32 edge | 128 |
34+
| Fat-tree K=12 | Fat-tree | 36 core + 72 aggregation + 72 edge | 144 |
3535

3636
The benchmark covers twelve canonical fault types:
3737

@@ -51,9 +51,9 @@ Each scale combines fault cases with healthy negative samples:
5151
| Small | 12 | 3 | 15 |
5252
| Medium | 24 | 4 | 28 |
5353
| Large | 48 | 4 | 52 |
54-
| XLarge | 66 | 4 | 70 |
55-
| Fat-tree K8 | 66 | 4 | 70 |
56-
| Fat-tree K12 | 66 | 4 | 70 |
54+
| Xlarge | 66 | 4 | 70 |
55+
| Fat-tree K=8 | 66 | 4 | 70 |
56+
| Fat-tree K=12 | 66 | 4 | 70 |
5757

5858
Scenarios are generated from the packaged canonical campaign spec. Running `netopsbench benchmark prepare --scales <scale>` materializes YAML files under `scenarios/generated/<scale>/`; `--spec` can select a custom campaign.
5959

@@ -153,3 +153,5 @@ The judge affects only fault-type classification. Verdict, device localization,
153153
| `avg_tool_calls` | Mean tool calls per case. |
154154

155155
The scoring implementation lives in `netopsbench.evaluator.scorer` and session scoring helpers under `netopsbench.platform.session.scoring`.
156+
157+
The large profiles and native client-agent lifecycle are documented in the [NetOpsBench v0.2.0 release notes](/docs/releases/v0.2.0).

0 commit comments

Comments
 (0)