You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b5dbb23
Browse filesBrowse the repository at this point in the historyBrowse files
- Publish a versioned DeepSeek validation snapshot across all seven supported scales. See the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx).
50
55
-**2026-05**: 🎉 **Initial Release** - NetOpsBench is now available as an open arena for agentic network troubleshooting.
51
56
- Provide public SDK with `run_scenario()` and `run_suite()` APIs to launch live network environments from Python.
52
57
- Equip native MCP tools of complete observability utilities and pre-configured SONiC-VS network covering XS, Small, Medium and Large scales.
NetOpsBench reports detection, fault type, device/interface localization, runtime, tool calls, and token usage so troubleshooting quality and operational cost can be compared together.
Read [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx) for scoring definitions and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for an example completed suite.
Public agent trajectory artifacts are available in the [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/netopsbench-trace), including Harbor/ATIF traces, run reports, and summary CSVs for reproducible analysis.
116
+
Read the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx), [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx), and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for the full validation snapshot and scoring definitions.
117
+
118
+
The public [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/netopsbench-trace) contains both the earlier cross-model snapshot and the [v0.2 seven-scale release](https://huggingface.co/datasets/yyyyyt/netopsbench-trace/tree/main/releases/netopsbench-0.2): 319 validated DeepSeek Harbor/ATIF trajectories across XS through Fat-tree K=12. Aggregate metrics and immutable publication provenance are recorded in the [v0.2 result snapshot](docs/public/assets/benchmark/deepseek_v02_release.json).
Copy file name to clipboardExpand all lines: docs/content/docs/debug-operate/observability.mdx
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -120,6 +120,8 @@ or deleted.
120
120
121
121
## Cleanup
122
122
123
+
Xlarge and Fat-tree labs can consume substantial CPU, memory, container, and telemetry capacity. Start with one worker, inspect `docker stats`, host memory and pressure, and confirm one complete baseline before increasing concurrency. Keeping a topology warm is useful for repeated experiments, but do not leave unused large workers running alongside a new campaign.
Prefer the exact runtime name. `teardown --all` is intended only when every runtime registered in the current workspace is owned by the caller; it is not a substitute for host-wide Docker cleanup. Do not use global `docker system prune`, broad container-name filters, or shared-network deletion on multi-user hosts.
Copy file name to clipboardExpand all lines: docs/content/docs/index.mdx
+5Lines changed: 5 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,10 @@ description: Documentation map for implementing, running, and evaluating NetOpsB
5
5
6
6
NetOpsBench evaluates troubleshooting agents on generated data-center network fault scenarios. A run provisions a SONiC-VS / Containerlab topology, injects a controlled fault or healthy episode, exposes runtime evidence, calls an agent, and scores the returned `DiagnosisResult` against scenario ground truth.
The current release adds Xlarge CLOS, Fat-tree K=8/K=12, and the native Rust Pingmesh/traffic client agent. Read the [v0.2.0 release notes](/docs/releases/v0.2.0).
10
+
</Callout>
11
+
8
12
These docs are organized around the agent-development workflow.
9
13
10
14
## Main path
@@ -72,3 +76,4 @@ These docs are organized around the agent-development workflow.
72
76
-[System Overview](/docs/architecture/system-overview) explains the runtime loop, evidence path, worker isolation, and report aggregation.
Copy file name to clipboardExpand all lines: docs/content/docs/quickstart.mdx
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,8 @@ description: The shortest path from a fresh clone to one completed NetOpsBench r
5
5
6
6
This page verifies that the host can run one benchmark case. It stops after the first `BenchmarkReport`; agent implementation and larger benchmark runs are covered separately.
7
7
8
+
Use XS for installation verification even when the eventual target is Xlarge or Fat-tree. It exercises the same SDK, native client-agent, observability, fault, scoring, and cleanup path with a much smaller resource footprint.
The run provisions an XS topology, starts observability, injects one generated fault, calls the reference agent, scores the returned diagnosis, and writes a `BenchmarkReport`.
85
87
88
+
`netopsbench benchmark prepare` only generates topology and scenario assets. It does not deploy a lab, start telemetry, or reserve resources; deployment begins when a session or runtime command is executed.
89
+
86
90
## After success
87
91
88
92
- Open saved agent trajectories with `netopsbench trace view`; it syncs trace-enabled runs into the local Harbor viewer cache automatically.
description: Xlarge CLOS, Fat-tree, and native client-agent release notes.
4
+
---
5
+
6
+
**Released: 2026-08**
7
+
8
+
NetOpsBench v0.2.0 extends the live benchmark from the original CLOS profiles to Xlarge CLOS and two multi-tier Fat-tree profiles. It also replaces the client-side Python Pingmesh and iperf process fleet with a native Rust client agent while preserving the public observation, tool, evaluator, and SDK contracts.
Each profile contains four healthy cases and fault cases spanning physical links, devices, impairments, routing, BGP, policy, and ACL behavior. Fault placement covers access and upper-fabric tiers; endpoint Pingmesh can remain healthy when ECMP successfully preserves forwarding, while interface and control-plane telemetry still expose the degraded redundancy.
19
+
20
+
## Runtime changes
21
+
22
+
- A single Rust binary runs independent Pingmesh and background-traffic processes in every client.
- Background traffic is runtime-owned and health-checked as a complete flow matrix instead of being recreated for every case.
25
+
- Fault injection and recovery use transactional state tracking, readback, convergence checks, quarantine, and exact resource cleanup.
26
+
- Large BGP collections preserve fast transition/index data while writing stable full snapshots at a lower cadence.
27
+
28
+
## Compatibility and results
29
+
30
+
The stable `netopsbench.sdk` agent, scenario, report, and tool entrypoints remain available. The v0.2 result snapshot is a release-validation reference, not a permanent model ranking; provider behavior and agent strategies can change independently of the benchmark contract.
31
+
32
+
See [Benchmark Results](/docs/run-benchmarks/results) for accuracy, localization, observability, cost, and trace inventory, the [public v0.2 trace release](https://huggingface.co/datasets/yyyyyt/netopsbench-trace/tree/main/releases/netopsbench-0.2) for all 319 ATIF trajectories, and [Benchmark Methodology](/docs/run-benchmarks/methodology) for the exact scoring and detector semantics.
The benchmark covers twelve canonical fault types:
37
37
@@ -51,9 +51,9 @@ Each scale combines fault cases with healthy negative samples:
51
51
| Small | 12 | 3 | 15 |
52
52
| Medium | 24 | 4 | 28 |
53
53
| Large | 48 | 4 | 52 |
54
-
|XLarge| 66 | 4 | 70 |
55
-
| Fat-tree K8| 66 | 4 | 70 |
56
-
| Fat-tree K12| 66 | 4 | 70 |
54
+
|Xlarge| 66 | 4 | 70 |
55
+
| Fat-tree K=8| 66 | 4 | 70 |
56
+
| Fat-tree K=12| 66 | 4 | 70 |
57
57
58
58
Scenarios are generated from the packaged canonical campaign spec. Running `netopsbench benchmark prepare --scales <scale>` materializes YAML files under `scenarios/generated/<scale>/`; `--spec` can select a custom campaign.
59
59
@@ -153,3 +153,5 @@ The judge affects only fault-type classification. Verdict, device localization,
153
153
|`avg_tool_calls`| Mean tool calls per case. |
154
154
155
155
The scoring implementation lives in `netopsbench.evaluator.scorer` and session scoring helpers under `netopsbench.platform.session.scoring`.
156
+
157
+
The large profiles and native client-agent lifecycle are documented in the [NetOpsBench v0.2.0 release notes](/docs/releases/v0.2.0).
0 commit comments