docs: refine release results presentation

Author: yyyyyt123

README.md

@@ -24,18 +24,11 @@ NetOpsBench is an open benchmark arena for agentic network troubleshooting — r
 
 ## Why NetOpsBench
 
-Developing and evaluating agentic root cause analysis methods for network troubleshooting remains challenging, with three core bottlenecks hindering further advancement:
-
-![NetOpsBench motivation](docs/public/assets/Motivation.png)
-
-| Gap | The problem | How NetOpsBench closes it |
-|---|---|---|
-| **No fair comparison** | Varied network topologies, fault sets, observability tools, and evaluation metrics hinder the comparison of agentic troubleshooting strategies across the research community. | NetOpsBench unifies fault scenarios, observability access and scoring rules to support agent comparison on a shared benchmark. |
-| **Non-reproducible faults** | Real network incidents cannot be reliably reproduced or labeled with consistent ground truth, slowing iterative improvement and evaluation of troubleshooting agents. | Containerlab + SONiC-VS inject controlled, reproducible faults with stable labels, so every run is an identical, repeatable episode. |
-| **Non-Interactive Environment** | Static topology snapshots and logs cannot provide live probing and telemetry signals required by agents for diagnostic work. | NetOpsBench offers an interactive environment for agents to operate within live networks, capturing real-time Pingmesh data, gNMI telemetry and switch CLI evidence during every episode. |
-
-
+Troubleshooting agents are difficult to compare when the network, incident, and evidence change from run to run. NetOpsBench turns those variables into a controlled live benchmark:
 
+- **Reproducible incidents** — labeled faults run against repeatable SONiC-VS and Containerlab topologies.
+- **Interactive evidence** — agents inspect live Pingmesh, BGP, gNMI, syslog, and switch state instead of static logs.
+- **Comparable outcomes** — one evaluator measures detection, localization, efficiency, and tool use across agent strategies.
 
 ## Overview
 
@@ -111,7 +104,13 @@ NetOpsBench reports detection, fault type, device/interface localization, runtim
 | Fat-tree K=8 | Fat-tree | 80 | 128 | 70 |
 | Fat-tree K=12 | Fat-tree | 180 | 144 | 70 |
 
-![NetOpsBench v0.2.0 DeepSeek validation](docs/public/assets/benchmark/fig_deepseek_v02_quality.png)
+**Diagnosis score** is the mean end-to-end case score: healthy cases require the correct verdict, while fault cases receive localization credit only after the fault is detected. **Fault detection F1** measures the fault-versus-healthy decision independently.
+
+![NetOpsBench v0.2.0 quality across seven scales](docs/public/assets/benchmark/fig_deepseek_v02_overview.svg)
+
+The largest validated Fat-tree profile provides a compact case-level view. Each square below is one K=12 case; detailed cross-topology observability analysis remains in the full results.
+
+![Fat-tree K=12 case outcomes grouped by fault family](docs/public/assets/benchmark/fig_deepseek_v02_k12_cases.svg)
 
 Read the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx), [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx), and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for the full validation snapshot and scoring definitions.
 
@@ -134,10 +133,6 @@ The public [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/ne
 - Global community: [NetOpsBench Slack](https://join.slack.com/t/netopsbench/shared_invite/zt-3zhhfangj-2U4dU_NSfCy1rcOM1dmuvQ)
 - Chinese-language community: [NetOpsBench Feishu group](https://applink.feishu.cn/client/chat/chatter/add_by_link?link_token=595v4390-2a51-4db0-baa4-811821b47448)
 
-## Contributing
-
-Contributions are welcome for benchmark scenarios, fault types, SDK ergonomics, documentation, and evaluation workflows.
-
 ## License
 
 NetOpsBench is released under the MIT License. See [LICENSE](LICENSE).

docs/content/docs/run-benchmarks/methodology.mdx

@@ -144,11 +144,11 @@ The judge affects only fault-type classification. Verdict, device localization,
 | Metric | Interpretation |
 |---|---|
 | `detection_accuracy` | Fault/healthy verdict accuracy. |
-| `detection_f1` | Fault/healthy verdict F1-score used in the results plots. |
+| `detection_f1` | Fault/healthy verdict F1-score, shown publicly as **Fault detection F1**. |
 | `device_localization_rate` | Device-level troubleshooting precision. |
 | `interface_localization_rate` | Interface-level precision for applicable cases. |
 | `localization_composite_score` | Combined localization metric used for comparison. |
-| `average_score` | Mean case score over the run. |
+| `average_score` | Mean verdict-gated case score, shown publicly as **Diagnosis score**. Healthy cases require the correct verdict; fault cases receive localization credit only after the fault is detected. |
 | `avg_time_seconds` | Mean diagnosis latency per case. |
 | `avg_tool_calls` | Mean tool calls per case. |
 

docs/content/docs/run-benchmarks/results.mdx

@@ -9,18 +9,20 @@ This page records versioned benchmark snapshots. Treat each snapshot as a refere
 
 This 2026-08 release snapshot used `minimal-deepagent` with `deepseek-v4-pro`, thinking disabled, temperature 0, and two runtime workers for the XS–Large campaigns. All 109 cases completed under the NetOpsBench v0.2.0 runtime, Rust client-agent, traffic, and Pingmesh contract.
 
-| Scale | Cases | Primary reward (%) | Detection F1 (%) | Device loc. (%) | Interface loc. (%) | Fault type (%) | Avg time (s) | Avg tools | Input tokens (K) |
+| Scale | Cases | Diagnosis score (%) | Fault detection F1 (%) | Device loc. (%) | Interface loc. (%) | Fault type (%) | Avg time (s) | Avg tools | Input tokens (K) |
 |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
 | XS | 14 | 75.0 | 95.7 | 75.0 | 57.1 | 41.7 | 61.3 | 22.6 | 174.3 |
 | Small | 15 | 83.3 | 95.7 | 83.3 | 57.1 | 58.3 | 65.1 | 20.9 | 221.4 |
 | Medium | 28 | 71.4 | 85.7 | 70.8 | 42.9 | 37.5 | 79.6 | 23.5 | 298.4 |
 | Large | 52 | 70.2 | 84.3 | 68.8 | 42.9 | 45.8 | 84.3 | 25.2 | 405.7 |
 
-![DeepSeek 0.2 release quality by scale](/assets/benchmark/fig_deepseek_v02_quality.png)
+![DeepSeek 0.2 release quality across seven scales](/assets/benchmark/fig_deepseek_v02_overview.svg)
 
-Across all scales:
+The overview spans all 319 v0.2 Agent-scored cases. The table and aggregate summary in this subsection focus on the 109 XS–Large cases; Xlarge and Fat-tree details follow below.
 
-- Primary reward was **72.9%** (`79.5 / 109`).
+Across XS–Large:
+
+- Diagnosis score was **72.9%** (`79.5 / 109`). This is a verdict-gated case score, not an RL reward: healthy cases require the correct verdict, while fault cases receive credit for correct device/interface localization only after detecting the fault.
 - Verdict accuracy was **80.7%** (`88 / 109`), including **13 / 13** healthy cases and no false positive.
 - The Agent detected **75 / 96** positive cases (**78.1%**).
 - Device localization was **69 / 96** (**71.9%**) over all positive cases and **69 / 75** (**92.0%**) conditional on the Agent reporting a fault.
@@ -59,13 +61,15 @@ The machine-readable [v0.2.0 result snapshot](/assets/benchmark/deepseek_v02_rel
 
 The large-topology campaigns use composite summaries because infrastructure failures were fixed and only the affected cases were rerun. Selection is by scenario ID: a valid retry replaces an invalid infrastructure/API attempt, while successful original attempts remain unchanged.
 
-| Topology | Catalog coverage | Agent-scored | Snapshot | Primary reward (%) | Detection F1 (%) | Device loc. (%) | Device loc. given detection (%) | Interface loc. (%) | Fault type (%) | Avg time (s) | Input tokens (K) |
+| Topology | Catalog coverage | Agent-scored | Snapshot | Diagnosis score (%) | Fault detection F1 (%) | Device loc. (%) | Device loc. given detection (%) | Interface loc. (%) | Fault type (%) | Avg time (s) | Input tokens (K) |
 |---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
 | Xlarge CLOS | 70 / 70 | 70 | Retry-adjusted 0.2 composite | 56.4 | 81.1 | 56.1 | 82.2 | 30.0 | 43.9 | 97.5 | 727.8 |
 | Fat-tree K=8 | 70 / 70 | 70 | Resumed 0.2 composite* | 63.6 | 80.0 | 63.6 | 95.5 | 45.0 | 34.8 | 102.9 | 594.7 |
 | Fat-tree K=12 | 70 / 70 | 70 | Retry-adjusted 0.2 composite | 55.0 | 73.1 | 56.1 | 97.4 | 27.5 | 30.3 | 102.9 | 784.3 |
 
-![DeepSeek large-topology results](/assets/benchmark/fig_deepseek_v02_large_topologies.png)
+![Fat-tree K=12 case outcomes grouped by fault family](/assets/benchmark/fig_deepseek_v02_k12_cases.svg)
+
+The K=12 figure is a representative case-level deep dive for the largest validated Fat-tree profile. Each square is one of its 70 Agent-scored cases; the outcome distribution must not be assumed to apply unchanged to the other topologies.
 
 \* The K=8 composite combines 16 complete results from the initial run with 54 missing-scenario results from a resumed run. Two orphan traces produced during the interrupted process were excluded because they did not complete post-recovery validation.