|
| 1 | +--- |
| 2 | +title: Conversation Metrics |
| 3 | +description: Retrieve cost and token usage metrics for conversations via CLI tool. |
| 4 | +icon: chart-line |
| 5 | +--- |
| 6 | + |
| 7 | +{/* GENERATED from OpenHands/enterprise-cookbook@b73c7c4dec2be0f1ed2258deebc2e49013064c38 (conversation-metrics/README.md). Edit the source, not this file. */} |
| 8 | + |
| 9 | +<Card title="View source on GitHub" icon="github" href="https://github.com/OpenHands/enterprise-cookbook/tree/b73c7c4dec2be0f1ed2258deebc2e49013064c38/conversation-metrics" horizontal /> |
| 10 | + |
| 11 | +CLI tool to retrieve cost and token usage metrics for OpenHands conversations. |
| 12 | + |
| 13 | +Supports both **V0** and **V1** APIs, automatically selecting the appropriate one based on the conversation version. |
| 14 | + |
| 15 | +## Features |
| 16 | + |
| 17 | +- **Auto-detects** conversation version (V0 vs V1) and uses the appropriate API |
| 18 | +- **Graceful fallback** chain for metric retrieval |
| 19 | +- **Displays** cost (USD), token counts, cache stats, and context window |
| 20 | +- **JSON output** for programmatic use |
| 21 | +- **API call logging** for debugging and development |
| 22 | +- **Fixture-based testing** - high coverage via recorded API responses |
| 23 | +- **Zero dependencies** - uses only Python standard library |
| 24 | + |
| 25 | +## Installation |
| 26 | + |
| 27 | +No installation required - just make the script executable: |
| 28 | + |
| 29 | +```bash |
| 30 | +chmod +x oh-metrics |
| 31 | +``` |
| 32 | + |
| 33 | +Requires Python 3.10+. |
| 34 | + |
| 35 | +## Usage |
| 36 | + |
| 37 | +### Set your API key |
| 38 | + |
| 39 | +```bash |
| 40 | +export OH_API_KEY="your-api-key" |
| 41 | +``` |
| 42 | + |
| 43 | +### Get metrics for a conversation |
| 44 | + |
| 45 | +```bash |
| 46 | +./oh-metrics <conversation_id> |
| 47 | +``` |
| 48 | + |
| 49 | +**Example output (V0 conversation):** |
| 50 | + |
| 51 | +```text |
| 52 | +──────────────────────────────────────────────────────────── |
| 53 | +Conversation: 7f3d57c4e5b2434d9ca78e5e27311137 |
| 54 | +Title: Hello and a Programming Joke |
| 55 | +API Version: V0 via V0 (events) |
| 56 | +──────────────────────────────────────────────────────────── |
| 57 | +💰 Total Cost: $0.083175 USD |
| 58 | +
|
| 59 | +📊 Token Usage: |
| 60 | + Prompt tokens: 12,865 |
| 61 | + Completion tokens: 117 |
| 62 | + Total tokens: 12,982 |
| 63 | +
|
| 64 | +🗄️ Cache: |
| 65 | + Cache read: 0 |
| 66 | + Cache write: 12,740 |
| 67 | +
|
| 68 | +📐 Context window: 200,000 |
| 69 | +──────────────────────────────────────────────────────────── |
| 70 | +``` |
| 71 | + |
| 72 | +**Example output (V1 conversation):** |
| 73 | + |
| 74 | +```text |
| 75 | +──────────────────────────────────────────────────────────── |
| 76 | +Conversation: 72d40619b8534f9b9de6c3f17a71072d |
| 77 | +Title: 📝 V0 API for Conversation Costs & Tokens |
| 78 | +API Version: V1 via V1 (events) |
| 79 | +──────────────────────────────────────────────────────────── |
| 80 | +💰 Total Cost: $13.576675 USD |
| 81 | +
|
| 82 | +📊 Token Usage: |
| 83 | + Prompt tokens: 17,577,146 |
| 84 | + Completion tokens: 63,985 |
| 85 | + Total tokens: 17,641,131 |
| 86 | +
|
| 87 | +🗄️ Cache: |
| 88 | + Cache read: 17,022,477 |
| 89 | + Cache write: 553,973 |
| 90 | +
|
| 91 | +🧠 Reasoning tokens: 1,169 |
| 92 | +
|
| 93 | +📐 Context window: 200,000 |
| 94 | +──────────────────────────────────────────────────────────── |
| 95 | +``` |
| 96 | + |
| 97 | +### JSON output |
| 98 | + |
| 99 | +```bash |
| 100 | +./oh-metrics <conversation_id> --json |
| 101 | +``` |
| 102 | + |
| 103 | +```json |
| 104 | +{ |
| 105 | + "conversation_id": "7f3d57c4e5b2434d9ca78e5e27311137", |
| 106 | + "title": "Hello and a Programming Joke", |
| 107 | + "api_version": "V0", |
| 108 | + "api_used": "V0 (events)", |
| 109 | + "metrics": { |
| 110 | + "accumulated_cost": 0.083175, |
| 111 | + "accumulated_token_usage": { |
| 112 | + "prompt_tokens": 12865, |
| 113 | + "completion_tokens": 117, |
| 114 | + "cache_read_tokens": 0, |
| 115 | + "cache_write_tokens": 12740, |
| 116 | + "context_window": 200000 |
| 117 | + } |
| 118 | + } |
| 119 | +} |
| 120 | +``` |
| 121 | + |
| 122 | +### Options |
| 123 | + |
| 124 | +| Option | Short | Description | |
| 125 | +| ----------------- | ----- | ------------------------------------------------- | |
| 126 | +| `--api-key KEY` | `-k` | API key (defaults to `OH_API_KEY` env var) | |
| 127 | +| `--base-url URL` | `-u` | Base URL (default: `https://app.all-hands.dev`) | |
| 128 | +| `--json` | `-j` | Output in JSON format | |
| 129 | +| `--log-api-calls` | `-l` | Log all API requests/responses to `.oh/api-logs/` | |
| 130 | +| `--help` | `-h` | Show help message | |
| 131 | + |
| 132 | +### Logging API Calls |
| 133 | + |
| 134 | +For debugging or development, you can log all API requests and responses: |
| 135 | + |
| 136 | +```bash |
| 137 | +./oh-metrics <conversation_id> --log-api-calls |
| 138 | +``` |
| 139 | + |
| 140 | +This creates timestamped files in `.oh/api-logs/YYYYMMDD-HHMMSS/`: |
| 141 | + |
| 142 | +```text |
| 143 | +.oh/api-logs/20260227-145230/ |
| 144 | +├── 0001-request.json |
| 145 | +├── 0001-response.json |
| 146 | +├── 0002-request.json |
| 147 | +└── 0002-response.json |
| 148 | +``` |
| 149 | + |
| 150 | +Each request/response pair is numbered sequentially. The Authorization header is redacted from logged requests. |
| 151 | + |
| 152 | +## API Endpoints Used |
| 153 | + |
| 154 | +### For V1 Conversations |
| 155 | + |
| 156 | +| Endpoint | Purpose | |
| 157 | +| --------------------------------------------- | -------------------------------------------------------- | |
| 158 | +| `GET /api/conversations/{id}` | Get conversation info (to determine version) | |
| 159 | +| `GET /api/v1/app-conversations?ids={id}` | Get conversation with metrics | |
| 160 | +| `GET /api/v1/conversation/{id}/events/search` | Fallback: find metrics in `ConversationStateUpdateEvent` | |
| 161 | + |
| 162 | +### For V0 Conversations |
| 163 | + |
| 164 | +| Endpoint | Purpose | |
| 165 | +| ---------------------------------------- | ----------------------------- | |
| 166 | +| `GET /api/conversations/{id}` | Get conversation info | |
| 167 | +| `GET /api/conversations/{id}/events` | Get events with `llm_metrics` | |
| 168 | +| `GET /api/conversations/{id}/trajectory` | Fallback for metrics | |
| 169 | + |
| 170 | +## How Metrics Are Retrieved |
| 171 | + |
| 172 | +The tool uses a fallback chain to find metrics: |
| 173 | + |
| 174 | +1. **Check conversation version** via `/api/conversations/{id}` |
| 175 | +2. **For V1 conversations**: |
| 176 | + - First try `/api/v1/app-conversations?ids={id}` which includes a `metrics` object |
| 177 | + - If metrics are all zeros, fall back to `/api/v1/conversation/{id}/events/search` and extract metrics from `ConversationStateUpdateEvent` at `value.stats.usage_to_metrics.agent` |
| 178 | +3. **For V0 conversations** (or if V1 fails): Use `/api/conversations/{id}/events` and find the latest event with `llm_metrics` |
| 179 | +4. **Last resort**: Use `/api/conversations/{id}/trajectory` and scan for `llm_metrics` |
| 180 | + |
| 181 | +> **Note:** Some V1 conversations have metrics stored only in events (not in the app-conversations response). The fallback chain ensures these are still retrieved correctly. |
| 182 | +
|
| 183 | +## Metrics Explained |
| 184 | + |
| 185 | +| Field | Description | |
| 186 | +| -------------------- | ----------------------------------------- | |
| 187 | +| `accumulated_cost` | Total cost in USD | |
| 188 | +| `prompt_tokens` | Tokens sent to the LLM | |
| 189 | +| `completion_tokens` | Tokens generated by the LLM | |
| 190 | +| `cache_read_tokens` | Tokens read from prompt cache | |
| 191 | +| `cache_write_tokens` | Tokens written to prompt cache | |
| 192 | +| `reasoning_tokens` | Tokens used for reasoning (if applicable) | |
| 193 | +| `context_window` | Model's context window size | |
| 194 | + |
| 195 | +## Architecture |
| 196 | + |
| 197 | +The library is organized into separate modules: |
| 198 | + |
| 199 | +```text |
| 200 | +oh_api/ |
| 201 | +├── __init__.py # Public API exports |
| 202 | +├── client.py # Base HTTP client with logging and fixture support |
| 203 | +├── v0.py # V0 API driver |
| 204 | +├── v1.py # V1 API driver |
| 205 | +└── metrics.py # High-level metrics retrieval with fallback chain |
| 206 | +``` |
| 207 | + |
| 208 | +### Using the Library Programmatically |
| 209 | + |
| 210 | +```python |
| 211 | +from oh_api import APIClient, get_conversation_metrics |
| 212 | + |
| 213 | +# Create a client |
| 214 | +client = APIClient( |
| 215 | + base_url="https://app.all-hands.dev", |
| 216 | + api_key="your-api-key", |
| 217 | + log_api_calls=True, # Optional: enable logging |
| 218 | +) |
| 219 | + |
| 220 | +# Get metrics |
| 221 | +metrics = get_conversation_metrics(client, "conversation-id") |
| 222 | +if metrics: |
| 223 | + print(f"Cost: ${metrics.accumulated_cost:.6f}") |
| 224 | + print(f"Tokens: {metrics.total_tokens:,}") |
| 225 | +``` |
| 226 | + |
| 227 | +### Using the V0/V1 Drivers Directly |
| 228 | + |
| 229 | +```python |
| 230 | +from oh_api import APIClient |
| 231 | +from oh_api.v0 import V0Driver |
| 232 | +from oh_api.v1 import V1Driver |
| 233 | + |
| 234 | +client = APIClient(base_url="...", api_key="...") |
| 235 | + |
| 236 | +# V0 API |
| 237 | +v0 = V0Driver(client) |
| 238 | +conv = v0.get_conversation("conv-id") |
| 239 | +events = v0.get_events("conv-id", limit=50) |
| 240 | + |
| 241 | +# V1 API |
| 242 | +v1 = V1Driver(client) |
| 243 | +conv = v1.get_conversation("conv-id") |
| 244 | +if conv and conv.metrics: |
| 245 | + print(f"Cost: ${conv.metrics.accumulated_cost}") |
| 246 | +``` |
| 247 | + |
| 248 | +## Testing |
| 249 | + |
| 250 | +The test suite uses recorded API fixtures to achieve high coverage without making real API calls: |
| 251 | + |
| 252 | +```bash |
| 253 | +# Run tests |
| 254 | +uv run pytest tests/ -v |
| 255 | + |
| 256 | +# Run with coverage |
| 257 | +uv run pytest tests/ --cov --cov-report=term-missing |
| 258 | +``` |
| 259 | + |
| 260 | +Current coverage: **89%** of library code. |
| 261 | + |
| 262 | +### Creating New Fixtures |
| 263 | + |
| 264 | +1. Run the CLI with `--log-api-calls` to capture real API responses |
| 265 | +2. Copy relevant response files to `tests/fixtures/` |
| 266 | +3. Rename following the pattern: `GET__api_path_q_param=value.json` |
| 267 | + |
| 268 | +## License |
| 269 | + |
| 270 | +MIT License - see [LICENSE](https://github.com/OpenHands/enterprise-cookbook/blob/b73c7c4dec2be0f1ed2258deebc2e49013064c38/LICENSE) for details. |
![openhands-release-bot[bot]](https://avatars.githubusercontent.com/u/225919603?v=4&size=40){kind=avatar}
0 commit comments