parseTokenUsage returns 0 for Copilot CLI JSONL output

[jafreck]

Problem

parseTokenUsage() in src/runtime/backend/backend.ts always returns 0 for Copilot CLI invocations because it doesn't handle JSONL output.

The function currently handles two formats:

1. Single JSON object (Claude Code) — JSON.parse(result.stdout) on the entire stdout
2. Plain text regex — looks for total_tokens, tokens_used, etc.

Copilot CLI (copilot --output-format json) emits JSONL (one JSON object per line). JSON.parse on the full stdout throws a SyntaxError and falls through. The regex patterns also don't match. Result: always returns 0.

Copilot CLI JSONL format

Copilot CLI emits these relevant event types:

{"type":"assistant.message","data":{"messageId":"...","content":"...","outputTokens":12},...}
{"type":"assistant.message","data":{"messageId":"...","content":"...","outputTokens":85},...}
{"type":"result","timestamp":"...","exitCode":0,"usage":{"premiumRequests":1,"totalApiDurationMs":1755,"sessionDurationMs":6114,"codeChanges":{...}}}

• assistant.message events include data.outputTokens (number) — the only token count Copilot CLI 1.0.x reports
• The result event's usage object does not include input/output token counts — only premiumRequests, totalApiDurationMs, sessionDurationMs, and codeChanges
• Input tokens are not reported by Copilot CLI at all

Required fix

When JSON.parse(result.stdout) fails (stdout is not a single JSON object), try parsing as JSONL before falling through to regex patterns:

1. Split result.stdout by newlines
2. For each line, try JSON.parse(line)
3. For assistant.message events: accumulate data.outputTokens into a running total
4. For result events: check if usage (or data.usage) contains any standard token fields (input, inputTokens, input_tokens, output, outputTokens, output_tokens)
5. If token data was found from either source, return a TokenUsageDetail with { input, output, model: '' } (or the numeric total for backward compatibility). Use accumulated outputTokens from assistant.message events as the output count when the result event has no token fields.

Return type consideration

parseTokenUsage currently returns number. The return value populates AgentResult.tokenUsage which accepts TokenUsageDetail | number | null. Return a TokenUsageDetail when structured data is available so downstream consumers get input/output breakdowns, or a number for backward compatibility.

Test cases

1. Claude single-JSON (existing): JSON.parse succeeds, returns token total — no change
2. Copilot JSONL with outputTokens: Sum outputTokens from assistant.message events → return { input: 0, output: totalOutputTokens, model: '' }
3. Copilot JSONL with future token fields in result: If result.usage ever includes input/output, extract those
4. Plain text patterns (existing): Regex fallback — no change
5. Empty/no output: Returns 0 — no change

Impact

AAMF currently works around this by re-parsing Copilot JSONL in its own agent-launcher.ts post-processing layer. Fixing this in the framework would eliminate the workaround and give all framework consumers correct token tracking for Copilot.