[zshenker]

Summary

Adds a new fastly-stats skill: a dedicated, self-contained reference for Fastly's stats/metrics APIs so an agent can get to the data without reading Fastly's docs — usable via the fastly CLI or by building raw URLs for curl/direct HTTP.

Stats were previously only lightly covered inside fastly-cli (references/stats.md, CLI-only). This skill goes deep on the full family of stats APIs across two hosts and three response shapes, with endpoints, query parameters, and response schemas documented inline.

What's included

skills/fastly-stats/
  SKILL.md                          # entrypoint: the two-hosts/three-shapes model,
                                    # auth + token safety, "choose the right API" table,
                                    # accurate CLI quick-reference, raw-URL templates,
                                    # workflows, POP-level diagnosis, limits
  references/
    historical-stats-api.md         # /stats* on api.fastly.com — from/to/by, flat envelope,
                                    # usage/billing (SI units, billable_units), regions
    inspector-api.md                # Origin & Domain Inspector, historical + real-time:
                                    # start/end/downsample, dimensions+values, cursor pagination
    realtime-api.md                 # rt.fastly.com /v1/channel polling model, Timestamp chaining
    fields.md                       # measurement/metric catalog (with TOC) + all_/compute_/waf_
    debugging.md                    # symptom-keyed troubleshooting

Also lists the skill in README.md. No manifest changes needed (skills are discovered by directory).

Design notes

• Two hosts: api.fastly.com (historical) vs rt.fastly.com (real-time).
• Three response shapes: classic flat {status, meta, msg, data}; Inspector dimensions+values with cursor pagination; real-time Data[].{recorded, aggregated, datacenter}.
• Fixes an inaccuracy in the existing CLI docs: the inspector subcommands use --downsample/--metric/--group-by/--limit/--cursor (not --by/--field).
• Complements the existing fastly-cli stats reference rather than replacing it (cross-referenced).
• Copy-paste CLI + curl templates throughout; no bundled scripts to maintain.
• Emphasizes the non-obvious pitfalls: NDJSON --json (use jq -s), decimal-SI vs binary units and raw-bandwidth-vs-billable-usage, UTC defaults, Origin/Domain Inspector as a paid add-on, real-time Timestamp chaining, cursor pagination, and preferring per-POP (datacenter) granularity (incl. flagging Shield POPs).

Validation

• make validate ✓ (markdownlint, yamllint, jsonlint, frontmatter checks, plugin/extension validate)
• make skillscheck (--strict) ✓ — 0 errors, 0 warnings

How it was tested

Developed and evaluated with the skill-creator workflow: with-skill vs. no-skill agent runs across three realistic tasks (cross-service bandwidth, live RPS/5xx watch, origin 5xx/latency), graded against explicit assertions. The skill scored 100% then 95% (against a stricter assertion set) vs. an 82% no-skill baseline, catching concrete errors the baseline made — mishandling NDJSON, missing cursor pagination, unprefixed Inspector metric names, and binary-GiB units.

🤖 Generated with Claude Code

[jedisct1]

I don't understand the purpose of that file.

It looks like a methodology essay on experimental design, with a thin Fastly costume.

It's a generic undergraduate lab-methods handout, telling the agent nothing about what Fastly actually returns., it seems to just tell the agent how to reason.

Figures seem to be based on some random test/example, and are not going to match a real users' account. None can be checked or reproduced.

I just teaches what the model already knows. There's zero unique Fastly content, so as Fastly documentation, it's not useful.

Can you clarify the intended purpose of that file?

[zshenker]

Purpose question answered by deleting it — you were right. It was a methodology essay in a Fastly costume, its figures came from one past investigation and can't be reproduced on anyone else's account, and it taught the model things it already knows. Its actionable residue (baseline first, control POPs, sigma not percent, replicate, sanity-check ranges) is six bullets, which is what survives at the end of references/debugging.md; debugging.md already carried a shorter version of the same list, so nothing unique was lost.

[jedisct1]

This is an agent skill, for agents, not a markdown document for humans.

There's too much prose (just increasing token usage) and formatting like ** and * is useless to agents.

[zshenker]

Agreed and applied across the whole skill, not just this file: all 189 **bold** spans are gone, prose is cut to ~5.0k words from ~10.7k, and the rules that decide correctness are a numbered list on the first screen of SKILL.md rather than emphasis buried in prose.

[jedisct1]

Evaluation https://github.com/fastly/fastly-agent-toolkit-evals/tree/main/additional-experiments/stats-pr12

Quoting the verdict:

1. Cut the size. 11,743 words to 4,465, 7 files to 4. Delete references/verification-method.md, merge the three API reference files into one, drop the DEN/IAD/BOS war stories and all 189 bold spans, and move the eight rules that actually decide correctness to the top of SKILL.md. Today "use decimal SI, not GiB" is a sub-bullet in the tenth section, and Part 2 shows it decides two of the five tasks.

2. Give every endpoint one owner. A session loads fastly, fastly-cli and fastly-stats together, so three copies of the /stats* table are three copies in the same context window. fastly-stats has the best version, so let it keep the material and cut the other two down to a pointer: observability.md 1,018 to 407 words, fastly-cli/references/stats.md 1,769 to 463. The routing data says the agent already behaves this way, opening fastly-stats and nothing else; the duplicates are paying rent for text nobody reads.

   This includes the unit, window and gauge rules that f6dc310 put into both neighbours. They are there because main has no fastly-stats to point at, and about 85 words of them are duplicated between observability.md and fastly-cli/references/stats.md. Merging PR Add fastly-stats skill #12 removes the reason: collapse them into fastly-stats, which owns the conventions, and revert both neighbours to pointer-only. skills-trim is already in that shape and scores 24/25 on both models, so nothing is lost by doing it.

3. Lead with the CLI. 15 curl invocations become 5, and the 5 that survive are the ones fastly stats genuinely cannot reach. Every avoidable curl costs a token substitution, a URL-encoded time string and a hand-written envelope walk, and PR Add fastly-stats skill #12 spends a whole debugging section on a failure that only exists because of the substitution.

4. Fix the three wrong claims in Part 4.4. An unentitled Inspector returns HTTP 200 with an empty array, not an error, and fastly-cli has said so correctly all along. from=1 day ago&by=day, the skill's own first worked call, returns zero rows. The realtime CLI example filters on .Data[]? and silently prints nothing. All three produce confident wrong answers rather than visible failures, which is the worst failure mode for a reference an agent trusts.

5. Narrow the description so three skills stop competing for the same question.

Recommendation: merge, but merge the trimmed version. The 82%-overlap number in Part 1 is an argument for consolidating ownership.

[jedisct1]

Token/wall time usage with Deepseek 4-Flash:

       skills mcp    trials   turns   wall s   prompt tok
       base   nomcp      25      15      182      176,173
       stats  nomcp      25      10      106      109,659
       trim   nomcp      25      11       85       93,445
       base   mcp        25      19      221      212,918
       stats  mcp        25      14      160      170,207
       trim   mcp        25      10       93       97,773

Trimming the fastly-stats skill significantly reduces wall time and price.

[zshenker]

Thanks for the campaign — the skills-trim arm made this easy to act on, since it turned "this is too long" into a measured claim with a control. All five points are addressed in 5377d16, rebased onto current main.

1. Size. 7 files to 4, ~10.7k words to ~5.0k. verification-method.md deleted — your read of it was right, it was a methodology essay whose actionable residue is six bullets, and debugging.md already carried a shorter version of the same list. The three API references are merged into references/api.md. All 189 bold spans are gone, and the eight rules are now a numbered list at the top of SKILL.md rather than sub-bullets in the tenth section.

2. One owner. fastly-stats keeps the endpoints and the unit, window and gauge conventions. observability.md is at 407 words, fastly-cli/references/stats.md at 530, both pointer-only for the material fastly-stats owns — including the three rules f6dc310 had put in both neighbours. stats.md keeps the flag matrix, since that is genuinely the CLI skill's job.

3. CLI first. Six curl invocations against eight fastly ones, and the raw API section is scoped to the five things the CLI cannot reach.

4. The three wrong claims. All three reproduced against a live account before fixing:

• Inspector off on both products → HTTP 200 {"status":"success","data":[]}. fastly-cli had it right; that version is now the only one, with the fastly products -s ID check alongside.
• The relative-window bucket drop reproduced: from=8 days ago&by=day returns one row where explicit UTC boundaries over the same span return two. The Aug 3 bucket is silently lost.
• The realtime filter: fastly stats realtime --json emits {aggregated, datacenter, recorded} per line, no Data. The old filter printed nothing; the replacement reports 17 requests over a 15 s span on the same captured stream.

5. Description narrowed to your wording.

Every code block in the skill was executed against a live account before commit, and that turned up two things worth adding:

• Compute services report traffic in compute_requests and leave requests at 0. Summing requests alone reports an idle service that is serving fine — the same silently-plausible-wrong failure mode as the unit bug. It is now rule 9, and the worked examples sum both counters. Worth noting for the eval tasks too: a requests-only grader would mark a correct Compute answer wrong.
• region + datacenter together returns a meta that omits the datacenter key entirely rather than setting it to null, so .meta.datacenter == "DEN" is the right assertion but != null is not.

Two places I did not follow skills-trim, both deliberate:

• Kept the publication-lag timings (minute within 2-15 min, hourly within 15 min, daily ~2am UTC) and the reason datacenter= is unreachable from the CLI — it is missing from the SDK input type, not just the flags, which is what stops someone hunting for a flag combination that works. Both were unique to main's neighbours and would have been lost by reverting them to pointers.
• Did not take "repeatable flags take one value per occurrence, not a comma-separated list." The CLI joins repeated flags with commas before the request goes out, so the two forms are equivalent — --metric responses --metric status_2xx and --metric responses,status_2xx both produce "metric":"responses,status_2xx" in meta. fastly-cli documented this correctly and I kept its wording.

make ci passes: 0 errors, 0 warnings. The only note is [4c.nesting] on api.md having one link to fields.md, which seems worth the merge.

[jedisct1]

Funky formatting.

[zshenker]

Fixed — Inspector param table, normalized by prettier along with the rest.

[zshenker]

Pushed e63bdb5 addressing the four formatting comments.

Formatting. Ran prettier@3.3.3 over the four fastly-stats files. SKILL.md and api.md needed changes; fields.md and debugging.md were already clean. Table pipe alignment is now 10 tables / 0 ragged. A word-per-line diff confirms the only content-bearing differences are separator-row dash counts.

These slipped through CI because .markdownlint.json checks table structure (MD055/MD056) but nothing checks pipe alignment. I deliberately did not add a formatter to the pipeline in this PR — it would reformat markdown across the whole repo and there are pre-existing ragged tables on main outside this skill. Happy to open that as a separate PR if you want it.

One prettier gotcha worth knowing for this file format: it dedents a numbered-list continuation line when an inline code span wraps across the newline, which breaks list rendering. That hit rule 2 in SKILL.md; I rewrapped so the span stays on one line.

Re-verification found a real bug, not just cosmetics. Re-running every code block after reformatting, the ts/h example in api.md aborted with null (null) cannot be subtracted on a service with no live traffic — max/min over an empty Data array both return null. That is precisely the HTTP 200 + empty-Data payload the same file documents two paragraphs above the example. Guarded the empty case and added the compute_requests term it was also missing.

Verified live against a real account: hit ratio, 5xx share, bandwidth ranking (its total reconciles exactly with account-wide fastly stats usage), usage totals, realtime rps, and ts/h against both a trafficked and a quiet service. make ci: 9 skills, 0 errors, 0 warnings.

[jedisct1]

This is a code comment, not useful here.

[jedisct1]

Re-reviewed at e63bdb5. The packaging is now right: 4 files, one owner per endpoint family, rebases onto main cleanly, make ci reports 9 skills / 0 errors / 0 warnings. The three claims you corrected against my version are all correct and I re-measured each one on my account — unentitled Inspector answers 200 + "status":"success" + "data":[], region + datacenter returns a meta that echoes region and omits the datacenter key entirely (507,437 whole-region requests where SJC alone is 4,083), and fastly stats realtime --json emits flat {aggregated, datacenter, recorded} with no wrapper. Same for the new /service/{id}/stats/summary material, the stats_region list, the four Canadian North-America POPs, compute_sandboxes / compute_req_header_bytes, the origin_offload formula (API 0.14530569423201706 against the documented expression's 0.1453056942320171), and the N-1 day-bucket rule (0, 1 and 6 rows for 1, 2 and 7 days ago, against 31 for explicit July boundaries).

I also ran a sixth eval task, since none of the five existing ones can see the case rule 9 introduces: error-share verbatim against a Compute service instead of a VCL one, 5 repeats per arm on the same omlx / Qwopus3.6-27B setup. base 4/5, the PR at e63bdb5 4/5, the PR with the corrections below 5/5, which at n=5 is p = 1.00 and not the interesting part. The traces are:

arm	pass	turns	wall s	prompt tok
base	4/5	13	157	15,131
PR at e63bdb5	4/5	12	122	10,792
PR + corrections	5/5	11	108	10,652

All five trials on the PR branch ran the SKILL.md 5xx recipe as their first data call. Four noticed that zero errors was implausible, dumped the row's key list and recovered; the fifth wrote 5xx_responses: 0 and answered "no server errors at all" for a service that returned 168,981. Two of the four that recovered reached for compute_resp_status_5xx, which is the mirror-image trap since it reads 0 on a VCL service — all_status_5xx is the only one of the three that is right on both. Nothing in the corrected arm touched the bare name and all five wrote the exact figure. It lands as Part 6 of the existing campaign once I push the task and grader.

Five things to fix. The first four are one family: rule 9 is right, and the worked examples did not follow it.

1. The 5xx example reports zero errors on a Compute service.

The denominator got .requests + .compute_requests in this commit. The numerator still reads status_5xx, which is 0 on a Compute service for exactly the reason rule 9 gives. Run as written against a Compute service of mine:

{"requests": 227654417, "status_5xx": 0, "pct": 0}

The true count is 168,981, a 0.0742% share. all_status_5xx carries both service types: 168,981 there, and 1,103 on a VCL service where status_5xx is also 1,103. One field name.

2. fields.md presents all_ / compute_ / waf_ as Inspector prefixes, which is why (1) happened.

Classic /stats rows carry ten of them: all_status_2xx through all_status_5xx, all_hit_requests, all_miss_requests, all_pass_requests, all_error_requests, all_edge_hit_requests, all_edge_miss_requests. And GET /stats/service/{id}/field/all_status_5xx answers 200, so the skill's own "ask for it alone to test whether a name is real" recipe contradicts the sentence above it. The Status codes section lists status_1xx..status_5xx without mentioning that they are 0 on Compute or that an all_-prefixed twin exists.

3. The account-usage example under-reports by 8x, with no tell.

It sums [.[].requests] and reports 27,724,178 for an account that handled 255,380,898. Every region object carries compute_requests beside requests. This one worries me more than (1): a 5xx count of 0 looks wrong, whereas 27.7M looks like an answer.

4. The live-rate example never got the guard its ts/h sibling got in the same commit.

A service with no traffic emits no lines at all — fastly stats realtime --json printed nothing in 20 s on a quiet service of mine — so head -20 blocks rather than returning a short sample, and an early return dies on null (null) cannot be subtracted. Worth the same if length == 0 guard plus a line saying to bound the stream or use ts/h.

5. billable_units=true does not give a different figure from raw, only a rescaled one.

For July on my account, /stats/usage bandwidth, the sum of per-service /stats bandwidth, and /stats/usage_by_month bandwidth are all 3,349,138,213,801 bytes, and billable_units=true returns 3,349.138213801 GB with requests 2,772.4178 against 27,724,178. So "That is raw edge bandwidth. For the invoice figure use ..." and api.md's "so they exceed raw edge bandwidth" both promise a difference that is not there. bandwidth already includes the origin-bound request bytes: on one service, 2,442,423,451,498 is exactly resp_header_bytes 24,439,478,239 + resp_body_bytes 2,416,961,117,478 + bereq_header_bytes 1,022,855,781 + bereq_body_bytes 0. What the flag buys is the unit conversion and the month keying.

Two worth adding, since they are the same class of trap the skill already documents:

• region= is a silent no-op on /stats/usage. ?region=europe returns all eleven regions, byte-identical to the unfiltered response, while fastly stats usage --region europe returns only the europe key — so the CLI and the raw URL disagree, and api.md lists region as a usage parameter without saying it does nothing there.
• Inspector errors use problem+json: {"type":"https://fastly.help/metrics/validation-error","title":...,"status":400,"errors":[{"property","reason"}],"detail":...}. status is the string "success" on the happy path and the number 400 on the error path, and there is no msg. That matters because the skill tells the agent to assert on the envelope.

Packaging, two small ones:

• The two neighbouring SKILL.md router tables were not updated. fastly/SKILL.md still sends "Querying stats, viewing real-time analytics, using domain/origin inspectors" to observability.md, which no longer has any of it, and fastly-cli/SKILL.md still sends "Historical/real-time metrics, cache hit ratios, error rates, bandwidth, regional traffic" to stats.md, which is now the flag matrix. Neither table names fastly-stats, so routing from the top of either skill costs a hop through a pointer file.
• SKILL.md points at fastly-cli/references/stats.md as a path, which resolves nowhere from inside skills/fastly-stats/. Everywhere else in the toolkit a cross-skill reference names the skill.

Nits: "Five things the CLI cannot do" is followed by a paragraph naming two more, so it is seven. debugging.md files "a relative string sent to Inspector" under "Empty data or zero rows", but it fails loudly — 400 failed to parse 'start' time on the raw API, an argument error from the CLI. And --metric (repeatable, max 10) is the one claim I could not confirm: eleven metrics returned 200 with all eleven echoed in meta.metric, though Origin Inspector is off on my account so the request may not be validated.

[jedisct1]

Let's merge this.

Leaving Claude on its own doesn't work. The remaining issues will be fixed separately.