Add shared TypeSpec naming skill for authoring by JoshLove-msft · Pull Request #17121 · Azure/azure-sdk-tools

JoshLove-msft · 2026-09-26T04:52:39Z

Summary

Partially implements #17120 with reusable TypeSpec SDK naming guidance, initially for C# ARM management SDKs. Additional language/service profiles remain follow-up work. This is proactive guidance independent of the linter work in Azure/typespec-azure#4442.

Add azsdk-common-typespec-naming, distributed through the existing shared-skill sync, with a dispatcher and authoritative C# member/model profiles.
Invoke shared naming during relevant authoring plans and authorized naming-only customization. Preserve the client-configuration opt-out and avoid SDK-language intake for unrelated or documentation-only work.
Apply authorized naming-only changes through scoped direct edits to client.tsp and necessary imports, not azsdk_customized_code_update. General SDK customization retains its existing workflow.
Preserve wire names, other language customizations, shipped APIs, and unresolved or unsupported decisions. Read-only review remains supported without edits or SDK generation; proposed/applied names are distinguished from verified generated SDK names.

Evaluation coverage

Flat authoring cases 007001–007008 compare trigger, forced, and no-skill task outcomes; 007009–007014 cover routing boundaries. Naming-only continuations may invoke naming directly; forced handoff cases still require authoring. Outcome assertions remain identical across modes.
Declare skills in .vally.yaml and mount definitions/references inside trial workspaces. The existing runner isolates no-skill runs, excludes evaluator content and fixture dependencies, and supports older SkillBranch sources without importing candidate naming rules. The separate bundle-preparation script is removed.
Preservation assertions accept insertions between existing customizations while rejecting deleted or changed originals. The Employee resource case checks the compiled ARM resource and its CRUD/list operations, accepting inline/imported declarations and rejecting unreachable files.
Remove duplicate preview-planning case 001014; existing 001005 coverage remains.

Deferred assessment integration

Assessment integration is excluded and tracked separately in #17297. Assessment is retained only as a competing skill in routing-boundary tests; this PR does not add an assessment naming-report dimension or naming handoff.

Validation

23 focused regression tests passed, including real Git baseline overlay, skill isolation, preservation insertion/deletion checks, compiler-backed Employee reachability, and existing customization/validation fixture behavior.
Strict Vally lint passed for the standalone naming specification, all 14 numbered naming cases, and resource case 002005.
Actual Vally workspace materialization verified readable workspace-local skill/reference paths and empty no-skill mounts. All eight naming cases retain identical outcome assertions across modes.
Native Copilot planning smoke passed against the mounted inputs: shipped Enabled stayed unchanged, new minute-based TTL planned as TimeToLiveInMinutes, no edits, and generatedNamesVerified: false. This is not a full Vally benchmark result.

Investigated naming failures in benchmark 6972773, including failed reference reads and an author-invocation requirement on naming-only continuations. Live-MCP benchmark 6974159 completed successfully on e753da21f: 98/98 published tests passed (100%) across the forced and trigger tracks, exceeding the requested >90% success rate. The two completed test runs report 52/52 and 46/46 passed, with no incomplete tests.

Formatting-only follow-up 3484b4725 fixes the four Prettier violations reported by sync-skills; the exact CI formatting command passes locally.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-09-26T04:53:34Z

Azure Pipelines: Successfully started running 1 pipeline(s). 68 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Copilot review overview

🟡 Changes recommended

The skill currently excludes direct client.tsp naming edits, and version-sensitive exact-name behavior lacks evaluation coverage.

Review effort: Balanced
Findings: 2

Open (2)

Route client.tsp naming edits through compatibility checks · New
Add coverage for numeric enums and exact fallback across TCGC versions · New

What changed in this PR

Adds authoring-time SDK naming guidance for C# ARM TypeSpec projects and integrates hermetic naming evaluations.

Changes:

Adds C# ARM member/model naming profiles and workflow integration.
Adds 17 routing and capability evaluation scenarios.
Registers the suite in local and shared CI discovery.

File	Description
`eng/common/pipelines/skill-eval.yml`	Discovers the naming eval suite.
`.github/skills/.vally.yaml`	Registers its eval directory.
`.github/skills/azure-typespec-author/SKILL.md`	Integrates naming into authoring.
`references/intake.md`	Selects naming profiles during intake.
`references/authoring-plan.md`	Adds naming decisions to plans.
`references/validation.md`	Adds naming validation.
`references/naming-conventions.md`	Defines dispatch and safeguards.
`references/csharp-naming.md`	Defines C# ARM member rules.
`references/csharp-model-naming.md`	Defines C# ARM model rules.
`evals/README.md`	Documents the eval suite.
`evals/eval.yaml`	Adds routing and naming scenarios.

azure-sdk-automation · 2026-09-26T04:59:17Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-09-29T03:23:22Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-09-29T20:44:06Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Haoling Dong (haolingdong-msft)

Hi JoshLove-msft , I like the naming skill, it looks good to me.
Also thanks for contributing to the authoring skill and assessment skill, really appreciate it! I think the key point for us to align on is this comment. It would be helpful to agree on that first so we have a shared direction for how to integrate.

Also two general comments:

if there is any changes to azure-typespec-author skill, please run the benchmark and make sure benchmark passes without regression. https://dev.azure.com/azure-sdk/internal/_build?definitionId=8178&_a=summary
As Ray Chen (@raych1) mentioned, the current suggestion from engsys is to commit the code directly to spec repo and make spec repo as source of truth for azure-typespec-author and azure-typespec-assessment skill code. I've looped you into the email thraed as well. Please feel free to let us know if you have any concern or comments. So if you decided to continue developing authoring skill and assessment skill in this PR, you may need to create a separate PR to sync them to spec repo. An alternative is to only include the naming skill in this PR and have another PR to integrate it with authoring skill and assessment skill.

…ions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-10-01T20:17:36Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-10-01T20:32:08Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

azure-sdk-automation · 2026-10-16T15:57:11Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Replace naming-only customization tool calls with scoped client.tsp edits and align authoring handoffs and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-10-18T23:45:52Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Restore source intake confirmations and strict assertions, use document-led versioning guidance, and align naming benchmarks with numbered cases and comparable no-skill baselines. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-10-19T03:57:54Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Resolve the TypeSpec author benchmark conflicts while preserving the naming workflow and current main state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6952059b-ae31-4269-80d8-07264176b55e

Haoling Dong (haolingdong-msft) · 2026-10-19T07:58:42Z

Hi Josh, quick update: I merged the TypeSpec authoring benchmark PR in azure-rest-api-specs and synced the applicable changes back to azure-sdk-tools through #17310. I also merged the latest main into this branch and resolved the benchmark/naming conflicts.

Looks your latest pipeline run fails on env set up. I retriggered the TypeSpec author benchmark pipeline on the updated branch using the live MCP configuration. The run is currently in progress.

azure-sdk-automation · 2026-10-19T08:07:36Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Haoling Dong (haolingdong-msft)

The changes on benchmark overall looks good to me, I've improved the benchmark stability, here is the latest benchmark result triggered from main, https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6972379&view=results, pass rate is 100%.

I've merged my improvements to your pr and help to solve conflicts now. And I've triggered two pipeline runs.
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6972773
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6972554
Let's make sure the benchmark looks good (success rate > 90%) before merge.

update:
I've checked the failed non-naming cases in this completed run, they are caused by network/test writing issues, not changes introduced by this pr. But there are a few naming cases failure that may need your investigation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-10-19T16:07:07Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-10-20T00:02:52Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

Haoling Dong (haolingdong-msft)

Thanks JoshLove-msft for the changes! The skill and latest benchmark both look great to me now!

I've approved the PR.
Just a friendly reminder: after this PR is merged, please also create a sync PR to the spec repo so the change can take effect. Thanks!

JoshLove-msft · 2026-10-20T02:50:21Z

/check-enforcer override

Include the authoring workflow, client configuration reference, validation safeguards, naming evaluations, and compiled-service grader from Azure/azure-sdk-tools#17121. Preserve REST specs' local evaluation setup and adapt the grader for type-aware linting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Extract shared TypeSpec naming skill and tool-based customization Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Cover exact numeric enum naming across TCGC capabilities Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Test generic customization routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refine TypeSpec naming orchestration Make client configuration a separate default authoring step with an explicit opt-out, clarify customization ownership and verification, and document assessment evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make client configuration opt-out non-blocking Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden TypeSpec author benchmark behavior Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix remaining author benchmark failures Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address naming review safeguards Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove naming MCP dependency Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore naming MCP workflow Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify TypeSpec naming skill routing Exclude naming-only client.tsp overrides from author routing and make shipped-name compatibility a hard gate before naming style. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix naming and preview-version benchmark failures Require naming profile reads during default client configuration planning and clarify preview replacement and excluded-default reversal. Add an offline planning regression without relaxing the existing live assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address C# naming review feedback Remove the underscore enum convention and automatic casing guidance, correct date/time examples, and incorporate applicable semantic naming checks from the .NET review rules. Replace obsolete enum expectations and add focused naming regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Defer TypeSpec assessment naming integration Restore the assessment skill to the PR base and remove its naming handoff cases and integration claims. Retain authoring, standalone read-only naming review, and routing boundaries. Track the deferred integration in #17297. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add fallback for unavailable naming customization tool Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Apply TypeSpec naming overrides directly Replace naming-only customization tool calls with scoped client.tsp edits and align authoring handoffs and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix TypeSpec naming benchmark feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Format shared TypeSpec naming skill fixtures Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Sync naming fixture lint fixes from Azure/azure-sdk-tools#17326 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Sync TypeSpec authoring skill naming integration Include the authoring workflow, client configuration reference, validation safeguards, naming evaluations, and compiled-service grader from Azure/azure-sdk-tools#17121. Preserve REST specs' local evaluation setup and adapt the grader for type-aware linting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Sync latest naming fixture scope validation from tools PR 17326 Import upstream 0996ff06f86f scope-specific customization path validation and expanded regression coverage. Preserve the authoring integration and explicitly type the test response helper for downstream TypeScript narrowing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add language-aware TypeSpec naming guidance

6a9fd14

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings September 26, 2026 04:52

JoshLove-msft requested review from a team, catalinaperalta, Helen Gao (helen229) and Praveen Kuttappan (praveenkuttappan) as code owners September 26, 2026 04:52

Copilot started reviewing on behalf of JoshLove-msft September 26, 2026 04:53

View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-author/SKILL.md Resolved

Comment thread .github/skills/azsdk-common-typespec-naming/references/csharp-naming.md Resolved

Ray Chen (raych1) reviewed Sep 28, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-author/SKILL.md Resolved

Crystal YU (chunyu3) reviewed Sep 29, 2026

View reviewed changes

Extract shared TypeSpec naming skill and tool-based customization

485ada7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

JoshLove-msft requested a review from Haoling Dong (haolingdong-msft) as a code owner September 29, 2026 03:10

JoshLove-msft changed the title ~~Add TypeSpec naming scaffolding and initial C# ARM support~~ Add shared TypeSpec naming skill for authoring and assessment Sep 29, 2026

Cover exact numeric enum naming across TCGC capabilities

128b700

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Haoling Dong (haolingdong-msft) reviewed Sep 30, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-author/references/authoring-plan.md Resolved

Haoling Dong (haolingdong-msft) reviewed Sep 30, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-author/references/authoring-plan.md Resolved

JoshLove-msft mentioned this pull request Oct 1, 2026

Validate shared SDK naming integration in TypeSpec assessment on the PR corpus #17164

Closed

Clarify TypeSpec naming integration and consolidate authoring evaluat…

e95426a

…ions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep unassessed SDK naming reports evidence-only

522c1a7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Crystal YU (chunyu3) reviewed Oct 8, 2026

View reviewed changes

Comment thread .github/skills/azsdk-common-typespec-naming/SKILL.md Resolved

Haoling Dong (haolingdong-msft) reviewed Oct 8, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-assessment/references/workflow/finalization.md Resolved

Haoling Dong (haolingdong-msft) reviewed Oct 8, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-assessment/SKILL.md Resolved

Sameeksha Vaity (samvaity) approved these changes Oct 16, 2026

View reviewed changes

Apply TypeSpec naming overrides directly

c2a3edc

Replace naming-only customization tool calls with scoped client.tsp edits and align authoring handoffs and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Wei Hu (live1206) reviewed Oct 19, 2026

View reviewed changes

Merge main into TypeSpec naming branch

8f54899

Resolve the TypeSpec author benchmark conflicts while preserving the naming workflow and current main state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6952059b-ae31-4269-80d8-07264176b55e

Haoling Dong (haolingdong-msft) reviewed Oct 19, 2026

View reviewed changes

Comment thread .github/skills/azure-typespec-author/evaluate/evals/001014.eval.yaml Resolved

Comment thread .github/skills/azure-typespec-author/evaluate/.vally.yaml Resolved

Fix TypeSpec naming benchmark feedback

e753da2

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Laurent Mazuel (lmazuel) approved these changes Oct 19, 2026

View reviewed changes

Format shared TypeSpec naming skill fixtures

3484b47

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Haoling Dong (haolingdong-msft) approved these changes Oct 20, 2026

View reviewed changes

JoshLove-msft merged commit 704e686 into main Oct 20, 2026
8 of 10 checks passed

JoshLove-msft deleted the joshlove-msft-typespec-naming-conventions branch October 20, 2026 02:50

This was referenced Oct 20, 2026

Sync .github/skills directory with azure-sdk-tools for PR 17121 Azure/azure-rest-api-specs#47437

Merged

Fix shared TypeSpec naming fixtures for downstream lint #17326

Merged

This was referenced Oct 20, 2026

[C# Naming] Run shared naming skill on spec PRs #17336

Open

[Epic] C# ARM SDK readiness before spec merge #17269

Open

Haoling Dong (haolingdong-msft) mentioned this pull request Oct 21, 2026

[TypeSpec] Migrate all authoring and assessment scripts to TypeScript #17348

Closed

m-nash mentioned this pull request Oct 21, 2026

[C# Naming] Consume C# naming rules from azure-sdk-for-net instead of restating them #17371

Open

3 tasks

Copilot AI mentioned this pull request Oct 21, 2026

C# naming consolidation blocked on canonical .NET rules #17372

Draft

Copilot AI mentioned this pull request Oct 27, 2026

[TypeSpec Authoring][benchmark] Migrate and stabilize authoring evaluation tooling Azure/azure-rest-api-specs#47635

Merged

Conversation

JoshLove-msft commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Evaluation coverage

Deferred assessment integration

Validation

Uh oh!

azure-pipelines Bot commented Sep 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

azure-sdk-automation Bot commented Sep 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

azure-sdk-automation Bot commented Sep 29, 2026

Uh oh!

azure-sdk-automation Bot commented Sep 29, 2026

Uh oh!

Haoling Dong (haolingdong-msft) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

azure-sdk-automation Bot commented Oct 1, 2026

Uh oh!

azure-sdk-automation Bot commented Oct 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

azure-sdk-automation Bot commented Oct 16, 2026

Uh oh!

azure-sdk-automation Bot commented Oct 18, 2026

Uh oh!

azure-sdk-automation Bot commented Oct 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Haoling Dong (haolingdong-msft) commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-sdk-automation Bot commented Oct 19, 2026

Uh oh!

Haoling Dong (haolingdong-msft) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

azure-sdk-automation Bot commented Oct 19, 2026

Uh oh!

azure-sdk-automation Bot commented Oct 20, 2026

Uh oh!

Haoling Dong (haolingdong-msft) left a comment

Choose a reason for hiding this comment

Uh oh!

JoshLove-msft commented Oct 20, 2026

Uh oh!

Uh oh!

JoshLove-msft commented Sep 26, 2026 •

edited

Loading

Haoling Dong (haolingdong-msft) left a comment •

edited

Loading

Haoling Dong (haolingdong-msft) commented Oct 19, 2026 •

edited

Loading

Haoling Dong (haolingdong-msft) left a comment •

edited

Loading