Skip to content

Validate shared SDK naming integration in TypeSpec assessment on the PR corpus #17164

Closed

Description

@JoshLove-msft

🧩 Name of the Activity

Validate the supplementary SDK naming review in azure-typespec-assessment against the existing PR assessment corpus.

🎯 Goal of this Activity

Follow up on the assessment integration feedback in #17121. The reviewer offered to help with integration and report-quality/performance validation through an assigned issue. The authoring and assessment integration is being retained in that PR.

Establish the quality and end-to-end performance impact before treating the new naming checks as validated across real PRs. Isolated naming eval duration must not be presented as average PR assessment duration or incremental overhead.

🌟 Champion Scenarios

  • Confirmed C# ARM changes receive scoped naming recommendations with release-compatibility evidence and unchanged wire/other-language names.
  • Unsupported SDK languages are reported as not-covered, without borrowing C# rules.
  • Missing target, service-type, or release evidence is surfaced rather than guessed.
  • Assessment remains read-only and never invokes SDK customization or generation for the supplementary naming pass.

🔧 How is this Activity Done Today?

The assessment report site currently lists 13 PRs (the review initially mentioned 12): Azure/azure-rest-api-specs#42435, Azure/azure-rest-api-specs#42853, Azure/azure-rest-api-specs#43308, Azure/azure-rest-api-specs#43718, Azure/azure-rest-api-specs#43745, Azure/azure-rest-api-specs#44200, Azure/azure-rest-api-specs#44454, Azure/azure-rest-api-specs#44742, Azure/azure-rest-api-specs#44882, Azure/azure-rest-api-specs#44988, Azure/azure-rest-api-specs#45162, Azure/azure-rest-api-specs#45348, and Azure/azure-rest-api-specs#45536.

#17121 keeps supplementary naming findings separate from assessment.json / assessment.html, whose existing schema is unchanged. It documents a sample supplementary report and adds hermetic coverage for supported, unsupported, and missing-context reporting. Those cases do not establish real-corpus report quality or full-assessment performance.

🛠️ Tools Needed to Support This Activity

Use the assessment skill's coordinator and guarded finalization workflow with pinned PR heads and target baselines. Record the model, skill revision, cache/dependency state, and environment for each run.

Acceptance criteria:

  • Run the agreed corpus with and without naming integration using comparable settings; retain both main reports and supplementary naming output.
  • Review recommendation accuracy, coverage, exceptions, compatibility blockers, and false positives.
  • Record total and phase timings, plus failures or unavailable evidence, per PR. Report paired deltas and mean/median/range; distinguish cold setup costs from the naming step.
  • Confirm that assessment does not modify source or invoke customization/generation during naming judgment.
  • Link the results and remaining limitations back to Add shared TypeSpec naming skill for authoring #17121; do not claim a regression-free result until the comparison is complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions