[Customization Tool] [Epic] Benchmark for customization tool azsdk_customized_code_update

Build Benchmark for customization tool.

The benchmark will group to three category according to the edit-scope:

SpecInputs : typespec customization only
CustomCode: Code customization only
All : both

And the customization tool input will contains:

customization request
feedback
build failure
breaking changes

Eval naming convention
Each eval file follows this naming pattern:
[entry-point-code][scenario-id].eval.yaml

For example: 001001.eval.yaml

001 represents the entry-point code.
001 represents the scenario ID within that entry point.

entry point	code
customization request	001
feedback	002
build failure	003
breaking changes	004

Each *.eval.yaml file contains three simulations, covering different editScope values, with one simulation for each editScope.

Each simulate has tag 'editScope'

Two-level tags:
eval-spec-level tag: input, it refer to the customization input, it can be customizationRequest, feedback, buildFailure, breakingChange
simulate-level tag: editScope, it refer to the edit scope of the customization, it can be SpecInputs, CustomCode, All

example: 001001.eval.yaml

name: azsdk-customized-code-update-tool-eval
description: Evaluation for the azsdk_customized_code_update mcp tool
version: "1.0"
type: capability

tags:
  tier: unit
  area: customization
  input: customRequest #

environment: azsdk-mcp-live

# Execution configuration
defaults:
  runs: 1 # Trials per stimulus
  timeout: "660s" # Seconds per trial
  model: gpt-6.1-sol # Model for agent execution
  executor: copilot-sdk # Which executor to use

scoring:
  threshold: 1.0

# Test cases
stimuli:
  - name: 001001-rename-model-SpecInputs
    tags:
      editScope: SpecInputs
    prompt: |
      First, use the shell to resolve
      specification/widget/resource-manager/Microsoft.Widget/Widget against
      your current working directory and obtain its absolute directory path.
      Then call azsdk_customized_code_update with parameters:
      - customizationRequest: API review feedback requires renaming SDK model
        'Employee' to 'Staff' for Go.
      - tspProjectPath: use the exact absolute path returned by the shell
      - editScope: "SpecInputs"

      Do not pass a relative path or literal placeholder text as
      tspProjectPath.
    environment:
      files:
        - src: ../../fixtures/Microsoft.Widget/Widget/main.tsp
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/main.tsp
        - src: ../../fixtures/Microsoft.Widget/Widget/employee.tsp
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/employee.tsp
        - src: ../../fixtures/Microsoft.Widget/Widget/tspconfig.yaml
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/tspconfig.yaml
        - src: ../../../eng/common/knowledge/customizing-client-tsp.md
          dest: eng/common/knowledge/customizing-client-tsp.md
      commands:
        - git init
    constraints:
      max_turns: 20
      max_tokens: 50000
    graders:
      - type: tool-calls
        config:
          required:
            - azsdk_customized_code_update
      - type: file-contains
        config:
          path: specification/widget/resource-manager/Microsoft.Widget/Widget/client.tsp
          value: '@@clientName(Microsoft.Widget.Employee, "Staff", "go")'
  
  - name: 001001-rename-model-CodeCustom
    tags:
      editScope: CustomCode
    prompt: |
      First, use the shell to resolve
      azure-sdk-for-js/sdk/widget/arm-widget against your current working
      directory and obtain its absolute directory path.
      Then call azsdk_customized_code_update with parameters:
      - customizationRequest: API review feedback requires renaming SDK model
        'Employee' to 'Staff' for Javascript.
      - packagePath: use the exact absolute path returned by the shell
      - editScope: "CustomCode"

      Do not pass a relative path or literal placeholder text as
      packagePath.
    environment:
      files:
        - src: ../../fixtures/Microsoft.Widget/Widget/main.tsp
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/main.tsp
        - src: ../../fixtures/Microsoft.Widget/Widget/employee.tsp
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/employee.tsp
        - src: ../../fixtures/Microsoft.Widget/Widget/tspconfig.yaml
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/tspconfig.yaml
        - src: ../../../eng/common/knowledge/customizing-client-tsp.md
          dest: eng/common/knowledge/customizing-client-tsp.md
        - src: ../../../artifacts/mcp/cli
          dest: .artifacts/mcp/cli
      commands:
        - git init
        - git remote add origin https://github.com/Azure/azure-rest-api-specs.git
        - git clone --depth 1 --filter=blob:none --sparse https://github.com/Azure/azure-sdk-for-js.git azure-sdk-for-js
        - git -C azure-sdk-for-js sparse-checkout set eng/common/tsp-client
        - npm ci --prefix azure-sdk-for-js/eng/common/tsp-client
        - .\.artifacts\mcp\cli\azsdk.exe package generate --local-sdk-repo-path "%CD%\azure-sdk-for-js" --tsp-config-path "%CD%\specification\widget\resource-manager\Microsoft.Widget\Widget\tspconfig.yaml" --output json
        - node -e "const fs=require('node:fs'); const p='azure-sdk-for-js/sdk/widget/arm-widget/package.json'; if (!fs.existsSync(p)) throw new Error('JavaScript SDK generation did not create '+p)"
    constraints:
      max_turns: 20
      max_tokens: 50000
    graders:
      - type: tool-calls
        config:
          required:
            - azsdk_customized_code_update
  
  - name: 001001-rename-model-ALL
    prompt: |
      First, use the shell to resolve
      azure-sdk-for-js/sdk/widget/arm-widget against your current working
      directory and obtain its absolute directory path.
      Then call azsdk_customized_code_update with parameters:
      - customizationRequest: API review feedback requires renaming SDK model
        'Employee' to 'Staff' for Javascript.
      - packagePath: use the exact absolute path returned by the shell
      - editScope: "All"

      Do not pass a relative path or literal placeholder text as
      packagePath.
    environment:
      files:
        - src: ../../fixtures/Microsoft.Widget/Widget/main.tsp
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/main.tsp
        - src: ../../fixtures/Microsoft.Widget/Widget/employee.tsp
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/employee.tsp
        - src: ../../fixtures/Microsoft.Widget/Widget/tspconfig.yaml
          dest: specification/widget/resource-manager/Microsoft.Widget/Widget/tspconfig.yaml
        - src: ../../../eng/common/knowledge/customizing-client-tsp.md
          dest: eng/common/knowledge/customizing-client-tsp.md
        - src: ../../../artifacts/mcp/cli
          dest: .artifacts/mcp/cli
      commands:
        - git init
        - git remote add origin https://github.com/Azure/azure-rest-api-specs.git
        - git clone --depth 1 --filter=blob:none --sparse https://github.com/Azure/azure-sdk-for-js.git azure-sdk-for-js
        - git -C azure-sdk-for-js sparse-checkout set eng/common/tsp-client
        - npm ci --prefix azure-sdk-for-js/eng/common/tsp-client
        - .\.artifacts\mcp\cli\azsdk.exe package generate --local-sdk-repo-path "%CD%\azure-sdk-for-js" --tsp-config-path "%CD%\specification\widget\resource-manager\Microsoft.Widget\Widget\tspconfig.yaml" --output json
        - node -e "const fs=require('node:fs'); const p='azure-sdk-for-js/sdk/widget/arm-widget/package.json'; if (!fs.existsSync(p)) throw new Error('JavaScript SDK generation did not create '+p)"
    constraints:
      max_turns: 20
      max_tokens: 50000
    graders:
      - type: tool-calls
        config:
          required:
            - azsdk_customized_code_update

Scenarios:

Breaking Change

scenario	SpecInputs	CodeCustom	All
model renamed	Yes	Yes	Yes
property renamed	Yes	Yes	Yes
Property Optionality Changed	Yes	Yes	Yes
Operation renamed	Yes	Yes	Yes
Operation parameter order change	Yes	Yes	Yes
Client renamed	Yes	Yes	Yes
Client hierarchy	Yes	-	Yes
Operation removed from one client to another	Yes	-	Yes

Navigation Menu

Repository navigation

[Customization Tool] [Epic] Benchmark for customization tool azsdk_customized_code_update #17064

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Customization Tool] [Epic] Benchmark for customization tool azsdk_customized_code_update #17064

Description

Activity

chunyu3 commented on Oct 10, 2026

chunyu3 commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions