The >40% rewrite rule introduces undocumented human intervention

[guanziju]

Summary

All six benchmark-specific skillopt-improve.prompt.md files instruct the optimizer to stop and ask the user for confirmation when a
proposed edit would rewrite more than 40% of the Skill.

For example:

If you would rewrite >40% of the skill, stop and ask the user to confirm.

This rule appears in:

• copilot_example/alfworld/.github/prompts/skillopt-improve.prompt.md
• copilot_example/docvqa/.github/prompts/skillopt-improve.prompt.md
• copilot_example/livemath/.github/prompts/skillopt-improve.prompt.md
• copilot_example/officeqa/.github/prompts/skillopt-improve.prompt.md
• copilot_example/searchqa/.github/prompts/skillopt-improve.prompt.md
• copilot_example/spreadsheetbench/.github/prompts/skillopt-improve.prompt.md

Reproducibility concern

This introduces a human decision into what otherwise appears to be an automated closed-loop optimization procedure.

If the condition is triggered, the resulting experiment depends on information that is not fully specified by the prompt or configuration:

1. How is the 40% rewrite ratio measured?
   • Changed characters?
   • Changed lines?
   • Diff size?
   • Semantic content?
2. Who provides the confirmation during formal experiments?
3. What response is given: approve, reject, or request a smaller edit?
4. Is the human response included in the released optimizer trajectory?
5. Was this condition triggered in any of the reported experiments?
6. Can unattended or headless experiments block indefinitely while waiting for confirmation?

The response itself can affect the candidate Skill and therefore becomes an additional optimization input. Different operators may make
different decisions, even when starting from the same Skill, samples, model, and random seed.

Expected behavior

A formal reproduction path should have a deterministic, non-interactive policy. For example, it could:

• reject edits exceeding a precisely defined threshold;
• automatically reduce the proposed edit;
• allow the rewrite without confirmation;
• or expose a frozen configuration option defining the behavior in advance.

If the interactive confirmation rule is intended only for exploratory use, it would help to distinguish the interactive Copilot examples from
the protocol used for the paper's reported experiments.

Questions

• Was the >40% confirmation rule part of the protocol used for the paper results?
• If so, how was the percentage calculated, and what confirmation policy was followed?
• Were the human confirmations and responses preserved in the released experiment logs?
• Is there a recommended non-interactive configuration for faithful reproduction?

[yshenaw]

40% rewrite never happens during the experiments.

[yshenaw]

You can add my wechat for a further communication.