[experiments] Daily Experiment Report — 2026-10-21

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-21

50 workflows with experiment state (53 experiments) were analysed via gh aw experiments analyze. 26 are 🟢 READY on sample size, but none reached a PROMOTE or REJECT decision: all are blocked by missing observations or unsupported guardrails, or by multi-variant tests the core cannot decide.

⚡ Quick Stats

Metric	Value
Active experiments	53 (50 workflows)
Ready for analysis	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 43 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 10
Interaction safety holds	0 (no workflow has more than one experiment with comparable cells, except smokecopilot* pairs, which are balanced 1:1 and not decidable)

Reason codes: insufficient_observations 16 · guardrail_unsupported 15 · insufficient_samples 12 · unsupported_multi_variant 10.

📈 Per-experiment status (core decision, samples vs min_samples)

Workflow	Experiment	Readiness	Samples / min	Core decision
agentperformanceanalyzer	prompt_compression	🟢 READY	caveman 83, verbose 69 / 14	EXTEND (guardrail_unsupported)
agentpersonaexplorer	sub_agent_strategy	🟡 COLLECTING	none / 14	EXTEND (insufficient_observations)
architectureguardian	sub_agent_strategy	🟡 COLLECTING	17, 21 / 20	EXTEND (insufficient_samples)
auditworkflows	audit_decomposition	🟡 COLLECTING	61, 45 / 264	EXTEND (insufficient_samples)
awfailureinvestigator	tone_variant	🟡 COLLECTING	none / 20	EXTEND (insufficient_observations)
blogauditor	prompt_style	🟡 COLLECTING	5, 9 / 20	EXTEND (insufficient_samples)
breakingchangechecker	tone_variant	🟡 COLLECTING	35, 32 / 194	EXTEND (insufficient_samples)
cicoach	prompt_style	🟢 READY	48, 65 / 20	EXTEND (guardrail_unsupported)
copilotagentanalysis	output_format	🟡 COLLECTING	42, 4, 49 / 30	INCONCLUSIVE (unsupported_multi_variant)
copilotprnlpanalysis	nlp_prompt_style	🟡 COLLECTING	0, 0 / 30	EXTEND (insufficient_samples)
dailyagentrxtraceoptimizer	sub_agent_strategy	🟢 READY	52, 50 / 20	EXTEND (guardrail_unsupported)
dailyarchitecturediagram	detail_level	🟡 COLLECTING	13, 8 / 10	EXTEND (insufficient_samples)
dailyastrostylelitemarkdownspellcheck	prompt_style	🟢 READY	87, 82 / 30	EXTEND (guardrail_unsupported)
dailycachestrategyanalyzer	model_size	🟢 READY	24, 21 / 20	EXTEND (guardrail_unsupported)
dailycavemanoptimizer	model_size	🟡 COLLECTING	82, 18 / 20	EXTEND (insufficient_samples)
dailycodemetrics	output_format	🟡 COLLECTING	54, 58, 17 / 20	INCONCLUSIVE (unsupported_multi_variant)
dailycommunityattribution	prompt_style	🟢 READY	78, 78 / 20	EXTEND (insufficient_observations)
dailycompilerquality	output_format	🟢 READY	56, 66, 22 / 20	INCONCLUSIVE (unsupported_multi_variant)
dailydochealer	model_size	🟢 READY	70, 23 / 20	EXTEND (guardrail_unsupported)
dailydocupdater	model_size	🟢 READY	56, 20 / 20	EXTEND (guardrail_unsupported)
dailyfact	reasoning_depth	🟢 READY	49, 64 / 30	EXTEND (guardrail_unsupported)
dailyfunctionnamer	model_size	🟡 COLLECTING	none / 20	EXTEND (insufficient_observations)
dailyissuesreport	output_format	🟡 COLLECTING	67, 64, 23 / 30	INCONCLUSIVE (unsupported_multi_variant)
dailynews	prompt_style	🟢 READY	68, 61 / 30	EXTEND (guardrail_unsupported)
dailyrenderingscriptsverifier	remove_redundant_context_v1	🟡 COLLECTING	0, 0 / 20	EXTEND (insufficient_samples)
dailysafeoutputoptimizer	log_fetch_strategy	🟡 COLLECTING	none / 20	EXTEND (insufficient_observations)
dailysecurityredteam	reasoning_depth	🟢 READY	91, 70 / 30	EXTEND (guardrail_unsupported)
dailysemgrepscan	semgrep_output_format	🟡 COLLECTING	28, 27, 22 / 30	INCONCLUSIVE (unsupported_multi_variant)
dailysubagentoptimizer	timeout_setting	🟡 COLLECTING	2, 1, 1 / 20	INCONCLUSIVE (unsupported_multi_variant)
dataflowprdiscussiondataset	caveman_mode	🟡 COLLECTING	8, 11 / 10	EXTEND (insufficient_samples)
deepreport	output_format	🟢 READY	71, 91, 71, 53 / 15	INCONCLUSIVE (unsupported_multi_variant)
dependabotcampaign	summary_detail	🟡 COLLECTING	6, 6 / 20	EXTEND (insufficient_samples)
dependabotgochecker	prompt_style	🟡 COLLECTING	22, 20, 20 / 30	INCONCLUSIVE (unsupported_multi_variant)
gofan	prompt_tightening_v1	🟡 COLLECTING	0, 0 / 20	EXTEND (insufficient_samples)
gpclean	tool_verbosity	🟢 READY	70, 79 / 30	EXTEND (guardrail_unsupported)
issuearborist	prompt_style	🟢 READY	58, 50 / 30	EXTEND (guardrail_unsupported)
plan	reasoning_depth	🟡 COLLECTING	4, 2, 1 / 137	INCONCLUSIVE (unsupported_multi_variant)
prsouschef	remove_redundant_context_v1	🟡 COLLECTING	none / 20	EXTEND (insufficient_observations)
smokeantigravity	sub_agent_strategy	🟢 READY	99, 109 / 20	EXTEND (insufficient_observations)
smokecopilot	caveman	🟢 READY	206, 206 / 20	EXTEND (insufficient_observations)
smokecopilot	subagent_model	🟢 READY	196, 196 / 20	EXTEND (insufficient_observations)
smokecopilotaoaiapikey	caveman	🟢 READY	104, 105 / 20	EXTEND (insufficient_observations)
smokecopilotaoaiapikey	subagent_model	🟢 READY	104, 105 / 20	EXTEND (insufficient_observations)
smokecopilotaoaientra	caveman	🟢 READY	91, 91 / 20	EXTEND (insufficient_observations)
smokecopilotaoaientra	subagent_model	🟢 READY	91, 91 / 20	EXTEND (insufficient_observations)
smokecopilotsubagents	sub_agent_strategy	🟡 COLLECTING	21, 21, 26 / 30	INCONCLUSIVE (unsupported_multi_variant)
smokegemini	sub_agent_strategy	🟢 READY	140, 153 / 30	EXTEND (guardrail_unsupported)
smokepi	sub_agent_decomposition	🟢 READY	42, 34 / 20	EXTEND (insufficient_observations)
smokeproject	prompt_style_test	🟡 COLLECTING	none / 20	EXTEND (insufficient_observations)
smoketemporaryid	sub_agent_strategy	🟡 COLLECTING	none / 20	EXTEND (insufficient_observations)
testqualitysentinel	model_size	🟢 READY	131, 154 / 70	EXTEND (guardrail_unsupported)
typist	tone_style	🟢 READY	58, 48 / 10	EXTEND (guardrail_unsupported)
weeklyblogpostwriter	prefetch_strategy	🟡 COLLECTING	12, 7 / 10	EXTEND (insufficient_samples)

⚠️ Data limitations

• Outcome charts (success rate, duration CI) and per-run outcome correlation were not produced: the sandbox gh could not list workflow runs/artifacts for this analysis, so no descriptive outcome statistics are reported. Core decisions are unaffected.
• Balance test fails (is_balanced=false) for copilotagentanalysis, dailycodemetrics, dailycompilerquality, dailyissuesreport, dailysemgrepscan, deepreport, dailydocupdater (the ste / claude-sonnet-5 arms are under-assigned) and dailycavemanoptimizer still reports balance true despite 82 vs 18 and a 0 p-value, so check its assignment weights.
• No tracking issue: was acted on, so no issue comments or labels were posted.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire native metric observations (run_success_rate, output_quality_score) so the 15 guardrail_unsupported experiments (cicoach, dailynews, gpclean, testqualitysentinel, typist, ...) can reach PROMOTE/REJECT.
2. Emit primary-metric observations for the 16 insufficient_observations experiments (smoke* family, dailycommunityattribution), which have enough assignments but no metric data.
3. Rebalance or retire stalled experiments: 0-sample ones (gofan, copilotprnlpanalysis, dailyrenderingscriptsverifier) and plan (4/137 required).

Top eval actions

1. Tighten output-format evals for the output_format experiments (deepreport, dailycompilerquality, dailyissuesreport) to score the ste variant on verbosity/readability.
2. Add quality-score graders for prompt_style workflows (cicoach, dailynews, issuearborist).
3. Add success-rate graders to the smoke workflows to supply observations.

Decision-pipeline gaps for the next PR

• Core analysis does not support K≥3 variants (unsupported_multi_variant, 10 experiments): add pairwise-vs-control decisions with Bonferroni correction.
• Make guardrail_unsupported metrics (run_success_rate, output_quality_score) native observations.
• Promote the factorial interaction diagnostic (smokecopilot* caveman × subagent_model) to a normalized core signal.
• experiments analyze reports is_balanced=true for dailycavemanoptimizer despite a skewed 82/18 split; verify.

Descriptive window: n/a · Decision thresholds: from decision_policy
Run: 39565432767

Generated by 🧪 daily-experiment-report · copilot · auto · 31.6 AIC · ⌖ 0.887 AIC · ⊞ 17.1K · ◷

• expires on Oct 24, 2026, 12:03 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #71083.