Investigate pocket vetoes (open PRs as implicit rejections) in merge rate

[jeffreyksmithjr]

Problem

The v3 scoring model computes merge rate as merged / (merged + closed). This excludes PRs that are still open, which means pocket vetoes (PRs that maintainers ignore rather than explicitly closing) are absent from both the numerator and denominator. A contributor whose PRs are routinely left to rot rather than closed would have an artificially high merge rate.

Pocket vetoes are common. Many maintainers use them as a softer alternative to explicit rejection, especially for PRs that aren't harmful but aren't wanted.

Prior work

The rejection awareness study (on experiments/ge-validation) and Bad Egg experiments (bad-egg branch) only considered MERGED and CLOSED states. Open PRs were never queried or analyzed. No prior experiment in this repo has addressed pocket vetoes.

Complexity

Defining a pocket veto requires a staleness threshold (e.g., open for 90+ days with no activity). But the right threshold varies by repo. A PR open for 6 months in a fast-moving repo is dead; the same in a slow-moving repo might just be waiting for a release cycle. This likely requires per-repo calibration based on each repo's typical PR lifecycle, which is infrastructurally more complicated than the current approach of pulling a single totalCount from the GraphQL API.

Proposed investigation

1. Query open PR counts and ages for the existing author population
2. Characterize the distribution of open-PR staleness across repos
3. Test whether a repo-calibrated staleness threshold (e.g., open longer than 2x the repo's median time-to-close) changes merge rate distributions meaningfully
4. Evaluate impact on v3 scoring accuracy if stale open PRs are counted as rejections
5. Determine whether a universal threshold is viable or per-repo calibration is required

[jeffreyksmithjr]

Closing after investigation. See #52 for the full evidence trail.

TL;DR: Counting stale open PRs as implicit rejections does not improve merge_rate signal quality against suspended-vs-active ground truth. Tested both age-based (crude proxy) and idle-time-based (updatedAt re-fetch) staleness, with universal and per-repo thresholds. CV AUC flat across all four variants; Cohen's d slightly worse in every case vs the v3 baseline.

Summary of the four variants (2-feature LR, 5-fold CV, 31,229 labeled authors):

Definition	CV AUC	Cohen's d	Authors shifted >0.10
v3 baseline	0.5494	0.096	n/a
age_universal (90d)	0.5488	0.081	1,836
age_per_repo	0.5486	0.073	2,111
idle_universal (90d)	0.5489	0.087	459
idle_per_repo	0.5488	0.085	529

Why: Suspended high-volume accounts in the dataset either are bots v3 already catches, or carry no open PRs at all (e.g. amd-jmacaran: 105 PRs, merge_rate=1.000 under every definition). The reclassification doesn't catch a failure mode the current model misses, it just redistributes mass among accounts the model already scores adequately.

Side-findings worth recording:

1. 79% of DB-OPEN PRs had been closed or merged since the snapshot (15,433 of 19,502). The age-based staleness proxy (forced by the DuckDB schema's lack of updatedAt) mis-classified all of them. Idle-time is the mechanically correct signal. The two-pass structure of the investigation was load-bearing; Pass 1 alone would have been an artifact of the crude proxy.
2. The per_repo calibration matches the hypothesis in this issue (roughly 2x median time-to-close, calibrated per repo) but doesn't meaningfully differ from a universal 90d cutoff on outcome.
3. Interpretability case survives: a drive-by contributor with a single ignored PR currently shows merge_rate=1.000, which reads as "always accepted" but isn't true. idle_per_repo corrects that asymmetry. If we ever revisit this, it should be framed as a fairness/interpretability fix with an acknowledged flat-to-slightly-negative detection metric, not as an accuracy improvement. Filing that separately is a product call, not a data call.

Closing as "won't do for detection accuracy, possible future revisit for interpretability."