Add Better Egg (v2) scoring model#31
Conversation
Implement the v2 scoring model based on the validation study (5,417 PRs, 49 repos). The v2 model simplifies the graph (removes 3 zero-signal dimensions) and adds a logistic regression combining graph score with merge rate and account age. - Add scoring_model config (v1 default, v2 opt-in) with V2Config block - Simplify graph builder: no self-contrib penalty, no language normalization, no same_language_weight, no diversity/volume adjustment - Add closed PR count to GraphQL query (zero API cost) - Add v2 scoring pipeline with sigmoid combined model - Add "Better Egg" branding and component score breakdown to all formatters (markdown, CLI, check-run, JSON) - Add --scoring-model CLI option, scoring-model action input/output, scoring_model MCP parameter, GOOD_EGG_SCORING_MODEL env var - Add training script for extracting model weights from validation data - Add 53 new tests (266 total), 93% coverage - Add documentation across methodology, config, action, library, MCP, troubleshooting, README, CHANGELOG, and examples
🥚 Better Egg: HIGH TrustScore: 79% Score Breakdown
Top Contributions
|
Summary of ChangesHello @jeffreyksmithjr, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the scoring capabilities by introducing a new, opt-in 'Better Egg' (v2) model. This model integrates additional signals beyond the contribution graph, such as a user's merge rate and account age, to provide a more robust and comprehensive assessment of trustworthiness. The changes also include extensive updates to configuration, documentation, and output formatting to support the new model, ensuring a seamless experience for users wishing to leverage these advanced scoring features. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new 'Better Egg' (v2) scoring model, which extends the existing graph-based scoring (v1) by incorporating merge rate and account age features via logistic regression. The changes involve updating the CHANGELOG.md, README.md, and various documentation files (docs/configuration.md, docs/github-action.md, docs/library.md, docs/mcp-server.md, docs/methodology.md, docs/troubleshooting.md) to reflect the new model, its configuration options, and output details, including a component score breakdown. The core implementation includes adding V2Config to handle v2-specific parameters, modifying the TrustScorer to dispatch between v1 and v2 logic, and updating the GitHubClient to fetch closed PR counts for the new merge rate feature. The GitHub Action, CLI, and MCP server are also updated to support the scoring_model parameter. However, a review comment highlights inconsistencies between the V2Config model's field names (e.g., temporal_merge_rate, graph_score_weight) and the corresponding names used in the documentation and examples (merge_rate, graph_weight), as well as differing default logistic regression weights, which could lead to user confusion and configuration issues.
PR #31 Review: Better Egg (v2) Scoring ModelCriticalC1. Documentation references a
|
| Severity | Count | Key Issues |
|---|---|---|
| Critical | 3 | Ghost combined_score in docs; field name mismatches; weight value mismatches |
| High | 6 | No temporal merge rate; language match removed despite spec; text dissimilarity absent; negative weight unexplained; incomplete V2GraphConfig vs docs; unsafe Pydantic mutation |
| Medium | 7 | Missing feature handling; wrong class name; dead formatter code; missing tests (x3); undocumented config split |
| Low | 4 | Misleading naming; missing weights_file; dataset count discrepancy; premature date |
| Informational | 6 | Pydantic extras; typing; duplication; query change; cache compat; export completeness |
The three critical issues (C1, C2, C3) will cause incorrect behavior or runtime errors for users following the documentation. These should be resolved before merge. The high-severity issues (particularly H1, H2, H4) represent design decisions that diverge from the issue specification and should at minimum be explicitly acknowledged.
Address all issues from the red team review (except text dissimilarity, which is intentionally excluded): Critical fixes: - Remove ghost `combined_score` key from all docs/examples (C1) - Fix config field names in docs to match code (C2) - Fix default weight values in docs to match trained values (C3) High-severity fixes: - Retain same_language_weight in v2 simplified mode per issue #30 (H2) - Explain negative merge_rate_weight in methodology docs (H4) - Remove incorrect v2.graph fields from docs/examples (H5) - Use model_copy() instead of direct Pydantic mutation (H6) Medium fixes: - Rename temporal_merge_rate to merge_rate (honest naming) (H1/L1) - Add closed_pr_count to v2 scoring_metadata (M8) - Fix V2ScoringConfig -> V2Config in docs (M2) - Document reduced model for missing merge rate (M1) - Add tests: all features disabled, negative weight, opposing signals Low fixes: - Clarify 5,129 vs 5,417 PR count in docs (L4/L5) - Update CHANGELOG date (L6)
Review Fixes (f6946d6)Addressed all findings from the red team review except text dissimilarity (intentionally excluded — the inverted signal is not yet well enough understood). Critical fixes
High-severity fixes
Medium fixes
Low fixes
Verification
|
Point action ref to @better-egg for self-testing on this branch. Will be updated to @v1 after release.
|
Note that the CI action has been run on the latest version of the code (as seen in the edit history here). |
Ensure all parameter tables, section headers, and comparison tables consistently pair version numbers with brand names.
- Remove INPUT_SCORING_MODEL/INPUT_SCORING-MODEL from load_config(); action.py already handles its own override via model_copy(), so load_config() only needs GOOD_EGG_SCORING_MODEL - Clarify "validation study data" wording in methodology docs
2 participants
Summary
scoring_modelconfig (v1default,v2opt-in) with--scoring-modelCLI option,scoring-modelaction input/output, MCP parameter, andGOOD_EGG_SCORING_MODELenv varTest plan
uv run pytest --cov=good_egg -v— 269 tests pass, 93% coverageuv run ruff check src/ tests/ scripts/— all checks passuv run mypy src/good_egg/— no issues foundgood-egg score jeffreyksmithjr --repo 2ndSetAI/good-egg --scoring-model v2 --verbose— Better Egg: HIGH (79%)--scoring-model v1— Good Egg: HIGH (85%), empty component_scores, scoring_model="v1"scorecheck passed using@better-eggref withscoring-model: v2Review findings addressed
Red team review identified 3 critical, 6 high, 7 medium, 4 low, and 6 informational issues. All were addressed in f6946d6 except text dissimilarity (intentionally excluded). Key fixes:
combined_scorefrom all docs (C1)same_language_weightin v2 per issue spec (H2)model_copy()(H6)Closes #30