Skip to content

Commit 3bfeaaa

Browse files
committed
Guard performance with Duel benchmarks
Add Duel, which runs the benchmarks in duel/ on the main branch and on a pull request side by side, and fails the pull request when the code got slower, allocates more, or leaves more objects alive. The benchmarks cover collating resultsets (absorbing a shard, merging two, materializing the merge, decoding a resultset), building the report (adapting Coverage.result, the merged Result, per-file statistics, the JSON and HTML report data), simulating coverage for unloaded files (classifying lines, Prism extraction) and track_tests (diffing snapshots, the test map), plus the cost of requiring simplecov. Inputs come from the collate benchmark's generator and from copies of lib/ files in duel/fixtures, all seeded. duel-retained guards the parse and intern caches that live for a whole test run. The Perf workflow compares each pull request with main, each push with the commit before it, and the last release nightly.
1 parent 4c6cfb6 commit 3bfeaaa

17 files changed

Lines changed: 2602 additions & 3 deletions

‎.duel.rb‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# frozen_string_literal: true
2+
3+
# Duel's configuration: https://github.com/sferik/duel-ruby/blob/main/docs/configuration.md
4+
#
5+
# Duel follows conventions, so set only what differs: benchmarks are in
6+
# duel/**/*_duel.rb, each version's lib/ is first on the load path, a
7+
# change is compared when it touches lib/ or ext/, and releases are
8+
# tagged v*.
9+
Duel.configure do |c|
10+
# SimpleCov runs inside a test suite's process for the whole run, and
11+
# keeps caches for that long: the branch and method keys it has parsed
12+
# and interned, and which files are under the project's root. Each is
13+
# bounded by the project's size, so a benchmark that leaves more alive
14+
# with every call has a leak.
15+
c.use :retained
16+
end

‎.github/workflows/perf.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
name: Perf
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches: [main]
7+
schedule:
8+
- cron: "0 4 * * *"
9+
workflow_dispatch:
10+
11+
permissions:
12+
contents: read
13+
pull-requests: write # to comment on a pull request with the report
14+
15+
jobs:
16+
duel:
17+
name: Duel
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@v7.0.1
21+
with:
22+
fetch-depth: 0 # the baseline comes from history, tags included
23+
# (And whatever else your tests need set up: packages, services.)
24+
- uses: ruby/setup-ruby@v1.329.0
25+
with:
26+
ruby-version: "3.4"
27+
bundler-cache: true
28+
# Against the main branch for a pull request, what was there before
29+
# for a push, and the last release on a schedule (slow creep). Duel
30+
# adds its report to the job's summary, and writes it to duel.md for
31+
# the comment.
32+
- name: Compare
33+
env:
34+
DUEL_BASE: ${{ github.event_name == 'schedule' && 'last-release' || github.event_name == 'push' && github.event.before || '' }}
35+
# (Run by hand, it compares anyway: with nothing changed, the code
36+
# with itself, which checks the setup.)
37+
DUEL_FORCE: ${{ github.event_name == 'workflow_dispatch' && '1' || '' }}
38+
run: |
39+
# (A branch's first push, or one that rewrote history, has no
40+
# commit before it here to compare against.)
41+
if [ "$GITHUB_EVENT_NAME" = push ] && ! git cat-file -e "$DUEL_BASE^{commit}" 2>/dev/null; then
42+
echo "::notice::No commit before this push to compare against"
43+
exit 0
44+
fi
45+
bundle exec duel --markdown "$RUNNER_TEMP/duel.md"
46+
# The report, as a comment on the pull request: one comment, updated
47+
# on every push. (A pull request from a fork gets no comment, as its
48+
# token can't write one: the job's summary has the report.)
49+
- name: Comment on the pull request
50+
if: ${{ !cancelled() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository }}
51+
env:
52+
GH_TOKEN: ${{ github.token }}
53+
PR: ${{ github.event.pull_request.number }}
54+
run: |
55+
[ -f "$RUNNER_TEMP/duel.md" ] || exit 0
56+
marker="<!-- duel -->"
57+
{ echo "$marker"; cat "$RUNNER_TEMP/duel.md"; } > "$RUNNER_TEMP/comment.md"
58+
ids=$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues/$PR/comments" \
59+
--jq ".[] | select(.user.login == \"github-actions[bot]\" and (.body | startswith(\"$marker\"))) | .id")
60+
if [ -n "$ids" ]; then
61+
gh api --method PATCH "repos/$GITHUB_REPOSITORY/issues/comments/${ids%%$'\n'*}" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
62+
else
63+
gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
64+
fi

‎.rubocop.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,15 @@ plugins:
1919
- rubocop-rspec
2020

2121
AllCops:
22+
# Duel's configuration is a dotfile, which RuboCop skips unless told.
23+
Include:
24+
- ".duel.rb"
2225
Exclude:
2326
# Fixtures are byte-stable test inputs: their line numbers, layout,
2427
# and sometimes their encodings are what the specs assert against,
25-
# so they are not subject to style.
28+
# so they are not subject to style. Duel's are copies of lib/ files
29+
# that its benchmarks parse, one of them several files run together.
30+
- "duel/fixtures/**/*"
2631
- "spec/fixtures/**/*"
2732
- "test_projects/**/*"
2833
- "tmp/**/*"
@@ -34,6 +39,7 @@ AllCops:
3439
inherit_mode:
3540
merge:
3641
- Exclude
42+
- Include
3743

3844
# spec/support/ holds helper modules, not a spec suite, so the RSpec
3945
# department shouldn't touch it. (Fixtures are excluded wholesale under

‎.standard.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@ ruby_version: 3.4
1212
ignore:
1313
# Fixtures are byte-stable test inputs: their line numbers, layout, and
1414
# sometimes their encodings are what the specs assert against, so they are
15-
# not subject to style.
15+
# not subject to style. Duel's are copies of lib/ files that its benchmarks
16+
# parse, one of them several files run together.
17+
- "duel/fixtures/**/*"
1618
- "spec/fixtures/**/*"
1719
- "test_projects/**/*"
1820
- "tmp/**/*"

‎Gemfile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,14 @@ group :development do
4343
gem "standard", ">= 1.35.1"
4444
gem "standard-performance"
4545
gem "standard-rspec"
46+
47+
# Performance regression guards (rake duel): the benchmarks in duel/ run
48+
# on the main branch and the candidate side by side, and a slowdown fails.
49+
# StackProf profiles a regression, and duel-retained fails a benchmark
50+
# that leaves objects alive, such as a parse cache that keeps growing.
51+
gem "duel"
52+
gem "duel-retained"
53+
gem "stackprof", platforms: :mri
4654
end
4755

4856
group :benchmark do

‎Gemfile.lock‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,15 @@ GEM
1414
concurrent-ruby (1.3.8)
1515
csv (3.3.6)
1616
diff-lcs (1.6.2)
17+
duel (0.1.0)
18+
fiddle (~> 1.1)
19+
duel-retained (0.1.0)
20+
duel (~> 0.1)
1721
erb (6.0.7)
1822
erb (6.0.7-java)
1923
ffi (1.17.4)
2024
ffi (1.17.4-java)
25+
fiddle (1.1.8)
2126
fileutils (1.8.1)
2227
hana (1.3.7)
2328
io-console (0.9.4)
@@ -144,6 +149,7 @@ GEM
144149
securerandom (0.4.1)
145150
simpleidn (0.3.0)
146151
sorbet-runtime (0.6.13571)
152+
stackprof (0.2.28)
147153
standard (1.57.0)
148154
language_server-protocol (~> 3.17.0.2)
149155
lint_roller (~> 1.0)
@@ -202,6 +208,8 @@ PLATFORMS
202208

203209
DEPENDENCIES
204210
benchmark-ips
211+
duel
212+
duel-retained
205213
json_schemer
206214
mutant
207215
mutant-rspec
@@ -215,6 +223,7 @@ DEPENDENCIES
215223
rubocop-rake
216224
rubocop-rspec
217225
simplecov!
226+
stackprof
218227
standard (>= 1.35.1)
219228
standard-performance
220229
standard-rspec

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
SimpleCov [![Gem Version](https://badge.fury.io/rb/simplecov.svg)](https://badge.fury.io/rb/simplecov) [![Build Status](https://github.com/simplecov-ruby/simplecov/actions/workflows/stable.yml/badge.svg?branch=main)][ci] [![Lint](https://github.com/simplecov-ruby/simplecov/actions/workflows/lint.yml/badge.svg?branch=main)](https://github.com/simplecov-ruby/simplecov/actions/workflows/lint.yml) [![Typecheck](https://github.com/simplecov-ruby/simplecov/actions/workflows/typecheck.yml/badge.svg?branch=main)](https://github.com/simplecov-ruby/simplecov/actions/workflows/typecheck.yml) [![Mutation](https://github.com/simplecov-ruby/simplecov/actions/workflows/mutation.yml/badge.svg?branch=main)](https://github.com/simplecov-ruby/simplecov/actions/workflows/mutation.yml) [![Maintainability](https://api.codeclimate.com/v1/badges/c071d197d61953a7e482/maintainability)](https://codeclimate.com/github/simplecov-ruby/simplecov/maintainability)
1+
SimpleCov [![Gem Version](https://badge.fury.io/rb/simplecov.svg)](https://badge.fury.io/rb/simplecov) [![Build Status](https://github.com/simplecov-ruby/simplecov/actions/workflows/stable.yml/badge.svg?branch=main)][ci] [![Lint](https://github.com/simplecov-ruby/simplecov/actions/workflows/lint.yml/badge.svg?branch=main)](https://github.com/simplecov-ruby/simplecov/actions/workflows/lint.yml) [![Typecheck](https://github.com/simplecov-ruby/simplecov/actions/workflows/typecheck.yml/badge.svg?branch=main)](https://github.com/simplecov-ruby/simplecov/actions/workflows/typecheck.yml) [![Mutation](https://github.com/simplecov-ruby/simplecov/actions/workflows/mutation.yml/badge.svg?branch=main)](https://github.com/simplecov-ruby/simplecov/actions/workflows/mutation.yml) [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby) [![Maintainability](https://api.codeclimate.com/v1/badges/c071d197d61953a7e482/maintainability)](https://codeclimate.com/github/simplecov-ruby/simplecov/maintainability)
22
=========
33

44
**Code coverage for Ruby**

‎Rakefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,3 +369,6 @@ namespace :assets do
369369
Dir.chdir(__dir__) { sh "git", "diff", "--exit-code", "--", template }
370370
end
371371
end
372+
373+
require "duel/rake_task"
374+
Duel::RakeTask.new

‎duel/boot_duel.rb‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# frozen_string_literal: true
2+
3+
# What `require "simplecov"` costs every test process, before any test
4+
# runs. Prism loads only when branch or method coverage needs it, and this
5+
# guards that it stays that way.
6+
duel_process "boot: require simplecov", ruby: %(require "simplecov")

‎duel/collate_duel.rb‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# frozen_string_literal: true
2+
3+
# Collating a parallel run's resultsets (`SimpleCov.collate`, and the merge
4+
# at the end of every parallel run), the cost that grows with a CI run's
5+
# worker count. Reading each resultset is JSON parsing, and merging it is
6+
# folding its counts into an accumulator that's updated in place, one
7+
# resultset at a time.
8+
#
9+
# The tuple combiners intern every branch key the first time they see it,
10+
# in caches that last the process's life: the first `once` of each
11+
# benchmark warms them, as the first resultset of a real collate does.
12+
13+
require_relative "duel_helper"
14+
15+
# What each further resultset of a run costs: its counts folded into what
16+
# the run's earlier resultsets merged. The counts grow with every
17+
# iteration, but the work doesn't.
18+
duel "collate: absorb a shard of 20 files" do
19+
once do
20+
@shards = DuelFixtures.shards(20, 3)
21+
@accumulator = SimpleCov::Combine::CoverageAccumulator.new
22+
@shards.first(2).each { |shard| @accumulator.absorb(shard) }
23+
end
24+
@accumulator.absorb(@shards.last)
25+
end
26+
27+
duel "collate: absorb a shard of 200 files" do
28+
once do
29+
@shards = DuelFixtures.shards(200, 3)
30+
@accumulator = SimpleCov::Combine::CoverageAccumulator.new
31+
@shards.first(2).each { |shard| @accumulator.absorb(shard) }
32+
end
33+
@accumulator.absorb(@shards.last)
34+
end
35+
36+
# A whole merge of two shards, from a fresh accumulator: each file's
37+
# first merge sets up its tables, and the merged coverage is turned back
38+
# into the shape a Result reads. The merge holds one resultset at a time,
39+
# whatever the run's size, so its memory is guarded too.
40+
duel "collate: merge two shards of 200 files", peak_memory: true do
41+
once { @shards = DuelFixtures.shards(200, 2).tap { |shards| SimpleCov::Combine::ResultsCombiner.combine(*shards) } }
42+
SimpleCov::Combine::ResultsCombiner.combine(*@shards)
43+
end
44+
45+
# The merged coverage, from tables keyed by interned identities back to
46+
# the tuple-keyed hashes the rest of SimpleCov reads. Done once a merge.
47+
duel "collate: materialize 200 merged files" do
48+
once do
49+
@accumulator = SimpleCov::Combine::CoverageAccumulator.new
50+
DuelFixtures.shards(200, 3).each { |shard| @accumulator.absorb(shard) }
51+
end
52+
@accumulator.result
53+
end
54+
55+
duel "collate: decode a resultset of 20 files" do
56+
once { @json = DuelFixtures.resultset_json(20) }
57+
SimpleCov::ResultMerger::ResultsetFile.decode(@json)
58+
end
59+
60+
duel "collate: decode a resultset of 200 files" do
61+
once { @json = DuelFixtures.resultset_json(200) }
62+
SimpleCov::ResultMerger::ResultsetFile.decode(@json)
63+
end

0 commit comments

Comments
 (0)