Skip to content

Commit aebaca0

Browse files
committed
Say why a run that tracked tests recorded no map
A process whose tests run concurrently in threads drops its test map, because per-test coverage cannot tell concurrent tests apart. That is the right call for threaded `parallelize`, but `simplecov affected` then said to enable `track_tests`, which was already on. A merge of results only some of which recorded a map is in the same position. The tracker now gives a reason when it drops its map. The result carries it through `.resultset.json` and every merge, and `coverage.json` records it as `contexts_dropped`. A merge that drops a map for want of one in every result gives its own reason. `affected`, `tests`, and `status` say the reason instead of pointing at `track_tests`. `contexts_dropped` is an optional key in schema 1.4, which is not yet released. Reported in #1316 (comment)
1 parent 65ea4c3 commit aebaca0

34 files changed

Lines changed: 344 additions & 32 deletions

‎CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1717

1818
- SimpleCov depends on the `prism` gem. Ruby 3.4 and 4.0 ship Prism as a default gem, so nothing new installs there, but Ruby 4.1 ships it as a bundled gem, which a bundle can load only when something in it depends on Prism. Without it, a project's bundle on Ruby 4.1 silently lost what needs Prism: static branch and method coverage for files the suite never loaded, telling a newly loaded file's load-time lines from its method bodies under `track_tests`, and narrowing `simplecov affected` to the changed methods. Prism still loads lazily, so a Ruby that cannot load it (JRuby on Windows) keeps working without those features.
1919
- The HTML report keeps a test file's own loading apart from its tests. The tests panel lists the files whose loading ran a line under their own note, and a line only loading ran counts as covered outside tests in the source view and the file list, like other load-time code. See [#1316].
20-
- `coverage.json` advances to schema 1.4, which changes no shape. It documents that a bare test file path in `contexts`, beside ids pointing into the same file, names that file's own loading rather than a test, now that `track_tests` records it. Schema 1.3 stays frozen for pinned consumers. See [#1316].
20+
- `coverage.json` advances to schema 1.4. It adds the optional `contexts_dropped` string, saying why a run that tracked tests carries no `contexts`, and documents that a bare test file path in `contexts`, beside ids pointing into the same file, names that file's own loading rather than a test, now that `track_tests` records it. Schema 1.3 stays frozen for pinned consumers. See [#1316].
2121
- `track_tests` credits a test file with what it runs while it loads, before any of its tests run, under the file's path with no line number (`spec/user_spec.rb`). Code a spec file runs to define its examples, like shared examples or helpers called in `describe` bodies, belonged to no test, so `simplecov affected` fell back to the full suite for a change to it. Files first loaded while a test file loads still belong to no test, and RSpec's `before(:suite)` hooks belong to no test either. `simplecov tests --redundant` never lists a test file's loading as a redundant test, and `simplecov status` does not count it as a recorded test. Watching files load, and parsing each file a test loads first with Prism, cost nothing measurable: on SimpleCov's own suite under Ruby 4.1 (6,172 examples, three runs each), `track_tests` adds 6% to the suite's CPU time, where it added 20% before this release's attribution changes. See [#1316].
2222
- SimpleCov no longer warns on JRuby that coverage may be inaccurate without `--debug` (full-trace mode). JRuby's coverage no longer depends on it: SimpleCov's own suite on JRuby 10.0.7 reports every line identically with and without `--debug`, so the warning only told JRuby users to slow their test runs down for nothing.
2323
- The HTML report is usable on phones. On narrow screens the theme toggles shrink to icons, a report with several groups picks them from a dropdown instead of a row of tabs, the file list drops its coverage bars and covered/total counts and wraps long paths, and the source view stacks its header. On small or short screens the source header and footer scroll away with the code instead of staying pinned. Below 960px wide the code font scales so 80 columns fit beside the line numbers and hit counts. See [#1312].
@@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3434
- `simplecov affected` falls back to the full suite when a changed statement outside any method ran only outside recorded tests, even if recorded tests ran other lines of the file, and names the changed line. A changed or added constant selected only the tests that ran the file's methods, leaving out tests that read the constant. Blank lines, comments, and added methods do not count, and a change to a heredoc's body counts through the statement that opens it. See [#1316].
3535
- `track_tests` no longer credits a test with the lines a file ran as the test first loaded it. A file's class body, constants, and `def` lines run once on behalf of every later test, but the test that happened to load it was recorded as their only owner, so `simplecov affected` selected that one test for a change to them. The loading test is now credited only with the method body lines it ran, and the rest is left to no test, like a file loaded before the suite ran. See [#1316].
3636
- A merge keeps the method and branch hits of a process that recorded no line hit for the file. A forked worker inherits the files its parent loaded with their counters cleared, and calling a method whose body shares its `def` line (a short endless method, for instance) then records a method hit and no line hit. The merge took that for a never-loaded file and dropped its hits in favor of the parent's zeroes, so methods the workers exercised were reported as uncovered. A file now counts as executed when any line, method, or branch arm was hit. See [#1310].
37+
- `simplecov affected`, `simplecov tests`, and `simplecov status` say why a run that tracked tests recorded no test map, instead of telling you to enable `track_tests`, which was already on. A process whose tests ran concurrently in threads (Rails' threaded `parallelize`) drops its map, since per-test coverage cannot tell concurrent tests apart, and so does a merge of results only some of which recorded one. The reason is carried through the resultset into `coverage.json` as `contexts_dropped`. See [#1316].
3738
- `track_tests` credits a test with a one-line method it called in a file it first loaded, such as `def odd? = (a || b) && c`. The loading test kept only method body lines, and a one-line method's body shares its `def` line, so the method was left to no test while a later test that called it claimed the line. A change to it then selected only that later test. The line is now the loading test's when it ran more often than loading the file accounts for. See [#1316].
3839
- A later run under the same command name keeps the test map an earlier run recorded for the tests it did not run. Rerunning one test file, or running `simplecov affected --run`, replaced the whole map, so a later `simplecov affected` selected nothing for a change that only the other tests covered, and exited 0. A test recorded again replaces its earlier lines. An earlier test whose file is gone is dropped, and one that touched a source file changed since is credited with every line of that file, since its recorded line numbers no longer hold. See [#1316].
3940
- The `strict` profile no longer makes JRuby warn that branch and method coverage are not supported. It enabled both on every engine, so `Coverage.start` was asked for criteria JRuby cannot measure. It now enables only the criteria the runtime supports and pins a 100% threshold for each of those.

‎docs/Formatters.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -171,8 +171,10 @@ complete context map. 1.2 added the optional `history` array (the recorded run h
171171
added the optional `production` section: the window and per-file lines a
172172
[production coverage store](Production.md) accumulated, present exactly when
173173
[`production_coverage`](Production.md#seeing-the-cross-in-the-html-and-json-reports) names a readable store at report
174-
time. 1.4 changed no shape: it documents that a bare test file path in `contexts`, beside ids pointing into the
175-
same file, is that file's own loading rather than a test. Top-level structure:
174+
time. 1.4 added the optional `contexts_dropped` string, saying why a run that tracked tests carries no `contexts`
175+
(tests that ran concurrently in threads, or a merge that mixed results with and without a map), and documents that
176+
a bare test file path in `contexts`, beside ids pointing into the same file, is that file's own loading rather than
177+
a test. Top-level structure:
176178

177179
```jsonc
178180
{
@@ -181,6 +183,7 @@ same file, is that file's own loading rather than a test. Top-level structure:
181183
"total": { /* aggregate stats for lines (and branches / methods when enabled) */ },
182184
"coverage": { "<project-relative path>": { /* per-file lines, source, branches, methods, contexts, etc. */ } },
183185
"contexts": [ /* recorded context ids (with track_tests: test definition locations); present only when recorded */ ],
186+
"contexts_dropped": "/* why a tracked run carries no contexts; present only when a map was dropped */",
184187
"history": [ /* recorded runs, oldest first, this run last; present only when past runs are recorded */ ],
185188
"production": { /* window + per-file lines and last_seen stamps from a production store; present only when configured */ },
186189
"groups": { "<group name>": { /* per-group stats + files */ } },

‎lib/simplecov/cli/command_helpers.rb‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,20 @@ def error_nil(stderr, message)
3535

3636
def recorded_contexts(document, opts, stderr)
3737
contexts = document["contexts"]
38-
if contexts.nil?
39-
return error_nil(stderr, "no test contexts recorded in #{opts.fetch(:input)}. Enable `track_tests` in " \
40-
"your `SimpleCov.start` block and rerun the suite to record them")
41-
end
38+
return error_nil(stderr, unrecorded_contexts(document, opts.fetch(:input))) if contexts.nil?
4239
return contexts if contexts.instance_of?(Array) && contexts.all?(String)
4340

4441
CoverageFile.report_invalid(stderr, command_name, opts.fetch(:input), '"contexts" must be an array of strings')
4542
end
4643

44+
def unrecorded_contexts(document, input)
45+
dropped = document["contexts_dropped"]
46+
return "no test contexts recorded in #{input}: the run dropped them because #{dropped}" if dropped.instance_of?(String)
47+
48+
"no test contexts recorded in #{input}. Enable `track_tests` in your `SimpleCov.start` block and rerun " \
49+
"the suite to record them"
50+
end
51+
4752
def common_options(parser, opts)
4853
parser.on("--input PATH") { |v| opts[:input] = v }
4954
parser.on("--json") { opts[:json] = true }

‎lib/simplecov/cli/status.rb‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ def report_lines(facts)
3636
provenance_words(facts),
3737
"commit #{commit_words(facts)}",
3838
(totals_words(facts.fetch(:totals)) unless facts.fetch(:totals).empty?),
39-
"tests recorded: #{contexts_words(facts.fetch(:contexts))}"
39+
"tests recorded: #{contexts_words(facts.fetch(:contexts), facts.fetch(:contexts_dropped))}"
4040
].compact
4141
end
4242

@@ -62,8 +62,11 @@ def totals_words(totals)
6262
totals.map { |name, percent| "#{name} #{format("%.2f%%", percent)}" }.join(", ")
6363
end
6464

65-
def contexts_words(count)
66-
count ? "#{count} (track_tests)" : "none (enable track_tests to select and re-run by test)"
65+
def contexts_words(count, dropped)
66+
return "#{count} (track_tests)" if count
67+
return "none (dropped because #{dropped})" if dropped
68+
69+
"none (enable track_tests to select and re-run by test)"
6770
end
6871

6972
def print_resultset(facts, stdout)

‎lib/simplecov/cli/status/facts.rb‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,11 @@ module Facts
88

99
def gather(document, resultset_path)
1010
contexts = document["contexts"]
11+
dropped = document["contexts_dropped"]
1112
meta_facts(document).merge(
1213
totals: totals(document["total"]),
1314
contexts: (tests_recorded(contexts) if contexts.instance_of?(Array)),
15+
contexts_dropped: (dropped if dropped.instance_of?(String)),
1416
resultset_path: resultset_path, resultset: readable_resultset(resultset_path)
1517
)
1618
end

‎lib/simplecov/context_map/union.rb‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ def absorb_resultset(resultset)
2929
def absorb_entry(data)
3030
@entries += 1
3131
map = ContextMap.from_hash(data["contexts"])
32-
return @complete = false unless map
32+
return drop_entry(data["contexts_dropped"]) unless map
3333

3434
@carrying += 1
3535
@map.absorb(map)
@@ -41,6 +41,7 @@ def absorb_union(other)
4141
@entries += other.entries
4242
@carrying += other.carrying
4343
@complete &&= other.complete?
44+
@reason ||= other.reason
4445
@map.absorb(other.partial_map)
4546
self
4647
end
@@ -59,15 +60,36 @@ def map
5960
warn_about_partial_maps
6061
end
6162

63+
# Why the merged result carries no map, for the report to say: the
64+
# first reason a merged result gave, or the merge's own when only some
65+
# merged results recorded one. nil when nothing was dropped.
66+
def dropped
67+
return if @complete
68+
69+
@reason || (partial_reason if @carrying.positive?)
70+
end
71+
6272
protected
6373

64-
# Protected: only another union folding this one in may read it.
74+
# Protected: only another union folding this one in may read these.
6575
def partial_map
6676
@map
6777
end
6878

79+
attr_reader :reason
80+
6981
private
7082

83+
def drop_entry(reason)
84+
@reason ||= reason if reason.instance_of?(String)
85+
@complete = false
86+
end
87+
88+
def partial_reason
89+
"only #{@carrying} of #{@entries} merged results recorded one, so enable `track_tests` in every suite " \
90+
"that should contribute"
91+
end
92+
7193
def warn_about_partial_maps
7294
return unless @carrying.positive? && SimpleCov.print_errors
7395

‎lib/simplecov/formatter/json_formatter/result_hash_formatter.rb‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ def format(result, include_source: true)
4040
# and production coverage only when a store is configured and readable.
4141
def add_optional_sections(document, result)
4242
document[:contexts] = result.contexts.contexts if result.contexts
43+
document[:contexts_dropped] = result.contexts_dropped if result.contexts_dropped
4344
history = History.entries_with(result)
4445
document[:history] = history if history.length > 1
4546
production = ProductionSectionFormatter.call

‎lib/simplecov/parallel_result_merger.rb‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ def merge_results(*file_paths, processes:, ignore_timeout: false)
3131
return ResultMerger.merge_results(*file_paths, ignore_timeout: ignore_timeout) unless pair
3232

3333
command_names, coverage = pair
34-
ResultMerger.create_result(command_names, coverage, tracked_files: tracked_files, contexts: context_maps.map)
34+
ResultMerger.create_result(command_names, coverage, tracked_files: tracked_files, contexts: context_maps.map,
35+
contexts_dropped: context_maps.dropped)
3536
end
3637

3738
# Answers nil when the work could not be fanned out and the caller should

‎lib/simplecov/result.rb‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@ class Result
2929
# carries none: tracking was off, or a merge dropped the map because not
3030
# every merged result recorded one.
3131
attr_reader :contexts
32+
# Why no map was recorded although `track_tests` was on, or nil.
33+
attr_reader :contexts_dropped
3234

3335
# The distinct run names behind this result, set by a merge so presentation
3436
# can say "A and N other runs" instead of reading the joined
@@ -50,12 +52,13 @@ class Result
5052
# `tracked_files` accepts any collection that answers `to_a` (the merge
5153
# passes a Set), and nil for a run that tracked nothing.
5254
def initialize(original_result, command_name: nil, created_at: nil, not_loaded_files: Set.new,
53-
tracked_files: nil, run_id: nil, worker_id: nil, contexts: nil, report: false,
55+
tracked_files: nil, run_id: nil, worker_id: nil, contexts: nil, contexts_dropped: nil, report: false,
5456
filter_config: FilterConfig.new)
5557
@original_result = original_result.freeze
5658
@command_name = command_name
5759
@created_at = created_at
5860
initialize_resultset_metadata(tracked_files, run_id, worker_id, contexts)
61+
@contexts_dropped = contexts_dropped
5962
@groups_config = filter_config.groups
6063
builder = SourceFileBuilder.new(original_result, not_loaded_files: not_loaded_files)
6164
@files = builder.call
@@ -122,7 +125,8 @@ def self.from_hash(hash)
122125
hash.map do |command_name, data|
123126
new(data.fetch("coverage"), command_name: command_name, created_at: Time.at(data.fetch("timestamp")),
124127
tracked_files: data["tracked_files"], run_id: data["run_id"],
125-
worker_id: data["worker_id"], contexts: ContextMap.from_hash(data["contexts"]))
128+
worker_id: data["worker_id"], contexts: ContextMap.from_hash(data["contexts"]),
129+
contexts_dropped: data["contexts_dropped"])
126130
end
127131
end
128132

‎lib/simplecov/result/serialization.rb‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ def to_hash
99
data["worker_id"] = worker_id if worker_id
1010
data["tracked_files"] = tracked_files unless tracked_files.empty?
1111
append_contexts(data)
12+
data["contexts_dropped"] = contexts_dropped if contexts_dropped
1213
{command_name => data}
1314
end
1415

0 commit comments

Comments
 (0)