Skip to content

Commit f4d9f36

Browse files
Merge pull request #85 from ktsu-dev/claude/magical-knuth-idzj5r
2 parents d575159 + 66be68d commit f4d9f36

9 files changed

Lines changed: 906 additions & 488 deletions

File tree

‎.github/workflows/benchmark-history.yml‎

Lines changed: 52 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,15 @@ name: Benchmark History
55
# which exists to get a full ad-hoc run on demand and publishes nothing.
66
#
77
# Two ways in:
8-
# * a published release, which measures that version and adds one point;
9-
# * a manual dispatch listing refs, which measures each of them in ONE job and backfills.
8+
# * a published release, which measures that version from its own tag and adds one point;
9+
# * a manual dispatch listing versions, which measures each published package in ONE job.
10+
#
11+
# A backfill measures packages rather than tags. Both are defensible until a benchmark is added: a
12+
# tag predating one cannot run it, so a new measurement could only ever start from the next release,
13+
# and AbstractionCostBenchmarks would have had no history at all. Measuring the published packages
14+
# with today's sources gives it the same history as everything else, and times every version by
15+
# identical code besides. ktsu.SignificantNumber and ktsu.Semantics do the same, so the three
16+
# charts are built the same way.
1017
#
1118
# The backfill running as a single job is the point rather than an optimisation. Separate runs land
1219
# on different CI hosts, and the difference between an x86-64-v3 and a v4 runner is larger than
@@ -18,15 +25,10 @@ on:
1825
types: [published]
1926
workflow_dispatch:
2027
inputs:
21-
refs:
22-
description: "Space-separated refs to backfill, oldest first (tags, branches, or SHAs)"
23-
required: false
24-
default: "cd9a8227793bbd6ea791be7f6c0579ae1242eec5 v2.0.0 v2.0.1 v2.0.2 v2.0.3 v2.0.4"
25-
type: string
26-
labels:
27-
description: "Optional space-separated version labels matching refs, when a ref is not a version"
28+
versions:
29+
description: "Space-separated released versions to backfill, oldest first"
2830
required: false
29-
default: "1.9.0 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4"
31+
default: "1.8.0 1.9.0 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4 2.0.5"
3032
type: string
3133

3234
permissions:
@@ -52,6 +54,7 @@ env:
5254
*ConstructionBenchmarks.Sanitizing
5355
*TextBenchmarks.Parse
5456
*ConversionBenchmarks.ToDouble
57+
*AbstractionCostBenchmarks.*
5558
# Short runs: three iterations is enough for a trend line, and a release should not tie up a
5659
# runner for half an hour. benchmarks.yml is still there for a full-length run.
5760
BENCHMARK_JOB: short
@@ -121,51 +124,55 @@ jobs:
121124
122125
git worktree remove --force "$work"
123126
124-
- name: Measure each backfill ref
127+
- name: Measure each backfill version
125128
if: github.event_name == 'workflow_dispatch'
126129
shell: bash
127130
env:
128-
REFS: ${{ inputs.refs }}
129-
LABELS: ${{ inputs.labels }}
131+
VERSIONS: ${{ inputs.versions }}
130132
BASELINE_NS: ${{ steps.baseline.outputs.ns }}
131133
run: |
132134
set -euo pipefail
133-
read -ra refs <<< "$REFS"
134-
read -ra labels <<< "$LABELS"
135-
136-
for index in "${!refs[@]}"; do
137-
ref="${refs[$index]}"
138-
label="${labels[$index]:-${ref#v}}"
139-
work="${RUNNER_TEMP}/bench-$label"
140-
141-
echo "::group::$label ($ref)"
142-
rm -rf "$work"
143-
git worktree add --detach "$work" "$ref"
144-
145-
if [ ! -f "$work/PreciseNumber.Benchmarks/PreciseNumber.Benchmarks.csproj" ]; then
146-
echo "::warning::$ref has no benchmark project; skipping"
147-
git worktree remove --force "$work"
135+
read -ra versions <<< "$VERSIONS"
136+
137+
for version in "${versions[@]}"; do
138+
echo "::group::$version"
139+
# Through the environment rather than a -p: switch, because BenchmarkDotNet generates
140+
# and builds a project of its own per run, which a property passed on the command line
141+
# does not reach. MSBuild reads environment variables as properties in every project.
142+
#
143+
# A version whose API the current benchmarks cannot express is reported and skipped,
144+
# rather than failing the whole backfill after the ones before it have been measured.
145+
if ! BenchmarkAgainstVersion="$version" dotnet run -c Release --project PreciseNumber.Benchmarks -- \
146+
--filter $HEADLINE_FILTER \
147+
--job "$BENCHMARK_JOB" \
148+
--artifacts "$GITHUB_WORKSPACE/$RUNS/$version"; then
149+
echo "::warning::$version could not be benchmarked by the current suite; skipping"
148150
echo "::endgroup::"
149151
continue
150152
fi
151153
152-
# Each ref is measured by its own benchmark sources. Between 2.0.0 and now those
153-
# sources are unchanged, so this compares library versions rather than harnesses.
154-
(cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \
155-
--filter $HEADLINE_FILTER \
156-
--job "$BENCHMARK_JOB" \
157-
--artifacts "$GITHUB_WORKSPACE/$RUNS/$label")
158-
159-
dotnet run scripts/benchmark-history.cs -- ingest \
160-
--history "$HISTORY" \
161-
--results "$RUNS/$label" \
162-
--version "$label" \
163-
--commit "$(git rev-parse --short "$ref^{commit}")" \
164-
--date "$(git log -1 --format=%cs "$ref")" \
165-
--run-id "${{ github.run_id }}" \
166-
--baseline-ns "$BASELINE_NS"
167-
168-
git worktree remove --force "$work"
154+
tag="v$version"
155+
commit=""
156+
date=""
157+
if git rev-parse -q --verify "$tag^{commit}" >/dev/null; then
158+
commit="$(git rev-parse --short "$tag^{commit}")"
159+
date="$(git log -1 --format=%cs "$tag")"
160+
fi
161+
162+
# Skipped here too, and for the same reason: a package can build against these
163+
# benchmarks and still throw from every one of them at run time, which BenchmarkDotNet
164+
# reports as a table of NA rather than as a failure. Ingest refuses such a run, and
165+
# the backfill carries on to the next version.
166+
if ! dotnet run scripts/benchmark-history.cs -- ingest \
167+
--history "$HISTORY" \
168+
--results "$RUNS/$version" \
169+
--version "$version" \
170+
--commit "$commit" \
171+
--date "$date" \
172+
--run-id "${{ github.run_id }}" \
173+
--baseline-ns "$BASELINE_NS"; then
174+
echo "::warning::$version produced no usable measurement; skipping"
175+
fi
169176
echo "::endgroup::"
170177
done
171178

‎Directory.Packages.props‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,10 @@
66
<PackageVersion Include="BenchmarkDotNet" Version="0.15.8" />
77
<PackageVersion Include="BenchmarkDotNet.Annotations" Version="0.15.8" />
88
<PackageVersion Include="Polyfill" Version="11.3.0" />
9+
<!-- Only the benchmark project references this, and only when BenchmarkAgainstVersion asks it
10+
to measure a published release rather than the working copy. The version here is a
11+
placeholder that VersionOverride replaces; central package management requires the entry
12+
to exist before a project may override it. -->
13+
<PackageVersion Include="ktsu.PreciseNumber" Version="2.0.5" />
914
</ItemGroup>
1015
</Project>
Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
// Copyright (c) 2023-2026 ktsu-dev contributors
2+
3+
namespace ktsu.PreciseNumber.Benchmarks;
4+
5+
using System.Globalization;
6+
7+
using BenchmarkDotNet.Attributes;
8+
using BenchmarkDotNet.Configs;
9+
10+
/// <summary>
11+
/// Measures what this type costs against the same arithmetic on a bare <see cref="double"/>.
12+
/// </summary>
13+
/// <remarks>
14+
/// <para>
15+
/// Every other class here answers "how long does this operation take", which is only readable next
16+
/// to something. This one supplies the something: the primitive a caller would otherwise have
17+
/// used. The same class, with the same loops and the same methodology, is in ktsu.SignificantNumber
18+
/// and ktsu.Semantics, so the three answers are comparable with each other as well as with
19+
/// <see cref="double"/>.
20+
/// </para>
21+
/// <para>
22+
/// The bare method is the BenchmarkDotNet baseline, so the answer is the <c>Ratio</c> column rather
23+
/// than two rows divided by hand. Unlike the wrapper types built on this one, a ratio here is not
24+
/// expected to be 1.00 and is not a defect when it is not: arbitrary precision is a cost paid for
25+
/// something a <see cref="double"/> cannot do at all. What the number is for is watching that cost
26+
/// across releases.
27+
/// </para>
28+
/// <para>
29+
/// <b>Why these are loops.</b> A single operation over operands that do not change is
30+
/// loop-invariant, and the JIT hoists it out of the measurement entirely — for
31+
/// <see cref="double"/> that leaves a method indistinguishable from an empty one, and a ratio
32+
/// against an empty method means nothing. Here each iteration feeds the next, so there is nothing
33+
/// to hoist and both sides are measurable.
34+
/// </para>
35+
/// <para>
36+
/// <b>Which way the loop biases the answer.</b> Both sides pay the same counter and branch, and it
37+
/// is a dependency chain, so most of that overlaps the arithmetic; whatever does not is added
38+
/// equally to numerator and denominator and pulls the ratio toward 1.00. A ratio here is therefore
39+
/// a floor on the real cost rather than the whole of it.
40+
/// </para>
41+
/// <para>
42+
/// <b>Why the operands stay bounded, and why they are short.</b> This type carries as many digits
43+
/// as the arithmetic produces, so a chain that compounded its operand would measure that growth
44+
/// rather than the operation; both loops accumulate instead. The operands are also chosen to be
45+
/// values a <see cref="double"/> can hold, so the two sides are doing the same arithmetic on the
46+
/// same numbers rather than being handed different problems. How the cost grows with digits is a
47+
/// different question, and <see cref="ArithmeticBenchmarks"/> answers it across its
48+
/// <c>Digits</c> axis.
49+
/// </para>
50+
/// </remarks>
51+
[MemoryDiagnoser]
52+
[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
53+
[CategoriesColumn]
54+
public class AbstractionCostBenchmarks
55+
{
56+
/// <summary>
57+
/// Operations per invocation. Enough that the loop's own cost is a small share of the work,
58+
/// few enough that the arbitrary-precision side still finishes an iteration promptly.
59+
/// </summary>
60+
private const int Operations = 256;
61+
62+
private const string SeedText = "1234.5678901234";
63+
private const string StepText = "0.0009765625";
64+
private const string OtherText = "3.14159265358979";
65+
66+
private double bareSeed;
67+
private double bareStep;
68+
private double bareOther;
69+
70+
// Assigned in GlobalSetup before anything is measured. Initialised here because this type
71+
// was a class before 2.0, where an unassigned field is a null reference the compiler
72+
// rejects; from 2.0 it is a struct and this is simply its default. The backfill measures
73+
// those releases too, so the file has to compile against both shapes.
74+
private PreciseNumber preciseSeed = default!;
75+
private PreciseNumber preciseStep = default!;
76+
private PreciseNumber preciseOther = default!;
77+
78+
/// <summary>
79+
/// Prepares the operands, parsed from the same text on both sides.
80+
/// </summary>
81+
[GlobalSetup]
82+
public void Setup()
83+
{
84+
bareSeed = double.Parse(SeedText, CultureInfo.InvariantCulture);
85+
bareStep = double.Parse(StepText, CultureInfo.InvariantCulture);
86+
bareOther = double.Parse(OtherText, CultureInfo.InvariantCulture);
87+
88+
preciseSeed = PreciseNumber.Parse(SeedText, CultureInfo.InvariantCulture);
89+
preciseStep = PreciseNumber.Parse(StepText, CultureInfo.InvariantCulture);
90+
preciseOther = PreciseNumber.Parse(OtherText, CultureInfo.InvariantCulture);
91+
}
92+
93+
/// <summary>Adds along a chain, on a bare double.</summary>
94+
/// <returns>The accumulated value.</returns>
95+
[BenchmarkCategory("Add")]
96+
[Benchmark(Baseline = true, OperationsPerInvoke = Operations)]
97+
public double BareAdd()
98+
{
99+
double accumulator = bareSeed;
100+
101+
for (int i = 0; i < Operations; i++)
102+
{
103+
accumulator += bareStep;
104+
}
105+
106+
return accumulator;
107+
}
108+
109+
/// <summary>Adds along the same chain, on this type.</summary>
110+
/// <returns>The accumulated value.</returns>
111+
[BenchmarkCategory("Add")]
112+
[Benchmark(OperationsPerInvoke = Operations)]
113+
public PreciseNumber PreciseAdd()
114+
{
115+
PreciseNumber accumulator = preciseSeed;
116+
117+
for (int i = 0; i < Operations; i++)
118+
{
119+
accumulator += preciseStep;
120+
}
121+
122+
return accumulator;
123+
}
124+
125+
/// <summary>Multiplies and accumulates, on a bare double.</summary>
126+
/// <returns>The accumulated value.</returns>
127+
[BenchmarkCategory("Multiply")]
128+
[Benchmark(Baseline = true, OperationsPerInvoke = Operations)]
129+
public double BareMultiply()
130+
{
131+
double accumulator = 0d;
132+
double value = bareSeed;
133+
134+
for (int i = 0; i < Operations; i++)
135+
{
136+
accumulator += value * bareOther;
137+
value += bareStep;
138+
}
139+
140+
return accumulator;
141+
}
142+
143+
/// <summary>Multiplies and accumulates over the same values, on this type.</summary>
144+
/// <returns>The accumulated value.</returns>
145+
[BenchmarkCategory("Multiply")]
146+
[Benchmark(OperationsPerInvoke = Operations)]
147+
public PreciseNumber PreciseMultiply()
148+
{
149+
PreciseNumber accumulator = PreciseNumber.Zero;
150+
PreciseNumber value = preciseSeed;
151+
152+
for (int i = 0; i < Operations; i++)
153+
{
154+
accumulator += value * preciseOther;
155+
value += preciseStep;
156+
}
157+
158+
return accumulator;
159+
}
160+
}

‎PreciseNumber.Benchmarks/PreciseNumber.Benchmarks.csproj‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,34 @@
2929
<PackageReference Include="BenchmarkDotNet.Annotations" />
3030
</ItemGroup>
3131

32-
<ItemGroup>
32+
<!-- Normally the working copy. Set BenchmarkAgainstVersion to measure a released package
33+
instead, in the environment of the same `dotnet run` that names this project, as
34+
`BenchmarkAgainstVersion=2.0.4 dotnet run -c Release ...`. An environment variable rather
35+
than a `-p:` switch, because BenchmarkDotNet builds each benchmark in an autogenerated
36+
project of its own, which a property set on the outer run never reaches. MSBuild reads
37+
environment variables as properties in every project, including that one.
38+
39+
That switch is how the release history is charted, and it replaced measuring each tag from
40+
its own worktree. Both are defensible until a benchmark is added: a tag predating it cannot
41+
run it, so a new measurement could only ever start from the next release. Measuring the
42+
published packages with today's sources gives the new benchmark the same history as the old
43+
ones, and is the better comparison besides, since every version is then timed by identical
44+
code rather than by whatever each tag happened to ship. It is also what ktsu.SignificantNumber
45+
and ktsu.Semantics do, so the three charts are built the same way. -->
46+
<ItemGroup Condition="'$(BenchmarkAgainstVersion)' == ''">
3347
<ProjectReference Include="..\PreciseNumber\PreciseNumber.csproj" />
3448
</ItemGroup>
49+
<ItemGroup Condition="'$(BenchmarkAgainstVersion)' != ''">
50+
<PackageReference Include="ktsu.PreciseNumber" VersionOverride="$(BenchmarkAgainstVersion)" />
51+
</ItemGroup>
52+
53+
<!-- Construction is the one area measured through an internal factory, because building a number
54+
from its components is not otherwise reachable and is exactly what the sanitising path costs.
55+
A published package exposes nothing, so that file cannot compile against one; it is left out
56+
of those builds and the chart draws its panel from the working copy onward. Everything else
57+
here is public surface and measures identically either way. -->
58+
<ItemGroup Condition="'$(BenchmarkAgainstVersion)' != ''">
59+
<Compile Remove="ConstructionBenchmarks.cs" />
60+
</ItemGroup>
3561

3662
</Project>

‎PreciseNumber.Benchmarks/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,39 @@ each digit separately looks fine at 8 digits and falls apart at 200. Reading a t
5050
apart from operands in the same decade, because aligning two exponents is its own cost, distinct
5151
from the size of the operands.
5252

53+
## What this type costs against a bare double
54+
55+
`AbstractionCostBenchmarks` is the one benchmark here whose answer is a ratio rather than a
56+
duration. Every other class says how long an operation takes, which is only readable beside
57+
something; this supplies the something — the primitive a caller would otherwise have used.
58+
59+
The same class, with the same loops and the same methodology, is in `ktsu.SignificantNumber` and
60+
`ktsu.Semantics`, so the three libraries answer one question the same way and their answers are
61+
comparable with each other as well as with `double`.
62+
63+
| release | `Add` | `Multiply` |
64+
|---|---|---|
65+
| 1.8.0 | 99.7× | 293.9× |
66+
| 1.9.0 | 99.7× | 288.2× |
67+
| 2.0.0 | 82.8× | 248.8× |
68+
| 2.0.5 | 83.6× | 249.8× |
69+
70+
Becoming a value type in 2.0 took about 15% off the price of arbitrary precision, and six releases
71+
have held it there. **The ratio is not expected to be 1 and is not a defect for being large** — a
72+
`double` cannot do this at all. What the chart's third section is for is noticing the day it moves.
73+
74+
Three things decide how the number should be read:
75+
76+
- **These are loops.** A single operation over operands that do not change is loop-invariant and
77+
the JIT hoists it out, which would leave the `double` side indistinguishable from an empty method
78+
and the ratio meaningless. Each iteration feeds the next, so there is nothing to hoist.
79+
- **The loop's own cost biases toward 1**, being paid by both sides, so a ratio is a floor on the
80+
real cost rather than the whole of it.
81+
- **Both loops accumulate rather than compound**, because this type carries as many digits as the
82+
arithmetic produces and a compounding chain would measure that growth instead of the operation.
83+
How the cost grows with digits is a different question, and `ArithmeticBenchmarks` answers it
84+
across the `Digits` axis.
85+
5386
## Reading the results
5487

5588
Allocation is reported next to time. Both matter here, and they trade against each other: every

0 commit comments

Comments
 (0)