Conversation
Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>
Codeowners resolved as |
This comment has been minimized.
This comment has been minimized.
|
I am not sure I understand this benchmark, why it is set up the way that it is. for example, we are measuring some timing data in the child process/target and we return an elapsed time back to the parent, but we don't do anything with it? the way our scenarios are set up, |
Performance SLOsComparing candidate juanjux/fork-benchmarks (392ecdb) with baseline main (937997f) ❌ Test Failures (1 suite)❌ forktime - 0/4❌ baselineTime: ❌ 1.937ms (SLO: <1.750ms 📈 +10.7%) vs baseline: +3.7% Memory: ❌ 29.452MB (SLO: <20.000MB 📈 +47.3%) vs baseline: +5.4% ❌ configuredTime: ❌ 8.616ms (SLO: <4.500ms 📈 +91.5%) vs baseline: +0.3% Memory: ❌ 57.338MB (SLO: <53.000MB +8.2%) vs baseline: +4.7% 📈 Performance Regressions (4 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 105.345µs (SLO: <130.000µs 📉 -19.0%) vs baseline: +4.1% Memory: ✅ 43.136MB (SLO: <46.000MB -6.2%) vs baseline: +5.3% ✅ add_inplace_aspectTime: ✅ 102.027µs (SLO: <130.000µs 📉 -21.5%) vs baseline: +0.2% Memory: ✅ 43.214MB (SLO: <46.000MB -6.1%) vs baseline: +5.4% ✅ add_inplace_noaspectTime: ✅ 28.141µs (SLO: <40.000µs 📉 -29.6%) vs baseline: +0.4% Memory: ✅ 43.195MB (SLO: <46.000MB -6.1%) vs baseline: +5.5% ✅ add_noaspectTime: ✅ 48.839µs (SLO: <70.000µs 📉 -30.2%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.8% ✅ bytearray_aspectTime: ✅ 251.916µs (SLO: <400.000µs 📉 -37.0%) vs baseline: +0.5% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ bytearray_extend_aspectTime: ✅ 651.687µs (SLO: <800.000µs 📉 -18.5%) vs baseline: +2.4% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ bytearray_extend_noaspectTime: ✅ 269.316µs (SLO: <400.000µs 📉 -32.7%) vs baseline: +1.2% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 139.616µs (SLO: <300.000µs 📉 -53.5%) vs baseline: +1.0% Memory: ✅ 43.175MB (SLO: <46.000MB -6.1%) vs baseline: +5.1% ✅ bytes_aspectTime: ✅ 219.688µs (SLO: <300.000µs 📉 -26.8%) vs baseline: +1.0% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.3% ✅ bytes_noaspectTime: ✅ 134.996µs (SLO: <200.000µs 📉 -32.5%) vs baseline: -0.5% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.1% ✅ bytesio_aspectTime: ✅ 3.863ms (SLO: <5.000ms 📉 -22.7%) vs baseline: +0.1% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ bytesio_noaspectTime: ✅ 320.376µs (SLO: <420.000µs 📉 -23.7%) vs baseline: +0.8% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ capitalize_aspectTime: ✅ 88.211µs (SLO: <300.000µs 📉 -70.6%) vs baseline: +0.4% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ capitalize_noaspectTime: ✅ 260.354µs (SLO: <300.000µs 📉 -13.2%) vs baseline: +3.5% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ casefold_aspectTime: ✅ 89.262µs (SLO: <500.000µs 📉 -82.1%) vs baseline: +1.6% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ casefold_noaspectTime: ✅ 312.226µs (SLO: <500.000µs 📉 -37.6%) vs baseline: +1.5% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.9% ✅ decode_aspectTime: ✅ 87.126µs (SLO: <100.000µs 📉 -12.9%) vs baseline: ~same Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ decode_noaspectTime: ✅ 154.570µs (SLO: <210.000µs 📉 -26.4%) vs baseline: +1.3% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 85.157µs (SLO: <200.000µs 📉 -57.4%) vs baseline: ~same Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.4% ✅ encode_noaspectTime: ✅ 144.552µs (SLO: <200.000µs 📉 -27.7%) vs baseline: +4.1% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.8% ✅ format_aspectTime: ✅ 14.679ms (SLO: <19.200ms 📉 -23.5%) vs baseline: ~same Memory: ✅ 43.175MB (SLO: <46.000MB -6.1%) vs baseline: +5.0% ✅ format_map_aspectTime: ✅ 16.442ms (SLO: <21.500ms 📉 -23.5%) vs baseline: +0.1% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +4.7% ✅ format_map_noaspectTime: ✅ 378.701µs (SLO: <500.000µs 📉 -24.3%) vs baseline: +2.2% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.1% ✅ format_noaspectTime: ✅ 306.985µs (SLO: <500.000µs 📉 -38.6%) vs baseline: +1.7% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ index_aspectTime: ✅ 127.738µs (SLO: <300.000µs 📉 -57.4%) vs baseline: +2.9% Memory: ✅ 43.136MB (SLO: <46.000MB -6.2%) vs baseline: +5.2% ✅ index_noaspectTime: ✅ 40.312µs (SLO: <300.000µs 📉 -86.6%) vs baseline: -0.2% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ join_aspectTime: ✅ 212.991µs (SLO: <300.000µs 📉 -29.0%) vs baseline: -1.6% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.2% ✅ join_noaspectTime: ✅ 140.495µs (SLO: <300.000µs 📉 -53.2%) vs baseline: -2.4% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ ljust_aspectTime: ✅ 602.977µs (SLO: <700.000µs 📉 -13.9%) vs baseline: 📈 +20.1% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ ljust_noaspectTime: ✅ 254.557µs (SLO: <300.000µs 📉 -15.1%) vs baseline: +0.9% Memory: ✅ 43.155MB (SLO: <46.000MB -6.2%) vs baseline: +5.4% ✅ lower_aspectTime: ✅ 298.337µs (SLO: <500.000µs 📉 -40.3%) vs baseline: +1.6% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ lower_noaspectTime: ✅ 235.354µs (SLO: <300.000µs 📉 -21.5%) vs baseline: +1.9% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ lstrip_aspectTime: ✅ 0.280ms (SLO: <3.000ms 📉 -90.7%) vs baseline: -0.2% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 0.180ms (SLO: <3.000ms 📉 -94.0%) vs baseline: -0.5% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +4.7% ✅ modulo_aspectTime: ✅ 14.368ms (SLO: <18.750ms 📉 -23.4%) vs baseline: ~same Memory: ✅ 43.195MB (SLO: <46.000MB -6.1%) vs baseline: +5.2% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.809ms (SLO: <19.350ms 📉 -23.5%) vs baseline: -0.8% Memory: ✅ 43.195MB (SLO: <46.000MB -6.1%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 14.401ms (SLO: <18.900ms 📉 -23.8%) vs baseline: -0.3% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.1% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.669ms (SLO: <19.150ms 📉 -23.4%) vs baseline: +0.3% Memory: ✅ 43.155MB (SLO: <46.000MB -6.2%) vs baseline: +4.9% ✅ modulo_noaspectTime: ✅ 0.361ms (SLO: <3.000ms 📉 -88.0%) vs baseline: +1.4% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ replace_aspectTime: ✅ 18.350ms (SLO: <24.000ms 📉 -23.5%) vs baseline: -0.7% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ replace_noaspectTime: ✅ 280.774µs (SLO: <300.000µs -6.4%) vs baseline: -1.1% Memory: ✅ 43.234MB (SLO: <46.000MB -6.0%) vs baseline: +5.3% ✅ repr_aspectTime: ✅ 314.337µs (SLO: <420.000µs 📉 -25.2%) vs baseline: -2.0% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ repr_noaspectTime: ✅ 46.522µs (SLO: <90.000µs 📉 -48.3%) vs baseline: -0.3% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.8% ✅ rstrip_aspectTime: ✅ 392.807µs (SLO: <500.000µs 📉 -21.4%) vs baseline: -1.2% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ rstrip_noaspectTime: ✅ 186.200µs (SLO: <300.000µs 📉 -37.9%) vs baseline: +0.8% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ slice_aspectTime: ✅ 183.523µs (SLO: <300.000µs 📉 -38.8%) vs baseline: -2.1% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ slice_noaspectTime: ✅ 54.317µs (SLO: <90.000µs 📉 -39.6%) vs baseline: +1.0% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ stringio_aspectTime: ✅ 4.530ms (SLO: <5.000ms -9.4%) vs baseline: 📈 +15.9% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 358.174µs (SLO: <500.000µs 📉 -28.4%) vs baseline: +3.6% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.8% ✅ strip_aspectTime: ✅ 274.717µs (SLO: <350.000µs 📉 -21.5%) vs baseline: -1.4% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.7% ✅ strip_noaspectTime: ✅ 177.882µs (SLO: <240.000µs 📉 -25.9%) vs baseline: -1.5% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +4.7% ✅ swapcase_aspectTime: ✅ 340.672µs (SLO: <500.000µs 📉 -31.9%) vs baseline: +2.5% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ swapcase_noaspectTime: ✅ 275.349µs (SLO: <400.000µs 📉 -31.2%) vs baseline: +1.1% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.3% ✅ title_aspectTime: ✅ 324.907µs (SLO: <500.000µs 📉 -35.0%) vs baseline: +1.1% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.3% ✅ title_noaspectTime: ✅ 262.713µs (SLO: <400.000µs 📉 -34.3%) vs baseline: +3.6% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ translate_aspectTime: ✅ 504.412µs (SLO: <700.000µs 📉 -27.9%) vs baseline: +3.7% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ translate_noaspectTime: ✅ 427.077µs (SLO: <500.000µs 📉 -14.6%) vs baseline: +1.5% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.8% ✅ upper_aspectTime: ✅ 298.495µs (SLO: <500.000µs 📉 -40.3%) vs baseline: +1.9% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ upper_noaspectTime: ✅ 235.383µs (SLO: <400.000µs 📉 -41.2%) vs baseline: +1.8% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 506.539µs (SLO: <700.000µs 📉 -27.6%) vs baseline: 📈 +18.4% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 428.977µs (SLO: <700.000µs 📉 -38.7%) vs baseline: -1.6% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.7% ✅ ospathjoin_aspectTime: ✅ 618.839µs (SLO: <700.000µs 📉 -11.6%) vs baseline: -0.3% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +5.3% ✅ ospathjoin_noaspectTime: ✅ 626.167µs (SLO: <700.000µs 📉 -10.5%) vs baseline: -0.7% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.8% ✅ ospathnormcase_aspectTime: ✅ 350.368µs (SLO: <700.000µs 📉 -49.9%) vs baseline: -2.5% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ ospathnormcase_noaspectTime: ✅ 356.095µs (SLO: <700.000µs 📉 -49.1%) vs baseline: -2.8% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ ospathsplit_aspectTime: ✅ 487.903µs (SLO: <700.000µs 📉 -30.3%) vs baseline: -0.9% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 498.533µs (SLO: <700.000µs 📉 -28.8%) vs baseline: +0.4% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 370.801µs (SLO: <700.000µs 📉 -47.0%) vs baseline: -1.6% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.7% ✅ ospathsplitdrive_noaspectTime: ✅ 73.149µs (SLO: <700.000µs 📉 -89.6%) vs baseline: +0.7% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ ospathsplitext_aspectTime: ✅ 460.933µs (SLO: <700.000µs 📉 -34.2%) vs baseline: -0.4% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ ospathsplitext_noaspectTime: ✅ 467.042µs (SLO: <700.000µs 📉 -33.3%) vs baseline: +0.4% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.1% 📈 iastaspectssplit - 12/12✅ rsplit_aspectTime: ✅ 168.920µs (SLO: <250.000µs 📉 -32.4%) vs baseline: 📈 +15.5% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ rsplit_noaspectTime: ✅ 167.743µs (SLO: <250.000µs 📉 -32.9%) vs baseline: +9.3% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ split_aspectTime: ✅ 151.930µs (SLO: <250.000µs 📉 -39.2%) vs baseline: +4.6% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.9% ✅ split_noaspectTime: ✅ 164.892µs (SLO: <250.000µs 📉 -34.0%) vs baseline: +9.0% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.8% ✅ splitlines_aspectTime: ✅ 151.330µs (SLO: <250.000µs 📉 -39.5%) vs baseline: +5.0% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.7% ✅ splitlines_noaspectTime: ✅ 161.809µs (SLO: <250.000µs 📉 -35.3%) vs baseline: +8.3% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.543µs (SLO: <20.000µs 📉 -82.3%) vs baseline: 📈 +22.8% Memory: ✅ 35.527MB (SLO: <38.000MB -6.5%) vs baseline: +4.7% ✅ 1-count-metrics-100-timesTime: ✅ 207.220µs (SLO: <220.000µs -5.8%) vs baseline: +3.5% Memory: ✅ 35.665MB (SLO: <38.000MB -6.1%) vs baseline: +5.2% ✅ 1-distribution-metric-1-timesTime: ✅ 3.388µs (SLO: <20.000µs 📉 -83.1%) vs baseline: +6.2% Memory: ✅ 35.625MB (SLO: <38.000MB -6.2%) vs baseline: +4.7% ✅ 1-distribution-metrics-100-timesTime: ✅ 219.680µs (SLO: <230.000µs -4.5%) vs baseline: +3.6% Memory: ✅ 35.665MB (SLO: <38.000MB -6.1%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.202µs (SLO: <20.000µs 📉 -89.0%) vs baseline: +1.2% Memory: ✅ 35.684MB (SLO: <38.000MB -6.1%) vs baseline: +5.0% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.390µs (SLO: <150.000µs -9.1%) vs baseline: -1.4% Memory: ✅ 35.704MB (SLO: <38.000MB -6.0%) vs baseline: +5.0% ✅ 1-rate-metric-1-timesTime: ✅ 3.200µs (SLO: <20.000µs 📉 -84.0%) vs baseline: +5.7% Memory: ✅ 35.704MB (SLO: <38.000MB -6.0%) vs baseline: +5.3% ✅ 1-rate-metrics-100-timesTime: ✅ 222.431µs (SLO: <250.000µs 📉 -11.0%) vs baseline: +4.5% Memory: ✅ 35.645MB (SLO: <38.000MB -6.2%) vs baseline: +4.9% ✅ 100-count-metrics-100-timesTime: ✅ 20.272ms (SLO: <22.000ms -7.9%) vs baseline: +1.5% Memory: ✅ 35.566MB (SLO: <38.000MB -6.4%) vs baseline: +4.7% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.279ms (SLO: <2.550ms 📉 -10.6%) vs baseline: +2.1% Memory: ✅ 35.665MB (SLO: <38.000MB -6.1%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.401ms (SLO: <1.550ms -9.6%) vs baseline: +0.4% Memory: ✅ 35.665MB (SLO: <38.000MB -6.1%) vs baseline: +5.0% ✅ 100-rate-metrics-100-timesTime: ✅ 2.222ms (SLO: <2.550ms 📉 -12.9%) vs baseline: +1.7% Memory: ✅ 35.606MB (SLO: <38.000MB -6.3%) vs baseline: +5.1% ✅ flush-1-metricTime: ✅ 4.642µs (SLO: <20.000µs 📉 -76.8%) vs baseline: +2.4% Memory: ✅ 35.625MB (SLO: <38.000MB -6.2%) vs baseline: +5.1% ✅ flush-100-metricsTime: ✅ 173.519µs (SLO: <250.000µs 📉 -30.6%) vs baseline: ~same Memory: ✅ 35.665MB (SLO: <38.000MB -6.1%) vs baseline: +5.2% ✅ flush-1000-metricsTime: ✅ 2.175ms (SLO: <2.500ms 📉 -13.0%) vs baseline: ~same Memory: ✅ 36.412MB (SLO: <38.750MB -6.0%) vs baseline: +4.9%
|
Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>
Damn, I swear I set this as draft, but it seem not. Anyway, I have just pushed the finished version. The idea is that the child sends the time to the parent (which started the "chronometer") so we know the fork time. And, as you suggested, I am overriding |
|
Added SLOs (the initial times are the ones on my machine which is probably faster than CI, so I will adjust it based on the first CI runs). |
Update SLO thresholds with realistic values for CI
Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>
Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>
Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>
brettlangdon
left a comment
There was a problem hiding this comment.
SLO check is showing this as near SLO breach, so we might want to adjust the starting SLOs to be a bit higher
Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>
Yeah, waiting for the new SLO values after reworking the scenario. |
Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>
Description
Add a benchmark to measure fork time.