chore(ci_visibility): upload coverage report from pytest plugin v2#16302
chore(ci_visibility): upload coverage report from pytest plugin v2#16302
Conversation
This comment has been minimized.
This comment has been minimized.
Performance SLOsComparing candidate gnufede/SDTEST-3005-4 (b15b91f) with baseline main (9e014a2) 📈 Performance Regressions (4 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 104.970µs (SLO: <130.000µs 📉 -19.3%) vs baseline: +3.0% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ add_inplace_aspectTime: ✅ 102.326µs (SLO: <130.000µs 📉 -21.3%) vs baseline: +0.4% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ add_inplace_noaspectTime: ✅ 28.144µs (SLO: <40.000µs 📉 -29.6%) vs baseline: -0.5% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ add_noaspectTime: ✅ 48.603µs (SLO: <70.000µs 📉 -30.6%) vs baseline: -0.9% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.1% ✅ bytearray_aspectTime: ✅ 251.628µs (SLO: <400.000µs 📉 -37.1%) vs baseline: -0.6% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.2% ✅ bytearray_extend_aspectTime: ✅ 647.987µs (SLO: <800.000µs 📉 -19.0%) vs baseline: +2.0% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ bytearray_extend_noaspectTime: ✅ 268.992µs (SLO: <400.000µs 📉 -32.8%) vs baseline: +0.7% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ bytearray_noaspectTime: ✅ 140.302µs (SLO: <300.000µs 📉 -53.2%) vs baseline: +1.2% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ bytes_aspectTime: ✅ 221.940µs (SLO: <300.000µs 📉 -26.0%) vs baseline: +0.7% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.3% ✅ bytes_noaspectTime: ✅ 135.634µs (SLO: <200.000µs 📉 -32.2%) vs baseline: -0.9% Memory: ✅ 43.195MB (SLO: <46.000MB -6.1%) vs baseline: +5.4% ✅ bytesio_aspectTime: ✅ 3.891ms (SLO: <5.000ms 📉 -22.2%) vs baseline: +0.9% Memory: ✅ 43.136MB (SLO: <46.000MB -6.2%) vs baseline: +5.3% ✅ bytesio_noaspectTime: ✅ 317.384µs (SLO: <420.000µs 📉 -24.4%) vs baseline: +0.9% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.2% ✅ capitalize_aspectTime: ✅ 89.014µs (SLO: <300.000µs 📉 -70.3%) vs baseline: +0.7% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ capitalize_noaspectTime: ✅ 260.141µs (SLO: <300.000µs 📉 -13.3%) vs baseline: +3.3% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ casefold_aspectTime: ✅ 88.981µs (SLO: <500.000µs 📉 -82.2%) vs baseline: +0.9% Memory: ✅ 43.175MB (SLO: <46.000MB -6.1%) vs baseline: +5.3% ✅ casefold_noaspectTime: ✅ 311.821µs (SLO: <500.000µs 📉 -37.6%) vs baseline: +1.2% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.7% ✅ decode_aspectTime: ✅ 86.706µs (SLO: <100.000µs 📉 -13.3%) vs baseline: -0.8% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.9% ✅ decode_noaspectTime: ✅ 155.353µs (SLO: <210.000µs 📉 -26.0%) vs baseline: +1.3% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ encode_aspectTime: ✅ 85.452µs (SLO: <200.000µs 📉 -57.3%) vs baseline: +0.5% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ encode_noaspectTime: ✅ 143.454µs (SLO: <200.000µs 📉 -28.3%) vs baseline: +4.3% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.3% ✅ format_aspectTime: ✅ 14.678ms (SLO: <19.200ms 📉 -23.6%) vs baseline: +0.5% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ format_map_aspectTime: ✅ 16.440ms (SLO: <21.500ms 📉 -23.5%) vs baseline: ~same Memory: ✅ 43.175MB (SLO: <46.000MB -6.1%) vs baseline: +5.0% ✅ format_map_noaspectTime: ✅ 378.699µs (SLO: <500.000µs 📉 -24.3%) vs baseline: +1.8% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ format_noaspectTime: ✅ 308.010µs (SLO: <500.000µs 📉 -38.4%) vs baseline: +1.2% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 125.973µs (SLO: <300.000µs 📉 -58.0%) vs baseline: +0.9% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ index_noaspectTime: ✅ 40.036µs (SLO: <300.000µs 📉 -86.7%) vs baseline: -0.5% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ join_aspectTime: ✅ 211.124µs (SLO: <300.000µs 📉 -29.6%) vs baseline: -3.4% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.7% ✅ join_noaspectTime: ✅ 140.709µs (SLO: <300.000µs 📉 -53.1%) vs baseline: -3.3% Memory: ✅ 43.155MB (SLO: <46.000MB -6.2%) vs baseline: +5.1% ✅ ljust_aspectTime: ✅ 603.335µs (SLO: <700.000µs 📉 -13.8%) vs baseline: 📈 +21.3% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ ljust_noaspectTime: ✅ 257.274µs (SLO: <300.000µs 📉 -14.2%) vs baseline: +0.7% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +4.9% ✅ lower_aspectTime: ✅ 297.990µs (SLO: <500.000µs 📉 -40.4%) vs baseline: +0.9% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ lower_noaspectTime: ✅ 236.426µs (SLO: <300.000µs 📉 -21.2%) vs baseline: +1.5% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ lstrip_aspectTime: ✅ 0.279ms (SLO: <3.000ms 📉 -90.7%) vs baseline: ~same Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.8% ✅ lstrip_noaspectTime: ✅ 0.178ms (SLO: <3.000ms 📉 -94.1%) vs baseline: -1.6% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ modulo_aspectTime: ✅ 14.389ms (SLO: <18.750ms 📉 -23.3%) vs baseline: +0.2% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.798ms (SLO: <19.350ms 📉 -23.5%) vs baseline: -0.1% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 14.410ms (SLO: <18.900ms 📉 -23.8%) vs baseline: -0.1% Memory: ✅ 43.195MB (SLO: <46.000MB -6.1%) vs baseline: +4.9% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.715ms (SLO: <19.150ms 📉 -23.2%) vs baseline: +0.4% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +4.7% ✅ modulo_noaspectTime: ✅ 0.358ms (SLO: <3.000ms 📉 -88.1%) vs baseline: +0.4% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ replace_aspectTime: ✅ 18.487ms (SLO: <24.000ms 📉 -23.0%) vs baseline: +0.2% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +4.7% ✅ replace_noaspectTime: ✅ 279.300µs (SLO: <300.000µs -6.9%) vs baseline: -0.8% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ repr_aspectTime: ✅ 313.843µs (SLO: <420.000µs 📉 -25.3%) vs baseline: -1.9% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ repr_noaspectTime: ✅ 47.055µs (SLO: <90.000µs 📉 -47.7%) vs baseline: +1.2% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ rstrip_aspectTime: ✅ 390.153µs (SLO: <500.000µs 📉 -22.0%) vs baseline: -1.1% Memory: ✅ 43.136MB (SLO: <46.000MB -6.2%) vs baseline: +5.3% ✅ rstrip_noaspectTime: ✅ 187.079µs (SLO: <300.000µs 📉 -37.6%) vs baseline: +2.7% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.3% ✅ slice_aspectTime: ✅ 183.546µs (SLO: <300.000µs 📉 -38.8%) vs baseline: -1.8% Memory: ✅ 43.116MB (SLO: <46.000MB -6.3%) vs baseline: +5.1% ✅ slice_noaspectTime: ✅ 53.613µs (SLO: <90.000µs 📉 -40.4%) vs baseline: -0.9% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ stringio_aspectTime: ✅ 4.544ms (SLO: <5.000ms -9.1%) vs baseline: 📈 +16.6% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ stringio_noaspectTime: ✅ 356.960µs (SLO: <500.000µs 📉 -28.6%) vs baseline: +3.5% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.5% ✅ strip_aspectTime: ✅ 280.222µs (SLO: <350.000µs 📉 -19.9%) vs baseline: +1.3% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.0% ✅ strip_noaspectTime: ✅ 178.397µs (SLO: <240.000µs 📉 -25.7%) vs baseline: -0.7% Memory: ✅ 43.313MB (SLO: <46.000MB -5.8%) vs baseline: +5.8% ✅ swapcase_aspectTime: ✅ 338.508µs (SLO: <500.000µs 📉 -32.3%) vs baseline: +1.9% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ swapcase_noaspectTime: ✅ 276.742µs (SLO: <400.000µs 📉 -30.8%) vs baseline: +2.1% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.2% ✅ title_aspectTime: ✅ 326.277µs (SLO: <500.000µs 📉 -34.7%) vs baseline: +1.5% Memory: ✅ 43.096MB (SLO: <46.000MB -6.3%) vs baseline: +5.0% ✅ title_noaspectTime: ✅ 262.014µs (SLO: <400.000µs 📉 -34.5%) vs baseline: +2.5% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ translate_aspectTime: ✅ 504.026µs (SLO: <700.000µs 📉 -28.0%) vs baseline: +3.1% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.7% ✅ translate_noaspectTime: ✅ 426.491µs (SLO: <500.000µs 📉 -14.7%) vs baseline: +1.9% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.8% ✅ upper_aspectTime: ✅ 296.420µs (SLO: <500.000µs 📉 -40.7%) vs baseline: +0.5% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ upper_noaspectTime: ✅ 236.037µs (SLO: <400.000µs 📉 -41.0%) vs baseline: +2.3% Memory: ✅ 43.234MB (SLO: <46.000MB -6.0%) vs baseline: +5.4% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 507.657µs (SLO: <700.000µs 📉 -27.5%) vs baseline: 📈 +20.2% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.7% ✅ ospathbasename_noaspectTime: ✅ 427.524µs (SLO: <700.000µs 📉 -38.9%) vs baseline: -1.0% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ ospathjoin_aspectTime: ✅ 621.856µs (SLO: <700.000µs 📉 -11.2%) vs baseline: -0.6% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.9% ✅ ospathjoin_noaspectTime: ✅ 621.603µs (SLO: <700.000µs 📉 -11.2%) vs baseline: -0.9% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ ospathnormcase_aspectTime: ✅ 352.358µs (SLO: <700.000µs 📉 -49.7%) vs baseline: -1.0% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ ospathnormcase_noaspectTime: ✅ 357.286µs (SLO: <700.000µs 📉 -49.0%) vs baseline: -3.1% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +5.0% ✅ ospathsplit_aspectTime: ✅ 488.036µs (SLO: <700.000µs 📉 -30.3%) vs baseline: ~same Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ ospathsplit_noaspectTime: ✅ 494.883µs (SLO: <700.000µs 📉 -29.3%) vs baseline: -1.0% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.1% ✅ ospathsplitdrive_aspectTime: ✅ 374.542µs (SLO: <700.000µs 📉 -46.5%) vs baseline: -0.7% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +5.2% ✅ ospathsplitdrive_noaspectTime: ✅ 73.129µs (SLO: <700.000µs 📉 -89.6%) vs baseline: +0.7% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +5.0% ✅ ospathsplitext_aspectTime: ✅ 461.841µs (SLO: <700.000µs 📉 -34.0%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.8% ✅ ospathsplitext_noaspectTime: ✅ 464.707µs (SLO: <700.000µs 📉 -33.6%) vs baseline: -1.7% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% 📈 iastaspectssplit - 12/12✅ rsplit_aspectTime: ✅ 172.172µs (SLO: <250.000µs 📉 -31.1%) vs baseline: 📈 +17.6% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.1% ✅ rsplit_noaspectTime: ✅ 169.092µs (SLO: <250.000µs 📉 -32.4%) vs baseline: 📈 +10.5% Memory: ✅ 43.037MB (SLO: <46.000MB -6.4%) vs baseline: +5.5% ✅ split_aspectTime: ✅ 151.122µs (SLO: <250.000µs 📉 -39.6%) vs baseline: +3.2% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.9% ✅ split_noaspectTime: ✅ 165.106µs (SLO: <250.000µs 📉 -34.0%) vs baseline: +9.9% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +5.8% ✅ splitlines_aspectTime: ✅ 150.454µs (SLO: <250.000µs 📉 -39.8%) vs baseline: +3.3% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +5.0% ✅ splitlines_noaspectTime: ✅ 159.563µs (SLO: <250.000µs 📉 -36.2%) vs baseline: +6.5% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.9% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.544µs (SLO: <20.000µs 📉 -82.3%) vs baseline: 📈 +23.5% Memory: ✅ 35.586MB (SLO: <38.000MB -6.4%) vs baseline: +5.0% ✅ 1-count-metrics-100-timesTime: ✅ 208.161µs (SLO: <220.000µs -5.4%) vs baseline: +4.6% Memory: ✅ 35.586MB (SLO: <38.000MB -6.4%) vs baseline: +5.1% ✅ 1-distribution-metric-1-timesTime: ✅ 3.403µs (SLO: <20.000µs 📉 -83.0%) vs baseline: +5.4% Memory: ✅ 35.606MB (SLO: <38.000MB -6.3%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 220.640µs (SLO: <230.000µs -4.1%) vs baseline: +4.2% Memory: ✅ 35.645MB (SLO: <38.000MB -6.2%) vs baseline: +5.1% ✅ 1-gauge-metric-1-timesTime: ✅ 2.196µs (SLO: <20.000µs 📉 -89.0%) vs baseline: +1.0% Memory: ✅ 35.645MB (SLO: <38.000MB -6.2%) vs baseline: +5.0% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.585µs (SLO: <150.000µs -8.9%) vs baseline: -0.8% Memory: ✅ 35.606MB (SLO: <38.000MB -6.3%) vs baseline: +5.3% ✅ 1-rate-metric-1-timesTime: ✅ 3.186µs (SLO: <20.000µs 📉 -84.1%) vs baseline: +5.4% Memory: ✅ 35.645MB (SLO: <38.000MB -6.2%) vs baseline: +5.2% ✅ 1-rate-metrics-100-timesTime: ✅ 221.048µs (SLO: <250.000µs 📉 -11.6%) vs baseline: +4.0% Memory: ✅ 35.586MB (SLO: <38.000MB -6.4%) vs baseline: +4.9% ✅ 100-count-metrics-100-timesTime: ✅ 20.406ms (SLO: <22.000ms -7.2%) vs baseline: +2.1% Memory: ✅ 35.566MB (SLO: <38.000MB -6.4%) vs baseline: +5.0% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.261ms (SLO: <2.550ms 📉 -11.3%) vs baseline: +1.2% Memory: ✅ 35.547MB (SLO: <38.000MB -6.5%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.403ms (SLO: <1.550ms -9.5%) vs baseline: -0.8% Memory: ✅ 35.566MB (SLO: <38.000MB -6.4%) vs baseline: +5.1% ✅ 100-rate-metrics-100-timesTime: ✅ 2.210ms (SLO: <2.550ms 📉 -13.3%) vs baseline: +1.3% Memory: ✅ 35.566MB (SLO: <38.000MB -6.4%) vs baseline: +5.0% ✅ flush-1-metricTime: ✅ 4.554µs (SLO: <20.000µs 📉 -77.2%) vs baseline: +3.3% Memory: ✅ 35.704MB (SLO: <38.000MB -6.0%) vs baseline: +5.3% ✅ flush-100-metricsTime: ✅ 173.858µs (SLO: <250.000µs 📉 -30.5%) vs baseline: +0.2% Memory: ✅ 35.566MB (SLO: <38.000MB -6.4%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.189ms (SLO: <2.500ms 📉 -12.4%) vs baseline: +0.2% Memory: ✅ 36.471MB (SLO: <38.750MB -5.9%) vs baseline: +5.1%
|
Codeowners resolved as |
Description
This PR implements coverage report upload for the V2 pytest plugin, enabling customers to view code coverage data in the Datadog UI alongside their test execution traces.
Core Functionality:
pytest-cov(when available) and standalonecoverage.pyDD_CIVISIBILITY_CODE_COVERAGE_REPORT_UPLOAD_ENABLEDKey Behaviors:
pytest-covis enabled (--cov): Uses the existingpytest-covcoverage instancepytest-covis NOT enabled: Automatically startscoverage.pyif coverage report upload is enabledpytest_sessionfinishafter all tests completeTesting
69 tests across 5 files (~1465 lines)
Unit Tests:
tests/ci_visibility/test_coverage_report_encoder.py(8 tests)tests/ci_visibility/test_writer_coverage_client.py(6 tests)tests/ci_visibility/test_recorder_coverage_upload.py(13 tests)tests/contrib/internal/coverage/test_utils.py(16 tests)Integration Tests:
tests/contrib/pytest/test_pytest_coverage_upload_v2.py(16 tests)tests/testing/internal/pytest/test_pytest_coverage_report_upload.py(10 tests, V3 plugin)Risks
Low Risk:
coverage_report_upload_enabled=Truein API settings or via env var