Last updated: 2026-05-23.
This document records how the corpus workloads were changed, what was tried, what was rejected, and which raw artifacts back each decision. The benchmark framework must keep writing raw measurements only. Slowdown numbers in this file are offline analysis results computed from saved workload stdout/stderr, not framework-side summaries.
Slowdown convention:
slowdown = 1 - mean(ebpf_no_state_workload_metric) / mean(no_ebpf_workload_metric)
For stress-ng workloads the offline metric is the sum of stress-ng
bogo ops/s (real time) rows. For pktgen workloads it is summed pps. For
OTEL it is summed interpreter loop ops= plus stress-ng bogo ops. Higher
slowdown means the eBPF application consumes a larger fraction of the workload
budget. The target for workload selection was at least 60% slowdown when
possible, because small eBPF overhead makes optimization speedups hard to
observe at the app-workload level.
The current corpus has one selected workload per supported app. There is no runtime workload-selection matrix in the framework.
| App | Current workload | Current status |
|---|---|---|
bcc/set |
stress_ng_bcc_hook_hot |
Runs and is close but still below 60% in the latest 60s x3 stats-off pair. |
tracee/monitor |
stress_ng_tracee_syscall_hot |
Runs and reaches 60% in 60s x1 pairs; needs 60s x3 confirmation. |
tetragon/observer |
stress_ng_tetragon_policy_hot |
Runs and reaches 60% in one 60s x1 pair, but earlier same-workload pair was much lower; needs 60s x3 confirmation. |
otelcol-ebpf-profiler/profiling |
otel_mixed_workload |
Runs and reaches 60% in 60s x3 stats-off. |
katran |
xdp_pktgen |
Runs and reaches 60% in the stable 4-thread pktgen candidate. |
cilium/agent |
cilium_endpoint_pktgen |
Runs, but current stats-off slowdown is still below 60%. This remains the main workload gap. |
Current code locations:
- App mapping:
runner/libs/benchmark_catalog.py - Generic and app-specific workloads:
runner/libs/workload.py - Katran pktgen runner path:
runner/libs/app_runners/katran.py - OTEL profiler runner setup:
runner/libs/app_runners/otel_profiler.py
| Date | Commit | Change | Reason / result |
|---|---|---|---|
| 2026-04-27 | 986ec3eb |
Switched many old handwritten synthetic loops to standard tools: stress-ng, fio, and real network workloads. | Improve paper credibility and avoid custom benchmark generators where standard tools exist. |
| 2026-04-27 | 5edf29e4 |
Expanded stress-ng workload coverage. | Raised BPF program triggering coverage before app-specific tuning. |
| 2026-05-04 | 56148a3d |
Ran stress-ng stressors concurrently instead of sequentially; moved Tracee from stress_ng_os to broader stress_ng_os_io_network; moved TCP tools to real tcp_connect. |
Concurrent stressors made duration semantics sane and improved trigger density. |
| 2026-05-06 | 8d679044 |
Replaced OTEL runner-internal language idlers and noop measurement with bounded otel_mixed_workload. Initial workload used SHA-256 loops in Python/Ruby/Node/Perl/PHP plus stress-ng --cpu 1. |
Made OTEL workload a raw workload payload instead of hidden runner state. |
| 2026-05-07 | db1564b4 |
Removed per-app workload duration defaults; kept one global WORKLOAD_DURATION. |
Avoided per-app duration tuning in the framework. |
| 2026-05-14 | 7e4a3838 |
Added Katran xdp_pktgen support. |
wrk/HTTP xdp_traffic was not BPF-dominant enough for Katran. |
| 2026-05-14 | 59b8e144 |
Switched the main Katran app from xdp_traffic to xdp_pktgen. |
Made pktgen the primary Katran workload instead of an extra variant. |
| 2026-05-22 | 50bf5e00 |
Added app-specific hot stress-ng groups for BCC, Tetragon, and Tracee; switched those apps off generic stress_ng_os_io_network. |
Generic coverage workload mixed too much non-BPF user/storage work into performance measurement. |
| 2026-05-22 | 9f9d57e4 |
Narrowed BCC hot workload to syscall, cap, set, and sockfd. |
Removed low-ratio file/block/scheduler stressors after measuring that they diluted BPF overhead. |
| 2026-05-22 | 361e40c9 |
Switched Cilium from network_lossy_multi to cilium_endpoint_pktgen; added plain endpoint topology and kernel pktgen. |
network_lossy_multi was dominated by HTTP, wrk, and netem latency, not Cilium datapath cost. |
| 2026-05-23 | f04689b4 |
Changed Cilium endpoint pktgen toward bidirectional traffic and varying UDP destination ports; kept clone_skb=0. |
Tried to increase Cilium policy/CT/datapath work per packet. Reproducible slowdown remains below target. |
| 2026-05-23 | d340a708 |
Made composite workload parent records raw-only by removing derived parent duration_s, returncode, stdout, and stderr. |
Result-format cleanup, not a workload-tuning change. Leaf component raw stdout/stderr remains available for offline analysis. |
All commands were run through make corpus; manifests below are only indexes
to raw result.json and app payloads.
Goal: drive BCC probes with syscall/capability/set/socket-fd paths while avoiding slow file/block/scheduler stressors that hide eBPF overhead.
| Experiment | Workload | Artifacts | Slowdown | Notes |
|---|---|---|---|---|
workload_tune_stressapps_20260522_211432 |
stress_ng_bcc_hook_hot early hot set |
no-eBPF x86_kvm_corpus_20260522_211742_883689; eBPF no-state x86_kvm_corpus_20260522_212048_090465 |
34.28% | First 60s x1 BCC hot test. |
workload_tune_60s_stressapps_20260522_212234 |
stress_ng_bcc_hook_hot broad hot set |
no-eBPF x86_kvm_corpus_20260522_213601_930201; eBPF no-state x86_kvm_corpus_20260522_213958_976252 |
45.27% | Still included too much low-ratio work. |
workload_tune_60s_refined_20260522_215403 |
refined BCC hot set | no-eBPF x86_kvm_corpus_20260522_215705_145573; eBPF no-state x86_kvm_corpus_20260522_220057_857310 |
43.94% | Single-run result did not improve. |
workload_perf_coverage_60s_20260522_155307 |
stress_ng_bcc_hook_hot |
no-eBPF x86_kvm_corpus_20260522_225556_087664; eBPF no-state x86_kvm_corpus_20260522_230217_359197 |
44.49% | 60s x3; eBPF CV 6.87%. |
workload_perf_bcc_hot4_60s_20260522_161113 |
hot4 variant | no-eBPF x86_kvm_corpus_20260522_231402_001368; eBPF no-state x86_kvm_corpus_20260522_231959_336238 |
44.71% | 60s x3; stable but below target. |
workload_perf_bcc_syscall_hot_60s_20260522_162712 |
final narrow set: syscall,cap,set,sockfd |
no-eBPF x86_kvm_corpus_20260522_233010_504479; eBPF no-state x86_kvm_corpus_20260522_233558_916058 |
54.35% | Best current 60s x3; still below 60%. |
Coverage diagnostics:
| Artifact | Workload | Loaded | Active | run_cnt >= 100 |
|---|---|---|---|---|
x86_kvm_corpus_20260522_222339_442883 |
stress_ng_bcc_hook_hot |
25 | 22 | 20 |
x86_kvm_corpus_20260522_234642_346506 |
stress_ng_bcc_hook_hot |
25 | 17 | 16 |
Decision: keep stress_ng_bcc_hook_hot because it is the best current
BPF-heavy BCC workload, but mark BCC as not yet meeting the 60% workload target.
Goal: maximize all-events syscall/raw-tracepoint/LSM dispatch with cheap syscall generators.
| Experiment | Workload | Artifacts | Slowdown | Notes |
|---|---|---|---|---|
workload_tune_tracee_20260522_210536 |
stress_ng_tracee_syscall_hot |
no-eBPF x86_kvm_corpus_20260522_210921_806671; eBPF no-state x86_kvm_corpus_20260522_211234_202742 |
86.14% | First 10s x1 smoke pair reached target. |
workload_tune_60s_stressapps_20260522_212234 |
stress_ng_tracee_syscall_hot |
no-eBPF x86_kvm_corpus_20260522_212550_405856; eBPF no-state x86_kvm_corpus_20260522_212946_730185 |
85.88% | 60s x1 reached target. |
Coverage diagnostic:
| Artifact | Workload | Loaded | Active | run_cnt >= 100 |
|---|---|---|---|---|
x86_kvm_corpus_20260522_224455_947112 |
stress_ng_tracee_syscall_hot |
167 | 63 | 41 |
Decision: keep stress_ng_tracee_syscall_hot. Remaining work is a 60s x3
confirmation pair.
Goal: drive the bundled policy hooks with socket/datagram/eventfd/mmap paths.
| Experiment | Workload | Artifacts | Slowdown | Notes |
|---|---|---|---|---|
workload_tune_60s_stressapps_20260522_212234 |
stress_ng_tetragon_policy_hot |
no-eBPF x86_kvm_corpus_20260522_214505_834979; eBPF no-state x86_kvm_corpus_20260522_214926_162024 |
34.16% | First 60s x1 pair was below target. |
workload_tune_60s_refined_20260522_215403 |
same policy-hot workload | no-eBPF x86_kvm_corpus_20260522_220606_309552; eBPF no-state x86_kvm_corpus_20260522_221005_946299 |
64.59% | Later 60s x1 pair reached target, but the gap between the two x1 pairs means this needs x3 confirmation. |
Coverage diagnostic:
| Artifact | Workload | Loaded | Active | run_cnt >= 100 |
|---|---|---|---|---|
x86_kvm_corpus_20260522_223420_827419 |
stress_ng_tetragon_policy_hot |
173 | 25 | 16 |
Decision: keep stress_ng_tetragon_policy_hot, but do not treat the 64.59%
number as final until a 60s x3 pair reproduces it.
Goal: make profiler overhead visible in workload throughput and ensure samples hit both native and interpreter unwind paths.
| Experiment | Workload | Artifacts | Slowdown | Notes |
|---|---|---|---|---|
git before 8d679044 |
runner-internal language idlers plus noop |
no current pair used | n/a | Rejected because measurement work was hidden in app runner state rather than in raw workload payload. |
workload_perf_otel_10khz_60s_20260522_192516 |
early profiler-frequency/intloop candidate | manifest only | n/a | Used while tuning profiler sampling pressure. |
workload_perf_otel_intloop_10khz_60s_20260522_193322 |
otel_mixed_workload, low-frequency/intloop candidate |
no-eBPF x86_kvm_corpus_20260523_023621_211760; eBPF no-state x86_kvm_corpus_20260523_024233_560597 |
20.34% | Too little profiler pressure. |
workload_perf_otel_intloop_100khz_60s_20260522_195146 |
otel_mixed_workload, higher sampling/intloop candidate |
no-eBPF x86_kvm_corpus_20260523_025542_000315; eBPF no-state x86_kvm_corpus_20260523_030252_187353 |
58.80% | Close but just below target. |
| final 60s x3 pair | current otel_mixed_workload with 2 workers per language |
no-eBPF x86_kvm_corpus_20260523_081204_776028; eBPF no-state x86_kvm_corpus_20260523_082356_404370 |
61.36% | Meets target. no-eBPF CV 1.46%, eBPF CV 2.89%. |
Coverage diagnostic:
| Artifact | Workload | Loaded | Active | run_cnt >= 100 |
|---|---|---|---|---|
x86_kvm_corpus_20260523_002121_691273 |
otel_mixed_workload |
13 | 2 | 1 |
Coverage note: OTEL interpreter unwind programs are tail-call targets, so their
own run_cnt_delta stays zero even when they execute. Their cost is measured
through the attached caller native_tracer_entry.
Decision: keep otel_mixed_workload.
Goal: make the workload packet/datapath dominated. The original workload was too much userspace HTTP and netem delay.
| Experiment | Workload | Artifacts | Slowdown | Notes |
|---|---|---|---|---|
workload_stats_remaining_60s_20260522_171746 |
old network_lossy_multi |
stats artifact x86_kvm_corpus_20260523_003100_940391 |
n/a | Coverage-only run. This old workload was dominated by wrk/HTTP/netem. |
workload_perf_network_pktgen_60s_20260522_202935 |
network_pktgen_udp shared network variant |
manifest exists | n/a | Early high-pps candidate before endpoint-specific traffic. |
workload_perf_cilium_endpoint_pktgen_noclone_60s_20260522_210958 |
endpoint pktgen, clone_skb=0 |
no-eBPF x86_kvm_corpus_20260523_041339_638255; eBPF no-state x86_kvm_corpus_20260523_042014_476148 |
34.31% | Stable but below target. |
workload_perf_cilium_endpoint_pktgen_bidir_60s_20260522_212749 |
bidirectional endpoint pktgen | no-eBPF x86_kvm_corpus_20260523_043053_421385; eBPF no-state x86_kvm_corpus_20260523_043702_423340 |
46.77% | Better but still below target. |
workload_perf_cilium_endpoint_pktgen_bidir_threads_60s_20260522_214411 |
bidirectional, separate pktgen threads | no-eBPF x86_kvm_corpus_20260523_044714_736491; eBPF no-state x86_kvm_corpus_20260523_045322_149815 |
63.79% | Rejected as unstable: eBPF CV was 17.84% and later runs did not reproduce this level. |
| final stats-off reference | current cilium_endpoint_pktgen |
no-eBPF x86_kvm_corpus_20260523_102334_880171; eBPF no-state x86_kvm_corpus_20260523_104422_931622 |
32.28% | Stable 60s x3, but far below target. |
| latest stats-on no-rejit | current cilium_endpoint_pktgen |
no-eBPF x86_kvm_corpus_20260523_102334_880171; stats-on app artifact x86_kvm_corpus_20260523_205640_203233 |
42.13% baseline, 36.66% post | Stats-on includes BPF stats accounting overhead, so this is diagnostic only, not pure workload overhead. |
Coverage diagnostic:
| Artifact | Workload | Loaded | Active | run_cnt >= 100 |
|---|---|---|---|---|
x86_kvm_corpus_20260523_003100_940391 |
old network_lossy_multi |
60 | 8 | 6 |
x86_kvm_corpus_20260523_203848_846964 |
cilium_endpoint_pktgen stats-on |
26 | 8 | 2 |
x86_kvm_corpus_20260523_205640_203233 |
cilium_endpoint_pktgen stats-on |
26 | 8 | 2 |
Decision: current Cilium workload is the best integrated endpoint pktgen path, but it is not good enough for the 60% target. The issue is workload design, not loadtime apply: the no-eBPF/eBPF-no-state gap is too small under stable runs. The next tuning direction should reduce non-BPF pktgen/veth/netns overhead or make Cilium do heavier per-packet datapath work without relying on BPF stats accounting to inflate slowdown.
Goal: avoid HTTP/wrk userspace bottlenecks and make XDP balancer cost visible.
| Experiment | Workload | Artifacts | Slowdown | Notes |
|---|---|---|---|---|
| old mapping | xdp_traffic / wrk HTTP |
older corpus artifacts | about 11.7% in tuning notes | Rejected because pktgen or wrk client work dominated and XDP program was too light. |
workload_perf_katran_clone_flags_60s_20260522_220024 |
pktgen clone/flag candidate | manifest only, workload-only failed | n/a | Rejected during pktgen setup. |
workload_perf_katran_heavy_flags_60s_20260522_220936 and retry manifests |
xdp_pktgen heavy flags, single pktgen thread |
no-eBPF x86_kvm_corpus_20260523_051235_885139; eBPF no-state attempts mostly status 2 until runtime module issues were fixed |
n/a | Several eBPF no-state attempts failed before the runtime image/module setup was corrected. |
workload_perf_katran_heavy_flags_ebpf_image_60s_20260522_223651 |
same candidate after runtime-image fix | no-eBPF x86_kvm_corpus_20260523_051235_885139; eBPF no-state x86_kvm_corpus_20260523_054010_203688 |
-2.53% | Rejected: eBPF baseline CV was 16.44%, noisy and not BPF-dominant. |
workload_perf_katran_heavy_flags_4pktgen_60s_20260522_224855 plus later successful eBPF run |
4 pktgen threads | no-eBPF x86_kvm_corpus_20260523_055252_246684; eBPF no-state x86_kvm_corpus_20260523_061505_495947 |
63.74% | Accepted. no-eBPF CV 2.00%, eBPF CV 0.41%. |
Coverage diagnostic:
| Artifact | Workload | Loaded | Active | run_cnt >= 100 |
|---|---|---|---|---|
x86_kvm_corpus_20260523_004124_511756 |
xdp_pktgen |
1 | 1 | 1 |
Decision: keep xdp_pktgen with the 4-thread pktgen path.
This index lists every corpus/results/workload_* tuning manifest found in the
tree at the time this note was written. Some entries were interrupted, failed,
or only contain one side of a no-eBPF/eBPF pair; those are intentionally kept
here so the failed tuning path is visible. Setup status is the manifest's
recorded command status; rows marked app error exited the outer command but
the per-app JSON recorded an application failure.
| Manifest | App(s) | Setup status | Result artifacts |
|---|---|---|---|
workload_perf_bcc_hot4_60s_20260522_161113 |
bcc/set |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260522_231402_001368, x86_kvm_corpus_20260522_231959_336238 |
workload_perf_bcc_syscall_hot_60s_20260522_162712 |
bcc/set |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260522_233010_504479, x86_kvm_corpus_20260522_233558_916058 |
workload_perf_cilium_endpoint_pktgen_60s_20260522_210155 |
cilium/agent |
workload_only:0 app error |
x86_kvm_corpus_20260523_040513_850094 |
workload_perf_cilium_endpoint_pktgen_bidir_60s_20260522_212749 |
cilium/agent |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260523_043053_421385, x86_kvm_corpus_20260523_043702_423340 |
workload_perf_cilium_endpoint_pktgen_bidir_threads_60s_20260522_214411 |
cilium/agent |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260523_044714_736491, x86_kvm_corpus_20260523_045322_149815 |
workload_perf_cilium_endpoint_pktgen_noclone_60s_20260522_210851 |
cilium/agent |
workload_only:2 app error |
x86_kvm_corpus_20260523_040513_850094 |
workload_perf_cilium_endpoint_pktgen_noclone_60s_20260522_210958 |
cilium/agent |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260523_041339_638255, x86_kvm_corpus_20260523_042014_476148 |
workload_perf_coverage_60s_20260522_155307 |
bcc/set |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260522_225556_087664, x86_kvm_corpus_20260522_230217_359197 |
workload_perf_katran_clone_flags_60s_20260522_220024 |
katran |
workload_only:0 app error |
x86_kvm_corpus_20260523_050519_591710 |
workload_perf_katran_heavy_flags_4pktgen_60s_20260522_224855 |
katran |
workload_only:0, ebpf_no_state:0 app error |
x86_kvm_corpus_20260523_055252_246684, x86_kvm_corpus_20260523_055952_009234 |
workload_perf_katran_heavy_flags_60s_20260522_220936 |
katran |
workload_only:0, ebpf_no_state:2 |
x86_kvm_corpus_20260523_051235_885139, x86_kvm_corpus_20260523_051235_885139 |
workload_perf_katran_heavy_flags_ebpf_after_btf2_60s_20260522_223108 |
katran |
ebpf_no_state:0 app error |
x86_kvm_corpus_20260523_053517_445489 |
workload_perf_katran_heavy_flags_ebpf_after_btf_60s_20260522_223013 |
katran |
ebpf_no_state:2 |
x86_kvm_corpus_20260523_051235_885139 |
workload_perf_katran_heavy_flags_ebpf_clean_60s_20260522_221954 |
katran |
ebpf_no_state:2 |
x86_kvm_corpus_20260523_051235_885139 |
workload_perf_katran_heavy_flags_ebpf_image_60s_20260522_223651 |
katran |
ebpf_no_state:0 |
x86_kvm_corpus_20260523_054010_203688 |
workload_perf_katran_heavy_flags_ebpf_jobs1_60s_20260522_222837 |
katran |
ebpf_no_state:2 |
x86_kvm_corpus_20260523_051235_885139 |
workload_perf_katran_heavy_flags_ebpf_rerun_60s_20260522_222444 |
katran |
ebpf_no_state:2 |
x86_kvm_corpus_20260523_051235_885139 |
workload_perf_katran_heavy_flags_ebpf_retry_60s_20260522_221805 |
katran |
ebpf_no_state:2 |
x86_kvm_corpus_20260523_051235_885139 |
workload_perf_network_pktgen_60s_20260522_202935 |
cilium/agent,katran |
workload_only:0, ebpf_no_state:2 |
x86_kvm_corpus_20260523_033326_529957, x86_kvm_corpus_20260523_033326_529957 |
workload_perf_network_pktgen_ebpf_retry_60s_20260522_204045 |
cilium/agent,katran |
ebpf_no_state:0 |
x86_kvm_corpus_20260523_034445_173649 |
workload_perf_otel_10khz_60s_20260522_192516 |
otelcol-ebpf-profiler/profiling |
interrupted, empty manifest rows | log only |
workload_perf_otel_intloop_100khz_60s_20260522_195146 |
otelcol-ebpf-profiler/profiling |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260523_025542_000315, x86_kvm_corpus_20260523_030252_187353 |
workload_perf_otel_intloop_10khz_60s_20260522_193322 |
otelcol-ebpf-profiler/profiling |
workload_only:0, ebpf_no_state:0 |
x86_kvm_corpus_20260523_023621_211760, x86_kvm_corpus_20260523_024233_560597 |
workload_stats_bcc_syscall_hot_60s_20260522_164348 |
bcc/set |
stats_on_norejit:0 |
x86_kvm_corpus_20260522_234642_346506 |
workload_stats_coverage_60s_20260522_152042 |
bcc/set,tetragon/observer,tracee/monitor |
all stats_on_norejit:0 |
x86_kvm_corpus_20260522_222339_442883, x86_kvm_corpus_20260522_223420_827419, x86_kvm_corpus_20260522_224455_947112 |
workload_stats_remaining_60s_20260522_171746 |
otelcol-ebpf-profiler/profiling,cilium/agent,katran |
all stats_on_norejit:0 |
x86_kvm_corpus_20260523_002121_691273, x86_kvm_corpus_20260523_003100_940391, x86_kvm_corpus_20260523_004124_511756 |
workload_tune_60s_refined_20260522_215403 |
bcc/set,tetragon/observer |
all ok | x86_kvm_corpus_20260522_215705_145573, x86_kvm_corpus_20260522_220057_857310, x86_kvm_corpus_20260522_220606_309552, x86_kvm_corpus_20260522_221005_946299 |
workload_tune_60s_stressapps_20260522_212234 |
tracee/monitor,bcc/set,tetragon/observer |
all ok | x86_kvm_corpus_20260522_212550_405856, x86_kvm_corpus_20260522_212946_730185, x86_kvm_corpus_20260522_213601_930201, x86_kvm_corpus_20260522_213958_976252, x86_kvm_corpus_20260522_214505_834979, x86_kvm_corpus_20260522_214926_162024 |
workload_tune_stressapps_20260522_211432 |
bcc/set |
all ok | x86_kvm_corpus_20260522_211742_883689, x86_kvm_corpus_20260522_212048_090465 |
workload_tune_tracee_20260522_210536 |
tracee/monitor |
all ok | x86_kvm_corpus_20260522_210921_806671, x86_kvm_corpus_20260522_211234_202742 |
- Cilium still needs a better workload. Current stable stats-off slowdown is 32.28%, and stats-on diagnostic slowdown is not a valid substitute.
- BCC needs either another workload refinement or acceptance as a below-target workload. Current best 60s x3 stats-off slowdown is 54.35%.
- Tracee needs a 60s x3 no-eBPF vs eBPF-no-state confirmation pair.
- Tetragon needs a 60s x3 confirmation pair because existing 60s x1 pairs disagree widely, 34.16% vs 64.59%.
rejit/docs/evaluation.mdis stale for workload names; it still describes older workloads such asstress_ng_os_io_network,network_lossy_multi, andxdp_trafficfor apps whose current workload has changed. This file should be used as the current workload-tuning record until that evaluation document is updated.