Skip to content

Commit b7e694f

Browse files
authored
test+perf: add bandwidth / cold-start / latency-curve baseline coverage (#7)
This PR establishes a measurement baseline that the open optimization PRs (decode_bytearray #5, Nagle #6) can be validated against, plus adds regression-guard coverage for several blind spots. Designed to be merged FIRST so subsequent perf PRs see the new scenarios as already-existing baselines in CodSpeed. ## CodSpeed scenarios (perf framework) * **X7 — bytes round-trip RECV** (1k / 16k / 256k). Each call round-trips `ByteBuffer.array()` over the wire and routes through OUTPUT_CONVERTER[BYTES_TYPE] -> decode_bytearray. Without X7 the decode_bytearray path is a measurement blind spot (every prior macro returns int / list / void / callback). X7-16k joins the CodSpeed CI scenario list. * **X8 — bytes SEND** (1k / 16k / 256k). Complements X7 on the encode_bytearray + Python -> Java direction via ByteArrayOutputStream.write(byte[], int, int). X8-16k joins the CodSpeed list. Together X7-16k and X8-16k cover both halves of the byte-codec bandwidth surface that's Nagle-sensitive on Linux. ## perf_coverage_test.py (run with `pytest -v -s`) * **ColdStartLatencyTest (issue py4j#557)** — three measurements with progressively more of the cold path excluded so a regression can be localized: - `test_full_cold_start` — fresh `java` subprocess + listening- socket wait + py4j handshake + first call. Includes JVM class loading; issue py4j#557's reported window. Caps <30 s. - `test_py4j_handshake_under_500ms` — JVM is already listening; only `JavaGateway()` + first call is timed. Isolates py4j's own contribution (<500 ms). - `test_first_vs_warm_call_ratio` — first call vs warm median, catches regressions in protocol-cache fill. * **StringArgLatencyCurveTest** — per-call latency at 16 / 256 / 4K / 16K / 64K byte string args. Fills the gap between M2b (~5 B arg) and X7-class large payloads. Surfaces the Nagle knee around the BufferedWriter buffer boundary (~8 K) on Linux. * **BandwidthSummaryTest** — MB/s at 1K / 16K / 256K for both directions (recv via ByteBuffer.array, send via BAOS.write). Reports the numbers; asserts >0.1 MB/s floor to catch protocol-layer regressions without flaking on hardware variance. ## Validation handle for the open PRs * PR #5 (decode_bytearray) — `BandwidthSummaryTest` recv MB/s should jump substantially after the list-comprehension fix. X7-16k drops on the CodSpeed dashboard. * PR #6 (Nagle) — `StringArgLatencyCurveTest` flattens across the 8 K boundary; X7-16k and X8-16k drop dramatically on Linux CodSpeed runners (issue py4j#516's 4.4s / 100 calls reproducer is exactly X7-16k's pattern). ## Local sample output (macOS loopback) ``` [full-cold-start] subprocess.Popen -> first call: 0.31 s [py4j-handshake-only] gateway+first-call: 1.4 ms [first-vs-warm] first: 196us warm-median: 108.4us ratio: 1.8x [string-arg latency curve] 16 B arg: 361.2 us/call 256 B arg: 270.6 us/call 4096 B arg: 339.6 us/call 16384 B arg: 387.6 us/call 65536 B arg: 462.9 us/call [bandwidth summary] size recv MB/s send MB/s 1024B 7.1 6.8 16384B 60.3 79.3 262144B 94.1 244.7 ``` macOS loopback shows no Nagle penalty (Linux CodSpeed runners will); send is ~2.5x faster than recv at 256K because recv pays the base64-decode + list-comprehension cost in decode_bytearray that PR #5 fixes. Co-authored-by: Isaac
1 parent c4dc138 commit b7e694f

3 files changed

Lines changed: 449 additions & 1 deletion

File tree

‎py4j-python/src/py4j/tests/perf/scenarios/codspeed_macros.py‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,22 @@
3939
X2_10k,
4040
X4_Callbacks,
4141
X6_PoolSaturation,
42+
X7_16k,
43+
X8_16k,
4244
)
4345

4446

45-
_MACRO_SCENARIOS = [X1_1Thread, X2_10k, X4_Callbacks, X6_PoolSaturation]
47+
# X7-16k exercises the bytes-decoding recv path (decode_bytearray);
48+
# X8-16k exercises the bytes-encoding send path. Together they cover
49+
# both halves of the byte-codec / Nagle-sensitive bandwidth surface
50+
# in CodSpeed CI. Without either, byte-codec optimizations are
51+
# invisible to the per-PR dashboard — every prior macro returns
52+
# int / list / void / callback and the byte path was a measurement
53+
# blind spot.
54+
_MACRO_SCENARIOS = [
55+
X1_1Thread, X2_10k, X4_Callbacks, X6_PoolSaturation,
56+
X7_16k, X8_16k,
57+
]
4658

4759

4860
@pytest.fixture(scope="function")

‎py4j-python/src/py4j/tests/perf/scenarios/macro.py‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -276,11 +276,117 @@ def worker():
276276
t.join()
277277

278278

279+
# ===================================================================== X7
280+
# Bytes round-trip RECV: exercises decode_bytearray on the recv path.
281+
# Java returns a byte[]; py4j routes it through OUTPUT_CONVERTER[BYTES_TYPE]
282+
# → decode_bytearray, which base64-decodes the ASCII wire payload back
283+
# to bytes. Without this scenario, decode_bytearray improvements (e.g.
284+
# issue #570's ~7.5x speedup on 256KB payloads) are invisible to
285+
# CodSpeed — every prior macro returns int / list / void / callback.
286+
287+
class _X7BytesRoundtripBase(MacroScenario):
288+
size = 1_024 # bytes per call
289+
iterations_per_round = 100
290+
bytes_per_iteration = property(lambda self: self.size)
291+
292+
def setup(self, gateway):
293+
# Allocate a ByteBuffer once. It stays as a JavaObject because
294+
# ByteBuffer has no Python equivalent (unlike byte[]/String,
295+
# which py4j auto-converts on return). Each .array() call below
296+
# round-trips the backing byte[] over the wire — py4j returns
297+
# it via BYTES_TYPE and decodes via decode_bytearray, exactly
298+
# the recv-side path optimized in issue #570.
299+
self._buf = gateway.jvm.java.nio.ByteBuffer.allocate(self.size)
300+
301+
def measure(self, gateway):
302+
buf = self._buf
303+
for _ in range(self.iterations_per_round):
304+
buf.array()
305+
306+
307+
class X7_1k(_X7BytesRoundtripBase):
308+
id = "X7-1k"
309+
name = "bytes_roundtrip_1k"
310+
size = 1_024
311+
312+
313+
class X7_16k(_X7BytesRoundtripBase):
314+
id = "X7-16k"
315+
name = "bytes_roundtrip_16k"
316+
size = 16 * 1_024
317+
318+
319+
class X7_256k(_X7BytesRoundtripBase):
320+
id = "X7-256k"
321+
name = "bytes_roundtrip_256k"
322+
size = 256 * 1_024
323+
# Larger payloads → fewer iterations to keep the round bounded.
324+
iterations_per_round = 25
325+
326+
327+
# ===================================================================== X8
328+
# Bytes SEND: Python -> Java byte[] payload throughput. Complements X7
329+
# (bytes recv). Each iteration sends `size` bytes via
330+
# ByteArrayOutputStream.write(byte[], int, int). py4j encodes Python
331+
# bytes as BYTES_TYPE on the wire; Java decodes and writes into the
332+
# BAOS. The BAOS is reset between iterations so memory stays bounded.
333+
#
334+
# Exercises the Python-side encode_bytearray path (mirror of X7's
335+
# Java-side encode + Python-side decode_bytearray exercised in X7).
336+
# Same Nagle write-write-read sensitivity at sizes that exceed the
337+
# BufferedWriter buffer.
338+
339+
class _X8BytesSendBase(MacroScenario):
340+
size = 1_024 # bytes per call
341+
iterations_per_round = 100
342+
bytes_per_iteration = property(lambda self: self.size)
343+
344+
def setup(self, gateway):
345+
# ByteArrayOutputStream is a JavaObject (not auto-converted
346+
# because it's not a known Python type). reset() between
347+
# iterations keeps it from growing unboundedly.
348+
self._baos = gateway.jvm.java.io.ByteArrayOutputStream(self.size)
349+
# Build the payload once Python-side; setup is not timed.
350+
self._payload = b"x" * self.size
351+
352+
def measure(self, gateway):
353+
baos = self._baos
354+
payload = self._payload
355+
length = self.size
356+
for _ in range(self.iterations_per_round):
357+
baos.reset()
358+
# Force the write(byte[], int, int) overload by passing
359+
# the explicit offset/length triple — avoids any chance
360+
# of py4j picking write(int) for a 1-arg call.
361+
baos.write(payload, 0, length)
362+
363+
364+
class X8_1k(_X8BytesSendBase):
365+
id = "X8-1k"
366+
name = "bytes_send_1k"
367+
size = 1_024
368+
369+
370+
class X8_16k(_X8BytesSendBase):
371+
id = "X8-16k"
372+
name = "bytes_send_16k"
373+
size = 16 * 1_024
374+
375+
376+
class X8_256k(_X8BytesSendBase):
377+
id = "X8-256k"
378+
name = "bytes_send_256k"
379+
size = 256 * 1_024
380+
iterations_per_round = 25
381+
382+
279383
ALL_MACRO_CLASSES = [
280384
X1_1Thread, X1_4Thread, X1_16Thread,
281385
X2_1k, X2_10k, X2_100k,
282386
X3_100, X3_1k, X3_10k,
283387
X4_Callbacks,
284388
X5_ErrorPath,
285389
X6_PoolSaturation,
390+
X7_1k, X7_16k, X7_256k,
391+
X8_1k, X8_16k, X8_256k,
286392
]

0 commit comments

Comments
 (0)