Optimize TCP throughput with zero-copy segments and faster checksums#5
Merged
Conversation
- tests/throughput.rs: in-memory two-stack transfer benchmark (forward/ reverse, current_thread/multi_thread), mirrors the tcp_proxy wiring at MTU 1420 so it can stand in for the iperf3 script where TUN/iperf3 are unavailable. - src/checksum.rs: RFC 1071 checksum with a u64 accumulator reading 8 bytes per step, replacing the scalar 16-bit pnet helpers on the hot paths (TCP/UDP egress build, ingress IPv4 header + L4 validation, egress IPv4 header). Differential tests against pnet cover random buffers, odd lengths, skipword edge cases and carry saturation. - examples/tcp_proxy.rs: fall back to an IPv4-only TUN when the environment has IPv6 disabled. - Fix stale mtu() builder test: the alias sets both MTUs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EUsV3w9pksrMkRP8z2odvw
After awaiting the first TransportPacket, drain everything already queued with try_recv() until the caller's buffers are full, instead of returning after a single packet. A packet that does not fit the remaining buffers is stashed in a new 'pending' slot and delivered on the next call. This turns one consumer wakeup per TCP segment into one wakeup per queue drain; try_recv never waits, so no latency is added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EUsV3w9pksrMkRP8z2odvw
max_cwnd and ssthresh were derived from the peer window seen during the handshake and never updated, permanently capping cwnd at ~the initial 64KB a Linux peer advertises before receive autotuning grows it, and ending slow start almost immediately (initial ssthresh ~34KB). - ssthresh now starts effectively infinite per RFC 5681 and is only set by loss events. - max_cwnd tracks the largest window the peer has ever advertised (updated on every window update); send_window() still bounds the in-flight data by the live advertised window. - Initial window is 10 MSS (RFC 6928). - on_ack() grows slow start by the acked byte count, capped at 2*MSS (RFC 3465 appropriate byte counting), since this stack ACKs in batches rather than per segment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EUsV3w9pksrMkRP8z2odvw
- Process up to 64 queued inbound segments (was 10) per task-loop iteration before sending one cumulative ACK; the loop still stops as soon as the queue is empty, so ACK latency is unchanged without backlog. - Let poll_write hand up to 32*MSS (was 10*MSS) to the stream task per call, reducing channel round trips for large application writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EUsV3w9pksrMkRP8z2odvw
TransportPacket and the whole inbound payload chain now carry refcounted Bytes instead of BytesMut, and the send path builds the TCP wire bytes exactly once: - create_data_segment allocates header+payload at exact size, copies the payload once, checksums, and freezes; the inflight entry keeps a zero-copy slice of the same allocation. Steady state per segment is 1 alloc + 1 copy (was 2 allocs + 2 copies). - The write path no longer appends into the trailing inflight buffer, so inflight entries are 1:1 with wire segments and the tiny top-up segments that path emitted are gone; a many-small-writes echo test covers the new shape. - Retransmission clones the payload refcount instead of copying it twice, then rebuilds the wire packet with fresh ack/window/checksum. - FixedBuffer is unused now and removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EUsV3w9pksrMkRP8z2odvw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR significantly improves TCP throughput by eliminating unnecessary buffer copies in the send path and replacing the checksum implementation with a faster u64-accumulator variant. The changes maintain API compatibility while improving performance on the hot path.
Key Changes
Zero-copy inflight packets: Replace
FixedBufferwithBytesfor inflight packet payloads, allowing the wire packet and retransmission buffer to share a single allocation viacreate_data_segment(). This eliminates the copy that occurred when building packets.Faster checksum implementation: Add
src/checksum.rswith RFC 1071 checksums using u64 accumulators (8 bytes per load instead of 2), providing identical results topnet_packet::utilbut with better performance. Replace all checksum calls throughout the codebase.Improved congestion window management: Update initial congestion window to RFC 6928 (10 MSS instead of 4), simplify the congestion window API, and add
update_max_cwnd()to track window growth from ACKs.Batched egress processing: Implement packet batching in
IpStackRecv::recv_ip_packet()to drain all queued packets in a single call, with proper handling of packets that don't fit the caller's buffers via apendingfield.Type safety improvements: Convert
BytesMuttoBytesin receive paths (UnreadPacket,TcpReceiveQueue, application layer) to enforce immutability and enable zero-copy slicing.Enhanced test coverage: Add
tests/throughput.rsfor in-memory throughput benchmarks (forward/reverse, single/multi-threaded) andtests/egress_batch.rsfor batching correctness. Addmany_small_writes_echotest for the new send path.Configuration fix: Correct
IpStackConfig::mtu()builder to set both IPv4 and IPv6 MTU (previously only IPv4).Implementation Details
The
create_data_segment()method builds the wire packet once and returns both theTransportPacketand a zero-copyBytesslice of the payload for inflight tracking, eliminating the separate buffer allocation.Checksum folding properly handles carries between words per RFC 1071, maintaining byte-order independence while using native-endian loads for speed.
Batching respects MTU constraints: packets are accumulated until the next packet would exceed available buffer space, then stashed in
pendingfor the next call.The send path no longer coalesces multiple writes into a single inflight packet; each segment is independent, which simplifies the code and allows better pipelining.
https://claude.ai/code/session_01EUsV3w9pksrMkRP8z2odvw