Skip to content

[release-8.0] TLog: reuse disk-space samples in the low-disk admission check - #14345

Open
saintstack wants to merge 1 commit into
apple:release-8.0from
saintstack:dev/stack/backport-14216-release-8.0
Open

saintstack wants to merge 1 commit into
apple:release-8.0from
saintstack:dev/stack/backport-14216-release-8.0

Conversation

@saintstack

@saintstack saintstack commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Backport of #14216 to release-8.0. Clean cherry-pick -x of c442deb; no conflicts. Same 3 files, +20/−2.

canTLogAcceptNewData runs for every version a remote TLog pulls, and it probed both disk stores each time: a realpath() of the parent directory plus a statvfs(). On XFS that statvfs() sums per-CPU counters, so it costs more on hosts with many CPUs. The probes ran even with TLOG_MIN_AVAILABLE_SPACE_RATIO at its default of 0, where the result is not used. The check now returns before probing when the ratio disables it. Otherwise it reuses a sample for up to TLOG_DISK_SPACE_CHECK_INTERVAL seconds (default 1.0; 0 probes every time). See #14216 for the full description.

Effect on 8.0.0 in a two-region deployment

Ran the same workload twice on a production-scale two-region cluster: once on stock 8.0.0, then on 8.0.0 plus this change. The workload was a sustained closed-loop write load, then a whole-machine exclude under load, then the re-include. The binary we deployed was the 8.0.0 tag plus this same cherry-pick, line-identical to this branch.

On stock 8.0.0 the remote-region TLogs spent so long probing the disk that spilling fell behind. Unspilled data piled up in memory, and three remote TLogs ran out of memory during the exclude. The primary region was unaffected. With this change:

remote-region TLogs 8.0.0 8.0.0 + this change
disk-check share of run-loop stall samples (canTLogAcceptNewData / statvfs) 18.1% 0
CPU per TLog, median / p95, load plateau 94.2% / 95.2% 89.8% / 94.1%
min-popped-version lag, median / p90, load plateau 53.2 s / 91.5 s 18.9 s / 49.4 s
unspilled window, median / p95 51.2 s / 95.8 s 9.8 s / 36.6 s
resident memory, median, ~45 min into the load ~4.2 GB (8.55 GB peak later) 1.41 GB, flat at 1.1–1.5 GB through the exclude and include
TLog out-of-memory events 3 0
whole-machine exclude duration ≤ 8h55m 6h45m

Primary-region TLogs were unchanged (CPU ~74% in both runs).

Caveats:

  • One run per arm.
  • The patched run's load phase carried about 22% more commits than the stock run. Its exclude carried about 10–20% fewer, so the exclude duration is somewhat in this change's favour.
  • Remote TLog CPU stayed high after this change. A CPU profile attributed about 8% of it to C++ exception unwinding from coroutine cancellation. That cost is separate from this change and will be addressed in its own PR.

…ackport)

Backport of apple#14216.

canTLogAcceptNewData runs for every version a remote TLog pulls and probed
both disk stores each time: getStorageBytes() is a realpath() of the parent
directory plus a statvfs() whose XFS implementation sums per-CPU counters. On
448-CPU hosts that was about 28% of a remote TLog's CPU at peak load (statfs
13%, readlink 15%), even with TLOG_MIN_AVAILABLE_SPACE_RATIO at its default of
0, where the result is not used.

The check now returns before probing when the ratio disables it, and
otherwise reuses a sample for up to TLOG_DISK_SPACE_CHECK_INTERVAL seconds
(default 1.0, the low-disk retry loop's polling period); 0 probes every time.

(cherry picked from commit c442deb)
@saintstack
saintstack requested a review from spraza as a code owner October 22, 2026 17:52
@saintstack
saintstack requested review from akankshamahajan15 and tclinkenbeard-oai and removed request for tclinkenbeard-oai October 22, 2026 17:58
@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: 948199e
  • Duration 0:47:00
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 948199e
  • Duration 0:47:19
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: 948199e
  • Duration 0:48:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 948199e
  • Duration 0:58:44
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: 948199e
  • Duration 1:38:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 948199e
  • Duration 1:38:59
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@ploxiln ploxiln changed the title [release-8.0] TLog: reuse disk-space samples in the low-disk admission check (#14216) [release-8.0] TLog: reuse disk-space samples in the low-disk admission check Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants