[spraza]

Description

Adds opt-in audit logging to transaction_profiling_analyzer.py (the hot shard tool). When on, every key, range and mutation it prints is appended as one JSON line to transaction_profiling_analyzer_audit.json.

• turn on with --audit-log or FDB_TRANSACTION_PROFILER_AUDIT_LOG_ENABLE
• the file goes in the FDB_NETWORK_OPTION_TRACE_ENABLE directory, next to libfdb_c's trace files (empty means the current directory). If audit is on and that's not set, the tool exits with an error instead of silently not logging
• one entry per output, and entries from the same input share a reqid
• each entry has ts, level, msg, cluster, ip_address (client IP, same one libfdb_c's Machine uses), reqid, operation, input (the sampled op's fields or the report's parameters), and the output fields (key, end_key, value, mutation_type, conflict_range, count, shard_addresses, ...). Values are always logged

Example entries from the test run (get, get_range, one commit's clear_range, set and write conflict range, then one row of each report):

{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "be86a9f31c604d39", "operation": "get", "input": {"start_timestamp": 1792104536.593214, "dc_id": "", "latency": 0.00010895729064941406, "value_size": 2}, "key": "hot_0"}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "c35d1e5e812a465c", "operation": "get_range", "input": {"start_timestamp": 1792104536.5934234, "dc_id": "", "latency": 8.225440979003906e-05, "range_size": 62}, "key": "hot_", "end_key": "hot_z"}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "20f60698673c4af6", "operation": "commit", "input": {"start_timestamp": 1792104543.52407, "dc_id": "", "latency": 0.0030279159545898438, "num_mutations": 2, "commit_bytes": 106, "commit_version": 13555923, "read_snapshot_version": 13351981, "report_conflicting_keys": false, "lock_aware": false}, "mutation_type": "CLEAR_RANGE", "key": "tmp_a", "value": "tmp_z"}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "20f60698673c4af6", "operation": "commit", "input": {"start_timestamp": 1792104543.52407, "dc_id": "", "latency": 0.0030279159545898438, "num_mutations": 2, "commit_bytes": 106, "commit_version": 13555923, "read_snapshot_version": 13351981, "report_conflicting_keys": false, "lock_aware": false}, "mutation_type": "SET_VALUE", "key": "hot_1", "value": "v39"}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "20f60698673c4af6", "operation": "commit", "input": {"start_timestamp": 1792104543.52407, "dc_id": "", "latency": 0.0030279159545898438, "num_mutations": 2, "commit_bytes": 106, "commit_version": 13555923, "read_snapshot_version": 13351981, "report_conflicting_keys": false, "lock_aware": false}, "conflict_range": "write_conflict_range", "key": "tmp_a", "end_key": "tmp_z"}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "8ee8d71b371841d6", "operation": "top_writes", "input": {"top_requests": 3, "filter_address": null, "min_timestamp": 1, "max_timestamp": 1792104556}, "key": "hot_1", "end_key": null, "count": 20, "shard_addresses": ["127.0.0.1:4810"]}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "b1d29f21f66948f2", "operation": "write_boundaries", "input": {"num_buckets": 3, "single_shard_ranges_only": false, "filter_address": null, "min_timestamp": 1, "max_timestamp": 1792104556}, "key": "hot_0", "end_key": "hot_0", "count": 20, "total_count": null, "shard_count": 1, "shard_addresses": ["127.0.0.1:4810"]}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "3c10116b0ee0496a", "operation": "top_reads", "input": {"top_requests": 3, "filter_address": null, "min_timestamp": 1, "max_timestamp": 1792104556}, "key": "hot_", "end_key": "hot_z", "count": 40, "shard_addresses": ["127.0.0.1:4810"]}
{"ts": "2026-10-15T22:49:16Z", "level": "info", "msg": "TransactionProfilingAnalyzerAudit", "cluster": "mycluster", "ip_address": "127.0.0.1", "reqid": "459f6aae6f5643b6", "operation": "read_boundaries", "input": {"num_buckets": 3, "single_shard_ranges_only": false, "filter_address": null, "min_timestamp": 1, "max_timestamp": 1792104556}, "key": "hot_", "end_key": "hot_0", "count": 40, "total_count": 40, "shard_count": 1, "shard_addresses": ["127.0.0.1:4810"]}

Testing

• Unit tests in transaction_profiling_analyzer_audit_tests.py
• End to end against a local cluster with client profiling at 100% and a small workload (sets, a clear_range, gets and get_ranges)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: bd5d093
• Duration 0:38:31
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[saintstack]

LGTM

[foundationdb-ci]

Result of foundationdb-pr-clang-73 on Linux CentOS 7

• Commit ID: bd5d093
• Duration 0:44:14
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: bd5d093
• Duration 0:45:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: bd5d093
• Duration 1:02:06
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: bd5d093
• Duration 1:05:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: bd5d093
• Duration 1:30:14
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: bd5d093
• Duration 1:51:31
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[ploxiln]

macos-m1 failure unrelated:

98% tests passed, 1 tests failed out of 56

Total Test time (real) = 420.64 sec

The following tests FAILED:
      4 - multi_process_fdbcli_tests (Timeout)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: bd5d093
• Duration 0:25:42
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)