The numbers uCache records about your jobs: what each one means, and the format of the JSON files it writes.
To simply see how your cache is doing, run ucache stats — it prints a
human-readable summary of these same numbers. This file is for reading the raw
JSON, or building tooling on it. The user guide's monitoring section explains
how to read the printed summary.
In this document "with failover on" means: for a file failover applies to --
with failover = on, the default, a file opened at a server that reports a
redirector's role, or whose URL has a host pattern in tried= (USER_GUIDE.md,
"Failover between copies").
| question | look at |
|---|---|
| Is the cache being used at all? | opens, files_opened, hit_bytes |
| Did a warm pass avoid the network? | origin_bytes — 0 means everything came from local disk |
| Is anything wrong? | crc_failures, failopen_events, validations_failed — all three should be 0 |
| Is the cache big enough? | admissions_bypassed, evicted_bytes, evicted_entries |
| Where did the bytes come from? | ram_hit_bytes, hit_bytes, replica_bytes_served, miss_bytes, relay_bytes |
| Is the job re-reading the same data? | opens / files_opened, first_touch_bytes against hit_bytes |
| What is the cache disk being asked to do? | hit_disk_reads, hit_disk_bytes, replica_reads, replica_read_bytes, hist_*_read_bytes |
| Is the cache disk keeping up? | buffer_stalls, buffer_stall_us, hist_hit_read_us, hist_flush_write_us, slow_cache_episodes |
| Did the replica tier get used? | replica_bytes_served, cold_replica_files |
| Were replicas created as files were read? | cold_replica_files, cold_replica_in_bytes, cold_replica_baskets_kept, cold_replica_declined, cold_replica_skipped, cold_replica_waits, cold_replica_completed |
| Did the cache take along the rest of the clusters a job touched? | cold_replica_completed, fill_completed and their _bytes |
| Were later passes shown the converted baskets at their real size? | slot_maps_made, slot_map_opens, slot_map_full |
These records are also what ucache summary and ucache history read: the
file name carries the store's start time and the last line carries its end, so
one process's work is bracketed without any periodic sampling, and the
.files.jsonl companion says which entries the bytes belonged to. Consumers
should take the file name's <start_ts> as the start of the run.
One JSON line is appended to $UCACHE_DIR/stats/<host>-<pid>-<start_ts>-<seq>.jsonl
at every CacheStore::dumpStats(): explicit calls, store destruction, the
plugin's atexit, and every meta_flush_seconds (default 30 s) while the
process runs. That periodic line is what keeps a run visible when its
process exits without running destructors — Python's multiprocessing and
concurrent.futures workers _exit(), which skips atexit too — so such a
run is at most one period short instead of absent. Counters are cumulative,
so consumers take the last complete line. External tooling consumes these —
the schema is a correctness surface; changes require updating this file and
the consumers together.
{"ts": 1783300000, "pid": 12345, "ts_ms": 1783300000123, "start_ms": 1783299400456,
"opens": 0, "validations_failed": 0,
"hit_bytes": 0, "miss_bytes": 0, "origin_bytes": 0, "served_bytes": 0,
"origin_reads": 0, "fetches_joined": 0, "origin_readvs": 0, "page_writes": 0,
"prefetch_issued_bytes": 0, "prefetch_served_bytes": 0, "prefetch_refetched_bytes": 0,
"prefetch_dropped_unread": 0, "prefetch_late_bytes": 0, "prefetch_parses": 0,
"prefetch_bridge_bytes": 0,
"prefetch_disabled": 0, "prefetch_fetch_errors": 0,
"crc_failures": 0, "meta_corrupt": 0,
"evicted_entries": 0, "evicted_bytes": 0,
"failopen_events": 0, "admissions_bypassed": 0,
"open_retries": 0, "open_retries_exhausted": 0,
"disabled_handles": 0, "copier_handles": 0, "copies_refused": 0,
"direct_read_files": 0, "direct_read_bytes": 0,
"replica_opens": 0, "replica_published": 0, "replica_invalid": 0,
"replica_crc_failures": 0, "replica_punched_bytes": 0, "replica_orphans_swept": 0,
"files_opened": 0, "ram_hit_bytes": 0, "first_touch_bytes": 0,
"hit_disk_reads": 0, "hit_disk_bytes": 0, "hit_disk_seq": 0,
"replica_bytes_served": 0,
"cold_replica_files": 0, "cold_replica_in_bytes": 0, "cold_replica_out_bytes": 0,
"cold_replica_baskets": 0, "cold_replica_baskets_kept": 0, "cold_replica_convert_us": 0,
"cold_replica_declined": 0, "cold_replica_skipped": 0, "cold_replica_waits": 0,
"cold_replica_wait_us": 0, "cold_replica_completed": 0, "cold_replica_completed_bytes": 0,
"fill_completed": 0, "fill_completed_bytes": 0, "slow_cache_episodes": 0, "slow_cache_us": 0,
"fill_not_kept_bytes": 0,
"cold_origin_failures": 0,
"slot_maps_made": 0, "slot_map_opens": 0, "slot_map_full": 0,
"slot_crc_failures": 0, "slot_read_fallbacks": 0, "slot_pages_decoded": 0,
"kept_reads": 0, "kept_read_bytes": 0,
"pool_tasks": 0, "pool_busy_us": 0, "pool_queue_high_water": 0, "pool_thread_failures": 0,
"reader_wait_byte_us": 0, "reader_wait_replica_us": 0, "reader_wait_slots_us": 0,
"reader_wait_origin_us": 0, "reader_threads": 0,
"replica_reads": 0, "replica_read_bytes": 0,
"relay_bytes": 0,
"readv_chunks": 0, "readv_calls": 0, "readv_mixed": 0,
"flush_runs": 0, "flush_run_bytes": 0,
"buffer_stalls": 0, "buffer_stall_us": 0,
"cpu_us": 0, "instructions": 0, "cycles": 0, "counter_source": "perf", "pmu_duty": 1.0000,
"hist_hit_read_us": [..], "hist_miss_read_us": [..], "hist_origin_rt_us": [..],
"hist_relay_rt_us": [..], "hist_open_us": [..], "hist_replica_read_us": [..],
"hist_flush_write_us": [..], "hist_meta_flush_us": [..],
"hist_req_read_bytes": [..], "hist_hit_read_bytes": [..],
"hist_replica_read_bytes": [..],
"hist_slot_read_us": [..], "hist_cold_request_us": [..], "hist_pool_queue_us": [..],
"origin_open_attempts": 0, "origin_open_failures": 0,
"origin_failures_client": 0, "origin_failures_site": 0, "origin_failures_notfound": 0,
"origin_failures_data": 0, "origin_failures_expired": 0, "origin_failures_other": 0,
"records_dropped": 0, "hist_origin_open_us": [..],
"open_failovers": 0, "opens_recovered": 0, "opens_failed_everywhere": 0,
"open_deadline_expiries": 0, "failover_locates": 0, "offline_copies_skipped": 0,
"open_check_bytes_kept": 0, "bad_entries_dropped": 0,
"site_name_queries": 0, "site_names_missing": 0, "requests_parked_for_parse": 0,
"site_faults_marked": 0, "site_marks_cleared": 0, "copy_faults_marked": 0,
"marked_copies_last": 0, "hosts_avoided": 0, "role_queries": 0, "role_redirectors": 0, "role_unanswered": 0,
"size_disagreements": 0, "name_size_mismatches": 0,
"origin_checks_passed": 0, "origin_checks_failed": 0, "origin_checks_none": 0,
"origin_checks_unjudged": 0, "origin_checks_skipped": 0, "origin_checks_not_examined": 0,
"entries": [{"key": "root://host:1094/path", "file_size": 0,
"cached_bytes": 0, "page_size": 4096}],
"run_settings": "v1 check_origin_integrity=off xrdcl.ConnectionWindow=120/default ... failover=on",
"redirector_hosts": "cms-redirector.example.org:1094"}Fields are added at the end of a line, never in the middle, and under names no other field uses: the readers find a field by its name, so an older reader skips a field it does not know.
What each counter means, and the reasoning behind the ones that are easy to
misread. Three words recur. The byte tier is the page-granular copy of the
original file — the ordinary cache. The replica tier is a file's slot
store: its baskets or pages recompressed (as a job reads them with
recompress = on, or by ucache recompress), which serves the same data with
less decompression work; a cache that never recompressed has none. A sidecar is
the small file beside each cached entry holding its page bitmap and checksums.
How it works covers the tiers, and
FORMAT.md the files.
Every read is served from somewhere, and the counters name the source. Bytes
served locally are hit_bytes (byte tier) and replica_bytes_served (replica
tier); bytes that had to be fetched are miss_bytes; bytes the cache never
touched at all are relay_bytes. On a warm pass the fetched figure should be
zero.
origin_bytes— bytes requested from the origin after page rounding, so it is the numerator of read amplification: divide by the bytes the job asked for to see how much extra the cache pulled to satisfy them.- Which layer maintains which counter, when tracing an unexpected value back to
its source:
miss_bytes/served_bytes/origin_reads/origin_readvsare incremented by the plugin layer; the core ownsopens,hit_bytes,page_writes,crc_failures,meta_corrupt,validations_failed,evicted_*,failopen_events.
These are the counters to read first when something looks off. crc_failures,
failopen_events and validations_failed should all be zero. By contrast
admissions_bypassed is not a fault at all — it records a deliberate decision
not to cache, and says the cache is under pressure rather than broken.
admissions_bypassed— files NOT cached because every resident entry was still inside the eviction protection window (in_use_seconds), so the cache had stopped growing rather than evict data a running job still needs. These reads SUCCEED, uncached. Deliberately separate fromfailopen_events: that counter means something went wrong, while this is a capacity decision. A non-zero value here means the cache is too small for the working set — seeucache status, which names the remedy.validations_failed— cached entries discarded because they no longer matched the file at the origin: a different size, mtime or checksum, or a changed page size. The entry is refetched from scratch, so a read still succeeds; a steadily rising count means the data is being rewritten underneath the cache.name_size_mismatches— in a cache that has logical names: times a job opened a URL at its origin and found its copy another size than the entry the URL reached by its name (the entry of its logical name, or another URL's). The URL is then cached apart, under its own key; that entry is left as it is. A URL whose own entry, cached before, shows the other size is cached apart without an open, and is not counted.size_disagreements— times a source reported a different file size than the cached entry it was about to extend (or, where the modification time is checked — a local file, orvalidate = size+mtime, for a file not keyed by a logical name — a different time while the process holds the entry). Bytes from that source are never mixed into the entry: a handle that already holds the cached size fails its read with a "data is corrupted" error naming both sizes and the server (and the next open checks the origin again); a handle opening while the process holds the entry reads that file uncached. Non-zero means two copies of one file disagree — usually one damaged or truncated copy among the copies a redirector chooses from — or a file rewritten while it was being read.crc_failures— pages that failed CRC on read and were quarantined (refetched later).meta_corrupt— sidecars that failed to load (torn or damaged) and were rebuilt.failopen_events— cache-side faults that degraded to pass-through (failed page writes, failed sidecar persists).disabled_handles— handles that trippedUCACHE_MAX_ERRORS(plugin layer).copier_handles(plugin layer) — file handles opened for a copy: by a copy tool (xrdcp,xrdfs,xrdadler32,edmCopyUtil), byhaddor one of ROOT's command-line tools, from inside XRootD's copy engine, from ROOT'sTFile::CporTFileMerger, or from gfal2's xrootd plugin. They are served as pure pass-through, so a copy is the origin's bytes: their reads are inrelay_bytes, and they add nothing to the cache. Also counted: a handle whose first request read the whole file at once, for a file with a slot store. Zero withcopy_detect = off.copies_refused(plugin layer) — requests refused because they would have completed a copy, in pieces up to the origin's size, of a file shown in a replica's layout (the handle read all of [0, origin size) without reading any of the replica's own part first): the copy fails with an error namingUCACHE_DISABLE=1instead of coming out corrupt. Not a cache error: never infailopen_events.direct_read_files/direct_read_bytes(plugin layer) — ROOT files this process decided to read straight from the origin because its reads covered branches holding more thanmax_read_fractionof the file's data, and the bytes it fetched for their data and did not keep (also inrelay_bytes). A file is counted once per process. Zero withmax_read_fraction = 100, and on a build without the codec libraries whatever the value; withmax_read_fraction = 0, every ROOT file with a TTree or RNTuple the process read from the origin, of any size, including those read directly whole because their records cannot be mapped.open_retries/open_retries_exhausted(plugin layer) — transient inner-open failures re-attempted (UCACHE_OPEN_RETRIES), and opens that ultimately gave up after retrying. Uncounted when caching is off (retry still works, just unrecorded — no store to hold the counters).
What the origin answered, as raw counts; each is also recorded line by line
in the per-file record file (below), with the server and the error. Nothing is
counted when there is no cache directory. Opens are counted for a file the
cache serves only; a pass-through handle's open (UCACHE_DISABLE=1, a write,
a copy) is not. Failed reads are counted for every read handle, pass-through
ones included, so a run with caching disabled shows its origin failures too.
origin_open_attempts/origin_open_failures— origin open attempts the plugin watched, and how many failed: the reader's own open of a file the cache serves (each retry withopen_retriescounts again), and the cache's own opens (a trusted handle's first miss, a handle whose trusted entry was gone, a forked child opening a handle again). An open the client refuses at once, before it reaches any server, counts as a failed attempt too, and so does each later open of that file the client refuses again. An open issued again on a file whose earlier open was sent (in progress, open, or failed in its answer), which the client answers from that file's own state, is not an attempt.hist_origin_open_us— each attempt's time from issue to answer (log2 µs).origin_failures_client,_site,_notfound,_data,_expired,_other— failed origin requests, opens and reads alike, by a first reading of the error alone; what a failure means often depends on what else happened at the time, so the records keep the raw code and error number too.client: this machine could not reach or talk to the server (socket, connection and TLS errors, failed authentication, a refused authorization).site: the server or its storage failed the request, or the redirectors sent it on past the redirect limit.notfound: no server holds the file.data: bytes refused (a basket or page that failed its checksum, a basket of a byte-cache fetch whose key is not where the tree's record puts it, or, with failover on, an empty copy of a file named*.root, or one whose first bytes show it is not a whole ROOT file, or a read a copy answered short inside the file's confirmed size).expired: a request that outlived its timeout, whose timeout not said; with failover on, among them an open that passed the open deadline (USER_GUIDE.md), itstimeout_sin its open line (FORMAT.md).other: the rest, among them a server's report that it could not reach a server behind it, which can be that site's network as well as this machine's. A read the cache fails open on (and relays) is counted when its first attempt fails, and again, as arelayfailure, if the relayed request fails too.records_dropped— record lines (an origin open, an origin failure) dropped because the 4096 lines waiting for the next checkpoint were already queued, or, for a failure line, because the process already keeps the account of 16384 distinct (file, server, class) failures it has recorded (each records at most 16 lines). The counters above still count them.open_failovers,opens_recovered,opens_failed_everywhere— with failover on (USER_GUIDE.md): attempts made at another site after an open failed at one, opens that succeeded after at least one such attempt, and opens that ended with every site they tried failing (stopped by the bound of sites or time, or by the redirector sending the open back to a server or site that failed already). An open whose first failure is final (no copy anywhere, a redirector that refuses the client itself) is in none of them. The open that moves a file's reads to another site after a read failed (USER_GUIDE.md) counts in them as an open does: each of its attempts is one inopen_failovers, its success one inopens_recovered, and its failure at every site one inopens_failed_everywhere.open_deadline_expiries— with failover on, open attempts that expired at their timeout: the open deadline, the job's own shorter open timeout, or what was left of an open's time budget (also counted inorigin_failures_expired).failover_locates,offline_copies_skipped— with failover on, requests to a regional redirector for the places a file is (made when the redirector sends an open back to a server or site that failed, or finds no copy after servers were left out; one request answers every open waiting for the same answer), and listed copies not opened because the server says they are on tape.open_check_bytes_kept— with failover on, for files the cache caches: bytes the checks made inside a job's open read of the copy the open took (its first page, and the pages of its list of top-level objects; whole pages, at most 1 MiB of a copy), kept in the file's entry so that the job's own reads of them right after the open are served from the cache. They are also counted inorigin_bytesandorigin_reads(one read each), which is where a read the job makes of them would otherwise have been counted. Nothing a refused copy's checks read is kept.bad_entries_dropped— cache entries dropped because what they hold came from a bad copy: with failover on, an entry made before the checks at open existed whose stored header states an end past its stored size (filled from a cut copy), found at an open the entry would have answered; and, withcheck_origin_integrity = on(which failover forces on), an entry holding bytes this process fetched from a copy that a check then found bad, at a read or at an open that refused the copy. The next open of the file fills a new entry (TROUBLESHOOTING.md).requests_parked_for_parse— with failover on, requests made where no move may be waited for (a client callback thread: a read issued from inside a completion handler) before their file was set up or had its map, made again on a thread of uCache's own (TROUBLESHOOTING.md).site_name_queries,site_names_missing— with failover on: servers asked for the name of their site (theirall.sitename, as the server's configuration valuesitename; USER_GUIDE.md: a server that answered an open, and a listed copy when its turn comes), once per server and process (again 5 minutes after a query that got no answer), and the asks that gave no name (none configured, an error, or no answer within 5 s): such a server's site is its domain.role_queries,role_redirectors,role_unanswered— withfailover = on(the default; USER_GUIDE.md, "Which files failover applies to"): servers asked what they are (their login flags, and for one that says manager there its configured role), at the first open of a file there, once per server and process (an open the cache answers from a trusted entry asks nothing); those found to be a redirector, which gives their files failover; and those that gave no answer (an error, or none within 5 s), whose files are read as with failover off. A server is asked again 10 minutes after its last outcome, at an open that needs it (each counted); a renewal that gets no answer is counted inrole_unansweredand leaves the earlier answer standing. The servers asked and counted in neither are not redirectors (a data server, a proxy, an EOS instance).redirector_hosts(a string, on the counter line afterrun_settings, only when there is one) — withfailover = on, the servers (host:port, lower-cased) found to be redirectors in this process (a forked child keeps its parent's), one space between them.site_faults_marked,site_marks_cleared,copy_faults_marked,marked_copies_last— with failover on (USER_GUIDE.md, "Remembering sources that failed"): servers marked in<dir>/sources(a stall while another site answered, or a fault of the server's own; each mark written, a repeat included), marks cleared by a success, copies proven missing (noted in the file's in-use record), and copies a regional redirector listed that were put after the others for a mark.hosts_avoided— with failover on and host patterns in the URL'stried=(USER_GUIDE.md, "Keeping reads away from some hosts"): copies at a host a pattern matches, each refused once per open: an open that landed there and was refused before anything was read, or a copy a regional redirector listed that was not tried. Each has ahost_avoidedline in the per-file record file (FORMAT.md). It is no failure count: none of these is inorigin_open_failuresororigin_failures_*.
run_settings, on the counter line, is the settings that decide how the
origin is reached, as the plugin found them at load, one string:
v1 check_origin_integrity=on|off, then xrdcl.<name>=<value>/<source> for
the XRootD client's ConnectionWindow, ConnectionRetry, StreamErrorWindow,
RequestTimeout, StreamTimeout and TimeoutResolution, then
failover=on|off. The source is env when the XRD_<NAME>
variable is set, default when the value is the client's built-in one, and
client when it differs and no variable is set (a client.conf, or the
application); uCache sets none of them. With logical_name_prefixes set
(and valid), the string ends with logical_names=on/<hash>, whatever
failover says: files whose path holds one of the prefixes are keyed by
their logical name; <hash> is the first 8 hex digits of the SHA-256 of
logical_name_prefixes as ucache settings shows it, one space between prefixes,
so two runs that keyed files two ways can be told apart. With it off there is no
such token. An application that changes a
value after the plugin has loaded is not seen. A token missing from the string
means the build that wrote it did not have that setting; tokens may be added
at the end.
replica_opens, replica_published, replica_invalid,
replica_crc_failures and replica_punched_bytes counted the compact replicas
earlier releases built beside the byte cache. This release neither builds nor
serves one, so they stay 0; they are kept so that every record has the same
fields. replica_orphans_swept still counts replica files removed by the
eviction pass: a slot store whose entry is gone, and every replica an earlier
release left (see FORMAT.md).
With recompress = on, a file that has no replica gets a slot store on its
first open, and every basket a job reads that the store does not hold yet is
converted as it arrives -- on the first pass, and later for branches read for
the first time. All zero otherwise, except fill_completed, fill_completed_bytes (byte-cache
fills) and the slow-disk counters (both kinds of fill). ucache stats prints the conversions on
one converted on read line, cold_replica_skipped on a not converted line, what
complete_chunks took along on completed, converted and completed, cached lines, the waits on
a waited for writes line and a cache disk too slow on a cache disk slow line. For an RNTuple
file, read "page" wherever these say "basket".
cold_replica_files— slot stores created.cold_replica_in_bytes/cold_replica_out_bytes— original basket bytes converted, and the converted records they became (ZSTD-1, or uncompressed where ZSTD could not shrink a basket).cold_replica_baskets— baskets converted.cold_replica_baskets_kept— baskets kept exactly as stored because their converted form did not fit their slot, or they could not be decoded (withrecompress = onorcheck_origin_integrityon, a basket that fails its own checksum is not kept: it is counted underorigin_checks_failed). Their original bytes are the byte cache's share of such a file. (Branches whose codec is not inrecompress_codecsare not counted: they are never converted.)origin_checks_passed,origin_checks_failed,origin_checks_none— withrecompress = on, whatevercheck_origin_integritysays, or withcheck_origin_integrityon: baskets and RNTuple pages a first pass converts whose own checksum held (ROOT's LZMA CRC32, zlib adler32, LZ4 XXH64, a ZSTD content checksum, the RNTuple page checksum); that failed it — never kept or served, the read fails with "data is corrupted" and the log names the file, the position, the codec and the server; or that carry no checksum this build can check (ZSTD as ROOT writes it, baskets stored uncompressed, a block this build cannot decode), which nothing can check.origin_checks_not_examinedcounts, withrecompress = onandcheck_origin_integrityoff, the baskets of a converted branch that a first pass did not decode, so that their own checksum, which they may carry, was not checked: a basket whose own codec is not inrecompress_codecsthough its branch's is (a file merged from inputs written with different codecs keeps each input's baskets as they were), or whose block tag the conversion does not know; one whose blocks name more than one codec; and one that cannot be read as a basket record, or does not fit its slot. Such a basket is kept and served as stored, as without the cache (shown only when not 0). Withcheck_origin_integrityon every such basket is checked by the byte path's rule below and counted there, so the counter stays 0. A basket left unconverted whose data carries no check at all (stored uncompressed, an object of no length, its first block ZSTD without a content checksum or ROOT's old algorithm) counts underorigin_checks_none, as on the byte path below. A branch whose own codec is not inrecompress_codecsis not converted at all: in a file that has a replica its baskets are fetched as original bytes, checked (and counted) only withcheck_origin_integrityon, below. Withcheck_origin_integrityoff, only what a first pass converts is checked by checksum; other bytes are not counted here except as follows. Withcheck_origin_integrityon, these counters count every basket wholly in a fetch from the origin, by one rule, whateverrecompresssays and whatever the codec: zlib and LZMA blocks, and ZSTD blocks written with a content checksum, are decompressed and checked; an LZ4 block is checked by its XXH64 over the compressed bytes, never decompressed; ZSTD written without a checksum (as ROOT writes it), an xz stream that declares no check, ROOT's old algorithm and baskets stored uncompressed are not decompressed at all (nothing could be checked), and count underorigin_checks_none. In a file that has a replica (recompress = on) that covers the original bytes a read fetches outside the converted baskets (a branch of a codec not converted among them) and a converted branch's basket the conversion leaves as stored; a basket the conversion decodes has its data checked once, by its conversion, and its key against the tree's record (counted underorigin_checks_failedwhen it fails). A basket a fetch does not hold whole (cut by its edge) is not judged by that fetch; it is judged only if another fetch holds it whole, and is otherwise not checked (USER_GUIDE, "Not checked"). Withcheck_origin_integrityon, these counters also count the byte path: the baskets of a fetch from the origin (a byte-cache miss, a read the cache relays to the origin for a handle that holds the file's map, read-ahead, the bytes the checks at open keep) that lie wholly in the fetched bytes, whose key passes and whose length the file map gives:origin_checks_passedthose whose own checksum held (ZLIB and LZMA, decoded; LZ4, its XXH64),origin_checks_nonethose that carry none (stored uncompressed, ZSTD without a content checksum, ROOT's old algorithm).origin_checks_unjudgedcounts the parts of such a fetch the check could not judge (a decoding job that ran out of memory): their bytes are served, as without the check, but nothing of that fetch is kept, and read-ahead stages none of it (shown only when not 0).origin_checks_skippedcounts the fetches from the origin (a byte-cache miss, single or vector, or the original bytes of a file in uCache's own layout) that could not be checked at all because the file's map could not be made: a read of the file's records failed (another is made a second later or more, at the next miss), or the parse ran out of memory. Such a fetch is served unchecked, as without the setting, and nothing of it is kept; the first one of a file in a process is also said in the log, as a warning naming the file (shown only when not 0). Read-ahead made while the map is not known stages nothing and is not counted here.origin_checks_failedcounts the baskets of such a fetch that the reader asked for whose key does not say they sit where the tree's record puts them, whose length differs from the one that record gives them where it changes ROOT's reading (the file map), or whose data fails its own checksum or does not decode; and, as one each, the reads a copy answered of the parts of a tree whose record showed the copy bad (its record failing its own checksum, or placing a basket past the copy's end; the tree's record and, where it could be read, its baskets: a read of another tree of that copy is not refused). Such a read fails with "failed its check (the reason) ... not served", naming the position and the server, and nothing of that fetch is kept; the fetch's bytes are not counted inorigin_bytes, and itsorigin_failurerecord, not awiretrace line, shows it. Read-ahead counts the baskets of its own fetches that fail (nothing of such a fetch is staged; no record, since no reader asked for them).cold_replica_convert_us— time spent converting, summed over the threads that did it; it is CPU a pass spends that a pass without recompression does not.cold_replica_declined— converted records not kept: they still did not fit above the free-space floor after an eviction pass, writing them failed (the log says which), the store was removed meanwhile, or the cache disk was too slow to take them (slow_cache_episodes). They were served to the job that read them; a later read converts them again.cold_replica_waits,cold_replica_wait_us— the times a first-pass request waited for the cache's writes before converting more (the records converted and not yet written had passed 512 MiB in the process), and the time they waited, summed. Each request waits 2 seconds at most, over all its tries. Nonzero means the cache disk, or the kernel's limit on unwritten data (vm.dirty_bytes), set the first pass's pace; the records were kept. The 512 MiB is where waiting starts, not a bound: requests converting when it is passed go on, and one that has waited its 2 seconds goes on too. After a commit that could not keep its records (no room, a failed write:cold_replica_declined) nothing waits until a commit keeps its records again.slow_cache_episodes,slow_cache_us— the times the cache disk was too slow to be filled, and how long, summed: a first-pass request waited its 2 seconds and the records were still waiting, or a write of the byte cache's staged pages made no progress for 2 seconds while more waited. Meanwhile the reads go on and what they fetch is served and not kept (converted records count undercold_replica_declined, byte-cache fills underfill_not_kept_bytes); it ends when what waited has been written (to half the limit). The log warns when it starts and when it ends (later episodes at most once a minute). Nonzero means the cache disk cannot keep up with the jobs: the cache fills less, and later passes fetch more from the origin.fill_not_kept_bytes— bytes byte-cache fills fetched and did not keep because the cache disk was too slow (above).cold_replica_completed,cold_replica_completed_bytes— basketscomplete_chunkstook along and converted on a first pass, beyond what the readers asked for, and their original bytes: the rest of each read branch's baskets in the clusters the readers touched, fetched from the origin or read from the byte cache. A basket a reader asks for later is not fetched again; one that failed to come or failed its check is not counted (nor kept).fill_completed,fill_completed_bytes— the same for byte-cache fills (a file with no replica store): baskets kept in the byte cache beyond what the readers asked for, and the bytes kept for them (basket bytes when they came with a vector read's miss, whole pages when fetched after a single read's).cold_origin_failures— origin reads sent on the first pass that failed. The request is served again from the start, twice at most, so a read the origin fails once or twice is answered; a third failure in a row reaches the reader, as it would without the cache. Nonzero means origin trouble during the pass.cold_replica_skipped— files whose first pass was declined by what the file itself holds: baskets in a codecrecompress_codecsdoes not name, baskets stored uncompressed, or a structure the first pass cannot lay out. Such a file is served as stored. Each file is counted once, by the process that decided it, when that decision is recorded in the cache; later reads of the file do not count it again. The plugin logs the first such file of each process as a warning, with the reason (see TROUBLESHOOTING.md, "recompress = onbut no replicas ever appear").slot_maps_made— maps this process made (mixed_maps = on): a later open is shown the baskets or RNTuple pages converted by then at their real size. One is made a moment after a process is done with a file, when enough has been converted since the newest (a twentieth of what the file's store holds, and after the first map 8 MiB) and nobody else is converting it; after the first, at most one an hour.slot_map_opens— opens shown a map. An open of a file with a slot store that is not counted here is shown the slot layout alone: no map yet, ormixed_maps = off.slot_map_refused— reads refused because they lay in the range of a compact map no longer live: replaced and, forin_use_seconds, neither held nor used; or in a map of a store that was replaced since (its range, or its tree record). The reader gets an error and the plugin logs one; reopening the file shows the current map.slot_map_full— maps not made because there was no place for one: the baskets converted since the newest map are read at their slot's size until a place frees: when no handle holds a replaced map and it has not been used forin_use_seconds.slot_crc_failures— records read from a slot store that failed their check (damage on the cache disk, or a torn write): each is dropped and converted again from the origin, and the read that met it is served again.slot_read_fallbacks— reads of neighbouring stored records (joined into one read of at most 1 MiB) that failed or came back short, and were read again one record at a time. Nonzero means a cache disk returning errors.slot_pages_decoded— RNTuple pages decoded from their stored record to be served. A page converted for the request that asked for it is served from the conversion's own decoded copy and is not counted, so a first pass counts here only pages it finds already stored. A later pass shown a map counts only the pages it reads in the slot layout -- column ranges not yet wholly converted, and the few an open reads past the file's original end; the pages of the map's range are served as stored and ROOT decodes them.kept_reads/kept_read_bytes— reads of baskets a slot store keeps as stored, served from the byte cache's copy of the original. They are not inhit_disk_reads(that counts the byte tier's own serving).hist_slot_read_us— time to read and check one stored slot record.hist_cold_request_us— a request served in the first-pass layout, from its arrival to the answer handed to the reader, retries included: what a reader waits for on the first pass.pool_tasks,pool_busy_us,pool_queue_high_water,hist_pool_queue_us— the serving pool, the threads every read is posted to: tasks it ran, time its threads spent running them, the most tasks waiting at once, and how long each waited before a thread took it. A long queue wait with the threads busy the whole time means the pool, not the disk or the origin, sets the pace.pool_thread_failures— threads of uCache's pools (workers and timers) that could not be started, because the process may make no more (an address-space limit,ulimit -v, or a limit on processes). Nothing fails for it: the work of a pool short of threads runs on the threads that did start, on another pool, or, when none could be started anywhere, on the thread that asked. The first failure is said once at WARN.reader_wait_byte_us,reader_wait_replica_us,reader_wait_slots_us,reader_wait_origin_us,reader_threads— how long the threads that read through the cache waited on it. For each reading thread, the time during which it had at least one read outstanding in the cache, from the moment it sent one while none was outstanding until the last one was answered (reads a thread sends together and waits for together count once). Each such wait is charged to the costliest place any of its reads needed: the origin (a fetch, or waiting for one already on the wire), then a slot store, then the byte cache. (The fourth slot, an earlier release's compact replica, stays 0.)reader_threadscounts the threads that read. For TTree the readers are the analysis threads, so the sum divided by threads x wall is the share of their time spent waiting on the cache; for RNTuple they are ROOT's own I/O threads, which wait by design while the analysis computes, so there the sum is request latency only. Handles that do not cache (UCACHE_DISABLE=1, a copy) record nothing here.
Records a job reads from a slot store count in replica_bytes_served (and
their reads from the cache disk in replica_reads / replica_read_bytes);
records it converts from what the origin sends count in miss_bytes, as any
first read does, so the pass that makes a store reads as a fill. The zeros
that pad each slot count in neither.
ucache summary never takes a run that converted baskets this way as its
baseline: its time includes the cache's own conversion work. A plain first pass
that only filled the byte cache is unaffected and still qualifies under the
usual rules.
ucache stats does the divisions in its workflow: block from the counters
below — how often each file was opened, which tier served what, how much of the
reading was re-reading, and what the cache disk was asked to do.
-
Workflow counters:
files_opened— distinct keys opened by this process (opens/files_opened≫ 1 = a reopen loop);ram_hit_bytes— hit bytes served from the fill buffer's staged RAM (subset ofhit_bytes);first_touch_bytes— bytes served for the first time in the entry's in-process lifetime (the rest are re-reads). -
Byte-tier disk reads:
hit_disk_reads/hit_disk_bytes/hit_disk_seq— byte-tier disk reads, their bytes, and how many continued sequentially from the previous read. Onehit_disk_readsis one pread covering a whole contiguous run of cached pages, not one per page — a request spanning ten adjacent pages costs one read op — sohit_disk_bytes / hit_disk_readsis the mean size the device sees andhit_disk_readsdivided by wall time is the op rate it is charged for (the currency on IOPS-quota storage). Bytes are counted at whole-page granularity, since each page is still checksummed in full.hit_disk_bytesis directly comparable to versions before reads were coalesced for every request that SUCCEEDS (verified byte-identical on the same workload). It differs only for a request that fails part way: the older per-page loop had already read some pages before discovering an absent one, whereas the planning pass now finds that page first and reads nothing — so a run with a non-zero acceptance triple can show slightly fewer bytes than an older one. -
Replica-tier reads:
replica_bytes_served/replica_reads/replica_read_bytes— bytes served from slot-store records, the physical.slotspreads that delivered them, and the bytes those preads moved (the replica-tier analogue of the byte-tier trio;replica_read_bytes≥replica_bytes_servedbecause whole records are read and verified, and a record read for two different user reads counts twice). -
Pass-through and vector reads:
relay_bytes— pure pass-through, the cache never touched them (copies included: seecopier_handles; files read directly too: seedirect_read_files);readv_chunks/readv_calls/readv_mixed— vector-read chunks seen, vector reads the cache handled (readv_chunks / readv_calls= mean batch width), and vectors mixing hits with misses. -
Fill path:
flush_runs/flush_run_bytes— coalesced fill-buffer pwrite runs and their bytes;buffer_stalls/buffer_stall_us— fills that had to drain synchronously at the buffer cap, and the wall time lost;fetches_joined— misses that joined an identical in-flight fetch instead of fetching again. -
Read-ahead (
prefetch, OFF by default; when switched on the plugin predicts a TTree reader's next fill from the file's own basket map and fetches it while the reader computes, into RAM only):prefetch_issued_bytes— asked of the origin ahead of demand;prefetch_served_bytes— of those, bytes the reader then demanded (they also count as hits);prefetch_refetched_bytes— served bytes a demand read had fetched again in the meantime;prefetch_dropped_unread— bytes read ahead that the reader never asked for, dropped from RAM and never written to the cache;prefetch_late_bytes— arrived after the demand read had already brought them;prefetch_parses— basket maps parsed (successes; a map is retried while the reader's header pages are still being staged, and those attempts do not count);prefetch_bridge_bytes— padding between two predicted ranges, fetched to save a request element and discarded on arrival (prefetch_bridge_kb): bandwidth only, never staged, never cached, and never counted against read-ahead's own accuracy;prefetch_disabled— 1 once a process switched read-ahead off, so a total over several processes is a count of how many did;prefetch_fetch_errors— read-ahead wire reads that failed (dropped; never a fail-open event).fetches_joinedcounts the reads that waited for a fetch already on the wire rather than sending their own, which is how read-ahead stops paying for the same bytes twice (prefetch_join). Read-ahead's wire traffic is counted inorigin_bytesandorigin_readvs, as any other origin read, but NOT inhist_origin_rt_us: that histogram is time a reader waited, and nobody waited for these. A run that read ahead at all is never taken as a no-cache baseline bysummary, whether or not the prediction was used. Per file,prefetch_issued/prefetch_served/prefetch_droppedin the.files.jsonlrecord. -
hist_relay_rt_us— answer times of the requests the plugin passed straight through to the origin (the ones whose bytes count inrelay_bytes, aUCACHE_DISABLErun's included), process-wide. Kept apart fromhist_origin_rt_us, which is the cache's own fetches and keeps meaning only that. -
Histograms are log2 buckets: bucket i counts samples with
floor(log2(v)) == i(bucket 0 = ≤1); arrays are trimmed of trailing zeros, 40 buckets max. The_usones bucket microseconds; the_bytesones bucket bytes (bucket 12 = 4 KiB, 16 = 64 KiB). -
Read-shape histograms answer "what is the storage actually asked for":
hist_req_read_bytes— sizes the client requested (one sample per read and per vector-read chunk, before the cache decides how to serve them);hist_hit_read_bytes/hist_replica_read_bytes— sizes of the preads each tier issued. Issued sizes smaller than requested are normal on a partially cached entry, because a read stops at any page that is absent or still staged in RAM, and at the coalescing cap; issued sizes larger than requested come from rounding out to whole pages. What the pair is for is the shape of the tail: many issued reads far smaller than the requests they serve means the entry is fragmented, and on op-priced storage that is what costs. -
entries— per-entry page coverage of entries live in the process at dump time (cached_bytes / file_size= the coverage signal gating replica materialization). Closed entries' coverage persists in their sidecars and is aggregated byucache statsfrom there. -
handles_high_water— the most cache-engaged handles open at once in this process. The origin's delivery is a non-monotone function of concurrency, so measurements at different widths are not interchangeable; consumers pool runs across time only when this agrees. Aggregated across processes it is a MAX, never a sum. -
disabled— present (as1) when the process ran withUCACHE_DISABLE: the cache was out of the loop and every byte was relayed. Such a run is a measured NO-CACHE BASELINE;ucache summary/historycompare cached runs against it, matched by key through the per-file records. -
threads_high_water— the most threads this process had alive at once, every thread it owns, not the ones doing the work. Recorded, not a width: a runtime's worker pool inflates it freely. -
peak_cores— the most cores busy in any one second, from CPU time sampled once a second on the read path. This IS a width: CPU seconds per wall second cannot exceed the cores actually executing. Sampled where reads happen, not where files open, because opens all land before the work starts and would report one core for a job running on thirty-two. -
cpu_us,instructions,cycles— work done, as opposed to time taken. Instructions are the stable one across routes: the same analysis over the same data executes very nearly the same number however the bytes reached it, which is what makes two runs comparable at all. Requires performance counters to be permitted; absent (zero) where they are not. -
counter_source— whereinstructionsandcyclescame from:perf(Linux performance counters: user space only, counted from when the cache engaged in the process) orrusage(macOS per-process counters: the whole process from its start, user and kernel). The two do not count the same thing, so compare counts only between runs with the same source. Absent when there are no counters. -
pmu_duty—perfonly, four decimals: the smaller, over the two counters, of the share of the time a counter was enabled that it was actually counting. Below 1 the processor's counters were shared (with other jobs on the node, or other counters in the process) and the kernel multiplexed them; theinstructionsandcycleswritten are already SCALED by enabled/running time, so they estimate the full count, and a low duty is how much of it was estimated. Like the counts, it covers the process from when the counters opened, so each line's includes the lines before it;ucache history --jsonreports the last line's, which is the whole run's. -
ts_ms,start_ms— the same clock astsand the file name's start, in milliseconds: when this line was written, and when this process's store started. Their difference is the run's wall to a millisecond, where the file name andtsgive it to a second. The final line's time, and itscpu_us,instructionsandcycles, are read as the final dump begins, before the dump writes the records of files still open: that work is not the job's, and is in neither. -
origin_reads_in_flight_high_water— the most reads the ORIGIN was being asked for at once. Counted where a read is issued to the origin and given back at the top of its completion handler, so it spans the READ. This is the width that sets the origin's delivery rate, and it is the one to use. A warm run reads 0, correctly: nothing was outstanding at the origin because nothing went there. It is recorded and displayed; nothing is decided from it. -
reads_in_flight_high_water— SUPERSEDED by the counter above, and kept only so the old name is not silently reused for the new meaning. It counted reads being SET UP at once: every route dispatches and returns while a synchronous caller waits above this library, so it spans the handoff, not the read, and a job with thirty-two reads genuinely outstanding can report a handful. It is emitted as a permanent 0. Do not read anything into it. -
Counters are process-lifetime monotonic; successive lines from one process supersede earlier ones (consumers take the last line per file).
-
<stem>.files.jsonl— one record per entry per process lifetime, emitted once (at last release, or at the final dump). A handle the cache never served — pass-through underUCACHE_DISABLE, a copy, or a write-opened file — also leaves one at close, with the relayed bytes underwire_bytesand the cache-side fields zero; that is what makes a baseline run matchable file by file. Consumed byucache stats --files:{"ts": 0, "ts_ms": 0, "key": "root://…", "opens": 0, "served_bytes": 0, "ram_bytes": 0, "replica_bytes": 0, "disk_reads": 0, "disk_seq": 0, "disk_bytes": 0, "first_touch_bytes": 0, "wire_bytes": 0, "direct_bytes": 0, "span_us": 0, "origin_size": 0, "read_sig": "", "read_buckets": 0, "mode": "cached", "orig_bytes": 0, "unique_bytes": 0, "origin_rt_us": [..], "data_server": "disk7.site.example", "data_servers": "disk7.site.example:1094=…"}keyis the file's cache key: its URL, normalized (no query), or, withlogical_name_prefixesset and a path that holds one of them,lfn:and its logical name (lfn:/store/mc/...; USER_GUIDE.md, "Keys by logical name"), for whatever URL the job opened.The same file holds lines of newer kinds, which carry
"kind"and no"key": one per origin open attempt ("kind":"open") and one per failed origin request ("kind":"origin_failure"); FORMAT.md gives their fields. They are written at the periodic checkpoint and at exit, never from the client's callback threads. A reader of per-file records skips them by the absence of"key", asucache stats --files,summaryandhistorydo.The fields from
orig_byteson are recorded for the report service (ucache publish);summaryandhistorydo not interpret them. Recording them never changes what is read, fetched, cached or served: whenever one cannot be had cheaply, it is left out.orig_bytes— the bytes the application asked this file for, in the ORIGINAL file's coordinates, re-reads counted, clipped toorigin_size.unique_bytes— the distinct bytes among them: the union of the same ranges.origin_rt_us— answer times of this file's requests to the origin, a histogram likehist_origin_rt_us: the cache's own fetches on the cached route, the relayed requests on the pass-through route. A cached entry's record also holds the requests its handles relayed past the cache (a file read directly undermax_read_fraction, a fail-open), so its sum ishist_origin_rt_usplus that share ofhist_relay_rt_us. Read-ahead fetches are left out, as they are fromhist_origin_rt_us. Absent when there were none.ts_ms— when the record was written, in milliseconds; for a record the final dump writes (a file still open at exit), the moment the dump began, like the counter line it goes with.data_server— the host, without its port, that delivered the most of this file's origin bytes in this process;""when nothing came from the origin (a file served from the cache).data_servers— every server this file's origin requests went to, ashost:port=bytes:reqs:rt_sum_us:rt_max_us:nc_max:fails,;between servers, most bytes first, at most 8 named and then+Nfor the others.bytesare what the server delivered,reqsthe requests it answered,rt_sum_usandrt_max_ustheir answer times (issue to completion) summed and at most,nc_maxthe most origin reads this process had in flight to that server when one of them was issued (itself included; the client keeps one connection per server in a process, so these are the reads on that connection; a read is counted against the server the file's handle was using when it was sent, and 0 when none was known), andfailsthe reads that failed (every one; the failure lines are capped; a failed open is in theopenlines, not here). The server is the last host of the request's answer, so a read the client recovered elsewhere is put down to the server that answered it; one whose answer names no host is put down to the server the file's open reached (on a pass-through handle, whose open is not watched, a failed one to the server that delivered most of its bytes). The same requests asorigin_rt_us; read-ahead's are in neither.
ucache summary,historyandpublishtake a file's origin from the host in the file's URL, as they always have, unless the run hadfailover = on(its counter line'srun_settingssay so) and the file's URL names a server (host and port, 1094 when the URL names none) found to be a redirector (the counter line'sredirector_hosts): failover applied to the file. For such a file they take it fromdata_serverwhen it names one, and from the host in the file's URL otherwise (files that read nothing from the origin): for a file opened through a redirector the origin is then the data server the reads went to, andhistory --jsonandpublishshow that server's domain underoriginsandorigins_rt, not the redirector's. A run whose records do not say (written beforerun_settingsexisted, or a job killed before it wrote a counter line) is taken as one with failover off. A file keyed by its logical name has no host in its key: its origin isdata_server, and a file of such a run that read nothing from the origin names none.With
logical_name_prefixesset in the settings the command reads (the conf or the state file), or when any run of the records keyed files by these prefixes (its counter line'srun_settingssaylogical_names=on/<hash>with this list's hash),summary,history,stats --filesandpublishread a record's key as the file is keyed now, by the command's ownlogical_name_prefixes: a key by URL whose path holds a prefix is read aslfn:and the name, so the records of one file, written by URL before the prefixes were set or through other servers, are one file; a run'ssig(the hash of its set of files) and the matching of a baseline's files are computed on those keys. Such a record's origin is still the host its URL named. The records of a run keyed by other prefixes (another hash) are read as written, and the command says on stderr how many such runs there are. Otherwise the keys are read as written.orig_bytesandunique_bytesare exact or absent, never estimated: absent means unknown, and a file opened and not read has 0. How each route gets them:- Byte cache, pass-through, cache disabled: the application's requests address the original file, and are counted as they are.
- A replica (a recompressed copy, or the slot store of
recompress = on) has its own layout, and a read of it is mapped back through the replica's map of what each piece holds. Bytes still where the original has them count as themselves. A relocated basket or page read WHOLE counts as its whole original record. A read that covers only PART of a relocated basket or page has no place in the original file -- its bytes are recompressed, or decoded -- so from that read on the file records neither field, for the rest of the process. ROOT reads a TTree's baskets and its tree record whole, so a TTree file that ROOT reads through a replica keeps them. Other readers mostly do not: measured, uproot's reads of a TTree through a replica leave neither, and so do ROOT's reads of an RNTuple, whose buffered reads near the footer take part of the pages beside it. - Either is absent too when the footprint was discarded;
orig_bytesalso when the file's size was not known well enough to clip the reads to it;unique_bytesalso when the file's range set outgrew its cap (100000 ranges per file, 128 MiB of ranges per process).
They count what the reader asked of the layout it was shown, and a reader does not ask a replica for the same bytes as the original: one that reads ahead in blocks counts the extra bytes of the original on the byte route, and on a replica either whole pieces or, once it takes part of one, no count at all. Compare them between runs of one route;
read_sig(below) is the field made to compare across routes.One key can have several records in a process (one per relayed handle, or an entry dropped and opened again), and the fields combine differently.
orig_bytesandunique_bytesdescribe the process's whole footprint on the URL, so later records hold what earlier ones did: take them, as one unit, from the record with the largestorig_bytes(on a tie, the later one) — never add them. A record withts_msand noorig_bytessays the footprint became unknown after the records before it (discarded, a replica read that could not be mapped exactly, or asked past the file's end more often than it could follow): the unit is then unknown, until a later record carriesorig_bytesagain.origin_rt_uscovers only the requests of the record's own handle or entry: add those bucket by bucket, as withwire_bytes.direct_bytesare the bytes a file read directly (max_read_fraction) fetched and did not keep.modeisfillwhen the process mostly fetched the file,relaywhen it mostly read it straight from the origin (a file read directly), andcachedotherwise.origin_sizeis the file's size at the origin — the one measure of a file that means the same thing whichever route served it, and therefore the only sound weight when combining files.read_sigandread_bucketssay WHICH parts of the file were read, and how many, in the ORIGINAL file's coordinates. The file is divided into 1 MiB blocks and the signature is a hash of the set that were touched; a read served from a replica is mapped back through the sidecar's origin map first, so the two routes are expressed in the same coordinates.That is not the same as agreeing, and MEASURED THEY DO NOT ALWAYS AGREE. A replica is a rewritten container, so the reader asks it for different spans than it asks the original -- not merely coalesced differently, requested differently. On a recorded file set, 26 files of 439 (94.1% agreement) had a whole block fall inside a gap the replica route never asks for: the byte route marked it, the replica route did not. Both are honest about what was read; they answer slightly different questions.
So equality is evidence of the same work, while inequality is NOT proof of different work -- which is why runs are compared on the SHARE of files that agree rather than on a single hash over the whole run, and why that share is required to be high rather than total.
An empty
read_sigmeans UNKNOWN, never "nothing was read" — a replica written before origin maps existed cannot answer, and so says nothing rather than something wrong. Treat it as no evidence.A consumer reading these must take the record with the most
read_buckets, not the last one. A file can be reported more than once in a process, and the footprint grows for as long as the process runs, so earlier records hold subsets of later ones. Taking the last record is right for every other field here and wrong for this one.span_usis the wall this file was live and working, from open to last activity. Where one worker handles one file at a time that is the worker's full cost for it — waits and the route's own CPU together — which is what makes files served by different routes comparable.modeiscached,fill(this process mostly fetched it) orrelay(passed through, including aUCACHE_DISABLErun). -
<stem>.trace.jsonl— sampled per-operation IO trace, written only withtrace = io(everytrace_sample-th operation per class). Records are{"t": <wall µs>, "w": <worker slot>, "op": "<operation>", "k": "<16-hex key hash>", "off": 0, "len": 0, "us": 0};wis a small per-thread slot id, assigned on that thread's first record and stable for its life, so a consumer can reconstruct each worker's timeline — the gap between one worker's completion and its next issue is that worker's compute. The first record for each key is a legend line{"op": "key", "k": "<hash>", "url": "<full url>"}mapping the hash back to the URL. A record of an origin request (wire: the byte cache's fetches, a replica's first pass and the reads of a file's own records;swire: a read-ahead fetch) has three more members afterus:ds, the server that answered it (host:port, as indata_serversabove;""when none is known),t0, when it was issued (µs of a steady clock, comparable between the records of one process only), andnc, the origin requests this process had in flight to that server at the issue, itself included (0 when the server was not known then).ncis counted against the server the file's handle was using when the request was sent, which differs fromdsonly when the client redirects or recovers the request.