Description
Cache::clockTick scans the per-cycle event buffer front-to-back and retries any event it cannot accept. An event that is rejected (e.g. MSHR full, bank busy) is skipped and left for a later cycle, while later events in the same scan continue to be evaluated. That is normally harmless, but if an earlier request to a line is rejected and a response later in the same scan frees the blocking resource, a later same-address request can be accepted ahead of the earlier one — inverting two same-address requests issued in program order by a single source.
For two same-address writes this silently drops an update: the older request is accepted last and overwrites the newer one, so memory ends up with the stale value and no error is raised. The reorder is protocol-independent but is most easily hit with coherence_protocol=none and a small MSHR, where nothing else re-serializes same-address requests.
Root cause
Paths on devel @ f782290:
cacheController.cc — Cache::clockTick (line ~101): iterates eventBuffer_ front-to-back (line ~154), calling processEvent and retrying events that return false.cacheController.cc — Cache::processEvent (line ~241): arbitrateAccess(addr) (line ~274) and the MSHR/coherence accept path can reject a request; the rejection is silent w.r.t. later same-address events in the same scan. There is no per-cycle record that a given address was already rejected, so a subsequent same-address request evaluated later in the scan can be accepted once an intervening response frees the resource — accepting the younger request before the older one.
How to reproduce
The defect is a within-cycle scan-ordering issue and is timing-dependent: it requires two same-address requests plus a resource-freeing response to land in the same cycle's event buffer while the MSHR is full. It is established by code inspection above; it does not reliably surface with stock synthetic traffic generators (they write data-independent payloads and don't consistently pack the required same-cycle sequence), so there is no simple deterministic script — the original observation was under a data-dependent workload.
A configuration that sets up the necessary conditions (single request source, coherence_protocol=none, small MSHR, repeated same-line writes interleaved with cold-line misses to churn the MSHR):
import sst
cpu = sst.Component("core", "memHierarchy.standardCPU")
cpu.addParams({"clock":"2GHz","memFreq":"1","rngseed":"101","memSize":"256KiB","verbose":0,
"maxOutstanding":16,"opCount":200000,"reqsPerIssue":8,"write_freq":80,"read_freq":20})
iface = cpu.setSubComponent("memory", "memHierarchy.standardInterface")
l1 = sst.Component("l1cache", "memHierarchy.Cache")
l1.addParams({"access_latency_cycles":"4","cache_frequency":"2 Ghz","replacement_policy":"lru",
"coherence_protocol":"none","associativity":"2","cache_line_size":"64","cache_size":"512 B","L1":"1",
"mshr_num_entries":2})
mc = sst.Component("memory", "memHierarchy.MemController")
mc.addParams({"clock":"1GHz","addr_range_end":512*1024*1024-1,"backing":"malloc","backing_init_zero":"true"})
be = mc.setSubComponent("backend", "memHierarchy.simpleMem")
be.addParams({"access_time":"100 ns","mem_size":"512MiB"})
sst.Link("a").connect((iface,"lowlink","1000ps"),(l1,"highlink","1000ps"))
sst.Link("b").connect((l1,"lowlink","10000ps"),(mc,"highlink","10000ps"))Environment
- OS: Ubuntu 22.04.5 LTS, kernel 6.8.0
- Compiler: gcc 11.4.0
- MPI: Open MPI 4.1.6
- sst-core: 15.0.0 (
87e9e3b)
- sst-elements: v15.0.0 (
d713776) and devel (f782290) — the code path is unchanged across the v16.0.0 release
Proposed fix (PR to follow)
Record, per cycle, the addresses whose request was rejected, and reject every later same-address request for the rest of that cycle's scan. On the next cycle the event buffer's FIFO order lets the earlier request claim the freed resource first, restoring program order. Responses are exempt so MSHR-draining events always flow and forward progress is preserved. Small, protocol-independent change in cacheController.{cc,h}.
Related issues
Same code area as the retry-ordering / bank-arbitration class of bugs (e.g. #2241, a livelock from retry ordering; #2156, a clockTick event-acceptance issue) — this is a distinct silent-data-corruption (lost-write) manifestation, not a hang.
Description
Cache::clockTickscans the per-cycle event buffer front-to-back and retries any event it cannot accept. An event that is rejected (e.g. MSHR full, bank busy) is skipped and left for a later cycle, while later events in the same scan continue to be evaluated. That is normally harmless, but if an earlier request to a line is rejected and a response later in the same scan frees the blocking resource, a later same-address request can be accepted ahead of the earlier one — inverting two same-address requests issued in program order by a single source.For two same-address writes this silently drops an update: the older request is accepted last and overwrites the newer one, so memory ends up with the stale value and no error is raised. The reorder is protocol-independent but is most easily hit with
coherence_protocol=noneand a small MSHR, where nothing else re-serializes same-address requests.Root cause
Paths on
devel@f782290:cacheController.cc—Cache::clockTick(line ~101): iterateseventBuffer_front-to-back (line ~154), callingprocessEventand retrying events that returnfalse.cacheController.cc—Cache::processEvent(line ~241):arbitrateAccess(addr)(line ~274) and the MSHR/coherence accept path can reject a request; the rejection is silent w.r.t. later same-address events in the same scan. There is no per-cycle record that a given address was already rejected, so a subsequent same-address request evaluated later in the scan can be accepted once an intervening response frees the resource — accepting the younger request before the older one.How to reproduce
The defect is a within-cycle scan-ordering issue and is timing-dependent: it requires two same-address requests plus a resource-freeing response to land in the same cycle's event buffer while the MSHR is full. It is established by code inspection above; it does not reliably surface with stock synthetic traffic generators (they write data-independent payloads and don't consistently pack the required same-cycle sequence), so there is no simple deterministic script — the original observation was under a data-dependent workload.
A configuration that sets up the necessary conditions (single request source,
coherence_protocol=none, small MSHR, repeated same-line writes interleaved with cold-line misses to churn the MSHR):Environment
87e9e3b)d713776) and devel (f782290) — the code path is unchanged across the v16.0.0 releaseProposed fix (PR to follow)
Record, per cycle, the addresses whose request was rejected, and reject every later same-address request for the rest of that cycle's scan. On the next cycle the event buffer's FIFO order lets the earlier request claim the freed resource first, restoring program order. Responses are exempt so MSHR-draining events always flow and forward progress is preserved. Small, protocol-independent change in
cacheController.{cc,h}.Related issues
Same code area as the retry-ordering / bank-arbitration class of bugs (e.g. #2241, a livelock from retry ordering; #2156, a clockTick event-acceptance issue) — this is a distinct silent-data-corruption (lost-write) manifestation, not a hang.