A CHI home-node agent, and a XiangShan cluster that boots by zarubaf · Pull Request #4 · silicon-org/foundry

zarubaf · 2026-08-20T16:26:52Z

Nothing in this repository drove //hardware/soc/xs_cluster's CHI port, so the design had only ever been elaborated by the lint aspect and the core had never fetched an instruction. This branch takes that to a XiangShan cluster executing a program out of a memory on the far side of a CHI link.

The lasting goal is agents where the interface is separate from the coherence protocol, so the same harness serves CHI now, AXI next, and TileLink when Spatz clusters arrive. See hardware/vip/README.md for the layering and tasks/xs-cluster-tb.md for the milestone-by-milestone record.

What lands

Toolchain. CIRCT and firtool built from source, C++20 host and target, and XiangShan's Chisel elaboration made reproducible — EnumUInt.scala discovered enum members by reflection in JDK-order, which renamed FIRRTL nodes and changed every byte of the 139 MB of generated SystemVerilog run to run.

The CHI stack, four layers with a test at each:

layer	what	tested by
`hardware/ip/chi`	Issue E.b package: opcodes, responses, link bundles	588 checks against CHIron, both directions
`vip/chi/rtl`	credit-based link layer, knows no opcode	7 cases on a loopback, no XiangShan in the build
`vip/chi/dpi`	one packed flit per call	—
`vip/chi`	`chi::HomeNode`, simulator-free	every opcode, no RTL in the build

Two upstream CHIron bugs found and patched along the way — the flit serializer had never worked, and two of twelve multi-word reads dropped the high half of an address.

The testbench boots the cluster. Link up at cycle 199, reset vector fetched at 1048 (CoupledL2 clearing its directory, not the core being slow), program executed, and the run ends on the store to ToHost.

The last four commits

refactor(vip): let Verilator generate main() — --main emits the same --timing loop we hand-wrote, so three main() files were boilerplate. Testbench directories are now SystemVerilog only. The verdict moved into the testbench with it: a generated main() always returns zero, so vip_test_failed() plus $fatal is what fails a run. Reviewer note: Verilator's gotError() no longer fails a run, so an $error without $fatal now passes.

feat(xs_cluster): dump FST waveforms — a second verilated model rather than a flag, so tracing never slows CI. $dumpvars off a plusarg, so it works on other simulators.

fix(vip): compute CHI DataCheck in the home node — the reason the core never executed. CHI DataCheck is odd parity per byte and CoupledL2 checks it; leaving the field zero claims even parity, wrong for 0x00 among others. Every line arrived marked corrupt → TileLink d.corrupt → hardware-error exception on the first instruction → and with mnstatus.NMIE zero out of reset, Smrnmi made that a critical error. Three lines to fix, and a test that pins it.

build(tools): pin tsunami-serve — the waveform query engine the debugging agents drive over MCP, pinned like every other CLI. Needed zarubaf/tsunami#3 upstream first.

Verification

bazel test //hardware/... — 15 tests pass
bazel build --config=lint //hardware/... — clean
Each of the last four commits was checked on its own tree, not just the tip

🤖 Generated with Claude Code

Retires the port-forward for the one dashboard that gets opened daily. An Ingress rather than a `tailscale.com/expose` annotation on the Service, because the difference is TLS: the annotation is a layer-4 proxy on plain HTTP, so the browser posts the login form in the clear and says so. The Ingress terminates with a real MagicDNS certificate. Not `type: LoadBalancer` -- the Hetzner cloud controller manager would see that and provision a public load balancer, which is invariant #4 breached by a field name. Prometheus and Alertmanager stay on the tunnel deliberately. Neither authenticates, so publishing them would put an unauthenticated read of every metric in front of anyone on the tailnet, and both are reached rarely and by an administrator -- which is what a tunnel is for. root_url follows, because Grafana builds share links and alert notification URLs from it and the default says localhost:3000. That was true when a port-forward was the only way in, and is now wrong in a way nothing reports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZdCMAkREhSzPRfdGxUn65

Three things need it, which is why it is one line in //.bazelrc rather than three sets of per-target copts. Verilator's --timing runtime is built on coroutines, and rules_verilator responds to `timing = True` by compiling the generated model at -std=c++20 whatever the rest of the build uses. A testbench links our C++ against that model, so leaving the two halves of one binary at two standards is not a configuration worth having. CHIron, the CHI flit library the verification IP is built on, is concepts and consteval throughout and does not compile at 17 at all. And it settles the question tasks/todo.md left open at M5 of the CIRCT work, where slang needs C++20 and the alternative was compiling MLIR headers at two standards inside one translation unit. The cost is that LLVM, MLIR and CIRCT are rebuilt at a standard upstream does not test them at, so that is checked rather than assumed: `bazel build @circt//:firtool` builds clean, 1883 actions, with no source changes anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

First piece of //hardware/vip: reusable bus agents, laid out so that the interface is separate from the protocol. vip/README.md describes the four layers and how a second protocol is added; this commit lands the bottom of the CHI stack, the flit encoding, and nothing that touches a wire yet. CHIron supplies the flit layouts and their pack/unpack, and is vendored rather than fetched. hardware/vip/chi/chiron/PROVENANCE.md gives the reasoning at length: it is a 700 KB subset of a 13 MB header-only library, the rest being a Qt trace viewer, its tests and two protocols we do not speak; and we need to patch it. refresh.sh re-copies from a named commit and re-applies the patches, so verifying the copy is one command and a diff. hardware/README.md records the exception against its own rule. The patching turned out not to be hypothetical. Two upstream bugs, both in chi_util_flit.hpp, both worth sending back: Serialize* has never worked. FlitAppender::Append32 omits the else that advances `offset`, so every field that does not cross a word boundary is written on top of the previous one; the crossing path advances `index` twice; and the mask shifts by 32 when a field ends on a boundary. The function's own assertion fires on the first flit. DeserializeREQ and DeserializeSNP drop the top of Addr -- the high half is OR-ed in with no << 32, so 0xabcdef012345 decodes as 0xef01abcd. The other ten multi-word reads in the file have the shift. All twelve now go through one Walk64 helper, which also removes the unsequenced `|` they shared. Only deserialization was ever exercised upstream: it arrived in the one human-authored pull request that repository has merged. chi_flit.h states the link configuration once -- 11-bit NodeID, 48-bit address, 256-bit data, DataCheck, Poison and MPAM present -- and asserts the four flit widths it produces against xs_cluster's port declarations. Everything later is written against these aliases rather than against CHIron. chi_flit_test pins the bit positions before it round-trips anything, because a packer and an unpacker wrong in the same way agree with each other: 67 fields, each set to all ones on its own, each required to occupy exactly the range the specification gives it. Both bugs above survived a round trip and died here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

//hardware/ip/chi describes the protocol once, for both the designs that speak it and the agents that verify them. Shaped after hw.riscv#2184 -- chi_pkg.sv for the encodings, chi_typedef.svh for the typedef macros -- and written from ARM IHI 0050 rather than copied, since that PR's files are marked confidential and this repository is public. Two things it has that #2184 does not. Link bundles. A CHI link is not valid/ready: a transmitter may send only while it holds an L-Credit, the receiver grants credits by pulsing LCRDV, and there is no ready anywhere. chi_rn_link_tx_t and chi_rn_link_rx_t group, per direction, the flits that direction sends and the credits it returns for what it receives, so xs_cluster's thirty loose CHI signals become two ports with a type. The valid/ready bundles remain the right shape for RTL sitting behind a link layer; these are for anything touching pins. And the opcodes are machine-checked rather than transcribed and hoped for. chi_pkg_test walks all 208 encodings of the four opcode spaces and compares this package's enum against CHIron's independent Issue-E.b tables by name, in both directions: an opcode defined here and not there fails, and so does one defined there and not here. That matters because the current specification is Issue H, which defines around thirty REQ opcodes E.b does not, and eyeballing which is which is exactly the sort of thing nobody does twice. 588 checks in total, with the field widths, the snoop classifiers over their whole 32-value domain, and the PassDirty helpers. chi_layout_test covers the other half: the package sets every field of a flit to a distinct value and hands the packed struct to CHIron to decode. 74 fields over four channels. Two flit decoders exist here for good reasons -- SystemVerilog for waveforms, RTL and assertions, C++ for the agents -- and this is the only thing that stops them drifting. Both testbenches are SystemVerilog driving C++ over DPI, which is the shape the agents need later. C++ owns the verdict because Verilator exposes no $error count to a main(), so the alternative is scraping stderr. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Thirty CHI signals become two ports, chi_pkg::chi_rn_link_tx_t out and chi_rn_link_rx_t in. The split is by direction rather than by channel because that is what a CHI link is: a transmitter may send only while it holds an L-Credit, so each side both sends flits and returns credits for what it receives, and there is no ready anywhere to pair them up by channel. The comment this replaces argued that a CHI package belongs with the interconnect that defines the address map and the node IDs, and that inventing one here would fix those choices before anyone had made them. The package now exists and fixes neither -- it describes a specification, the address map is still nobody's, and node_id_i is still an input -- so the comment is rewritten rather than left to mislead. The width check is deliberately not in the wrapper. It was, as a generate block with $error, until a deliberately false condition showed the build succeeding anyway: Verilator treats that as %Warning-USERERROR, and warnings are non-fatal here, so the check passed whatever the widths were. XSTop's four literal port widths went into chi_pkg_test instead, beside the ones chi_pkg computes and the ones CHIron computes, where a wrong one does fail -- confirmed the other way round. tasks/lessons.md records both halves. Lint elaborates the wrapper with the struct ports and reports no width warning on any CHI connection, which is the check that carries the weight here: nothing else in this repository would catch a mistyped port name against 1868 generated modules. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The build-risk milestone, on its own so its cost is visible: 676 s and 8187 actions to Verilate 139 MB of generated SystemVerilog with --timing, and 12 s to run 1000 cycles. A Verilator-only change afterwards rebuilds in about 130 s. --output-split 20000 is what keeps the C++ compiles parallel. The clock and the reset are SystemVerilog processes, which is the whole reason for --timing. Two things follow and both are worth the cost: the testbench reads like hardware rather than like a program poking a model, and it will run unchanged under a commercial simulator because nothing in it is Verilator's. //hardware/vip/common:sim_main is the evaluation loop every testbench here will share. Its deadlock detector found its own bug on the first run: gotFinish() has to be tested immediately after eval(), because $finish leaves nothing scheduled and the other order reports every successful run as a hang. The CHI port is held inactive, so the cluster comes out of reset, raises tx_linkactivereq, finds nobody there, and waits -- which is the correct thing for it to do and is what the test asserts, along with critical_error_o staying low. //hardware/vip/chi arrives on this port next. One scare, and it was ours rather than the design's: %Error-DIDNOTCONVERGE from `forever #(ClkPeriod / 2)`, where `time` is an integer type and the division rounded to zero, making an infinite loop in zero time. --timing on 1868 generated modules is fine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The interface half of the CHI agent: credits, the LINKACTIVE handshake, and nothing about what a flit means. chi_link_hn and chi_link_rn present six channels as valid/ready and never read an opcode except to recognise the L-Credit return, which is flow control rather than a message. What a ReadNotSharedDirty means belongs on the far side of a DPI boundary, in C++ that no simulator owns. Two channel modules do the work, parameterised by flit width so one serves all four channels. A receiving channel's credit count and its buffer depth are the same number, because a credit is a promise of somewhere to put the flit it will bring -- which is how XiangShan's own receiver is built, and the only arrangement that works when the receiving end cannot push back. Deactivation returns unspent credits as LCrdReturn flits. That was written because the test for it hung: a receiver may not withdraw its acknowledgement while it is still owed credits, so without it the link reaches DEACTIVATE and stops there. Seven tests over one loopback model, with no XiangShan in the build and seconds to compile. bringup, transfer -- six flits over six channels with payloads compared at the far end -- credit exhaustion and recovery, deactivation, and two negative cases that drive the pins directly with flits no credit paid for. The negative cases take the message they expect to see, because the first version of the overrun case passed on an unrelated assertion in a different module: the testbench had frozen a credit signal and three transmitters overflowed at once. A negative test that fails for the wrong reason is worth nothing, and a check nobody has watched fire is decoration. Two things learned the hard way and written down in tasks/lessons.md: reading a design's combinational `ready` from a procedural block after @(posedge clk) reads it after the design's own registers have moved, which silently dropped every flit on one channel; and `assert ... else $error` lets Verilator exit zero, so an interface invariant has to be $fatal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The protocol half of the CHI agent, joined to the link by DPI. chi::HomeNode is a function from flits to flits with no wires and no simulator in it, so chi_home_node_test covers every opcode it supports -- reads, copyback and immediate writes, the dataless requests, CompAck, WriteDataCancel, and an error response for anything else rather than silence -- in milliseconds, with no RTL in the build. One RN-F on this link means nothing else holds a copy of a line, which is why there is no directory and no snoop path. The whole of a test's setup crosses the DPI boundary, so the testbench is the test: xs_cluster_tb.sv holds the program, creates the memory, sets the address that ends the run and creates the home node, and its C++ is the one line every testbench shares. That is a correction -- the first version had the program and the watchpoint in a per-testbench main(), which put half of each test somewhere nobody would look for it. xs_cluster.sv brings the trace-encoder port out rather than tying it off. It is the only window into what the core is doing that is not a waveform, and the investigation below needed it. What the cluster testbench establishes: the CHI link comes up at cycle 199, the core asks for the line its reset vector is in at cycle 1048, and the home node serves it. The thousand-cycle wait is CoupledL2 walking its directory to clear it, not the core being slow. What it does not yet establish is that the core executes what it is served. It does not, and tasks/xs-cluster-tb.md records the investigation: the delivered line is correct at the pins, the failure is at the same cycle for three different programs including one that cannot trap, and the mechanism is a trap taken while mnstatus.NMIE is still zero, which Smrnmi makes a critical error. Which trap is the next question. Two tests were added that the plan did not have, and should have. chi_hn_agent_tb covers the seam -- link, DPI, protocol model and memory together, against a request node's link with no design in the build -- and it is what proved the flit data reaching the core was correct, which halved the search. The link had a test with no protocol and the protocol had a test with no link, and nothing covered the boundary between them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Verilator's --main emits the same --timing evaluation loop we hand-wrote, so the three main() files were boilerplate. Testbench directories are now SystemVerilog only, which is what a commercial simulator needs anyway. A generated main() always returns zero, so the verdict moves into the testbench: vip_test_failed() is a new DPI import and $fatal is the only thing that can fail a run. Verilator's gotError() no longer does, so an $error without $fatal now passes. sim_main.cc survives as verilated_shims.cc, the sc_time_stamp() stub.

Nothing here produced a waveform, which left no way to see inside the cluster when it misbehaves. A second verilated model rather than a flag on the test, so tracing never slows the target CI runs. $dumpvars off a plusarg rather than a Verilator API, so it works on other simulators too. bazel run //hardware/soc/xs_cluster/tb:xs_cluster_tb_traced -- \ +dump +dumpfile=/tmp/xs.fst

CHI DataCheck is odd parity per byte, and CoupledL2 checks it. Leaving the field zero does not mean "unused", it claims even parity, which is wrong for 0x00 among others. So every line the cluster fetched arrived marked corrupt, reached the L1I as TileLink d.corrupt, and became a hardware-error exception on the first instruction. mnstatus.NMIE is zero out of reset, so Smrnmi turned that trap into a critical error -- the symptom that had looked like the core's fault. The cluster now executes its program and ends on the store to ToHost, so the testbench waits for that instead of concluding once a line arrives.

Waveform query engine the debugging agents drive over MCP. Pinned like every other CLI here rather than built in the graph: it is a tool people and agents type at, not something a build action invokes. Upstream shipped no binaries until zarubaf/tsunami#3; the Linux builds are static musl, so the pin has no glibc floor and runs on the Talos workers as well as a laptop.

Verilating this design is one process over 139 MB of SystemVerilog and needs more than 16 GiB on its own. `bazel test //...` builds every target in the pattern rather than only the tests, so CI was asking the worker to elaborate the traced model alongside the untraced one -- two of the largest actions in the repository, concurrently, on a 30 GiB node. No limit that node can offer satisfies that, and it does not fail as an OOM: the runner container is killed and the worker beside it reports "dial unix /worker/runner: connection refused", which names a socket and reads as a cluster fault. Nothing is lost. The traced model exists for `bazel run ... -- +dump`, which names it explicitly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZdCMAkREhSzPRfdGxUn65

zarubaf force-pushed the xs-cluster-tb branch from cc3eefd to 8b9d08d Compare August 20, 2026 18:10

zarubaf and others added 13 commits August 23, 2026 09:33

docs: ask for brief comments that explain intent

5510089

zarubaf force-pushed the xs-cluster-tb branch from 8b9d08d to 8ed9be5 Compare August 23, 2026 07:35

zarubaf merged commit a10e0b1 into main Aug 23, 2026
3 of 4 checks passed

Navigation Menu

Repository navigation

A CHI home-node agent, and a XiangShan cluster that boots - #4

A CHI home-node agent, and a XiangShan cluster that boots#4
zarubaf merged 13 commits into
mainfrom
xs-cluster-tb

zarubaf commented Aug 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zarubaf commented Aug 20, 2026

What lands

The last four commits

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant