CI / Packaging

[josibake]

Starting this discussion as a general dumping ground for some ideas.

I've been playing around with a strict nix packaging repo here: https://github.com/josibake/champix. The result is a much cleaner CMake file in the source repo and clear separation of concern.

The next thing I want to investigate is fully moving over CI into the packaging repo. Could easily start with just a 1:1 translation of the existing CI, or we could start small and only add platforms we run on / support and keep things focused.Additionally, since its all nix native, we could spin up a Hydra build server and a cache.

One reason I'm hesitant to port the existing CI as is: I'd like to have a separation between "correctness" and integration. What I mean here is when a PR is pushed it should have correctness tests ran. These tests should be fast, and are mainly to make sure you don't break code for the testing environment. So a few multi platform checks, the unit tests, etc.

If the CI is green and their is conceptual agreement on the change, it auto promotes to a "staging" environment where it is now tested against all other recently merged to staging changes. This is where the heavy tests run: IBD, tsan, fuzz, benchmarks, etc. As errors are found here, they are opened as bug fixes reference the specific PR, or standalone bugfixes considering many bugs will surface based on interactions between this staged changes (see recent levelDB stuff in core).

After awhile , we could have stuff auto promote to master for a CI/CD style, but in a real world application I would expect it to be similar to Bitcoin Core where there is a time released and things are elected to be included in a release. This means they would be branched off the staging branch, and go into yet another round of testing before being included in a tagged release.

Curious to hear thoughts / improvements / disagreements / etc. I think I'll start by first replicating the champix repo, and then start building out a CI there and we can punt the actual correctness -> staging -> release stuff out later.

[josibake]

Alright, I've been hacking away at a design that I'm pretty happy with. Inspired by what I did with https://github.com/josibake/champy and https://github.com/josibake/champix. The basic idea is the source repository should keep owning implementation details, such as:

• build files
• tests
• install rules
• source branches
• source review

What I am calling Ironworks should own the external test, regression and packaging pipeline:

• nix dependency pins
• project specific adapters
• package profiles (e.g., hardening, RelWithDebug, etc)
• ci stages
• hydra jobsets (multi platform building)
• binary build caches (e.g., cachix)
• promotion tooling
• release manifests
• scheduled jobs (e.g., IBD, long running benchmark orchestration, stress testing)

stage vocabulary

You can think of the process of getting something from an idea to production ready in the following stages:

Stage	Purpose
spark	fast PR correctness checks.
forge	staging integration for reviewed changes.
harden	scheduled heavy validation: IBD, fuzz corpus, long benchmarks, compatibility.
temper	release candidate validation.
stamp	final tag, manifest, and release artifact publication.

pipeline

flowchart TD
    PR["Source PR"]
    Spark["spark: fast PR correctness"]
    Review["review approval and ready-for-staging"]
    Staging["promote to staging"]
    Hydra["Hydra build farm"]
    Cache["signed binary cache"]
    ForgeJobs["forge jobs: full build, tests, ASan/UBSan, TSan, MSan build"]
    ForgeGate["forge green: staging is eligible for harden"]
    Harden["harden: scheduled heavy validation"]
    IBD["deterministic IBD replay"]
    Fuzz["fuzz corpus and long fuzz budgets"]
    Compat["previous-release compatibility"]
    Bench["Benchkit long benchmarks"]
    BenchStore["benchmark metrics, logs, dashboard"]
    BenchReport["benchmark report: temper review input"]
    HardenGate["harden green: required IBD/fuzz/compat signal"]
    Temper["temper: release candidate validation"]
    Stamp["stamp: tag, manifest, artifacts"]

    PR --> Spark
    Spark --> Review
    Review --> Staging
    Staging --> Hydra
    Hydra --> Cache
    Hydra --> ForgeJobs
    ForgeJobs --> ForgeGate
    ForgeGate --> Harden
    Harden --> IBD
    Harden --> Fuzz
    Harden --> Compat
    Harden --> Bench
    Cache --> Bench
    Bench --> BenchStore
    BenchStore --> BenchReport
    IBD --> HardenGate
    Fuzz --> HardenGate
    Compat --> HardenGate
    HardenGate --> Temper
    BenchReport --> Temper
    Temper --> Stamp

Loading

project adaptors

Ironworks cand and should be generic. This works by having a top level flake that is common, and then writing a project specific adaptor in nix:

flowchart TD
    Ironworks["Ironworks flake"]
    Adapter["project adapter"]
    Packages["packages"]
    Checks["checks"]
    HydraJobs["Hydra jobs"]

    Node2140["projects/2140-node"]
    BitcoinCore["projects/bitcoin-core"]
    Btcd["projects/btcd"]
    Libbitcoin["projects/libbitcoin"]

    Ironworks --> Adapter
    Adapter --> Packages
    Adapter --> Checks
    Adapter --> HydraJobs

    Node2140 --> Adapter
    BitcoinCore -. "future" .-> Adapter
    Btcd -. "future" .-> Adapter
    Libbitcoin -. "future" .-> Adapter

Loading

I've only done this with the 2140-node so far, but we already have nix packaging for bitcoin core and libbitcoin and future adapters can map the same stage model onto different implementations:

projects/bitcoin-core/default.nix
projects/btcd/default.nix
projects/libbitcoin/default.nix

Each adapter defines how that implementation builds, tests, smokes, fuzzes,
benchmarks, and releases. Release here should be seen more as "promote to running in prod," not as "we manage release for other projects." I decided , however, to keep the name because I'd like this project to be usable for others to actually run their own CI/CD and release pipelines.

infra

PRs run spark in GitHub Actions. The source checkout is passed into Ironworks as a Nix input override, so the source repo does not need to own the full CI environment. Worth mentioning that, because this is nix, it would be trivial to run the actions on self hosted hardware, or to bypass github completely and orchestrate your own runners. Reviewed PRs that pass spark can be explicitly promoted to staging. The most important thing at this stage is conceptual review, so this should be a fast promotion, especially if the design of the change has already been discussed.

forge runs integration CI against the staging branch, ideally through Hydra. This catches interaction bugs between recently accepted changes. This would be on a per merge (promotion) to staging, but could also be scheduled.

harden jobs are scheduled, not per-merge. This is where deterministic IBD runs, fuzz corpus runs, long benchmarks, and previous-release compatibility belong. Stress testing could also go here.

Benchmarks should be Hydra-adjacent (i.e., not scheduled by hydra or managed as hydra jobs):

• hydra builds
• benchkit runs measurements on dedicated hardware
• results go to a database, artifact store, and dashboard, etc

Releases are cut from known-good staging commits. A release candidate passes temper, then stamp publishes tags, manifests, and artifacts.

why

This keeps PR feedback fast and gives better testing than individual , per PR tests do. It also separates responsibilities:

• source repos own source-level concerns
• Ironworks owns production build/release infrastructure
• project adapters capture implementation specifics
• hydra builds with dedicated hardware
• scheduled jobs watch for IBD regressions, capture more interactions in testing
• benchmarks focus on reproducibility

I stubbed out the basic repo that runs the CI and sets up hydra jobs, have yet to build out the hydra server and the remaining bits of orchestration but wanted to throw this up in case folks have thoughts on the design. I am very proud of the name, so pls no feedback on that 🥹

EDIT: fixed broken links, thanks @nymius !

[nymius]

Alright, I've been hacking away at a design that I'm pretty happy with. Inspired by what I did with https://github.com/champy and https://github.com/champix. The basic idea is the source repository should keep owning implementation details, such as:

The links above are pointing somewhere else:
*https://github.com/josibake/champy *https://github.com/josibake/champix

[josibake]

Fixed, thanks!