Skip to content

CONFORMANCE.md

Latest commit

 

History

History
542 lines (445 loc) · 30 KB

CONFORMANCE.md

File metadata and controls

542 lines (445 loc) · 30 KB
Raw

CONFORMANCE: testing a node, and what a test can prove

This preamble is background — it is about this document. Three levels follow, each cheap and each proving something the one before it does not, and then a section on what none of them proves, which matters more than it sounds: a suite that claims more than it establishes teaches wrong code onward.

Labels as elsewhere — RULE, BEHAVIOUR, RECOMMENDATION — and each one governs the statement it opens, including any list or table that continues it, until the next label or the next heading. The paragraph you are reading now is background, and that is the convention working: it describes this document rather than the platform. Two other kinds of paragraph here are background for the same reason and carry no label: the ones headed What ... proves / What it does not prove and the ones saying how a list here was produced or how complete it is, which are about the method and the evidence rather than about the platform; and the captions that introduce a fixture or a command. See README.md.

Level 1: run it with a hand-written envelope

RECOMMENDATION, for this whole section. No orchestrator, no network, no docker: build the fixture, write the two documents by hand, and run your program. Everything below is literal — copy it and run it.

First the input file — this paragraph is background about the fixture. It is twelve bytes, and the hash below is its real sha256; check it yourself with the last line.

mkdir -p /tmp/job/in /tmp/job/out/attempts/1/gen-1
printf 'a,b\n1,2\n3,4\n' > /tmp/job/in/rows.csv

wc -c /tmp/job/in/rows.csv     # -> 12
sha256sum /tmp/job/in/rows.csv # -> b9485148546419a0f6a85e8d708c923557c15d7f3c7d078ef1fa7f7c0f57d5a5

/tmp/job/creds.json:

{
  "schema_version": 1,
  "scheme": "local",
  "credentials_file": "/tmp/job/creds.json",
  "expires_at": "2099-01-01T00:00:00+00:00",
  "manifest_path": "/tmp/job/invocation.json",
  "inputs": [
    {"port": "input", "relpath": "rows.csv", "name": "rows.csv",
     "sha256": "b9485148546419a0f6a85e8d708c923557c15d7f3c7d078ef1fa7f7c0f57d5a5", "size": 12,
     "local_path": "/tmp/job/in/rows.csv"}
  ],
  "staging": {"mode": "local_path", "path": "/tmp/job/out/attempts/1/gen-1"}
}

/tmp/job/invocation.json:

{
  "schema_version": 1,
  "execution_id": 1, "pipeline_id": 1, "deployment_id": 1, "revision_id": 1,
  "image_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
  "params": {},
  "env_names": [],
  "inputs": [
    {"name": "input", "payload_kind": "csv", "layout": "file", "cardinality": "many",
     "objects": [{"uri": "file:///tmp/job/in/rows.csv",
                  "sha256": "b9485148546419a0f6a85e8d708c923557c15d7f3c7d078ef1fa7f7c0f57d5a5",
                  "size": 12, "relpath": "rows.csv"}]}
  ],
  "staging_prefix": "file:///tmp/job/out/attempts/1/gen-1",
  "credentials_file": "/tmp/job/creds.json",
  "timeout_seconds": 900,
  "idempotency_key": "execution-1-attempt-1",
  "attempt": 1,
  "generation": 1
}

RECOMMENDATION, and it is the whole point of splitting this in two. Then two separate runs, from the root of this repository. They are not variants of one command and neither substitutes for the other: the first shows a complete program working end to end, and the second is the one that actually tests whether your node reads its credentials from the right place. Run both.

Run A — the whole cycle, against this repository's node

LSPO_CREDENTIALS_FILE=/tmp/job/creds.json LSPO_CREDENTIALS=/tmp/job/creds.json \
LSPO_EXECUTION_ID=1 LSPO_ATTEMPT=1 LSPO_GENERATION=1 \
LSPO_IDEMPOTENCY_KEY=execution-1-attempt-1 LSPO_CONTRACT_VERSION=1 \
LSPO_STAGING_PREFIX=file:///tmp/job/out/attempts/1/gen-1 \
LSPO_INVOCATION_URI=file:///tmp/job/invocation.json \
python3 node.py

BEHAVIOUR, with one RECOMMENDATION inside it: why two credentials variables, when the platform only sets one. The agent sets LSPO_CREDENTIALS_FILE and nothing else (PROTOCOL.md), and that is the one your node should read. This run sets the obsolete LSPO_CREDENTIALS beside it deliberately, for one reason: it is the arrangement in which a node reading the wrong name still works, which is the whole point run A is here to make. The node.py in this repository reads LSPO_CREDENTIALS_FILE, falls back to the obsolete name only when that is the only one set — older images of this repository baked it, and some are still in the field — and prefers the one the agent sets when the two disagree. Your own node should read LSPO_CREDENTIALS_FILE and nothing else, and then it needs only the first of the two.

This is what run A prints, verbatim. The two @lspo:progress lines are the progress protocol (PROTOCOL.md) — a run that emits none is indistinguishable from a stuck one:

INFO execution 1 attempt 1
@lspo:progress {"fraction": 0.0, "phase": "copying inputs"}
@lspo:progress {"fraction": 1.0, "phase": "done"}
INFO done — 1 file(s), 12 bytes

it exits 0, and it leaves exactly three files behind:

/tmp/job/out/attempts/1/gen-1/outputs/rows.csv
/tmp/job/out/attempts/1/gen-1/result.json
/tmp/job/out/attempts/1/gen-1/__lspo_complete.json

What run A proves, and it is less than it looks. That a program in the right shape can complete the cycle against these two fixtures. It proves nothing whatever about which environment variable a node reads, because it sets both of them — a node that reads only the obsolete name passes it just as happily as a correct one. A green run here is compatibility evidence, not conformance evidence, and it is split out for exactly that reason: a check that cannot fail for the thing it is named after is worse than no check, because it reports success.

Run B — the check that the credentials variable is right

Delete the staging directory, recreate it empty, then run with only the real name:

rm -rf /tmp/job/out/attempts/1/gen-1 && mkdir -p /tmp/job/out/attempts/1/gen-1

env -u LSPO_CREDENTIALS \
LSPO_CREDENTIALS_FILE=/tmp/job/creds.json \
LSPO_EXECUTION_ID=1 LSPO_ATTEMPT=1 LSPO_GENERATION=1 \
LSPO_IDEMPOTENCY_KEY=execution-1-attempt-1 LSPO_CONTRACT_VERSION=1 \
LSPO_STAGING_PREFIX=file:///tmp/job/out/attempts/1/gen-1 \
LSPO_INVOCATION_URI=file:///tmp/job/invocation.json \
python3 node.py

RECOMMENDATION. This is the run to put in your own suite. A correct node produces the same output and the same three files as run A. Assert the exit code, the three filenames, and that the staging directory is otherwise empty.

BEHAVIOUR of this repository's example, with one RECOMMENDATION at the end. node.py passes run B: the same four stdout lines and the same three files as run A. It did not always. Until it was rewritten it read the obsolete variable and failed here, exiting 1 with the staging directory empty — no outputs, no result.json, no marker — and the failure arrived as an uncaught traceback, because the bootstrap sat outside the try. Both mistakes, and eighteen others, are recorded with what each one cost in CONFORMANCE-BASELINE.md. If your own node fails run B, do not treat it as a broken fixture; treat it as the check doing its job.

The second of those two is still worth reproducing, because it is about how a node reports a failure it cannot recover from. Point LSPO_CREDENTIALS_FILE at a file that is not there and run again: before credentials exist there is nowhere to write a marker, so the only honest ending is one line on stderr and an exit code, never a stack trace an operator has to interpret. This is verbatim what happens now:

hello-node: could not start: no credentials file at the path LSPO_CREDENTIALS_FILE names; the agent did not mount one, or this job is no longer its

It exits 1 and leaves the staging directory empty. Assert that shape in your own suite: a bootstrap failure that escapes as a traceback tells the operator nothing about what to fix.

timeout_seconds is 900 in the fixture because that is what a registered node really gets; see PROTOCOL.md. Nothing in either offline run enforces it — it is there so the number your code reads offline is the number it will read in production.

RULE, worth knowing before you hand-write a manifest. staging_prefix must end with attempts/<attempt>/gen-<generation> matching the attempt and generation fields, or the document is invalid. A single trailing slash is tolerated, since the check strips trailing slashes before comparing. This is the first thing that rejects a hand-made manifest, and it is not arbitrary: that suffix is the fence keeping a superseded runner out of the live attempt's area (external/contract.py).

The offline examples describe this repository's reference node, which copies inputs. They are not an output-shape requirement for a node that transforms its inputs. The historical parser/refusal measurements are described in README.md. The original reference-node defect measurements and the current audit's verification are recorded separately in CONFORMANCE-BASELINE.md. No production run or real S3 upload was performed for this update. Both documented commands (runs A and B) and the missing-envelope case were rerun on 2026-10-13 at hello-node branch commit 0f19d5d (pull request #12 on top of main at a9ca5b8), reproducing the transcripts, exit codes and output files shown here. Their manifest, marker and result.json also parsed with the platform's own parser at 86461801.

What level 1 proves. From run B: that your program finds its credentials through the one variable the platform sets. From either run: that it parses both documents, verifies input pins, writes objects, and writes a marker with a consistent inventory. Neither command alone proves every validation branch: changed input bytes, missing pins and malformed documents need separate negative cases in the harness.

What it does not prove. Anything to do with expiry, cancellation, object-store refusals, or the collection side.

Validating your own marker

RECOMMENDATION. You do not have the orchestrator's parser, so write a short checker and run it over your marker in the test. Everything in the two lists below is a RULE — each one is a refusal the real parser makes, so a marker breaking any of them fails your run. That is why these are worth reproducing in a test at all; the recommendations elsewhere in these documents are not checkable this way.

How this list was produced, because it matters for how much you should trust it. It was not written from memory. A harness read the parser's own model definition, enumerated every field it declares and every cross-field check it runs, and then fed it about ninety mutated markers one at a time, recording which the parser accepted and which it refused. Every field and every check came back with at least one exercised refusal, and nothing in the model was left without one. See README.md for the commit and for what "exercised" means here.

RULE, for every item in the list that follows. Each one is a refusal the marker parser really makes (external/contract.py), so a marker breaking any of them fails your run. RECOMMENDATION, separately and for the whole list: reproduce them in your own checker, since you do not have that parser.

The document as a whole

  • a payload that is not a JSON object at all — an array, a bare string, null;
  • a schema_version that is not a real integer, in particular JSON true (which in Python compares equal to 1 and would otherwise be read as version 1), the string "1", or 1.0;
  • a schema_version that is a whole number this contract has no parser for, such as 2;
  • a document over 8 MiB, refused by the reader before it is parsed at all.

Fields that must be there

  • a missing execution_id, attempt, generation or status. All four are required and a document without any one of them is invalid.

Numbers

  • execution_id, attempt or generation that is null, a string such as "1", JSON true, a float such as 1.0, zero, or negative — they are strict integers of at least 1;
  • an object size that is null, a string, a boolean, a float, or negative. Zero is accepted: an empty file is a real file;
  • an exit_code that is a string, a boolean or a float. It is optional, and a negative value is accepted deliberately — that is how a signal death is reported.

Text

  • a status that is anything other than exactly succeeded, failed or cancelled. "SUCCEEDED" is refused: the match is case-sensitive;
  • an idempotency_key that is empty, whitespace-only, or not a string. It is optional and may be absent or null;
  • an error that is not a string — a number, an object, a list. It is optional. Note that control characters inside it are cleaned, not refused (see the second list);
  • a sha256 that is not exactly 64 lowercase hexadecimal characters: no sha256: prefix, no uppercase, no trailing newline, not 63 or 65 characters, no non-hex letters, and not a non-string.

Paths and port names

  • a relpath that is not a string, or is empty, absolute, contains a backslash, contains any control character in U+0000–U+001F — tab, newline, NUL and escape included — has a . or .. component, or has an empty component such as a//b or a trailing slash;
  • the same relpath rules applied to the relpaths listed under produced_ports, which are validated in their own right and not merely looked up in the inventory;
  • an output port name that is empty, whitespace-only, contains a control character, or is not a string.

Shape

  • an objects that is not a list (null, or an object keyed by relpath), or a list whose entries are not objects (a bare string, null);
  • a produced_ports that is not an object (a list), or a port whose value is not a list of relpaths (a bare string, null, an object);
  • an object entry missing its relpath, its sha256 or its size.

The three cross-field checks

  • a duplicate relpath in objects — one file, one entry;
  • a relpath in produced_ports that is absent from objects;
  • a relpath repeated within one port.

BEHAVIOUR, for the whole list below: the platform ACCEPTS every one of these. That is a different kind of fact from the list above — those name a check that refuses you, these name the absence of one — and it is why this list does not carry the RULE label even though it sits beside one. RECOMMENDATION, for the same list: do not let your own checker refuse them, because a checker stricter than the platform sends you rewriting correct code.

  • a negative exit_code, and an absent or null one;
  • an object whose size is 0;
  • a field the checker has never heard of, anywhere in the document. Unknown fields are ignored by design, so that adding one is not a breaking change;
  • control characters inside error — they are stripped out as the document is parsed, and the rest of your sentence survives. A checker that refuses a marker over its error text is stricter than the platform;
  • the same relpath under two different ports. Only repeating it within one port is refused;
  • a port whose relpath list is empty;
  • a failed or cancelled marker that still declares produced_ports.

BEHAVIOUR, and it is the one surprise in the set. A port name is checked but not trimmed. " output " with its spaces is accepted exactly as written and becomes the artifact kind downstream steps have to match, spaces and all. Trim your own port names.

How complete this is, stated exactly. Every field the marker parser declares and every cross-field check it runs is represented above by at least one refusal that was actually executed against it, and the field list was read off the parser rather than typed from memory — so no field or check is missing. That is a different claim from "every value that would ever be refused is named here", which no list can make. The parser is the authority. If your checker passes and collection still refuses your marker, that disagreement is a defect in this list and is worth reporting.

Level 2: a fake object-store endpoint

RECOMMENDATION, for this whole section. Run your node against a small HTTP server that stands in for the storage service: it serves the inputs and the job description on presigned-looking GET URLs, and accepts form POSTs to a fake upload endpoint.

RECOMMENDATION. Have the fake refuse a POST whose key does not start with the declared key_prefix, and one whose body exceeds the per-object ceiling. Those are the two conditions the real policy carries, and a node that gets them wrong fails in production and passes against a permissive fake.

Testing credential rotation without a clock

RECOMMENDATION. Model generations plus a fake clock, not wall-clock time. A time-based test is either slow or flaky.

RECOMMENDATION, and do not confuse rotation with revocation, or the test fails a correct node. Installing a new envelope revokes nothing: a presigned URL from envelope A stays valid until its own expires_at, whatever happens to the file it arrived in (PROTOCOL.md). A fake that starts refusing A's signatures the moment B is written is modelling a platform we do not have, and it fails exactly the nodes that are right: caching A until A says it is stale is correct behaviour, explicitly permitted, and arguably the better implementation.

So the fake needs a clock it controls, and A has to actually expire before it is refused:

  1. The harness owns a clock the node reads — an injected time source, a monkeypatched time.time, or a faketime-style shim. Give envelope A an expires_at a few simulated minutes ahead.
  2. Start the node. Let it use A for its inputs. The fake accepts A's signatures, because they are valid.
  3. Atomically replace creds.json with envelope B, whose expires_at is later. The fake keeps accepting A: nothing has expired yet.
  4. Advance the fake clock past A's expires_at. Now the fake refuses A's signatures, with whatever the storage service really returns for an expired policy, and accepts B's.
  5. A node that cached A and never looks again fails deterministically at its first upload. A node that re-reads at or near expires_at picks up B and succeeds. A node that cached A but watches its expiry also succeeds — and must, because that is a correct implementation.

Then rotate B to C, again with an expiry the harness steps past, after the first output upload. That separates "reloads once, after processing" from "reloads whenever the envelope says it is stale", which is the property that actually matters at the end of a long run.

RECOMMENDATION. Have the fake refuse an expired signature the way the storage service does — an HTTP error, not a Python exception of your own invention — and assert that your node recognises it. A node that treats "expired" as an unknown transport error will retry blindly instead of re-reading its credentials.

RECOMMENDATION, and it is a property of the test harness rather than of your node. Mount the directory, not the file, if you run this inside docker. A bind-mounted file keeps pointing at the replaced inode and will never appear to change, so a harness that mounts the file will report every node as broken. That is exactly why the platform mounts the directory (agent/creds.py).

Testing the stop path

RECOMMENDATION. Send SIGTERM during a transfer, not only between work units. A cooperative flag cannot be checked while the process is blocked in a socket read, and that is the case a real stop hits. Assert three things: a cancelled marker exists, it inventories the objects that were already uploaded, and the process exited within a few seconds. Pick that threshold yourself and keep it small: there is no grace period to measure it against, because no interval on any stop path is guaranteed (PROTOCOL.md). A node that needs the whole of a stop's notional grace is a node that gets nothing done on a stop that gives it none.

RECOMMENDATION, and it is the one this repository learned the hard way. Make the store in your test hold a response open longer than the stop is nominally given, and stop the container during that. A store that eventually answers lets a step which merely waited look exactly like a step that stopped — every timing assertion passes, for the wrong reason. Held open past the grace, the two are no longer confusable, and it is the shape that caught node.py handling a stop correctly in every respect except noticing it: 12.2 seconds to go, because its handler closed the wrong object. See tests/test_cancellation.py for both halves — the timing, and the receipt — and AUTHORING.md for what a handler has to do to pass them.

RECOMMENDATION, and it is worth more than the rest of this section. Test every wait your step can be stopped inside, not the one you thought of first. There are four — getting a connection (DNS, TCP, TLS), waiting for the store to begin answering, reading the body, and waiting for an upload to be acknowledged — and they are reached through different objects, so a node can be interruptible in one and not the others. This repository fixed the second, shipped it, and still paid the full 25-second timeout on the first and the third. Two of the four need no store at all to test: a listener that accepts a connection and then says nothing stalls a TLS handshake (no certificate is involved — a handshake stalls before any certificate is offered), and the same listener sending response headers that promise more body than it delivers stalls a read mid-body. Roughly forty lines, no new dependency; conformance/stalling.py is the whole of it.

BEHAVIOUR, and it caps what any of these tests can prove. A local SIGTERM followed by a kill after N seconds is your harness's number, not the platform's. The agent controls stopping, and a fence grants no grace. The envelope's expires_at is limited by its own lifetime and the reporting-window ceiling; near the end it can reflect that ceiling, with signing conservatively understating the expiry. It does not identify the workload kill cutoff, which precedes the reporting cutoff by the configured margin (60 seconds by default). Earlier envelopes can expire well before either cutoff and be refreshed. The heartbeat's stop instruction is not forwarded to the container, and the envelope is not a stop-timing contract. A timing test therefore proves your node bounds ITSELF; it cannot prove the platform will leave that much time (runners/credentials.py, runners/jobs.py, agent/runner.py).

RECOMMENDATION. Assert that a marker exists even when your step had produced nothing yet. That case is easy to leave untested and it is where the hole hides: a step that exits the instant it is signalled, writing nothing at all, satisfies every assertion about promptness and every assertion about not over-claiming. This suite had that hole for a release — a stop landing during the first download was measured for its speed and never for its account of itself.

BEHAVIOUR. A local SIGTERM tests whether this node stops and leaves a receipt. It does not test the deployed agent's lease, stop-window timing or collection. Runtime expiry now permits reporting within the configured window, so an accepted failure report can salvage a valid marker as diagnostics. Operator cancellation of running work still normally fences the container and does not collect its objects. See PROTOCOL.md for those separate paths.

BEHAVIOUR to keep in mind while reading the result. When the agent classifies at all, it calls the run cancelled regardless of your exit code, because it asks "was cancellation requested?" before it looks at the code — three other questions are asked ahead of even that one (was the job fenced, was the agent forced to exit, had the deadline passed), and any of them wins. So a test that only asserts "the run is cancelled" passes even for a node that ignores SIGTERM completely. Assert the marker and the timing.

Level 3: the real thing

RECOMMENDATION. Register the node, start an agent, run the pipeline. See OPERATIONS.md. A first run needs no upstream step: upload a test file to a Data Source node, wire that node straight into yours and start the run with Run node on your node (OPERATIONS.md).

What only level 3 proves. That your container can read the credentials a real agent wrote — which is not a question about your image's uid (any uid can open them) but about that machine: a state directory on a share that reports permissions without enforcing them, or an image that makes /lspo non-traversable, both fail here and nowhere else. Also that the agent's environment allowlist permits every variable your deployment declares; that your output ports arrive downstream as the artifact kinds you expected; that collection accepts your hashes.

RECOMMENDATION, and it takes one deliberate step to set up. Run at least one job whose work takes longer than fifteen minutes before you trust the node in production. That is the single threshold that separates a node which reloads its credentials from one which does not, and nothing shorter will reveal the difference.

BEHAVIOUR. Registration defaults the runtime budget to 900 seconds. A test that needs more than fifteen minutes of useful work must increase that budget first. The default stop window permits an expired job to report and retain diagnostics; it does not turn the stop period into more processing time. A lost lease or unavailable agent can still leave a job parked; see PROTOCOL.md.

RECOMMENDATION, and nothing in the platform requires it — it is a precondition of the test, not a rule about your node. Set timeout_seconds on the pipeline node (for example 5400) before you try it; that takes effect immediately, without publishing a new revision. See PROTOCOL.md for how the number is resolved and OPERATIONS.md for where to put it.

That step is not an inconvenience in the test, it is the test: raising the budget is exactly the change an operator makes the first time a real job needs more than a quarter of an hour, and it is the moment a node that reads its credentials once stops working.

What testing cannot prove

BEHAVIOUR — write order. Nothing can establish, after the fact, that you wrote the marker last. The platform does not try: it checks the consequences, by copying every object your marker names into a place your credentials cannot reach and re-reading it there (pipelines/external_finalize.py). A marker written too early shows up as a size or hash mismatch, or as a missing object, and never as "you wrote things in the wrong order". A local test can record the order your program wrote things in, which is useful, but it is a statement about your test double and not about the contract. This is why "write the marker last" is labelled a recommendation everywhere in these documents: it is the most important thing in the set that nothing enforces.

RECOMMENDATION — that a fake endpoint behaves like the real one. A fake is either stricter or laxer than the storage service, never identical. Stricter is the safer error, and it is still an error: a suite that refuses something the platform permits will send you refactoring correct code. Say which of the two yours is, in the harness, so a failure can be read.

RECOMMENDATION — that a green run means the contract is satisfied. Every test has a subject, and a suite that does not say which is which is unreadable. Three kinds:

  1. tests whose subject is your node, which you can make pass by editing your node;
  2. tests whose subject is the platform, the agent or the collector, which need an emulator or a fixture and can never be turned green by editing your node;
  3. tests whose subject is the harness itself.

Without those labels, an implementer cannot tell a real defect from a gap in the harness, and a green run proves nothing in particular.

RECOMMENDATION — anything a check validates against itself. Ask of every check: what independent thing does this compare against? Five real examples from this project's own history. Byte accounting measured with the encoder its own test hand-picked. A guard test that set only the obsolete environment variable it existed to catch, so it encoded the bug. A proposal to diff generated documentation against its own generator. A drift guard comparing a snapshot with a version file sitting beside it. And the fifth is on this page: the offline example above used to be one command that set both credentials variables while claiming to prove the node read the right one, so a node reading only the obsolete name passed the check that existed to catch it. All five are green forever and prove nothing.

RECOMMENDATION. Distrust a conformance claim you cannot trace to a specific check in the platform. A suite built from a prose description of this contract once reported 23 defects in the reference node, of which 6 were contract violations. The rest were the suite's author's preferences, presented as rules. That is the failure this document set is organised to avoid, and it is why every statement here carries a label.

If a conformance/ directory exists in this repository

This section is background, about the harness shipped beside these documents. Read its own README first. It will say which of the three subjects above each test has, and whether its fake endpoint is stricter or laxer than the storage service. A test in that suite that fails your node is a reason to read the check, not automatically a reason to change your node: check it against PROTOCOL.md, and if the two disagree, one of them is wrong and it is worth finding out which.