[P1][L] Automate permanent RNode/LoRa HIL from existing field evidence

[brothercorvo]

Objective

Turn the existing physical-carrier and product HIL evidence into a permanent, repeatable HIL capability, inspired by Leviculum's scheduled multi-RNode hardware testing.

This is not the first proof that LXMF-rs works on physical radio hardware. REM 1.4 already demonstrated bidirectional LXMF over two physical RNode/LoRa links on POCO + Pixel at 915 MHz using LXMF-rs revision 56ea9f06c474426b2245739e9cb5e2c325cdb1e2. Earlier REM releases also retained real two-phone TCP evidence, and LXMF-rs history contains Sideband, Columba, and RCH application interoperability evidence. See docs/status/real-world-interoperability.md.

The gap addressed here is automation and repeatability: the project does not yet have a permanently connected unattended hardware rack that reruns the declared physical matrix and retains release-candidate artifacts.

This issue is an automation extension of #616, not a duplicate operational-acceptance issue and not a requirement to run radios on every PR.

Initial HIL topology

Start with a small owned test rack that can remain connected to a dedicated runner:

• 2 physical RNodes as the minimum always-available topology
• support expansion to 3–4 nodes for multi-hop/contention scenarios
• serial RNode first; add supported TCP/Wi-Fi/BLE rows where hardware permits
• record device model, MCU, firmware, frequency/bandwidth/SF/CR and runner OS

Do not hard-code one vendor or board into the harness.

Automated scenarios

At minimum:

• discovery/announce
• bidirectional packet delivery
• proofs/receipts
• link establishment and teardown
• LXMF message exchange
• small and larger payload transfer
• Resource transfer
• reconnect/recovery after interface restart
• multi-hop when >=3 nodes are available
• contention/concurrent traffic when >=3 nodes are available
• bounded soak test
• configurable LoRa profiles, including the bandwidths supported by the declared test hardware

Runner architecture

Use a dedicated self-hosted runner or equivalent authorized host. Integrate through existing cargo xtask hil and prepared-host tooling.

The runner must:

• detect attached devices before testing
• identify devices deterministically
• prevent concurrent jobs from fighting over radios
• reset/recover interfaces without destructive firmware operations
• distinguish infrastructure failure from protocol failure
• redact secrets/private traffic
• upload machine-readable results and logs

Scheduling

• lightweight hardware smoke test on a controlled schedule
• fuller multi-node suite nightly or several times per week
• release-candidate HIL evidence for the exact candidate
• manual dispatch for diagnosis
• no HIL requirement on ordinary PRs

Acceptance criteria

• A permanently connected two-RNode topology runs unattended.
• Hardware absence/disconnection reports BLOCKED/INFRA FAILURE, never PASS.
• Rust↔Rust and Python↔Rust physical exchanges are represented where applicable.
• >=3-node topology automatically enables multi-hop/contention tests when available.
• Test manifests capture exact software, firmware, device and RF configuration.
• A failed scenario can be reproduced locally from the emitted command/configuration.
• Results are retained as artifacts and consumed by [P1][XL] Complete physical-carrier, platform, client and soak evidence matrix #616 operational evidence.
• Runner locking prevents simultaneous jobs from corrupting results.
• No unattended flashing, erase or destructive hardware management is introduced.

Relationship

Parent/umbrella: #605
Operational acceptance: #616
Software conformance: #615

Inspiration

Leviculum demonstrates the value of scheduled physical LoRa testing with multiple RNodes. Adopt the same engineering principle while using LXMF-rs's existing HIL/xtask infrastructure and supported hardware matrix.