Feature request: Boot loop detection

[mattzzw]

In solar-powered mesh networks like Meshcore, the transition between "low battery" and "operational" states can be treacherous. When a solar panel provides just enough current to start the processor but not enough to sustain the surge of a LoRa transmission or a full boot sequence, the device undergoes a brownout. This creates a cycle where the device boots, attempts to communicate, crashes due to voltage drop, and restarts immediately.

The Problem: Flooding the Mesh

By default, Meshcore repeaters are designed to announce their presence to the network. If a repeater is caught in a rapid boot loop, it may successfully reach the stage of sending a flood advert (broadcasted to all nodes) before it crashes.

• Airtime Saturation: A looping node can send dozens of high-priority flood packets per minute.
• Cascading Congestion: Since other repeaters are programmed to repeat these adverts, a single unstable node can trigger a "broadcast storm," effectively silencing legitimate traffic across the entire mesh.
• Battery Drain on Neighbors: Healthy nodes waste power receiving and re-transmitting the garbage packets from the looping node.

---

Proposal: Boot Loop Mitigation for Meshcore

This proposal introduces a persistent Boot Counter and a Cooldown Timer to distinguish between a healthy reboot and a power-instability loop. This would fix #1091.

1. Mechanism Design

We utilize a small segment of Non-Volatile Storage (NVM) or EEPROM to track the boot state.

• The Boot Counter: Incremented immediately upon CPU initialization.
• The Stability Threshold: Set to 10 minutes. If the device stays alive longer than this, the power is considered "stable."
• The Limit: Set to 3-10 boots. (tbd)

2. Logic Flow

1. On Boot: * Read from NVM.

• Increment and write back to NVM.
• Start a timer for (10 minutes).

2. Check Condition: * If :

• Action: Set flood.advert.interval to 0 (Disabled).
• Action: Potentially do the same for zero hop advert, set set advert.interval to 0
• Action: (Optional) Enter "Low Power Mode" or increase RX-only time to allow the battery to charge.

3. On Timer Expiry (Stable Operation):

• If the device reaches 10 minutes without crashing, reset in NVM.
• Restore the original advert.interval if it was previously suppressed.

3. Pseudo-Code Implementation

// Define storage and constants
#define BOOT_LIMIT 10
#define STABILITY_TIME_MS 600000 // 10 minutes

void setup() {
    int boot_count = nvm_read_boot_count();
    boot_count++;
    nvm_write_boot_count(boot_count);

    if (boot_count > BOOT_LIMIT) {
        // Suppress flood adverts to protect the mesh
        meshcore_set_flood_advert_interval(0); 
        serial_println("Boot loop detected! Adverts disabled.");
    }

    // Schedule the reset of the counter
    timer_run_after(STABILITY_TIME_MS, reset_boot_counter);
}

void reset_boot_counter() {
    nvm_write_boot_count(0);
    // Optionally restore default advert interval here
}

Impact Assessment

By implementing this "circuit breaker," a failing solar node will only "hit" the mesh 10 times before being silenced. This allows the node to potentially recover once the sun provides enough peak current to sustain the system for more than 10 minutes, at which point it will automatically resume its role in the network.

[oltaco]

It's very dangerous (at least on NRF52) to be attempting to write to the flash in a brownout situation, it basically would guarantee you end up with filesystem corruption and a non-booting node. There's better ways to achieve this using non-initialised RAM or registers that survive though and I've put together something that can track resets and state at reset time as well, maybe it's time to dust that off.

Brownout detection is hopefully not far off, @entr0p1 has been working on this and maybe we could combine the two and get something really useful going for NRF52 based nodes.

[luigi311]

I agree with @oltaco i think a better solution to this is to actually add a check on the startup prior to lora startup to check the battery voltage/percentage and if it does not meet a threshold sleep for X amount of time and try again. The big thing that causes the brownout is the surge from lora system kicking on so if we can put that off until a threshold is met then the chances of a brownout should be reduced. This method would not require any writing as it will just be read and condition checking.

[IoTThinks]

For now, the flood advert is only sent at 18th seconds from boot.
I believe that flood advert will not be sent in case of boot loop as boot loop likely resets the board in 1 second.

May be changing to 30th or 60th second to send the flood advert will mitigate the problem?

[luigi311]

hmm yeah i guess it would depend on how fast its browning out. With that 18th second from boot before the flood currently in place then if it is crashing prior to 18 seconds then it doesnt seem like its related. If it is crashing within a second or so then it would have to be in relation to powering up some of the components. If it is this then id rather not adjust the flood advert further down as it wont really make a difference depending on which of the two instances is happening.

[mattzzw]

The boot loops we observed in our mesh from a supposedly dying repeater were resulting in flood advert intervals of about 2 minutes. This went on for days until the repeater either finally died or was switched off.

[timdj]

In my opinion we should:
1 For repeaters that reset the clock

• don't send an immediate advert until clock is set 'Y = default_year'. If someone is setting the repeater up the admin will do a time sync. After time sync we could trigger an advert. If not we could still send an automatic advert after X minutes.

2 For repeaters that retain the clock

• If clock is set 'Y > default_year' we we can probably wait a few minutes before sending an advert. Since clock is set these have probably already been part of the network.

Also wouldn't it be beter to do a zero hop advert. Flood adverts will be send by set interval anyway. That would prevent traffic as well if there are issues with booting or an admin is trying out different settings / firmware

[entr0p1]

FYI the PR is now in as a draft. I had to clean up some embarrassingly sloppy code from the original implementation I had, so it just needs to be tested again before I'm happy to mark it ready for review.

If you're interested in testing, here is the PR: #1413