Skip to content

Fix bootloop caused by orphaned pool workers and solis debug logs - #3718

Merged
springfall2008 merged 3 commits into
mainfrom
copilot/fix-bootloop-issue-8-35-2
Apr 2, 2026
Merged

springfall2008 merged 3 commits into
mainfrom
copilot/fix-bootloop-issue-8-35-2

Conversation

Copilot AI commented Apr 2, 2026 •

Copy link
Copy Markdown
Contributor

After the v8.35.2 solis_fix PR, a bootloop was observed where AppDaemon restarts predbat after each prediction run, leaving multiprocessing pool workers orphaned. Additionally, debug log statements were accidentally left in production code.

Changes

solis.py — Remove debug logs

  • Removed 4 # Debug log statements accidentally committed in read_cid, read_batch, and write_cid that logged full API payloads and responses on every call
  • Fixed misleading field_description in startup_reset_registers: the third read_and_write_cid call restores the original mode but was labelled identically to the first call that sets the new mode

predbat.py — Robust pool cleanup

Added cleanup_pool() helper that safely terminates and joins any active multiprocessing.Pool:

def cleanup_pool(self):
    if getattr(self, "pool", None):
        try:
            self.pool.terminate()
            self.pool.join()
        except Exception as e:
            self.log("Warn: Failed to terminate thread pool: {}".format(e))
        self.pool = None

Called in three places to prevent orphaned worker processes:

  • reset() — terminates any pool left over from a prior crashed run when the app restarts
  • update_time_loop / run_time_loop finally blocks — terminates the pool when an exception propagates out of update_pred before calculate_plan's normal cleanup (pool.close() + pool.join()) is reached

Previously, an exception inside the prediction loop would leave self.pool pointing to live worker processes. On the next AppDaemon restart reset() would silently overwrite self.pool = None, orphaning those workers. On constrained hardware this could exhaust resources and trigger another kill, creating the bootloop.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • api.octopus.energy
    • Triggering command: /home/REDACTED/work/batpred/batpred/coverage/venv/bin/python3 python3 ../apps/predbat/unit_test.py --quick (dns block)
    • Triggering command: /home/REDACTED/work/batpred/batpred/coverage/venv/bin/python3 python3 ../apps/predbat/unit_test.py --quick git conf�� get --global rgo/bin/git HooksPath (dns block)
    • Triggering command: /home/REDACTED/work/batpred/batpred/coverage/venv/bin/python3 python3 ../apps/predbat/unit_test.py --quick git conf�� get --global git core.hooksPath (dns block)
  • gitlab.com
    • Triggering command: /usr/lib/git-core/git-remote-https /usr/lib/git-core/git-remote-https origin REDACTED (dns block)
  • https://api.github.com/repos/springfall2008/batpred/contents/apps/predbat
    • Triggering command: /home/REDACTED/work/batpred/batpred/coverage/venv/bin/python3 python3 ../apps/predbat/unit_test.py --quick (http block)
    • Triggering command: /home/REDACTED/work/batpred/batpred/coverage/venv/bin/python3 python3 ../apps/predbat/unit_test.py --quick git conf�� get --global rgo/bin/git HooksPath (http block)
    • Triggering command: /home/REDACTED/work/batpred/batpred/coverage/venv/bin/python3 python3 ../apps/predbat/unit_test.py --quick git conf�� get --global git core.hooksPath (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue Apr 2, 2026 that may be closed by this pull request
…ception

- Remove accidental debug log lines from solis.py read_cid, read_batch, and
  write_cid methods that were left in from the solis_fix PR (#3716)
- Fix misleading description in startup_reset_registers third API call
- Add cleanup_pool() helper method to PredBat to properly terminate
  multiprocessing pool workers
- Call cleanup_pool() in reset() to clean up any orphaned pool from a
  previous failed run on restart
- Call cleanup_pool() in update_time_loop and run_time_loop finally blocks
  to ensure pool workers are terminated when an exception propagates out
  of the prediction loop (prevents orphaned worker processes that could
  contribute to the bootloop)

Agent-Logs-Url: https://github.com/springfall2008/batpred/sessions/67d7d029-d664-4901-979c-e2b278b7a45b

Co-authored-by: springfall2008 <48591903+springfall2008@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix bootloop issue after updating to v8.35.2 Fix bootloop caused by orphaned pool workers and solis debug logs Apr 2, 2026
Copilot AI requested a review from springfall2008 April 2, 2026 07:31
@springfall2008
springfall2008 marked this pull request as ready for review April 2, 2026 11:17
Copilot AI review requested due to automatic review settings April 2, 2026 11:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes an AppDaemon bootloop/resource exhaustion scenario by ensuring multiprocessing.Pool workers are force-cleaned up when the prediction loop exits unexpectedly, and removes accidentally committed Solis API debug logging (plus a small log-description correction).

Changes:

  • Removed verbose Solis debug logs that printed full request/response payloads on every CID read/write.
  • Added a cleanup_pool() helper to terminate/join an active multiprocessing.Pool and invoked it during reset and after prediction-loop runs.
  • Corrected startup_reset_registers’s field_description to reflect restoring the prior mode.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
apps/predbat/solis.py Removes production debug logging of Solis API payloads/responses and fixes a misleading field_description string.
apps/predbat/predbat.py Adds centralized pool cleanup via cleanup_pool() and calls it from reset() and prediction-loop finally blocks to prevent orphaned workers.

Comment thread apps/predbat/predbat.py
Comment on lines +394 to +398
def cleanup_pool(self):
"""
Terminate and clean up the multiprocessing pool if it is active.

Ensures worker processes are properly terminated to prevent orphaned

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New pool-cleanup behavior is critical to preventing the bootloop, but there doesn’t appear to be any unit test coverage exercising it (e.g., simulating an exception during prediction with a mocked Pool and asserting terminate/join are invoked and self.pool is cleared). Adding a lightweight test that patches the Pool object would help prevent regressions without needing to spawn real worker processes.

Copilot generated this review using guidance from repository custom instructions.
Comment thread apps/predbat/predbat.py Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@springfall2008
springfall2008 merged commit 08da5d9 into main Apr 2, 2026
1 check passed
@springfall2008
springfall2008 deleted the copilot/fix-bootloop-issue-8-35-2 branch April 2, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8.35.2 - Bootloop

3 participants