Export threshold can hide whole periods from the planner: slot visibility, not cost

[chalfontchubby]

Summary

When a high-priced export window enters the plan horizon — an Axle VPP event, and likely any alert-priced or saving-session slot — the export rate threshold is raised to that window's price. On a flat export tariff this excludes every ordinary export window from high_export_rates, so the optimiser has no export slot to consider for the rest of the horizon. The battery then sits full through peak PV and clips.

Reported as part of #4865 (PV clipping), but it is a separate mechanism with a much wider blast radius: it suppresses all rate-arbitrage exports, not just clipping-avoidance ones. Splitting it out so it doesn't get closed along with that issue.

Mechanism

set_rate_thresholds() picks a sensible threshold in automatic mode (fetch.py:2492-2496) — for a flat 20p export tariff with a 23.8p import minimum, that's rate_export_min + 0.5 = 20.5p.

fetch.py:1315-1319 then rescans and overwrites it with lowest, the lowest average among the windows that already cleared it:

self.high_export_rates, lowest, highest = self.rate_scan_window(self.rate_export, 5, self.rate_export_cost_threshold, True, alt_rates=self.rate_import)
if self.rate_high_threshold == 0 and lowest <= self.rate_export_max:
    self.rate_export_cost_threshold = lowest

With a 100p VPP window in the scan, lowest becomes the VPP price rather than the base tariff's. The threshold ratchets up and every 20p window falls below it.

I assume this overwrite is deliberate for Agile-style tariffs, where adapting to what's actually on offer is right. It goes wrong when a rare high-price window sits alongside a flat base rate — the threshold stops describing "a good export rate" and starts describing "the VPP price". Worth a maintainer's ruling on the intent before picking a fix.

Evidence

Two independent instances from the same reporter, three months apart, both in #4865.

1. 14 June 2026 — logs attached to #4036. The threshold flips mid-day exactly when the event is fetched:

05:45  High export rate found rates in range 20.0p to 20.0p
...
18:40  AxleAPI: Successfully fetched event data - export event 2026-06-15 06:00-07:00
18:45  Setting Axle VPP session ... pence_per_kwh 100
...
20:45  High export rate found rates in range 99p to 120.0p

Same day, same system, unchanged PV data — ordinary export windows vanish the moment the event enters the horizon. The reporter's comment at the time: "with an Axle event tomorrow, Predbat doesn't want to force export at all", with ~13 kWh of predicted clipping.

2. 4 September 2026 — predbat_debug.yaml on #4865, v8.54.3. Replayed against current main:

• rate_export_cost_threshold = 120.0, against a flat daytime export rate of 20.0p
• high_export_rates contains only the two Axle windows, at +1695 and +1725 minutes
• PV peak is at +20 minutes, with no candidate export window anywhere near it
• clipping 2.10 kWh in the incoming plan, rising to 3.26 kWh after calculate_plan()

Inserting an export window over the PV peak by hand makes no difference, because the only windows the optimiser can reach are 28 hours away.

The reporter guessed the cause both times — "this might be caused by the Axle event later in the day, but I don't know why that would affect it" — and was right.

Reproduction

Replay the 4 September file from #4865 against main and inspect rate_export_cost_threshold and high_export_rates. It makes a ready-made regression case.

Not yet explained

Clipping getting worse after recalculation on the September file (2.10 → 3.26 kWh) is a separate oddity I haven't accounted for. It may be a consequence of having no reachable export window, or it may be its own bug.

Scope

Anyone on a flat export tariff with occasional high-price windows: Axle VPP, Octopus saving sessions, alert-priced slots. The clipping in these reports is how it surfaced, but the same suppression would block ordinary rate-arbitrage exports too.

Filed by Claude on chalfontchubby's behalf, from analysis of the debug files and logs attached to #4865.

[springfall2008]

Automated first-pass triage — a maintainer will review before any action is taken.

Classification: bug (confirmed) · priority_medium. No duplicate found — the split out of #4865 stands. Workaround exists (manual export times, which the reporter used in June).

Independently verified against current main (v9.2.0-18-g130596bc) by replaying the 4 September debug file from #4865 through a temporary test probe that runs the exact fetch.py sequence — set_rate_thresholds() → the rate_scan_window() rescan → the threshold overwrite. The probe was temporary and has been removed; nothing was committed.

What reproduces:

• The dump (v8.54.3, minutes_now 915) stores rate_export_cost_threshold 120.0 and high_export_rates = only the two Axle windows at minutes 2610–2670 (+1695/+1725), average 120.0. PV peaks at minute 930 — 15 minutes after the snapshot — with no candidate export window anywhere near it. calculate_best_export and set_export_window are both on, so export_window_best (plan.py:1422) is exactly those event windows.
• Running the fresh sequence on the dump's raw rates: set_rate_thresholds() → 20.5p (the rate_export_min + 0.5 branch); the rescan then finds only the two event windows; the overwrite (fetch.py:1289-1293) latches the threshold at 99 — because rate_scan_window initialises lowest = 99 (fetch.py:2420) and the found windows' averages (120) never lower it. Every 20p window is now below threshold.

One correction to the issue's mechanism write-up: the ratchet value is that lowest = 99 sentinel, not the event price. The dump's 120.0 is a display-time copy — publish_rate_and_threshold() (output.py:3723) overwrites rate_export_cost_threshold from rate_best_cost_threshold_export (the plan's own price level, 120 here because the plan exports nowhere but the event). The June log line "High export rate found rates in range 99p to 120.0p" is that sentinel pair printed live. It doesn't change the outcome — 99 or 120, everything at 20p is excluded either way.

Counterfactual, same dump with the event minutes stripped: the threshold settles at 20.0 and the scan finds 95 export windows spanning the horizon, the first at minutes 900–930 — directly over the PV peak. Nothing else changed; the event's presence alone is what removes every ordinary export candidate.

Related in-flight work — PR #5163 (open, filed for #5050) rewires set_rate_thresholds() to compute its stats from event-free rates. I simulated that behaviour on this dump (event-free stats → threshold 19.9 → rescan over the full rates including the event): the scan then finds 97 windows — ordinary 20p windows and the event — the first window covers the PV peak, and the threshold settles at 20.0. So #5163 as written should incidentally resolve this symptom. It does not touch the rescan overwrite itself, so it may be worth a maintainer decision on whether the lowest-based overwrite also needs a guard against latching above the base tariff rate in cases the event exclusion doesn't cover.

Not covered by this run: the issue's "not yet explained" item — clipping growing 2.10 → 3.26 kWh after recalculation. The probe stopped at the threshold layer, so that remains open and would need a full plan replay to account for.

All four bug-report items are available via #4865 (version v8.54.3, environment, June logs, 4 September debug file), so nothing further is needed from the reporter.

[chalfontchubby]

Related: #5163 (open) fixes the same family of bug — saving-session/Axle boosted minutes leaking into the automatic rate thresholds — but tested against the 4 September debug file here, it does not fix this case on its own.

On that branch:

rate_export_cost_threshold = 20.5          <- correct, #5163 working
High export rate found rates in range 99p to 120.0p
AFTER ratchet: threshold = 99              <- fetch.py:1319 overwrites it
high_export_rates: only the two 120p Axle windows

Same end state as main: no ordinary export window anywhere in the horizon.

The reason is that #5163 corrects the stats feeding set_rate_thresholds() (rate_max/rate_min/rate_average via rate_minmax_excluding_saving()), but the ratchet at fetch.py:1315-1319 doesn't use those — it uses lowest, returned from scanning the live rate_export, which still carries the boosted minutes:

self.high_export_rates, lowest, highest = self.rate_scan_window(self.rate_export, 5, self.rate_export_cost_threshold, True, alt_rates=self.rate_import)
if self.rate_high_threshold == 0 and lowest <= self.rate_export_max:
    self.rate_export_cost_threshold = lowest

So the correct 20.5p threshold survives only until the next line. Note the scan output — "99p to 120.0p" — matches the June log in the earlier report exactly.

A complete fix probably means scanning the same saving-excluded series #5163 already builds, or not letting the lowest overwrite raise the threshold above the base tariff's own maximum. That looks like a small extension to #5163 rather than a separate change, since the mapping it needs is already there.

Caveat on method: the debug harness doesn't run fetch(), so I called set_rate_thresholds() and rate_scan_window() by hand in the same sequence as fetch.py:1313-1319. Worth confirming against a real fetch cycle before treating it as settled.

Posted by Claude on chalfontchubby's behalf.

[chalfontchubby]

Retitling and re-scoping this: the Axle framing was too narrow, and the mechanism is not really about VPP events at all.

The general form

The export rate threshold is derived from price statistics alone. It has no knowledge of how much energy has to move, or when. So it can exclude an entire period the plan needs, whenever that period happens to be cheaper than the rest of the horizon.

The clearest case needs no VPP event, no saving session, no alert and no price spike - just an ordinary two-level export tariff:

export tariff: 20p morning (00:00-12:00), 90p afternoon (12:00-24:00)
import: flat 23.8p

set_rate_thresholds() -> rate_export_cost_threshold = 20.5
rate_scan_window()    -> 24 windows, every one of them 12:00 or later
                         windows covering any morning minute: 0

The automatic branch picks rate_export_min + 0.5, which lands half a penny above the morning rate, and the whole morning disappears from high_export_rates. If PV fills the battery by 11:00, Predbat has no export slot it is allowed to consider before the afternoon, and simply curtails.

The Axle case reported here is the same defect with a more visible trigger: a 100p event window makes every ordinary 20p window fall below the threshold. Same outcome, same cause.

This is about slot visibility, not cost

Worth being precise, because it changes what the fix is. Clipping is already costed correctly - clipped PV never earns its export revenue in the prediction, so a plan that curtails scores worse than one that does not. The optimiser is not mispricing anything.

The problem is upstream of the metric: the windows the optimiser would need in order to act on that cost are never put in front of it. high_export_rates is built from a price threshold before the planner runs, so a window excluded there is invisible for the whole cycle no matter how much the metric would have liked it.

Admission is decided by price. The need is decided by energy and timing. Those are different things, and right now only the first one is consulted.

Why a purely statistical threshold cannot be patched into correctness

Any rule of the form "threshold = some function of min/max/average" has this failure somewhere, because it cannot know the plan's energy constraints. Tuning +0.5 / -0.1 moves which tariff shapes break, not whether they break.

A threshold that reflected the actual constraint would be demand-driven:

• total PV exceeding total load means some minimum energy must be exported, whatever the prices
• rank candidate slots by rate and admit downward until the admitted export capacity covers that surplus
• do the accumulation causally, sweeping forward in time against battery headroom, since energy can only be deferred while there is room to store it:
  surplus arriving by T <= export capacity admitted before T + battery headroom

The causal part is what catches the morning/afternoon case. Globally there is plenty of afternoon capacity; the trouble is the battery is full by 11:00, so morning surplus has nowhere to go. A threshold that drops far enough to admit morning slots - even though they are the worst-priced in the horizon - is the correct answer.

Erring wide is safe here: an admitted slot that goes unused costs nothing, while an excluded slot is invisible forever. So the sizing should round up generously.

Relationship to #4865

These look like one problem seen from two ends. #4865 asks for battery headroom to be created before peak PV; this asks for export slots to be visible before peak PV. Both are "capacity ahead of time", and neither is a pricing question. Worth considering together rather than as two features.

Note on #5163

#5163 fixes the case where the high-priced window is tagged as a saving/Axle event, by excluding those minutes from the threshold statistics. It does not address - and cannot address - the untagged cases above, where the expensive period is simply the tariff. Verified: with no saving minutes present, rate_minmax_excluding_saving() is a no-op and the morning still vanishes.

Posted by Claude on chalfontchubby's behalf.

[chalfontchubby]

Prototyped a fix for this. Posting the design and numbers before writing any production code, since it is a design change rather than a patch.

Prototype is standalone - no production files were touched.

The approach

Rather than trying to make the price threshold smarter, admit extra export windows ahead of predicted clipping, leaving the threshold alone entirely.

A static forward walk over PV vs load, with no planner involvement:

1. Walk the "do nothing" SoC forward in 5-minute steps, recording at each step: SoC, clipped energy, export rate, and spare pipe capacity (what the export limit can still take after PV has used its share).
2. Walk backwards applying a retro-cap: an earlier step can only give up what every later step can still spare, otherwise exporting then would drive SoC negative later. Respects best_soc_keep.
3. For each clip, admit the best-rate eligible earlier slot and consume its budget. Eligible means after the battery last filled - exporting before that point is simply undone by PV refilling the battery, so it cannot relieve this clip.
4. Coalesce the 5-minute admissions onto the real 30-minute slot grid and merge adjacent same-rate slots.

The output is windows added to high_export_rates. Nothing is removed, no threshold changes, and the optimiser is still free to reject every one of them.

Two properties that fell out of the design rather than being aimed at:

• The binding constraint is shared pipe capacity, not just battery energy. During the peak the export pipe is already full of PV (measured spare capacity 0.000 at the peak vs 0.283 earlier), which is precisely why early export works - it uses pipe capacity at a time PV is not yet competing for it.
• It self-limits. Admission only happens where clipping is predicted and there is spare pipe capacity earlier, so on well-behaved systems it admits nothing at all.

Results across 12 debug files

Every coverage/debug/*.yaml and coverage/cases/*.yaml, plus the two files from #4865:

cases where a plan got worse	0
cases unchanged (no windows admitted, plan byte-identical)	11
cases improved	1

The improved case is the one reported here:

daviessm_20260904.yaml
  before:  1 export window, metric -1659.39, clipping 3.26 kWh
  after:   2 export windows, metric -1671.94, clipping 2.08 kWh
  plan:    adds an export at +990 to 51%, leaves the 120p event export at +1695 untouched

That is the shape that should happen: export early while pipe capacity is free, refill from PV, and still export at full rate into the high-priced window in the evening.

Cost of admitting a window the optimiser then rejects

Measured: ~10ms. Going from 8 to 9 candidate windows moved plan time 0.11s to 0.12s, and the resulting plan was identical. Since an admitted-but-unused window costs about a millisecond and an excluded window is invisible for the entire cycle, the asymmetry strongly favours erring wide.

What this does NOT fix

Spiky-cloud clipping. The walk keys off predicted clipping, and a 30-minute-averaged forecast cannot represent a 10-minute cloud spike. Walking the same file at different cloud factors:

metric_cloud_coverage	modelled peak	predicted clip	admitted
0	5.51 kW	1.59 kWh	0.06 kWh
0.27 (the file's value)	7.00 kW	3.70 kWh	2.17 kWh
0.5	8.27 kW	8.59 kWh	7.27 kWh

A 120x swing on identical PV data. And it is worse than simply "no buffer built": a smooth curve that only grazes the pipe limit also leaves almost no spare pipe capacity earlier to export into, so smoothing suppresses both the signal and the remedy.

Any scheme driven by predicted clipping inherits this, #4036 included if it sizes from a forecast percentile. Options if it is worth chasing later: admit against a clear-sky envelope rather than the central case (cheap, and purely additive so unused admissions cost nothing); calibrate metric_cloud_coverage against observed intra-slot variance from the 1-minute generation history; or a standing buffer that ignores the forecast's fine structure entirely. Those are separate work.

Honest limitations of the evidence

• Only one positive case, and it is the file the algorithm was developed against. The other 11 either have no clipping or already had the slots visible. This is a no-regression check, not independent validation.
• Every structural floor in the corpus is 0.00 - none of these are days where PV genuinely exceeds pipe-plus-battery, so the avoidable/unavoidable split is untested on the case it exists for. (That split matters: on such a day residual clipping is physics, not a failure, and any evaluation has to report avoidable clip relieved rather than clip remaining.)
• Plan time roughly doubled on the one case that used an admitted window (0.28s -> 0.50s). That is the planner doing real work on a window it now uses, not overhead, but it has not been measured on a file with many windows.

Where it would live

The walk needs only PV, load, rates and battery limits - no plan - so it fits as a pre-pass in fetch_sensor_data() after the window scan, appending to high_export_rates. That keeps it out of the planner and avoids the admission-depends-on-plan circularity.

Happy to turn this into a PR if the direction looks right, or to leave it here if you would rather shape it differently - it is a behavioural change to slot selection, so worth your view before I build it.

Prototype and analysis by Claude, run and reviewed by chalfontchubby.