WIP: upgrade to rtc-tools 2.7.3, casadi 3.7.2, rtctools-highs 0.1.3 by jarsarasty · Pull Request #483 · Multi-Energy-Systems-Optimization/mesido

jarsarasty · 2026-06-04T12:00:23Z

Replace rtc-tools-gil-comp and casadi-gil-comp with rtc-tools 2.7.3 and rtctools-highs 0.1.3 (casadi 3.7.2 pinned transitively)
Register HiGHS 1.14.0 CasADi plugin via import in mesido/init.py
Update python_requires to >=3.10,<3.13 (pandas<2.0 has no wheels for 3.13+)
Update solver docstrings from cbc to HiGHS
Fix __state_vector_scaled duplication: delegate to BaseProblemMixin
Add BaseProblemMixin to FinancialMixin bases
Add presolve=off workaround for HiGHS 1.14.0 in HeatProblem and HeatProblemSetPoints; HeatProblemPlacingOverTime explicitly removes it as it needs presolve on for correct solution
Fix test_multicommodity: use asset IDs and correct .__disabled key
Fix test_electricity_storage: use asset IDs, relax tolerance
Fix test_gas_pipe_topology_optimization: relax atol 1e-10 -> 1e-6
Add test_highs_solver.py: verify HiGHS 1.14.0 and GIL release
Raise tox.ini end_scenario_sizing timeout 120s -> 300s
Remove CBC solver override from test_multicommodity price_profile test

Known failing (WIP):

test_head_loss::test_heat_network_pipe_split_head_loss: HiGHS 1.14.0 finds different valid flow split; test assumption too strict
test_end_scenario_sizing_staged: IndexError on _priorities_output[1], suggests a priority solve is failing with HiGHS 1.14.0

- Replace rtc-tools-gil-comp and casadi-gil-comp with rtc-tools 2.7.3 and rtctools-highs 0.1.3 (casadi 3.7.2 pinned transitively) - Register HiGHS 1.14.0 CasADi plugin via import in mesido/__init__.py - Widen python_requires to >=3.10,<3.15 - Update solver docstrings from cbc to HiGHS - Fix __state_vector_scaled duplication: delegate to BaseProblemMixin - Add BaseProblemMixin to FinancialMixin bases - Add presolve=off workaround for HiGHS 1.14.0 (ERGO-Code/HiGHS#2388) in HeatProblem and HeatProblemSetPoints; HeatProblemPlacingOverTime explicitly removes it as it needs presolve on for correct solution - Fix test_multicommodity: use asset IDs and correct .__disabled key - Fix test_electricity_storage: use asset IDs, relax tolerance - Fix test_gas_pipe_topology_optimization: relax atol 1e-10 -> 1e-6 - Add test_highs_solver.py: verify HiGHS 1.14.0 and GIL release - Raise tox.ini end_scenario_sizing timeout 120s -> 300s - Remove CBC solver override from test_multicommodity price_profile test - Tighten python_requires to <3.13 (pandas<2.0 has no wheels for 3.13+) - Add comment on CBC removal in test_multicommodity explaining why HiGHS 1.14.0 via rtctools-highs 0.1.3 now passes correctly Known failing (WIP): - test_head_loss::test_heat_network_pipe_split_head_loss: HiGHS 1.14.0 finds different valid flow split; test assumption too strict - test_end_scenario_sizing_staged: IndexError on _priorities_output[1], suggests a priority solve is failing with HiGHS 1.14.0

…th pandas<2.0

jarsarasty · 2026-06-05T11:11:33Z

Hi @KobusVanRooyen,

With the changes in this PR, we can finally run the latest Highs version (1.14) in mesido.

However, two tests are failing due to numerical issues or testing assumptions, and some adjustments should be implemented in a separate PR before we merge this PR:

test_head_loss::test_heat_network_pipe_split_head_loss

The piecewise-linear DW approximation introduces near-zero constraint coefficients that can cause numerical ill-conditioning in HiGHS 1.14.0, potentially leading to a suboptimal solution. The assertion could be relaxed to a lower-bound check pending investigation

test_asset_is_realized::test_asset_is_realized

Preliminary investigation identified three compounding issues:
1. t=0 investment cap bound appears unscaled in run_ates.py
2. asset_is_realized big-M constraint in financial_mixin.py may be too loose
3. HeatProducer_1 has zero capex in the test ESDL, making the constraint chain untestable for that asset

KobusVanRooyen · 2026-06-11T08:03:39Z

Hi @KobusVanRooyen,

With the changes in this PR, we can finally run the latest Highs version (1.14) in mesido.

However, two tests are failing due to numerical issues or testing assumptions, and some adjustments should be implemented in a separate PR before we merge this PR:

test_head_loss::test_heat_network_pipe_split_head_loss

The piecewise-linear DW approximation introduces near-zero constraint coefficients that can cause numerical ill-conditioning in HiGHS 1.14.0, potentially leading to a suboptimal solution. The assertion could be relaxed to a lower-bound check pending investigation

test_asset_is_realized::test_asset_is_realized
Preliminary investigation identified three compounding issues:

t=0 investment cap bound appears unscaled in run_ates.py

asset_is_realized big-M constraint in financial_mixin.py may be too loose

HeatProducer_1 has zero capex in the test ESDL, making the constraint chain untestable for that asset

Hi @jarsarasty. Some feedback:

I see you updated your message regarding the pipes being identically ;-) Please update with main first before doing further checks re this one. And the parallel pipes are not identically so the shortest pipe should be chosen.
test_asset_is_realized::test_asset_is_realized:
1: Nice catch, In run_ates.py in HeatProblemPlacingOverTime constraints the value 200000.0 should be 200000.0 / nominal ?
2: Yes. The factor of 1.5 in the Big M can probably be reduced to a smaller number (1.0-1.1 ?) in __cumulative_investments_made_in_eur_path_constraints (financial_mixin.py)
3: HeatProducer_1 does have capex and opex in the esdl file, but this assets is not selected for use therefor the cost is 0 in the results. You will also see __max_size = 0 for this asset.

And by the way the only test that I see currently failing is tests/test_head_loss.py::TestHeadLoss::test_heat_network_pipe_split_head_loss (with all the presolve turned in the branch dated 2026/06/10)

KobusVanRooyen · 2026-06-11T07:33:15Z

        "pyesdl == 26.3",
        "pandas >= 1.3.1, < 2.0",
-        "casadi-gil-comp == 3.6.7",
+        "rtctools-highs == 0.1.3",


Currently the specific HIGHS and CASADI version cannot be specified in MESIDO. Would it be possible to change the rtc-tools-casadi-plugins such that the HIGHS and CASADI version numbers are inputs?

KobusVanRooyen · 2026-06-11T07:34:08Z

 Operating System :: MacOS
 """

-if sys.version_info < (3, 10):


I would not remove this but rather update the version numbers

KobusVanRooyen · 2026-06-11T07:37:14Z

    achieved by having an upper limit on the investment per time-step.
    """

+    def solver_options(self):


I am aware that you will still investigate potential solutions such that we do not have to turn of presolve. But this part is not needed, since presolve is already turned off in the inherited problem class (HeatProblem)

KobusVanRooyen · 2026-06-11T07:40:13Z

 Operating System :: MacOS
 """

-if sys.version_info < (3, 10):


Please update with main before continuing, since there has been some fixes regarding network settings which might have an impact on the head loss testing

KobusVanRooyen · 2026-06-11T07:43:37Z

        highs_options["mip_rel_gap"] = 0.02
-        highs_options["presolve"] = "on"
-
+        # workaround for HiGHS 1.14.0 presolve bug (ERGO-Code/HiGHS#2388)


Not ideal. Any other insights in this one to potential get rid of this?

So this one is here for test_setpoint_constraints.py (test_run_small_ates_timed_setpoints_2_changes, test_run_small_ates_timed_setpoints_0_changes, test_run_small_ates_timed_setpoints_multiple_constraint --> presolve_infeasible_or_unbounded_priority_1)

KobusVanRooyen · 2026-06-11T07:52:05Z

        options["casadi_solver"] = self._qpsol
+        highs_options = options.setdefault("highs", {})
+        # workaround for HiGHS 1.14.0 presolve bug (ERGO-Code/HiGHS#2388)
+        highs_options["presolve"] = "off"


HiGHS#2388 -> I assume this refers to a specific issue number, but I cannot find it in https://github.com/ERGO-Code/HiGHS/issues?q=is%3Aissue%20state%3Aopen

Can you please clarify?

KobusVanRooyen · 2026-06-11T08:06:24Z

        """
        options = super().solver_options()
        options["casadi_solver"] = self._qpsol
+        highs_options = options.setdefault("highs", {})


Not ideal. Any other insights to potentially not turn off presolve?

This one is needed for tests/test_ates.py::TestAtes::test_ates

KobusVanRooyen · 2026-06-11T08:35:28Z

@@ -0,0 +1,77 @@
+"""Tests for HiGHS 1.14.0 integration via rtctools-highs.


I am a bit uncertain about this test. Since the idea originally was to be able to specify the version number of casadi and highs in mesido (de-linked via mesido). I will think about this topic a bit futher and get back to you

But here is some feedback:
In this test all version numbers are hard-coded and specific version numbers are also mentioned in the comments/doc strings. I think this needs to be made a bit more generic with no version numbers mentioned in the text and all version numbers in the test itself should ideally be a variable where one sets the version number.

jarsarasty marked this pull request as ready for review June 4, 2026 12:06

jarsarasty requested review from FJanssen-TNO, KobusVanRooyen and jimrojerTNO as code owners June 4, 2026 12:06

jarsarasty marked this pull request as draft June 4, 2026 12:11

fix(ci): pin numpy<2.0 in tox deps to avoid binary incompatibility wi…

c2efeb7

…th pandas<2.0

jarsarasty marked this pull request as ready for review June 4, 2026 14:58

KobusVanRooyen reviewed Jun 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: upgrade to rtc-tools 2.7.3, casadi 3.7.2, rtctools-highs 0.1.3#483

WIP: upgrade to rtc-tools 2.7.3, casadi 3.7.2, rtctools-highs 0.1.3#483
jarsarasty wants to merge 2 commits into
mainfrom
feature/upgrade-to-rtc-tools-2.7.3-casadi-3.7.2

jarsarasty commented Jun 4, 2026 •

edited

Loading

Uh oh!

jarsarasty commented Jun 5, 2026 •

edited

Loading

Uh oh!

KobusVanRooyen commented Jun 11, 2026 •

edited

Loading

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

KobusVanRooyen Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,77 @@
		"""Tests for HiGHS 1.14.0 integration via rtctools-highs.

Uh oh!

Conversation

jarsarasty commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jarsarasty commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KobusVanRooyen commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jarsarasty commented Jun 4, 2026 •

edited

Loading

jarsarasty commented Jun 5, 2026 •

edited

Loading

KobusVanRooyen commented Jun 11, 2026 •

edited

Loading