CPU and Max RSS Analysis tools by ChrisPaulBennett · Pull Request #675 · cylc/cylc-uiserver

ChrisPaulBennett · 2025-03-12T09:14:27Z

This apart of 3 pull requests for adding CPU time and Max RSS analysis to the Cylc UI.

This adds the Max RSS and CPU time (as measured by cgroups) to the table view, box plot and time series views.

Linked to;
cylc/cylc-flow#6663
cylc/cylc-ui#2100

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
Changelog entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

ChrisPaulBennett · 2025-04-15T07:48:58Z

I've merged in the previous changes to the SQL infrastructure that deals with task states. @oliver-sanders I've had to change some of your code that creates parts of the SQL query, mostly just to take into account the new join.
The codecov failures are from code that I haven't touched.

MetRonnie

These changes appear to have broken compatibility with workflows run in earlier versions of Cylc. I tested the tasks GraphQL query on a workflow that ran in 8.4.4.dev and got an empty result

query {
  tasks(live: false, workflows: ["wflow"]) {
    name
  }
}

{
  "data": {
    "tasks": []
  }
}

ChrisPaulBennett · 2025-07-18T10:10:43Z

This been fixed. Max RSS and CPU time will be 0, if no data is present. An update to the cylc-ui needs to be made to not show cpu time / max rss charts if there is no data

MetRonnie · 2025-08-12T15:01:53Z

This been fixed. Max RSS and CPU time will be 0, if no data is present. An update to the cylc-ui needs to be made to not show cpu time / max rss charts if there is no data

Is there a way to make this null instead of 0? Because currently the UI can't tell the difference between no data and times that are actually zero (to the nearest whole second).

ChrisPaulBennett · 2025-10-20T13:35:28Z

This been fixed. Max RSS and CPU time will be 0, if no data is present. An update to the cylc-ui needs to be made to not show cpu time / max rss charts if there is no data

Is there a way to make this null instead of 0? Because currently the UI can't tell the difference between no data and times that are actually zero (to the nearest whole second).

This has been fixed in the UI repo

oliver-sanders

Some fields removed from the tests, not sure if this was purposeful?

oliver-sanders · 2026-09-08T13:39:20Z

-    assert ret['started_time'] == '2022-12-14T15:01:00Z'
-    assert ret['state'] == 'succeeded'
    assert ret['std_dev_queue_time'] == pytest.approx(0.0, 0.01)
    assert ret['std_dev_run_time'] == pytest.approx(0.0, 0.01)
    assert ret['std_dev_total_time'] == pytest.approx(0.0, 0.01)
-    assert ret['submit_num'] == 1
-    assert ret['submitted_time'] == '2022-12-14T15:00:00Z'


Are we not testing started_time/submitted_time/state/submit_num any more?

oliver-sanders · 2026-09-08T13:39:47Z

@@ -187,87 +249,115 @@
    assert ret['min_total_time'] == 600
    assert ret['name'] == 'Task_1'
    assert ret['platform'] == 'MyPlatform'
-    assert ret['started_time'] == '2022-12-15T15:01:16Z'
-    assert ret['state'] == 'succeeded'
    assert ret['std_dev_queue_time'] == pytest.approx(8.00, 0.01)
    assert ret['std_dev_run_time'] == pytest.approx(52.0, 0.01)
    assert ret['std_dev_total_time'] == pytest.approx(60.0, 0.01)
-    assert ret['submit_num'] == 1
-    assert ret['submitted_time'] == '2022-12-15T15:00:00Z'


Are we not testing these fields any more?

oliver-sanders · 2026-09-08T13:40:19Z

@@ -279,43 +369,29 @@
    assert ret['min_total_time'] == 600
    assert ret['name'] == 'Task_1'
    assert ret['platform'] == 'MyPlatform'
-    assert ret['started_time'] == '2022-12-16T15:01:16Z'
-    assert ret['state'] == 'succeeded'
    assert ret['std_dev_queue_time'] == pytest.approx(7.54, 0.01)
    assert ret['std_dev_run_time'] == pytest.approx(49.02, 0.01)
    assert ret['std_dev_total_time'] == pytest.approx(56.56, 0.01)
-    assert ret['submit_num'] == 1
-    assert ret['submitted_time'] == '2022-12-16T15:00:00Z'


Are we not testing these fields any more?

No. They we aren't. These values are not apart of the task query anymore. I've also removed submit_num.

I think we've removed some of the fields that these functions used to return. E.g, id was returned before, but isn't now.

Note, this is unrelated to tidying unused parts of the SQL queries used within the functions.

MetRonnie · 2026-10-06T14:25:36Z

+        cylc_version = CylcVersion(description=sstrip('''
                Set the Cylc version that the workflow starts with.
-            ''')
-        )
-        initial_cycle_point = CyclePoint(
-            description=sstrip('''
+            '''))


Can you revert this un-blackening?

Re-blackened the file

That's not what I meant. And it hasn't reverted these changes, which was all I was asking for

To make this easier to review, I've reverted the formatting changes that were made to unrelated lines (blackened or un-blackened): acebd40

MetRonnie · 2026-10-06T14:30:12Z

-            else row[f"{prop}_quartile_3"]
-        ),
-    ]
+def get_quartiles(row, prop: str) -> List[Optional[float]]:


(Try not to re-introduce the deprecated typing classes going forwards)

list[float | None]

Not done? But let's leave it as is now

MetRonnie · 2026-10-29T14:57:11Z

+    # fill forwards from the nearest lower quartile
+    previous = None
+    for index, value in enumerate(quartiles):
+        if value is None:
+            quartiles[index] = previous
+        else:
+            previous = value
+    # fill any leading gaps backwards from the first available quartile
+    fallback = next((value for value in quartiles if value is not None), None)
+    return [fallback if value is None else value for value in quartiles]


This can be simpler with a rearrangement:

Suggested change

# fill forwards from the nearest lower quartile

previous = None

for index, value in enumerate(quartiles):

if value is None:

quartiles[index] = previous

else:

previous = value

# fill any leading gaps backwards from the first available quartile

fallback = next((value for value in quartiles if value is not None), None)

return [fallback if value is None else value for value in quartiles]

# find the first existing quartile to use as the initial value

previous = next((value for value in quartiles if value is not None), None)

# fill forwards from the nearest lower quartile

for index, value in enumerate(quartiles):

if value is None:

quartiles[index] = previous

else:

previous = value

return quartiles

ChrisPaulBennett marked this pull request as draft March 12, 2025 09:19

oliver-sanders reviewed Mar 12, 2025

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

This was referenced Mar 13, 2025

CPU and Max RSS Analysis tools cylc/cylc-flow#6663

Merged

CPU and Max RSS Analysis tools cylc/cylc-ui#2100

Open

ChrisPaulBennett marked this pull request as ready for review April 2, 2025 14:15

This comment was marked as resolved.

Sign in to view

ChrisPaulBennett requested a review from oliver-sanders April 15, 2025 07:49

This comment was marked as resolved.

Sign in to view

MetRonnie reviewed May 27, 2025

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

MetRonnie reviewed Jul 14, 2025

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

MetRonnie reviewed Jul 14, 2025

View reviewed changes

ChrisPaulBennett self-assigned this Jul 18, 2025

MetRonnie added this to the 1.8.0 milestone Aug 4, 2025

oliver-sanders modified the milestones: 1.8.0, 1.9.0 Sep 17, 2025

This comment was marked as resolved.

Sign in to view

ChrisPaulBennett force-pushed the memory_view branch from e14ad5c to 83dbbd6 Compare December 5, 2025 09:18

CPU/Memory Logging working

0d31a9f

ChrisPaulBennett force-pushed the memory_view branch from 83dbbd6 to 0d31a9f Compare December 5, 2025 09:19

Updated unit tests

75b3ce1

ChrisPaulBennett requested a review from MetRonnie December 30, 2025 15:33

oliver-sanders requested review from wxtim and removed request for MetRonnie January 13, 2026 15:38

wxtim reviewed Jan 27, 2026

View reviewed changes

Comment thread cylc/uiserver/tests/test_workflow_retrieval.py Resolved

wxtim reviewed Jan 28, 2026

View reviewed changes

Comment thread cylc/uiserver/tests/test_workflow_retrieval.py Resolved

wxtim reviewed Jan 28, 2026

View reviewed changes

Comment thread cylc/uiserver/tests/test_workflow_retrieval.py Resolved

ChrisPaulBennett added 2 commits September 7, 2026 15:28

Using SUM over TOTAL so zero's don't get sent

d31e4df

Fixed unintended NULL bug

aeeb9d3

ChrisPaulBennett marked this pull request as ready for review September 8, 2026 12:33

oliver-sanders reviewed Sep 8, 2026

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

optimised sql query

0a454ca

oliver-sanders reviewed Sep 8, 2026

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

oliver-sanders reviewed Sep 8, 2026

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

Removed usage of Bigint

d02141c

ChrisPaulBennett mentioned this pull request Sep 10, 2026

Changed max rss values to MiB to decrease number size cylc/cylc-flow#7451

Closed

8 tasks done

ChrisPaulBennett added 2 commits September 10, 2026 11:24

Replacing unintentionally removed asserts

46781fc

Altered Unit tests

032ed5e

ChrisPaulBennett requested review from MetRonnie and oliver-sanders September 18, 2026 13:14

MetRonnie reviewed Sep 21, 2026

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

ChrisPaulBennett added 2 commits September 21, 2026 14:06

Changed RSS values to use floats

09a057e

Added more uses of get_quartiles helper method

b1a6d47

MetRonnie reviewed Sep 21, 2026

View reviewed changes

Comment thread cylc/uiserver/schema.py Resolved

Comment thread cylc/uiserver/schema.py Resolved

ChrisPaulBennett added 2 commits September 23, 2026 15:41

Merge branch 'master' into memory_view

3ef2eda

Fixing quartiles and changing a few ints to floats

542e736

oliver-sanders modified the milestones: 1.10.0, 1.11.0 Sep 30, 2026

MetRonnie reviewed Oct 6, 2026

View reviewed changes

ChrisPaulBennett and others added 5 commits October 27, 2026 16:09

Fixed bug with quartiles

63ca699

Linting

b5cccb6

Merge remote-tracking branch 'origin/master' into memory_view

455c37c

Add te (SQL alias) to codespell ignore list

9d117d2

Revert formatting changes to unrelated lines

acebd40

MetRonnie force-pushed the memory_view branch from 4608862 to acebd40 Compare October 29, 2026 14:40

MetRonnie reviewed Oct 29, 2026

View reviewed changes

Conversation

ChrisPaulBennett commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

ChrisPaulBennett commented Apr 15, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MetRonnie left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChrisPaulBennett commented Jul 18, 2025

Uh oh!

MetRonnie commented Aug 12, 2025

Uh oh!

ChrisPaulBennett commented Oct 20, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

oliver-sanders left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MetRonnie Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MetRonnie Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChrisPaulBennett commented Mar 12, 2025 •

edited

Loading

MetRonnie left a comment •

edited

Loading

MetRonnie Oct 29, 2026 •

edited

Loading

MetRonnie Oct 29, 2026 •

edited

Loading