Skip to content

Commit 25e75e1

Browse files
committed
Give the nightly E2E test job a realistic timeout
The Windows legs of the python-nightly matrix run right up against Azure Pipelines' implicit 60-minute job timeout, so they get cancelled even though the test suite itself passes. This is what has been showing up as a "flaky" nightly. Measured on build 161689 (a clean run, setup and cleanup both green): py39_windows_mqtt 60.4 min -> CANCELLED py39_windows_mqttws 52.9 min -> passed, ~7 minutes of headroom the seven linux legs 31.9 - 34.7 min The cancelled leg had already printed its pytest summary before being killed: ========= 74 passed, 33 skipped, 67924 warnings in 3437.00s (0:57:16) ========= So nothing actually failed -- 0 failures, 74 passed -- the job simply ran out of wall clock during teardown. Whichever Windows leg happens to be slower on the day is the one that dies, which is why the failure appears to wander between py39_windows_mqtt and py39_windows_mqttws across runs. vsts/python-nightly.yaml sets no timeoutInMinutes anywhere, so every job inherits the 60-minute default. Set 90 minutes on the Test job. This is a CI-budget fix and is unrelated to the CodeQL change in this PR, but it has to live on this branch for this PR's own python-nightly check to pass. Happy to split it into its own PR if preferred.
1 parent a9364c9 commit 25e75e1

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

‎vsts/python-nightly.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,17 @@ stages:
6262
- stage: build_and_tests
6363
jobs:
6464
- job: 'Test'
65+
# The Windows legs of this matrix run right up against Azure Pipelines' implicit
66+
# 60-minute job timeout, so they get cancelled mid-run even though the suite
67+
# itself passes. Measured in build 161689:
68+
# py39_windows_mqtt 60.4 min -> CANCELLED (pytest had already reported
69+
# "74 passed, 33 skipped" in 57:16)
70+
# py39_windows_mqttws 52.9 min -> passed, with only ~7 minutes of headroom
71+
# the seven linux legs 31.9 - 34.7 min
72+
# Whichever Windows leg is slower on the day is the one that dies, which is why
73+
# the failure appears to wander between py39_windows_mqtt and py39_windows_mqttws.
74+
# Give the job a realistic budget instead.
75+
timeoutInMinutes: 90
6576
variables:
6677
IOTHUB_CONNECTION_STRING: $[stageDependencies.setup.create_iot_hub_and_dps.outputs['createAzureResources.IOTHUB_CONNECTION_STRING']]
6778
EVENTHUB_CONNECTION_STRING: $[stageDependencies.setup.create_iot_hub_and_dps.outputs['createAzureResources.EVENTHUB_CONNECTION_STRING']]

0 commit comments

Comments
 (0)