Skip to content

e2e: replace timing waits with bounded synchronization - #1253

Merged
Carter Tinney (cartertinney) merged 7 commits into
mainfrom
agents/e2e-test-stability
Sep 1, 2026
Merged

Carter Tinney (cartertinney) merged 7 commits into
mainfrom
agents/e2e-test-stability

Conversation

@cartertinney

@cartertinney Carter Tinney (cartertinney) commented Sep 1, 2026 •

Copy link
Copy Markdown
Member

Stack

What changed

  • replace one-second connection polling and unbounded event/future waits with bounded condition-based synchronization
  • remove method subscription sleeps because receive-handler setters synchronously wait for feature subscription
  • signal Event Hub readiness after the first active receive cycle instead of sleeping three seconds
  • start each helper from its construction timestamp so later partitions retain new events without replaying prior test traffic
  • filter raw-string telemetry waits by expected payload so delayed unrelated events cannot satisfy them
  • guarantee service-helper shutdown if readiness fails before fixture yield
  • run the Event Hub receiver on a daemon-backed future so its 30-second shutdown timeout cannot be bypassed
  • bound twin matching/retry loops with absolute deadlines
  • remove import-time iptables mutation; existing per-test setup restores network state
  • co-locate focused wait-helper and Event Hub lifecycle tests under the IoT Hub E2E suite

Validation

  • Python 3.12 unit suite: 5437 passed, 6 skipped
  • E2E infrastructure tests: 13 passed
  • Black and Ruff passed
  • all 132 IoT Hub E2E tests collected without cloud access
  • previous natural build, Python E2E, DPS E2E, and Horton E2E gates passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Builds on #1252 by replacing fragile IoT Hub E2E timing waits with bounded synchronization and safer Event Hub lifecycle handling.

Changes:

  • Adds bounded sync/async wait helpers and daemon-backed operations.
  • Improves Event Hub readiness, filtering, startup position, and shutdown.
  • Replaces fixed sleeps and unbounded waits across E2E tests.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/e2e/iothub_e2e/wait_helpers.py Adds bounded polling helpers.
tests/e2e/iothub_e2e/test_wait_helpers.py Tests polling and timeout behavior.
tests/e2e/iothub_e2e/test_service_helper.py Tests Event Hub lifecycle behavior.
tests/e2e/iothub_e2e/sync/test_sync_twin.py Bounds twin synchronization waits.
tests/e2e/iothub_e2e/sync/test_sync_send_message.py Bounds sends and filters telemetry.
tests/e2e/iothub_e2e/sync/test_sync_sas_renewal.py Bounds renewal event waits.
tests/e2e/iothub_e2e/sync/test_sync_methods.py Removes redundant subscription sleep.
tests/e2e/iothub_e2e/sync/test_sync_connect_disconnect.py Bounds connection-state waits.
tests/e2e/iothub_e2e/sync/test_sync_c2d.py Bounds C2D receipt waits.
tests/e2e/iothub_e2e/sync/conftest.py Adds readiness and guaranteed shutdown.
tests/e2e/iothub_e2e/drop_fixtures.py Removes import-time network mutation.
tests/e2e/iothub_e2e/const.py Defines the shared E2E timeout.
tests/e2e/iothub_e2e/conftest.py Adds daemon-thread execution fixture.
tests/e2e/iothub_e2e/aio/test_twin.py Bounds asynchronous twin waits.
tests/e2e/iothub_e2e/aio/test_twin_stress.py Adds deadlines to stress synchronization.
tests/e2e/iothub_e2e/aio/test_send_message.py Bounds sends and filters telemetry.
tests/e2e/iothub_e2e/aio/test_sas_renewal.py Bounds renewal event waits.
tests/e2e/iothub_e2e/aio/test_methods.py Removes redundant subscription sleep.
tests/e2e/iothub_e2e/aio/test_connect_disconnect.py Bounds connection-state waits.
tests/e2e/iothub_e2e/aio/test_c2d.py Bounds C2D receipt waits.
tests/e2e/iothub_e2e/aio/conftest.py Adds readiness and guaranteed shutdown.
dev_utils/dev_utils/service_helper.py Exposes async readiness and filtering.
dev_utils/dev_utils/service_helper_sync.py Implements Event Hub lifecycle synchronization.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/e2e/iothub_e2e/aio/test_twin_stress.py
stack merge was automatically disabled September 1, 2026 18:56

Pull Request is not mergeable

stack merge was automatically disabled September 1, 2026 19:22

Pull Request is not mergeable

Base automatically changed from agents/unit-test-failure-analysis-python312 to main September 1, 2026 19:29
Use explicit condition, event, and future deadlines instead of fixed sleeps and unbounded polling. Signal Event Hub readiness from an active receive cycle, guarantee fixture cleanup, and use a daemon-backed receiver so shutdown timeouts cannot be bypassed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The service helper imports Azure IoT Hub service dependencies that are intentionally absent from unit-test jobs. Move its focused tests under the E2E suite while keeping them runnable without E2E fixtures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Provide the daemon-thread fixture within the E2E test root and start Event Hub consumption from the helper construction timestamp. This preserves events for partitions that attach later without replaying prior async-test traffic into sync tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow Event Hub waits without message IDs to filter arrivals, so delayed telemetry from earlier tests cannot satisfy raw-string send assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep tests for IoT Hub E2E infrastructure together under the E2E suite instead of splitting the wait-helper tests into the unit-test tree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use one absolute deadline for each reported-property batch and create only batch_size updates per batch. This prevents unrelated patches from extending the timeout and restores the intended total update count.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use one absolute deadline per expected serial reported-property update so delayed unrelated patches cannot renew the service wait indefinitely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cartertinney
Carter Tinney (cartertinney) merged commit fefdb59 into main Sep 1, 2026
12 of 20 checks passed
@cartertinney
Carter Tinney (cartertinney) deleted the agents/e2e-test-stability branch September 1, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants