Summary
In some cases (approx 2%) attempts to launch remote jobs via Parameterized Remote Trigger results in a hang.
Key Symptoms
- The AQA_Test_Pipeline_JCK job displays many instances of "Waiting for 240 seconds until next poll."
- The AQA_Test_Pipeline_JCK job makes no reference to "Remote build started", or a build ID, for a given test target.
- The AQA_Test_Pipeline_JCK job runs forever (one instance ran for 150+ hours), or until terminated by the user.
Links
Short-term workaround
I propose to put a timeout block around the triggerRemoteJob call in aqaTestPipeline.groovy. Say, 4 days?
If timeout then kill and retry.
Long-term fix
Some form of upstream fix into parameterized-remote-trigger-plugin. Bob suggests that one cause for this problem could be in QueueItemData.java:
This is a well-known Jenkins core race. When a build transitions from BUILDABLE → LEFT (executor assigned), the queue item is garbage-collected from Jenkins' internal queue after a short TTL (~5 minutes). There is a window — measurable under load or slow agent handshakes — where the item's status is LEFT but the executable block hasn't been flushed to the API yet. In QueueItemData.update():
if (isLeft()) {
JSONObject remoteJobInfo = queueResponse.getJSONObject("executable");
if (!(remoteJobInfo.isNullObject())) {
buildNumber = remoteJobInfo.getInt("number");
buildURL = new URL(remoteJobInfo.getString("url"));
}
if (buildNumber != 0 && buildURL != null) status = QueueItemStatus.EXECUTED;
// ↑ if executable is null/absent, status stays LEFT, but RemoteBuildInfo never advances
}
Summary
In some cases (approx 2%) attempts to launch remote jobs via Parameterized Remote Trigger results in a hang.
Key Symptoms
Links
Short-term workaround
I propose to put a timeout block around the triggerRemoteJob call in aqaTestPipeline.groovy. Say, 4 days?
If timeout then kill and retry.
Long-term fix
Some form of upstream fix into parameterized-remote-trigger-plugin. Bob suggests that one cause for this problem could be in QueueItemData.java:
This is a well-known Jenkins core race. When a build transitions from BUILDABLE → LEFT (executor assigned), the queue item is garbage-collected from Jenkins' internal queue after a short TTL (~5 minutes). There is a window — measurable under load or slow agent handshakes — where the item's status is LEFT but the executable block hasn't been flushed to the API yet. In QueueItemData.update():