feat: drain sender and receiver children before shutdown - #500
Open
DeamonMV wants to merge 1 commit into
Open
Conversation
On SIGTERM/SIGINT the master closes the queue server, but forked sender and receiver children only find out when their queue-server socket drops -- logged as an unexpected connection-closed error, and for senders, in-flight deliveries are cut off mid-send. Before tearing down the queue server, the master now notifies every child over the fork IPC channel. Each child marks itself closing (so the socket drop is expected, not logged as an error) and drains before exiting: senders finish their current delivery and stop fetching new work, receivers let open SMTP sessions complete via smtp-server's close(). The queue server (and the Mongo connection behind it) is not torn down until every notified child has actually exited. Server.close() (lib/transport/ server.js) doesn't just stop listening -- it immediately force-closes every connected child socket, so closing it right away would sever an in-flight sender's only channel back to the queue mid-delivery, before it can report the outcome, leaving it hung forever instead of exiting. There is still no per-child drain timeout beyond that: a child that never finishes is caught by the master's existing ~10s force-exit. The sender's shutdown handler captures its drain target up front: a sender spawned by the staggered startup delay could otherwise be added to the set after the drain loop already started, and never be waited for.
Author
|
@NickOvt would you have time to review changes ? |
DeamonMV
force-pushed
the
feat/graceful-shutdown
branch
from
July 27, 2026 11:43
0ce3dc5 to
311df98
Compare
Author
|
Sorry, I have overlook my latest changes, had to force push |
NickOvt
self-requested a review
July 27, 2026 20:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On SIGTERM/SIGINT the master closes the queue server, but forked sender and receiver
children only find out when their queue-server socket drops — logged as an unexpected
QUEUE_CONNECTION_CLOSEDerror, and for senders, in-flight deliveries are cut offmid-send.
This PR notifies every child over the fork IPC channel before the queue server is
torn down, so each drains and exits quietly instead.
Problem
In production, every rollout of the sender/receiver deployments produces a burst of
QUEUE_CONNECTION_CLOSEDerrors — one per worker process, all within the same fewminutes — because Kubernetes only sends SIGTERM to the master (PID 1), never to the
forked children, so each child only learns the server is stopping once its socket to
the queue server drops.
Example log lines (trimmed to the fields the application itself emits; infra
metadata such as pod name/IP, node, cluster, and image were not application output
and are omitted):
{"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"Sender/bad/42","_zone":"bad","_pid":42,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}{"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"Sender/bad/66","_zone":"bad","_pid":66,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}Reproduced locally too, on this repo's pre-fix code (
cf94b99, in a throwawayworktree, against real MongoDB/Redis): start the master, SIGTERM it, and every
receiver plus the sender log the exact same error the instant the queue server
closes — no active delivery required, since it's the queue-socket drop itself
that's unexpected, not anything about the delivery:
Fix
Before tearing down the queue server, the master now notifies every child over the
fork IPC channel. Each child marks itself closing (so the socket drop is expected,
not logged as an error) and drains before exiting:
close()There is no per-child drain timeout — a hung child is still caught by the master's
existing ~10s force-exit, so no mail is lost either way (delivery/ingest are
at-least-once).
Test plan
grunt(eslint + nodeunit) passes, including a newtest/graceful-shutdown-test.jscovering the IPC-notify and drain-handshake primitives
MongoDB/Redis, confirmed 2 receiver children and all sender-zone children
(11 sender instances across 2 zones) drain and exit cleanly on SIGTERM — no
QUEUE_CONNECTION_CLOSED, no respawn, master exits in under 1sSMTP relay that holds its response for 9s, sent a message through the feeder,
and sent SIGTERM while the sender was mid-delivery (DATA sent, awaiting the
response). This first surfaced the queue-server-teardown race above — the
sender hung indefinitely, never exiting, because its channel back to the
queue was cut before it could report the delivery outcome. After the fix,
the sender waits out the full delivery, reports success, and exits cleanly;
the master's own exit now correctly follows the child, not the other way
around
How the active-delivery test above was run, and what it caught
Real MongoDB 6 + Redis (Docker), the
defaultzone pointed at a stub SMTPrelay that accepts the message but holds its final response for 9s, a message
sent through the feeder, SIGTERM sent to the master right after the sender's
DATA reached the stub — i.e. genuinely mid-delivery, not idle.
The first time I ran this (locally, before this commit existed), it caught the
queue-server-teardown race described above:
queueServer.close()ranimmediately and destroyed the sender's socket back to the queue before it
could report the delivery outcome, so it hung forever instead of exiting —
confirmed by
kill -0still finding the process alive minutes later. That'swhat the "queue server ... isn't torn down until every notified child has
actually exited" paragraph above fixes; it was never a separate commit, just
a bug caught in testing before this PR's single commit was written.
Re-running the same scenario against the code in this PR, the sender now
waits out the full 9s delivery, reports success, and exits on its own:
QS Service closed— the queue server tearing down — now correctly comeslast, after the sender's own drain, not before it. Master reported
Master exited after 9s, matching the stub's delay exactly, and allsender/receiver child processes were confirmed gone (
pgrepfound zero)shortly after.