Skip to content

feat: drain sender and receiver children before shutdown - #500

Open
DeamonMV wants to merge 1 commit into
zone-eu:masterfrom
DeamonMV:feat/graceful-shutdown
Open

feat: drain sender and receiver children before shutdown#500
DeamonMV wants to merge 1 commit into
zone-eu:masterfrom
DeamonMV:feat/graceful-shutdown

Conversation

@DeamonMV

Copy link
Copy Markdown

Summary

On SIGTERM/SIGINT the master closes the queue server, but forked sender and receiver
children only find out when their queue-server socket drops — logged as an unexpected
QUEUE_CONNECTION_CLOSED error, and for senders, in-flight deliveries are cut off
mid-send.

This PR notifies every child over the fork IPC channel before the queue server is
torn down, so each drains and exits quietly instead.

Problem

In production, every rollout of the sender/receiver deployments produces a burst of
QUEUE_CONNECTION_CLOSED errors — one per worker process, all within the same few
minutes — because Kubernetes only sends SIGTERM to the master (PID 1), never to the
forked children, so each child only learns the server is stopping once its socket to
the queue server drops.

Example log lines (trimmed to the fields the application itself emits; infra
metadata such as pod name/IP, node, cluster, and image were not application output
and are omitted):

{"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"Sender/bad/42","_zone":"bad","_pid":42,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}
{"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"Sender/bad/66","_zone":"bad","_pid":66,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}

Reproduced locally too, on this repo's pre-fix code (cf94b99, in a throwaway
worktree, against real MongoDB/Redis): start the master, SIGTERM it, and every
receiver plus the sender log the exact same error the instant the queue server
closes — no active delivery required, since it's the queue-socket drop itself
that's unexpected, not anything about the delivery:

ERR! SMTP/feeder/67098 Connection to Queue server closed unexpectedly
ERR! Sender/default/67100 Connection to Queue server closed unexpectedly
info Gelf {"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"SMTP/feeder/67098","_interface":"feeder","_pid":67098,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}
info Gelf {"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"Sender/default/67100","_zone":"default","_pid":67100,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}
ERR! SMTP/feeder/67099 Connection to Queue server closed unexpectedly
info Gelf {"short_message":"MTA [QUEUE_CONNECTION_CLOSED] Queue server connection closed unexpectedly","_logger":"SMTP/feeder/67099","_interface":"feeder","_pid":67099,"full_message":"Queue server connection closed unexpectedly","_error":"Queue server connection closed unexpectedly","facility":"mta","_component":"mta"}

Fix

Before tearing down the queue server, the master now notifies every child over the
fork IPC channel. Each child marks itself closing (so the socket drop is expected,
not logged as an error) and drains before exiting:

  • senders finish their current delivery and stop fetching new work
  • receivers let open SMTP sessions complete via smtp-server's close()

There is no per-child drain timeout — a hung child is still caught by the master's
existing ~10s force-exit, so no mail is lost either way (delivery/ingest are
at-least-once).

Test plan

  • grunt (eslint + nodeunit) passes, including a new test/graceful-shutdown-test.js
    covering the IPC-notify and drain-handshake primitives
  • Verified end-to-end with idle workers: started the master against real
    MongoDB/Redis, confirmed 2 receiver children and all sender-zone children
    (11 sender instances across 2 zones) drain and exit cleanly on SIGTERM — no
    QUEUE_CONNECTION_CLOSED, no respawn, master exits in under 1s
  • Verified end-to-end with an active delivery: pointed a zone at a stub
    SMTP relay that holds its response for 9s, sent a message through the feeder,
    and sent SIGTERM while the sender was mid-delivery (DATA sent, awaiting the
    response). This first surfaced the queue-server-teardown race above — the
    sender hung indefinitely, never exiting, because its channel back to the
    queue was cut before it could report the delivery outcome. After the fix,
    the sender waits out the full delivery, reports success, and exits cleanly;
    the master's own exit now correctly follows the child, not the other way
    around
How the active-delivery test above was run, and what it caught

Real MongoDB 6 + Redis (Docker), the default zone pointed at a stub SMTP
relay that accepts the message but holds its final response for 9s, a message
sent through the feeder, SIGTERM sent to the master right after the sender's
DATA reached the stub — i.e. genuinely mid-delivery, not idle.

The first time I ran this (locally, before this commit existed), it caught the
queue-server-teardown race described above: queueServer.close() ran
immediately and destroyed the sender's socket back to the queue before it
could report the delivery outcome, so it hung forever instead of exiting —
confirmed by kill -0 still finding the process alive minutes later. That's
what the "queue server ... isn't torn down until every notified child has
actually exited" paragraph above fixes; it was never a separate commit, just
a bug caught in testing before this PR's single commit was written.

Re-running the same scenario against the code in this PR, the sender now
waits out the full 9s delivery, reports success, and exits on its own:

info Process Server closing down...
info Sender/bounces/46035 Received shutdown from master, draining 1 sender(s)
info Sender/default/46029 Received shutdown from master, draining 1 sender(s)
info SMTP/feeder/46027 Received shutdown from master, draining SMTP sessions
info API Service closed
info Sender/default/46029[1] Closing sender instance for default
info Sender/bounces/46035[1] Closing sender instance for bounces
info SMTP/feeder/46027 Graceful shutdown, draining complete, exiting
info SMTP/feeder/46028 Received shutdown from master, draining SMTP sessions
info SMTP/feeder/46028 Graceful shutdown, draining complete, exiting
info Service closed
info Sender/bounces/46035 Graceful shutdown, draining complete, exiting
info Sender/default/46029[1] ... ACCEPTED ... (250 2.0.0 OK queued as stub-1)
info Sender/default/46016 All SMTP sessions end id=ih71ln9copf45rre
info Sender/default/46029 Graceful shutdown, draining complete, exiting
info QS Service closed

QS Service closed — the queue server tearing down — now correctly comes
last, after the sender's own drain, not before it. Master reported
Master exited after 9s, matching the stub's delay exactly, and all
sender/receiver child processes were confirmed gone (pgrep found zero)
shortly after.

On SIGTERM/SIGINT the master closes the queue server, but forked sender and
receiver children only find out when their queue-server socket drops --
logged as an unexpected connection-closed error, and for senders, in-flight
deliveries are cut off mid-send.

Before tearing down the queue server, the master now notifies every child
over the fork IPC channel. Each child marks itself closing (so the socket
drop is expected, not logged as an error) and drains before exiting: senders
finish their current delivery and stop fetching new work, receivers let open
SMTP sessions complete via smtp-server's close().

The queue server (and the Mongo connection behind it) is not torn down until
every notified child has actually exited. Server.close() (lib/transport/
server.js) doesn't just stop listening -- it immediately force-closes every
connected child socket, so closing it right away would sever an in-flight
sender's only channel back to the queue mid-delivery, before it can report
the outcome, leaving it hung forever instead of exiting. There is still no
per-child drain timeout beyond that: a child that never finishes is caught
by the master's existing ~10s force-exit.

The sender's shutdown handler captures its drain target up front: a sender
spawned by the staggered startup delay could otherwise be added to the set
after the drain loop already started, and never be waited for.
@DeamonMV

Copy link
Copy Markdown
Author

@NickOvt would you have time to review changes ?
Thank you

@DeamonMV
DeamonMV force-pushed the feat/graceful-shutdown branch from 0ce3dc5 to 311df98 Compare July 27, 2026 11:43
@DeamonMV

DeamonMV commented Jul 27, 2026

Copy link
Copy Markdown
Author

Sorry, I have overlook my latest changes, had to force push

@NickOvt
NickOvt self-requested a review July 27, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant