88. pepsi-failure-bouncer¶
Funnel failed/timeout messages to the bounce stage.
88.1. Role¶
When its stage gives up, a message in the post-ingress pipeline ends up
failed or timeout. It is failed when a stage gave a permanent error,
an error outlived MAX_LIFETIME at a stage with no BOUNCE_STAGE, the
message crashed its worker three times, or it was pending at a stage that the
configuration no longer names. It is timeout when it held its worker past
MAX_RUNTIME three times or, in a pepsi-stage-relay-to-* stage, was still
being killed at MAX_LIFETIME. The dispatcher never
re-queues such a row, so without operator action the message is stuck in
pepsi.workqueue and the sender is never notified.
pepsi-failure-bouncer moves each stuck message to the configured
[pepsi-failure-bouncer] BOUNCE_STAGE and resets it to pending, so the
dispatcher runs the bounce stage (pepsi-stage-bounce), which emits a
delivery-status notification as the sender’s RFC 3461 NOTIFY requests. A message
already at the bounce stage is left alone, so a bounce failing at that stage
does not loop; one failing downstream (an unrelayable null-sender DSN) is moved
back and simply dropped (a bounce is never re-bounced). BOUNCE_STAGE must
therefore lead to a real bounce stage: pepsi-setup refuses one that names
no stage and warns about one that does not run pepsi-stage-bounce directly.
It waits first: a message is moved only once it has been failed for
MIN_AGE (default one hour, counted from state.failed_at, which the
database stamps). This gives the operator time to put it back with
pepsi-queue. Each
message it moves is logged as a warning with its sender, the stage it failed
at and the recorded error, which the bounce stage does not copy into the DSN;
the stage is kept as state.failed_stage. A member’s copy of a list post (a row
carrying state.list.member) is deleted instead: its envelope sender is
the list’s own bounce address, so a DSN would count this host’s fault against
the member. A post on its way to a list has the poster as its envelope sender
and is bounced like any other message. A message whose content came out of, or failed to open from, an
S/MIME EnvelopedData is deleted too (telemetry decrypt-dropped): a DSN would tell a
sender who modified the CBC ciphertext whether its padding was valid
(CBC padding and format oracles against unauthenticated S/MIME). Having moved rows, it wakes the dispatcher itself.
It connects through [pepsi-postgres] and is configured by
[pepsi-failure-bouncer]: BOUNCE_STAGE (required, the stage that stuck
messages are moved to) and MIN_AGE. It is not a stage. Started as
root, it continues as the pepsi service account, the role that owns the
queue, before connecting. Reference: pepsi-failure-bouncer(1).
88.2. Modes¶
service (the default, no
--once) — a long-lived process thatLISTENs on theworkqueue_failedchannel (fired by theworkqueue_failed_notifytrigger whenever a row becomes failed/timeout) and bounces each message once it isMIN_AGEold, waking for the oldest waiting failure when no notification comes first. It sweeps all already-stuck messages on start-up and on every reconnect, so nothing is missed, and runs untilSIGINT/SIGTERM. No shipped unit runs this mode.–once — sweep the currently-stuck messages a single time and exit. This is what
pepsi-failure-bouncer.timerruns, as thepepsiaccount, every ten minutes;pepsi.targetwants the timer. A test host that wants to keep failed messages for diagnosis masks it (systemctl mask --now pepsi-failure-bouncer.timer), since the target would start a merely disabled timer again.
--failed-only / --timeout-only restrict either mode to one of the two
terminal states.
Why a failure cannot loop through the bounce stage is part of the availability analysis (Availability); which DSNs the bounce stage then withholds is described under pepsi-stage-bounce.
88.3. See also¶
pepsi-dispatch, pepsi-stage-bounce, pepsi-queue, pepsi-status, pepsi-failure-bouncer(1), pepsi.conf(5).