88. pepsi-failure-bouncer

Funnel failed/timeout messages to the bounce stage.

88.1. Role

When its stage gives up, a message in the post-ingress pipeline ends up failed or timeout. It is failed when a stage gave a permanent error, an error outlived MAX_LIFETIME at a stage with no BOUNCE_STAGE, the message crashed its worker three times, or it was pending at a stage that the configuration no longer names. It is timeout when it held its worker past MAX_RUNTIME three times or, in a pepsi-stage-relay-to-* stage, was still being killed at MAX_LIFETIME. The dispatcher never re-queues such a row, so without operator action the message is stuck in pepsi.workqueue and the sender is never notified.

pepsi-failure-bouncer moves each stuck message to the configured [pepsi-failure-bouncer] BOUNCE_STAGE and resets it to pending, so the dispatcher runs the bounce stage (pepsi-stage-bounce), which emits a delivery-status notification as the sender’s RFC 3461 NOTIFY requests. A message already at the bounce stage is left alone, so a bounce failing at that stage does not loop; one failing downstream (an unrelayable null-sender DSN) is moved back and simply dropped (a bounce is never re-bounced). BOUNCE_STAGE must therefore lead to a real bounce stage: pepsi-setup refuses one that names no stage and warns about one that does not run pepsi-stage-bounce directly.

It waits first: a message is moved only once it has been failed for MIN_AGE (default one hour, counted from state.failed_at, which the database stamps). This gives the operator time to put it back with pepsi-queue. Each message it moves is logged as a warning with its sender, the stage it failed at and the recorded error, which the bounce stage does not copy into the DSN; the stage is kept as state.failed_stage. A member’s copy of a list post (a row carrying state.list.member) is deleted instead: its envelope sender is the list’s own bounce address, so a DSN would count this host’s fault against the member. A post on its way to a list has the poster as its envelope sender and is bounced like any other message. A message whose content came out of, or failed to open from, an S/MIME EnvelopedData is deleted too (telemetry decrypt-dropped): a DSN would tell a sender who modified the CBC ciphertext whether its padding was valid (CBC padding and format oracles against unauthenticated S/MIME). Having moved rows, it wakes the dispatcher itself.

It connects through [pepsi-postgres] and is configured by [pepsi-failure-bouncer]: BOUNCE_STAGE (required, the stage that stuck messages are moved to) and MIN_AGE. It is not a stage. Started as root, it continues as the pepsi service account, the role that owns the queue, before connecting. Reference: pepsi-failure-bouncer(1).

88.2. Modes

  • service (the default, no --once) — a long-lived process that LISTENs on the workqueue_failed channel (fired by the workqueue_failed_notify trigger whenever a row becomes failed/timeout) and bounces each message once it is MIN_AGE old, waking for the oldest waiting failure when no notification comes first. It sweeps all already-stuck messages on start-up and on every reconnect, so nothing is missed, and runs until SIGINT/SIGTERM. No shipped unit runs this mode.

  • –once — sweep the currently-stuck messages a single time and exit. This is what pepsi-failure-bouncer.timer runs, as the pepsi account, every ten minutes; pepsi.target wants the timer. A test host that wants to keep failed messages for diagnosis masks it (systemctl mask --now pepsi-failure-bouncer.timer), since the target would start a merely disabled timer again.

--failed-only / --timeout-only restrict either mode to one of the two terminal states.

Why a failure cannot loop through the bounce stage is part of the availability analysis (Availability); which DSNs the bounce stage then withholds is described under pepsi-stage-bounce.

88.3. See also

pepsi-dispatch, pepsi-stage-bounce, pepsi-queue, pepsi-status, pepsi-failure-bouncer(1), pepsi.conf(5).