85. pepsi-stage-list-bounce¶
Attribute an inbound bounce to the subscription it concerns, and score it.
85.1. Role¶
pepsi-stage-list-bounce consumes bounces. When a delivery failure comes
back to a list’s -bounces address, the stage attributes it to the member
that it concerns and scores it. Once the score crosses the list’s threshold,
that member’s delivery is disabled, they are warned, and eventually they are
unsubscribed.
Reference: pepsi-stage-list-bounce(1).
Warning
Three things in this tree are called “bounce” and they are not the same
thing. This one reads an inbound failure report.
pepsi-stage-bounce generates a DSN for a message that Pepsi itself
could not deliver. pepsi-failure-bouncer rescues messages that the
pipeline gave up on and hands them to that generator. A configuration that points this
stage’s NEXT_STAGE at the generator answers a bounce with a bounce, and
pepsi-setup refuses it.
This subsystem is a reimplementation of GNU Mailman 3; see Mailing lists, which names what was taken from upstream.
85.2. Attribution¶
Attribution takes four steps, in decreasing order of certainty. The first two are exact, because Pepsi wrote the address that they read.
The VERP envelope. Every copy that a list sends carries an envelope sender of
<list>-bounces+<local>=<domain>=<code>@<host>, so a bounce of it names the member unambiguously. The<list>-bouncespart must match this list’s own bounces mailbox, and the code must verify before the address is believed. The code is eight base32 characters of an HMAC keyed with[pepsi-list] VERP_SECRETover the mailbox, the host and the member. Without the first check, anybody could score a bounce against a member of a list that they are not on; without the second, anybody who knows a list and a member could write the address and have five days of forged reports disable that member. A member-shaped address whose code is missing or wrong is dropped unscored (logged, counted aslist-bounce-verp-rejected) and not forwarded either. A VERP match still has to look like a failure: a temporary failure is ignored outright, and a VERP’d message with no recognised failure at all is forwarded rather than scored, because it may be a vacation reply or a virus scanner’s report that arrived here only because that is what was on the envelope.A probe token (
<list>-bounces+<token>@<host>), whenBOUNCE_PROBESis on.The heuristic detectors, over the whole message.
Nothing resolved: the bounce is wrapped up and forwarded to a human.
85.3. Scoring¶
The machine is GNU Mailman 3’s, reproduced step for step, because every attribute
that it reads is visible through the REST API and a site’s documentation says what each
one does: a list whose process_bounces is off drops every bounce unread; a
missing list or a non-member is dropped; a probe bounce disables the member
immediately, without scoring, because a probe bounce is decisive; a member already
disabled by bounces is a residual bounce and is dropped; a second bounce for the
same member on the same calendar day (UTC) updates the timestamp only; a first
bounce, or one older than bounce_info_stale_after, resets the score to 1,
and any other bounce increments it; the owners are told if
bounce_notify_owner_on_bounce_increment is set; and at or above
bounce_score_threshold, a probe is sent if probes are on (which resets the
score to zero), and otherwise the score is zeroed and delivery is disabled.
The at-most-one-increment-per-day rule is why a bounce storm cannot purge a list in an afternoon: a member with the default threshold of 5 needs failures on five different days.
85.4. Warning and removal¶
Disabling a member is not unsubscribing them. Two periodic passes, run by
pepsi-list tasks --once from a systemd timer (see pepsi-list),
complete the process: a member disabled by bounces who has had fewer than
bounce_you_are_disabled_warnings warnings, the last of them at least
bounce_you_are_disabled_warnings_interval ago, is sent one; and once the
warnings have run out and the interval has elapsed again the member is
unsubscribed, with the owners told if bounce_notify_owner_on_removal is set. A
member disabled by themselves or by a moderator is never touched by either pass.
Setting bounce_you_are_disabled_warnings to zero removes a disabled member
on the next sweep with no warning mail at all. That is upstream’s documented
behaviour and it is deliberate, not an edge case.
The notices these passes send are injected at [pepsi-list] NOTICE_STAGE,
because the passes run from a timer rather than from a stage. pepsi-setup
refuses a configuration that runs this stage without it: with no stage to inject
into, a member would be warned three times in the database and never once by mail.
85.5. Probes¶
With [pepsi-list] BOUNCE_PROBES = yes, a member who crosses the
threshold is sent a probe instead of being disabled: a message addressed to
them alone, from an envelope sender that carries a one-use token, with the
triggering bounce attached. Their score is reset to zero while the probe is
outstanding. If the probe
bounces, the token identifies them exactly and delivery is disabled at once; if it
does not, nothing happens and they keep receiving mail.
Probes are on by default (upstream’s are off): the probe’s token is the one
thing about a bounce that nobody can forge, so it is the final check before a
member is disabled on reports that somebody else sent. The trade is one more message to an
address that is probably dead, against not disabling a member whose provider had
one bad afternoon; BOUNCE_PROBES = no restores upstream’s behaviour. A probe
token is good for ten days and carries no
Precedence: bulk header. A probe exists in order to be bounced, and that
header invites some systems not to bounce it.
85.6. Unrecognised bounces¶
A bounce that nothing recognised is wrapped in an explanatory notice with the original
attached verbatim as message/rfc822 and sent to whoever the list’s
forward_unrecognized_bounces_to names: administrators (the default),
site_owner or discard.
This is a feature rather than a fallback. Every bounce that falls through is
in a format that the corpus does not have, and the person who receives it can
send the sample to the Pepsi maintainers so that the detector can be written.
Under every disposition, discard included, the fall-through is counted in
pepsi.event_log as list.bounce.unrecognized, because its frequency shows
whether a site needs more detectors at all.
85.7. Recognised bounce formats¶
There are seventeen detectors, tried in a fixed order with the first match
winning. They are ported from flufl.bounce (Apache-2.0, copyright Barry
Warsaw), and each is tested against that library’s own fixtures: dsn, exim, sina, yahoo,
yale, smtp32, postfix, qmail, groupwise, microsoft,
caiwireless, exchange, netscape, aol, simplematch,
simplewarning, llnl.
Exactly one of them, dsn (RFC 3464), is a standard; the rest name a vendor
or a site. In practice they matter far less here than upstream, because Pepsi
VERPs every copy of every post, so a bounce of a message that Pepsi sent is
attributed by its envelope without any of this being consulted. The detectors
read a bounce that reached a list some other way.
A message that no detector recognises is not a bounce as far as this stage is concerned, and is forwarded rather than guessed at. That distinction is worth knowing when reading a log: “Pepsi did not recognise this” and “Pepsi decided this was not a failure” are different sentences.
85.8. Configuration¶
[stage-<name>]: PROGRAM = pepsi-stage-list-bounce, NEXT_STAGE (where a
forwarded non-bounce goes; not the DSN generator) and RESPONSE_STAGE,
both required.
[pepsi-list] supplies VERP_SECRET (required; the stage refuses to start
without it), NOTICE_STAGE, BOUNCE_PROBES (default on) and
BOUNCE_EVENT_RETENTION; the per-list bounce_* attributes are in the
database. See pepsi-list and Mailing lists.
85.9. State¶
Inputs:
state.list.id, written by pepsi-stage-list, and the envelopercpt_to(then theTo,Delivered-To,Envelope-ToandApparently-Toheaders). The VERP detail or probe token is decoded from the address itself, and believed only when its<list>-bouncespart is this list’s own and, for a member detail, its authentication code verifies.Outputs: bounce events and member delivery status in the database; notices and probes injected as new messages.
Transitions: finish, always. A bounce that is scored, ignored or dropped is consumed; one that is forwarded to a human is consumed too, because the forward is a new notice injected at
NEXT_STAGE.
85.10. See also¶
Mailing lists, pepsi-stage-list, pepsi-stage-bounce, pepsi-failure-bouncer, pepsi-list, pepsi-stage-list-bounce(1).