75. pepsi-stage-list-bounce

Attribute an inbound bounce to the subscription it concerns, and score it.

75.1. Role

pepsi-stage-list-bounce consumes bounces: a delivery failure that came back to a list’s -bounces address is attributed to the member it concerns, scored, and — once the score crosses the list’s threshold — that member’s delivery is disabled, they are warned, and eventually they are unsubscribed. Reference: pepsi-stage-list-bounce(1).

Warning

Three things in this tree are called “bounce” and they are not the same thing. This one reads an inbound failure report. pepsi-stage-bounce generates a DSN for a message Pepsi itself could not deliver. pepsi-failure-bouncer rescues messages the pipeline gave up on and hands them to that generator. A configuration that points this stage’s NEXT_STAGE at the generator answers a bounce with a bounce, and pepsi-setup refuses it.

This subsystem is a reimplementation of GNU Mailman 3; see Mailing lists, which names what was taken from upstream.

75.2. Attribution

Four steps, in decreasing order of certainty. The first two are exact, because Pepsi wrote the address they read.

  1. The VERP envelope. Every copy a list sends carries an envelope sender of <list>-bounces+<local>=<domain>@<host>, so a bounce of it names the member unambiguously. The <list>-bounces part must match this list’s own bounces mailbox before the address is believed: without that check anybody could score a bounce against any member of any list by sending mail to an address they made up. A VERP match still has to look like a failure — a temporary failure is ignored outright, and a VERP’d message with no recognised failure at all is forwarded rather than scored, because it may be a vacation reply or a virus scanner’s report that arrived here only because that is what was on the envelope.

  2. A probe token (<list>-bounces+<token>@<host>), when BOUNCE_PROBES is on.

  3. The heuristic detectors, over the whole message.

  4. Nothing resolved — the bounce is wrapped up and forwarded to a human.

75.3. Scoring

The machine is GNU Mailman 3’s, reproduced step for step, because every attribute it reads is visible through the REST API and a site’s documentation says what each one does: a missing list or a non-member is dropped; a probe bounce disables immediately, without scoring, because a probe bounce is decisive; a member already disabled by bounces is a residual bounce and is dropped; a second bounce for the same member on the same calendar day updates the timestamp only; a first bounce, or one older than bounce_info_stale_after, resets the score to 1 and otherwise it is incremented; the owners are told if bounce_notify_owner_on_bounce_increment is set; and at or above bounce_score_threshold a probe is sent if probes are on (which resets the score to zero) and otherwise the score is zeroed and delivery disabled.

The at-most-one-increment-per-day rule is why a bounce storm cannot purge a list in an afternoon: a member with the default threshold of 5 needs failures on five different days.

75.4. Warning and removal

Disabling a member is not unsubscribing them. Two periodic passes, run by pepsi-list tasks --once from a systemd timer (see pepsi-list), finish the story: a member disabled by bounces who has had fewer than bounce_you_are_disabled_warnings warnings, the last of them at least bounce_you_are_disabled_warnings_interval ago, is sent one; and once the warnings have run out and the interval has elapsed again the member is unsubscribed, with the owners told if bounce_notify_owner_on_removal is set. A member disabled by themselves or by a moderator is never touched by either pass.

Setting bounce_you_are_disabled_warnings to zero removes a disabled member on the next sweep with no warning mail at all. That is upstream’s documented behaviour and it is deliberate, not an edge case.

The notices these passes send are injected at [pepsi-list] NOTICE_STAGE, because the passes run from a timer rather than from a stage. pepsi-setup refuses a configuration that runs this stage without it: with no stage to inject into, a member would be warned three times in the database and never once by mail.

75.5. Probes

With [pepsi-list] BOUNCE_PROBES = yes, crossing the threshold sends the member a probe instead of disabling them: a message addressed to them alone, from an envelope sender carrying a one-use token, with the triggering bounce attached, and their score reset to zero while it is outstanding. If the probe bounces, the token identifies them exactly and delivery is disabled at once; if it does not, nothing happens and they keep receiving mail.

Probes are off by default, as upstream. The trade is one more message to an address that is probably dead, against not disabling a member whose provider had one bad afternoon. A probe token is good for ten days and carries no Precedence: bulk header — a probe exists in order to be bounced, and that header invites some systems not to bounce it.

75.6. Unrecognised bounces

A bounce nothing recognised is wrapped in an explanatory notice with the original attached verbatim as message/rfc822 and sent to whoever the list’s forward_unrecognized_bounces_to names: administrators (the default), site_owner or discard.

This is a feature rather than a fallback. Every bounce that falls through is a format the corpus does not have, and the person who receives it can send the sample to the Pepsi maintainers so that the detector can be written. Under every disposition — discard included — the fall-through is counted in pepsi.event_log as list.bounce.unrecognized, because how often it happens is what says whether a site needs more detectors at all.

75.7. Recognised bounce formats

Seventeen detectors, tried in a fixed order with the first match winning, ported from flufl.bounce (Apache-2.0, copyright Barry Warsaw) and each tested against that library’s own fixtures: dsn, exim, sina, yahoo, yale, smtp32, postfix, qmail, groupwise, microsoft, caiwireless, exchange, netscape, aol, simplematch, simplewarning, llnl.

Exactly one of them — dsn, RFC 3464 — is a standard; the rest name a vendor or a site. In practice they matter far less here than upstream, because Pepsi VERPs every copy of every post, so a bounce of a message Pepsi sent is attributed by its envelope without any of this being consulted. The detectors are what reads a bounce that reached a list some other way.

A message no detector recognises is not a bounce as far as this stage is concerned, and is forwarded rather than guessed at. That distinction is worth knowing when reading a log: “Pepsi did not recognise this” and “Pepsi decided this was not a failure” are different sentences.

75.8. Configuration

[stage-<name>]: PROGRAM = pepsi-stage-list-bounce, NEXT_STAGE (where a forwarded non-bounce goes — not the DSN generator) and RESPONSE_STAGE, both required. [pepsi-list] supplies NOTICE_STAGE, BOUNCE_PROBES and BOUNCE_EVENT_RETENTION; the per-list bounce_* attributes are in the database. See pepsi-list and Mailing lists.

75.9. State

  • Inputs: state.list — the list id, the -bounces role and any VERP detail the address carried — written by pepsi-stage-list.

  • Outputs: bounce events and member delivery status in the database; notices and probes injected as new messages.

  • Transitions: finish (a scored bounce is consumed) or advance (a forwarded non-bounce).

75.10. See also

Mailing lists, pepsi-stage-list, pepsi-stage-bounce, pepsi-failure-bouncer, pepsi-list, pepsi-stage-list-bounce(1).