72. pepsi-stage-list-post

Moderate a post to a mailing list, transform it, and fan it out to the members.

72.1. Role

pepsi-stage-list-post receives a message that pepsi-stage-list decided is a post to a list, and does two things to it in this order and with these names: a chain of rules decides whether the post is accepted, held, rejected or discarded, and then a pipeline of handlers decides how an accepted post is transformed and delivered. Moderation and processing are different steps with different vocabularies, and conflating them produces something that is not Mailman — the rule, chain, handler and pipeline names are a compatibility identifier that the REST API reports and that a list’s posting_chain and posting_pipeline name. Reference: pepsi-stage-list-post(1).

This subsystem is a reimplementation of GNU Mailman 3; see Mailing lists, which names what was taken from upstream.

72.2. Moderation

The default posting chain runs the DMARC mitigation check, the no-senders check, the moderator-password bypass (approved), the loop and banned-address rules, the site and per-list header matches, the emergency flag, and the member and non-member moderation rules. Then eight rules only record what they find — administrivia, implicit-destination, maximum-recipients, maximum-size, news-moderation, empty-subject, digests and suspicious-header — and a final any holds the post if one of them hit: eighteen rules and four terminal chains (accept, hold, reject, discard).

Warning

A correct moderator password bypasses the ban list. That is upstream’s behaviour and compatibility requires it, but it is a sharp edge: an Approved: header carrying the list’s moderator password jumps straight to accept, over the ban list, the header matches, the emergency flag and both moderation rules.

The approved password never survives. A moderator may approve a post with the password in an Approved: header or in the first line of the body; whichever form it arrived in, it is removed before the message is archived or delivered — in the header, in the plain-text body, and in the HTML alternative a mail client generated alongside it. A password that reaches a public archive is published permanently, and purging the archived copy does not recall the copies already in five thousand mailboxes. The rule strips what it matched, and the cleanse handler then deletes all four header names (Approve, Approved, X-Approve, X-Approved) unconditionally, which is the only thing that removes a wrong password.

72.3. Handlers

The pipeline is upstream’s, in upstream’s order: validate-authenticity, mime-delete, tagger, member-recipients, avoid-duplicates, cleanse, cleanse-dkim, cook-headers, subject-prefix, rfc-2369, to-archive, to-digest, to-usenet, after-delivery, acknowledge, dmarc, to-outgoing.

decorate and arc-sign are absent because upstream’s own default pipeline has them commented out, for upstream’s reasons: all decoration is done in delivery, and decoration in delivery would break a signature applied here. Both belong to pepsi-stage-list-deliver and the signing tail. to-usenet is an explicit, logged no-op — gateway_to_news stays settable because removing it would change the resource a REST client sees, but Pepsi does not gateway to NNTP, and a silent no-op would leave an operator believing their gateway works.

72.4. Headers written

Mailman’s X-Mailman-* namespace is renamed to X-Pepsi-List-* — Version, Rule-Hits, Rule-Misses, Approved-At, Hash-ID, Duplicate, Copy — because nothing in Postorius or HyperKitty reads any of it and the only consumer is a human reading a held post. Message-ID-Hash and its legacy X-Message-ID-Hash spelling keep their names: they carry no project’s name, every archive URL is derived from them, and they are the ones a third party is likely to have built something on.

X-Pepsi-List-Hops has no upstream counterpart and is the only one the software reads back in. It is the backstop against a subscription cycle through two or more lists — the loop rule catches only a message returning to the same list — and it travels in a header because a sibling row leaves over SMTP and re-enters as ordinary mail, where row state does not survive. Because a header is forgeable, pepsi-ingress strips an inbound copy from an unauthenticated sender.

72.5. The fan-out

Every regular member with delivery enabled and not on digest gets one row, unconditionally, each with its own single recipient and its own VERP envelope sender <list>-bounces+<local>=<domain>@<host> — which is what lets the bounce path attribute a failure to exactly one subscription without parsing the bounce. What crosses the wire is the recipient list, not the message: headers and body are copied inside the database, so a five-thousand-member post sends one message and five thousand small JSON objects to PostgreSQL.

The fan-out is batched: at most [pepsi-list] LIST_FANOUT_BATCH rows (default 256) are materialised at a time and the message pauses between batches with the remaining recipients in its own state. Peak transient storage is the batch times the message size rather than the membership times the message size — a 5 000-member post with a 5 MB attachment peaks near 1.3 GB instead of 25 GB — and pepsi-queue shows a fan-out in progress rather than a database that has suddenly grown. The recipient list is snapshotted at the first batch, so an unsubscribe halfway through a large send can neither skip nor double a neighbour, and the source row is deleted only by the batch that finishes the list, so a crash replays that batch and nothing more.

72.6. The posting autoresponder

autorespond_postings is the third of the three autoresponders, on the same grace-period machinery as -request and -owner, and the one worth reading about: it fires on the posting address, so a misconfigured one answers everybody who writes to the list. On a posting address respond_and_discard means the post is answered and not distributed — occasionally what an owner wants (a retired list that answers “we have moved”) and never what they want by accident, so the discard is logged at warn and the owner console labels the value plainly. Two rules that are not obvious: an unrecognised value is ``none`` (a response action nobody can spell is not a licence to mail a membership), and a post that could not be answered is still delivered — when the autoresponder declines, respond_and_discard does not discard, because a list that silently swallowed every message from an autoresponder would be the worse failure. Every autoresponse passes the shared RFC 3834 rule set (pepsi_common::autoreply) first; see pepsi-stage-vacation.

72.7. Configuration

[stage-<name>]: PROGRAM = pepsi-stage-list-post, DELIVERY_STAGE (required, and it must run pepsi-stage-list-deliver), BOUNCE_STAGE (required — here it means what it means everywhere except pepsi-stage-list), RESPONSE_STAGE (required, and it must be the signing tail: the delivery stage accepts only a member’s copy from the fan-out, so a notice sent there never goes out), REMOVE_DKIM_HEADERS (default no), SITE_HEADER_MATCH_CHAIN (default hold, site-wide so that a list owner cannot turn the site’s spam rules into a discard), and the two availability limits SENDER_POSTS_PER_HOUR (default 30: a sender’s posts to one list beyond that in an hour are held for the moderator instead of sent to every member) and MAX_HELD_MESSAGES (default 1000 per list: a post that would be held beyond that is discarded, so an open list cannot be used to fill the disk; the new post is dropped rather than the oldest, so a flood cannot flush out what a moderator has not yet seen). pepsi-setup checks all three targets and refuses a ``NEXT_STAGE``: every path out of this stage is a terminal.

REMOVE_DKIM_HEADERS is off by default because a list that rewrites nothing leaves a valid signature and stripping it discards real evidence; turn it on for lists that add a footer or a subject prefix, where the surviving signature would fail — and a failing signature is worse than none, because a failure is evidence of tampering and an absence is not.

72.8. State

• Inputs: state.list.id (written by pepsi-stage-list), state.auth (the SPF/DKIM/DMARC verdicts ingress recorded), and state.list.pending_rcpt on a resumed fan-out.

• Outputs: a state.list descriptor on each sibling row — the member id, their language, their unsubscribe serial and the duplicate flag.

• Transitions: finish (the last batch of a fan-out, a hold, a discard), finish with a side clone (a rejection’s DSN), or pause (a fan-out with recipients remaining).

72.9. See also

Mailing lists, Archives, pepsi-stage-list, pepsi-stage-list-deliver, pepsi-list, pepsi-archive, pepsi-stage-list-post(1).