82. pepsi-stage-list-post

Moderate a post to a mailing list, transform it, and fan it out to the members.

82.1. Role

pepsi-stage-list-post receives a message that pepsi-stage-list decided is a post to a list, and does two things to it in this order and with these names: a chain of rules decides whether the post is accepted, held, rejected or discarded, and then a pipeline of handlers decides how an accepted post is transformed and delivered. Moderation and processing are different steps with different vocabularies, and conflating them produces something that is not Mailman. The rule, chain, handler and pipeline names are a compatibility identifier that the REST API reports and that a list’s posting_chain and posting_pipeline name. Reference: pepsi-stage-list-post(1).

This subsystem is a reimplementation of GNU Mailman 3; see Mailing lists, which names what was taken from upstream.

82.2. Moderation

The default posting chain runs the DMARC mitigation check, the no-senders check, the moderator-password bypass (approved), the loop and banned-address rules, Pepsi’s dmarc-failure rule, the site and per-list header matches, the emergency flag, and the member and non-member moderation rules. Then eight rules only record what they find (administrivia, implicit-dest, max-recipients, max-size, news-moderation, no-subject, digests and suspicious-header), and a final any holds the post if one of them hit. In all, these are upstream’s eighteen rules, one of Pepsi’s, and four terminal chains (accept, hold, reject, discard).

A post that failed DMARC is held. A list re-signs and re-sends what it distributes, so a post whose From: domain did not authenticate it would otherwise reach every member looking authenticated, and the list would launder the forgery. dmarc-failure therefore holds a post whose ingress DMARC verdict is fail on a list with hold_dmarc_failures on (the default for lists Pepsi creates; imported lists get no, see Settings Pepsi invented), whatever policy the domain publishes and whether or not the poster is a member. A failure that a trusted ARC sealer vouched for (ARC_TRUSTED_SEALERS) counts as a pass. A post with no verdict is not held, and neither is one submitted here by an authenticated sender (state.local_origin), whatever verdict ingress recorded beside it. A moderator’s Approved: password releases a held post, as it releases every hold.

Warning

A correct moderator password bypasses the ban list. That is upstream’s behaviour and compatibility requires it, but it is a sharp edge: an Approved: header carrying the list’s moderator password jumps straight to accept, over the ban list, the header matches, the emergency flag and both moderation rules.

The approved password never survives. A moderator may approve a post with the password in an Approved: header or in the first line of the body; whichever form it arrived in, it is removed before the message is archived or delivered: from the header, from the plain-text body, and from the HTML alternative that a mail client generated alongside it. A password that reaches a public archive is published permanently, and purging the archived copy does not recall the copies already in five thousand mailboxes. The rule strips what it matched, and the cleanse handler then deletes all four header names (Approve, Approved, X-Approve, X-Approved) unconditionally, which is the only thing that removes a wrong password. If the password is right but its stored hash was made with weaker Argon2id parameters than new hashes get, the password is hashed again and stored, just as a console sign-in upgrades its own hash; the header is the one moment when the plaintext is seen.

82.3. Handlers

The pipeline is upstream’s, in upstream’s order: validate-authenticity, mime-delete, tagger, member-recipients, avoid-duplicates, cleanse, cleanse-dkim, cook-headers, subject-prefix, rfc-2369, to-archive, to-digest, to-usenet, after-delivery, acknowledge, dmarc, to-outgoing.

A post from which mime-delete removes everything goes no further down the pipeline. The list’s filter_action then decides what happens to it (When content filtering leaves nothing).

decorate and arc-sign are absent because upstream’s own default pipeline has them commented out, for upstream’s reasons: all decoration is done in delivery, and decoration in delivery would break a signature applied here. Both belong to pepsi-stage-list-deliver and the signing tail. to-usenet is an explicit, logged no-op. gateway_to_news stays settable because removing it would change the resource a REST client sees, but Pepsi does not gateway to NNTP, and a silent no-op would leave an operator believing their gateway works.

82.4. Headers written

Mailman’s X-Mailman-* namespace is renamed to X-Pepsi-List-*, because nothing in Postorius or HyperKitty reads any of it and the only consumer is a human reading a held post. The renamed fields are Version, Rule-Hits, Rule-Misses, Approved-At, Hash-ID, Duplicate and Copy; upstream’s X-Content-Filtered-By becomes X-Pepsi-List-Content-Filtered-By. Message-ID-Hash and its legacy X-Message-ID-Hash spelling keep their names: they carry no project’s name, every archive URL is derived from them, and they are the ones that a third party is likely to have built something on.

X-Pepsi-List-Hops has no upstream counterpart and is the only one that the software reads back in. It is the backstop against a subscription cycle through two or more lists, since the loop rule catches only a message returning to the same list. It travels in a header because a sibling row leaves over SMTP and re-enters as ordinary mail, where row state does not survive. Because a header is forgeable, pepsi-ingress strips an inbound copy from mail that did not arrive over an authenticated or local submission path.

82.5. The fan-out

Every regular member with delivery enabled and not on digest gets one row, unconditionally. Each row has its own single recipient and its own VERP envelope sender <list>-bounces+<local>=<domain>=<code>@<host> (the code keyed with [pepsi-list] VERP_SECRET). That is what lets the bounce path attribute a failure to exactly one subscription without parsing the bounce, and refuse to attribute one to an address that somebody made up. Each row is held paused for a random time of up to [pepsi] REFLECTION_HOLD_MAX (30 s), drawn per copy, before the delivery stage takes it up, since a post is reflected mail (Unknown recipients and backscatter). What crosses the wire is the recipient list, not the message: the headers are copied inside the database and the body is shared by reference, so a five-thousand-member post sends one message and five thousand small JSON objects to PostgreSQL.

The fan-out is batched: at most [pepsi-list] LIST_FANOUT_BATCH rows (default 256) are materialised at a time and the message pauses between batches with the remaining recipients in its own state. Every sibling row references one stored copy of the cooked body in pepsi.workqueue_body, so the fan-out stores a 5 MB attachment once rather than once per member; a member’s copy gets a body of its own only when pepsi-stage-list-deliver rewrites it. The batching bounds how many rows each step adds, and pepsi-queue shows a fan-out in progress rather than a queue that has suddenly grown. When the queue is at one of the [pepsi] admission limits, the stage emits an empty batch and looks again after QUEUE_THROTTLE_DELAY. The recipient list is snapshotted at the first batch, so an unsubscribe halfway through a large send can neither skip nor double a neighbour, and the source row is deleted only by the batch that finishes the list, so a crash replays that batch and nothing more.

A digest issue takes the same path from its second step: pepsi-list digest injects it here (as [pepsi-list] RELEASE_STAGE) with its digest members already in state as the snapshot. It is therefore fanned out like a paused post and runs none of the posting chain: no moderation, archive, digest append or counters. Each sibling is marked so that pepsi-stage-list-deliver does not decorate it.

82.6. The posting autoresponder

autorespond_postings answers posts on the per-list grace-period machinery (autoresponse_grace_period) that also rate-limits the command stage’s results mail. (The sibling attributes autorespond_requests and autorespond_owner are stored and reported over REST, but no stage acts on them.) Of the three, autorespond_postings is the one worth reading about. It fires on the posting address, so a misconfigured autoresponder answers everybody who writes to the list. On a posting address respond_and_discard means the post is answered and not distributed — occasionally what an owner wants (a retired list that answers “we have moved”) and never what they want by accident, so the discard is logged at warn and the owner console labels the value plainly. Two rules that are not obvious: an unrecognised value is ``none`` (a response action nobody can spell is not a licence to mail a membership), and a post that could not be answered is still delivered: when the autoresponder declines, respond_and_discard does not discard, because a list that silently dropped every message from an autoresponder would be the worse failure. Every autoresponse passes the shared RFC 3834 rule set (pepsi_common::autoreply) first; see pepsi-stage-vacation.

82.7. Configuration

[stage-<name>]: PROGRAM = pepsi-stage-list-post, DELIVERY_STAGE (required, and it must run pepsi-stage-list-deliver), BOUNCE_STAGE (required — here it means what it means everywhere except pepsi-stage-list), RESPONSE_STAGE (required, and it must be the signing tail: the delivery stage accepts only a member’s copy from the fan-out, so a notice sent there never goes out), REMOVE_DKIM_HEADERS (default no), SITE_HEADER_MATCH_CHAIN (default hold, site-wide so that a list owner cannot turn the site’s spam rules into a discard), and the two availability limits SENDER_POSTS_PER_HOUR (default 30: a sender’s posts to one list beyond that in an hour are held for the moderator instead of sent to every member) and MAX_HELD_MESSAGES (default 1000 per list: a post that would be held beyond that is discarded, so an open list cannot be used to fill the disk; the new post is dropped rather than the oldest, so a flood cannot flush out what a moderator has not yet seen). These limits, the [pepsi-list] MAX_LIST_HOPS ceiling (a post that has passed through that many lists is held rather than delivered) and the fan-out’s admission throttle are the mailing-list entries of Availability; Security model (“Resource limits”) says where each is enforced. pepsi-setup checks all three targets and refuses a ``NEXT_STAGE``: every path out of this stage is a terminal.

REMOVE_DKIM_HEADERS is off by default because a list that rewrites nothing leaves a valid signature and stripping it discards real evidence; turn it on for lists that add a footer or a subject prefix, where the surviving signature would fail. A failing signature is worse than none, because a failure is evidence of tampering and an absence is not.

82.8. State

  • Inputs: state.list.id (written by pepsi-stage-list), state.auth (the SPF/DKIM/DMARC verdicts that ingress recorded), state.list.pending_rcpt on a resumed fan-out, and state.list.moderator_approved on a held post a moderator released.

  • Outputs: a state.list descriptor on each sibling row — the member id, their language, their unsubscribe serial and the duplicate flag.

  • Transitions: finish (the last batch of a fan-out, a hold, a discard), finish with a side clone (a rejection’s DSN), or pause (a fan-out with recipients remaining).

82.9. See also

Mailing lists, Archives, pepsi-stage-list, pepsi-stage-list-deliver, pepsi-list, pepsi-archive, pepsi-stage-list-post(1).