95.1.21. pepsi-stage-list-post

moderate, transform and fan out a post to a mailing list

Manual section:

1

95.1.21.1.1. Name

pepsi-stage-list-post - the mailing-list posting stage of the Pepsi pipeline.

95.1.21.1.2. Synopsis

pepsi-stage-list-post [GLOBAL-OPTIONS] worker

95.1.21.1.3. Description

pepsi-stage-list-post is a stage program run by pepsi-dispatch(1). A message that pepsi-stage-list(1) decided is a post to a list arrives here, and two things happen to it, in this order and with these names:

  1. a chain of rules decides whether the post is accepted, held, rejected or discarded;

  2. a pipeline of handlers decides how an accepted post is transformed and delivered.

Moderation and processing are different stages with different vocabularies, and conflating them produces something that is not Mailman. The rule, chain, handler and pipeline names are a compatibility identifier: the REST API reports them and a list’s posting_chain and posting_pipeline name them.

The stage reads the database once, at the start; computes everything in memory; and writes once, in a single transaction that also takes the message’s own terminal. A crash anywhere in the middle replays the whole thing.

95.1.21.1.4. Chains and rules

The default posting chain runs, in order: the DMARC mitigation check, the no-senders check, the moderator-password bypass (approved), the loop and banned-address rules, Pepsi’s dmarc-failure rule, the site and per-list header matches, the emergency flag, and the member and non-member moderation rules. Then eight rules only record what they find, and a final any holds the post if one of them hit. These eight are administrivia, implicit-dest, max-recipients, max-size, news-moderation, no-subject, digests and suspicious-header. Upstream’s eighteen rules are the seventeen named here and truth, which always matches. Pepsi adds one rule, dmarc-failure, and there are four terminal chains (accept, hold, reject, discard).

The DMARC mitigation check does not look the policy up again: it uses the policy that ingress recorded for the From: domain (state.auth.dmarc_policy). A domain publishing quarantine or reject is mitigated, one publishing none or no record at all is not, and a post for which nothing was recorded (a local submission) is mitigated, as upstream does when its lookup fails. dmarc_mitigate_unconditionally mitigates every post.

dmarc-failure holds a post whose From: domain failed DMARC at intake (state.auth.dmarc is fail) on a list whose hold_dmarc_failures setting is on. That setting is on by default for every list that Pepsi creates; lists imported from GNU Mailman are given no. It holds whatever policy the domain publishes and whether or not the poster is a member, with the reason “The message failed DMARC”. A failure that a trusted ARC sealer vouched for at ingress (state.auth.arc_override) counts as a pass, and a post submitted here by an authenticated sender (state.local_origin) is never held by this rule, whatever verdict ingress recorded beside it: SMTP AUTH proved the sender, and a roaming member’s client is in nobody’s SPF record. A list re-signs and re-sends every post it distributes, so without this rule a forged post would reach every member looking authenticated. Like every other hold, a moderator’s Approved: password releases it.

A correct moderator password bypasses the ban list. This is upstream’s behaviour and compatibility requires it, but it is a sharp edge and it is said here rather than three chapters away: an Approved: header carrying the list’s moderator password jumps straight to accept, over the ban list, the header matches, the emergency flag and both moderation rules.

95.1.21.1.5. The approved-password invariant

A moderator may approve a post by putting the list’s moderator password in an Approved: header or in the first line of the body. Whichever form it arrived in, it is removed before the message is archived or delivered: from the header, from the plain-text body, and from the HTML alternative that a mail client generated alongside it. A password that survives into a public archive is published permanently, and purging the archived message does not recall the copies already in five thousand mailboxes.

The rule strips what it matched; cleanse then deletes all four header names (Approve, Approved, X-Approve, X-Approved) unconditionally, which is the only thing that removes a wrong password.

When a password verifies against a stored hash that was made with weaker Argon2id parameters than new hashes get, the stage hashes it again with the current parameters and stores the result, as a console or member sign-in upgrades its own. It stores it only while the stored hash is still the one that it verified against. The Approved: header is the only moment at which the stage sees the plaintext.

95.1.21.1.6. Handlers

The pipeline is upstream’s, in upstream’s order: validate-authenticity, mime-delete, tagger, member-recipients, avoid-duplicates, cleanse, cleanse-dkim, cook-headers, subject-prefix, rfc-2369, to-archive, to-digest, to-usenet, after-delivery, acknowledge, dmarc, to-outgoing.

A post from which mime-delete removes everything goes no further down the pipeline, and the list’s filter_action decides what happens to it. Every action works from the post as it arrived:

hold

The post waits for a moderator like any held post, and the moderators and the sender are told, subject to admin_immed_notify and respond_to_post_requests.

preserve

The post is kept the same way, but the sender is not told. Moderators find it among the list’s held messages, in the owner console and at /lists/<id>/held of the REST API, with a reason that says it was preserved. max_days_to_hold applies to it too.

reject

The sender gets a DSN that gives the reason.

forward

The list’s owners get a notice with the post attached, and the sender is not told. A list with no owner keeps the post as with preserve.

discard

The post is deleted and nobody is told. This is the default.

A moderator who releases a post held by hold or preserve has decided, so the post is delivered as it arrived, without the filter. A post that was held for another reason still passes the filter, and if nothing is left of it under reject or discard, the owners get it as with forward, because a released post has no sender to tell. accept and defer are refused for a new setting. A list that already carries one of them behaves as with hold. If the moderation queue is at MAX_HELD_MESSAGES, hold and preserve cannot keep the post, and the sender gets a DSN instead.

decorate and arc-sign are absent because upstream’s own default pipeline has them commented out, with upstream’s reasons: all decoration is done in delivery, and decoration in delivery would break a signature applied here. Both are pepsi-stage-list-deliver(1)’s and the signing tail’s.

to-usenet is an explicit, logged no-op: gateway_to_news stays settable because removing it would change the resource a REST client sees, but Pepsi does not gateway to NNTP, and a silent no-op would leave an operator believing their gateway works.

95.1.21.1.7. Headers written

Mailman’s X-Mailman-* namespace is renamed. Nothing in Postorius or HyperKitty reads any of it, and the only consumer of any of them is a human reading a held post:

Mailman 3

Pepsi

X-Mailman-Version

X-Pepsi-List-Version

X-Mailman-Rule-Hits

X-Pepsi-List-Rule-Hits

X-Mailman-Rule-Misses

X-Pepsi-List-Rule-Misses

X-Mailman-Approved-At

X-Pepsi-List-Approved-At

X-Mailman-Hash-ID

X-Pepsi-List-Hash-ID

X-Mailman-Duplicate

X-Pepsi-List-Duplicate

X-Mailman-Copy

X-Pepsi-List-Copy

X-Content-Filtered-By

X-Pepsi-List-Content-Filtered-By

(no counterpart)

X-Pepsi-List-Hops

X-Message-ID-Hash

X-Message-ID-Hash (unchanged)

Message-ID-Hash and its legacy X- spelling keep their names: they carry no project’s name, they are what every archive URL is derived from, and they are the ones a third party is likely to have built something on.

X-Pepsi-List-Hops is the only one that the software ever reads back in. It is the backstop against a subscription cycle through two or more lists, since the loop rule catches only a message returning to the same list. It travels in a header because a sibling row leaves over SMTP and re-enters as ordinary mail, where row state does not survive. A post whose count has reached [pepsi-list] MAX_LIST_HOPS (default 5) is held for the moderators instead of delivered, because a cycle that is delivered is a cycle that grows. Because a header is forgeable, pepsi-ingress(1) strips an inbound copy from mail that did not arrive over an authenticated or local submission path.

95.1.21.1.8. The fan-out

Every regular member with delivery enabled and not on digest gets one row, unconditionally. Each carries its own single recipient and its own VERP envelope sender, <list>-bounces+<local>=<domain>=<code>@<host>. This is upstream’s form with an authentication code appended, keyed with [pepsi-list] VERP_SECRET (the stage refuses to start without it). It lets the bounce path attribute a failure to exactly one subscription without parsing the bounce, and refuse an address that somebody made up. Each row starts paused at DELIVERY_STAGE for a random time of up to [pepsi] REFLECTION_HOLD_MAX (default 30 s; pepsi.conf(5)), drawn per copy, and the dispatcher releases each copy when its time comes. A post is reflected mail, and spreading the fan-out over the window keeps a stranger from timing posts to many lists so that they land on one host at once.

What crosses the wire is the recipient list, not the message: the headers are copied inside the database and the body is shared by reference, so a five-thousand-member post sends one message and five thousand small JSON objects to PostgreSQL rather than five thousand messages.

The fan-out is batched: at most [pepsi-list] LIST_FANOUT_BATCH rows (default 256) are materialised at a time, and the message pauses between batches with the remaining recipients in its own state. Every sibling row references one stored copy of the (cooked) body in pepsi.workqueue_body rather than carrying its own, so the fan-out of a 5 000-member post with a 5 MB attachment stores the attachment once, not 5 000 times (a member’s copy gets a body of its own only when pepsi-stage-list-deliver(1) rewrites it). The batching bounds how many rows each step adds, and pepsi-queue(1) shows a fan-out in progress rather than a queue that has suddenly grown.

The fan-out also obeys the queue admission limits ([pepsi] MAX_QUEUE_ROWS, MAX_QUEUE_BYTES, MIN_FREE_SPACE; see pepsi.conf(5)). When the queue is at one of them the stage emits an empty batch, keeping the whole remainder in state, and pauses for [pepsi] QUEUE_THROTTLE_DELAY (default 60 s) before looking again, so a post to a large list cannot keep filling a queue that pepsi-ingress(1) is already refusing mail for. A first batch that is held back still archives the post and moves its counters, exactly once, as usual. The recipient list is snapshotted at the first batch, so an unsubscribe halfway through a large send cannot skip or double a neighbour, and the source row is deleted only by the batch that finishes the list, so a crash replays that batch and nothing more.

A digest issue takes the same path from its second step. pepsi-list digest injects each issue here (as [pepsi-list] RELEASE_STAGE) with its digest members already in state as the recipient snapshot, so the stage fans it out batch by batch exactly like a paused post and runs none of the posting chain: an issue is not moderated, archived, appended to the next digest or counted. Each sibling is marked as a digest, which tells pepsi-stage-list-deliver(1) to leave the issue’s own header and footer alone.

95.1.21.1.9. The posting autoresponder

autorespond_postings answers posts on the per-list grace-period machinery (autoresponse_grace_period, claimed through list_autoresponse_claim) that also rate-limits pepsi-stage-list-command(1)’s results mail. It is the one worth reading about: it fires on the posting address, so a misconfigured one answers everybody who writes to the list. (The sibling attributes autorespond_requests and autorespond_owner are stored and reported over REST, but no stage acts on them.)

Three values, as everywhere: none, respond_and_continue and respond_and_discard. On a posting address the last means the post is answered and not distributed: the list drops every post while replying politely to each sender. That is occasionally what an owner wants (a retired list that answers “we have moved”) and never what they want by accident, so the discard is logged at warn and the owner console labels the value plainly.

Two rules that are not obvious:

  • An unrecognised value is ``none``. A response action that nobody can spell is not a licence to mail a membership.

  • A post that could not be answered is still delivered. When the autoresponder declines (because RFC 3834 said no, the grace period had not expired, or the envelope sender was null), respond_and_discard does not discard. A list that silently dropped every message from an autoresponder would be a worse failure than a missing reply.

Every autoresponse goes through pepsi_common::autoreply’s RFC 3834 rule set first: the null sender, an Auto-Submitted that is not no, Precedence: bulk|list|junk, any List-*, a multipart/report, self-mail, a post an upstream stage called spam, and a post whose sender failed SPF or DMARC outright (state.auth.spf or state.auth.dmarc is fail). That rule set is shared rather than restated here, because getting it wrong is how an auto-responder builds a mail loop, and a loop on a posting address is a loop with a whole membership in it.

95.1.21.1.10. Configuration

DELIVERY_STAGE (required)

Where each member’s copy goes. It must run pepsi-stage-list-deliver(1), and pepsi-setup checks that it does.

BOUNCE_STAGE (required)

Where a rejected post is turned into a DSN for its sender. The DSN gives the list’s reason and follows each recipient’s NOTIFY, so NOTIFY=NEVER means no DSN. Here this means what it means everywhere except pepsi-stage-list(1).

RESPONSE_STAGE (required)

Where notices are injected: the moderators’ held-post notification, the poster’s “your message is held” notice and acknowledgement, and the posting autoresponse. The signing tail, not the list delivery stage: pepsi-stage-list-deliver(1) only accepts a member’s copy from the fan-out and fails anything else, so a notice sent there never goes out. pepsi-setup(1) refuses the wrong value.

REMOVE_DKIM_HEADERS (default no)

Strip the originator’s DKIM-Signature, DomainKey-Signature and Authentication-Results. Upstream’s [mailman] remove_dkim_headers. Off by default: a list that rewrites nothing leaves a valid signature and stripping it discards real evidence. Turn it on for lists that add a footer or a subject prefix, where the surviving signature would fail. A failing signature is worse than none, because a failure is evidence of tampering and an absence is not.

SITE_HEADER_MATCH_CHAIN (default hold)

Which chain a site-wide header match jumps to; upstream’s [antispam] jump_chain. Site-wide because the matches it applies to are the site’s, and a list owner must not be able to turn the site’s spam rules into a discard.

SENDER_POSTS_PER_HOUR (default 30)

How many posts from one sender one list distributes per hour. A post over the limit that the chain would have accepted is held for the moderator instead, with the reason saying so; a moderator’s release is never counted. The window opens at the sender’s first post and lasts an hour (pepsi.list_post_rate). Site-wide rather than a list attribute because it is an availability limit: a list multiplies every accepted post by its membership. 0 disables it.

MAX_HELD_MESSAGES (default 1000)

How many messages one list may hold for moderation. A post that would be held beyond it is refused with a DSN and a warning in the log, and it is not stored. Held posts are kept as their full wire bytes until a moderator acts or the list’s max_days_to_hold (default: never) expires them, so without a cap an open list would store whatever anybody sent it. The new post is refused rather than the oldest so that a flood cannot flush out the posts still waiting for a moderator. Concurrent workers may overshoot by a few. 0 disables it.

A held post is stored under the hash of its Message-ID. A sender chooses that field, so a second post with the same Message-ID and other content is stored under a key that also depends on its content, and it gets a moderator request of its own.

Both limits, the hop ceiling and the admission throttle of the fan-out are the mailing-list entries of “Loops and amplification” in the threat-model chapter of the Pepsi manual; “Resource limits” in its security chapter says where each is enforced.

NEXT_STAGE is meaningless here and pepsi-setup refuses a section that sets it: every path out of this stage is a terminal.

95.1.21.1.11. State

Inputs: state.list.id (written by pepsi-stage-list(1)), state.auth (the SPF/DKIM/DMARC verdicts that pepsi-ingress(1) recorded), state.list.pending_rcpt on a resumed fan-out, and state.list.moderator_approved on a held post a moderator released.

Outputs: a state.list descriptor on each sibling row: the member id, their language, their unsubscribe serial and the duplicate flag.

Transitions: finishes (the last batch of a fan-out, a hold, a discard), finishes with a side clone (a rejection’s DSN), or pauses (a fan-out with recipients remaining).

95.1.21.1.12. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

95.1.21.1.13. Global Options

The usual set — -c/–config, -L/–log, -v/–verbose, -h/–help and -V/–version — behaving as they do for every Pepsi program; see pepsi-config(1).

95.1.21.1.14. Errors and retries

A post that names a list which no longer exists (it was deleted after the router routed the message) is reported to the sender: one DSN per recipient through BOUNCE_STAGE, saying the mailing list no longer exists, and the post is gone. A message the MIME parser refuses (nested or split too deeply to process), or one that reached the stage without the router’s state.list.id (a stage wired by hand), is failed at once. Any other error, such as one from the database, a template or the archive, is a fault of the host: the post is paused and retried with back-off until MAX_LIFETIME (see pepsi-dispatch(1)). A section that does not parse keeps the worker from starting at all, so the queue waits for the fix instead of failing post after post.

A retry must not change what the post does. SENDER_POSTS_PER_HOUR and the posting autoresponder’s grace period are both claims recorded before the post is committed, so both are keyed on the message’s queue token: a post retried after its commit failed gets the answers that its first attempt got. It is neither counted twice (and held for our own retry) nor told that it was already answered today. The autoresponse itself is committed in the same transaction as the post, so a post is never answered without being handled, or handled without its answer. That transaction is the one with the first fan-out batch, or the one that deletes a respond_and_discard post.

95.1.21.1.15. Exit Status

The outcome of each message is reported to pepsi-dispatch(1) on the worker’s status line, not as the exit status.

0

The worker ran until its standard input was closed.

1

A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.

78

The worker refused to start: a [stage-*] section it serves, or [pepsi-list], does not parse (e.g. a missing DELIVERY_STAGE). The dispatcher requeues what it handed over and retries the stage later.

95.1.21.1.16. Examples

[stage-list-post]
PROGRAM = pepsi-stage-list-post
DELIVERY_STAGE = list-deliver
BOUNCE_STAGE = bounce
RESPONSE_STAGE = dkim-sign
REMOVE_DKIM_HEADERS = no

95.1.21.1.17. See Also

pepsi-list(1), pepsi-stage-list(1), pepsi-stage-list-deliver(1), pepsi-stage-dkim-sign(1), pepsi-ingress(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

95.1.21.1.18. Bugs

Report bugs to the Pepsi issue tracker.