85.1.20. pepsi-stage-list-post¶
moderate, transform and fan out a post to a mailing list
- Manual section:
1
85.1.20.1.1. Name¶
pepsi-stage-list-post - the mailing-list posting stage of the Pepsi pipeline.
85.1.20.1.2. Synopsis¶
pepsi-stage-list-post [GLOBAL-OPTIONS] worker
85.1.20.1.3. Description¶
pepsi-stage-list-post is a stage program run by pepsi-dispatch(1). A message that pepsi-stage-list(1) decided is a post to a list arrives here, and two things happen to it, in this order and with these names:
a chain of rules decides whether the post is accepted, held, rejected or discarded;
a pipeline of handlers decides how an accepted post is transformed and delivered.
Moderation and processing are different stages with different vocabularies, and
conflating them produces something that is not Mailman. The rule, chain,
handler and pipeline names are a compatibility identifier: the REST API
reports them and a list’s posting_chain and posting_pipeline name them.
The stage reads the database once, at the start; computes everything in memory; and writes once, in a single transaction that also takes the message’s own terminal. A crash anywhere in the middle replays the whole thing.
85.1.20.1.4. Chains and rules¶
The default posting chain runs, in order: the DMARC mitigation check, the
no-senders check, the moderator-password bypass (approved), the loop and
banned-address rules, the site and per-list header matches, the emergency flag,
and the member and non-member moderation rules. Then eight rules only
record what they find — administrivia, implicit-destination,
maximum-recipients, maximum-size, news-moderation, empty-subject, digests and
suspicious-header — and a final any holds the post if one of them hit.
There are eighteen rules and four terminal chains (accept, hold,
reject, discard).
A correct moderator password bypasses the ban list. This is upstream’s
behaviour and compatibility requires it, but it is a sharp edge and it is said
here rather than three chapters away: an Approved: header carrying the
list’s moderator password jumps straight to accept, over the ban list, the
header matches, the emergency flag and both moderation rules.
85.1.20.1.5. The approved-password invariant¶
A moderator may approve a post by putting the list’s moderator password in an
Approved: header or in the first line of the body. Whichever form it
arrived in, it is removed before the message is archived or delivered — in
the header, in the plain-text body, and in the HTML alternative a mail client
generated alongside it. A password that survives into a public archive is
published permanently, and purging the archived message does not recall the
copies already in five thousand mailboxes.
The rule strips what it matched; cleanse then deletes all four header names
(Approve, Approved, X-Approve, X-Approved) unconditionally,
which is the only thing that removes a wrong password.
85.1.20.1.6. Handlers¶
The pipeline is upstream’s, in upstream’s order:
validate-authenticity, mime-delete, tagger, member-recipients,
avoid-duplicates, cleanse, cleanse-dkim, cook-headers,
subject-prefix, rfc-2369, to-archive, to-digest, to-usenet,
after-delivery, acknowledge, dmarc, to-outgoing.
decorate and arc-sign are absent because upstream’s own default
pipeline has them commented out, with upstream’s reasons: all decoration is
done in delivery, and decoration in delivery would break a signature applied
here. Both are pepsi-stage-list-deliver(1)’s and the signing tail’s.
to-usenet is an explicit, logged no-op: gateway_to_news stays settable
because removing it would change the resource a REST client sees, but Pepsi
does not gateway to NNTP, and a silent no-op would leave an operator believing
their gateway works.
85.1.20.1.7. Headers written¶
Mailman’s X-Mailman-* namespace is renamed. Nothing in Postorius or
HyperKitty reads any of it, and the only consumer of any of them is a human
reading a held post:
Mailman 3 |
Pepsi |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(no counterpart) |
|
|
|
Message-ID-Hash and its legacy X- spelling keep their names: they carry
no project’s name, they are what every archive URL is derived from, and they
are the ones a third party is likely to have built something on.
X-Pepsi-List-Hops is the only one the software ever reads back in. It is
the backstop against a subscription cycle through two or more lists — the
loop rule catches only a message returning to the same list — and it
travels in a header because a sibling row leaves over SMTP and re-enters as
ordinary mail, where row state does not survive. Because a header is forgeable,
pepsi-ingress(1) strips an inbound copy from an unauthenticated sender.
85.1.20.1.8. The fan-out¶
Every regular member with delivery enabled and not on digest gets one row,
unconditionally. Each carries its own single recipient and its own VERP
envelope sender, <list>-bounces+<local>=<domain>@<host>, which is what lets
the bounce path attribute a failure to exactly one subscription without parsing
the bounce.
What crosses the wire is the recipient list, not the message: the headers are copied inside the database and the body is shared by reference, so a five-thousand-member post sends one message and five thousand small JSON objects to PostgreSQL rather than five thousand messages.
The fan-out is batched: at most [pepsi-list] LIST_FANOUT_BATCH rows (default 256)
are materialised at a time, and the message pauses between batches with the
remaining recipients in its own state. Every sibling row references one
stored copy of the (cooked) body in pepsi.workqueue_body rather than carrying
its own, so a 5 000-member post with a 5 MB attachment stores the attachment
once, not 5 000 times; the batching bounds the number of rows in flight, and
pepsi-queue(1) shows a fan-out in progress rather than a queue that has
suddenly grown.
The fan-out also obeys the queue admission limits ([pepsi]
MAX_QUEUE_ROWS, MAX_QUEUE_BYTES, MIN_FREE_SPACE; see
pepsi.conf(5)). When the queue is at one of them the stage emits an
empty batch — the whole remainder stays in state — and pauses for
[pepsi] QUEUE_THROTTLE_DELAY (default 60 s) before looking again, so a post
to a large list cannot keep filling a queue that pepsi-ingress(1) is
already refusing mail for. A first batch that is held back still archives the
post and moves its counters, exactly once, as usual. The recipient list is snapshotted at the
first batch, so an unsubscribe halfway through a large send cannot skip or
double a neighbour, and the source row is deleted only by the batch that
finishes the list, so a crash replays that batch and nothing more.
85.1.20.1.9. The posting autoresponder¶
autorespond_postings is the third of the three autoresponders, on the same
grace-period machinery as -request and -owner, and the one worth reading
about: it fires on the posting address, so a misconfigured one answers
everybody who writes to the list.
Three values, as everywhere: none, respond_and_continue and
respond_and_discard. On a posting address the last means the post is answered
and not distributed — a list that swallows every post while replying politely
to each sender. That is occasionally what an owner wants (a retired list that
answers “we have moved”) and never what they want by accident, so the discard is
logged at warn and the owner console labels the value plainly.
Two rules that are not obvious:
An unrecognised value is ``none``. A response action nobody can spell is not a licence to mail a membership.
A post that could not be answered is still delivered. When the autoresponder declines — RFC 3834 said no, the grace period had not expired, the envelope sender was null —
respond_and_discarddoes not discard. A list that silently swallowed every message from an autoresponder would be a worse failure than a missing reply.
Every autoresponse goes through pepsi_common::autoreply’s RFC 3834 rule set
first — the null sender, an Auto-Submitted that is not no,
Precedence: bulk|list|junk, any List-*, a multipart/report,
self-mail, and a post an upstream stage called spam. That rule set is shared
rather than restated here, because getting it wrong is how an auto-responder
builds a mail loop, and a loop on a posting address is a loop with a whole
membership in it.
85.1.20.1.10. Configuration¶
- DELIVERY_STAGE (required)
Where each member’s copy goes. It must run pepsi-stage-list-deliver(1), and pepsi-setup checks that it does.
- BOUNCE_STAGE (required)
Where a rejected post is turned into a DSN for its sender. Here this means what it means everywhere except pepsi-stage-list(1).
- RESPONSE_STAGE (required)
Where notices are injected: the moderator’s held-post notification and the poster’s acknowledgement. The signing tail, not the list delivery stage: pepsi-stage-list-deliver(1) only accepts a member’s copy from the fan-out and fails anything else, so a notice sent there never goes out. pepsi-setup(1) refuses the wrong value.
- REMOVE_DKIM_HEADERS (default
no) Strip the originator’s
DKIM-Signature,DomainKey-SignatureandAuthentication-Results. Upstream’s[mailman] remove_dkim_headers. Off by default: a list that rewrites nothing leaves a valid signature and stripping it discards real evidence. Turn it on for lists that add a footer or a subject prefix, where the surviving signature would fail — and a failing signature is worse than none, because a failure is evidence of tampering and an absence is not.- SITE_HEADER_MATCH_CHAIN (default
hold) Which chain a site-wide header match jumps to; upstream’s
[antispam] jump_chain. Site-wide because the matches it applies to are the site’s, and a list owner must not be able to turn the site’s spam rules into a discard.- SENDER_POSTS_PER_HOUR (default
30) How many posts from one sender one list distributes per hour. A post over the limit that the chain would have accepted is held for the moderator instead, with the reason saying so; a moderator’s release is never counted. The window opens at the sender’s first post and lasts an hour (
pepsi.list_post_rate). Site-wide rather than a list attribute because it is an availability limit: a list multiplies every accepted post by its membership.0disables it.- MAX_HELD_MESSAGES (default
1000) How many messages one list may hold for moderation. A post that would be held beyond it is discarded — not stored, and no notice sent — with a warning in the log. Held posts are kept as their full wire bytes until a moderator acts or the list’s
max_days_to_hold(default: never) expires them, so without a cap an open list would store whatever anybody sent it. The new post is dropped rather than the oldest so that a flood cannot flush out the posts still waiting for a moderator. Concurrent workers may overshoot by a few.0disables it.
NEXT_STAGE is meaningless here and pepsi-setup refuses a section that sets it: every path out of this stage is a terminal.
85.1.20.1.11. State¶
Inputs: state.list.id (written by pepsi-stage-list(1)),
state.auth (the SPF/DKIM/DMARC verdicts pepsi-ingress(1) recorded),
and state.list.pending_rcpt on a resumed fan-out.
Outputs: a state.list descriptor on each sibling row — the member id,
their language, their unsubscribe serial and the duplicate flag.
Transitions: finishes (the last batch of a fan-out, a hold, a discard), finishes with a side clone (a rejection’s DSN), or pauses (a fan-out with recipients remaining).
85.1.20.1.12. Commands¶
- worker
Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.
85.1.20.1.13. Global Options¶
The usual set — -c/–config, -L/–log, -v/–verbose, -h/–help and -V/–version — behaving as they do for every Pepsi program; see pepsi-config(1).
85.1.20.1.14. Errors and retries¶
A post that names a list which no longer exists (it was deleted after the router routed the message) is reported to the sender: one DSN per recipient through BOUNCE_STAGE, saying the mailing list no longer exists, and the post is gone. A message the MIME parser refuses (nested or split too deeply to process) is failed at once. Any other error – the database, a template, the archive – is a fault of the host: the post is paused and retried with back-off until MAX_LIFETIME (see pepsi-dispatch(1)). A section that does not parse keeps the worker from starting at all, so the queue waits for the fix instead of failing post after post.
A retry must not change what the post does. SENDER_POSTS_PER_HOUR and the
posting autoresponder’s grace period are both claims recorded before the post is
committed, so both are keyed on the message’s queue token: a post retried after
its commit failed gets the answers its first attempt got, and is neither counted
twice (and held for our own retry) nor told it was already answered today. The
autoresponse itself is committed in the same transaction as the post – with the
first fan-out batch, or with the deletion of a respond_and_discard post –
so a post is never answered without being handled, or handled without its
answer.
85.1.20.1.15. Exit Status¶
- 0
The post was accepted, held, rejected or discarded.
- 1
The worker could not run (its database could not be opened, or standard input/output failed). The reason is written to the log.
85.1.20.1.16. Examples¶
[stage-list-post]
PROGRAM = pepsi-stage-list-post
DELIVERY_STAGE = list-deliver
BOUNCE_STAGE = bounce
RESPONSE_STAGE = dkim-sign
REMOVE_DKIM_HEADERS = no
85.1.20.1.17. See Also¶
pepsi-list(1), pepsi-stage-list(1), pepsi-stage-list-deliver(1), pepsi-stage-dkim-sign(1), pepsi-ingress(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)
85.1.20.1.18. Bugs¶
Report bugs to the Pepsi issue tracker.
