95.1.21. pepsi-stage-list-post¶
moderate, transform and fan out a post to a mailing list
- Manual section:
1
95.1.21.1.1. Name¶
pepsi-stage-list-post - the mailing-list posting stage of the Pepsi pipeline.
95.1.21.1.2. Synopsis¶
pepsi-stage-list-post [GLOBAL-OPTIONS] worker
95.1.21.1.3. Description¶
pepsi-stage-list-post is a stage program run by pepsi-dispatch(1). A message that pepsi-stage-list(1) decided is a post to a list arrives here, and two things happen to it, in this order and with these names:
a chain of rules decides whether the post is accepted, held, rejected or discarded;
a pipeline of handlers decides how an accepted post is transformed and delivered.
Moderation and processing are different stages with different vocabularies, and
conflating them produces something that is not Mailman. The rule, chain,
handler and pipeline names are a compatibility identifier: the REST API
reports them and a list’s posting_chain and posting_pipeline name them.
The stage reads the database once, at the start; computes everything in memory; and writes once, in a single transaction that also takes the message’s own terminal. A crash anywhere in the middle replays the whole thing.
95.1.21.1.4. Chains and rules¶
The default posting chain runs, in order: the DMARC mitigation check, the
no-senders check, the moderator-password bypass (approved), the loop and
banned-address rules, Pepsi’s dmarc-failure rule, the site and per-list
header matches, the emergency flag, and the member and non-member moderation
rules. Then eight rules only record what they find, and a final any holds
the post if one of them hit. These eight are administrivia,
implicit-dest, max-recipients, max-size, news-moderation,
no-subject, digests and suspicious-header. Upstream’s eighteen
rules are the seventeen named here and truth, which always matches. Pepsi
adds one rule, dmarc-failure, and there are four terminal chains
(accept, hold, reject, discard).
The DMARC mitigation check does not look the policy up again: it uses the
policy that ingress recorded for the From: domain (state.auth.dmarc_policy).
A domain publishing quarantine or reject is mitigated, one publishing
none or no record at all is not, and a post for which nothing was recorded
(a local submission) is mitigated, as upstream does when its lookup fails.
dmarc_mitigate_unconditionally mitigates every post.
dmarc-failure holds a post whose From: domain failed DMARC at intake
(state.auth.dmarc is fail) on a list whose hold_dmarc_failures
setting is on. That setting is on by default for every list that Pepsi creates;
lists imported from GNU Mailman are given no. It holds whatever policy the domain publishes and
whether or not the poster is a member, with the reason “The message failed
DMARC”. A failure that a trusted ARC sealer vouched for at ingress
(state.auth.arc_override) counts as a pass, and a post submitted here by an
authenticated sender (state.local_origin) is never held by this rule, whatever
verdict ingress recorded beside it: SMTP AUTH proved the sender, and a roaming
member’s client is in nobody’s SPF record. A list re-signs and re-sends every post it
distributes, so without this rule a forged post would reach every member
looking authenticated. Like every other hold, a moderator’s Approved:
password releases it.
A correct moderator password bypasses the ban list. This is upstream’s
behaviour and compatibility requires it, but it is a sharp edge and it is said
here rather than three chapters away: an Approved: header carrying the
list’s moderator password jumps straight to accept, over the ban list, the
header matches, the emergency flag and both moderation rules.
95.1.21.1.5. The approved-password invariant¶
A moderator may approve a post by putting the list’s moderator password in an
Approved: header or in the first line of the body. Whichever form it
arrived in, it is removed before the message is archived or delivered: from
the header, from the plain-text body, and from the HTML alternative that a mail
client generated alongside it. A password that survives into a public archive is
published permanently, and purging the archived message does not recall the
copies already in five thousand mailboxes.
The rule strips what it matched; cleanse then deletes all four header names
(Approve, Approved, X-Approve, X-Approved) unconditionally,
which is the only thing that removes a wrong password.
When a password verifies against a stored hash that was made with weaker
Argon2id parameters than new hashes get, the stage hashes it again with the
current parameters and stores the result, as a console or member sign-in
upgrades its own. It stores it only while the stored hash is still the one that
it verified against. The Approved: header is the only moment at which the
stage sees the plaintext.
95.1.21.1.6. Handlers¶
The pipeline is upstream’s, in upstream’s order:
validate-authenticity, mime-delete, tagger, member-recipients,
avoid-duplicates, cleanse, cleanse-dkim, cook-headers,
subject-prefix, rfc-2369, to-archive, to-digest, to-usenet,
after-delivery, acknowledge, dmarc, to-outgoing.
A post from which mime-delete removes everything goes no further down the
pipeline, and the list’s filter_action decides what happens to it. Every
action works from the post as it arrived:
holdThe post waits for a moderator like any held post, and the moderators and the sender are told, subject to
admin_immed_notifyandrespond_to_post_requests.preserveThe post is kept the same way, but the sender is not told. Moderators find it among the list’s held messages, in the owner console and at
/lists/<id>/heldof the REST API, with a reason that says it was preserved.max_days_to_holdapplies to it too.rejectThe sender gets a DSN that gives the reason.
forwardThe list’s owners get a notice with the post attached, and the sender is not told. A list with no owner keeps the post as with
preserve.discardThe post is deleted and nobody is told. This is the default.
A moderator who releases a post held by hold or preserve has decided, so
the post is delivered as it arrived, without the filter. A post that was held for
another reason still passes the filter, and if nothing is left of it under
reject or discard, the owners get it as with forward, because a
released post has no sender to tell. accept and defer
are refused for a new setting. A list that already carries one of them behaves as
with hold. If the moderation queue is at MAX_HELD_MESSAGES, hold and
preserve cannot keep the post, and the sender gets a DSN instead.
decorate and arc-sign are absent because upstream’s own default
pipeline has them commented out, with upstream’s reasons: all decoration is
done in delivery, and decoration in delivery would break a signature applied
here. Both are pepsi-stage-list-deliver(1)’s and the signing tail’s.
to-usenet is an explicit, logged no-op: gateway_to_news stays settable
because removing it would change the resource a REST client sees, but Pepsi
does not gateway to NNTP, and a silent no-op would leave an operator believing
their gateway works.
95.1.21.1.7. Headers written¶
Mailman’s X-Mailman-* namespace is renamed. Nothing in Postorius or
HyperKitty reads any of it, and the only consumer of any of them is a human
reading a held post:
Mailman 3 |
Pepsi |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(no counterpart) |
|
|
|
Message-ID-Hash and its legacy X- spelling keep their names: they carry
no project’s name, they are what every archive URL is derived from, and they
are the ones a third party is likely to have built something on.
X-Pepsi-List-Hops is the only one that the software ever reads back in. It
is the backstop against a subscription cycle through two or more lists, since
the loop rule catches only a message returning to the same list. It
travels in a header because a sibling row leaves over SMTP and re-enters as
ordinary mail, where row state does not survive. A post whose count has
reached [pepsi-list] MAX_LIST_HOPS (default 5) is held for the
moderators instead of delivered, because a cycle that is delivered is a cycle
that grows. Because a header is forgeable, pepsi-ingress(1) strips an
inbound copy from mail that did not arrive over an authenticated or local
submission path.
95.1.21.1.8. The fan-out¶
Every regular member with delivery enabled and not on digest gets one row,
unconditionally. Each carries its own single recipient and its own VERP
envelope sender, <list>-bounces+<local>=<domain>=<code>@<host>. This is
upstream’s form with an authentication code appended, keyed with
[pepsi-list] VERP_SECRET (the stage refuses to start without it). It lets
the bounce path attribute a failure to exactly one subscription without parsing
the bounce, and refuse an address that somebody made up. Each row starts
paused at DELIVERY_STAGE for a random time of up to [pepsi]
REFLECTION_HOLD_MAX (default 30 s; pepsi.conf(5)), drawn per copy, and
the dispatcher releases each copy when its time comes. A post is reflected mail,
and spreading the fan-out over the window keeps a stranger from timing posts to
many lists so that they land on one host at once.
What crosses the wire is the recipient list, not the message: the headers are copied inside the database and the body is shared by reference, so a five-thousand-member post sends one message and five thousand small JSON objects to PostgreSQL rather than five thousand messages.
The fan-out is batched: at most [pepsi-list] LIST_FANOUT_BATCH rows (default 256)
are materialised at a time, and the message pauses between batches with the
remaining recipients in its own state. Every sibling row references one
stored copy of the (cooked) body in pepsi.workqueue_body rather than carrying
its own, so the fan-out of a 5 000-member post with a 5 MB attachment stores the
attachment once, not 5 000 times (a member’s copy gets a body of its own only
when pepsi-stage-list-deliver(1) rewrites it). The batching bounds how
many rows each step adds, and pepsi-queue(1) shows a fan-out in progress
rather than a queue that has suddenly grown.
The fan-out also obeys the queue admission limits ([pepsi]
MAX_QUEUE_ROWS, MAX_QUEUE_BYTES, MIN_FREE_SPACE; see
pepsi.conf(5)). When the queue is at one of them the stage emits an
empty batch, keeping the whole remainder in state, and pauses for
[pepsi] QUEUE_THROTTLE_DELAY (default 60 s) before looking again, so a post
to a large list cannot keep filling a queue that pepsi-ingress(1) is
already refusing mail for. A first batch that is held back still archives the
post and moves its counters, exactly once, as usual. The recipient list is snapshotted at the
first batch, so an unsubscribe halfway through a large send cannot skip or
double a neighbour, and the source row is deleted only by the batch that
finishes the list, so a crash replays that batch and nothing more.
A digest issue takes the same path from its second step. pepsi-list
digest injects each issue here (as [pepsi-list] RELEASE_STAGE) with
its digest members already in state as the recipient snapshot, so the stage
fans it out batch by batch exactly like a paused post and runs none of the
posting chain: an issue is not moderated, archived, appended to the next digest
or counted. Each sibling is marked as a digest, which tells
pepsi-stage-list-deliver(1) to leave the issue’s own header and footer
alone.
95.1.21.1.9. The posting autoresponder¶
autorespond_postings answers posts on the per-list grace-period machinery
(autoresponse_grace_period, claimed through list_autoresponse_claim)
that also rate-limits pepsi-stage-list-command(1)’s results mail. It is
the one worth reading about: it fires on the posting address, so a
misconfigured one answers everybody who writes to the list. (The sibling
attributes autorespond_requests and autorespond_owner are stored and
reported over REST, but no stage acts on them.)
Three values, as everywhere: none, respond_and_continue and
respond_and_discard. On a posting address the last means the post is answered
and not distributed: the list drops every post while replying politely to
each sender. That is occasionally what an owner wants (a retired list that
answers “we have moved”) and never what they want by accident, so the discard is
logged at warn and the owner console labels the value plainly.
Two rules that are not obvious:
An unrecognised value is ``none``. A response action that nobody can spell is not a licence to mail a membership.
A post that could not be answered is still delivered. When the autoresponder declines (because RFC 3834 said no, the grace period had not expired, or the envelope sender was null),
respond_and_discarddoes not discard. A list that silently dropped every message from an autoresponder would be a worse failure than a missing reply.
Every autoresponse goes through pepsi_common::autoreply’s RFC 3834 rule set
first: the null sender, an Auto-Submitted that is not no,
Precedence: bulk|list|junk, any List-*, a multipart/report,
self-mail, a post an upstream stage called spam, and a post whose sender failed
SPF or DMARC outright (state.auth.spf or state.auth.dmarc is fail).
That rule set is shared
rather than restated here, because getting it wrong is how an auto-responder
builds a mail loop, and a loop on a posting address is a loop with a whole
membership in it.
95.1.21.1.10. Configuration¶
- DELIVERY_STAGE (required)
Where each member’s copy goes. It must run pepsi-stage-list-deliver(1), and pepsi-setup checks that it does.
- BOUNCE_STAGE (required)
Where a rejected post is turned into a DSN for its sender. The DSN gives the list’s reason and follows each recipient’s
NOTIFY, soNOTIFY=NEVERmeans no DSN. Here this means what it means everywhere except pepsi-stage-list(1).- RESPONSE_STAGE (required)
Where notices are injected: the moderators’ held-post notification, the poster’s “your message is held” notice and acknowledgement, and the posting autoresponse. The signing tail, not the list delivery stage: pepsi-stage-list-deliver(1) only accepts a member’s copy from the fan-out and fails anything else, so a notice sent there never goes out. pepsi-setup(1) refuses the wrong value.
- REMOVE_DKIM_HEADERS (default
no) Strip the originator’s
DKIM-Signature,DomainKey-SignatureandAuthentication-Results. Upstream’s[mailman] remove_dkim_headers. Off by default: a list that rewrites nothing leaves a valid signature and stripping it discards real evidence. Turn it on for lists that add a footer or a subject prefix, where the surviving signature would fail. A failing signature is worse than none, because a failure is evidence of tampering and an absence is not.- SITE_HEADER_MATCH_CHAIN (default
hold) Which chain a site-wide header match jumps to; upstream’s
[antispam] jump_chain. Site-wide because the matches it applies to are the site’s, and a list owner must not be able to turn the site’s spam rules into a discard.- SENDER_POSTS_PER_HOUR (default
30) How many posts from one sender one list distributes per hour. A post over the limit that the chain would have accepted is held for the moderator instead, with the reason saying so; a moderator’s release is never counted. The window opens at the sender’s first post and lasts an hour (
pepsi.list_post_rate). Site-wide rather than a list attribute because it is an availability limit: a list multiplies every accepted post by its membership.0disables it.- MAX_HELD_MESSAGES (default
1000) How many messages one list may hold for moderation. A post that would be held beyond it is refused with a DSN and a warning in the log, and it is not stored. Held posts are kept as their full wire bytes until a moderator acts or the list’s
max_days_to_hold(default: never) expires them, so without a cap an open list would store whatever anybody sent it. The new post is refused rather than the oldest so that a flood cannot flush out the posts still waiting for a moderator. Concurrent workers may overshoot by a few.0disables it.
A held post is stored under the hash of its Message-ID. A sender chooses that
field, so a second post with the same Message-ID and other content is stored
under a key that also depends on its content, and it gets a moderator request of
its own.
Both limits, the hop ceiling and the admission throttle of the fan-out are the mailing-list entries of “Loops and amplification” in the threat-model chapter of the Pepsi manual; “Resource limits” in its security chapter says where each is enforced.
NEXT_STAGE is meaningless here and pepsi-setup refuses a section that sets it: every path out of this stage is a terminal.
95.1.21.1.11. State¶
Inputs: state.list.id (written by pepsi-stage-list(1)),
state.auth (the SPF/DKIM/DMARC verdicts that pepsi-ingress(1) recorded),
state.list.pending_rcpt on a resumed fan-out, and
state.list.moderator_approved on a held post a moderator released.
Outputs: a state.list descriptor on each sibling row: the member id,
their language, their unsubscribe serial and the duplicate flag.
Transitions: finishes (the last batch of a fan-out, a hold, a discard), finishes with a side clone (a rejection’s DSN), or pauses (a fan-out with recipients remaining).
95.1.21.1.12. Commands¶
- worker
Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.
95.1.21.1.13. Global Options¶
The usual set — -c/–config, -L/–log, -v/–verbose, -h/–help and -V/–version — behaving as they do for every Pepsi program; see pepsi-config(1).
95.1.21.1.14. Errors and retries¶
A post that names a list which no longer exists (it was deleted after the
router routed the message) is reported to the sender: one DSN per recipient
through BOUNCE_STAGE, saying the mailing list no longer exists, and the post
is gone. A message the MIME parser refuses (nested or split too deeply to
process), or one that reached the stage without the router’s state.list.id
(a stage wired by hand), is failed at once. Any other error, such as one from
the database, a template or the archive, is a fault of the host: the post is paused and retried with back-off
until MAX_LIFETIME (see pepsi-dispatch(1)). A section that does not
parse keeps the worker from starting at all, so the queue waits for the fix
instead of failing post after post.
A retry must not change what the post does. SENDER_POSTS_PER_HOUR and the
posting autoresponder’s grace period are both claims recorded before the post is
committed, so both are keyed on the message’s queue token: a post retried after
its commit failed gets the answers that its first attempt got. It is neither
counted twice (and held for our own retry) nor told that it was already answered
today. The autoresponse itself is committed in the same transaction as the
post, so a post is never answered without being handled, or handled without its
answer. That transaction is the one with the first fan-out batch, or the one
that deletes a respond_and_discard post.
95.1.21.1.15. Exit Status¶
The outcome of each message is reported to pepsi-dispatch(1) on the worker’s status line, not as the exit status.
- 0
The worker ran until its standard input was closed.
- 1
A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.
- 78
The worker refused to start: a
[stage-*]section it serves, or[pepsi-list], does not parse (e.g. a missing DELIVERY_STAGE). The dispatcher requeues what it handed over and retries the stage later.
95.1.21.1.16. Examples¶
[stage-list-post]
PROGRAM = pepsi-stage-list-post
DELIVERY_STAGE = list-deliver
BOUNCE_STAGE = bounce
RESPONSE_STAGE = dkim-sign
REMOVE_DKIM_HEADERS = no
95.1.21.1.17. See Also¶
pepsi-list(1), pepsi-stage-list(1), pepsi-stage-list-deliver(1), pepsi-stage-dkim-sign(1), pepsi-ingress(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)
95.1.21.1.18. Bugs¶
Report bugs to the Pepsi issue tracker.