82. pepsi-stage-list-post¶
Moderate a post to a mailing list, transform it, and fan it out to the members.
82.1. Role¶
pepsi-stage-list-post receives a message that pepsi-stage-list
decided is a post to a list, and does two things to it in this order and with
these names: a chain of rules decides whether the post is accepted,
held, rejected or discarded, and then a pipeline of handlers decides
how an accepted post is transformed and delivered. Moderation and processing
are different steps with different vocabularies, and conflating them produces
something that is not Mailman. The rule, chain, handler and pipeline names
are a compatibility identifier that the REST API reports and that a list’s
posting_chain and posting_pipeline name. Reference:
pepsi-stage-list-post(1).
This subsystem is a reimplementation of GNU Mailman 3; see Mailing lists, which names what was taken from upstream.
82.2. Moderation¶
The default posting chain runs the DMARC mitigation check, the no-senders
check, the moderator-password bypass (approved), the loop and banned-address
rules, Pepsi’s dmarc-failure rule, the site and per-list header matches, the
emergency flag, and the member and non-member moderation rules. Then eight rules
only record what they find (administrivia, implicit-dest,
max-recipients, max-size, news-moderation, no-subject,
digests and suspicious-header), and a final any holds the post if
one of them hit. In all, these are upstream’s eighteen rules, one of Pepsi’s,
and four terminal chains (accept, hold, reject, discard).
A post that failed DMARC is held. A list re-signs and re-sends what it
distributes, so a post whose From: domain did not authenticate it would
otherwise reach every member looking authenticated, and the list would launder
the forgery. dmarc-failure therefore holds a post whose ingress DMARC verdict is
fail on a list with hold_dmarc_failures on (the default for lists Pepsi
creates; imported lists get no, see Settings Pepsi invented),
whatever policy the domain publishes and whether or not the poster is a
member. A failure that a trusted ARC sealer vouched for
(ARC_TRUSTED_SEALERS) counts as a pass. A post with no verdict is not held,
and neither is one submitted here by an authenticated sender
(state.local_origin), whatever verdict ingress recorded beside it. A
moderator’s Approved: password releases a held post, as it releases every
hold.
Warning
A correct moderator password bypasses the ban list. That is upstream’s
behaviour and compatibility requires it, but it is a sharp edge: an
Approved: header carrying the list’s moderator password jumps straight to
accept, over the ban list, the header matches, the emergency flag and
both moderation rules.
The approved password never survives. A moderator may approve a post with
the password in an Approved: header or in the first line of the body;
whichever form it arrived in, it is removed before the message is archived or
delivered: from the header, from the plain-text body, and from the HTML
alternative that a mail client generated alongside it. A password that reaches a public archive is
published permanently, and purging the archived copy does not recall the copies
already in five thousand mailboxes. The rule strips what it matched, and the
cleanse handler then deletes all four header names (Approve,
Approved, X-Approve, X-Approved) unconditionally, which is the only
thing that removes a wrong password. If the password is right but its stored hash
was made with weaker Argon2id parameters than new hashes get, the password is
hashed again and stored, just as a console sign-in upgrades its own hash; the
header is the one moment when the plaintext is seen.
82.3. Handlers¶
The pipeline is upstream’s, in upstream’s order: validate-authenticity,
mime-delete, tagger, member-recipients, avoid-duplicates,
cleanse, cleanse-dkim, cook-headers, subject-prefix,
rfc-2369, to-archive, to-digest, to-usenet, after-delivery,
acknowledge, dmarc, to-outgoing.
A post from which mime-delete removes everything goes no further down the
pipeline. The list’s filter_action then decides what happens to it
(When content filtering leaves nothing).
decorate and arc-sign are absent because upstream’s own default pipeline
has them commented out, for upstream’s reasons: all decoration is done in
delivery, and decoration in delivery would break a signature applied here. Both
belong to pepsi-stage-list-deliver and the signing tail.
to-usenet is an explicit, logged no-op. gateway_to_news stays settable
because removing it would change the resource a REST client sees, but Pepsi does
not gateway to NNTP, and a silent no-op would leave an operator believing their
gateway works.
82.4. Headers written¶
Mailman’s X-Mailman-* namespace is renamed to X-Pepsi-List-*, because
nothing in Postorius or HyperKitty reads any of it and the only consumer is a
human reading a held post. The renamed fields are Version, Rule-Hits,
Rule-Misses, Approved-At, Hash-ID, Duplicate and Copy;
upstream’s X-Content-Filtered-By becomes
X-Pepsi-List-Content-Filtered-By. Message-ID-Hash and
its legacy X-Message-ID-Hash spelling keep their names: they carry no
project’s name, every archive URL is derived from them, and they are the ones that a
third party is likely to have built something on.
X-Pepsi-List-Hops has no upstream counterpart and is the only one that the
software reads back in. It is the backstop against a subscription cycle through
two or more lists, since the loop rule catches only a message returning to
the same list. It travels in a header because a sibling row leaves over SMTP
and re-enters as ordinary mail, where row state does not survive. Because a
header is forgeable, pepsi-ingress strips an inbound copy from mail that
did not arrive over an authenticated or local submission path.
82.5. The fan-out¶
Every regular member with delivery enabled and not on digest gets one row,
unconditionally. Each row has its own single recipient and its own VERP envelope
sender <list>-bounces+<local>=<domain>=<code>@<host> (the code keyed with
[pepsi-list] VERP_SECRET). That is what lets the bounce path attribute a
failure to exactly one subscription without parsing the bounce, and refuse to
attribute one to an address that somebody made up. Each row is held paused
for a random time of up to [pepsi] REFLECTION_HOLD_MAX (30 s), drawn per
copy, before the delivery stage takes it up, since a post is reflected mail
(Unknown recipients and backscatter).
What crosses the wire is the recipient list, not the message: the headers are
copied inside the database and the body is shared by reference, so a
five-thousand-member post sends one message and five thousand small JSON objects
to PostgreSQL.
The fan-out is batched: at most [pepsi-list] LIST_FANOUT_BATCH rows
(default 256) are materialised at a time and the message pauses between batches with the remaining
recipients in its own state. Every sibling row references one stored
copy of the cooked body in pepsi.workqueue_body, so the fan-out stores a
5 MB attachment once rather than once per member; a member’s copy gets a body of
its own only when pepsi-stage-list-deliver rewrites it. The batching
bounds how many rows each step adds, and pepsi-queue shows a fan-out in
progress rather than a queue that has suddenly grown. When the queue is at one
of the [pepsi] admission limits, the stage emits an empty batch and looks
again after QUEUE_THROTTLE_DELAY. The recipient list is snapshotted at the
first batch, so an unsubscribe halfway through a large send can neither skip nor double a
neighbour, and the source row is deleted only by the batch that finishes the
list, so a crash replays that batch and nothing more.
A digest issue takes the same path from its second step: pepsi-list
digest injects it here (as [pepsi-list] RELEASE_STAGE) with its digest
members already in state as the snapshot. It is therefore fanned out like a
paused post and runs none of the posting chain: no moderation, archive, digest
append or counters. Each sibling is marked so that
pepsi-stage-list-deliver does not decorate it.
82.6. The posting autoresponder¶
autorespond_postings answers posts on the per-list grace-period machinery
(autoresponse_grace_period) that also rate-limits the command stage’s
results mail. (The sibling attributes autorespond_requests and
autorespond_owner are stored and reported over REST, but no stage acts on
them.) Of the three, autorespond_postings is the one worth reading about. It fires
on the posting address, so a misconfigured autoresponder answers everybody
who writes to the list. On a posting address respond_and_discard
means the post is answered and not distributed — occasionally what an owner
wants (a retired list that answers “we have moved”) and never what they want by
accident, so the discard is logged at warn and the owner console labels the
value plainly. Two rules that are not obvious: an unrecognised value is
``none`` (a response action nobody can spell is not a licence to mail a
membership), and a post that could not be answered is still delivered: when
the autoresponder declines, respond_and_discard does not discard, because a
list that silently dropped every message from an autoresponder would be the
worse failure. Every autoresponse passes the shared RFC 3834 rule set
(pepsi_common::autoreply) first; see pepsi-stage-vacation.
82.7. Configuration¶
[stage-<name>]: PROGRAM = pepsi-stage-list-post, DELIVERY_STAGE
(required, and it must run pepsi-stage-list-deliver),
BOUNCE_STAGE (required — here it means what it means everywhere except
pepsi-stage-list), RESPONSE_STAGE (required, and it must be the
signing tail: the delivery stage accepts only a member’s copy from the
fan-out, so a notice sent there never goes out), REMOVE_DKIM_HEADERS
(default no), SITE_HEADER_MATCH_CHAIN (default hold, site-wide so
that a list owner cannot turn the site’s spam rules into a discard), and the two
availability limits SENDER_POSTS_PER_HOUR (default 30: a sender’s posts to
one list beyond that in an hour are held for the moderator instead of sent to
every member) and MAX_HELD_MESSAGES (default 1000 per list: a post that
would be held beyond that is discarded, so an open list cannot be used to fill
the disk; the new post is dropped rather than the oldest, so a flood cannot
flush out what a moderator has not yet seen). These limits, the
[pepsi-list] MAX_LIST_HOPS ceiling (a post that has passed through that many
lists is held rather than delivered) and the fan-out’s admission throttle are
the mailing-list entries of Availability; Security model (“Resource
limits”) says where each is enforced.
pepsi-setup checks all three targets and refuses a ``NEXT_STAGE``: every
path out of this stage is a terminal.
REMOVE_DKIM_HEADERS is off by default because a list that rewrites nothing
leaves a valid signature and stripping it discards real evidence; turn it on
for lists that add a footer or a subject prefix, where the surviving signature
would fail. A failing signature is worse than none, because a failure is
evidence of tampering and an absence is not.
82.8. State¶
Inputs:
state.list.id(written by pepsi-stage-list),state.auth(the SPF/DKIM/DMARC verdicts that ingress recorded),state.list.pending_rcpton a resumed fan-out, andstate.list.moderator_approvedon a held post a moderator released.Outputs: a
state.listdescriptor on each sibling row — the member id, their language, their unsubscribe serial and the duplicate flag.Transitions: finish (the last batch of a fan-out, a hold, a discard), finish with a side clone (a rejection’s DSN), or pause (a fan-out with recipients remaining).
82.9. See also¶
Mailing lists, Archives, pepsi-stage-list, pepsi-stage-list-deliver, pepsi-list, pepsi-archive, pepsi-stage-list-post(1).