21. Mailing lists¶
21.1. This subsystem is a reimplementation of GNU Mailman 3¶
Before anything else, and before the first configuration example, the debt has to be stated plainly, because it is large.
Pepsi’s mailing-list subsystem is a reimplementation of GNU Mailman 3. Its
data model, its rule/chain and handler/pipeline architecture, its REST API, its
e-mail command vocabulary and its notice-template names are the GNU Mailman
project’s design and not ours. The GNU Mailman project and the Free Software
Foundation hold the copyright on the original. Pepsi exists to be compatible
with it: an unmodified mailmanclient, Postorius or HyperKitty must be able
to drive this implementation. That is the acceptance test around which the
whole design is built.
Upstream’s documentation at <https://docs.mailman3.org> describes concepts that
apply directly to Pepsi’s implementation, and is worth reading alongside this
manual. Where this chapter says “the hold action” or “the
list:user:notice:welcome template”, it means exactly what Mailman means.
Some of Pepsi’s files were not written by us at all. The notice templates, the
settings explanations and a number of test fixtures come from GNU Mailman,
Postorius and HyperKitty; they carry their original copyright and licence
notices, they remain under the GNU General Public License version 3 or later,
and they are listed in vendor/PEPSI-VENDORING.md. GPL version 3’s section
13 permits combining such a work with Pepsi’s AGPLv3+; it permits nothing about
relicensing it, and Pepsi has not. COPYING says the same thing for anyone
auditing the licensing rather than reading the manual.
Why reuse the text rather than write our own: those notices have been read, complained about and repaired by real list members for two decades, in thirty-four languages. Rewriting them from scratch would take weeks of work, and at best it would arrive back where upstream already is, in fewer languages.
21.2. What a list is¶
A mailing list belongs to a domain and is identified two ways, which are not interchangeable:
- name
@domain The posting address, which is what people type and what appears in
To:.- name
.domain The list id, with a dot. This is what the REST API uses as a resource identifier.
pepsi-list accepts either spelling everywhere a list is named. The
distinction matters when you talk to the API directly, and it is upstream’s.
Every list answers on nine addresses: the posting address itself, and eight
sub-addresses — -request, -join, -leave, -subscribe,
-unsubscribe, -owner, -bounces and -confirm+token. Unlike
GNU Mailman, Pepsi generates no alias file. Mailman has to write Postfix or
Exim alias and transport maps for those nine addresses of every list and tell
the MTA to reload; Pepsi routes on the recipient inside its own pipeline, so
there is nothing to regenerate, nothing to reload and nothing to get out of
sync.
21.3. How the subsystem is put together¶
The subsystem consists of five stages, two libraries and three surfaces. Everything else in this chapter is a detail of one of them.
inbound mail
|
v
+------------------+ not a list +----------------------------+
| pepsi-stage-list | -------------> | the rest of your pipeline |
+------------------+ +----------------------------+
| | |
post | -owner | -bounces|
| -join | |
| etc. | |
v v v
+----------+ +---------+ +--------+
| ...-post | | command | | bounce |
+----------+ +---------+ +--------+
| | |
| one row \ | scores, disables,
| per member \ | warns, unsubscribes
v v v
+-------------+ +------------------------+
| ...-deliver | | replies, notices and |
+-------------+ | probes, injected at |
| | RESPONSE/NOTICE_STAGE |
v +------------------------+
pepsi-stage-dkim-sign -> relay -> the member
|
+--> the archive (pepsi-archive), and the digest accumulator
The five stages. pepsi-stage-list routes; it is the only one that runs on every message. pepsi-stage-list-post moderates a post and fans it out one row per member. pepsi-stage-list-deliver turns one of those rows into that member’s copy. pepsi-stage-list-command answers everything a list does by mail that is not a post. pepsi-stage-list-bounce consumes the failures that come back.
The two libraries. pepsi-list holds the list model, the attribute table,
the moderation decisions, the digest builder and the importers. The CLI
pepsi-list, the REST API and the owner console all call it, so
the three surfaces cannot disagree about what a setting means.
pepsi-archive holds the archive: storage, threading, search, export, import
and purge, behind the CLI pepsi-archive.
The three surfaces, each on a listener flag of its own, are the reason why a single hostname is not a single trust boundary:
Surface |
Listener flag |
Who it is for |
|---|---|---|
|
|
the operator — |
|
|
programs — |
|
|
list owners, moderators and members — |
21.3.1. Three identities, and the words for them¶
The chapter uses these three words in exactly these senses, and so do the interface, the notices and the man pages.
- Operator
Whoever runs the Pepsi installation. Holds a
pepsi.admin_account, reaches/uiand/api/v1, editspepsi.conf, and can do anything to any list. There is no per-list operator.- List owner (and moderator)
Whoever administers one list. Holds a
list_useraccount and alist_memberrow withrole = ownerormoderator. Authority is a roster query, not a granted permission. This is Postorius’ model, and it means that adding an owner is subscribing them. An owner changes a list’s settings; a moderator decides its held messages.- Member
One subscription, not one person: an address on a list’s roster. Somebody with two addresses on one list is two members, with two delivery modes, two bounce scores and two unsubscribe URIs. The word subscriber means the same thing in prose; member is what the API, the interface and this manual use in reference material.
A list owner is not a Pepsi operator, and the two account systems never meet; see Two account systems below, and Two account systems, and neither grants the other anything for the tables, cookies and the one hostname that can confuse them.
21.3.2. Where to go next¶
Running a list, its settings and its digests — the rest of this chapter.
Migrating from GNU Mailman 2.1 or 3 — Installation, then What a migration does not bring.
Driving Pepsi with upstream’s own clients — The GNU Mailman 3 REST API.
The archive, its search and its purge — Archives.
The browser surfaces — The administration console.
The standards involved — RFC Index, which collects RFC 2369, RFC 5064, RFC 1153, RFC 3464 and RFC 8058 in one place.
21.4. Getting started¶
Register a domain, create a list, give it an owner:
# pepsi-list domain add lists.example.org --base-url https://lists.example.org
added list domain lists.example.org
# pepsi-list list create announce@lists.example.org --style announce-only
created announce@lists.example.org with style legacy-announce
# pepsi-list owner add announce@lists.example.org alice@example.org
alice@example.org is now an owner of announce@lists.example.org
# pepsi-list owner reset-password announce@lists.example.org alice@example.org
password for alice@example.org: xzfkh-9gehq-r7y57-6np8r
(printed rather than mailed: this path has to work when mail is broken)
# pepsi-list check
note: site: archive search: full text and trigram (substring, fuzzy)
1 finding, none fatal
Two warnings about that transcript.
The domain must also be one for which Pepsi accepts mail. pepsi-list
domain add registers a domain for lists; [pepsi-ingress]
ACCEPTED_DOMAINS decides what the server takes at RCPT. Mail to a list in
a domain that ingress does not accept is refused at the SMTP boundary, and
nothing in the list subsystem’s logs mentions it, because the message never
reached it.
``members add`` bypasses the subscription policy. A command-line tool is
there for exactly that, for example to migrate a roster or to fix a mistake.
It is also the one thing in this subsystem that can turn a mailing list into a
tool for sending spam. The confirmation workflow exists so that an address
proves that it wants to be there; the only paths that skip it are this command
and an
explicitly pre_verified + pre_confirmed call from an authenticated REST
client.
21.5. Styles¶
A style is a named set of attribute defaults applied when a list is
created. GET /3.0/lists/styles reports three, which are upstream’s, with
upstream’s names, descriptions and default:
Name |
Also accepted as |
What it is |
|---|---|---|
|
|
Ordinary discussion list. The default. |
|
|
Announce only: members may not post. |
|
|
Discussion list with private archives; not advertised, and subscriptions need both a confirmation and a moderator. |
One further style is Pepsi’s own. It is accepted by pepsi-list and by the
owner console, and is not listed over REST:
moderatedA discussion list on which every member’s post is held for a moderator.
An unknown style name is an error. Upstream’s create_list silently
applies no style at all when it does not recognise a name, which leaves a list
with every attribute at its column default and nothing said anywhere; a
migrating site’s provisioning script can fail that way for months.
21.6. Per-list settings¶
A list carries the 99 attributes that GNU Mailman 3.3.10 exposes over REST.
pepsi-list list show prints them; --explain adds a line of prose for
each:
$ pepsi-list list show announce@lists.example.org | head -6
accept_these_nonmembers
acceptable_aliases
admin_immed_notify yes
admin_notify_mchanges no
administrivia yes
advertised yes
Seventeen of them are read-only, including the eight derived addresses
(posting_address, owner_address, bounces_address and the rest),
which are computed from the list’s identity and never stored. A second copy of
a list’s identity would be a second place where it can be wrong.
Six more, the *_uri template attributes, exist only in API version 3.0;
version 3.1 reaches them through the template manager instead. list show
marks both kinds.
Warning
``unsubscription_policy`` does not govern the one-click unsubscribe
link. When [pepsi-list] BASE_URL and UNSUBSCRIBE_SECRET are both
set, every message that Pepsi sends to a list carries an RFC 8058
List-Unsubscribe-Post header, and a mailbox provider’s “unsubscribe”
button calls that target with a single POST. A valid token removes the
member immediately, whatever this attribute says — including
moderate and confirm. The POST must carry the body
List-Unsubscribe=One-Click (as multipart/form-data or
application/x-www-form-urlencoded, RFC 8058 section 3.2); without it
the request is refused with 400 and nobody is removed, so a link
scanner that finds the URL in a message and posts to it unsubscribes no
one.
That is deliberate and it is not a gap. RFC 8058 allows no further interaction: a provider that gets a redirect, a confirmation page or a “your request is awaiting approval” has no way to act on any of them, and many will simply mark the mail as spam instead. A header that promises one-click and then does not honour it is worse than not sending one.
The policy still governs everything that a person does: the -leave
address, the unsubscribe form on the list’s page, and the REST
DELETE. If a list genuinely must moderate every departure, the answer
is to stop emitting the header and to accept that the list will look worse
to every mailbox provider that checks. Leave [pepsi-list]
UNSUBSCRIBE_SECRET unset, and no https: form is emitted at all.
21.6.1. Three settings that are asked about more than the other ninety-six¶
advertisedWhether the list appears on the public index. It is not a privacy setting: an unadvertised list is still reachable at its own URL, still accepts posts, and its archive is governed by
archive_policyand not by this. That is upstream’s meaning and changing it would be a compatibility break. “Unadvertised” reads like “hidden”; it means “not in the catalogue”.archive_policynever,privateorpublic.neverstores nothing — the archive is not written, so there is nothing to make public later.privatestores the messages and refuses anonymous readers: the archive URLs answer as though the list did not exist, because a “forbidden” would confirm that it does.publicis public to the internet, including search engines, and/robots.txtasks them to index the archive and leave the search endpoint alone.member_roster_visibilityWho may see who is subscribed:
public,membersormoderators. The list’s own page shows the roster only forpublic. This is the attribute to reach for when somebody says “unadvertised” and means “the subscribers should not be a public list of who corresponds with us”.
All ninety-nine live on eleven screens of the owner console, in Postorius’
grouping (a twelfth holds the settings below). The screens are generated from
the same declaration that drives list show and the REST API, so a value
that the console refuses is refused by both of the others, in the same words.
Two account systems, and neither grants the other anything gives a tour.
21.6.2. Settings Pepsi invented¶
pepsi-list list set-ext manages a second, much smaller set of per-list
settings that Pepsi added and that the REST attribute set never shows:
hold_dmarc_failuresyesorno— hold for moderation every post whoseFrom:domain failed DMARC at intake, whatever policy the domain publishes and whether or not the poster is a member; a failure that a trusted ARC sealer (ARC_TRUSTED_SEALERS) vouched for counts as a pass. A list re-signs and re-sends what it distributes, so a forged post would otherwise reach every member looking authenticated. On for every list Pepsi creates (no row meansyes); the importers writenofor a list they create from GNU Mailman 2.1 or 3, which has no such rule, so a migrated list keeps distributing what it distributed before until its owner turns it on.search_trigramoff,shortorfull— how far this list’s substring and fuzzy search index reaches. Clamped to the site’sSEARCH_TRIGRAMceiling.archive_show_addressesShow posters’ full e-mail addresses to anonymous readers of this list’s public archive. Off by default.
archive_retention_daysDelete archived messages older than this. Absent means the site default.
pepsi-list list show lists all of them, with the value that applies when the
list has none of its own; the owner console has a Pepsi settings screen for
them, and a REST client reaches them at GET/PATCH
/<api>/lists/<id>/pepsi (The GNU Mailman 3 REST API) — a resource of its own, outside
/config.
They are absent from /config on purpose, and it is not a stylistic
choice. Upstream’s whole-resource PUT /3.0/lists/<id>/config requires
every writable attribute to be present in the request. A client built against
Mailman 3.3.10 does not know about anything that Pepsi added, so it would omit
ours and be rejected: one invented attribute would break every such PUT
from every existing client. So Pepsi’s own settings live outside the attribute
set entirely, under a path that no Mailman client asks for.
21.7. Configuration¶
Per-list configuration lives in the database. pepsi.conf holds only what the
operator owns:
[pepsi-list]
SITE_OWNER = postmaster@example.org
BASE_URL = https://lists.example.org
API_USER = restadmin
API_PASS = a long random string
SEARCH_TRIGRAM = short
ARCHIVE_PARTITIONS = 32
NOTICE_STAGE = dkim-sign
See pepsi.conf(5) for the full set. Three of them deserve a note here.
NOTICE_STAGE is where the periodic passes put their mail. Most notices
are sent by a stage, which has a RESPONSE_STAGE of its own; the bounce
warn-and-remove passes are run by a timer, which does not. Without this option
those passes can change a member’s state but cannot tell the member, so
pepsi-setup refuses a configuration that processes bounces without it.
ARCHIVE_PARTITIONS is fixed at install. The archive is hash-partitioned
on the list, and PostgreSQL has no online re-partitioning: changing the modulus
means a new table and a full copy of every archived message. pepsi-setup
creates the partitions once and refuses to change them silently afterwards, and
pepsi-list check reports a configuration that disagrees with what is on
disk. Pick it once. Thirty-two is comfortable well past the point where
something else becomes the bottleneck.
SEARCH_TRIGRAM is a ceiling, not a default. Each list chooses off,
short or full up to it. Lowering the site value therefore shrinks the
index at the next reindex, rather than only affecting lists created afterwards.
21.8. Substring search is optional¶
Pepsi’s archive search uses two PostgreSQL mechanisms, and they answer
different questions. Full-text search, with stemming, ranking and snippets,
answers “find me the thread about certificates”. Trigram search, from the
pg_trgm extension, answers what full text is bad at: a substring, a
misspelling, a hostname, a config key, a line from a traceback. On a technical
mailing list that is a large share of what people actually search for, and a
stemming dictionary throws exactly that material away.
pg_trgm ships in postgresql-contrib, which minimal container images
often omit. A site without it still installs and still works; it simply has
full-text search only. pepsi-list check says which mode the site is in:
$ pepsi-list check
warning: site: archive search: full text only -- pg_trgm is not installed,
so substring and fuzzy search are unavailable
install postgresql-contrib and re-run pepsi-setup; the archive then
needs `pepsi-archive reindex` to populate trgm_text
21.9. Two account systems¶
List owners are not Pepsi operators, and the two identities never meet.
pepsi.admin_account and the /ui console are the site operator’s, which
the manual tells you to bind to loopback and reach over an SSH tunnel. List
owners obviously cannot work that way. So the owner console lives on the public
listener and derives its authority from the list roster, that is, from a row in
list_member with role = owner or moderator. This is precisely
Postorius’ model, where authorisation is a roster query rather than a
permission grant.
There are two account systems on purpose, with separate tables, separate session cookies and separate password hashes, and neither grants the other anything. A member session can never satisfy an administrative guard, in either direction. That matters more than it looks: cookies are scoped by host rather than by listener, so an operator who serves both surfaces from one hostname is relying on the cookie names and the guard checks alone.
A member password must be at least ten characters long, must not be on the
list of common passwords that Pepsi ships (the passwords that an online
guesser tries first, compared without regard to case), and must not contain
the member’s address or its local part. This holds wherever the password is
chosen: on the verification and reset page /lists/verify/<token>, through
the List API’s password and cleartext_password attributes, or with
pepsi-list owner reset-password --password. Upstream Mailman accepts any
password; the List API answers these with 400 like any other refused value.
Two account systems, and neither grants the other anything sets the two side by side, names the tables and cookies, and says what happens if you serve both from one hostname. An owner who wants to know what they can actually do finds it in the same chapter’s account of the console.
21.10. Bounce processing¶
A mailing list sends to addresses that stop working, and somebody has to notice. Pepsi’s answer is GNU Mailman 3’s, step for step, because every knob in it is visible through the REST API and a site’s documentation has to keep being true.
The shape of it: a delivery failure comes back to the list’s -bounces
address, it is attributed to the member that it concerns, that member’s
bounce score goes up by at most one per calendar day, and when the score
reaches the list’s threshold their delivery is disabled. Being disabled is
not being unsubscribed: a disabled member is warned a few times over the
following weeks, and only then removed.
A member whose delivery shows as unknown was not disabled by bounces: the address belongs to a local login that changed owner (A login that changes owner), and its subscriptions stop receiving so that the new owner of the login does not read the previous owner’s lists. The address is also unverified and unlinked from the previous owner’s list account. A list owner can turn delivery back on, or the new owner can subscribe afresh.
21.10.1. Attribution is not guesswork¶
Every copy of every post that Pepsi sends carries an envelope sender naming the
member for whom it is meant:
announce-bounces+alice=example.net=<code>@lists.example.org, where
<code> authenticates the address (see Probes below). A bounce of that
copy therefore says who it is about without anything having to read the bounce
itself. Mailman can only do this when an operator turns VERP on; Pepsi’s fan-out
is per recipient always, so it is never off. The + is the site’s [pepsi]
RECIPIENT_DELIMITER. The fan-out writes the address with this one option and
the router splits on it, so the two cannot disagree.
That matters for a practical reason. Upstream has seventeen heuristic bounce
detectors: one standard (RFC 3464) and sixteen that name a vendor or a site.
They exist because attribution by envelope was not available. Pepsi ports all
seventeen (they are flufl.bounce’s, Apache-2.0, with their own fixtures),
but they only ever read a bounce that arrived some other way. How often that
happens is counted in pepsi.event_log under list.bounce.unrecognized,
and that count is the honest answer to “does this site need more detectors”.
A bounce that nothing recognises reaches a human, with the original
attached verbatim, unless the list’s forward_unrecognized_bounces_to says
discard. That forward is deliberate: without such a sample, nobody can
write the next detector.
21.10.2. The nine attributes¶
Attribute |
What it decides |
|---|---|
|
Whether this list scores bounces at all. Off means they are dropped unread. |
|
How many scoring days disable a member (default 5). |
|
How long a score survives without a new bounce (default 7 days). Past it, the next bounce starts again at 1. |
|
How many warnings a disabled member gets before removal (default 3). Zero means removal with no warning at all. |
|
How long between those warnings (default 7 days). |
|
Tell the owners about every increment (default off — on a large list this is a lot of mail). |
|
Tell the owners when a member is disabled (default on). |
|
Tell the owners when a member is removed (default on). |
|
|
21.10.3. A worked example¶
A list with the defaults, and an address that has stopped accepting mail:
Day 1 |
a permanent failure arrives. Score 1. Nobody is told. |
Day 1 |
four more failures the same day. Still score 1 — at most one increment per calendar day. |
Days 2–4 |
one failure each day. Score 4. |
Day 5 |
one more. Score 5 = the threshold: delivery is disabled and the owners are told, with the triggering DSN attached. |
Day 5 |
the member is sent a warning saying their subscription has been disabled and why. Warnings sent: 1. |
Day 12 |
warning 2. |
Day 19 |
warning 3. |
Day 26 |
the warnings have run out and the interval has elapsed again: the member is unsubscribed and the owners are told. |
So an address that dies takes about four weeks to leave the list, and a bad afternoon at a provider costs one point. A bounce storm of a thousand failures in an hour also costs exactly one point.
Everything after “disabled” is done by pepsi-list tasks --once, which a
timer runs daily. If nothing runs it, members are disabled and then never
warned and never removed.
21.10.4. Probes¶
[pepsi-list] BOUNCE_PROBES = yes changes what crossing the threshold does:
instead of disabling the member, Pepsi sends them a message with a one-use
token in its envelope sender, and resets their score to zero. If that message
bounces, the token names them exactly and they are disabled at once. If it does
not, they were fine and nothing happens.
It is on by default (upstream’s is off): every copy’s VERP envelope sender
carries an authentication code keyed with [pepsi-list] VERP_SECRET, so a
bounce to an address that somebody made up is dropped without being scored.
The probe, whose token nobody can forge, is the final check before a member is
disabled because of reports that somebody else sent. The trade is one more
message to an address that is probably dead, against not disabling somebody
whose provider had one bad afternoon; BOUNCE_PROBES = no restores
upstream’s behaviour.
21.10.5. What a migration does not bring¶
A site imported from Mailman starts every member at score zero. Upstream’s own importer does not read bounce history either, so this matches; but it means that a long-dead address gets one more delivery attempt after the cutover.
21.11. When content filtering leaves nothing¶
A list with filter_content removes the parts of a post that its filter
settings name. When nothing is left, filter_action decides what happens to the
post. Every action works from the post as it arrived.
Value |
What happens to the post |
Who is told |
|---|---|---|
|
It waits for a moderator like any held post. |
The moderators and the sender, subject to |
|
It is kept the same way, with a reason that says it was preserved. |
The moderators, subject to |
|
It is refused. |
The sender, by a DSN that gives the reason. |
|
It goes to the list’s owners as an attachment and is not kept. |
The owners. The sender is not told. |
|
It is deleted. This is the default. |
Nobody. |
You find a preserved post where you find any held post: in the owner console, or
at /lists/<id>/held of the REST API. A preserved post is therefore stored as
its exact bytes, it counts towards MAX_HELD_MESSAGES, and max_days_to_hold
expires it. A list with no owner keeps a post that forward could not send, as
with preserve.
If you release a post that hold or preserve kept, you have decided, so it
goes to the members as it arrived, without the filter. If the moderation queue is
full, hold and preserve cannot keep the post, so its sender gets a DSN.
accept and defer are refused for a new setting: they have no meaning for
a post with nothing left in it. A list that already carries one of them keeps it
and behaves as with hold.
21.12. What a sender can make a list do¶
A list multiplies whatever it accepts by its membership, and its posting and command addresses take mail from anybody. In the threat model, list members, owners and moderators are hostile beyond their own lists (Principals and how far each is trusted), and every remote sender is hostile outright. The bounds that follow are the mailing-list entries of Availability; Security model (“Resource limits”) says where each is enforced.
Posts per sender.
[stage-<name>] SENDER_POSTS_PER_HOURon the posting stage (default 30): a sender’s accepted posts to one list beyond it in an hour are held for the moderator rather than sent to every member. The count is decided and recorded in one statement, so parallel workers cannot both take the last slot, and a moderator’s release is never counted.The moderation queue.
MAX_HELD_MESSAGES(default 1000 per list): a post that would be held beyond it is refused to its sender, the new one rather than the oldest, so a flood cannot flush out the posts a moderator has not yet seen.Cycles between lists.
[pepsi-list] MAX_LIST_HOPS(default 5): a post that has passed through that many lists is held rather than delivered. The counter travels inX-Pepsi-List-Hops, which ingress strips from mail that did not arrive over an authenticated or local submission path.The queue. A fan-out obeys the
[pepsi]queue admission limits: while the queue is at one, the posting stage emits an empty batch and waitsQUEUE_THROTTLE_DELAYrather than adding rows while ingress is already refusing mail.Unsubscribing a stranger.
leavetakes no address argument: it acts only on the sender’s own address (the first inFrom:, never an envelope sender that differs from it), and only when that address is verified here.joinfor another address costs that address one confirmation message and subscribes nobody.The web forms (sign-up, password reset) each mail one target address at most twice an hour, and are rate-limited per client (
[pepsi-list] WEB_RATE_LIMIT,WEB_SEARCH_RATE_LIMIT).
21.13. Known limitations¶
The subsystem covers the data model and the command-line tool, the posting path with its moderation chain and per-member fan-out, the e-mail command interface, bounce processing, the archive and its search (Archives), the GNU Mailman 3 REST API (The GNU Mailman 3 REST API), the public web interface, the member accounts and owner console (The administration console), the importers (Installation) and both digest formats.
What it does not do: /plugins is always empty and there is no plugin
interface, there is no NNTP gateway (gateway_to_news is settable and logged
as the no-op it is), thread tagging
and categories are not in the archive, and a rejected held message is dropped
without the rejection notice that upstream sends. Each of those is stated again
where a reader would meet it.
21.14. Digests¶
Pepsi produces both formats, MIME multipart/digest and RFC 1153 plain,
because a client can read digest_is_default and mime_is_default_digest
back and a member can ask for plaintext_digests. Shipping one format and
accepting the attribute for the other would be the quiet kind of lie that the
compatibility contract exists to prevent.
21.14.1. How a digest accumulates and how it is sent¶
Accumulation is part of delivering a post: when digests_enabled is on, the
posting path appends the message to one row per list, inside the same transaction
as everything else the post does. Upstream appends to an MMDF mailbox file under
a lock; here a database row is simply the better tool.
The row carries the messages themselves, not references to an archive. A list
may perfectly well have digests_enabled with archive_policy = never, and
pepsi-list check even points out that digests are then the only record of
what was posted. A digest built from archive references would be empty for
exactly the lists that need it most. digest_size_threshold bounds how much
accumulates.
Sending has two triggers, and upstream has no periodic runner at all: its
maybe_send_digest_now fires synchronously on a post once the size threshold is
crossed, and mailman digests --send is meant to be put in the operator’s cron.
Pepsi keeps both halves and puts the second where it belongs:
size —
digest_size_threshold, in kilobytes (upstream’s unit, which the column name does not say). Zero means size never triggers a send, so a list can be periodic-only.periodic —
pepsi-list digest periodicfrom a systemd timer (pepsi-list-digest.timer, hourly), honouringdigest_send_periodicper list.pepsi-list tasksuses the same arrangement, which is why there is no long-lived digest service.
pepsi-list digest send forces an issue regardless of the threshold, and
digest bump advances the volume without sending — upstream’s two flags.
An issue removes only the messages that it carries, in the same transaction that puts it in the queue. A post that arrives while an issue is being built stays in the row for the next issue, and two runs that send at the same moment produce one issue.
21.14.2. Volume and issue number¶
digest_volume_frequency (yearly, monthly, quarterly, weekly,
daily) decides whether the period has advanced since
digest_last_sent_at:
a list that has never sent a digest keeps its current volume and number, so a list’s first issue is issue 1 rather than issue 2;
if the period has advanced, the volume increments and the number resets to 1;
if it has not, the number increments.
The period is compared rather than the elapsed time, which is what makes the answer the same on the 1st of a month as on the 31st: two posts a day apart across a month boundary are in different months, and two posts thirty days apart inside one are not.
21.14.3. What the two formats look like¶
The MIME digest is a multipart/mixed of five parts in order: the masthead
(carrying the issue’s subject as its Content-Description, which is what makes
a digest legible in a client that lists parts), the digest header, the table of
contents, an inner multipart/digest whose children are the original messages
as message/rfc822, and the footer. The header and footer parts are omitted
when their template is empty, and list:member:digest:header is empty,
upstream and here.
The RFC 1153 digest is one flat text/plain body, and every count in it
matters, because readers have split digests on these lines since 1990: seventy
hyphens once after the table of contents, thirty before every message except
the first, the footer as a fake extra message with its own Subject:
Digest Footer rather than as a trailer, and a sign-off followed by a line of
asterisks of the same length.
RFC 1153 is plain text, so an attachment cannot survive it. Every non-text part
therefore becomes a placeholder naming what was removed (its filename, its
media type and its size), because a plain digest that silently dropped a
spreadsheet would leave a reader believing they had seen the whole message. A
multipart/alternative yields its plain text and says nothing about the HTML
twin, since nothing was taken away.
21.14.4. Who gets which¶
Digest members with delivery enabled split by delivery_mode:
plaintext_digests get the RFC 1153 issue, and mime_digests and
``summary_digests`` both get the MIME one. Upstream has no separate summary
format and treats the two identically; inventing one would mean that a member
could ask for a format that no other Mailman produces.
Each format is built once and fanned out per recipient, on the same terms as any
other delivery: a digest carries a List-Unsubscribe too. An issue is
injected at [pepsi-list] RELEASE_STAGE (the posting stage) with its
recipients already attached, so that stage fans it out without running the
posting chain: an issue is not moderated, archived, digested again or counted as
a post. Each member’s copy then passes the delivery stage like any other, which
adds the one-click unsubscribe pair but not the regular header and footer —
an issue carries its own masthead, header and footer.
21.15. Declared differences from GNU Mailman 3¶
The differences are collected here, because an operator who evaluates the two systems needs them in one place, and a support conversation needs something to point at. Each of them is a decision with a reason, not an accident.
Difference |
Why |
|---|---|
``filter_action`` does what it says |
Upstream takes seven values and silently discards under |
RFC 8058 one-click unsubscribe |
We emit |
The fan-out is always per recipient |
|
Archived addresses are obscured |
For anonymous viewers: the sender’s and those in message bodies and search snippets, except the list’s own addresses. Not a security control: what it stops is the bulk harvest that makes a public archive a spam source. A signed-in member sees the whole address. |
A purge leaves a 90-day tombstone |
So that a re-delivery or a re-run import cannot bring back a message
that somebody deleted on purpose. |
``max_days_to_hold`` is honoured |
It is inert upstream: a grep of Mailman 3’s whole non-test tree finds
the column, the style default and the REST validator, and no job that
reads it. Here |
An unknown ``style_name`` is an error |
Upstream’s |
Template override URIs are fetched under limits |
A list owner, who is not the operator, can set one, so a |
``archive_rendering_mode = markdown`` is stored and not rendered |
The attribute exists because the attribute set is the contract; both values render as text. Rendering user-supplied markdown into our own origin is what the pages’ Content-Security-Policy exists to make impossible, and with no renderer there is no sanitiser to get wrong (Hand-written or single-pass HTML sanitisers let active content survive). |
``autorespond_requests`` and ``autorespond_owner`` are stored and not acted on |
Both are accepted, validated and reported over REST, and no stage reads
them: mail to |
No JavaScript anywhere |
On either browser surface, except on the pages that run a WebAuthn
ceremony (one static script of Pepsi’s own, allowed by hash). The cost
is the interactive parts of HyperKitty; the benefit is a policy with no
|
``X-Mailman-*`` is renamed ``X-Pepsi-List-*`` |
With a mapping table, because procmail and Sieve rules are a migrating user’s real cost. |
No NNTP gateway, no pluggable archivers, no plugin API |
The six NNTP attributes stay in the attribute set and are inert, and
|
Member passwords are checked |
At least ten characters, not on the shipped list of common passwords,
not containing the address or its local part — through the List API as
on the member pages; upstream accepts any password. An online guesser
tries the common passwords first, and with a stolen hash the guessing
happens offline (Offline guessing of weak password hashes, key passphrases and derived keys).
The upstream client suites create their users with passwords such as
|
``/system/configuration`` has four sections |
|
``/queues`` has four queues, and no |
|
A post that failed DMARC is held ( |
A Pepsi setting and a rule of Pepsi’s own, |
A weak moderator password hash is upgraded |
When an |
Passwords, bounce scores and in-flight tokens are not migrated |
Upstream’s own importer carries none of them either, and an upstream password hash is refused rather than stored beside our salted Argon2id ones (Offline guessing of weak password hashes, key passphrases and derived keys). See Migrating from GNU Mailman. |