17. Mailing lists¶
17.1. This subsystem is a reimplementation of GNU Mailman 3¶
Before anything else, and before the first configuration example, the debt has to be stated plainly, because it is large.
Pepsi’s mailing-list subsystem is a reimplementation of GNU Mailman 3. Its
data model, its rule/chain and handler/pipeline architecture, its REST API, its
e-mail command vocabulary and its notice-template names are the GNU Mailman
project’s design and not ours. The GNU Mailman project and the Free Software
Foundation hold the copyright on the original. Pepsi exists to be compatible
with it: an unmodified mailmanclient, Postorius or HyperKitty must be able
to drive this implementation, which is the acceptance test the whole design is
built around.
Upstream’s documentation at <https://docs.mailman3.org> describes concepts that
apply directly to Pepsi’s implementation, and is worth reading alongside this
manual. Where this chapter says “the hold action” or “the
list:user:notice:welcome template”, it means exactly what Mailman means.
Some of Pepsi’s files were not written by us at all. The notice templates, the
settings explanations and a number of test fixtures come from GNU Mailman,
Postorius and HyperKitty; they carry their original copyright and licence
notices, they remain under the GNU General Public License version 3 or later,
and they are listed in vendor/PEPSI-VENDORING.md. GPL version 3’s section
13 permits combining such a work with Pepsi’s AGPLv3+; it permits nothing about
relicensing it, and Pepsi has not. COPYING says the same thing for anyone
auditing the licensing rather than reading the manual.
Why reuse the text rather than write our own: those notices have been read, complained about and repaired by real list members for two decades, in thirty-four languages. Rewriting them from scratch would have been weeks of work whose best possible outcome was to arrive back where upstream already is, in fewer languages.
17.2. What a list is¶
A mailing list belongs to a domain and is identified two ways, which are not interchangeable:
- name
@domain The posting address, which is what people type and what appears in
To:.- name
.domain The list id, with a dot. This is what the REST API uses as a resource identifier.
pepsi-list accepts either spelling everywhere a list is named. The
distinction matters when you talk to the API directly, and it is upstream’s.
Every list answers on nine addresses: the posting address itself, and eight
sub-addresses — -request, -join, -leave, -subscribe,
-unsubscribe, -owner, -bounces and -confirm+token. Unlike
GNU Mailman, Pepsi generates no alias file. Mailman has to write Postfix or
Exim alias and transport maps for those nine addresses of every list and tell
the MTA to reload; Pepsi routes on the recipient inside its own pipeline, so
there is nothing to regenerate, nothing to reload and nothing to get out of
sync.
17.3. How the subsystem is put together¶
Five stages, two libraries and three surfaces. Everything else in this chapter is a detail of one of them.
inbound mail
|
v
+------------------+ not a list +----------------------------+
| pepsi-stage-list | -------------> | the rest of your pipeline |
+------------------+ +----------------------------+
| | |
post | -owner | -bounces|
| -join | |
| etc. | |
v v v
+----------+ +---------+ +--------+
| ...-post | | command | | bounce |
+----------+ +---------+ +--------+
| | |
| one row \ | scores, disables,
| per member \ | warns, unsubscribes
v v v
+-------------+ +------------------------+
| ...-deliver | | replies, notices and |
+-------------+ | probes, injected at |
| | RESPONSE/NOTICE_STAGE |
v +------------------------+
pepsi-stage-dkim-sign -> relay -> the member
|
+--> the archive (pepsi-archive), and the digest accumulator
The five stages. pepsi-stage-list routes; it is the only one that runs on every message. pepsi-stage-list-post moderates a post and fans it out one row per member. pepsi-stage-list-deliver turns one of those rows into that member’s copy. pepsi-stage-list-command answers everything a list does by mail that is not a post. pepsi-stage-list-bounce consumes the failures that come back.
The two libraries. pepsi-list holds the list model, the attribute table,
the moderation decisions, the digest builder and the importers — and is what the
CLI pepsi-list, the REST API and the owner console all call, so
the three surfaces cannot disagree about what a setting means.
pepsi-archive holds the archive: storage, threading, search, export, import
and purge, behind the CLI pepsi-archive.
The three surfaces, each on a listener flag of its own, are the reason a single hostname is not a single trust boundary:
Surface |
Listener flag |
Who it is for |
|---|---|---|
|
|
the operator — |
|
|
programs — |
|
|
list owners, moderators and members — |
17.3.1. Three identities, and the words for them¶
The chapter uses these three words in exactly these senses, and so do the interface, the notices and the man pages.
- Operator
Whoever runs the Pepsi installation. Holds a
pepsi.admin_account, reaches/uiand/api/v1, editspepsi.conf, and can do anything to any list. There is no per-list operator.- List owner (and moderator)
Whoever administers one list. Holds a
list_useraccount and alist_memberrow withrole = ownerormoderator— authority is a roster query, not a granted permission, which is Postorius’ model and means adding an owner is subscribing them. An owner changes a list’s settings; a moderator decides its held messages.- Member
One subscription, not one person: an address on a list’s roster. Somebody with two addresses on one list is two members, with two delivery modes, two bounce scores and two unsubscribe URIs. The word subscriber means the same thing in prose; member is what the API, the interface and this manual use in reference material.
A list owner is not a Pepsi operator, and the two account systems never meet; see Two account systems below, and Two account systems, and neither grants the other anything for the tables, cookies and the one hostname that can confuse them.
17.3.2. Where to go next¶
Running a list, its settings and its digests — the rest of this chapter.
Migrating from GNU Mailman 2.1 or 3 — Installation, then What a migration does not bring.
Driving Pepsi with upstream’s own clients — The GNU Mailman 3 REST API.
The archive, its search and its purge — Archives.
The browser surfaces — The administration console.
The standards involved — RFC Index, which collects RFC 2369, RFC 5064, RFC 1153, RFC 3464 and RFC 8058 in one place.
17.4. Getting started¶
Register a domain, create a list, give it an owner:
# pepsi-list domain add lists.example.org --base-url https://lists.example.org
added list domain lists.example.org
# pepsi-list list create announce@lists.example.org --style announce-only
created announce@lists.example.org with style legacy-announce
# pepsi-list owner add announce@lists.example.org alice@example.org
alice@example.org is now an owner of announce@lists.example.org
# pepsi-list owner reset-password announce@lists.example.org alice@example.org
password for alice@example.org: xzfkh-9gehq-r7y57-6np8r
(printed rather than mailed: this path has to work when mail is broken)
# pepsi-list check
note: site: archive search: full text and trigram (substring, fuzzy)
1 finding, none fatal
Two warnings about that transcript.
The domain must also be one Pepsi accepts mail for. pepsi-list domain
add registers a domain for lists; [pepsi-ingress] ACCEPTED_DOMAINS
decides what the server takes at RCPT. A list in a domain ingress does not
accept is refused at the SMTP boundary, and nothing in the list subsystem’s
logs mentions it, because the message never reached it.
``members add`` bypasses the subscription policy. That is what a
command-line tool is for — migrating a roster, fixing a mistake — and it is
also the one thing in this subsystem that can turn a mailing list into a
spam cannon. The confirmation workflow exists so that an address proves it
wants to be there; the only paths that skip it are this command and an
explicitly pre_verified + pre_confirmed call from an authenticated REST
client.
17.5. Styles¶
A style is a named set of attribute defaults applied when a list is
created. GET /3.0/lists/styles reports three, which are upstream’s, with
upstream’s names, descriptions and default:
Name |
Also accepted as |
What it is |
|---|---|---|
|
|
Ordinary discussion list. The default. |
|
|
Announce only: members may not post. |
|
|
Discussion list with private archives; not advertised, and subscriptions need both a confirmation and a moderator. |
One further style is Pepsi’s own. It is accepted by pepsi-list and by the
owner console, and is not listed over REST:
moderatedA discussion list on which every member’s post is held for a moderator.
An unknown style name is an error. Upstream’s create_list silently
applies no style at all when it does not recognise a name, which leaves a list
with every attribute at its column default and nothing said anywhere; a
migrating site’s provisioning script can fail that way for months.
17.6. Per-list settings¶
A list carries the 99 attributes GNU Mailman 3.3.10 exposes over REST.
pepsi-list list show prints them; --explain adds a line of prose for
each:
$ pepsi-list list show announce@lists.example.org | head -6
accept_these_nonmembers
acceptable_aliases
admin_immed_notify yes
admin_notify_mchanges no
administrivia yes
advertised yes
Seventeen of them are read-only, including the eight derived addresses
(posting_address, owner_address, bounces_address and the rest),
which are computed from the list’s identity and never stored — a second copy of
a list’s identity is a second place it can be wrong.
Six more, the *_uri template attributes, exist only in API version 3.0;
version 3.1 reaches them through the template manager instead. list show
marks both kinds.
Warning
``unsubscription_policy`` does not govern the one-click unsubscribe
link. When [pepsi-list] BASE_URL and UNSUBSCRIBE_SECRET are both
set, every message Pepsi sends to a list carries an RFC 8058
List-Unsubscribe-Post header, and a mailbox provider’s “unsubscribe”
button calls that target with a single POST. A valid token removes the
member immediately, whatever this attribute says — including
moderate and confirm.
That is deliberate and it is not a gap. RFC 8058 allows no further interaction: a provider that gets a redirect, a confirmation page or a “your request is awaiting approval” has no way to act on any of them, and many will simply mark the mail as spam instead. A header that promises one-click and then does not honour it is worse than not sending one.
The policy still governs everything a person does: the -leave
address, the unsubscribe form on the list’s page, and the REST
DELETE. If a list genuinely must moderate every departure, the answer
is to stop emitting the header — leave [pepsi-list]
UNSUBSCRIBE_SECRET unset and no https: form is emitted at all — and
to accept that the list will look worse to every mailbox provider that
checks.
17.6.1. Three settings that are asked about more than the other ninety-six¶
advertisedWhether the list appears on the public index. It is not a privacy setting: an unadvertised list is still reachable at its own URL, still accepts posts, and its archive is governed by
archive_policyand not by this. That is upstream’s meaning and changing it would be a compatibility break. “Unadvertised” reads like “hidden”; it means “not in the catalogue”.archive_policynever,privateorpublic.neverstores nothing — the archive is not written, so there is nothing to make public later.privatestores the messages and refuses anonymous readers: the archive URLs answer as though the list did not exist, because a “forbidden” would confirm that it does.publicis public to the internet, including search engines, and/robots.txtasks them to index the archive and leave the search endpoint alone.member_roster_visibilityWho may see who is subscribed:
public,membersormoderators. The list’s own page shows the roster only forpublic. This is the attribute to reach for when somebody says “unadvertised” and means “the subscribers should not be a public list of who corresponds with us”.
All ninety-nine live on eleven screens of the owner console, in Postorius’
grouping, and the screens are generated from the same declaration that drives
list show and the REST API — so a value the console refuses is refused by
both of the others, in the same words. Two account systems, and neither grants the other anything has the tour.
17.6.2. Settings Pepsi invented¶
pepsi-list list set-ext manages a second, much smaller set of per-list
settings that Pepsi added and that the REST API never shows:
search_trigramoff,shortorfull— how far this list’s substring and fuzzy search index reaches. Clamped to the site’sSEARCH_TRIGRAMceiling.archive_show_addressesShow posters’ full e-mail addresses to anonymous readers of this list’s public archive. Off by default.
archive_retention_daysDelete archived messages older than this. Absent means the site default.
They are invisible to REST on purpose, and it is not a stylistic choice.
Upstream’s whole-resource PUT /3.0/lists/<id>/config requires every
writable attribute to be present in the request. A client built against Mailman
3.3.10 does not know about anything Pepsi added, so it would omit ours and be
rejected — one invented attribute would break every such PUT from every
existing client. So Pepsi’s own settings live outside the attribute set
entirely, reachable from the command line and the owner console and from
nowhere a Mailman client can see.
17.7. Configuration¶
Per-list configuration lives in the database. pepsi.conf holds only what the
operator owns:
[pepsi-list]
SITE_OWNER = postmaster@example.org
BASE_URL = https://lists.example.org
API_USER = restadmin
API_PASS = a long random string
SEARCH_TRIGRAM = short
ARCHIVE_PARTITIONS = 32
NOTICE_STAGE = dkim-sign
See pepsi.conf(5) for the full set. Three of them deserve a note here.
NOTICE_STAGE is where the periodic passes put their mail. Most notices
are sent by a stage, which has a RESPONSE_STAGE of its own; the bounce
warn-and-remove passes are run by a timer, which does not. Without this option
those passes can change a member’s state but cannot tell the member, so
pepsi-setup refuses a configuration that processes bounces without it.
ARCHIVE_PARTITIONS is fixed at install. The archive is hash-partitioned
on the list, and PostgreSQL has no online re-partitioning: changing the modulus
means a new table and a full copy of every archived message. pepsi-setup
creates the partitions once and refuses to change them silently afterwards, and
pepsi-list check reports a configuration that disagrees with what is on
disk. Pick it once. Thirty-two is comfortable well past the point where
something else becomes the bottleneck.
SEARCH_TRIGRAM is a ceiling, not a default. Each list chooses off,
short or full up to it. Lowering the site value therefore shrinks the
index at the next reindex, rather than only affecting lists created afterwards.
17.8. Substring search is optional¶
Pepsi’s archive search uses two PostgreSQL mechanisms, and they answer
different questions. Full-text search — stemming, ranking, snippets — answers
“find me the thread about certificates”. Trigram search, from the pg_trgm
extension, answers what full text is bad at: a substring, a misspelling, a
hostname, a config key, a line from a traceback. On a technical mailing list
that is a large share of what people actually search for, and a stemming
dictionary throws exactly that material away.
pg_trgm ships in postgresql-contrib, which minimal container images
often omit. A site without it still installs and still works; it simply has
full-text search only. pepsi-list check says which mode the site is in:
$ pepsi-list check
warning: site: archive search: full text only -- pg_trgm is not installed,
so substring and fuzzy search are unavailable
install postgresql-contrib and re-run pepsi-setup; the archive then
needs `pepsi-archive reindex` to populate trgm_text
17.9. Two account systems¶
List owners are not Pepsi operators, and the two identities never meet.
pepsi.admin_account and the /ui console are the site operator’s, which
the manual tells you to bind to loopback and reach over an SSH tunnel. List
owners obviously cannot work that way. So the owner console lives on the public
listener and derives its authority from the list roster — a row in
list_member with role = owner or moderator — which is precisely
Postorius’ model, where authorisation is a roster query rather than a
permission grant.
Two account systems, on purpose, with separate tables, separate session cookies and separate password hashes, and neither grants the other anything. A member session can never satisfy an administrative guard, in either direction. That matters more than it looks: cookies are scoped by host rather than by listener, so an operator who serves both surfaces from one hostname is relying on the cookie names and the guard checks alone.
Two account systems, and neither grants the other anything sets the two side by side, names the tables and cookies, and says what happens if you serve both from one hostname. An owner looking for what they can actually do wants the same chapter’s account of the console.
17.10. Bounce processing¶
A mailing list sends to addresses that stop working, and somebody has to notice. Pepsi’s answer is GNU Mailman 3’s, step for step, because every knob in it is visible through the REST API and a site’s documentation has to keep being true.
The shape of it: a delivery failure comes back to the list’s -bounces
address, it is attributed to the member it concerns, that member’s bounce
score goes up by at most one per calendar day, and when the score reaches the
list’s threshold their delivery is disabled. Being disabled is not being
unsubscribed — a disabled member is warned a few times over the following
weeks, and only then removed.
17.10.1. Attribution is not guesswork¶
Every copy of every post Pepsi sends carries an envelope sender naming the
member it is for: announce-bounces+alice=example.net@lists.example.org. A
bounce of that copy therefore says who it is about without anything having to
read the bounce itself. Mailman can only do this when an operator turns VERP
on; Pepsi’s fan-out is per recipient always, so it is never off. The + is
the site’s [pepsi] RECIPIENT_DELIMITER — the one option the fan-out writes
with and the router splits on, so the two cannot disagree.
That matters for a practical reason. Upstream’s seventeen heuristic bounce
detectors — one standard (RFC 3464) and sixteen that name a vendor or a site —
exist because attribution by envelope was not available. Pepsi ports all
seventeen (they are flufl.bounce’s, Apache-2.0, with their own fixtures),
but they only ever read a bounce that arrived some other way. How often that
happens is counted in pepsi.event_log under list.bounce.unrecognized,
and that count is the honest answer to “does this site need more detectors”.
A bounce nothing recognises reaches a human, with the original attached
verbatim, unless the list’s forward_unrecognized_bounces_to says
discard. That forward is deliberate: the sample is the only way the next
detector gets written.
17.10.2. The nine attributes¶
Attribute |
What it decides |
|---|---|
|
Whether this list scores bounces at all. Off means they are dropped unread. |
|
How many scoring days disable a member (default 5). |
|
How long a score survives without a new bounce (default 7 days). Past it, the next bounce starts again at 1. |
|
How many warnings a disabled member gets before removal (default 3). Zero means removal with no warning at all. |
|
How long between those warnings (default 7 days). |
|
Tell the owners about every increment (default off — on a large list this is a lot of mail). |
|
Tell the owners when a member is disabled (default on). |
|
Tell the owners when a member is removed (default on). |
|
|
17.10.3. A worked example¶
A list with the defaults, and an address that has stopped accepting mail:
Day 1 |
a permanent failure arrives. Score 1. Nobody is told. |
Day 1 |
four more failures the same day. Still score 1 — at most one increment per calendar day. |
Days 2–4 |
one failure each day. Score 4. |
Day 5 |
one more. Score 5 = the threshold: delivery is disabled and the owners are told, with the triggering DSN attached. |
Day 5 |
the member is sent a warning saying their subscription has been disabled and why. Warnings sent: 1. |
Day 12 |
warning 2. |
Day 19 |
warning 3. |
Day 26 |
the warnings have run out and the interval has elapsed again: the member is unsubscribed and the owners are told. |
So an address that dies takes about four weeks to leave the list, and a bad afternoon at a provider costs one point. A bounce storm — a thousand failures in an hour — costs exactly one point too.
Everything after “disabled” is done by pepsi-list tasks --once, which a
timer runs daily. If nothing runs it, members are disabled and then never
warned and never removed.
17.10.4. Probes¶
[pepsi-list] BOUNCE_PROBES = yes changes what crossing the threshold does:
instead of disabling the member, Pepsi sends them a message with a one-use
token in its envelope sender, and resets their score to zero. If that message
bounces, the token names them exactly and they are disabled at once. If it does
not, they were fine and nothing happens.
It is off by default, as it is upstream. The trade is one more message to an address that is probably dead, against not disabling somebody whose provider had one bad afternoon.
17.10.5. What a migration does not bring¶
A site imported from Mailman starts every member at score zero. Upstream’s own importer does not read bounce history either, so this matches — but it means a long-dead address gets one more delivery attempt after the cutover.
17.11. Known limitations¶
The subsystem covers the data model and the command-line tool, the posting path with its moderation chain and per-member fan-out, the e-mail command interface, bounce processing, the archive and its search (Archives), the GNU Mailman 3 REST API (The GNU Mailman 3 REST API), the public web interface, the member accounts and owner console (The administration console), the importers (Installation) and both digest formats.
What it does not do: /plugins is always empty and there is no plugin
interface, there is no NNTP gateway (gateway_to_news is settable and logged
as the no-op it is), thread tagging
and categories are not in the archive, and a rejected held message is dropped
without the rejection notice upstream sends. Each of those is stated again where
a reader would meet it.
17.12. Digests¶
Both formats, MIME multipart/digest and RFC 1153 plain, because a client can
read digest_is_default and mime_is_default_digest back and a member can
ask for plaintext_digests — shipping one format and accepting the attribute
for the other would be the quiet kind of lie the compatibility contract exists to
prevent.
17.12.1. How a digest accumulates and how it is sent¶
Accumulation is part of delivering a post: when digests_enabled is on, the
posting path appends the message to one row per list, inside the same transaction
as everything else the post does. Upstream appends to an MMDF mailbox file under
a lock; a row is one of the places a database is simply better.
The row carries the messages themselves, not references to an archive. A list
may perfectly well have digests_enabled with archive_policy = never —
pepsi-list check even points out that digests are then the only record of what
was posted — so a digest built from archive references would be empty for exactly
the lists that need it most. digest_size_threshold bounds how much
accumulates.
Sending has two triggers, and upstream has no periodic runner at all: its
maybe_send_digest_now fires synchronously on a post once the size threshold is
crossed, and mailman digests --send is meant to be put in the operator’s cron.
Pepsi keeps both halves and puts the second where it belongs:
size —
digest_size_threshold, in kilobytes (upstream’s unit, which the column name does not say). Zero means size never triggers a send, so a list can be periodic-only.periodic —
pepsi-list digest periodicfrom a systemd timer, honouringdigest_send_periodicper list. The same arrangementpepsi-list taskshas, and the reason no long-lived digest service is written.
pepsi-list digest send forces an issue regardless of the threshold, and
digest bump advances the volume without sending — upstream’s two flags.
17.12.2. Volume and issue number¶
digest_volume_frequency (yearly, monthly, quarterly, weekly,
daily) decides whether the period has advanced since
digest_last_sent_at:
a list that has never sent a digest keeps its current volume and number, so a list’s first issue is issue 1 rather than issue 2;
if the period has advanced, the volume increments and the number resets to 1;
if it has not, the number increments.
The period is compared rather than the elapsed time, which is what makes the answer the same on the 1st of a month as on the 31st: two posts a day apart across a month boundary are in different months, and two posts thirty days apart inside one are not.
17.12.3. What the two formats look like¶
The MIME digest is a multipart/mixed of five parts in order: the masthead
(carrying the issue’s subject as its Content-Description, which is what makes
a digest legible in a client that lists parts), the digest header, the table of
contents, an inner multipart/digest whose children are the original messages
as message/rfc822, and the footer. The header and footer parts are omitted
when their template is empty — which list:member:digest:header is, upstream
and here.
The RFC 1153 digest is one flat text/plain body, and every count in it is
load-bearing because readers have been splitting digests on these lines since
1990: seventy hyphens once after the table of contents, thirty before every
message except the first, the footer as a fake extra message with its own
Subject: Digest Footer rather than as a trailer, and a sign-off followed by a
line of asterisks of the same length.
RFC 1153 is plain text, so an attachment cannot survive it. Every non-text part
therefore becomes a placeholder naming what was removed — its filename, its
media type and its size — because a plain digest that silently dropped a
spreadsheet would leave a reader believing they had seen the whole message. A
multipart/alternative yields its plain text and says nothing about the HTML
twin, since nothing was taken away.
17.12.4. Who gets which¶
Digest members with delivery enabled split by delivery_mode:
plaintext_digests get the RFC 1153 issue, and mime_digests and
``summary_digests`` both get the MIME one. Upstream has no separate summary
format and treats the two identically; inventing one would mean a member could
ask for a format no other Mailman produces.
Each format is built once and fanned out per recipient, on the same terms as any
other delivery — a digest carries a List-Unsubscribe too.
17.13. Declared differences from GNU Mailman 3¶
Collected in one place, because an operator evaluating the two systems needs it in one place and a support conversation needs something to point at. Each of these is a decision with a reason, not an accident.
Difference |
Why |
|---|---|
RFC 8058 one-click unsubscribe |
We emit |
The fan-out is always per recipient |
|
Archived sender addresses are obscured |
For anonymous viewers, by default, with a per-list override. Not a security control: what it stops is the bulk harvest that makes a public archive a spam source. A signed-in member sees the whole address. |
A purge leaves a 90-day tombstone |
So that a re-delivery or a re-run import cannot resurrect a message
somebody deleted on purpose. |
``max_days_to_hold`` is honoured |
It is inert upstream: a grep of Mailman 3’s whole non-test tree finds
the column, the style default and the REST validator, and no job that
reads it. Here |
An unknown ``style_name`` is an error |
Upstream’s |
Template override URIs are fetched under limits |
A list owner — not the operator — can set one, so a |
``archive_rendering_mode = markdown`` is stored and not rendered |
The attribute exists because the attribute set is the contract; both values render as text. Rendering user-supplied markdown into our own origin is what the pages’ Content-Security-Policy exists to make impossible. |
No JavaScript anywhere |
On either browser surface. The cost is the interactive parts of
HyperKitty; the benefit is a policy with no |
``X-Mailman-*`` is renamed ``X-Pepsi-List-*`` |
With a mapping table, because procmail and Sieve rules are a migrating user’s real cost. |
No NNTP gateway, no pluggable archivers, no plugin API |
The six NNTP attributes stay in the attribute set and are inert, and
|
Passwords, bounce scores and in-flight tokens are not migrated |
Upstream’s own importer carries none of them either. See Migrating from GNU Mailman. |