21. Mailing lists

21.1. This subsystem is a reimplementation of GNU Mailman 3

Before anything else, and before the first configuration example, the debt has to be stated plainly, because it is large.

Pepsi’s mailing-list subsystem is a reimplementation of GNU Mailman 3. Its data model, its rule/chain and handler/pipeline architecture, its REST API, its e-mail command vocabulary and its notice-template names are the GNU Mailman project’s design and not ours. The GNU Mailman project and the Free Software Foundation hold the copyright on the original. Pepsi exists to be compatible with it: an unmodified mailmanclient, Postorius or HyperKitty must be able to drive this implementation. That is the acceptance test around which the whole design is built.

Upstream’s documentation at <https://docs.mailman3.org> describes concepts that apply directly to Pepsi’s implementation, and is worth reading alongside this manual. Where this chapter says “the hold action” or “the list:user:notice:welcome template”, it means exactly what Mailman means.

Some of Pepsi’s files were not written by us at all. The notice templates, the settings explanations and a number of test fixtures come from GNU Mailman, Postorius and HyperKitty; they carry their original copyright and licence notices, they remain under the GNU General Public License version 3 or later, and they are listed in vendor/PEPSI-VENDORING.md. GPL version 3’s section 13 permits combining such a work with Pepsi’s AGPLv3+; it permits nothing about relicensing it, and Pepsi has not. COPYING says the same thing for anyone auditing the licensing rather than reading the manual.

Why reuse the text rather than write our own: those notices have been read, complained about and repaired by real list members for two decades, in thirty-four languages. Rewriting them from scratch would take weeks of work, and at best it would arrive back where upstream already is, in fewer languages.

21.2. What a list is

A mailing list belongs to a domain and is identified two ways, which are not interchangeable:

name@domain

The posting address, which is what people type and what appears in To:.

name.domain

The list id, with a dot. This is what the REST API uses as a resource identifier.

pepsi-list accepts either spelling everywhere a list is named. The distinction matters when you talk to the API directly, and it is upstream’s.

Every list answers on nine addresses: the posting address itself, and eight sub-addresses — -request, -join, -leave, -subscribe, -unsubscribe, -owner, -bounces and -confirm+token. Unlike GNU Mailman, Pepsi generates no alias file. Mailman has to write Postfix or Exim alias and transport maps for those nine addresses of every list and tell the MTA to reload; Pepsi routes on the recipient inside its own pipeline, so there is nothing to regenerate, nothing to reload and nothing to get out of sync.

21.3. How the subsystem is put together

The subsystem consists of five stages, two libraries and three surfaces. Everything else in this chapter is a detail of one of them.

inbound mail
     |
     v
+------------------+   not a list   +----------------------------+
| pepsi-stage-list | -------------> | the rest of your pipeline  |
+------------------+                +----------------------------+
     |         |          |
post |  -owner |  -bounces|
     |  -join  |          |
     |  etc.   |          |
     v         v          v
+----------+ +---------+ +--------+
| ...-post | | command | | bounce |
+----------+ +---------+ +--------+
     |            |          |
     | one row     \        | scores, disables,
     | per member    \      | warns, unsubscribes
     v                v      v
+-------------+   +------------------------+
| ...-deliver |   | replies, notices and   |
+-------------+   | probes, injected at    |
     |            | RESPONSE/NOTICE_STAGE  |
     v            +------------------------+
pepsi-stage-dkim-sign  ->  relay  ->  the member
     |
     +--> the archive (pepsi-archive), and the digest accumulator

The five stages. pepsi-stage-list routes; it is the only one that runs on every message. pepsi-stage-list-post moderates a post and fans it out one row per member. pepsi-stage-list-deliver turns one of those rows into that member’s copy. pepsi-stage-list-command answers everything a list does by mail that is not a post. pepsi-stage-list-bounce consumes the failures that come back.

The two libraries. pepsi-list holds the list model, the attribute table, the moderation decisions, the digest builder and the importers. The CLI pepsi-list, the REST API and the owner console all call it, so the three surfaces cannot disagree about what a setting means. pepsi-archive holds the archive: storage, threading, search, export, import and purge, behind the CLI pepsi-archive.

The three surfaces, each on a listener flag of its own, are the reason why a single hostname is not a single trust boundary:

Surface

Listener flag

Who it is for

/api/v1, /ui

ADMIN = yes

the operator — pepsi.admin_account, the pepsi_session cookie; bind it to loopback. See The administration console.

/3.0, /3.1

LIST_API = yes

programs — mailmanclient, Postorius, HyperKitty, over HTTP Basic. See The GNU Mailman 3 REST API.

/lists, /archives

LISTS = yes

list owners, moderators and members — list_user, the pepsi_list_session cookie. See The administration console and Archives.

21.3.1. Three identities, and the words for them

The chapter uses these three words in exactly these senses, and so do the interface, the notices and the man pages.

Operator

Whoever runs the Pepsi installation. Holds a pepsi.admin_account, reaches /ui and /api/v1, edits pepsi.conf, and can do anything to any list. There is no per-list operator.

List owner (and moderator)

Whoever administers one list. Holds a list_user account and a list_member row with role = owner or moderator. Authority is a roster query, not a granted permission. This is Postorius’ model, and it means that adding an owner is subscribing them. An owner changes a list’s settings; a moderator decides its held messages.

Member

One subscription, not one person: an address on a list’s roster. Somebody with two addresses on one list is two members, with two delivery modes, two bounce scores and two unsubscribe URIs. The word subscriber means the same thing in prose; member is what the API, the interface and this manual use in reference material.

A list owner is not a Pepsi operator, and the two account systems never meet; see Two account systems below, and Two account systems, and neither grants the other anything for the tables, cookies and the one hostname that can confuse them.

21.3.2. Where to go next

21.4. Getting started

Register a domain, create a list, give it an owner:

# pepsi-list domain add lists.example.org --base-url https://lists.example.org
added list domain lists.example.org

# pepsi-list list create announce@lists.example.org --style announce-only
created announce@lists.example.org with style legacy-announce

# pepsi-list owner add announce@lists.example.org alice@example.org
alice@example.org is now an owner of announce@lists.example.org

# pepsi-list owner reset-password announce@lists.example.org alice@example.org
password for alice@example.org: xzfkh-9gehq-r7y57-6np8r
(printed rather than mailed: this path has to work when mail is broken)

# pepsi-list check
note: site: archive search: full text and trigram (substring, fuzzy)
1 finding, none fatal

Two warnings about that transcript.

The domain must also be one for which Pepsi accepts mail. pepsi-list domain add registers a domain for lists; [pepsi-ingress] ACCEPTED_DOMAINS decides what the server takes at RCPT. Mail to a list in a domain that ingress does not accept is refused at the SMTP boundary, and nothing in the list subsystem’s logs mentions it, because the message never reached it.

``members add`` bypasses the subscription policy. A command-line tool is there for exactly that, for example to migrate a roster or to fix a mistake. It is also the one thing in this subsystem that can turn a mailing list into a tool for sending spam. The confirmation workflow exists so that an address proves that it wants to be there; the only paths that skip it are this command and an explicitly pre_verified + pre_confirmed call from an authenticated REST client.

21.5. Styles

A style is a named set of attribute defaults applied when a list is created. GET /3.0/lists/styles reports three, which are upstream’s, with upstream’s names, descriptions and default:

Name

Also accepted as

What it is

legacy-default

discussion, default

Ordinary discussion list. The default.

legacy-announce

announce-only, announce

Announce only: members may not post.

private-default

private

Discussion list with private archives; not advertised, and subscriptions need both a confirmation and a moderator.

One further style is Pepsi’s own. It is accepted by pepsi-list and by the owner console, and is not listed over REST:

moderated

A discussion list on which every member’s post is held for a moderator.

An unknown style name is an error. Upstream’s create_list silently applies no style at all when it does not recognise a name, which leaves a list with every attribute at its column default and nothing said anywhere; a migrating site’s provisioning script can fail that way for months.

21.6. Per-list settings

A list carries the 99 attributes that GNU Mailman 3.3.10 exposes over REST. pepsi-list list show prints them; --explain adds a line of prose for each:

$ pepsi-list list show announce@lists.example.org | head -6
accept_these_nonmembers
acceptable_aliases
admin_immed_notify                         yes
admin_notify_mchanges                      no
administrivia                              yes
advertised                                 yes

Seventeen of them are read-only, including the eight derived addresses (posting_address, owner_address, bounces_address and the rest), which are computed from the list’s identity and never stored. A second copy of a list’s identity would be a second place where it can be wrong.

Six more, the *_uri template attributes, exist only in API version 3.0; version 3.1 reaches them through the template manager instead. list show marks both kinds.

Warning

``unsubscription_policy`` does not govern the one-click unsubscribe link. When [pepsi-list] BASE_URL and UNSUBSCRIBE_SECRET are both set, every message that Pepsi sends to a list carries an RFC 8058 List-Unsubscribe-Post header, and a mailbox provider’s “unsubscribe” button calls that target with a single POST. A valid token removes the member immediately, whatever this attribute says — including moderate and confirm. The POST must carry the body List-Unsubscribe=One-Click (as multipart/form-data or application/x-www-form-urlencoded, RFC 8058 section 3.2); without it the request is refused with 400 and nobody is removed, so a link scanner that finds the URL in a message and posts to it unsubscribes no one.

That is deliberate and it is not a gap. RFC 8058 allows no further interaction: a provider that gets a redirect, a confirmation page or a “your request is awaiting approval” has no way to act on any of them, and many will simply mark the mail as spam instead. A header that promises one-click and then does not honour it is worse than not sending one.

The policy still governs everything that a person does: the -leave address, the unsubscribe form on the list’s page, and the REST DELETE. If a list genuinely must moderate every departure, the answer is to stop emitting the header and to accept that the list will look worse to every mailbox provider that checks. Leave [pepsi-list] UNSUBSCRIBE_SECRET unset, and no https: form is emitted at all.

21.6.1. Three settings that are asked about more than the other ninety-six

advertised

Whether the list appears on the public index. It is not a privacy setting: an unadvertised list is still reachable at its own URL, still accepts posts, and its archive is governed by archive_policy and not by this. That is upstream’s meaning and changing it would be a compatibility break. “Unadvertised” reads like “hidden”; it means “not in the catalogue”.

archive_policy

never, private or public. never stores nothing — the archive is not written, so there is nothing to make public later. private stores the messages and refuses anonymous readers: the archive URLs answer as though the list did not exist, because a “forbidden” would confirm that it does. public is public to the internet, including search engines, and /robots.txt asks them to index the archive and leave the search endpoint alone.

member_roster_visibility

Who may see who is subscribed: public, members or moderators. The list’s own page shows the roster only for public. This is the attribute to reach for when somebody says “unadvertised” and means “the subscribers should not be a public list of who corresponds with us”.

All ninety-nine live on eleven screens of the owner console, in Postorius’ grouping (a twelfth holds the settings below). The screens are generated from the same declaration that drives list show and the REST API, so a value that the console refuses is refused by both of the others, in the same words. Two account systems, and neither grants the other anything gives a tour.

21.6.2. Settings Pepsi invented

pepsi-list list set-ext manages a second, much smaller set of per-list settings that Pepsi added and that the REST attribute set never shows:

hold_dmarc_failures

yes or no — hold for moderation every post whose From: domain failed DMARC at intake, whatever policy the domain publishes and whether or not the poster is a member; a failure that a trusted ARC sealer (ARC_TRUSTED_SEALERS) vouched for counts as a pass. A list re-signs and re-sends what it distributes, so a forged post would otherwise reach every member looking authenticated. On for every list Pepsi creates (no row means yes); the importers write no for a list they create from GNU Mailman 2.1 or 3, which has no such rule, so a migrated list keeps distributing what it distributed before until its owner turns it on.

search_trigram

off, short or full — how far this list’s substring and fuzzy search index reaches. Clamped to the site’s SEARCH_TRIGRAM ceiling.

archive_show_addresses

Show posters’ full e-mail addresses to anonymous readers of this list’s public archive. Off by default.

archive_retention_days

Delete archived messages older than this. Absent means the site default.

pepsi-list list show lists all of them, with the value that applies when the list has none of its own; the owner console has a Pepsi settings screen for them, and a REST client reaches them at GET/PATCH /<api>/lists/<id>/pepsi (The GNU Mailman 3 REST API) — a resource of its own, outside /config.

They are absent from /config on purpose, and it is not a stylistic choice. Upstream’s whole-resource PUT /3.0/lists/<id>/config requires every writable attribute to be present in the request. A client built against Mailman 3.3.10 does not know about anything that Pepsi added, so it would omit ours and be rejected: one invented attribute would break every such PUT from every existing client. So Pepsi’s own settings live outside the attribute set entirely, under a path that no Mailman client asks for.

21.7. Configuration

Per-list configuration lives in the database. pepsi.conf holds only what the operator owns:

[pepsi-list]
SITE_OWNER = postmaster@example.org
BASE_URL = https://lists.example.org
API_USER = restadmin
API_PASS = a long random string
SEARCH_TRIGRAM = short
ARCHIVE_PARTITIONS = 32
NOTICE_STAGE = dkim-sign

See pepsi.conf(5) for the full set. Three of them deserve a note here.

NOTICE_STAGE is where the periodic passes put their mail. Most notices are sent by a stage, which has a RESPONSE_STAGE of its own; the bounce warn-and-remove passes are run by a timer, which does not. Without this option those passes can change a member’s state but cannot tell the member, so pepsi-setup refuses a configuration that processes bounces without it.

ARCHIVE_PARTITIONS is fixed at install. The archive is hash-partitioned on the list, and PostgreSQL has no online re-partitioning: changing the modulus means a new table and a full copy of every archived message. pepsi-setup creates the partitions once and refuses to change them silently afterwards, and pepsi-list check reports a configuration that disagrees with what is on disk. Pick it once. Thirty-two is comfortable well past the point where something else becomes the bottleneck.

SEARCH_TRIGRAM is a ceiling, not a default. Each list chooses off, short or full up to it. Lowering the site value therefore shrinks the index at the next reindex, rather than only affecting lists created afterwards.

21.8. Substring search is optional

Pepsi’s archive search uses two PostgreSQL mechanisms, and they answer different questions. Full-text search, with stemming, ranking and snippets, answers “find me the thread about certificates”. Trigram search, from the pg_trgm extension, answers what full text is bad at: a substring, a misspelling, a hostname, a config key, a line from a traceback. On a technical mailing list that is a large share of what people actually search for, and a stemming dictionary throws exactly that material away.

pg_trgm ships in postgresql-contrib, which minimal container images often omit. A site without it still installs and still works; it simply has full-text search only. pepsi-list check says which mode the site is in:

$ pepsi-list check
warning: site: archive search: full text only -- pg_trgm is not installed,
         so substring and fuzzy search are unavailable
    install postgresql-contrib and re-run pepsi-setup; the archive then
    needs `pepsi-archive reindex` to populate trgm_text

21.9. Two account systems

List owners are not Pepsi operators, and the two identities never meet.

pepsi.admin_account and the /ui console are the site operator’s, which the manual tells you to bind to loopback and reach over an SSH tunnel. List owners obviously cannot work that way. So the owner console lives on the public listener and derives its authority from the list roster, that is, from a row in list_member with role = owner or moderator. This is precisely Postorius’ model, where authorisation is a roster query rather than a permission grant.

There are two account systems on purpose, with separate tables, separate session cookies and separate password hashes, and neither grants the other anything. A member session can never satisfy an administrative guard, in either direction. That matters more than it looks: cookies are scoped by host rather than by listener, so an operator who serves both surfaces from one hostname is relying on the cookie names and the guard checks alone.

A member password must be at least ten characters long, must not be on the list of common passwords that Pepsi ships (the passwords that an online guesser tries first, compared without regard to case), and must not contain the member’s address or its local part. This holds wherever the password is chosen: on the verification and reset page /lists/verify/<token>, through the List API’s password and cleartext_password attributes, or with pepsi-list owner reset-password --password. Upstream Mailman accepts any password; the List API answers these with 400 like any other refused value.

Two account systems, and neither grants the other anything sets the two side by side, names the tables and cookies, and says what happens if you serve both from one hostname. An owner who wants to know what they can actually do finds it in the same chapter’s account of the console.

21.10. Bounce processing

A mailing list sends to addresses that stop working, and somebody has to notice. Pepsi’s answer is GNU Mailman 3’s, step for step, because every knob in it is visible through the REST API and a site’s documentation has to keep being true.

The shape of it: a delivery failure comes back to the list’s -bounces address, it is attributed to the member that it concerns, that member’s bounce score goes up by at most one per calendar day, and when the score reaches the list’s threshold their delivery is disabled. Being disabled is not being unsubscribed: a disabled member is warned a few times over the following weeks, and only then removed.

A member whose delivery shows as unknown was not disabled by bounces: the address belongs to a local login that changed owner (A login that changes owner), and its subscriptions stop receiving so that the new owner of the login does not read the previous owner’s lists. The address is also unverified and unlinked from the previous owner’s list account. A list owner can turn delivery back on, or the new owner can subscribe afresh.

21.10.1. Attribution is not guesswork

Every copy of every post that Pepsi sends carries an envelope sender naming the member for whom it is meant: announce-bounces+alice=example.net=<code>@lists.example.org, where <code> authenticates the address (see Probes below). A bounce of that copy therefore says who it is about without anything having to read the bounce itself. Mailman can only do this when an operator turns VERP on; Pepsi’s fan-out is per recipient always, so it is never off. The + is the site’s [pepsi] RECIPIENT_DELIMITER. The fan-out writes the address with this one option and the router splits on it, so the two cannot disagree.

That matters for a practical reason. Upstream has seventeen heuristic bounce detectors: one standard (RFC 3464) and sixteen that name a vendor or a site. They exist because attribution by envelope was not available. Pepsi ports all seventeen (they are flufl.bounce’s, Apache-2.0, with their own fixtures), but they only ever read a bounce that arrived some other way. How often that happens is counted in pepsi.event_log under list.bounce.unrecognized, and that count is the honest answer to “does this site need more detectors”.

A bounce that nothing recognises reaches a human, with the original attached verbatim, unless the list’s forward_unrecognized_bounces_to says discard. That forward is deliberate: without such a sample, nobody can write the next detector.

21.10.2. The nine attributes

Attribute

What it decides

process_bounces

Whether this list scores bounces at all. Off means they are dropped unread.

bounce_score_threshold

How many scoring days disable a member (default 5).

bounce_info_stale_after

How long a score survives without a new bounce (default 7 days). Past it, the next bounce starts again at 1.

bounce_you_are_disabled_warnings

How many warnings a disabled member gets before removal (default 3). Zero means removal with no warning at all.

bounce_you_are_disabled_warnings_interval

How long between those warnings (default 7 days).

bounce_notify_owner_on_bounce_increment

Tell the owners about every increment (default off — on a large list this is a lot of mail).

bounce_notify_owner_on_disable

Tell the owners when a member is disabled (default on).

bounce_notify_owner_on_removal

Tell the owners when a member is removed (default on).

forward_unrecognized_bounces_to

administrators (default), site_owner or discard.

21.10.3. A worked example

A list with the defaults, and an address that has stopped accepting mail:

Day 1

a permanent failure arrives. Score 1. Nobody is told.

Day 1

four more failures the same day. Still score 1 — at most one increment per calendar day.

Days 2–4

one failure each day. Score 4.

Day 5

one more. Score 5 = the threshold: delivery is disabled and the owners are told, with the triggering DSN attached.

Day 5

the member is sent a warning saying their subscription has been disabled and why. Warnings sent: 1.

Day 12

warning 2.

Day 19

warning 3.

Day 26

the warnings have run out and the interval has elapsed again: the member is unsubscribed and the owners are told.

So an address that dies takes about four weeks to leave the list, and a bad afternoon at a provider costs one point. A bounce storm of a thousand failures in an hour also costs exactly one point.

Everything after “disabled” is done by pepsi-list tasks --once, which a timer runs daily. If nothing runs it, members are disabled and then never warned and never removed.

21.10.4. Probes

[pepsi-list] BOUNCE_PROBES = yes changes what crossing the threshold does: instead of disabling the member, Pepsi sends them a message with a one-use token in its envelope sender, and resets their score to zero. If that message bounces, the token names them exactly and they are disabled at once. If it does not, they were fine and nothing happens.

It is on by default (upstream’s is off): every copy’s VERP envelope sender carries an authentication code keyed with [pepsi-list] VERP_SECRET, so a bounce to an address that somebody made up is dropped without being scored. The probe, whose token nobody can forge, is the final check before a member is disabled because of reports that somebody else sent. The trade is one more message to an address that is probably dead, against not disabling somebody whose provider had one bad afternoon; BOUNCE_PROBES = no restores upstream’s behaviour.

21.10.5. What a migration does not bring

A site imported from Mailman starts every member at score zero. Upstream’s own importer does not read bounce history either, so this matches; but it means that a long-dead address gets one more delivery attempt after the cutover.

21.11. When content filtering leaves nothing

A list with filter_content removes the parts of a post that its filter settings name. When nothing is left, filter_action decides what happens to the post. Every action works from the post as it arrived.

Value

What happens to the post

Who is told

hold

It waits for a moderator like any held post.

The moderators and the sender, subject to admin_immed_notify and respond_to_post_requests.

preserve

It is kept the same way, with a reason that says it was preserved.

The moderators, subject to admin_immed_notify. The sender is not told.

reject

It is refused.

The sender, by a DSN that gives the reason.

forward

It goes to the list’s owners as an attachment and is not kept.

The owners. The sender is not told.

discard

It is deleted. This is the default.

Nobody.

You find a preserved post where you find any held post: in the owner console, or at /lists/<id>/held of the REST API. A preserved post is therefore stored as its exact bytes, it counts towards MAX_HELD_MESSAGES, and max_days_to_hold expires it. A list with no owner keeps a post that forward could not send, as with preserve.

If you release a post that hold or preserve kept, you have decided, so it goes to the members as it arrived, without the filter. If the moderation queue is full, hold and preserve cannot keep the post, so its sender gets a DSN.

accept and defer are refused for a new setting: they have no meaning for a post with nothing left in it. A list that already carries one of them keeps it and behaves as with hold.

21.12. What a sender can make a list do

A list multiplies whatever it accepts by its membership, and its posting and command addresses take mail from anybody. In the threat model, list members, owners and moderators are hostile beyond their own lists (Principals and how far each is trusted), and every remote sender is hostile outright. The bounds that follow are the mailing-list entries of Availability; Security model (“Resource limits”) says where each is enforced.

  • Posts per sender. [stage-<name>] SENDER_POSTS_PER_HOUR on the posting stage (default 30): a sender’s accepted posts to one list beyond it in an hour are held for the moderator rather than sent to every member. The count is decided and recorded in one statement, so parallel workers cannot both take the last slot, and a moderator’s release is never counted.

  • The moderation queue. MAX_HELD_MESSAGES (default 1000 per list): a post that would be held beyond it is refused to its sender, the new one rather than the oldest, so a flood cannot flush out the posts a moderator has not yet seen.

  • Cycles between lists. [pepsi-list] MAX_LIST_HOPS (default 5): a post that has passed through that many lists is held rather than delivered. The counter travels in X-Pepsi-List-Hops, which ingress strips from mail that did not arrive over an authenticated or local submission path.

  • The queue. A fan-out obeys the [pepsi] queue admission limits: while the queue is at one, the posting stage emits an empty batch and waits QUEUE_THROTTLE_DELAY rather than adding rows while ingress is already refusing mail.

  • Unsubscribing a stranger. leave takes no address argument: it acts only on the sender’s own address (the first in From:, never an envelope sender that differs from it), and only when that address is verified here. join for another address costs that address one confirmation message and subscribes nobody.

  • The web forms (sign-up, password reset) each mail one target address at most twice an hour, and are rate-limited per client ([pepsi-list] WEB_RATE_LIMIT, WEB_SEARCH_RATE_LIMIT).

21.13. Known limitations

The subsystem covers the data model and the command-line tool, the posting path with its moderation chain and per-member fan-out, the e-mail command interface, bounce processing, the archive and its search (Archives), the GNU Mailman 3 REST API (The GNU Mailman 3 REST API), the public web interface, the member accounts and owner console (The administration console), the importers (Installation) and both digest formats.

What it does not do: /plugins is always empty and there is no plugin interface, there is no NNTP gateway (gateway_to_news is settable and logged as the no-op it is), thread tagging and categories are not in the archive, and a rejected held message is dropped without the rejection notice that upstream sends. Each of those is stated again where a reader would meet it.

21.14. Digests

Pepsi produces both formats, MIME multipart/digest and RFC 1153 plain, because a client can read digest_is_default and mime_is_default_digest back and a member can ask for plaintext_digests. Shipping one format and accepting the attribute for the other would be the quiet kind of lie that the compatibility contract exists to prevent.

21.14.1. How a digest accumulates and how it is sent

Accumulation is part of delivering a post: when digests_enabled is on, the posting path appends the message to one row per list, inside the same transaction as everything else the post does. Upstream appends to an MMDF mailbox file under a lock; here a database row is simply the better tool.

The row carries the messages themselves, not references to an archive. A list may perfectly well have digests_enabled with archive_policy = never, and pepsi-list check even points out that digests are then the only record of what was posted. A digest built from archive references would be empty for exactly the lists that need it most. digest_size_threshold bounds how much accumulates.

Sending has two triggers, and upstream has no periodic runner at all: its maybe_send_digest_now fires synchronously on a post once the size threshold is crossed, and mailman digests --send is meant to be put in the operator’s cron. Pepsi keeps both halves and puts the second where it belongs:

  • size — digest_size_threshold, in kilobytes (upstream’s unit, which the column name does not say). Zero means size never triggers a send, so a list can be periodic-only.

  • periodic — pepsi-list digest periodic from a systemd timer (pepsi-list-digest.timer, hourly), honouring digest_send_periodic per list. pepsi-list tasks uses the same arrangement, which is why there is no long-lived digest service.

pepsi-list digest send forces an issue regardless of the threshold, and digest bump advances the volume without sending — upstream’s two flags.

An issue removes only the messages that it carries, in the same transaction that puts it in the queue. A post that arrives while an issue is being built stays in the row for the next issue, and two runs that send at the same moment produce one issue.

21.14.2. Volume and issue number

digest_volume_frequency (yearly, monthly, quarterly, weekly, daily) decides whether the period has advanced since digest_last_sent_at:

  • a list that has never sent a digest keeps its current volume and number, so a list’s first issue is issue 1 rather than issue 2;

  • if the period has advanced, the volume increments and the number resets to 1;

  • if it has not, the number increments.

The period is compared rather than the elapsed time, which is what makes the answer the same on the 1st of a month as on the 31st: two posts a day apart across a month boundary are in different months, and two posts thirty days apart inside one are not.

21.14.3. What the two formats look like

The MIME digest is a multipart/mixed of five parts in order: the masthead (carrying the issue’s subject as its Content-Description, which is what makes a digest legible in a client that lists parts), the digest header, the table of contents, an inner multipart/digest whose children are the original messages as message/rfc822, and the footer. The header and footer parts are omitted when their template is empty, and list:member:digest:header is empty, upstream and here.

The RFC 1153 digest is one flat text/plain body, and every count in it matters, because readers have split digests on these lines since 1990: seventy hyphens once after the table of contents, thirty before every message except the first, the footer as a fake extra message with its own Subject: Digest Footer rather than as a trailer, and a sign-off followed by a line of asterisks of the same length.

RFC 1153 is plain text, so an attachment cannot survive it. Every non-text part therefore becomes a placeholder naming what was removed (its filename, its media type and its size), because a plain digest that silently dropped a spreadsheet would leave a reader believing they had seen the whole message. A multipart/alternative yields its plain text and says nothing about the HTML twin, since nothing was taken away.

21.14.4. Who gets which

Digest members with delivery enabled split by delivery_mode: plaintext_digests get the RFC 1153 issue, and mime_digests and ``summary_digests`` both get the MIME one. Upstream has no separate summary format and treats the two identically; inventing one would mean that a member could ask for a format that no other Mailman produces.

Each format is built once and fanned out per recipient, on the same terms as any other delivery: a digest carries a List-Unsubscribe too. An issue is injected at [pepsi-list] RELEASE_STAGE (the posting stage) with its recipients already attached, so that stage fans it out without running the posting chain: an issue is not moderated, archived, digested again or counted as a post. Each member’s copy then passes the delivery stage like any other, which adds the one-click unsubscribe pair but not the regular header and footer — an issue carries its own masthead, header and footer.

21.15. Declared differences from GNU Mailman 3

The differences are collected here, because an operator who evaluates the two systems needs them in one place, and a support conversation needs something to point at. Each of them is a decision with a reason, not an accident.

Difference

Why

``filter_action`` does what it says

Upstream takes seven values and silently discards under hold, accept and defer. Here no value but discard loses a post (When content filtering leaves nothing), and accept and defer are refused for a new setting, so a REST client gets a 400. An imported list that carries one of them gets hold, and the import report says so.

RFC 8058 one-click unsubscribe

We emit List-Unsubscribe-Post and upstream does not, and a valid token removes the member immediately whatever ``unsubscription_policy`` says. RFC 8058 allows no further interaction, and a header that promises one click and then does not honour it is worse than no header.

The fan-out is always per recipient

personalize controls whether the message is rewritten for each member, not whether each member gets their own row — because every copy carries that member’s own unsubscribe URI.

Archived addresses are obscured

For anonymous viewers: the sender’s and those in message bodies and search snippets, except the list’s own addresses. Not a security control: what it stops is the bulk harvest that makes a public archive a spam source. A signed-in member sees the whole address.

A purge leaves a 90-day tombstone

So that a re-delivery or a re-run import cannot bring back a message that somebody deleted on purpose. --forget skips it.

``max_days_to_hold`` is honoured

It is inert upstream: a grep of Mailman 3’s whole non-test tree finds the column, the style default and the REST validator, and no job that reads it. Here pepsi-list tasks --once discards held messages past the limit, logs each one and mails the moderators once per list with the count and each post’s sender and subject. A strict superset — the default 0 means “never expire”.

An unknown ``style_name`` is an error

Upstream’s create_list looks the style up, gets nothing, and creates the list with no style applied at all — every attribute at its column default, no error anywhere. A provisioning script with a typo would silently produce lists with none of the intended defaults. We refuse, which cannot break a client that passes a name it read from /lists/styles.

Template override URIs are fetched under limits

A list owner, who is not the operator, can set one, so a file: URI is refused on every write path and never read, the size and time are bounded, and redirects are constrained. Upstream has no such limits.

``archive_rendering_mode = markdown`` is stored and not rendered

The attribute exists because the attribute set is the contract; both values render as text. Rendering user-supplied markdown into our own origin is what the pages’ Content-Security-Policy exists to make impossible, and with no renderer there is no sanitiser to get wrong (Hand-written or single-pass HTML sanitisers let active content survive).

``autorespond_requests`` and ``autorespond_owner`` are stored and not acted on

Both are accepted, validated and reported over REST, and no stage reads them: mail to -request gets the command results and mail to -owner reaches the owners, whatever they say. Only autorespond_postings (and the autoresponse_*_text that goes with it) answers anybody.

No JavaScript anywhere

On either browser surface, except on the pages that run a WebAuthn ceremony (one static script of Pepsi’s own, allowed by hash). The cost is the interactive parts of HyperKitty; the benefit is a policy with no script-src at all, and no third-party script library to fall out of date (Missing HTTPS and HTTP security-header hardening on webmail, Reflected and stored XSS or HTML injection in portals and admin mails).

``X-Mailman-*`` is renamed ``X-Pepsi-List-*``

With a mapping table, because procmail and Sieve rules are a migrating user’s real cost.

No NNTP gateway, no pluggable archivers, no plugin API

The six NNTP attributes stay in the attribute set and are inert, and gateway_to_news is logged when it would have applied rather than ignored — a silent no-op would leave an operator believing their gateway works. A list’s archivers resource names the one archive there is, pepsi, where upstream names the archivers that its configuration enables (mail-archive, mhonarc and prototype in its test configuration).

Member passwords are checked

At least ten characters, not on the shipped list of common passwords, not containing the address or its local part — through the List API as on the member pages; upstream accepts any password. An online guesser tries the common passwords first, and with a stolen hash the guessing happens offline (Offline guessing of weak password hashes, key passphrases and derived keys). The upstream client suites create their users with passwords such as xxx; contrib/compat lengthens them on the way out, because none of those tests is about the password.

``/system/configuration`` has four sections

mailman, webservice, mta and language.en, filled from Pepsi’s own configuration. Upstream answers with every section of its mailman.cfg (runners, logging, paths, archivers), which describe a configuration that Pepsi does not have; inventing values for them would be worse than leaving them out.

``/queues`` has four queues, and no virgin

in, retry, bad and shunt count the work-queue rows in the matching state (The GNU Mailman 3 REST API). A message that Pepsi composes itself is an ordinary row at its response stage, so there is no separate queue of generated mail to report. The List API does not send a welcome message for a subscription that it completes either; send_welcome_message is accepted and has no effect (docs/ROADMAP.md).

A post that failed DMARC is held (hold_dmarc_failures)

A Pepsi setting and a rule of Pepsi’s own, dmarc-failure, in the built-in posting chain after banned-address. Upstream’s DMARC mitigation only rewrites or refuses posts from domains that publish p=reject or p=quarantine; it distributes a failing post from a p=none domain re-signed under the list’s name, which turns a forgery into authenticated mail (Forwarders, relays and lists launder spoofed mail into authenticated mail). On for lists that Pepsi creates, no for imported ones, so a migrated list behaves as it did.

A weak moderator password hash is upgraded

When an Approved: header verifies against a hash made with weaker parameters than new hashes get, the stage stores a fresh hash of it, as the sign-ins do (Offline guessing of weak password hashes, key passphrases and derived keys).

Passwords, bounce scores and in-flight tokens are not migrated

Upstream’s own importer carries none of them either, and an upstream password hash is refused rather than stored beside our salted Argon2id ones (Offline guessing of weak password hashes, key passphrases and derived keys). See Migrating from GNU Mailman.