11. Supported Features¶
This chapter lists the protocol- and policy-level features that Pepsi supports, with the governing RFC for each: the SMTP and authentication machinery, the end-to-end cryptography, and the operational surface around both. The RFC Index cross-references every RFC to the code and configuration; the per-program chapters list which program realises each feature.
11.1. Inbound SMTP reception¶
pepsi-ingress is a standards SMTP server (RFC 5321) with these extensions:
Transport security. Cleartext, STARTTLS (RFC 3207) upgrade, and implicit TLS (RFC 8314, e.g. the submissions port 465) — selected per listener via
MODE. Submission listeners (RFC 6409,SUBMISSION = yes) may be bound — see Message submission.Message body transfer. Classic
DATAand CHUNKING/BDAT (RFC 3030) for large messages.BINARYMIMEis not advertised, and aMAIL FROMcarryingBODY=BINARYMIMEis refused with501 5.5.4.SIZE advertisement and enforcement (
MAX_MESSAGE_SIZE; oversize → 552).PIPELINING and ENHANCEDSTATUSCODES.
8BITMIME (RFC 6152) and SMTPUTF8 (RFC 6531) — see Internationalised and 8-bit content.
DSN (RFC 3461) — see Delivery Status Notifications.
RRVS (RFC 7293) — see Recycled addresses.
REQUIRETLS (RFC 8689), on TLS-protected sessions — see Require TLS.
SMTP AUTH (RFC 4954) for submission — see Client authentication.
Acceptance policy: mail is accepted only for
ACCEPTED_DOMAINS(others are rejected as relaying with550); the domainless<Postmaster>mailbox is always accepted (RFC 5321 §4.5.1); recipients that are valid SRS tokens are reverse-decoded and relayed (see Sender Rewriting Scheme (SRS)).Durability: the client is told
250only after the row is committed; if the database is unavailable the client gets a transient451and retries.
11.2. Trace headers¶
Ingress prepends an RFC 5321 §4.4 Received: header to every message before
authentication (so the ARC seal covers it). The with clause records the
transmission type per RFC 3848: SMTP, ESMTP, ESMTPS, ESMTPA or
ESMTPSA. The bare ESMTPA, authenticated but not over TLS, is the local
UNIX submission socket. A for clause names the recipient for
single-recipient transactions.
11.3. Client authentication¶
Each listener can authenticate the connecting client and mark its mail as
locally originated (which lets it relay onward to any domain and stamps
ESMTPSA in the Received: trace). A session is authenticated when any of
four per-listener mechanisms succeeds:
the peer IP is in the listener’s
MYNETWORKS(trusted up front; the addresses that such a network may send as are bound withSENDER_MAP, and an entry outside loopback that the map does not cover is refused at start-up);an SMTP AUTH exchange succeeds (RFC 4954,
PLAIN/LOGIN), verified against a configured SASL backend (SASL_TYPE/SASL_PATH; Dovecot’s auth-client socket today).AUTHis advertised and accepted only over TLS: a cleartext attempt is refused with504 5.5.4, the code that RFC 4954 §4 prescribes for a mechanism that “requires an encryption layer” (§1 deprecates the older538);a presented TLS client certificate is a pinned key (
TLS_AUTH_CLIENT) or validates, constraints and revocation included, to a configured CA (TLS_AUTH_CLIENT_CA), andSENDER_MAPbinds it to its addresses (an unbound certificate authenticates nothing);the peer of a UNIX socket is identified by
SO_PEERCRED(AUTH_PEERCRED, the local submission socket) and its login is mapped to at least one address. This one carries a username, so it feeds the sameUSERNAME_MAPenforcement that a SASL session gets.
This is distinct from Boundary authentication below, which verifies the message (SPF/DKIM/DMARC) regardless of how the client itself authenticated, or whether it did. See pepsi-ingress.
11.4. Message submission¶
A listener marked SUBMISSION = yes behaves as a Message Submission Agent
(RFC 6409) instead of an MX:
authentication is mandatory — an unauthenticated
MAIL FROMis refused with530 5.7.0(so the flag requires one of the client-auth mechanisms above, and TLS on any listener that is not a local UNIX-domain socket); andsubmission fixups are applied to each accepted message: a missing
Date:(§8.2) andMessage-ID:(§8.3) are added before the message is stored; andsubmission-identity enforcement (§6) via
USERNAME_MAP: the envelopeMAIL FROMand theFrom:header must be addresses that the authenticated user is permitted to use (otherwise550), and an allowedFrom:that differs from the user’s canonical identity gains aSender:header naming it (§8.1). EveryFrom:mailbox is read with a parser that skips quoted strings and RFC 5322 comments, so a comment cannot hide the address actually used (Ambiguous From headers checked as one address and displayed as another). A message without aFrom:, with more than one, with only a group or with a mailbox lacking a full address is refused (550 5.6.0), and so is aFrom:whose display name names an address other than its own, because many clients show only the name.
The flag has no effect on an MX listener. A local shell user and the The submission identity state what this binding does and does not protect.
11.5. Boundary authentication¶
Before storing a message, ingress authenticates it, so that the eventual receiver can still see the verdict that Pepsi reached at the boundary (forwarding will break the original SPF/DKIM signals):
SPF (RFC 7208) on the
MAIL FROMidentity using the client IP.DKIM signature verification (RFC 6376 / RFC 8463).
DMARC evaluation (RFC 7489), with SMTP-time enforcement (
DMARC_ENFORCE→550on a definite failure under aquarantine/rejectpolicy; the wizard writes it for a new installation) and an ARC override for the forwarders and lists named inARC_TRUSTED_SEALERS(a validating chain sealed last by one of them is accepted, recorded asstate.auth.arc_override). Authentication is otherwise fail-open: a DNS or parse error is recorded and the mail accepted.iprev / FCrDNS (RFC 8601): whether the client’s PTR record forward-confirms to its IP.
An Authentication-Results header (RFC 8601) recording all of the above, stamped with the ingress hostname as
authserv-id.
The other half of DMARC is the reports that other receivers send about mail
under our domains. pepsi-stage-dmarc-report reads the
aggregate reports (RFC 7489 §7.2) mailed to dmarc-reports@<domain>, which
is the rua= address of the record that pepsi-setup suggests. From the
authenticated reports, pepsi-setup check, pepsi-status and
the console say whether every sending source aligned, so that p=reject is
safe, or which sources it refuses.
Under DMARC_ENFORCE DMARC is evaluated per author domain (at most
MAX_DMARC_AUTHOR_DOMAINS, 10), so a second From: mailbox cannot carry a
forgery past the first. Mail arriving from outside is also cleaned at this
boundary: a message with more than one From: field is refused, and a
Bcc: field is removed (a reply-all would otherwise quote decrypted text to an
address that the attacker listed there). Submitted mail loses its Bcc: field
too, so a client that leaves one in cannot reveal the hidden recipients. A From: whose display name names a
different address is marked with X-Pepsi-Warning:
display-name-claims-other-address, and one carrying bidi controls or
invisible characters is marked with X-Pepsi-Warning:
from-has-bidi-or-invisible-characters; any X-Pepsi-Warning that the sender
wrote is removed first. A From: that reads more than one way or names
mailboxes in several domains (from-ambiguous), and a Sender: at a served
domain that nobody authenticated (served-sender-not-authenticated), are
marked and tagged [external] like a forged served From:. A From:
domain that looks like a served one (from-domain-resembles-served-domain)
and an attachment whose file name hides bidi or invisible characters
(attachment-name-has-bidi-or-invisible-characters) are marked in the header
only. The adversary this answers is A remote sender.
11.6. Authenticated Received Chain (ARC)¶
The pepsi-stage-arc stage implements ARC (RFC 8617). It
verifies any inbound ARC chain and seals the message as our ADMD by prepending an
ARC-Authentication-Results / ARC-Message-Signature / ARC-Seal set,
signed with the single ARC_ALGORITHM (ARC permits one signature per hop).
This lets a downstream receiver trust the boundary verdict after forwarding
breaks SPF/DKIM alignment. Sealing is fail-open. Because ARC preserves an
upstream sender’s authentication across the forwarding hop, it applies only to
mail that Pepsi receives. Locally-originated submissions
(state.local_origin) are skipped, because the DKIM-signing stage
authenticates them as the author domain. The generated pipeline therefore
places ARC on the inbound branch, after the state.local_origin split.
11.7. Sender Rewriting Scheme (SRS)¶
The pepsi-stage-srs stage rewrites the envelope sender into a
local address of a Pepsi-controlled SRS_DOMAIN that HMAC-encodes the original
sender (the truncated MAC is base32-encoded, RFC 4648), so SPF passes at the
next hop. The null sender and an address already in the SRS domain are left
unchanged; an already-SRS address is re-signed in the compact SRS1 form.
Ingress performs the reverse direction: a bounce returned to a valid SRS address
is verified and relayed to the original sender, while a forged or expired token
is rejected (550).
A copy that an alias or a ~/.forward sends off-site gets a forward return
address instead (fwd+<sealed>=<tag> in the SRS domain), which seals the
original sender and the address that the sender used. A remote report about
such a copy is answered, not relayed: ingress accepts it only from the null
sender, logs it for the operator and drops it, and the bounce stage sends the
original sender Pepsi’s own DSN about the address that they used. So the
forwarding target, the reply of the host behind it and, for mail that arrived
encrypted, the plaintext stay with the user (Forwarding addresses in delivery reports).
11.8. End-to-end encryption and signing¶
pepsi-stage-encrypt encrypts a locally submitted message to each recipient, in OpenPGP (PGP/MIME, RFC 3156) or S/MIME (CMS, RFC 8551 / 5083), signed with the From: author’s own key. The signature goes inside the ciphertext.
ENABLE_PEP (on by default) is a preset of defaults modelled on the pretty
Easy privacy project: every local sender at a served domain gets an OpenPGP key
automatically, a message is signed only when it is encrypted (SIGN =
encrypted-only), and cleartext mail carries the sender’s key in an
Autocrypt: header but no signature. Any option written out explicitly still
wins; ENABLE_PEP = no makes signing opportunistic and turns the automatic
key creation off.
Every encrypted recipient gets their own ciphertext on their own queue row:
no ciphertext names the other recipients, and no content key is shared between
recipients. Neither a ciphertext nor an Autocrypt gossip header can reveal a
Bcc recipient, because gossip names only the addresses in To: and
Cc:. Only the recipients without a key share a row, because they all get
the same no-key treatment.
Recipient keys come from the Key management store, filled by the discovery layer (WKD, VKS, DANE, LDAP, inbound harvesting). The stage itself performs no network I/O: a recipient whose key is not cached pauses the message until a discovery service settles the request, and one with a fresh negative cache entry takes the no-key path at once.
What happens to a recipient with no key is a matter of policy: ON_NO_KEY is
cleartext, secure-link or bounce, and takes its default from
ENCRYPT rather than having one of its own. ENCRYPT = required never
degrades to cleartext, and no cleartext outcome is ever silent: each is logged
and recorded per recipient in state.crypto.out, together with the content
container actually used and whether it was a downgrade. A lookup that failed
in the DNS does not count as “no key” where the outcome would be cleartext.
The message is then deferred; when the domain answers for its mail servers but
not for its key records, it is sent to the portal or bounced instead. Neither
happens if the sender accepted cleartext with [clear-ok] in the subject or
with CLEAR_ON_DNS_FAILURE
(A key lookup that fails in the DNS). Which S/MIME container a correspondent gets is
learnt per peer (Which S/MIME container a correspondent gets).
The stage runs before DKIM signing, so DKIM covers the bytes actually transmitted; see Configuration for why that ordering is not optional.
11.9. Server-side decryption and verification¶
pepsi-stage-decrypt is the inbound half. For a message addressed to a recipient that this host serves, it opens whatever ciphertext the message carries, verifies whatever signature it carries, records the verdict and removes any security indicator the sender forged. The user reads ordinary mail in their usual client, with no plugin involved. The exception is mail encrypted to a key that the user registered from their own mail client (a client-custody key, whose private half Pepsi never holds): that is passed through unopened, for the client to decrypt.
It runs only for recipients that this host serves, and for anything else it does not even try. Decryption rewrites the body, which invalidates the sender’s DKIM signature; that is harmless for a message about to be filed in a mailbox here and is not for one being relayed onward. A message with both kinds of recipient is split, so the forwarded copy is the one that arrived.
The verdict vocabulary is precise. valid means that the signature is sound
and that the key was anchored, by an X.509 chain to a configured CA or as an
OpenPGP key from a ranked discovery source. valid-untrusted means sound with
a key that could not be tied to anything, which is the state of most of the
world’s signed mail and is never reported as valid. invalid,
unverifiable and none complete the set, and where a message carries
several signatures over the same content the worst of them wins. (A signature
made outside the ciphertext is a separate class: it answers for the message
only when nothing signs the plaintext, and then never better than
valid-untrusted — see pepsi-stage-decrypt.) Only valid
sets state.signature_verified, on which
pepsi-stage-check-whitelist gates a whitelist row. A signature
whose protected From: differs from the outer one is demoted rather than
credited to the outer author (Protected headers against the outer ones), and what a verdict
promises the reader is stated in What a security indicator means. The verdict reaches the
reader as the subject tags, and, in Outlook and Thunderbird, through an
extension that shows the X-Pepsi-Crypto field outside the message area
(Mail client extensions).
Both failure paths deliver by default, matching Pepsi’s fail-open posture on inbound SPF, DKIM and DMARC: a message we could not open is delivered still encrypted (the user may hold the key in their own client), and a bad signature is a recorded verdict rather than a delivery failure (mailing lists that rewrite bodies produce them on entirely legitimate mail). Quarantine and bounce routes exist for deployments that want them. The cost of that default is that a bad or unverifiable signature is silent in the subject (Replies, drafts and mailto turning clients into decryption/signing oracles).
The result reaches the user two ways: an X-Pepsi-Crypto header with a
documented grammar, and, by default, [decrypted][verified] prepended to the
Subject in nesting order, which for most users is the only signal they will
ever see. Both depend on the same discipline: every X-Pepsi-* field and
every one of our own subject tags is removed from inbound mail before ours is
added. A private header is forgeable by definition, and that removal is the
entire basis on which it can be believed (Security status spoofed in-band in message body or headers,
Forged security tags in the Subject bypass gateway sanitisation). Our tags go at the very
front, so a client that truncates a long subject still shows them
(Forged security tags in the Subject bypass gateway sanitisation).
The key material that a message carries (S/MIME signer certificates,
application/pgp-keys parts, Autocrypt headers) is read in memory to
check that message’s own signature, because most signed mail carries the
certificate that signed it. When it does not, the message pauses on a discovery
request rather than blocking on a key server, so even the first message from a
new correspondent gets a real verdict.
The decrypt stage stores none of it. All peer-key learning is a stage of its
own, pepsi-stage-autocrypt-learn, placed after the spam
verdict. This placement is what makes the rule of Autocrypt Level 1 §5.3,
“ignore messages the MUA believes to be spam”, implementable at all
(LEARN_FROM_SPAM, default off).
An inbound pipeline without that stage verifies signatures perfectly well and
never learns a correspondent’s key.
What the decrypt stage opens is filed as plaintext, unless the recipient
registered their own mail-client key, in which case
pepsi-stage-reencrypt, placed immediately before local delivery,
seals it to that key once every reading stage is done. A per-recipient policy
(ON_NO_CLIENT_KEY) can refuse mail for a user who has no such key rather than
file it readable. Mail sealed this way is readable only with that client key, so
losing it loses the filed mail (Key management too hard: end-to-end mail clients error-prone or unused).
Before either, pepsi-stage-defang removes from the HTML of a
user who turned it on everything that the HTML would load from the network by
itself. That includes the tracking pixel that reports an open, and an
<img src="... in which a decrypted part could carry its plaintext. Links,
cid: parts, inline images and anything that a signature covers stay as they
are.
11.10. Key management, discovery and publication¶
Neither crypto stage performs network I/O of its own. Both draw on a key store in the shared schema, filled from two directions: by pepsi-keydisc, which runs one service instance per discovery method so that a slow key server delays nobody, and by pepsi-stage-autocrypt-learn from the mail that arrives. A stage that needs a key that it does not have commits what it may keep, pauses the message and enqueues a request; the service that answers releases every message parked on that address in the same round-trip.
Discovery sources, each a separate concurrently-run lookup with its own rank on a trust ladder: the Web Key Directory (advanced and direct forms), DANE
OPENPGPKEY(RFC 7929) andSMIMEA(RFC 8162) (accepted only when the resolver’s AD bit says that the answer was DNSSEC-validated), the VKS protocol of a verifying key server, and LDAP. Below all of them on the ladder sit the two rungs that are not discovery services and have no instance. The higher one is material harvested from inbound mail (S/MIME signer certificates,application/pgp-keysparts andAutocryptheaders). One rung lower still, at the very bottom, is a third party’s key introduced byAutocrypt-Gossip:in a message that arrived encrypted. Both are written inline by pepsi-stage-autocrypt-learn; naming either inSOURCESis refused.Publication of our own users’ keys: the WKD endpoints served by pepsi-httpd,
OPENPGPKEY/SMIMEArecords printed by pepsi-keys and pepsi-setup, and an upload to a verifying key server, made per identity and only when asked for; pepsi-stage-vks-confirm answers that server’s confirmation mail.Custody. Private material is stored wrapped under a key-encryption key that lives outside the database, and is reachable only by the one database role that the crypto stages run as (Unauthenticated private-key encryption lets the storage holder maul keys); those stages therefore hold every server-held key while they parse hostile input (Mail appliances store secrets world-readable or unprotected).
A negative cache means only the first message to or from an unknown correspondent ever waits.
Key management is the chapter on all of this; the option reference is pepsi.conf(5).
11.11. The secure-link fallback¶
Most correspondents publish no key, so ENCRYPT = required would otherwise
leave only “bounce it” or “send it in the clear”. The secure-link portal is
the third answer: pepsi-stage-secure-link stores the message on
the server, encrypted under a freshly generated PIN, and mails the recipient a
link, while the PIN reaches them by another channel (by default it is mailed to
the sender, who relays it). The recipient reads the message in a browser, can
download its attachments, and, where the operator enables it, reply.
The content key is Argon2id over the PIN, a per-message salt and a pepper that
lives only in secrets.d, and only the AEAD ciphertext is stored, with no
PIN verifier beside it (Offline guessing of weak password hashes, key passphrases and derived keys,
Offline guessing of weak password hashes, key passphrases and derived keys). So a stolen database or a
copy of the server’s disk yields no stored message without its PIN. The server
did handle the plaintext when the message was submitted, so an operator, who is
trusted in the Threat model, could have kept it then. The subject travels
inside the ciphertext with the body; the sender, recipient, timestamps and
lifecycle counters are in the clear because delivery, expiry and “did this reach
them?” need them. A lost PIN is a lost message, and the recovery path is that
the sender re-sends.
Messages are rendered as text, never as HTML, so there is no sanitiser to be
bypassed and no remote content to phone home (Hand-written or single-pass HTML sanitisers let active content survive,
Hand-written or single-pass HTML sanitisers let active content survive). Wrong PINs are bounded by a
per-token lockout and a per-source rate limit, and a locked token is refused
before any PIN is evaluated (Login and recovery rate limits defeated by races and logic errors;
parallel guesses can still overshoot the limit slightly,
Login and recovery rate limits defeated by races and logic errors). A reply is injected without
state.local_origin, addressed only to the stored sender, so the public form
cannot become an open relay. See The secure-link fallback portal for the flow, the threat model
and the configuration.
11.12. Outbound DKIM signing¶
The pepsi-stage-dkim-sign stage prepends DKIM signatures
(RFC 6376) under the SIGNING_DOMAIN or the message’s From: domain. By
default it adds both an RSA (2048 bits by default) and an Ed25519
(RFC 8463) signature; [pepsi] DKIM_ALGORITHMS selects either alone. The
private keys are held by the signing daemon pepsi-dkim, under
its own account, which signs for the stage and can rotate the keys on a
schedule (Rotating DKIM keys); no stage can read a key. All the same,
Pepsi’s signatures are only as strong as the account that the pipeline runs
as: any stage may ask the daemon for a signature, or rewrite a message after
ingress checked its From: (A compromised component). Gmail, Microsoft 365
and Yahoo do not verify Ed25519, and Google lists such a signature as fail
in its DMARC aggregate reports; DMARC still passes on the RSA signature. The
signature always covers the whole body: RFC 6376’s l= body-length tag is
never emitted, since strict verifiers reject such a signature outright rather
than tolerating the shorter coverage. Signing is fail-closed: a message from a
domain for which the daemon does not sign is refused rather than sent on
unsigned, and while the daemon cannot be reached the message waits and is
retried. Because header selection is bottom-up, the signature does not disturb
existing signatures.
11.13. Delivery Status Notifications¶
Pepsi implements DSN end to end (RFC 3461 / RFC 3463 / RFC 3464):
Ingress advertises
DSNand validates/storesRET/ENVID(onMAIL FROM) andNOTIFY/ORCPT(onRCPT TO) understate.dsn.The relay stages propagate those parameters to a next hop that also advertises
DSN, and omit them otherwise (RFC 3461 §6: Pepsi does not become the DSN-responsible relay).pepsi-stage-bounce emits the report as an RFC 3464
multipart/reportwith RFC 3463 enhanced status codes:a failure report when
NOTIFYrequestsFAILURE(the default when absent) —NOTIFY=NEVERdrops silently;a success report (
Action: delivered) only when the globalORIGINATE_SUCCESS_DSNis set andNOTIFY=SUCCESSwas requested;a delay report (
Action: delayed) when a relay stage’sDELAY_DSN_AFTERelapses on a still-queued message that asked forNOTIFY=DELAY(sent at most once).
A null-sender message (a bounce) is never itself bounced (RFC 5321 §6.1). When
the return path refuses one, both relay stages keep it until a copy reaches
POSTMASTER, if one is set. For a message that arrived encrypted, a report never returns the body or the subject,
whatever RET asked for, so a bounce cannot hand decrypted content back to
whoever sent the ciphertext (Bounces after gateway decryption reveal decrypted content).
11.14. Recycled addresses¶
A login that an operator deletes and later gives to somebody else would receive
mail meant for its previous owner, and would inherit their state. Pepsi
implements RFC 7293 (Require-Recipient-Valid-Since) against that, and
defines what a recycled login loses (the mechanism is A login that changes owner):
The record.
pepsi.login_assignmentholds, per passwd login, the uid it has and when that assignment was first seen. Local delivery records every login to which it delivers (over LMTP, every accepted recipient that passwd knows; the MDA’s own virtual users are not covered);pepsi-setup runand the hourlypepsi-queue gcre-check every recorded login against passwd. A login seen under another uid, or seen again after a check found it gone, has a new owner. A login seen for the first time is not a reassignment, because what happened before Pepsi was watching is unknown.What a new owner does not inherit. When the change is noticed, the database drops the previous owner’s
pepsi.settingsoverrides, own<login>/...whitelists (and pending confirm-to-send challenges), mailbox quota row, vacation/payment-request reply logs, pending mailing-list confirmation requests, and console enrolment links and sessions, unverifies the login’s mailing-list addresses, detaches them from the previous owner’s list account and stops delivery to their plain memberships, and logs anaccount.reassignedevent (naming any list owner or moderator role that the addresses still hold). Key material is kept: key-store identities, cached peer keys and a TOTP second factor are reported bypepsi-setup runandcheckwith the command that disposes of each, because a deleted private key cannot be restored and may be the only way to open the previous owner’s archived mail.The SMTP extension. Ingress advertises
RRVSand acceptsRRVS=<date-time>[;C|R]onRCPT TO. A recipient whose login has a recorded change of owner after that date is refused with550 5.7.17. A login with one owner as far as Pepsi knows passes for any date, so the answer reveals nothing about addresses that were never recycled; role addresses (postmaster,abuse, …) are exempt, and refusals count towardsMAX_INVALID_RECIPIENTS.The header field. At local delivery (
pepsi-stage-relay-to-maildir) and before running a~/.forward(pepsi-stage-dot-forward), the parameter is checked again with a fresh passwd lookup (without a parameter, aRequire-Recipient-Valid-Since:field naming that recipient is checked), and a failing recipient is bounced with5.7.17instead of delivered. The field is removed from the delivered copy.Relaying. The relay stages pass the parameter on to a next hop that advertises
RRVS. Towards one that does not, the default no-support actionRfails the recipient with5.7.19;Ccontinues with the request written as a header field for that single recipient. Alias targets and forwarded addresses never inherit a request.
11.15. Require TLS¶
Transport policy is normally the recipient’s to publish (MTA-STS, DANE) and the sender’s to honour; RFC 8689 lets the originator of one message demand validated TLS on every hop, or waive the recipient’s policy for it:
The SMTP extension. Ingress advertises
REQUIRETLSonly once a session is TLS-protected, and accepts theMAIL FROMparameter only there (530 5.7.10in cleartext): advertising it promises to carry the requirement onward. A message that arrives with it is taggedstate.tls_required = "required", and every split or clone inherits the tag. A message that arrives without it but with oneTLS-Required: Noheader field is tagged"no".Relaying a required message. The relay stages send it only to a next hop whose name DNSSEC or an MTA-STS policy validates (direct delivery) or that the configuration names (a smarthost), over mandatory
STARTTLSwith the certificate authenticated by DANE or PKIX, and only if the hop advertisesREQUIRETLSafter the upgrade; the parameter is passed on. Every MX host is tried; when none qualifies the message bounces with5.7.30(noREQUIRETLSthere) or5.7.10(no validated TLS), per RFC 8689 §4.2.1. A smarthost entry’sTLS_VERIFY = nois overridden andMODE = plainrefuses the message.Relaying a ``TLS-Required: No`` message. Direct delivery uses opportunistic
STARTTLSto every MX host, anenforceMTA-STS policy and DANE advisory; a handshake that fails is still never retried in cleartext. Smarthost settings are the site’s own policy and stay in force.Bounces. A DSN for a required message returns headers only (
RET=FULLis disregarded), carries the tag, is returned over validated TLS with the parameter where the return path advertises it, and is not discarded for a hop that lacks the extension (§5).
The mechanism and its limits are in Require TLS for one message (RFC 8689); what it changes for the network adversary is in An active network attacker.
11.16. Internationalised and 8-bit content¶
Ingress advertises 8BITMIME (RFC 6152) and SMTPUTF8 (RFC 6531) and records
the BODY= declaration. Because a next hop’s capabilities are unknown until
after connection, the outbound SMTP client decides per hop:
If the hop supports the extension, Pepsi re-advertises
BODY=8BITMIME/SMTPUTF8.Otherwise it downgrades: 8-bit MIME leaf parts are re-encoded to a 7-bit transfer-encoding (RFC 2045: quoted-printable for
text/*, base64 otherwise), on the evidence of the octets rather than of the declared encoding; UTF-8 header fields are rewritten as RFC 2047 encoded-words, in the three places RFC 2047 §5 allows one (an unstructured field, a display-name phrase, the inside of a comment); a non-ASCIIContent-Type/Content-Dispositionparameter, such as an attachment’s filename in any MIME part, takes the RFC 2231 form (filename*=UTF-8''caf%C3%A9.txt), and a non-ASCIIboundary=, which has no encoded form, is replaced by a fresh ASCII one together with the delimiter lines.The downgraded bytes are then re-tested: conversion is best-effort over a message that nobody validated, and RFC 6152 §3 forbids offering 8-bit content to a hop that did not advertise
8BITMIME“under any circumstances”, so a body that is still 8-bit afterwards is a permanent failure (bounce) instead.A non-ASCII address (envelope, or inside a header address) that cannot be represented to a non-
SMTPUTF8hop is a permanent failure (bounce).
Body re-encoding necessarily breaks a body-covering DKIM/ARC signature already
on the message; this is unavoidable (capabilities are late-bound) and rare.
Pepsi’s own DKIM signature is made over a body that pepsi-stage-dkim-sign
has already converted to 7-bit (RFC 6376 section 5.3), so the relay leaves it
intact.
11.17. Outbound relay¶
Two interchangeable relay stages send mail off-site:
pepsi-stage-relay-to-internet — direct-to-MX delivery (RFC 5321 §5): its own
MXlookup with preference ordering and Happy-Eyeballs address selection (cached per address inpepsi.dns_address), implicit MX via address records (§5.1), and Null MX handling (RFC 7505). TLS is authenticated by MTA-STS (RFC 8461) with certificate identity checks (RFC 6125);enforcepolicies require STARTTLS to a listed MX.pepsi-stage-relay-to-smarthost — relay through a configured smarthost, chosen by recipient domain (or a catch-all), with per-MTA transport (
plain/tls/starttls), certificate verification and the full SMTP AUTH (RFC 4954) suite (PLAIN/LOGIN,CRAM-MD5,SCRAM-SHA-1/-SHA-256with optional-PLUSchannel binding,OAUTHBEARER/XOAUTH2, SASLEXTERNALwith a TLS client certificate,NTLM,GSSAPI/Kerberos and the deprecatedDIGEST-MD5) orautonegotiation. See Client authentication and the RFC index.
Both stages apply the loop guard, counting the message’s Received:
headers against their own MAX_HOP_COUNT option. It is a relay-stage
setting in each stage’s [stage-<name>] section, not an ingress one, so a
message is stopped when it is about to be sent on again rather than when it
arrives.
Both stages authenticate the next hop’s TLS with DANE (RFC 7672, DANE =
off|warn|strict, strict by default as RFC 7672 §2.2 requires). They look
up the hop’s TLSA records and match them against the presented chain. DANE
takes precedence over MTA-STS, so a usable TLSA record makes TLS mandatory
and a mismatch defers delivery (The network adversary). When [pepsi-tlsrpt]
SEND_REPORTS is on (it is off by default), both stages also record every
outbound TLS session for TLS Reporting (RFC 8460), and
pepsi-tlsrpt ships the daily aggregate reports.
The two stages share retry semantics: a transient failure
pauses the message with exponential backoff (RETRY_INITIAL/
RETRY_MAX_INTERVAL/RETRY_FACTOR) until MAX_LIFETIME, after which it is
bounced or failed; a permanent failure routes to BOUNCE_STAGE (or marks the
row failed).
11.18. Local delivery¶
For recipients in a local domain, three stages file the mail on the host instead
of relaying it (each forwards the recipients it cannot handle to its
NEXT_STAGE, so they compose):
pepsi-stage-relay-to-maildir — writes the message directly into each local user’s
Maildir/new/. Locality is decided byLOCAL_DOMAINSandTARGETS(passwduidranges); the privileged write is done by the setuid-rootpepsi-helper-maildir-writer, reached through the stage’s ownpepsi-maildirset-group-id bit.pepsi-stage-relay-to-lmtp — hands the message to a local Mail Delivery Agent (typically Dovecot) over LMTP (RFC 2033), which runs each recipient’s Sieve (RFC 5228) filter as it files the mail. It delivers one recipient per transaction and routes each one that the MDA rejects onward by its RFC 3463 status. Needs no privileged helper (the MDA drops privilege).
pepsi-stage-dot-forward — processes each local user’s
~/.forwardfile (the classic sendmail/Postfix mechanism) via the setuid-rootpepsi-helper-dot-forward, which drops to the user before reading it: forwarded addresses restart the pipeline,|pipe//filedirectives run as the user, and a recipient with no~/.forwardpasses through unchanged.
pepsi-stage-aliases complements these by expanding envelope
recipients through a Postfix virtual(5)-style map (full-address or
@domain catch-all, expanded transitively) before local delivery.
pepsi-stage-vacation answers mail that arrives while a recipient is
away, and tags the forwarded copy’s subject so that they can see on their return
which mail was answered for them. It is the one user-filtering action that Pepsi
implements itself rather than leaving it to an MDA’s Sieve. The reply is a new
message that has to be signed and relayed, and the decision not to send one
needs knowledge that only the pipeline has: RFC 3834 says never answer a bounce,
mailing-list mail, anything already marked automatic, or a service address, and
Pepsi adds its own “never answer spam”, “never answer a sender whose SPF or
DMARC definitely failed” (the reply would go to a forged address) and a
per-correspondent rate limit. Leave dates are ordinary per-address
configuration, so the same option is a national holiday at global scope and one
person’s holiday in their own pepsi.settings row.
pepsi-stage-route complements them differently: instead of
delivering, it chooses a next hop per recipient from that recipient’s domain,
splitting the message when its recipients disagree. It is what lets Pepsi front
an existing mail system: the domains behind the gateway go to that system, and
the rest go to the internet. It is also where pepsi-setup proves that no
domain that this host serves can be routed back into its own ingress. See
Microsoft Exchange as a gateway.
11.19. Mailing lists and archives¶
Pepsi’s mailing-list subsystem is a reimplementation of GNU Mailman 3: its
data model, its rule/chain and handler/pipeline architecture, its REST API, its
e-mail command vocabulary and its notice-template names are the GNU Mailman
project’s design, copyright the Free Software Foundation. Files taken from GNU
Mailman, Postorius or HyperKitty remain under the GNU General Public License
(vendor/PEPSI-VENDORING.md lists them). The acceptance test is that an
unmodified mailmanclient, Postorius or HyperKitty drives it. See
Mailing lists, Archives and The GNU Mailman 3 REST API.
Five pipeline stages rather than a daemon of its own: routing, posting, per-member delivery, e-mail commands and bounce processing. A list post is therefore an ordinary queue row that ARC, DKIM, SRS, TLS and the relay stages already handle. How the subsystem is put together is the map.
Nine addresses per list and no alias file. Mailman writes Postfix or Exim maps for every list’s nine addresses and reloads the MTA; Pepsi routes on the recipient inside its own pipeline, so there is nothing to regenerate and nothing to get out of sync.
Moderation with upstream’s vocabulary: eighteen rules, four terminal chains (
accept,hold,reject,discard) and a seventeen-handler pipeline, with the names that the REST API reports. Pepsi adds one rule of its own,dmarc-failure, which holds a post that failed DMARC on a list withhold_dmarc_failureson (the default for lists that Pepsi creates).RFC 2369 ``List-*`` headers and RFC 5064 ``Archived-At`` on every post.
RFC 8058 one-click unsubscribe —
List-Unsubscribe-Postplus a keyedhttps:URI, honoured with a singlePOST. Upstream does not have it. Declared differences from GNU Mailman 3 lists this and the other places where Pepsi goes beyond upstream.Digests in both formats: the RFC 1153 plain digest and the MIME
multipart/digest, per subscriber, with volume/issue numbering, a size threshold and a periodic timer.Bounce processing with upstream’s scoring machine, VERP attribution on every copy (so the seventeen heuristic detectors are the fallback rather than the mechanism), optional probes, warnings and eventual unsubscription.
An archive with full-text search, optional trigram substring search, mbox import and export, and a
purgethat leaves a tombstone. It is a reimplementation of HyperKitty and keeps its Message-ID hash and URL scheme, so existing archive links survive a migration.Migration from Mailman 2.1 and 3: the 2.1
config.pckis read with a restricted pickle machine that cannot instantiate a class (so an untrusted pickle is data, not code), and a Mailman 3 site is read over its own REST API.No JavaScript on any list or archive page (Missing HTTPS and HTTP security-header hardening on webmail) except the two member pages that run a WebAuthn ceremony, which load one static script allowed by hash; and no
403on a public surface: a refusal is the same404that a nonexistent list gets, because a403discloses existence.
11.20. Unsolicited mail¶
Pepsi filters after it has accepted a message, so everything below decides what
happens to mail already in the queue. Three mechanisms cooperate through one
verdict, state.spam, and one table, pepsi.whitelist:
The correspondent whitelist. pepsi-stage-auto-whitelist records the recipients of mail your users send; pepsi-stage-check-whitelist recognises their replies and sets
state.spam = false. Optionally it does so only when DKIM, an aligned DMARC pass (aligned SPF included), a verified signature or a named ARC sealer vouches for them. The operator and users can manage it by hand with pepsi-whitelist.Pay-to-send (pepsi-stage-anti-spam): mail from anybody else is held until its sender pays a small GNU Taler amount. The other side of it is pepsi-stage-auto-pay, which pays the demands that come back to your own users’ outbound mail, within their limits.
Confirm-to-send (pepsi-stage-secretary): mail from anybody else is held until its sender replies once to a challenge. The reply whitelists them, so a correspondent is asked once, ever.
Milters (SpamAssassin, rspamd, ClamAV, …) can set or inform the verdict as well: pepsi-ingress consults them during the SMTP session, and pepsi-stage-milter runs one as a pipeline stage. So can the language filter: pepsi-stage-detect-language records the language, and pepsi-stage-block-language scores it. A pepsi-stage-if stage branches on any of these recorded values, and pepsi-stage-discard is the sink in which a branch can end.
11.20.2. Mailing lists your users subscribe to¶
A list’s postings carry their authors’ From: addresses, so a list is
whitelisted as a whole by its List-Id: (a list-id row). Users need not
type those identifiers:
Offline,
pepsi-whitelist importproposes every list for which the user answered at least one posting in the scanned mailbox, and adds each list that the user confirms (one question per list, all with--yes, none when run unattended). See pepsi-whitelist.Live, with
[stage-check-whitelist] RECORD_LIST_CONFIRMATIONS = yesand[stage-auto-whitelist] LIST_WHITELIST_NAME = {login}/lists, a user who answers a list’s subscription confirmation request gets that list whitelisted in their own<login>/lists. The inbound request is recorded inpepsi.list_confirmationand the outgoing reply must name it inIn-Reply-To:and go to its control address; nothing weaker counts, so an attacker cannot plant a rule by getting an ordinary message answered. Leaving a list sends no signal, so a learnt row expires afterLIST_MAX_IDLE_DAYS(180) without a matching posting. Add{login}/liststo the check stage’sWHITELIST_NAMEso that it is consulted.
Either way the row keeps dkim_required (and, when learnt, dmarc_required
from DMARC_REQUIRED): a List-Id: is text that any sender can copy, so a
posting must still be authenticated to match. Adding --sealer by hand
(pepsi-whitelist) narrows the row further, to mail that the
list’s own ARC sealer handled.
11.20.3. Pay-to-send and confirm-to-send compared¶
Both hold unknown senders’ mail and both send the sender one null-sender auto-reply, in their language. They differ in what the sender has to prove:
Pay-to-send |
Confirm-to-send |
|
|---|---|---|
The sender proves |
they will spend money on this message |
they can read mail at the address they wrote from |
Stops |
bulk mail at any volume, including from working mailboxes |
bulk mail from addresses that cannot receive (most of it) |
Does not stop |
a sender willing to pay |
a spammer with a working mailbox, who can automate the reply |
Deterrent |
economic: every message costs |
legal only: |
Needs |
a GNU Taler merchant backend, and a wallet on the sender’s side |
nothing but a reply |
Stated plainly: confirm-to-send is the weaker filter. It is still worth running,
because it costs a legitimate correspondent one reply and costs a mail server
nothing to operate. And the two combine. With both enabled, the secretary runs
first and its UNCHALLENGEABLE_STAGE points at the paywall, so mail that cannot
be asked to reply (a list posting, an unauthenticated or forged sender, a sender
over the daily challenge cap) is asked to pay instead, while a confirmed sender
reaches the paywall with state.spam = false and passes it untouched.
Confirm-to-send is careful about backscatter, since its challenge goes to the
envelope sender and spam’s envelope sender is usually forged: it challenges only a
sender that passed SPF or DMARC (REQUIRE_AUTHENTICATED) and whose From: is
that same address, at most MAX_CHALLENGES_PER_SENDER times a day across all
users; it never answers what RFC 3834 says not to; and its challenge quotes nothing
of the held message but a short hint of its subject (SUBJECT_HINT_LENGTH,
eight characters by default), and not even that for a message that arrived
encrypted, so the challenge cannot reveal a decryption outcome
(CBC padding and format oracles against unauthenticated S/MIME).
11.20.4. Spam filters and the secretary¶
By default spam-scoring milters (SpamAssassin through spamass-milter, rspamd,
MIMEDefang, …) are consulted by pepsi-ingress during the SMTP session, so
they run ahead of the whitelist check, and pepsi-setup’s
wizard generates that order:
pepsi-ingress (MILTERS = spamassassin) → … → check-whitelist → secretary → local
UNCHALLENGEABLE_STAGE = local
A message that the filter rejects is refused in the session and never reaches
the secretary, so obvious spam draws no challenge to its forged sender, and no
backscatter arises. What does reach the secretary has passed the filter,
which is why the wizard sets UNCHALLENGEABLE_STAGE to the secretary’s own
NEXT_STAGE: running the filter again would scan twice.
The alternative is to run the filter behind the secretary, as a pepsi-stage-milter stage, so it scans only mail that could not be challenged:
… → check-whitelist → secretary → local
└─ UNCHALLENGEABLE_STAGE → milter-spamassassin → local
The configuration edit, for a filter at [pepsi-ingress-milter-spamassassin]:
Take the filter out of the session: remove
spamassassinfrom[pepsi-ingress] MILTERS, and the section with it.Write
[stage-milter-spamassassin]withPROGRAM = pepsi-stage-milter, the sameSOCKETandALLOW_ACTIONS, andNEXT_STAGEset to the secretary’sNEXT_STAGE(herelocal).In
[stage-secretary], setUNCHALLENGEABLE_STAGE = milter-spamassassin.
Then run pepsi-setup check (or run) to validate the graph.
The trade-off cuts both ways:
For: the filter’s CPU is spent only on the mail that needs it. Whitelisted mail and mail from confirmed senders, which is the bulk of a personal mailbox, is never scanned.
Against: challenges go out unscanned. Spam that the filter would have rejected draws a challenge first, which is more backscatter to forged senders (bounded by
REQUIRE_AUTHENTICATEDand the daily cap, but not zero). And spam whose sender does confirm is delivered without the filter ever seeing it; that is the one kind of spam that confirm-to-send cannot stop.
Whichever layout you choose, virus and policy milters stay in the session (ahead of the whitelist check): a confirmed or whitelisted correspondent can still send malware, from a compromised account or unknowingly, and a policy filter’s verdict applies to everybody.
11.21. Operational features¶
Single-table pipeline with crash recovery: orphaned
runningrows are reset on dispatcher start-up; in-flight children are reset on shutdown.Elastic, pipelined worker pools and watchdog: each stage runs persistent worker processes started on demand up to its
PARALLELISMand reaped afterWORKER_IDLE_TIMEOUT, recycled afterMAX_MESSAGES; each worker is fed up toQUEUE_LIMITmessages at once (in-flight capacityQUEUE_LIMIT × PARALLELISM, at no extra connection cost), and a worker exceedingMAX_RUNTIMEon the message it is running is killed and replaced.Queue tooling: pepsi-queue lists, deletes, re-stages and bulk-unsticks messages.
Health monitoring: pepsi-status prints a read-only summary of the queue, stuck messages, cumulative delivery/failure counters and recent outbound TLS outcomes (also as
--json), suitable for running over SSH.Provisioning and DNS verification: pepsi-setup installs the schema, has the DKIM keys made and generates the others, and prints/validates DNS (DKIM, SPF, MTA-STS).
HTTP server: pepsi-httpd serves, over one or more TLS (SNI-selected) or plaintext listeners, the MTA-STS policy file (
/.well-known/mta-sts.txt), a Prometheus/metricspage (on anADMIN = yeslistener only, because it describes the pipeline), the Web Key Directory endpoints that publish this deployment’s own identities (/.well-known/openpgpkey/…, direct and advanced forms), the The secure-link fallback portal portal, and the mail autoconfiguration document below.Mail client autoconfiguration (draft-ietf-mailmaint-autoconfig): a client given nothing but
fred@example.orgfetcheshttps://autoconfig.example.org/mail/config-v1.1.xmland configures itself — submission host, port, transport security and authentication, and the same for the mailbox server. The submission half is derived from the running submission listener, so it cannot drift from the server it describes; the mailbox half names whatever MDA the deployment pairs with. A server without TLS is published only once the operator acknowledges it (ALLOW_PLAIN_AUTOCONFIG); otherwise the configuration is refused, naming the server. The document is served publicly and without authentication, as the draft requires, because a client must read it before it can know how to authenticate.Administrative API and console: on a listener explicitly flagged
ADMIN = yes, the same server exposes the The administrative API under/api/v1(queue, health, configuration, key store, audit and mail logs) and the browser The administration console under/ui, which is a client of exactly that API. It never does so on a listener that would carry them in cleartext off the host. Everywhere else those paths answer the same404that an unknown path gets. Authentication is by UNIX socket peer credentials, a session cookie or a bearer token, and every principal carries scopes; all of it is audited. End users may sign in by e-mail: a one-time link mailed to an address that the server serves opens a session confined to that address (Signing in by e-mail).Metrics:
pepsi-dispatchrecords per-stage throughput, kills/timeouts, crashes and processing time and the global stage/message totals (flushed to the database in one transaction roughly once a minute), exported bypepsi-httpdalongside live active/paused gauges read from the queue.Structured logging via
tracingat configurable levels.
For standards that Pepsi does not implement (BINARYMIME among them), see
Applicable RFCs not yet implemented.
11.22. Feature stability¶
Every feature described in this chapter is tracked in a single
Feature stability table, which records for each feature its governing
RFC, whether it has an automated test (and of what kind: U unit, I
integration, I/U both), whether it has been verified by hand, and how widely
it is deployed and exercised according to anonymous, opt-in usage telemetry
(pepsi-telemetry).
That table is generated: contrib/update-feature-stability.sh merges the
hand-maintained registry contrib/feature-registry.tsv (name, RFC, test and
manual-test columns) with a pepsi-telemetry GET /telemetry/report (the
deployment and usage counts). See Extending the Pipeline for how to keep it current
when you add a feature, write a test, or refresh the telemetry counts.