95.1.25. pepsi-stage-check-whitelist¶
mark trusted senders’ mail as non-spam
- Manual section:
1
95.1.25.1.1. Name¶
pepsi-stage-check-whitelist - the sender-whitelist stage of the Pepsi pipeline.
95.1.25.1.2. Synopsis¶
pepsi-stage-check-whitelist [GLOBAL-OPTIONS] worker
95.1.25.1.3. Description¶
This stage defaults to FUSION = yes: when stage fusion is enabled ([pepsi]
ALLOW_FUSION, the default) and this stage is folded into the unified pepsi
binary, a predecessor may run it in its own worker process instead of
dispatching it separately. Because this stage needs the message’s header block
(for List-Id:), only a predecessor that loaded the headers or the whole
message can fuse it; a predecessor that loaded envelope metadata alone dispatches
it normally. See pepsi-dispatch(1) and pepsi.conf(5).
pepsi-stage-check-whitelist is a stage program run by pepsi-dispatch(1)
as a persistent worker that reads message ids on standard input. For each id it
loads the pepsi.workqueue row (refusing to act unless its status is
running), reads its
[stage-<stage>] section, and consults a named whitelist of trusted
senders. If the message matches the whitelist, the stage
records state.spam = false so a later pepsi-stage-anti-spam(1) lets it
through without payment. Either way the message then advances to
NEXT_STAGE: this stage never drops, bounces or pauses a message.
The whitelist lives in the pepsi.whitelist table, populated out of band by
the operator (pepsi-whitelist(1)), by the outbound
pepsi-stage-auto-whitelist(1), or by a user importing their own mailbox
(pepsi-whitelist import). Each row pairs a whitelist_name (a group label)
with a whitelist_regex, a match_field that names what the expression is
matched against, and the conditions that gate the match. The stage’s
WHITELIST_NAME option selects the group or groups. PostgreSQL decides in one
query whether any of their rows matches, subject to the row’s conditions. The
whitelist_regex is an extended POSIX regular expression, matched
case-insensitively (it is evaluated by PostgreSQL’s ~* operator, so the
dialect is exactly POSIX ERE). The expression is unanchored; anchor it with
^/$ to match the whole subject. A from row’s subject is the
From: addr-spec in canonical form (Unicode NFC, local part case-folded,
domain as its A-label). pepsi-whitelist and
pepsi-stage-auto-whitelist(1) build their patterns from the same form, so
a row matches every spelling of its address.
A row whose expression is one exact address or list identifier, or any mailbox at one domain (the shapes pepsi-whitelist and pepsi-stage-auto-whitelist(1) write, when the text is ASCII), is found through an index rather than by evaluating the expression. The result is the same; only the other rows are evaluated one by one, so the cost of the check does not grow with the number of learnt correspondents.
A row whose expression does not compile matches nothing; it does not fail
the check. Every tool that writes the table validates an expression first, but
a row that got in anyway (a hand-written INSERT) would otherwise make
PostgreSQL refuse the whole query. Since every applicable name is checked in
that one query, one user’s bad row would then hold up the mail of everybody who
shares a whitelist name with that user. The stage logs a warning naming the
whitelists concerned; pepsi-whitelist list finds the row.
The query runs under a five-second statement_timeout. An expression
that PostgreSQL has to backtrack through, such as the back-reference
^(.*)\1\1$, can take seconds against a long From: and would hold a
database connection the whole time. When the limit fires, the message is treated
as not whitelisted (the same outcome as no match) and the stage logs a
warning naming the whitelists concerned.
95.1.25.1.3.1. What is matched¶
match_field selects the subject of the match:
from(the default): the addr-spec parsed out of the message’sFrom:header, not the raw header value. A whitelisted address written into the display-name position therefore cannot spoof a match, and RFC 5322 comments and folding whitespace are removed. A message with noFrom:header matches nothing, and so does one whoseFrom:names more than one mailbox. (The DKIM/DMARC verdict thatdkim_requiredconsults was computed for the first address, so matching any other would decide two questions about two different senders.)list-id: the RFC 2919 list identifier of the message’sList-Id:header, reduced to the bracketed token, trimmed and lower-cased (List-Id: Rust users <users.rust-lang.org>matches againstusers.rust-lang.org). A message with noList-Id:matches nothing.
A list-id row exists because a mailing list’s postings carry arbitrary
From: addresses. No pattern over From: can say “anything that came through
this list”, so a subscriber would otherwise have to whitelist every individual
poster, or pay the gate for every message that the list relays. Matching
List-Id: says it once. Because that header is unauthenticated text that
anyone can copy, pair such a row with sealer_domain (below).
95.1.25.1.3.2. Several groups, and per-user whitelists¶
WHITELIST_NAME is a comma-separated list, and an entry may be a template
containing {localpart} or {login}, which is expanded per envelope
recipient:
WHITELIST_NAME = correspondents, {localpart}/correspondents
That consults the operator’s shared list and each recipient’s own. The
<login>/... namespace is the one that pepsi-whitelist(1) lets an
ordinary user manage and seed from their mailbox, so a template is what makes a
per-user whitelist take effect. Without it, such a name is only reachable
through a per-address pepsi.settings override (pepsi-settings(1)).
{localpart} expands to the recipient’s local part with any sub-address stripped
(alice+lists@ → alice). {login} first resolves the recipient to a passwd
login, which is what lines up with the namespace when the address is not itself a
login (first.last@ → alice); it therefore needs the recipient to resolve
locally, so place the stage after pepsi-stage-aliases(1) when using it. A
recipient for which a template cannot be expanded is skipped, as is any
expansion outside the whitelist-name grammar (letters, digits, _, . and - per
/-separated segment, and no segment made only of dots).
Only a recipient at a served domain names a whitelist. Both placeholders are
expanded solely for recipients whose domain is in LOCAL_DOMAINS (defaulting to
[pepsi-ingress] ACCEPTED_DOMAINS); a recipient elsewhere is skipped. This
matters because a match in any consulted list clears the whole message: the
local part of a foreign address is chosen by whoever wrote the envelope, so without
the test a sender already in one user’s whitelist could add
RCPT TO:<thatuser@anything.invalid> to a message for somebody else and have it
cleared.
All applicable names are checked in one query, and a match in any of them marks
the message non-spam: the verdict is per message, and a queue row is one message.
Note that this pools trust between the served recipients of one message: a message
addressed to two of your users is non-spam if it matches either one’s list. The
locality options LOCAL_DOMAINS, TARGETS and RECIPIENT_DELIMITER are
read only to expand these placeholders.
A row may further restrict when it counts as a match:
dkim_required: the row matches only if the message’sFrom:domain is authenticated, that is, if ingress recordedstate.auth.dkim = pass, or pepsi-stage-arc(1) recordedstate.auth.arc = passand DMARC also passed (state.auth.dmarc = pass). An ARC pass on its own shows only the integrity of the chain: a sender who controls any single domain can mint a valid one-hop ARC chain over a forgedFrom:. It is therefore accepted only when DMARC confirms theFrom:alignment.dmarc_required: the row matches only if the message passed DMARC at ingress on either aligned identifier, an aligned DKIM signature or an aligned SPF pass (state.auth.dmarc = pass), or if a trusted ARC sealer let it through a DMARC failure (state.auth.arc_override). This is the level for a correspondent whose domain authenticates only through SPF, whichdkim_requireddoes not accept; an unaligned SPF pass does not count. Every message that satisfiesdkim_requiredsatisfies it too.signature_required: the row matches only ifstate.signature_verifiedistrueand the signature did not come out of a ciphertext without integrity protection (an S/MIMEEnvelopedData, AES-CBC). pepsi-stage-decrypt(1) setsstate.signature_verifiedfor a message that carries a signature that it verified against a trusted key. Whether such a ciphertext, modified by its sender, still yields a verifying signature depends on its padding, and the verdict decides whether pay-to-send or confirm-to-send answers the sender; counting it would make that answer a padding oracle.sealer_domain: the row matches only if the message arrived carrying an ARC chain that validated (state.auth.arc = pass) and the named ADMD is one of that chain’s sealers (state.auth.arc_sealers, recorded by pepsi-stage-arc(1)).
When a row sets several conditions, all of them must hold.
sealer_domain is what makes a rule about an intermediary safe to write, and
neither half suffices alone. List-Id: proves nothing: it is text that a
sender copies out of a genuine posting. A valid ARC chain proves nothing either.
RFC 8617 §8.4 is explicit that it conveys no trust, only that the ADMDs
named in it really did handle these bytes, and that is exactly why anyone can
mint one over a forged message. Naming the sealer is the local policy that the protocol
deliberately leaves to the receiver: the operator (or the user) states which
intermediary they accept, and the chain then proves that intermediary handled
the message. It applies to from rows too, where it narrows a sender pattern
to mail relayed through a known forwarder.
Note that the domains in state.auth.arc_sealers are recorded whether or not
the chain validated, because they are a record of what was claimed; the stage
discards all of them unless state.auth.arc is pass, so a forged chain can
never satisfy a sealer_domain row. For a validated chain the list holds only
the sealers whose own ARC-Message-Signature still verifies over the message, so
a genuine set copied off another message onto forged content, and sealed by the
forger, does not name the copied sealer either.
A row that pepsi-stage-auto-whitelist(1) learnt from a subscription
confirmation reply also carries last_seen and max_idle. It matches only
until last_seen + max_idle. Whenever it matches an authenticated posting,
the stage moves last_seen forward, at most once a day. (Authenticated means
the same conditions as for the match itself, so a forged posting that fails
dkim_required cannot keep the row alive.) Leaving a list sends no signal;
this expiry is what retires the row of a list that nobody posts to any more. Rows written by hand or at an import prompt have
no last_seen and never expire.
If any row matches (subject to its conditions), the stage merges
spam: false into the message state and advances. Otherwise it advances the
message unchanged.
95.1.25.1.3.3. Mailing-list confirmation requests¶
With RECORD_LIST_CONFIRMATIONS (default no) the stage also records every
message that looks like a mailing list’s subscription confirmation request
addressed to a local recipient, in pepsi.list_confirmation, so that
pepsi-stage-auto-whitelist(1) (its LIST_WHITELIST_NAME) can whitelist
the list when that recipient answers it. The stage recognises such a request
from the header block alone, and deliberately narrowly; all of these must hold:
exactly one
Message-ID:identifier;a
Subject:containingconfirmin any case (Mailman 2’sconfirm <cookie>, Mailman 3’s “Your confirmation is needed to join …”, and Pepsi’s own list notices all do);a
From:,Reply-To:orSender:address whose local part, without its+detail, ends in-request,-confirm,-joinor-subscribe(a list control address);a list identifier: the
List-Id:if the message has a valid one, else the<name>.<domain>the control address implies (users-confirm+tok@lists.example.orggivesusers.lists.example.org, the default identifier of Mailman and of Pepsi’s own lists; Mailman’s requests carry noList-Id:).
One row is written per local envelope recipient (a recipient outside
LOCAL_DOMAINS is skipped), valid for LIST_CONFIRMATION_DAYS (default 7)
and then deleted by pepsi-queue gc. The first record of a request wins, so a
re-delivered copy can neither change its list nor extend its window. Recording is
fail-open, and the message is unaffected either way. Anybody can send a message
that passes this test; recording it is not trusting it. Nothing is whitelisted
until the recipient’s own outgoing reply names the request in In-Reply-To:
and goes to its control address, and the learnt row still requires an
authenticated posting.
95.1.25.1.4. Configuration¶
Options live in the stage’s own [stage-<name>] section
(PROGRAM = pepsi-stage-check-whitelist): the required WHITELIST_NAME
list of groups to consult; a NEXT_STAGE, which is also required (the stage
always hands the message on, so pepsi-setup rejects a section without one);
the locality options LOCAL_DOMAINS / TARGETS / RECIPIENT_DELIMITER;
and the mailing-list options RECORD_LIST_CONFIRMATIONS (default no) and
LIST_CONFIRMATION_DAYS (default 7, 1 to 90). The locality options serve to
expand a placeholder in a WHITELIST_NAME entry, and LOCAL_DOMAINS also
decides for which recipients a confirmation request is recorded.
LIST_CONFIRMATION_DAYS matters only when RECORD_LIST_CONFIRMATIONS is
on. The stage checks all of these options when it starts, whether it uses them
or not. All these options are documented in
pepsi.conf(5); the whitelist rows themselves are managed with
pepsi-whitelist(1).
95.1.25.1.5. State¶
Inputs: state.auth.dkim / state.auth.arc / state.auth.dmarc (for
dkim_required rows), state.auth.dmarc / state.auth.arc_override /
state.auth.dkim (for dmarc_required rows), state.signature_verified (for
signature_required rows), which pepsi-stage-decrypt(1) sets only for its
valid signature verdict (so a sound signature by a key that could not be
anchored, valid-untrusted, does not open the gate), and
state.auth.arc / state.auth.arc_sealers (for sealer_domain rows),
both written by pepsi-stage-arc(1). The From: header
comes from the from_header column, not from state; List-Id: is read
from the message’s header block, which is why this stage loads it (the body is
never loaded).
Outputs: on a match the stage merges spam: false into state (the key
that pepsi-stage-anti-spam(1) reads to skip the payment gate) and advances; on no
match state is left untouched. Side effects: a matched learnt list row’s
last_seen is refreshed, and with RECORD_LIST_CONFIRMATIONS a confirmation
request is recorded in pepsi.list_confirmation. The state layout is described in
pepsi.state(7).
Transitions: always advances to NEXT_STAGE, whether or not the sender
matched the whitelist. A match is committed with the reroute helper, which
merges spam: false and moves the stage in the same single UPDATE; a
non-match is a plain advance. There is no branch; the stage never pauses, fails,
bounces or finishes. The actual payment/drop decision is left to a later stage
(pepsi-stage-anti-spam(1)).
95.1.25.1.6. Commands¶
- worker
Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.
95.1.25.1.7. Global Options¶
- -c FILE, –config FILE
Read the configuration from FILE instead of searching the default locations.
- -L LOGLEVEL, –log LOGLEVEL
Set the logging verbosity (default
[pepsi] LOG, elseinfo).- -v, –verbose
Show log messages from all sources.
- -h, –help; -V, –version
Print a usage summary / the version and exit.
95.1.25.1.8. Exit Status¶
The outcome of each message is reported to pepsi-dispatch(1) on the
worker’s status line, not as the exit status. A message that cannot be checked
(a database error, a {login} expansion that could not ask the passwd
database) is paused and retried, as for every stage; a malformed or too
expensive whitelist_regex is not an error at all: that row matches nothing
(see above).
- 0
The worker ran until its standard input was closed.
- 1
A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.
- 78
The worker refused to start: a
[stage-*]section it serves does not parse (e.g. a missing WHITELIST_NAME, or LIST_CONFIRMATION_DAYS outside 1 to 90). The dispatcher requeues what it handed over and retries the stage later.
95.1.25.1.9. Examples¶
Check message 42 against the whitelist (it must be running; its outcome is
printed as the status line):
echo 42 | pepsi-stage-check-whitelist -c /etc/pepsi/pepsi.conf worker
A pipeline section that whitelists trusted senders before the payment gate:
[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = anti-spam
WHITELIST_NAME = trusted-senders
Consulting the shared list and each recipient’s own, so that a whitelist that a
user seeded with pepsi-whitelist import is honoured:
[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = anti-spam
WHITELIST_NAME = trusted-senders, {localpart}/correspondents
Whitelisting a partner domain, but only when its mail is DKIM/ARC-signed:
INSERT INTO pepsi.whitelist (whitelist_name, whitelist_regex, dkim_required)
VALUES ('trusted-senders', '@partner\.example\.org$', TRUE);
dkim_required defaults to TRUE when an INSERT leaves it out, as it
does for pepsi-whitelist add; a row trusting unsigned mail must say
dkim_required = FALSE explicitly. A partner whose domain does not sign but
publishes DMARC and SPF is better held to dmarc_required:
pepsi-whitelist add trusted-senders --no-dkim-required --dmarc-required '@partner\.example\.org$'
Letting a mailing list past the payment gate, whoever posted to it, but only when the list’s own ADMD sealed the message’s ARC chain:
pepsi-whitelist add-list alice/lists users.rust-lang.org \
--sealer mail.rust-lang.org
Learning, live, the lists that users subscribe to by answering a confirmation
request. This stage records the request,
pepsi-stage-auto-whitelist(1) learns the list on the outbound path into
each user’s <login>/lists, and this stage consults it:
[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = anti-spam
WHITELIST_NAME = trusted-senders, {login}/correspondents, {login}/lists
RECORD_LIST_CONFIRMATIONS = yes
[stage-auto-whitelist]
PROGRAM = pepsi-stage-auto-whitelist
NEXT_STAGE = dkim-sign
WHITELIST_NAME = trusted-senders
LIST_WHITELIST_NAME = {login}/lists
Deliberately trusting the header alone, for a list that does not ARC-seal. An
attacker only has to copy a List-Id: to satisfy such a row, so scope it to
one user’s own whitelist rather than a shared one:
pepsi-whitelist add-list alice/lists announce.example.org --no-dkim-required
95.1.25.1.10. See Also¶
pepsi-config(1), pepsi-whitelist(1), pepsi-stage-anti-spam(1), pepsi-stage-arc(1), pepsi-settings(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)
95.1.25.1.11. Bugs¶
Report bugs to the Pepsi issue tracker.