95.1.25. pepsi-stage-check-whitelist

mark trusted senders’ mail as non-spam

Manual section:

1

95.1.25.1.1. Name

pepsi-stage-check-whitelist - the sender-whitelist stage of the Pepsi pipeline.

95.1.25.1.2. Synopsis

pepsi-stage-check-whitelist [GLOBAL-OPTIONS] worker

95.1.25.1.3. Description

This stage defaults to FUSION = yes: when stage fusion is enabled ([pepsi] ALLOW_FUSION, the default) and this stage is folded into the unified pepsi binary, a predecessor may run it in its own worker process instead of dispatching it separately. Because this stage needs the message’s header block (for List-Id:), only a predecessor that loaded the headers or the whole message can fuse it; a predecessor that loaded envelope metadata alone dispatches it normally. See pepsi-dispatch(1) and pepsi.conf(5).

pepsi-stage-check-whitelist is a stage program run by pepsi-dispatch(1) as a persistent worker that reads message ids on standard input. For each id it loads the pepsi.workqueue row (refusing to act unless its status is running), reads its [stage-<stage>] section, and consults a named whitelist of trusted senders. If the message matches the whitelist, the stage records state.spam = false so a later pepsi-stage-anti-spam(1) lets it through without payment. Either way the message then advances to NEXT_STAGE: this stage never drops, bounces or pauses a message.

The whitelist lives in the pepsi.whitelist table, populated out of band by the operator (pepsi-whitelist(1)), by the outbound pepsi-stage-auto-whitelist(1), or by a user importing their own mailbox (pepsi-whitelist import). Each row pairs a whitelist_name (a group label) with a whitelist_regex, a match_field that names what the expression is matched against, and the conditions that gate the match. The stage’s WHITELIST_NAME option selects the group or groups. PostgreSQL decides in one query whether any of their rows matches, subject to the row’s conditions. The whitelist_regex is an extended POSIX regular expression, matched case-insensitively (it is evaluated by PostgreSQL’s ~* operator, so the dialect is exactly POSIX ERE). The expression is unanchored; anchor it with ^/$ to match the whole subject. A from row’s subject is the From: addr-spec in canonical form (Unicode NFC, local part case-folded, domain as its A-label). pepsi-whitelist and pepsi-stage-auto-whitelist(1) build their patterns from the same form, so a row matches every spelling of its address.

A row whose expression is one exact address or list identifier, or any mailbox at one domain (the shapes pepsi-whitelist and pepsi-stage-auto-whitelist(1) write, when the text is ASCII), is found through an index rather than by evaluating the expression. The result is the same; only the other rows are evaluated one by one, so the cost of the check does not grow with the number of learnt correspondents.

A row whose expression does not compile matches nothing; it does not fail the check. Every tool that writes the table validates an expression first, but a row that got in anyway (a hand-written INSERT) would otherwise make PostgreSQL refuse the whole query. Since every applicable name is checked in that one query, one user’s bad row would then hold up the mail of everybody who shares a whitelist name with that user. The stage logs a warning naming the whitelists concerned; pepsi-whitelist list finds the row.

The query runs under a five-second statement_timeout. An expression that PostgreSQL has to backtrack through, such as the back-reference ^(.*)\1\1$, can take seconds against a long From: and would hold a database connection the whole time. When the limit fires, the message is treated as not whitelisted (the same outcome as no match) and the stage logs a warning naming the whitelists concerned.

95.1.25.1.3.1. What is matched

match_field selects the subject of the match:

  • from (the default): the addr-spec parsed out of the message’s From: header, not the raw header value. A whitelisted address written into the display-name position therefore cannot spoof a match, and RFC 5322 comments and folding whitespace are removed. A message with no From: header matches nothing, and so does one whose From: names more than one mailbox. (The DKIM/DMARC verdict that dkim_required consults was computed for the first address, so matching any other would decide two questions about two different senders.)

  • list-id: the RFC 2919 list identifier of the message’s List-Id: header, reduced to the bracketed token, trimmed and lower-cased (List-Id: Rust users <users.rust-lang.org> matches against users.rust-lang.org). A message with no List-Id: matches nothing.

A list-id row exists because a mailing list’s postings carry arbitrary From: addresses. No pattern over From: can say “anything that came through this list”, so a subscriber would otherwise have to whitelist every individual poster, or pay the gate for every message that the list relays. Matching List-Id: says it once. Because that header is unauthenticated text that anyone can copy, pair such a row with sealer_domain (below).

95.1.25.1.3.2. Several groups, and per-user whitelists

WHITELIST_NAME is a comma-separated list, and an entry may be a template containing {localpart} or {login}, which is expanded per envelope recipient:

WHITELIST_NAME = correspondents, {localpart}/correspondents

That consults the operator’s shared list and each recipient’s own. The <login>/... namespace is the one that pepsi-whitelist(1) lets an ordinary user manage and seed from their mailbox, so a template is what makes a per-user whitelist take effect. Without it, such a name is only reachable through a per-address pepsi.settings override (pepsi-settings(1)).

{localpart} expands to the recipient’s local part with any sub-address stripped (alice+lists@ → alice). {login} first resolves the recipient to a passwd login, which is what lines up with the namespace when the address is not itself a login (first.last@ → alice); it therefore needs the recipient to resolve locally, so place the stage after pepsi-stage-aliases(1) when using it. A recipient for which a template cannot be expanded is skipped, as is any expansion outside the whitelist-name grammar (letters, digits, _, . and - per /-separated segment, and no segment made only of dots).

Only a recipient at a served domain names a whitelist. Both placeholders are expanded solely for recipients whose domain is in LOCAL_DOMAINS (defaulting to [pepsi-ingress] ACCEPTED_DOMAINS); a recipient elsewhere is skipped. This matters because a match in any consulted list clears the whole message: the local part of a foreign address is chosen by whoever wrote the envelope, so without the test a sender already in one user’s whitelist could add RCPT TO:<thatuser@anything.invalid> to a message for somebody else and have it cleared.

All applicable names are checked in one query, and a match in any of them marks the message non-spam: the verdict is per message, and a queue row is one message. Note that this pools trust between the served recipients of one message: a message addressed to two of your users is non-spam if it matches either one’s list. The locality options LOCAL_DOMAINS, TARGETS and RECIPIENT_DELIMITER are read only to expand these placeholders.

A row may further restrict when it counts as a match:

  • dkim_required: the row matches only if the message’s From: domain is authenticated, that is, if ingress recorded state.auth.dkim = pass, or pepsi-stage-arc(1) recorded state.auth.arc = pass and DMARC also passed (state.auth.dmarc = pass). An ARC pass on its own shows only the integrity of the chain: a sender who controls any single domain can mint a valid one-hop ARC chain over a forged From:. It is therefore accepted only when DMARC confirms the From: alignment.

  • dmarc_required: the row matches only if the message passed DMARC at ingress on either aligned identifier, an aligned DKIM signature or an aligned SPF pass (state.auth.dmarc = pass), or if a trusted ARC sealer let it through a DMARC failure (state.auth.arc_override). This is the level for a correspondent whose domain authenticates only through SPF, which dkim_required does not accept; an unaligned SPF pass does not count. Every message that satisfies dkim_required satisfies it too.

  • signature_required: the row matches only if state.signature_verified is true and the signature did not come out of a ciphertext without integrity protection (an S/MIME EnvelopedData, AES-CBC). pepsi-stage-decrypt(1) sets state.signature_verified for a message that carries a signature that it verified against a trusted key. Whether such a ciphertext, modified by its sender, still yields a verifying signature depends on its padding, and the verdict decides whether pay-to-send or confirm-to-send answers the sender; counting it would make that answer a padding oracle.

  • sealer_domain: the row matches only if the message arrived carrying an ARC chain that validated (state.auth.arc = pass) and the named ADMD is one of that chain’s sealers (state.auth.arc_sealers, recorded by pepsi-stage-arc(1)).

When a row sets several conditions, all of them must hold.

sealer_domain is what makes a rule about an intermediary safe to write, and neither half suffices alone. List-Id: proves nothing: it is text that a sender copies out of a genuine posting. A valid ARC chain proves nothing either. RFC 8617 §8.4 is explicit that it conveys no trust, only that the ADMDs named in it really did handle these bytes, and that is exactly why anyone can mint one over a forged message. Naming the sealer is the local policy that the protocol deliberately leaves to the receiver: the operator (or the user) states which intermediary they accept, and the chain then proves that intermediary handled the message. It applies to from rows too, where it narrows a sender pattern to mail relayed through a known forwarder.

Note that the domains in state.auth.arc_sealers are recorded whether or not the chain validated, because they are a record of what was claimed; the stage discards all of them unless state.auth.arc is pass, so a forged chain can never satisfy a sealer_domain row. For a validated chain the list holds only the sealers whose own ARC-Message-Signature still verifies over the message, so a genuine set copied off another message onto forged content, and sealed by the forger, does not name the copied sealer either.

A row that pepsi-stage-auto-whitelist(1) learnt from a subscription confirmation reply also carries last_seen and max_idle. It matches only until last_seen + max_idle. Whenever it matches an authenticated posting, the stage moves last_seen forward, at most once a day. (Authenticated means the same conditions as for the match itself, so a forged posting that fails dkim_required cannot keep the row alive.) Leaving a list sends no signal; this expiry is what retires the row of a list that nobody posts to any more. Rows written by hand or at an import prompt have no last_seen and never expire.

If any row matches (subject to its conditions), the stage merges spam: false into the message state and advances. Otherwise it advances the message unchanged.

95.1.25.1.3.3. Mailing-list confirmation requests

With RECORD_LIST_CONFIRMATIONS (default no) the stage also records every message that looks like a mailing list’s subscription confirmation request addressed to a local recipient, in pepsi.list_confirmation, so that pepsi-stage-auto-whitelist(1) (its LIST_WHITELIST_NAME) can whitelist the list when that recipient answers it. The stage recognises such a request from the header block alone, and deliberately narrowly; all of these must hold:

  • exactly one Message-ID: identifier;

  • a Subject: containing confirm in any case (Mailman 2’s confirm <cookie>, Mailman 3’s “Your confirmation is needed to join …”, and Pepsi’s own list notices all do);

  • a From:, Reply-To: or Sender: address whose local part, without its +detail, ends in -request, -confirm, -join or -subscribe (a list control address);

  • a list identifier: the List-Id: if the message has a valid one, else the <name>.<domain> the control address implies (users-confirm+tok@lists.example.org gives users.lists.example.org, the default identifier of Mailman and of Pepsi’s own lists; Mailman’s requests carry no List-Id:).

One row is written per local envelope recipient (a recipient outside LOCAL_DOMAINS is skipped), valid for LIST_CONFIRMATION_DAYS (default 7) and then deleted by pepsi-queue gc. The first record of a request wins, so a re-delivered copy can neither change its list nor extend its window. Recording is fail-open, and the message is unaffected either way. Anybody can send a message that passes this test; recording it is not trusting it. Nothing is whitelisted until the recipient’s own outgoing reply names the request in In-Reply-To: and goes to its control address, and the learnt row still requires an authenticated posting.

95.1.25.1.4. Configuration

Options live in the stage’s own [stage-<name>] section (PROGRAM = pepsi-stage-check-whitelist): the required WHITELIST_NAME list of groups to consult; a NEXT_STAGE, which is also required (the stage always hands the message on, so pepsi-setup rejects a section without one); the locality options LOCAL_DOMAINS / TARGETS / RECIPIENT_DELIMITER; and the mailing-list options RECORD_LIST_CONFIRMATIONS (default no) and LIST_CONFIRMATION_DAYS (default 7, 1 to 90). The locality options serve to expand a placeholder in a WHITELIST_NAME entry, and LOCAL_DOMAINS also decides for which recipients a confirmation request is recorded. LIST_CONFIRMATION_DAYS matters only when RECORD_LIST_CONFIRMATIONS is on. The stage checks all of these options when it starts, whether it uses them or not. All these options are documented in pepsi.conf(5); the whitelist rows themselves are managed with pepsi-whitelist(1).

95.1.25.1.5. State

Inputs: state.auth.dkim / state.auth.arc / state.auth.dmarc (for dkim_required rows), state.auth.dmarc / state.auth.arc_override / state.auth.dkim (for dmarc_required rows), state.signature_verified (for signature_required rows), which pepsi-stage-decrypt(1) sets only for its valid signature verdict (so a sound signature by a key that could not be anchored, valid-untrusted, does not open the gate), and state.auth.arc / state.auth.arc_sealers (for sealer_domain rows), both written by pepsi-stage-arc(1). The From: header comes from the from_header column, not from state; List-Id: is read from the message’s header block, which is why this stage loads it (the body is never loaded).

Outputs: on a match the stage merges spam: false into state (the key that pepsi-stage-anti-spam(1) reads to skip the payment gate) and advances; on no match state is left untouched. Side effects: a matched learnt list row’s last_seen is refreshed, and with RECORD_LIST_CONFIRMATIONS a confirmation request is recorded in pepsi.list_confirmation. The state layout is described in pepsi.state(7).

Transitions: always advances to NEXT_STAGE, whether or not the sender matched the whitelist. A match is committed with the reroute helper, which merges spam: false and moves the stage in the same single UPDATE; a non-match is a plain advance. There is no branch; the stage never pauses, fails, bounces or finishes. The actual payment/drop decision is left to a later stage (pepsi-stage-anti-spam(1)).

95.1.25.1.6. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

95.1.25.1.7. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default [pepsi] LOG, else info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

95.1.25.1.8. Exit Status

The outcome of each message is reported to pepsi-dispatch(1) on the worker’s status line, not as the exit status. A message that cannot be checked (a database error, a {login} expansion that could not ask the passwd database) is paused and retried, as for every stage; a malformed or too expensive whitelist_regex is not an error at all: that row matches nothing (see above).

0

The worker ran until its standard input was closed.

1

A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.

78

The worker refused to start: a [stage-*] section it serves does not parse (e.g. a missing WHITELIST_NAME, or LIST_CONFIRMATION_DAYS outside 1 to 90). The dispatcher requeues what it handed over and retries the stage later.

95.1.25.1.9. Examples

Check message 42 against the whitelist (it must be running; its outcome is printed as the status line):

echo 42 | pepsi-stage-check-whitelist -c /etc/pepsi/pepsi.conf worker

A pipeline section that whitelists trusted senders before the payment gate:

[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = anti-spam
WHITELIST_NAME = trusted-senders

Consulting the shared list and each recipient’s own, so that a whitelist that a user seeded with pepsi-whitelist import is honoured:

[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = anti-spam
WHITELIST_NAME = trusted-senders, {localpart}/correspondents

Whitelisting a partner domain, but only when its mail is DKIM/ARC-signed:

INSERT INTO pepsi.whitelist (whitelist_name, whitelist_regex, dkim_required)
VALUES ('trusted-senders', '@partner\.example\.org$', TRUE);

dkim_required defaults to TRUE when an INSERT leaves it out, as it does for pepsi-whitelist add; a row trusting unsigned mail must say dkim_required = FALSE explicitly. A partner whose domain does not sign but publishes DMARC and SPF is better held to dmarc_required:

pepsi-whitelist add trusted-senders --no-dkim-required --dmarc-required '@partner\.example\.org$'

Letting a mailing list past the payment gate, whoever posted to it, but only when the list’s own ADMD sealed the message’s ARC chain:

pepsi-whitelist add-list alice/lists users.rust-lang.org \
    --sealer mail.rust-lang.org

Learning, live, the lists that users subscribe to by answering a confirmation request. This stage records the request, pepsi-stage-auto-whitelist(1) learns the list on the outbound path into each user’s <login>/lists, and this stage consults it:

[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = anti-spam
WHITELIST_NAME = trusted-senders, {login}/correspondents, {login}/lists
RECORD_LIST_CONFIRMATIONS = yes

[stage-auto-whitelist]
PROGRAM = pepsi-stage-auto-whitelist
NEXT_STAGE = dkim-sign
WHITELIST_NAME = trusted-senders
LIST_WHITELIST_NAME = {login}/lists

Deliberately trusting the header alone, for a list that does not ARC-seal. An attacker only has to copy a List-Id: to satisfy such a row, so scope it to one user’s own whitelist rather than a shared one:

pepsi-whitelist add-list alice/lists announce.example.org --no-dkim-required

95.1.25.1.10. See Also

pepsi-config(1), pepsi-whitelist(1), pepsi-stage-anti-spam(1), pepsi-stage-arc(1), pepsi-settings(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

95.1.25.1.11. Bugs

Report bugs to the Pepsi issue tracker.