95.1.49. pepsi-whitelist

manage the sender whitelist, by hand or by mailbox

Manual section:

1

95.1.49.1.1. Name

pepsi-whitelist - add, list, remove and import sender whitelist entries.

95.1.49.1.2. Synopsis

pepsi-whitelist [GLOBAL-OPTIONS] add [–no-dkim-required] [–dmarc-required] [–signature-required] [–sealer DOMAIN] NAME REGEX

pepsi-whitelist [GLOBAL-OPTIONS] add-list [–pattern] [–no-dkim-required] [–dmarc-required] [–signature-required] [–sealer DOMAIN] NAME LIST-ID

pepsi-whitelist [GLOBAL-OPTIONS] list [NAME] [–json]

pepsi-whitelist [GLOBAL-OPTIONS] remove NAME REGEX

pepsi-whitelist [GLOBAL-OPTIONS] remove-list [–pattern] NAME LIST-ID

pepsi-whitelist [GLOBAL-OPTIONS] import [IMPORT-OPTIONS] [PATH…]

95.1.49.1.3. Description

pepsi-whitelist is the operator’s tool for the pepsi.whitelist table — the named sets of patterns that mark a message as trusted (non-spam). Each row has a whitelist_name (the group it belongs to), a whitelist_regex (a POSIX extended regular expression matched case-insensitively, via the ~* operator), a match_field and four conditions: dkim_required, dmarc_required, signature_required and sealer_domain. The match_field names what the expression is matched against, the message’s From: addr-spec or its List-Id:; a row with several conditions needs all of them. A From: address is matched in canonical form: Unicode NFC, local part case-folded, domain as its A-label (xn--bcher-kva.example, not bücher.example); the generated patterns are built from the same form, and a hand-written expression must name an internationalised domain by its A-label. The rows and their conditions are described in pepsi-stage-check-whitelist(1), the stage that evaluates them.

Every pattern is checked before it is stored: the program asks PostgreSQL, the engine that will run the pattern, to compile it. add, add-list and import refuse an expression that does not compile, with PostgreSQL’s reason, and an import writes nothing if any of its patterns is refused.

The whitelist is consulted by pepsi-stage-check-whitelist(1) and populated automatically from outgoing recipients by pepsi-stage-auto-whitelist(1); this tool lets an administrator manage rows directly, and lets each user seed a whitelist of their own from the mail they have already sent (import). It is not a stage. It connects to the same database as the other components, through the shared [pepsi-postgres] section, and reads the optional [pepsi-whitelist] section described in pepsi.conf(5).

Unlike pepsi-stage-auto-whitelist(1), which escapes a recipient address into a boundary-anchored pattern, add stores the whitelist_regex exactly as given, so the operator controls the matching directly. import escapes the addresses it finds, exactly as the stage does.

95.1.49.1.3.1. Mailing lists

add-list and remove-list manage the list-id rows that whitelist a mailing list as a whole: every posting it relays, whoever wrote it. A From: pattern cannot express this, because a list’s postings carry arbitrary sender addresses.

They are separate subcommands rather than a flag on add because they take a different kind of argument. add stores its REGEX verbatim, which is right for a pattern an operator composed deliberately; a list identifier typed into that slot would be an unanchored regular expression, so users.example.org would also match users.example.org.attacker.example and whitelist a list of the attacker’s choosing. add-list takes the identifier as it appears in the header and anchors it itself; the brackets are optional, and case and any surrounding phrase are ignored. –pattern is the escape hatch for the deliberate wildcard (every list at a domain), and then the argument is a regular expression again, as with add.

A List-Id: header is unauthenticated text that any sender can copy out of a genuine posting, so a rule resting on it alone is a rule anyone can satisfy. –sealer DOMAIN is the other half: the row then also requires that the message arrived with an ARC chain that validated and that DOMAIN was one of its sealers, as recorded by pepsi-stage-arc(1). DOMAIN is normalised (lower-cased, trailing dot removed) to the same form the ARC stage records, so the two cannot disagree. It works on add rows too, where it restricts a sender pattern to mail relayed through a known forwarder.

Typing identifiers is not the only way in. import proposes the lists whose postings the user answered (see below), and pepsi-stage-auto-whitelist(1) learns a list live when the user answers the list’s request to confirm a subscription. A row learnt that way expires (last_seen + max_idle, shown as EXPIRES by list) unless postings of the list keep refreshing it, since leaving a list sends no signal; rows written by add-list or confirmed at an import prompt never expire.

95.1.49.1.3.2. Whitelist names and who may use them

A whitelist_name is either

global — a single segment such as correspondents, which only the operator

may manage; or

owned by a user — <login>/<segment> such as alice/correspondents, and

alice/lists/rust-lang for further sub-lists.

A segment consists of letters, digits, _, . and -; the separator is /. A segment made of nothing but dots is refused, so . and .. cannot be segments and a whitelist name can never be a relative path. A dot is otherwise allowed because the owning segment is the account’s passwd login, and alice.smith is an ordinary login on any LDAP/SSSD/FreeIPA deployment — as is a {localpart} of first.last in pepsi-stage-check-whitelist(1). / is the separator because no POSIX-ish system permits it in a login name, whereas -, _, . and a trailing $ are all legal in one and so could not delimit unambiguously. (A login holding a character outside the segment grammar still owns no namespace; import –all-users skips such an account with a message instead of failing the whole run.)

Because this program is installed setuid (see below), any local user can run it. An unprivileged caller may only read and write names in their own namespace — the login of the real uid, taken from the passwd database, never from $USER or an argument. That restriction covers list as well: the operator’s shared whitelist is a record of who the organisation corresponds with, and this program must not hand it to every account on the host. root, and the pepsi, pepsi-owner and pepsi-whitelist accounts, may manage every whitelist.

For a per-user whitelist to have any effect, some stage must consult it; see the {localpart} and {login} placeholders of WHITELIST_NAME in pepsi-stage-check-whitelist(1).

95.1.49.1.3.3. The setuid model

Managing the shared pepsi.whitelist table needs a database identity that ordinary users do not have, while scanning a user’s mailbox needs their identity. The program is therefore installed setuid to the pepsi-whitelist account (mode 4755) and immediately lowers its effective ids back to the invoking user, keeping the owner only in the saved set-user-id. It raises them again only for the few milliseconds in which it talks to PostgreSQL, because peer authentication keys off the effective uid.

The same mechanism is what lets root use the tool. When started as root, the other operator commands simply become the pepsi service account, permanently, because they have nothing to do as root: these are pepsi-queue(1), pepsi-status(1), pepsi-settings(1), pepsi-tlsrpt(1) and pepsi-failure-bouncer(1). This one does have work as root: import –user and –all-users run the scan helper under another user’s credentials, which only root can arrange. It therefore keeps the invoking identity, root included. For the duration of each database call it switches only its effective ids to the pepsi-whitelist account, whose database role is granted the whitelist table alone, and returns to root afterwards.

That account is deliberately not the pepsi service account: pepsi-setup(1) grants its database role SELECT, INSERT and DELETE on the pepsi.whitelist table and nothing more. A compromise of a binary that every local user may execute therefore cannot reach the message queue, the statistics or the signing keys.

The mailbox parsing does not happen in this process at all. It is the one part that reads data an attacker controls, namely every message that anybody ever sent the user. It runs in pepsi-helper-mailbox-scan(1), spawned with setresuid setting the real, effective and saved ids to the target user, so that process has no privileged id left anywhere in its credentials and cannot become pepsi-whitelist even in principle.

A setuid program must not be pointed at a configuration file of the caller’s choosing. -c/–config is therefore rejected for unprivileged callers. Before the configuration is read, the program also removes the environment variables that would otherwise redirect the configuration search (HOME, XDG_CONFIG_HOME), the database connection (PG*) or a ${NAME} expansion in a configuration value (TALER_*, PEPSI_*). It replaces PATH by a fixed system search path, because ${DATADIR} is derived from PATH and the scan helper is found through it.

95.1.49.1.4. Commands

add [–no-dkim-required] [–dmarc-required] [–signature-required] [–sealer DOMAIN] NAME REGEX

Insert a from entry storing REGEX under whitelist NAME. The insert is idempotent on the UNIQUE(whitelist_name, match_field, whitelist_regex) constraint: re-adding an existing triple is a no-op and leaves its conditions unchanged.

–no-dkim-required

Clear the row’s dkim_required flag. By default it is set, so a match counts only if the message’s From: domain is authenticated: a DKIM pass, or an ARC pass that DMARC also confirms.

–dmarc-required

Set the row’s dmarc_required flag. By default it is clear. When set, a match counts only if the message passed DMARC on an aligned identifier — an aligned DKIM signature or an aligned SPF pass — or a trusted ARC sealer vouched for it. Use it with –no-dkim-required for a correspondent whose domain does not DKIM-sign but publishes DMARC: the row is then bound to an authenticated sender instead of to an address anyone can forge.

–signature-required

Set the row’s signature_required flag. By default it is clear. When set, a match counts only if pepsi-stage-decrypt(1) verified the message’s end-to-end signature against a trusted key (state.signature_verified).

–sealer DOMAIN

Set the row’s sealer_domain. By default it is empty (any path). When set, a match counts only if the message arrived with an ARC chain that validated and DOMAIN was one of its sealers. DOMAIN is lower-cased and stripped of a trailing dot, matching what pepsi-stage-arc(1) records.

add-list [–pattern] [CONDITION-OPTIONS] NAME LIST-ID

Insert a list-id entry under whitelist NAME, matching every message whose List-Id: header carries the identifier LIST-ID. The identifier may be given bracketed or bare, in any case, with or without a descriptive phrase; it is reduced to the bare lower-cased form and anchored, so it matches that list and no other. Takes the same condition options as add, and pairing it with –sealer is strongly advised — see Mailing lists above.

–pattern

Treat LIST-ID as a POSIX extended regular expression to store verbatim, rather than as one identifier to anchor. For the deliberate wildcard case (every list at a domain); an unanchored expression matches any identifier containing it, so write the anchors yourself.

list [NAME] [–json]

List entries ordered by name, match field, then regex. With NAME, only that whitelist group is shown.

–json

Emit a JSON array of objects (whitelist_id, whitelist_name, whitelist_regex, match_field, dkim_required, dmarc_required, signature_required, sealer_domain, expires) instead of the human-readable table. expires is the day a learnt list row lapses unless refreshed, and null for a row that never expires.

remove NAME REGEX

Delete the from entry whose whitelist_name is NAME and whose whitelist_regex is exactly REGEX. A list-id row with the same pattern text is a different entry and is left alone; use remove-list for it.

remove-list [–pattern] NAME LIST-ID

Delete the list-id entry add-list would have created for LIST-ID, so the identifier that added a list also removes it. –pattern has the same meaning as in add-list, and must be given if it was given then.

import [OPTIONS] [PATH…]

Seed a whitelist from the recipients of the mail the user has sent, and offer the mailing lists the user took part in.

A mailbox is scanned for messages that were sent by the user, and the To:, Cc: and Bcc: addresses of those messages become whitelist entries, so that when those people write back their replies are recognised. Recipients at a local domain are skipped — local senders never pass through the inbound gate.

A message counts as sent by the user when its From: (or Sender:) is at one of the local domains and its header block carries no Received:, Return-Path:, Delivered-To: or X-Original-To: field. Both halves are needed: From: is unauthenticated, so on its own it is a claim anybody can make, and a stranger who sends you one message with your own address in From: and their own in To: would otherwise have that address inserted into your whitelist by the next import, and the whitelist would then let their mail through. Requiring a signature does not close this hole, because the forger signs their own domain’s mail. The four fields above are written by a receiving MTA or MDA and never by a mail user agent composing a message, so their absence is the evidence that the message was filed here rather than delivered here.

Received mail is therefore ignored, and so is anything else that arrived. –include-delivered turns that test off; see below before using it.

Where a message is filed still does not matter: a Maildir scan walks the whole tree, so ~/Maildir covers .Sent and every archive folder in one run. Only the header block of each message is read, never the body.

Mailing lists. The scan also remembers every message, received mail included, that carries a List-Id: and a Message-ID:. It proposes a list when a message that the user sent (by the rule above) names one of those messages in its In-Reply-To:, because the user then answered a posting of that list. Merely receiving list mail proposes nothing, because anybody can send a message with any List-Id:; the list and the reply must both be in the scanned mailbox, which is why scanning the whole ~/Maildir rather than only the Sent folder is what finds them. Each proposal becomes one exact list-id row (the one add-list writes for the identifier) in the same whitelist, carrying the same –no-dkim-required / –dmarc-required / –signature-required conditions and no sealer_domain, and is confirmed per list exactly like a wildcard: one question per list on a terminal, all accepted with –yes, all declined when standard input is not a terminal, none applied by –dry-run. The identifiers the helper reports are re-validated as RFC 2919 list identifiers before they are offered. Rows confirmed here never expire.

PATH… are mailbox files or Maildir directories. With none, ~/Maildir is scanned if it exists, otherwise /var/mail/<login>; [pepsi-whitelist] MAILBOX overrides that default. Note that /var/mail/<login> is a delivery spool and holds nothing but received mail, so on a host with no Maildir the default finds nothing and a Sent folder must be named explicitly.

–name NAME

Whitelist to seed. Defaults to <login>/correspondents (or the global correspondents for a privileged caller). With –user or –all-users an explicit –name replaces the per-account namespace: every scanned account’s rows go into NAME, so --all-users --name correspondents builds one list from everybody’s correspondence – a global list that admits any of those senders for every user who consults it. Omit –name to keep each account’s rows in its own list.

–include-delivered

Count a message as sent on its From: alone, dropping the delivery-trace test above. Unsafe on any mailbox that holds received mail: a forged From: is then enough to plant a whitelist entry. It exists for the unusual store whose sent copies really did go through delivery, and should be pointed at that Sent folder only, never at an inbox.

–format auto|mbox|maildir

Force the mailbox format instead of detecting it per path (a directory is a Maildir, a file an mbox).

–auto-wildcard

Offer to replace the individual addresses at a domain with one *@domain entry when at least N distinct addresses were seen there (N is --threshold, default [pepsi-whitelist] WILDCARD_THRESHOLD, 10).

Proposals are printed and confirmed one at a time on a terminal; declining one simply keeps that domain’s addresses individual. With –yes all are accepted; with no terminal and no –yes all are declined, so an unattended run can never widen a whitelist to a whole domain by itself.

Public e-mail hosters are never proposed, however many correspondents the user has there: “everyone at gmail.com” is not a set of people the user knows. The list of those domains is [pepsi-whitelist] HOSTERS_FILE (/usr/share/pepsi/hosters.txt).

–threshold N

Distinct addresses at one domain before a wildcard is proposed.

–hosters FILE

Use FILE as the public-hoster list instead of the configured one.

–no-hosters

Do not exclude any domain from wildcarding.

–dry-run

Report what would be added and change nothing.

-y, –yes

Accept every wildcard and mailing-list proposal without asking.

–max-entries N

Most rows this import may add (default [pepsi-whitelist] MAX_ENTRIES, 5000). Every row of a whitelist is matched against the From: of every inbound message that consults it, so an unbounded import is a permanent per-message cost.

–max-messages N

Examine at most N messages. This caps how many are parsed and classified, not how much of the mailbox is read: the file, tree or IMAP account is still walked to the end.

–own-address ADDRESS

Treat only mail sent from this exact address as the user’s own, instead of any sender at a local domain. Repeatable.

–imap URL

Scan a mail store that is not on this filesystem, e.g. imaps://alice@mail.example.com/. imaps:// is implicit TLS (port 993), imap:// is STARTTLS (port 143); a path component is a mailbox pattern (default *, every mailbox). Mailboxes are opened read-only (EXAMINE) and only header blocks are fetched, so a scan does not mark anything \Seen.

IMAP is the network option because LMTP cannot do this: LMTP is a delivery protocol with no verb that lists or fetches anything.

–imap-user NAME

IMAP login, if it is not in the URL.

–imap-password-file FILE

Read the password from the first line of FILE instead of prompting. Keep it mode 0600: this path reads the file itself and does not check its permissions, unlike pepsi-helper-mailbox-scan(1)’s option of the same name, which refuses a file that any other user can read. The password is never passed on a command line, where every other user could read it out of ps.

–doveadm

Read a local Dovecot store through doveadm, for formats no file scanner can parse (mdbox/sdbox). doveadm is run by this program (it needs privilege) and its output is piped into the unprivileged scan helper.

–doveadm-path PATH

The doveadm binary to use.

–user LOGIN

Import for another user, into their namespace. Requires a privileged caller; the scan helper is dropped to that user’s credentials.

–all-users

Import for every local account that [pepsi-whitelist] TARGETS permits, each into its own <login>/correspondents (unless –name names one list for all of them, see above). Requires a privileged caller. Only accounts in the local passwd file are found; name a directory-service account with –user.

–no-dkim-required, –dmarc-required, –signature-required

As for add, applied to every row the import adds.

95.1.49.1.5. Global Options

These global options precede the subcommand (a trailing flag is rejected).

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations. Rejected for unprivileged callers when this program is installed setuid: the configuration decides the database connection, so a setuid process reads only a root-owned one.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity. LOGLEVEL is one of error, warn, info, debug or trace (default: [pepsi] LOG, else info).

-v, –verbose

Show log messages from all sources, including third-party libraries.

-h, –help

Print a usage summary and exit.

-V, –version

Print the version and exit.

95.1.49.1.6. Exit Status

0

Successful completion.

1

An error occurred: a malformed configuration file, a failed database connection or query, a pattern PostgreSQL refuses to compile, a whitelist name outside the caller’s namespace, or -c given by an unprivileged caller. The reason is written to the log.

2

The command line itself was wrong (an unknown subcommand or option, or a global option given after the subcommand).

78

The database schema belongs to another Pepsi release (not installed, older, newer or modified); run pepsi-setup schema (see pepsi-setup(1)).

95.1.49.1.7. Files

When –config is not given, the first existing file from the following list is used. Every Pepsi component shares the same configuration file, so this is the same list each one searches:

  • $XDG_CONFIG_HOME/pepsi.conf

  • $HOME/.config/pepsi.conf

  • /etc/pepsi/pepsi.conf

  • /etc/pepsi.conf

When the program runs setuid for an unprivileged caller, XDG_CONFIG_HOME and HOME have been removed from the environment, so only the system-wide files are found.

${DATADIR}/hosters.txt

The list of public hosters for which import –auto-wildcard never proposes a wildcard, installed by make install from contrib/hosters.txt (/usr/share/pepsi/hosters.txt on a default installation). Overridden by [pepsi-whitelist] HOSTERS_FILE or, for one run, by –hosters.

~/Maildir, /var/mail/login

The mailboxes import scans when no PATH is given, in that order; [pepsi-whitelist] MAILBOX overrides the pair.

95.1.49.1.8. Examples

Whitelist a sender (an authenticated From: domain required by default):

pepsi-whitelist -c /etc/pepsi/pepsi.conf add trusted-senders '^alice@example\.com$'

The pattern is stored verbatim, so anchoring it is the operator’s job. An unanchored alice@example\.com also matches alice@example.com.attacker.example and "alice@example.com!"@attacker.example, both of which an attacker can put in a From: header. The generated patterns (import and pepsi-stage-auto-whitelist(1)) always anchor on ^/$ for this reason.

Whitelist a whole domain without requiring a signature:

pepsi-whitelist -c /etc/pepsi/pepsi.conf add trusted-senders --no-dkim-required '@example\.org$'

List the trusted-senders group as JSON:

pepsi-whitelist -c /etc/pepsi/pepsi.conf list trusted-senders --json

Remove an entry:

pepsi-whitelist -c /etc/pepsi/pepsi.conf remove trusted-senders '@example\.org$'

As an ordinary user, whitelist a mailing list you subscribe to, so its postings reach you whoever wrote them, but only when the list’s own ADMD sealed them:

pepsi-whitelist add-list alice/lists users.example.org \
    --sealer mail.example.org

Every list run at one domain, which needs the anchors written by hand:

pepsi-whitelist add-list alice/lists --pattern '^[a-z0-9-]+\.lists\.example\.org$' \
    --sealer lists.example.org

Unsubscribed; drop the rule again with the identifier that created it:

pepsi-whitelist remove-list alice/lists users.example.org

As an ordinary user, seed your own whitelist from your mailbox, seeing first what it would do:

pepsi-whitelist import --dry-run
pepsi-whitelist import

Same, but collapsing the domains you write to widely into one entry each (public hosters are never collapsed):

pepsi-whitelist import --auto-wildcard

Scan only your Sent folder, and a remote account over IMAP:

pepsi-whitelist import ~/Maildir/.Sent
pepsi-whitelist import --imap imaps://alice@mail.example.com/Sent

As root, seed every local account’s own whitelist from a Dovecot store:

pepsi-whitelist import --all-users --doveadm --auto-wildcard --yes

95.1.49.1.9. See Also

pepsi-config(1), pepsi-helper-mailbox-scan(1), pepsi-stage-check-whitelist(1), pepsi-stage-auto-whitelist(1), pepsi-queue(1), pepsi.conf(5), pepsi-setup(1)

95.1.49.1.10. Bugs

Report bugs to the Pepsi issue tracker.