95.1.49. pepsi-whitelist¶
manage the sender whitelist, by hand or by mailbox
- Manual section:
1
95.1.49.1.1. Name¶
pepsi-whitelist - add, list, remove and import sender whitelist entries.
95.1.49.1.2. Synopsis¶
pepsi-whitelist [GLOBAL-OPTIONS] add [–no-dkim-required] [–dmarc-required] [–signature-required] [–sealer DOMAIN] NAME REGEX
pepsi-whitelist [GLOBAL-OPTIONS] add-list [–pattern] [–no-dkim-required] [–dmarc-required] [–signature-required] [–sealer DOMAIN] NAME LIST-ID
pepsi-whitelist [GLOBAL-OPTIONS] list [NAME] [–json]
pepsi-whitelist [GLOBAL-OPTIONS] remove NAME REGEX
pepsi-whitelist [GLOBAL-OPTIONS] remove-list [–pattern] NAME LIST-ID
pepsi-whitelist [GLOBAL-OPTIONS] import [IMPORT-OPTIONS] [PATH…]
95.1.49.1.3. Description¶
pepsi-whitelist is the operator’s tool for the pepsi.whitelist table —
the named sets of patterns that mark a message as trusted (non-spam).
Each row has a whitelist_name (the group it belongs to), a
whitelist_regex (a POSIX extended regular expression matched
case-insensitively, via the ~* operator), a match_field and four
conditions: dkim_required, dmarc_required, signature_required and
sealer_domain. The match_field names what the expression is matched
against, the message’s From: addr-spec or its List-Id:; a row with
several conditions needs all of them. A From: address is matched in canonical form: Unicode NFC, local part
case-folded, domain as its A-label (xn--bcher-kva.example, not
bücher.example); the generated patterns are built from the same form, and a
hand-written expression must name an internationalised domain by its A-label.
The rows and their conditions are described in
pepsi-stage-check-whitelist(1), the stage that evaluates them.
Every pattern is checked before it is stored: the program asks PostgreSQL, the engine that will run the pattern, to compile it. add, add-list and import refuse an expression that does not compile, with PostgreSQL’s reason, and an import writes nothing if any of its patterns is refused.
The whitelist is consulted by pepsi-stage-check-whitelist(1) and populated
automatically from outgoing recipients by pepsi-stage-auto-whitelist(1);
this tool lets an administrator manage rows directly, and lets each user seed a
whitelist of their own from the mail they have already sent (import). It is
not a stage. It connects to the same database as the other components, through
the shared [pepsi-postgres] section, and reads the optional
[pepsi-whitelist] section described in pepsi.conf(5).
Unlike pepsi-stage-auto-whitelist(1), which escapes a recipient address
into a boundary-anchored pattern, add stores the whitelist_regex exactly
as given, so the operator controls the matching directly. import escapes the
addresses it finds, exactly as the stage does.
95.1.49.1.3.1. Mailing lists¶
add-list and remove-list manage the list-id rows that whitelist a
mailing list as a whole: every posting it relays, whoever wrote it. A From:
pattern cannot express this, because a list’s postings carry arbitrary sender
addresses.
They are separate subcommands rather than a flag on add because they take a
different kind of argument. add stores its REGEX verbatim, which is right
for a pattern an operator composed deliberately; a list identifier typed into
that slot would be an unanchored regular expression, so users.example.org
would also match users.example.org.attacker.example and whitelist a list of
the attacker’s choosing. add-list takes the identifier as it appears in the
header and anchors it itself; the brackets are optional, and case and any
surrounding phrase are ignored. –pattern is the escape hatch for the deliberate wildcard (every list
at a domain), and then the argument is a regular expression again, as with
add.
A List-Id: header is unauthenticated text that any sender can copy out of a
genuine posting, so a rule resting on it alone is a rule anyone can satisfy.
–sealer DOMAIN is the other half: the row then also requires that the
message arrived with an ARC chain that validated and that DOMAIN was one of its
sealers, as recorded by pepsi-stage-arc(1). DOMAIN is normalised
(lower-cased, trailing dot removed) to the same form the ARC stage records, so
the two cannot disagree. It works on add rows too, where it restricts a
sender pattern to mail relayed through a known forwarder.
Typing identifiers is not the only way in. import proposes the lists whose
postings the user answered (see below), and pepsi-stage-auto-whitelist(1)
learns a list live when the user answers the list’s request to confirm a
subscription.
A row learnt that way expires (last_seen + max_idle, shown as
EXPIRES by list) unless postings of the list keep refreshing it, since
leaving a list sends no signal; rows written by add-list or confirmed at an
import prompt never expire.
95.1.49.1.3.2. Whitelist names and who may use them¶
A whitelist_name is either
- global — a single segment such as
correspondents, which only the operator may manage; or
- owned by a user —
<login>/<segment>such asalice/correspondents, and alice/lists/rust-langfor further sub-lists.
A segment consists of letters, digits, _, . and -; the separator is
/. A segment made of nothing but dots is refused, so . and ..
cannot be segments and a whitelist name can never be a relative path. A dot is
otherwise allowed because the owning segment is the account’s passwd login, and
alice.smith is an ordinary login on any LDAP/SSSD/FreeIPA deployment — as is a
{localpart} of first.last in pepsi-stage-check-whitelist(1). / is
the separator because no POSIX-ish system permits it in a login name, whereas
-, _, . and a trailing $ are all legal in one and so could not
delimit unambiguously. (A login holding a character outside the segment grammar
still owns no namespace; import –all-users skips such an account with a
message instead of failing the whole run.)
Because this program is installed setuid (see below), any local user can run
it. An unprivileged caller may only read and write names in their own namespace —
the login of the real uid, taken from the passwd database, never from
$USER or an argument. That restriction covers list as well: the operator’s
shared whitelist is a record of who the organisation corresponds with, and this
program must not hand it to every account on the host. root, and the
pepsi, pepsi-owner and pepsi-whitelist accounts, may manage every
whitelist.
For a per-user whitelist to have any effect, some stage must consult it; see the
{localpart} and {login} placeholders of WHITELIST_NAME in
pepsi-stage-check-whitelist(1).
95.1.49.1.3.3. The setuid model¶
Managing the shared pepsi.whitelist table needs a database identity that
ordinary users do not have, while scanning a user’s mailbox needs their
identity. The program is therefore installed setuid to the pepsi-whitelist
account (mode 4755) and immediately lowers its effective ids back to the invoking
user, keeping the owner only in the saved set-user-id. It raises them again only
for the few milliseconds in which it talks to PostgreSQL, because peer
authentication keys off the effective uid.
The same mechanism is what lets root use the tool. When started as
root, the other operator commands simply become the pepsi service
account, permanently, because they have nothing to do as root: these are
pepsi-queue(1), pepsi-status(1), pepsi-settings(1),
pepsi-tlsrpt(1) and pepsi-failure-bouncer(1). This one does have
work as root: import –user and –all-users run the scan helper
under another user’s credentials, which only root can arrange. It
therefore keeps the invoking identity, root included. For the duration of
each database call it switches only its effective ids to the
pepsi-whitelist account, whose database role is granted the whitelist table
alone, and returns to root afterwards.
That account is deliberately not the pepsi service account:
pepsi-setup(1) grants its database role SELECT, INSERT and
DELETE on the pepsi.whitelist table and nothing more. A compromise of a
binary that every local user may execute therefore cannot reach the message
queue, the statistics or the signing keys.
The mailbox parsing does not happen in this process at all. It is the one part
that reads data an attacker controls, namely every message that anybody ever
sent the user.
It runs in pepsi-helper-mailbox-scan(1), spawned with setresuid setting
the real, effective and saved ids to the target user, so that process has no
privileged id left anywhere in its credentials and cannot become
pepsi-whitelist even in principle.
A setuid program must not be pointed at a configuration file of the caller’s
choosing. -c/–config is therefore rejected for unprivileged callers.
Before the configuration is read, the program also removes the environment
variables that would otherwise redirect the configuration search (HOME,
XDG_CONFIG_HOME), the database connection (PG*) or a ${NAME}
expansion in a configuration value (TALER_*, PEPSI_*). It replaces
PATH by a fixed system search path, because ${DATADIR} is derived from
PATH and the scan helper is found through it.
95.1.49.1.4. Commands¶
- add [–no-dkim-required] [–dmarc-required] [–signature-required] [–sealer DOMAIN] NAME REGEX
Insert a
fromentry storing REGEX under whitelist NAME. The insert is idempotent on theUNIQUE(whitelist_name, match_field, whitelist_regex)constraint: re-adding an existing triple is a no-op and leaves its conditions unchanged.- –no-dkim-required
Clear the row’s
dkim_requiredflag. By default it is set, so a match counts only if the message’sFrom:domain is authenticated: a DKIM pass, or an ARC pass that DMARC also confirms.- –dmarc-required
Set the row’s
dmarc_requiredflag. By default it is clear. When set, a match counts only if the message passed DMARC on an aligned identifier — an aligned DKIM signature or an aligned SPF pass — or a trusted ARC sealer vouched for it. Use it with –no-dkim-required for a correspondent whose domain does not DKIM-sign but publishes DMARC: the row is then bound to an authenticated sender instead of to an address anyone can forge.- –signature-required
Set the row’s
signature_requiredflag. By default it is clear. When set, a match counts only if pepsi-stage-decrypt(1) verified the message’s end-to-end signature against a trusted key (state.signature_verified).- –sealer DOMAIN
Set the row’s
sealer_domain. By default it is empty (any path). When set, a match counts only if the message arrived with an ARC chain that validated and DOMAIN was one of its sealers. DOMAIN is lower-cased and stripped of a trailing dot, matching what pepsi-stage-arc(1) records.
- add-list [–pattern] [CONDITION-OPTIONS] NAME LIST-ID
Insert a
list-identry under whitelist NAME, matching every message whoseList-Id:header carries the identifier LIST-ID. The identifier may be given bracketed or bare, in any case, with or without a descriptive phrase; it is reduced to the bare lower-cased form and anchored, so it matches that list and no other. Takes the same condition options as add, and pairing it with –sealer is strongly advised — see Mailing lists above.- –pattern
Treat LIST-ID as a POSIX extended regular expression to store verbatim, rather than as one identifier to anchor. For the deliberate wildcard case (every list at a domain); an unanchored expression matches any identifier containing it, so write the anchors yourself.
- list [NAME] [–json]
List entries ordered by name, match field, then regex. With NAME, only that whitelist group is shown.
- –json
Emit a JSON array of objects (
whitelist_id,whitelist_name,whitelist_regex,match_field,dkim_required,dmarc_required,signature_required,sealer_domain,expires) instead of the human-readable table.expiresis the day a learnt list row lapses unless refreshed, andnullfor a row that never expires.
- remove NAME REGEX
Delete the
fromentry whosewhitelist_nameis NAME and whosewhitelist_regexis exactly REGEX. Alist-idrow with the same pattern text is a different entry and is left alone; use remove-list for it.- remove-list [–pattern] NAME LIST-ID
Delete the
list-identry add-list would have created for LIST-ID, so the identifier that added a list also removes it. –pattern has the same meaning as in add-list, and must be given if it was given then.- import [OPTIONS] [PATH…]
Seed a whitelist from the recipients of the mail the user has sent, and offer the mailing lists the user took part in.
A mailbox is scanned for messages that were sent by the user, and the
To:,Cc:andBcc:addresses of those messages become whitelist entries, so that when those people write back their replies are recognised. Recipients at a local domain are skipped — local senders never pass through the inbound gate.A message counts as sent by the user when its
From:(orSender:) is at one of the local domains and its header block carries noReceived:,Return-Path:,Delivered-To:orX-Original-To:field. Both halves are needed:From:is unauthenticated, so on its own it is a claim anybody can make, and a stranger who sends you one message with your own address inFrom:and their own inTo:would otherwise have that address inserted into your whitelist by the next import, and the whitelist would then let their mail through. Requiring a signature does not close this hole, because the forger signs their own domain’s mail. The four fields above are written by a receiving MTA or MDA and never by a mail user agent composing a message, so their absence is the evidence that the message was filed here rather than delivered here.Received mail is therefore ignored, and so is anything else that arrived. –include-delivered turns that test off; see below before using it.
Where a message is filed still does not matter: a Maildir scan walks the whole tree, so
~/Maildircovers.Sentand every archive folder in one run. Only the header block of each message is read, never the body.Mailing lists. The scan also remembers every message, received mail included, that carries a
List-Id:and aMessage-ID:. It proposes a list when a message that the user sent (by the rule above) names one of those messages in itsIn-Reply-To:, because the user then answered a posting of that list. Merely receiving list mail proposes nothing, because anybody can send a message with anyList-Id:; the list and the reply must both be in the scanned mailbox, which is why scanning the whole~/Maildirrather than only the Sent folder is what finds them. Each proposal becomes one exactlist-idrow (the one add-list writes for the identifier) in the same whitelist, carrying the same –no-dkim-required / –dmarc-required / –signature-required conditions and nosealer_domain, and is confirmed per list exactly like a wildcard: one question per list on a terminal, all accepted with –yes, all declined when standard input is not a terminal, none applied by –dry-run. The identifiers the helper reports are re-validated as RFC 2919 list identifiers before they are offered. Rows confirmed here never expire.PATH… are mailbox files or Maildir directories. With none,
~/Maildiris scanned if it exists, otherwise/var/mail/<login>;[pepsi-whitelist] MAILBOXoverrides that default. Note that/var/mail/<login>is a delivery spool and holds nothing but received mail, so on a host with no Maildir the default finds nothing and a Sent folder must be named explicitly.- –name NAME
Whitelist to seed. Defaults to
<login>/correspondents(or the globalcorrespondentsfor a privileged caller). With –user or –all-users an explicit –name replaces the per-account namespace: every scanned account’s rows go into NAME, so--all-users --name correspondentsbuilds one list from everybody’s correspondence – a global list that admits any of those senders for every user who consults it. Omit –name to keep each account’s rows in its own list.- –include-delivered
Count a message as sent on its
From:alone, dropping the delivery-trace test above. Unsafe on any mailbox that holds received mail: a forgedFrom:is then enough to plant a whitelist entry. It exists for the unusual store whose sent copies really did go through delivery, and should be pointed at that Sent folder only, never at an inbox.- –format
auto|mbox|maildir Force the mailbox format instead of detecting it per path (a directory is a Maildir, a file an mbox).
- –auto-wildcard
Offer to replace the individual addresses at a domain with one
*@domainentry when at least N distinct addresses were seen there (N is--threshold, default[pepsi-whitelist] WILDCARD_THRESHOLD, 10).Proposals are printed and confirmed one at a time on a terminal; declining one simply keeps that domain’s addresses individual. With –yes all are accepted; with no terminal and no –yes all are declined, so an unattended run can never widen a whitelist to a whole domain by itself.
Public e-mail hosters are never proposed, however many correspondents the user has there: “everyone at
gmail.com” is not a set of people the user knows. The list of those domains is[pepsi-whitelist] HOSTERS_FILE(/usr/share/pepsi/hosters.txt).- –threshold N
Distinct addresses at one domain before a wildcard is proposed.
- –hosters FILE
Use FILE as the public-hoster list instead of the configured one.
- –no-hosters
Do not exclude any domain from wildcarding.
- –dry-run
Report what would be added and change nothing.
- -y, –yes
Accept every wildcard and mailing-list proposal without asking.
- –max-entries N
Most rows this import may add (default
[pepsi-whitelist] MAX_ENTRIES, 5000). Every row of a whitelist is matched against theFrom:of every inbound message that consults it, so an unbounded import is a permanent per-message cost.- –max-messages N
Examine at most N messages. This caps how many are parsed and classified, not how much of the mailbox is read: the file, tree or IMAP account is still walked to the end.
- –own-address ADDRESS
Treat only mail sent from this exact address as the user’s own, instead of any sender at a local domain. Repeatable.
- –imap URL
Scan a mail store that is not on this filesystem, e.g.
imaps://alice@mail.example.com/.imaps://is implicit TLS (port 993),imap://is STARTTLS (port 143); a path component is a mailbox pattern (default*, every mailbox). Mailboxes are opened read-only (EXAMINE) and only header blocks are fetched, so a scan does not mark anything\Seen.IMAP is the network option because LMTP cannot do this: LMTP is a delivery protocol with no verb that lists or fetches anything.
- –imap-user NAME
IMAP login, if it is not in the URL.
- –imap-password-file FILE
Read the password from the first line of FILE instead of prompting. Keep it mode
0600: this path reads the file itself and does not check its permissions, unlike pepsi-helper-mailbox-scan(1)’s option of the same name, which refuses a file that any other user can read. The password is never passed on a command line, where every other user could read it out ofps.- –doveadm
Read a local Dovecot store through
doveadm, for formats no file scanner can parse (mdbox/sdbox).doveadmis run by this program (it needs privilege) and its output is piped into the unprivileged scan helper.- –doveadm-path PATH
The
doveadmbinary to use.- –user LOGIN
Import for another user, into their namespace. Requires a privileged caller; the scan helper is dropped to that user’s credentials.
- –all-users
Import for every local account that
[pepsi-whitelist] TARGETSpermits, each into its own<login>/correspondents(unless –name names one list for all of them, see above). Requires a privileged caller. Only accounts in the local passwd file are found; name a directory-service account with –user.- –no-dkim-required, –dmarc-required, –signature-required
As for add, applied to every row the import adds.
95.1.49.1.5. Global Options¶
These global options precede the subcommand (a trailing flag is rejected).
- -c FILE, –config FILE
Read the configuration from FILE instead of searching the default locations. Rejected for unprivileged callers when this program is installed setuid: the configuration decides the database connection, so a setuid process reads only a root-owned one.
- -L LOGLEVEL, –log LOGLEVEL
Set the logging verbosity. LOGLEVEL is one of
error,warn,info,debugortrace(default:[pepsi] LOG, elseinfo).- -v, –verbose
Show log messages from all sources, including third-party libraries.
- -h, –help
Print a usage summary and exit.
- -V, –version
Print the version and exit.
95.1.49.1.6. Exit Status¶
- 0
Successful completion.
- 1
An error occurred: a malformed configuration file, a failed database connection or query, a pattern PostgreSQL refuses to compile, a whitelist name outside the caller’s namespace, or -c given by an unprivileged caller. The reason is written to the log.
- 2
The command line itself was wrong (an unknown subcommand or option, or a global option given after the subcommand).
- 78
The database schema belongs to another Pepsi release (not installed, older, newer or modified); run
pepsi-setup schema(see pepsi-setup(1)).
95.1.49.1.7. Files¶
When –config is not given, the first existing file from the following list is used. Every Pepsi component shares the same configuration file, so this is the same list each one searches:
$XDG_CONFIG_HOME/pepsi.conf$HOME/.config/pepsi.conf/etc/pepsi/pepsi.conf/etc/pepsi.conf
When the program runs setuid for an unprivileged caller, XDG_CONFIG_HOME and
HOME have been removed from the environment, so only the system-wide files
are found.
${DATADIR}/hosters.txtThe list of public hosters for which import –auto-wildcard never proposes a wildcard, installed by
make installfromcontrib/hosters.txt(/usr/share/pepsi/hosters.txton a default installation). Overridden by[pepsi-whitelist] HOSTERS_FILEor, for one run, by –hosters.~/Maildir,/var/mail/loginThe mailboxes import scans when no PATH is given, in that order;
[pepsi-whitelist] MAILBOXoverrides the pair.
95.1.49.1.8. Examples¶
Whitelist a sender (an authenticated From: domain required by default):
pepsi-whitelist -c /etc/pepsi/pepsi.conf add trusted-senders '^alice@example\.com$'
The pattern is stored verbatim, so anchoring it is the operator’s job. An
unanchored alice@example\.com also matches
alice@example.com.attacker.example and
"alice@example.com!"@attacker.example, both of which an attacker can put in a
From: header. The generated patterns (import and
pepsi-stage-auto-whitelist(1)) always anchor on ^/$ for this
reason.
Whitelist a whole domain without requiring a signature:
pepsi-whitelist -c /etc/pepsi/pepsi.conf add trusted-senders --no-dkim-required '@example\.org$'
List the trusted-senders group as JSON:
pepsi-whitelist -c /etc/pepsi/pepsi.conf list trusted-senders --json
Remove an entry:
pepsi-whitelist -c /etc/pepsi/pepsi.conf remove trusted-senders '@example\.org$'
As an ordinary user, whitelist a mailing list you subscribe to, so its postings reach you whoever wrote them, but only when the list’s own ADMD sealed them:
pepsi-whitelist add-list alice/lists users.example.org \
--sealer mail.example.org
Every list run at one domain, which needs the anchors written by hand:
pepsi-whitelist add-list alice/lists --pattern '^[a-z0-9-]+\.lists\.example\.org$' \
--sealer lists.example.org
Unsubscribed; drop the rule again with the identifier that created it:
pepsi-whitelist remove-list alice/lists users.example.org
As an ordinary user, seed your own whitelist from your mailbox, seeing first what it would do:
pepsi-whitelist import --dry-run
pepsi-whitelist import
Same, but collapsing the domains you write to widely into one entry each (public hosters are never collapsed):
pepsi-whitelist import --auto-wildcard
Scan only your Sent folder, and a remote account over IMAP:
pepsi-whitelist import ~/Maildir/.Sent
pepsi-whitelist import --imap imaps://alice@mail.example.com/Sent
As root, seed every local account’s own whitelist from a Dovecot store:
pepsi-whitelist import --all-users --doveadm --auto-wildcard --yes
95.1.49.1.9. See Also¶
pepsi-config(1), pepsi-helper-mailbox-scan(1), pepsi-stage-check-whitelist(1), pepsi-stage-auto-whitelist(1), pepsi-queue(1), pepsi.conf(5), pepsi-setup(1)
95.1.49.1.10. Bugs¶
Report bugs to the Pepsi issue tracker.