8. Operating Pepsi¶
This chapter is about the weeks after installation: where the logs are, what to watch, and what to back up so that a lost disk costs you an afternoon rather than every user’s keys. Upgrading is described in Upgrading, and what to do when something goes wrong in Troubleshooting. What your users see and can do, and what the protection they get means, is the next chapter, Using Pepsi: a guide for users, written for them and free to copy into your own help pages.
8.1. Logs¶
Every Pepsi program logs to standard error. Under the shipped systemd units that is the journal, one unit per long-lived program:
journalctl -u pepsi-ingress -u pepsi-dispatch -u pepsi-httpd --since today
journalctl -u pepsi-dispatch -f # follow the pipeline live
journalctl -u 'pepsi-*' -p warning # warnings and errors, every unit
The stage programs have no units of their own: the dispatcher starts them as
workers and they inherit its standard error, so everything that a stage logs
appears under pepsi-dispatch. Each record names the program that wrote it.
The timers (key refresh, quota reconciliation, log pruning, …) log under their
own .service names; systemctl list-timers 'pepsi-*' lists them with
their last and next run.
8.1.1. Log levels¶
The levels are error, warn, info (the default), debug and
trace.
[pepsi] LOGsets the level for every program at once, the stage workers included, which take no command-line flags from the dispatcher. Restart the services after changing it (the dispatcher restarts its workers itself).-L LEVEL(before the subcommand, e.g.pepsi-queue -L debug list) overrides it for one process, which is the usual way to look closer at one tool run by hand.-valso shows the records that the database, HTTP and TLS libraries write themselves, which are hidden otherwise. This is useful for a connection problem, but very noisy.[pepsi] LOG_JSON = yeswrites one JSON object per record, for a log aggregator.
info records one line or a few per message and stage. warn keeps what
needs attention and suits a busy host. debug and trace can include
envelope addresses and protocol detail; do not leave them on. No level writes
message content, so decrypted plaintext does not end up in the journal (see
Decrypted mail, keys or subjects persisted to logs and local storage). The journal is also where
the address of a submitting client is found: a submission listener keeps it
out of the message’s Received: header (RECEIVED_PRIVACY). So if you
receive an abuse report about a message that one of your users sent, you trace
it here, by the queue id and the time that pepsi-ingress logged when it
accepted the message, not from the header.
8.1.2. What is kept, and for how long¶
The journal’s retention is systemd’s (journald.conf). Inside the database
Pepsi keeps:
the audit log (
pepsi.event_log): who changed what, from the command-line tools, the API and the console, pruned daily bypepsi-log-prune.timer;no per-message delivery log, unless you turn on
[pepsi] MAIL_LOG(see pepsi.conf(5) and the privacy warning there). A delivered message is deleted from the queue, so without it the journal is the only record that a message passed through;TLS reporting counters (
pepsi.tls_session), pruned bypepsi-tlsrpt-prune.timer;the DMARC aggregate reports that other receivers sent about the served domains (
pepsi.dmarc_report, read by pepsi-stage-dmarc-report), deleted bypepsi-queue gc(pepsi-origin-gc.timer, hourly) once their period ended more than[pepsi-dmarc-reports] RETAIN_DAYS(90) ago.
8.2. Monitoring¶
Two things tell you whether mail is flowing: the queue, and the messages that the pipeline has given up on.
8.2.1. pepsi-status¶
pepsi-status is a read-only summary of the queue, the stuck
messages with the reason for each, cumulative per-stage counters, outbound TLS
results, mailbox quotas, recent key changes, what the DMARC aggregate reports
say about the served domains, and what the Web Key Directory and the DNS serve
for every published identity. That last section is the only one that queries
the network (--offline skips it). The command is safe to run at any time, and --json
gives the same report to a script:
pepsi-status
pepsi-status --json | jq .problem_total
problem_total is the number of messages in failed or timeout. The
dispatcher never retries them; pepsi-failure-bouncer.timer bounces them to
their senders once they have been failed for MIN_AGE (an hour; see
Messages in failed or timeout), so the count normally drops back to zero by
itself. If it keeps rising, the same stage is failing message after message and
the cause needs fixing. A host fault does not show here at first: its messages
wait paused at the failing stage (pepsi_pause_backlog) with the error
in state.last_error, and are retried until the stage’s MAX_LIFETIME.
A WARNING under Published keys as served means that a correspondent who
looks that address up is given a key that the key store does not publish for it
(Checking what is actually served). The cause may be another web server that answers the
domain’s apex for the WKD direct method, a DNS record left over from a rotation,
or somebody who could write to the zone or to a web server on the domain.
Compare pepsi-keys identity list --served --address ADDRESS with
pepsi-keys dns ADDRESS and remove whatever serves the foreign key; until
then, the holder of that key can read the user’s future mail.
A WARNING under DMARC aggregate reports means that receivers saw mail
under a served domain from a source that aligned neither DKIM nor SPF, which a
p=reject policy refuses. The source may be a service that sends as the
domain (a newsletter tool, a ticket system, a hosted CRM); such a service needs
an aligned DKIM key, or SPF with an envelope sender at the domain, before you
tighten the policy. The source may also be a forwarder, or a forgery, which is
refused as intended. pepsi-setup check gives the same verdict per domain,
and the console lists every source (Logs, DMARC reports). The reports
arrive only at a domain whose DMARC record names the stage’s address in
rua=, and they are read only where a stage runs
pepsi-stage-dmarc-report. A configuration without such a stage
reads none. To change that, add a [stage-dmarc-reports] to the inbound path
(right after decrypt, before the gates, where pepsi-setup --wizard puts
it), and add rua=mailto:dmarc-reports@<domain> to each domain’s record,
which pepsi-setup run then suggests.
8.2.2. /metrics¶
pepsi-httpd serves Prometheus metrics at /metrics on a listener flagged
ADMIN = yes, and nowhere else. The endpoint takes no credential, so put the
flag only on a listener that no one but your monitoring system can reach;
The administrative API says which listeners may carry the flag. The series:
Series |
Type |
Meaning |
|---|---|---|
|
gauge |
Messages being processed right now. |
|
gauge |
Messages waiting for a retry: a remote server that deferred, a key being looked up, a payment awaited. |
|
counter |
Messages that a stage has processed. |
|
counter |
Time spent in the stage; dividing it by the previous series gives the average. |
|
counter |
Workers killed for exceeding the stage’s |
|
counter |
Workers that died while handling a message. |
|
counter |
Messages that left the pipeline. |
|
counter |
Stage passes, all stages together. |
|
gauge |
Messages that a quarantine stage holds for a decision (a part of
|
|
gauge |
Greylisting decisions in the last 24 hours: the current hour and the 23
before it. The outcomes and reasons are those of |
The counters are written by the dispatcher every STATS_INTERVAL and when it
goes idle, so they lag by up to that interval. They are totals since the schema
was created, not per scrape; use rate(). The greylisting series is a gauge
over a sliding window instead, because the daemon keeps hourly counters and the
prune drops old ones, so a total would fall. The SMTP-time milter counters of
pepsi-ingress (ingress-milter-reject and the like) are telemetry keys,
not series here.
8.2.3. Managed-PKI certificates¶
Where a domain orders its users’ certificates from a CA
(Certificates from a commercial CA: the managed PKI), pepsi-keys-mpki.timer runs a pass every five
minutes, and every step that may need you writes a notice. Three commands show
where things stand:
pepsi-keys mpki notices --unread
pepsi-keys mpki order list --open
pepsi-keys mpki user list --domain example.org
A held order waits for you; order list names the reason, and
A user’s CA certificate does not arrive says what to do about each one. A failed
order carries the CA’s message or Pepsi’s reason in last error. Mark the
notices read once they are dealt with (mpki notices --read ID or
--read-all, or in the console, Managed certificates): a notice marked read
is read for every administrator. With a
notify address for the domain (mpki domain set DOMAIN --notify ADDRESS) and
[pepsi-mpki] NOTIFY_STAGE, each pass also mails that domain’s new notices.
8.2.4. Quarantined mail¶
A message that a milter quarantines, or that another stage routes to a
pepsi-stage-quarantine stage, waits there until you decide.
Pepsi tells nobody: the sender does not learn that the message is held, and
the recipient does not see it. pepsi-status counts the held messages under the
queue table (quarantined in --json), and pepsi-queue lists and acts
on them by token:
pepsi-queue quarantine list
pepsi-queue quarantine show TOKEN
pepsi-queue --headers --body WORKQUEUE_ID > held.eml
pepsi-queue quarantine release TOKEN
pepsi-queue quarantine delete TOKEN
list shows when each message arrived, since when it is held, its envelope,
and the by and reason of its record: who sent it to the quarantine
(milter:<name> for a filter) and why. Look at the content with list
--headers --body and the workqueue_id that show prints. release
sends the message on to the stage’s RELEASE_STAGE, which is normally
init for mail that ingress quarantined, so the message then takes the whole
pipeline. delete asks first (--yes in a script) and records the
deletion in the opt-in mail log. Both are in the audit log
(queue.quarantine-release, queue.quarantine-delete).
A hold does not last for ever. With MAX_HOLD the stage ends its own holds
and sends the message to EXPIRE_STAGE, or deletes it; without it,
pepsi-queue gc bounces a message that is still held
[pepsi-queue] QUARANTINE_MAX_AGE_DAYS (14) days after it arrived, without
its content. Held mail is plaintext in the database for as long as it is held.
pepsi-queue clear-all leaves held mail alone, and so does the payment
webhook POST /resume.
The console’s Quarantine page and /api/v1/quarantine list, release and
delete held mail the same way, with the same audit kinds (queue:read to
look, queue:write to act; Quarantine), and
pepsi_quarantine_held_messages on /metrics counts it.
8.2.5. Greylisting¶
Where pepsi-greylist runs, pepsi-status has a
Greylisting section: whether the daemon answers on its socket, and the last
24 hours by outcome. pepsi-greylist stats prints the same for any number
of hours, and --json for a script:
pepsi-greylist stats --hours 168
pepsi-greylist list --pending --rcpt alice@example.org
pepsi-greylist list --never-retried --domain example.net
Deferred counts first attempts; passed on retry and passed from another
host the retries that came back; never retried the first attempts whose
retry window ended without a retry. A high share of never-retried attempts is
what greylisting is for, unless they come from a real correspondent: a domain
that appears there with an spf or ptr identity is a real mail system
that does not retry, and its mail is lost to greylisting. Let it through:
pepsi-greylist allow domain example.net --note 'does not retry'
pepsi-greylist allow network 198.51.100.0/24 --expires '90 days'
pepsi-greylist rules
pepsi-greylist unrule 7
A domain rule also matches the envelope sender’s domain, which anybody can
write; for a sender without SPF, a network rule is the narrower choice. A
user can stop the delay for one correspondent by whitelisting them, which the
daemon reads (WHITELIST_BYPASS). Every rule change is in the audit log
(greylist.rule.added, greylist.rule.removed).
Errors are recipients let through because the database or the DNS did not answer in time. A few during a database restart are expected; a steady stream means greylisting is mostly off.
The console’s Greylisting page (Greylisting) shows the same
counters, the triplets with the same filters and the rules, and with
config:write it adds and removes rules (origin = console). Users signed
in for their own address see what was deferred to them on Deferred mail, and
can put a sender they know on their own whitelist there, which the daemon then
reads (When the first message from someone new arrives late). /metrics has the counters of the last
24 hours as pepsi_greylist_decisions_24h.
8.2.6. What to alert on¶
Condition |
Why |
|---|---|
|
A message has failed and has not been bounced:
|
|
A host fault at that stage (an unreadable key or template, a helper
that cannot start, a refused statement). Its messages are held, not
bounced, until |
|
Mail waits at a quarantine stage for a decision (see Quarantined mail). Nobody else is told, and the hold ends with a bounce or a deletion when nobody decides. |
The dispatcher logging “the worker refused to start” for a stage |
The stage’s configuration or a secret that it reads does not parse; its messages are held until it does. |
|
The next hop, DNS or outbound port 25 is unreachable. Retries continue
until |
|
A stage program is failing; its messages end as |
|
Nothing leaves the pipeline: the dispatcher is down or a stage is stuck. |
|
A key that the store does not publish is handed out for that address, so whoever holds that key can read future mail to the address (see above). Run the check on the monitoring host’s schedule, not every minute: each run fetches every published identity. |
A |
The long-running units retry forever, so a failed one is usually a timer or a one-shot job. |
|
The DNSSEC chain of trust of a served domain is broken, for example by
an expired signature, or by a DS record at the parent that matches no
published DNSKEY after a key roll. Every validating resolver then refuses the domain:
senders behind one cannot find its MX, and senders using DANE defer
mail to it. |
|
The daily |
A “DKIM: DNS changes needed” mail from |
A DKIM record is missing (receivers cannot verify mail signed with
that key), a rotation waits for its records, or an RFC 2136 update was
refused. |
|
The signing stages cannot sign; outbound mail waits in the queue (it is never sent unsigned) until the daemon is back. |
|
Mail is accepted without greylisting (an MTA treats the missing milter
as a failure, and |
|
The daemon cannot reach its database or its resolver in time
( |
A domain with an |
A real mail system that does not retry: its mail never arrives. Allow it (see above). |
The never-retried share well above its usual level:
|
A correspondent that sends much mail has stopped retrying, or started
sending from addresses that the identity ladder cannot tie together.
|
|
The same as the |
Journal records at |
Anything logged as an error needs looking at. |
The TLS certificates’ expiry |
|
Free space where PostgreSQL keeps its data |
Below |
Unread managed-PKI notices of severity |
A user’s certificate waits for you (a name to confirm, the daily
limit, a missing licence or credential) or could not be ordered. The
notice names the order and the command that releases it; see
A user’s CA certificate does not arrive. With a notify address and |
|
A pass of the managed PKI met an error ( |
An |
A login came back under another uid, or after it was removed: its
previous owner’s settings, whitelists and quota were dropped, and
|
8.3. Changing the key behind a TLSA record¶
The DANE record that pepsi-setup run prints for each TLS ingress listener
(_<port>._tcp.<HOSTNAME> IN TLSA 3 1 1 <hash>) pins the mail certificate’s
public key, not the certificate. pepsi-setup therefore obtains certbot
certificates with --reuse-key and keeps reuse_key = True in their
renewal configuration, so an ordinary renewal never invalidates the record.
A deliberate key change (a suspected compromise, a move to another key type
or size) is different: a sender that enforces DANE refuses to deliver to a host
whose certificate matches no published record, and resolvers keep serving the
old record set for its TTL. Pepsi has no rollover workflow of its own. You make
the change by hand, and publish the record of the next key before the key is
used:
Create the next key and its certificate beside the current ones, as a separate certbot lineage that the services do not use yet (same host name, same challenge method as the original, here the standalone one):
certbot certonly --cert-name mx.example.org-next -d mx.example.org \ --standalone --reuse-key
Publish the next key’s record alongside the current one, under every name at which the current record is published (one per TLS port). Its hash is the SHA-256 of the new key’s
SubjectPublicKeyInfo:openssl x509 -in /etc/letsencrypt/live/mx.example.org-next/cert.pem \ -noout -pubkey | openssl pkey -pubin -outform DER \ | openssl dgst -sha256 -hex
_25._tcp.mx.example.org. IN TLSA 3 1 1 <current hash> _25._tcp.mx.example.org. IN TLSA 3 1 1 <next hash>
Both records are valid at once: a sender accepts a certificate that matches either, so the current certificate keeps working.
Wait out the TTL of the TLSA record set (and let the signed zone reach every authoritative server). Until then a resolver may still hold the old set, which does not yet name the next key.
Switch to the new key. Point every section that names the old lineage (
TLS_CERT/TLS_KEYof the ingress listeners) at/etc/letsencrypt/live/mx.example.org-next/. Thepepsi-httpdlisteners have a lineage of their own,mx.example.org-web, which noTLSArecord pins. Then runpepsi-setup -c /etc/pepsi/pepsi.conf run(which keepsreuse_keyset for the new lineage and hands the certificate to the services) and restart the services.pepsi-setup checkthen reports the published records against the certificate that is now served.Remove the old record, and the old lineage with
certbot delete --cert-name mx.example.org. From here on the new lineage renews with its key kept, like the old one did.
8.4. Rotating DKIM keys¶
The DKIM and ARC private keys belong to the signing daemon,
pepsi-dkim (pepsi-dkim.service, account pepsi-dkim).
Each signing domain has a current generation of keys, one per algorithm of
[pepsi] DKIM_ALGORITHMS (the ARC_DOMAIN also one of ARC_ALGORITHM),
under selectors named after the month in which the generation was made:
pepsi-202610._domainkey.example.org for RSA and
pepsi-202610-ed25519._domainkey.example.org for Ed25519, with pepsi
being [pepsi] DKIM_SELECTOR. A rotation makes a next generation under
new selectors, switches signing to it once its records are published, keeps
the old records for RETIRE_AFTER_DAYS (14 by default) so that mail signed just
before the switch still verifies, and then retires the old keys.
8.4.1. Seeing what DNS needs¶
The daemon checks every authoritative server of each zone, hourly
(DNS_CHECK_INTERVAL), for the record of every key that has to be in DNS,
and reports the result everywhere:
pepsi-dkim statuslists every key with its state (next,current,retiring,retired) and its DNS verdict (visible,missing,disagreewhile a secondary lags,unknown);pepsi-dkim recordsprints the records to publish as zone-file lines and the removable ones as comments;pepsi-setup runprints the missing records with the rest of the zone, andpepsi-setup checkgrades each published record, including one that a DNS host has truncated (a 3072- or 4096-bit key needs a TXT record split into several strings, which some hosting panels mangle);pepsi-status (“DKIM keys”) and the console’s DKIM page (
/ui/dkim) show the same;the daemon mails the postmaster (
NOTIFY,NOTIFY_ADDRESS) at most once a day while a record has to be added, a record of a retired key may be removed, a rotation waits for its records, or an RFC 2136 update was refused. Each mail names the records and repeats daily until the change is made.
A domain that has no keys yet gets a generation at once, and it signs at once.
This happens on a fresh installation, for a domain added to the configuration,
and on a host restored without the daemon’s state directory. Until its records are published,
receivers cannot verify those signatures and DMARC rests on SPF alone, so
publish what pepsi-setup run prints right away.
8.4.2. Scheduled rotation¶
Rotation is off by default (ROTATION_DAYS = 0): a generation signs
until somebody runs pepsi-dkim rotate. To rotate on a schedule, set, for
example, the following (M3AAWG advises rotating at least every six months):
[pepsi-dkim]
ROTATION_DAYS = 180
# made this long before the switch, to give time to publish (default 28)
ROTATION_LEAD_DAYS = 28
and restart pepsi-dkim.service. ROTATION_LEAD_DAYS before a
generation has signed for ROTATION_DAYS, the daemon makes the next
generation and asks for its records. With WAIT_FOR_DNS = yes (the default)
the switch waits until every authoritative server serves every record of it;
the old keys go on signing meanwhile, and the delay is reported and mailed.
WAIT_FOR_DNS = no switches on schedule whatever DNS serves, which is right
only where something else guarantees that the records are there.
RETIRE_AFTER_DAYS must exceed the SIGNATURE_EXPIRATION_DAYS of every
pepsi-stage-dkim-sign and pepsi-stage-arc stage (seven days by
default); pepsi-setup validate checks this. With
SIGNATURE_EXPIRATION_DAYS = 0 the signatures carry no expiry, and RETIRE_AFTER_DAYS should cover
the delivery stages’ MAX_LIFETIME plus a few days for the receivers’ own
queues. After it, the old private keys are deleted and the old records may be
removed.
[pepsi-dkim] RSA_BITS (2048, 3072 or 4096; default 2048) sets the size of
the RSA keys that the next generation gets; existing keys keep theirs.
8.4.3. Rotating on demand¶
pepsi-dkim -c /etc/pepsi/pepsi.conf rotate example.org
pepsi-dkim -c /etc/pepsi/pepsi.conf rotate --hard example.org
Run as root (or pepsi-dkim). Without a domain, every signing domain
rotates. The first form makes a next generation and switches as soon as its
records are visible (or at once under WAIT_FOR_DNS = no); publish what
pepsi-dkim records prints. --hard switches at once, before anybody can
have published the new records, so mail signed with the new keys fails DKIM
until the records are published. It is meant for a key that may have been
exposed
(After a key exposure). The console’s DKIM page offers both as rotate
now; the button queues a dkim-rotate task that the root applier
(pepsi-setup apply) runs, so it needs setup:write and a running
applier.
8.4.4. Publishing the records automatically¶
Where the zone accepts dynamic updates, the daemon can add the records of its
next and current keys and remove those of retired keys itself, by RFC
2136 UPDATE authenticated with TSIG:
[pepsi-dkim]
DNS_UPDATE = rfc2136
TSIG_KEY_NAME = pepsi-dkim.example.org.
TSIG_ALGORITHM = hmac-sha256
TSIG_SECRET = <base64 of the key>
# default: the zone's primary, the SOA MNAME
DNS_UPDATE_SERVER = ns1.example.org:53
pepsi-setup moves TSIG_SECRET into
/etc/pepsi/secrets.d/pepsi-dkim.secret, readable by pepsi-dkim alone.
Grant the key on the DNS server nothing but updates of TXT records under
_domainkey (for BIND, an update-policy with subdomain or
wildcard rules limited to TXT): whoever holds the TSIG key can change
whatever the key is allowed to change. A server that cannot be reached is tried again a
minute later. A refusal (a wrong key, a policy that forbids the name) is
recorded on the key, mailed to the postmaster and not retried until
pepsi-dkim.service restarts; fix the settings or make the change by hand,
then restart it.
8.4.5. Publishing retired keys¶
A DKIM signature lets anybody holding a copy of a message prove to a third
party that the domain sent it, long after delivery
(DKIM signatures make stolen mail verifiably attributable). With [pepsi-dkim] PUBLISH_RETIRED_KEYS =
yes a retired private key is not deleted but written to
/var/lib/pepsi-dkim-public/retired/<domain>/<selector>.pem, and
pepsi-httpd serves it at
https://<host>/.well-known/pepsi-dkim-retired/<domain>/<selector>.pem. Once
anybody can make such a signature, an old one proves nothing. The feature
needs rotation, since keys retire only by rotating, and a public HTTPS
listener; pepsi-setup validate warns if either is missing.
A retired key is published only once every authoritative server has stopped
serving its records; until then the daemon keeps it unpublished, because
anybody could sign as the domain with a published key whose record still
resolves. So remove the records when the daemon reports them removable
(DNS_UPDATE = rfc2136 does that itself). A key whose records are never
removed is deleted, not published, 90 days after it retired.
Note
Resolvers may still hold a removed record until its TTL runs out, so for a few hours after publication a signature made with the published key can still verify at some receivers. Keep the TTL of the selector records short where this matters.
8.5. Rotating the SRS key¶
The SRS secret ([pepsi-srs] SECRET, in
/etc/pepsi/secrets.d/pepsi-ingress.secret) signs every rewritten envelope
sender, and a bounce returns to that address up to MAX_AGE_DAYS (default
21) later; the same holds for the forward return address of a forwarded copy,
whose keys are derived from the secret. Replacing the secret outright would
make every address still in circulation fail verification, and their bounces would be refused. The old
secret is therefore kept for verification only, in [pepsi-srs]
PREVIOUS_SECRET_FILE, until the last address it signed has expired:
Move the current secret to its own file, readable by root only until step 3, and point
PREVIOUS_SECRET_FILEat it:install -m 0600 /dev/null /etc/pepsi/secrets.d/srs-previous.secret sed -n 's/^SECRET *= *//p' /etc/pepsi/secrets.d/pepsi-ingress.secret \ > /etc/pepsi/secrets.d/srs-previous.secret
[pepsi-srs] PREVIOUS_SECRET_FILE = /etc/pepsi/secrets.d/srs-previous.secret
Write a new secret into the
[pepsi-srs]section ofsecrets.d/pepsi-ingress.secret(SECRET = `` followed by, for example, the output of ``openssl rand -hex 32), leaving every other line of the file as it is.Run
pepsi-setup -c /etc/pepsi/pepsi.conf run. It gives the previous secret’s file the ownership and mode of the current one (pepsi-ingress,0640) and checks that both can be read.Restart
pepsi-ingressright away, because it reads the secrets only once, at start.pepsi-stage-srsreads them for every message, so from that moment on it signs with the new secret. Returning addresses verify under either secret.After
MAX_AGE_DAYSdays, removePREVIOUS_SECRET_FILEand the file, and restartpepsi-ingressagain.pepsi-setup runandpepsi-setup checksay when that time has come: they warn about a previous-secret file last changed more thanMAX_AGE_DAYSago.
8.6. After a key exposure¶
A key may have left your control, for example through a stolen laptop with a mail client’s key on it, a backup that went astray, or a compromised stage. Such a key stays valid until somebody says otherwise; nothing expires it, and correspondents keep using what they cached. The steps below depend on what was exposed. Each one is idempotent, so when in doubt, do them all:
A user’s or the gateway’s OpenPGP key (an identity in the key store): revoke it and publish the revocation, in that order of importance:
pepsi-keys -c /etc/pepsi/pepsi.conf identity revoke ID --reason 'laptop stolen' --upload
Every revocation, whether by this command, by the user’s
retiree-mail command or in the console, builds a revocation certificate while the private key still exists, and stores it as the key’s public half. From then on, the Web Key Directory serves the key as revoked. If the key had been uploaded to a key server, the revocation is uploaded there too, because a key server cannot be told to forget a key; this command does that at once, otherwisepepsi-keys identity publish --retrydoes it.--uploadalso sends the revocation to a key server that never had the key. If the certificate cannot be made, nothing is revoked and the command can be re-run. The revoked key’s public half is kept so that old signatures still verify, and the next message from the address gets a fresh key under the usual policy (Key management). Correspondents learn of the revocation through the key server and the Web Key Directory when they refresh, and through the Autocrypt header of the user’s next mail. A correspondent who refreshes nothing keeps encrypting to the old key, and there is no remedy for that. A user’s own MUA key is revoked here only locally: its private half, and so its revocation certificate, is in their client.A user’s S/MIME certificate: ask the issuing CA to revoke it (Pepsi cannot; the CA’s CRL is what correspondents that check will see), then revoke the identity as above (no revocation certificate exists for S/MIME, and
--uploadhas no meaning) so this host stops signing with it.A correspondent’s key that you learnt is exposed (they told you, or their revocation never reached you): remove the cached copy, and the message in hand is then handled as for a correspondent without a key, under the encrypt stage’s
ON_NO_KEY:pepsi-keys -c /etc/pepsi/pepsi.conf peer list --address ADDRESS pepsi-keys -c /etc/pepsi/pepsi.conf peer remove PEER-KEY-ID
A CA whose key is exposed is disabled or removed the same way (
pepsi-keys ca disable CA-ID,ca remove CA-ID), which stops every certificate that it issued from verifying here. An S/MIME certificate that its CA has listed is retired automatically oncepepsi-keydiscfetches that CRL (pepsi-keydisc).The key-encryption key (
secrets.d/pepsi-crypto.secret) or thepepsi-cryptorole’s process: every MTA private key was readable. Rotate the KEK (pepsi-keys wrap rotate), then revoke and re-create each affected identity as in step 1; the rotation alone re-wraps keys that an attacker may already hold in the clear.The TLS key of a listener: obtain a certificate for a new key and publish its TLSA record first, as Changing the key behind a TLSA record describes, then have the old certificate revoked by its CA (
certbot revoke --cert-name mx.example.org --reason keycompromisefor a certbot lineage). Peers that validate by DANE stop accepting the old key once the old record is gone. Peers that validate by PKIX see the revocation only if they check; Pepsi itself checks a stapled OCSP response and refuses a revoked certificate on a validating session, but fetches no status. The web server’s key (themx.example.org-weblineage ofpepsi-httpd) is a separate key that noTLSArecord pins: replace it withcertbot renew --cert-name mx.example.org-web --force-renewal --new-keyand revoke the old certificate; the mail key is not affected. Its certificate does nameHOSTNAME, so until the revocation reaches them, senders and mail clients that authenticate the MX by PKIX alone can be deceived with it.A DKIM or ARC key, or the
pepsi-dkimaccount: rotate at once withpepsi-dkim rotate --hard DOMAIN(Rotating DKIM keys), publish the new records thatpepsi-dkim recordsprints, and remove the old selector’s records right away instead of waitingRETIRE_AFTER_DAYS. Mail signed before the switch then fails DKIM on the way, which costs less than letting the holder of the key sign as the domain. Do not publish an exposed key withPUBLISH_RETIRED_KEYSbefore its records are gone. A compromised stage running aspepsicould not read the keys, but could have had mail signed while it ran: the daemon’s log names every signature and the requesting process.
Whatever was exposed, the audit log (GET /api/v1/events; pepsi-keys
and pepsi-queue write to it) shows what the key was used for after the
suspected time, and A login that changes owner what a recycled login loses.
8.7. Backup and restore¶
Warning
A database dump alone is not a backup of Pepsi. The private keys in the
database are encrypted under a key-encryption key that is kept, on purpose,
outside the database, in /etc/pepsi/secrets.d/pepsi-crypto.secret. Lose
that file and every user’s stored private keys are lost with it: nothing can
open them again, and pepsi-setup will not generate a replacement while
encrypted keys exist (see Key management too hard: end-to-end mail clients error-prone or unused). The custody
boundary that this split upholds is described in The private-key custody boundary, and what a copy
of the database alone does and does not yield in Somebody holding a copy of the database.
8.7.1. What to back up¶
What |
What is lost without it |
|---|---|
|
|
The PostgreSQL database ( |
The queue, per-user settings, whitelists, mailing lists and their archive, the key store, quotas, administrator accounts and API tokens, the audit log. |
|
The configuration. It can be written again with the wizard, but not quickly. |
|
Leave it out. It holds the DKIM and ARC private keys, which do not
belong in a backup: a copy is one more place they can leak from. After
a restore without it, the daemon generates fresh keys under new
selectors and signs with them at once, which is a rotation; publish the
records |
|
With |
|
Nothing permanent; certbot obtains new certificates. Keep it anyway if you publish DANE (TLSA) records for the current key. |
|
Only with an OAuth or client-certificate smarthost: the refresh token has to be authorised again. |
The users’ mailboxes (Maildir, or the MDA’s store behind LMTP) are not
Pepsi’s and belong in your ordinary backup of home directories.
Keep the files and the database dump together, and from the same moment: a
dump newer than secrets.d may hold keys wrapped under a key-encryption key
that the file backup does not have. Both hold secrets; store them encrypted and
readable by root only. The database dump also holds queued messages, which are
plaintext for mail that Pepsi decrypted (see
Decrypted mail, keys or subjects persisted to logs and local storage).
8.7.2. Taking a backup¶
sudo -u pepsi-owner pg_dump --format=custom pepsi > pepsi-$(date +%F).dump
tar -C / -czpf pepsi-files-$(date +%F).tar.gz \
etc/pepsi var/pepsi var/lib/pepsi-wallets
Run as root, so that tar can read secrets.d. A [pepsi-mpki]
CREDENTIAL_DIR outside /etc/pepsi needs adding to the tar command.
pg_dump takes a
consistent snapshot while the services run; the files change rarely (a new key,
a configuration change), so taking them just before or after the dump is
enough. The database name and its owner are those in [pepsi-postgres].
The dump includes the known-compromised-key list (pepsi.compromised_key,
about 240 MB). It is the one large table a restore could do without: leaving
it out (--exclude-table-data=pepsi.compromised_key) keeps the dump small,
and after a restore pepsi-keys blocklist update --force fetches the list
again. The moduli that pepsi-keys peer screen factored (source screen)
are then lost, however, until the next screen finds them again.
The dump that pepsi-setup schema --backup-dir takes before an upgrade (see
Backing up before a schema upgrade) covers the database only, and only the pepsi
schema, and only the two newest such dumps are kept; it does not replace this.
8.7.3. Restoring¶
Restore on a host running the same Pepsi release that the backup came from (upgrade afterwards, if you want to), in this order:
Install Pepsi. The Debian package creates the service accounts and database roles; on a source install create them as in Installation.
Stop everything:
systemctl stop pepsi.target.Restore the files, secrets.d first.
tarkeeps owners by name, so the accounts must exist before you unpack:tar -C / -xzpf pepsi-files-2026-10-01.tar.gzRestore the database into an empty one owned by the schema owner:
sudo -u postgres dropdb --if-exists pepsi sudo -u postgres createdb -O pepsi-owner pepsi sudo -u pepsi-owner pg_restore -d pepsi pepsi-2026-10-01.dump
Run
pepsi-setup -c /etc/pepsi/pepsi.conf run. It re-applies the role grants, writes again what lives outside both backups (the systemd drop-ins that hand the TLS certificates to the services) and haspepsi-dkimmake fresh DKIM and ARC keys, since those are not in the backup. It does not replace a key or secret that already exists. Publish the DKIM records it prints (new selectors) and remove the old selectors’ records.Start the services:
systemctl start pepsi.target.
If pepsi-setup says it refuses to generate a key-encryption key because
identities exist, secrets.d/pepsi-crypto.secret was not restored, or is not
the one that goes with this database. Do not work around it: find the right
file.
8.7.4. Cloned and restored virtual machines¶
Every key, nonce and secret that Pepsi makes comes from the operating system’s
random number generator. On a virtual machine that generator’s state is part of
every snapshot: two instances started from one snapshot, or one instance
restored twice, can draw the same “random” bytes until something reseeds them.
Under ENABLE_PEP one host generates every user’s long-term keys, and
pepsi-setup generates the key-encryption key, so a repeated state would
make all of them predictable at once. Waiting for the generator to be seeded
does not help: after a clone it is seeded, with the state of the original.
Reboot a cloned or restored instance before it generates anything, and before it starts the Pepsi services: a fresh boot reseeds the kernel’s generator, and processes started afterwards draw from it. Restoring a backup onto a newly installed host does not have this problem; restoring a whole virtual machine image does.
Let the hypervisor make it automatic. A hypervisor that exposes a VM generation ID (QEMU/KVM
-device vmgenid, Hyper-V, VMware) lets the Linuxvmgeniddriver notice a clone or a restore and reseed the kernel’s generator at once. Check that the device is present (ls /sys/bus/acpi/devices/ | grep -i -e VMGENID -e QEMUVGID) and turn it on where it is not.Pepsi warns about what it generated just after a change. The driver announces the change with a udev event, on which Pepsi’s udev rule (
60-pepsi.rules) touches/run/pepsi-vmgenid-changed. Generating an OpenPGP or S/MIME key (pepsi-keys, the encrypt stage underENABLE_PEP), a one-time-password secret, the key-encryption key or a new wrapping of the private keys within ten minutes of that logs a warning naming what was made. If the instance was not rebooted since the change, reboot it, then revoke and regenerate what the warnings name (After a key exposure). Without the driver, or without the rule, nothing is recorded and nothing is warned about.