8. Operating Pepsi

This chapter is about the weeks after installation: where the logs are, what to watch, and what to back up so that a lost disk costs you an afternoon rather than every user’s keys. Upgrading is described in Upgrading, and what to do when something goes wrong in Troubleshooting. What your users see and can do, and what the protection they get means, is the next chapter, Using Pepsi: a guide for users, written for them and free to copy into your own help pages.

8.1. Logs

Every Pepsi program logs to standard error. Under the shipped systemd units that is the journal, one unit per long-lived program:

journalctl -u pepsi-ingress -u pepsi-dispatch -u pepsi-httpd --since today
journalctl -u pepsi-dispatch -f          # follow the pipeline live
journalctl -u 'pepsi-*' -p warning       # warnings and errors, every unit

The stage programs have no units of their own: the dispatcher starts them as workers and they inherit its standard error, so everything that a stage logs appears under pepsi-dispatch. Each record names the program that wrote it. The timers (key refresh, quota reconciliation, log pruning, …) log under their own .service names; systemctl list-timers 'pepsi-*' lists them with their last and next run.

8.1.1. Log levels

The levels are error, warn, info (the default), debug and trace.

  • [pepsi] LOG sets the level for every program at once, the stage workers included, which take no command-line flags from the dispatcher. Restart the services after changing it (the dispatcher restarts its workers itself).

  • -L LEVEL (before the subcommand, e.g. pepsi-queue -L debug list) overrides it for one process, which is the usual way to look closer at one tool run by hand.

  • -v also shows the records that the database, HTTP and TLS libraries write themselves, which are hidden otherwise. This is useful for a connection problem, but very noisy.

  • [pepsi] LOG_JSON = yes writes one JSON object per record, for a log aggregator.

info records one line or a few per message and stage. warn keeps what needs attention and suits a busy host. debug and trace can include envelope addresses and protocol detail; do not leave them on. No level writes message content, so decrypted plaintext does not end up in the journal (see Decrypted mail, keys or subjects persisted to logs and local storage). The journal is also where the address of a submitting client is found: a submission listener keeps it out of the message’s Received: header (RECEIVED_PRIVACY). So if you receive an abuse report about a message that one of your users sent, you trace it here, by the queue id and the time that pepsi-ingress logged when it accepted the message, not from the header.

8.1.2. What is kept, and for how long

The journal’s retention is systemd’s (journald.conf). Inside the database Pepsi keeps:

  • the audit log (pepsi.event_log): who changed what, from the command-line tools, the API and the console, pruned daily by pepsi-log-prune.timer;

  • no per-message delivery log, unless you turn on [pepsi] MAIL_LOG (see pepsi.conf(5) and the privacy warning there). A delivered message is deleted from the queue, so without it the journal is the only record that a message passed through;

  • TLS reporting counters (pepsi.tls_session), pruned by pepsi-tlsrpt-prune.timer;

  • the DMARC aggregate reports that other receivers sent about the served domains (pepsi.dmarc_report, read by pepsi-stage-dmarc-report), deleted by pepsi-queue gc (pepsi-origin-gc.timer, hourly) once their period ended more than [pepsi-dmarc-reports] RETAIN_DAYS (90) ago.

8.2. Monitoring

Two things tell you whether mail is flowing: the queue, and the messages that the pipeline has given up on.

8.2.1. pepsi-status

pepsi-status is a read-only summary of the queue, the stuck messages with the reason for each, cumulative per-stage counters, outbound TLS results, mailbox quotas, recent key changes, what the DMARC aggregate reports say about the served domains, and what the Web Key Directory and the DNS serve for every published identity. That last section is the only one that queries the network (--offline skips it). The command is safe to run at any time, and --json gives the same report to a script:

pepsi-status
pepsi-status --json | jq .problem_total

problem_total is the number of messages in failed or timeout. The dispatcher never retries them; pepsi-failure-bouncer.timer bounces them to their senders once they have been failed for MIN_AGE (an hour; see Messages in failed or timeout), so the count normally drops back to zero by itself. If it keeps rising, the same stage is failing message after message and the cause needs fixing. A host fault does not show here at first: its messages wait paused at the failing stage (pepsi_pause_backlog) with the error in state.last_error, and are retried until the stage’s MAX_LIFETIME.

A WARNING under Published keys as served means that a correspondent who looks that address up is given a key that the key store does not publish for it (Checking what is actually served). The cause may be another web server that answers the domain’s apex for the WKD direct method, a DNS record left over from a rotation, or somebody who could write to the zone or to a web server on the domain. Compare pepsi-keys identity list --served --address ADDRESS with pepsi-keys dns ADDRESS and remove whatever serves the foreign key; until then, the holder of that key can read the user’s future mail.

A WARNING under DMARC aggregate reports means that receivers saw mail under a served domain from a source that aligned neither DKIM nor SPF, which a p=reject policy refuses. The source may be a service that sends as the domain (a newsletter tool, a ticket system, a hosted CRM); such a service needs an aligned DKIM key, or SPF with an envelope sender at the domain, before you tighten the policy. The source may also be a forwarder, or a forgery, which is refused as intended. pepsi-setup check gives the same verdict per domain, and the console lists every source (Logs, DMARC reports). The reports arrive only at a domain whose DMARC record names the stage’s address in rua=, and they are read only where a stage runs pepsi-stage-dmarc-report. A configuration without such a stage reads none. To change that, add a [stage-dmarc-reports] to the inbound path (right after decrypt, before the gates, where pepsi-setup --wizard puts it), and add rua=mailto:dmarc-reports@<domain> to each domain’s record, which pepsi-setup run then suggests.

8.2.2. /metrics

pepsi-httpd serves Prometheus metrics at /metrics on a listener flagged ADMIN = yes, and nowhere else. The endpoint takes no credential, so put the flag only on a listener that no one but your monitoring system can reach; The administrative API says which listeners may carry the flag. The series:

Series

Type

Meaning

pepsi_stage_active_messages{stage}

gauge

Messages being processed right now.

pepsi_pause_backlog{stage}

gauge

Messages waiting for a retry: a remote server that deferred, a key being looked up, a payment awaited.

pepsi_stage_messages_total{stage}

counter

Messages that a stage has processed.

pepsi_stage_duration_seconds_total{stage}

counter

Time spent in the stage; dividing it by the previous series gives the average.

pepsi_stage_timeouts_total{stage}

counter

Workers killed for exceeding the stage’s MAX_RUNTIME.

pepsi_stage_crashes_total{stage}

counter

Workers that died while handling a message.

pepsi_messages_processed_total

counter

Messages that left the pipeline.

pepsi_stages_executed_total

counter

Stage passes, all stages together.

pepsi_quarantine_held_messages{stage}

gauge

Messages that a quarantine stage holds for a decision (a part of pepsi_pause_backlog); see Quarantined mail.

pepsi_greylist_decisions_24h{outcome,reason}

gauge

Greylisting decisions in the last 24 hours: the current hour and the 23 before it. The outcomes and reasons are those of pepsi-greylist stats --json; outcome="expired",reason="never-retried" counts the first attempts that the prune removed because no retry came.

The counters are written by the dispatcher every STATS_INTERVAL and when it goes idle, so they lag by up to that interval. They are totals since the schema was created, not per scrape; use rate(). The greylisting series is a gauge over a sliding window instead, because the daemon keeps hourly counters and the prune drops old ones, so a total would fall. The SMTP-time milter counters of pepsi-ingress (ingress-milter-reject and the like) are telemetry keys, not series here.

8.2.3. Managed-PKI certificates

Where a domain orders its users’ certificates from a CA (Certificates from a commercial CA: the managed PKI), pepsi-keys-mpki.timer runs a pass every five minutes, and every step that may need you writes a notice. Three commands show where things stand:

pepsi-keys mpki notices --unread
pepsi-keys mpki order list --open
pepsi-keys mpki user list --domain example.org

A held order waits for you; order list names the reason, and A user’s CA certificate does not arrive says what to do about each one. A failed order carries the CA’s message or Pepsi’s reason in last error. Mark the notices read once they are dealt with (mpki notices --read ID or --read-all, or in the console, Managed certificates): a notice marked read is read for every administrator. With a notify address for the domain (mpki domain set DOMAIN --notify ADDRESS) and [pepsi-mpki] NOTIFY_STAGE, each pass also mails that domain’s new notices.

8.2.4. Quarantined mail

A message that a milter quarantines, or that another stage routes to a pepsi-stage-quarantine stage, waits there until you decide. Pepsi tells nobody: the sender does not learn that the message is held, and the recipient does not see it. pepsi-status counts the held messages under the queue table (quarantined in --json), and pepsi-queue lists and acts on them by token:

pepsi-queue quarantine list
pepsi-queue quarantine show TOKEN
pepsi-queue --headers --body WORKQUEUE_ID > held.eml
pepsi-queue quarantine release TOKEN
pepsi-queue quarantine delete TOKEN

list shows when each message arrived, since when it is held, its envelope, and the by and reason of its record: who sent it to the quarantine (milter:<name> for a filter) and why. Look at the content with list --headers --body and the workqueue_id that show prints. release sends the message on to the stage’s RELEASE_STAGE, which is normally init for mail that ingress quarantined, so the message then takes the whole pipeline. delete asks first (--yes in a script) and records the deletion in the opt-in mail log. Both are in the audit log (queue.quarantine-release, queue.quarantine-delete).

A hold does not last for ever. With MAX_HOLD the stage ends its own holds and sends the message to EXPIRE_STAGE, or deletes it; without it, pepsi-queue gc bounces a message that is still held [pepsi-queue] QUARANTINE_MAX_AGE_DAYS (14) days after it arrived, without its content. Held mail is plaintext in the database for as long as it is held. pepsi-queue clear-all leaves held mail alone, and so does the payment webhook POST /resume.

The console’s Quarantine page and /api/v1/quarantine list, release and delete held mail the same way, with the same audit kinds (queue:read to look, queue:write to act; Quarantine), and pepsi_quarantine_held_messages on /metrics counts it.

8.2.5. Greylisting

Where pepsi-greylist runs, pepsi-status has a Greylisting section: whether the daemon answers on its socket, and the last 24 hours by outcome. pepsi-greylist stats prints the same for any number of hours, and --json for a script:

pepsi-greylist stats --hours 168
pepsi-greylist list --pending --rcpt alice@example.org
pepsi-greylist list --never-retried --domain example.net

Deferred counts first attempts; passed on retry and passed from another host the retries that came back; never retried the first attempts whose retry window ended without a retry. A high share of never-retried attempts is what greylisting is for, unless they come from a real correspondent: a domain that appears there with an spf or ptr identity is a real mail system that does not retry, and its mail is lost to greylisting. Let it through:

pepsi-greylist allow domain example.net --note 'does not retry'
pepsi-greylist allow network 198.51.100.0/24 --expires '90 days'
pepsi-greylist rules
pepsi-greylist unrule 7

A domain rule also matches the envelope sender’s domain, which anybody can write; for a sender without SPF, a network rule is the narrower choice. A user can stop the delay for one correspondent by whitelisting them, which the daemon reads (WHITELIST_BYPASS). Every rule change is in the audit log (greylist.rule.added, greylist.rule.removed).

Errors are recipients let through because the database or the DNS did not answer in time. A few during a database restart are expected; a steady stream means greylisting is mostly off.

The console’s Greylisting page (Greylisting) shows the same counters, the triplets with the same filters and the rules, and with config:write it adds and removes rules (origin = console). Users signed in for their own address see what was deferred to them on Deferred mail, and can put a sender they know on their own whitelist there, which the daemon then reads (When the first message from someone new arrives late). /metrics has the counters of the last 24 hours as pepsi_greylist_decisions_24h.

8.2.6. What to alert on

Condition

Why

pepsi-status --json reports problem_total above zero for longer than MIN_AGE plus ten minutes

A message has failed and has not been bounced: pepsi-failure-bouncer.timer is not running (systemctl list-timers), or the bounce itself failed. Messages that fail at the bounce stage are not bounced again and wait for you.

pepsi_pause_backlog of a non-delivery stage growing, or the journal repeating “failed at stage … retrying”

A host fault at that stage (an unreadable key or template, a helper that cannot start, a refused statement). Its messages are held, not bounced, until MAX_LIFETIME; state.last_error says why.

pepsi-status --json reports quarantined above zero, or min_over_time(sum(pepsi_quarantine_held_messages)[24h:1h]) > 0 (the quarantine has not been empty for a day)

Mail waits at a quarantine stage for a decision (see Quarantined mail). Nobody else is told, and the hold ends with a bounce or a deletion when nobody decides.

The dispatcher logging “the worker refused to start” for a stage

The stage’s configuration or a secret that it reads does not parse; its messages are held until it does.

pepsi_pause_backlog of a relay stage growing for hours

The next hop, DNS or outbound port 25 is unreachable. Retries continue until MAX_LIFETIME, then the senders get bounces.

rate(pepsi_stage_crashes_total[15m]) or rate(pepsi_stage_timeouts_total[15m]) above zero

A stage program is failing; its messages end as failed or timeout.

rate(pepsi_messages_processed_total[1h]) at zero when mail is expected

Nothing leaves the pipeline: the dispatcher is down or a stage is stuck.

pepsi-status --json lists a served_keys address with a channel whose verdict is mismatch

A key that the store does not publish is handed out for that address, so whoever holds that key can read future mail to the address (see above). Run the check on the monitoring host’s schedule, not every minute: each run fetches every published identity.

A pepsi-* unit in failed state (systemctl --failed)

The long-running units retry forever, so a failed one is usually a timer or a one-shot job.

pepsi-status --json lists an own_dnssec record whose verdict is bogus (or servfail)

The DNSSEC chain of trust of a served domain is broken, for example by an expired signature, or by a DS record at the parent that matches no published DNSKEY after a key roll. Every validating resolver then refuses the domain: senders behind one cannot find its MX, and senders using DANE defer mail to it. detail names the Extended DNS Error; repair the zone’s signatures or the DS record. The same finding fails pepsi-setup-check.service.

pepsi-setup-check.service failed

The daily pepsi-setup check --fail-on-findings found a DNS record missing, wrong, invalid or unreachable from the Internet, or this host on a DNS blocklist; the journal (journalctl -u pepsi-setup-check) has one warning per finding and the full report. Receivers then refuse the domain’s mail, or file it as spam, long before anybody complains. A lookup that merely failed does not fail the unit. Providers’ feedback loops (Microsoft SNDS/JMRP, Gmail Postmaster Tools) are not read by Pepsi; check them by hand.

A “DKIM: DNS changes needed” mail from pepsi-dkim, or pepsi-status warning in its “DKIM keys” section

A DKIM record is missing (receivers cannot verify mail signed with that key), a rotation waits for its records, or an RFC 2136 update was refused. pepsi-dkim records prints what to publish (Rotating DKIM keys).

pepsi-dkim.service not running

The signing stages cannot sign; outbound mail waits in the queue (it is never sent unsigned) until the daemon is back.

pepsi-greylist.service not running where greylisting is on, or pepsi-status warning that the daemon does not answer

Mail is accepted without greylisting (an MTA treats the missing milter as a failure, and ON_FAILURE = accept lets the mail through).

pepsi-greylist stats --json counting error outcomes steadily

The daemon cannot reach its database or its resolver in time (DNS_TIMEOUT), and lets those recipients through ungreylisted.

A domain with an spf or ptr identity in pepsi-greylist list --never-retried again and again

A real mail system that does not retry: its mail never arrives. Allow it (see above).

The never-retried share well above its usual level: sum(pepsi_greylist_decisions_24h{outcome="expired",reason="never-retried"}) / sum(pepsi_greylist_decisions_24h{outcome="deferred",reason="new"}). Most first attempts that are never retried are spam, so the share is high anyway; alert on a rise against what your site usually sees

A correspondent that sends much mail has stopped retrying, or started sending from addresses that the identity ladder cannot tie together. pepsi-greylist list --never-retried or the console’s Greylisting page (state tried only once) names the domains; allow the real ones.

pepsi_greylist_decisions_24h{outcome="error"} growing

The same as the error outcomes above, on /metrics.

Journal records at error

Anything logged as an error needs looking at.

The TLS certificates’ expiry

certbot renews them; a renewal that fails is logged by certbot, not by Pepsi.

Free space where PostgreSQL keeps its data

Below [pepsi] MIN_FREE_SPACE (default 1 GiB), pepsi-ingress refuses new mail with a temporary error.

Unread managed-PKI notices of severity warning or error (pepsi-keys mpki notices --unread), or a failed order in pepsi-keys mpki order list

A user’s certificate waits for you (a name to confirm, the daily limit, a missing licence or credential) or could not be ordered. The notice names the order and the command that releases it; see A user’s CA certificate does not arrive. With a notify address and [pepsi-mpki] NOTIFY_STAGE, the same notices arrive by mail.

pepsi-keys-mpki.service failed

A pass of the managed PKI met an error (journalctl -u pepsi-keys-mpki). Orders that the CA already has are retried by the next pass; nothing is lost, but certificates may arrive late.

An account.reassigned warning in the audit log

A login came back under another uid, or after it was removed: its previous owner’s settings, whitelists and quota were dropped, and pepsi-setup check lists any key material still held for the address, which you have to dispose of (A login that changes owner).

8.3. Changing the key behind a TLSA record

The DANE record that pepsi-setup run prints for each TLS ingress listener (_<port>._tcp.<HOSTNAME> IN TLSA 3 1 1 <hash>) pins the mail certificate’s public key, not the certificate. pepsi-setup therefore obtains certbot certificates with --reuse-key and keeps reuse_key = True in their renewal configuration, so an ordinary renewal never invalidates the record. A deliberate key change (a suspected compromise, a move to another key type or size) is different: a sender that enforces DANE refuses to deliver to a host whose certificate matches no published record, and resolvers keep serving the old record set for its TTL. Pepsi has no rollover workflow of its own. You make the change by hand, and publish the record of the next key before the key is used:

  1. Create the next key and its certificate beside the current ones, as a separate certbot lineage that the services do not use yet (same host name, same challenge method as the original, here the standalone one):

    certbot certonly --cert-name mx.example.org-next -d mx.example.org \
        --standalone --reuse-key
    
  2. Publish the next key’s record alongside the current one, under every name at which the current record is published (one per TLS port). Its hash is the SHA-256 of the new key’s SubjectPublicKeyInfo:

    openssl x509 -in /etc/letsencrypt/live/mx.example.org-next/cert.pem \
        -noout -pubkey | openssl pkey -pubin -outform DER \
        | openssl dgst -sha256 -hex
    
    _25._tcp.mx.example.org. IN TLSA 3 1 1 <current hash>
    _25._tcp.mx.example.org. IN TLSA 3 1 1 <next hash>
    

    Both records are valid at once: a sender accepts a certificate that matches either, so the current certificate keeps working.

  3. Wait out the TTL of the TLSA record set (and let the signed zone reach every authoritative server). Until then a resolver may still hold the old set, which does not yet name the next key.

  4. Switch to the new key. Point every section that names the old lineage (TLS_CERT/TLS_KEY of the ingress listeners) at /etc/letsencrypt/live/mx.example.org-next/. The pepsi-httpd listeners have a lineage of their own, mx.example.org-web, which no TLSA record pins. Then run pepsi-setup -c /etc/pepsi/pepsi.conf run (which keeps reuse_key set for the new lineage and hands the certificate to the services) and restart the services. pepsi-setup check then reports the published records against the certificate that is now served.

  5. Remove the old record, and the old lineage with certbot delete --cert-name mx.example.org. From here on the new lineage renews with its key kept, like the old one did.

8.4. Rotating DKIM keys

The DKIM and ARC private keys belong to the signing daemon, pepsi-dkim (pepsi-dkim.service, account pepsi-dkim). Each signing domain has a current generation of keys, one per algorithm of [pepsi] DKIM_ALGORITHMS (the ARC_DOMAIN also one of ARC_ALGORITHM), under selectors named after the month in which the generation was made: pepsi-202610._domainkey.example.org for RSA and pepsi-202610-ed25519._domainkey.example.org for Ed25519, with pepsi being [pepsi] DKIM_SELECTOR. A rotation makes a next generation under new selectors, switches signing to it once its records are published, keeps the old records for RETIRE_AFTER_DAYS (14 by default) so that mail signed just before the switch still verifies, and then retires the old keys.

8.4.1. Seeing what DNS needs

The daemon checks every authoritative server of each zone, hourly (DNS_CHECK_INTERVAL), for the record of every key that has to be in DNS, and reports the result everywhere:

  • pepsi-dkim status lists every key with its state (next, current, retiring, retired) and its DNS verdict (visible, missing, disagree while a secondary lags, unknown); pepsi-dkim records prints the records to publish as zone-file lines and the removable ones as comments;

  • pepsi-setup run prints the missing records with the rest of the zone, and pepsi-setup check grades each published record, including one that a DNS host has truncated (a 3072- or 4096-bit key needs a TXT record split into several strings, which some hosting panels mangle);

  • pepsi-status (“DKIM keys”) and the console’s DKIM page (/ui/dkim) show the same;

  • the daemon mails the postmaster (NOTIFY, NOTIFY_ADDRESS) at most once a day while a record has to be added, a record of a retired key may be removed, a rotation waits for its records, or an RFC 2136 update was refused. Each mail names the records and repeats daily until the change is made.

A domain that has no keys yet gets a generation at once, and it signs at once. This happens on a fresh installation, for a domain added to the configuration, and on a host restored without the daemon’s state directory. Until its records are published, receivers cannot verify those signatures and DMARC rests on SPF alone, so publish what pepsi-setup run prints right away.

8.4.2. Scheduled rotation

Rotation is off by default (ROTATION_DAYS = 0): a generation signs until somebody runs pepsi-dkim rotate. To rotate on a schedule, set, for example, the following (M3AAWG advises rotating at least every six months):

[pepsi-dkim]
ROTATION_DAYS = 180
# made this long before the switch, to give time to publish (default 28)
ROTATION_LEAD_DAYS = 28

and restart pepsi-dkim.service. ROTATION_LEAD_DAYS before a generation has signed for ROTATION_DAYS, the daemon makes the next generation and asks for its records. With WAIT_FOR_DNS = yes (the default) the switch waits until every authoritative server serves every record of it; the old keys go on signing meanwhile, and the delay is reported and mailed. WAIT_FOR_DNS = no switches on schedule whatever DNS serves, which is right only where something else guarantees that the records are there.

RETIRE_AFTER_DAYS must exceed the SIGNATURE_EXPIRATION_DAYS of every pepsi-stage-dkim-sign and pepsi-stage-arc stage (seven days by default); pepsi-setup validate checks this. With SIGNATURE_EXPIRATION_DAYS = 0 the signatures carry no expiry, and RETIRE_AFTER_DAYS should cover the delivery stages’ MAX_LIFETIME plus a few days for the receivers’ own queues. After it, the old private keys are deleted and the old records may be removed.

[pepsi-dkim] RSA_BITS (2048, 3072 or 4096; default 2048) sets the size of the RSA keys that the next generation gets; existing keys keep theirs.

8.4.3. Rotating on demand

pepsi-dkim -c /etc/pepsi/pepsi.conf rotate example.org
pepsi-dkim -c /etc/pepsi/pepsi.conf rotate --hard example.org

Run as root (or pepsi-dkim). Without a domain, every signing domain rotates. The first form makes a next generation and switches as soon as its records are visible (or at once under WAIT_FOR_DNS = no); publish what pepsi-dkim records prints. --hard switches at once, before anybody can have published the new records, so mail signed with the new keys fails DKIM until the records are published. It is meant for a key that may have been exposed (After a key exposure). The console’s DKIM page offers both as rotate now; the button queues a dkim-rotate task that the root applier (pepsi-setup apply) runs, so it needs setup:write and a running applier.

8.4.4. Publishing the records automatically

Where the zone accepts dynamic updates, the daemon can add the records of its next and current keys and remove those of retired keys itself, by RFC 2136 UPDATE authenticated with TSIG:

[pepsi-dkim]
DNS_UPDATE = rfc2136
TSIG_KEY_NAME = pepsi-dkim.example.org.
TSIG_ALGORITHM = hmac-sha256
TSIG_SECRET = <base64 of the key>
# default: the zone's primary, the SOA MNAME
DNS_UPDATE_SERVER = ns1.example.org:53

pepsi-setup moves TSIG_SECRET into /etc/pepsi/secrets.d/pepsi-dkim.secret, readable by pepsi-dkim alone. Grant the key on the DNS server nothing but updates of TXT records under _domainkey (for BIND, an update-policy with subdomain or wildcard rules limited to TXT): whoever holds the TSIG key can change whatever the key is allowed to change. A server that cannot be reached is tried again a minute later. A refusal (a wrong key, a policy that forbids the name) is recorded on the key, mailed to the postmaster and not retried until pepsi-dkim.service restarts; fix the settings or make the change by hand, then restart it.

8.4.5. Publishing retired keys

A DKIM signature lets anybody holding a copy of a message prove to a third party that the domain sent it, long after delivery (DKIM signatures make stolen mail verifiably attributable). With [pepsi-dkim] PUBLISH_RETIRED_KEYS = yes a retired private key is not deleted but written to /var/lib/pepsi-dkim-public/retired/<domain>/<selector>.pem, and pepsi-httpd serves it at https://<host>/.well-known/pepsi-dkim-retired/<domain>/<selector>.pem. Once anybody can make such a signature, an old one proves nothing. The feature needs rotation, since keys retire only by rotating, and a public HTTPS listener; pepsi-setup validate warns if either is missing.

A retired key is published only once every authoritative server has stopped serving its records; until then the daemon keeps it unpublished, because anybody could sign as the domain with a published key whose record still resolves. So remove the records when the daemon reports them removable (DNS_UPDATE = rfc2136 does that itself). A key whose records are never removed is deleted, not published, 90 days after it retired.

Note

Resolvers may still hold a removed record until its TTL runs out, so for a few hours after publication a signature made with the published key can still verify at some receivers. Keep the TTL of the selector records short where this matters.

8.5. Rotating the SRS key

The SRS secret ([pepsi-srs] SECRET, in /etc/pepsi/secrets.d/pepsi-ingress.secret) signs every rewritten envelope sender, and a bounce returns to that address up to MAX_AGE_DAYS (default 21) later; the same holds for the forward return address of a forwarded copy, whose keys are derived from the secret. Replacing the secret outright would make every address still in circulation fail verification, and their bounces would be refused. The old secret is therefore kept for verification only, in [pepsi-srs] PREVIOUS_SECRET_FILE, until the last address it signed has expired:

  1. Move the current secret to its own file, readable by root only until step 3, and point PREVIOUS_SECRET_FILE at it:

    install -m 0600 /dev/null /etc/pepsi/secrets.d/srs-previous.secret
    sed -n 's/^SECRET *= *//p' /etc/pepsi/secrets.d/pepsi-ingress.secret \
        > /etc/pepsi/secrets.d/srs-previous.secret
    
    [pepsi-srs]
    PREVIOUS_SECRET_FILE = /etc/pepsi/secrets.d/srs-previous.secret
    
  2. Write a new secret into the [pepsi-srs] section of secrets.d/pepsi-ingress.secret (SECRET = `` followed by, for example, the output of ``openssl rand -hex 32), leaving every other line of the file as it is.

  3. Run pepsi-setup -c /etc/pepsi/pepsi.conf run. It gives the previous secret’s file the ownership and mode of the current one (pepsi-ingress, 0640) and checks that both can be read.

  4. Restart pepsi-ingress right away, because it reads the secrets only once, at start. pepsi-stage-srs reads them for every message, so from that moment on it signs with the new secret. Returning addresses verify under either secret.

  5. After MAX_AGE_DAYS days, remove PREVIOUS_SECRET_FILE and the file, and restart pepsi-ingress again. pepsi-setup run and pepsi-setup check say when that time has come: they warn about a previous-secret file last changed more than MAX_AGE_DAYS ago.

8.6. After a key exposure

A key may have left your control, for example through a stolen laptop with a mail client’s key on it, a backup that went astray, or a compromised stage. Such a key stays valid until somebody says otherwise; nothing expires it, and correspondents keep using what they cached. The steps below depend on what was exposed. Each one is idempotent, so when in doubt, do them all:

  1. A user’s or the gateway’s OpenPGP key (an identity in the key store): revoke it and publish the revocation, in that order of importance:

    pepsi-keys -c /etc/pepsi/pepsi.conf identity revoke ID --reason 'laptop stolen' --upload
    

    Every revocation, whether by this command, by the user’s retire e-mail command or in the console, builds a revocation certificate while the private key still exists, and stores it as the key’s public half. From then on, the Web Key Directory serves the key as revoked. If the key had been uploaded to a key server, the revocation is uploaded there too, because a key server cannot be told to forget a key; this command does that at once, otherwise pepsi-keys identity publish --retry does it. --upload also sends the revocation to a key server that never had the key. If the certificate cannot be made, nothing is revoked and the command can be re-run. The revoked key’s public half is kept so that old signatures still verify, and the next message from the address gets a fresh key under the usual policy (Key management). Correspondents learn of the revocation through the key server and the Web Key Directory when they refresh, and through the Autocrypt header of the user’s next mail. A correspondent who refreshes nothing keeps encrypting to the old key, and there is no remedy for that. A user’s own MUA key is revoked here only locally: its private half, and so its revocation certificate, is in their client.

  2. A user’s S/MIME certificate: ask the issuing CA to revoke it (Pepsi cannot; the CA’s CRL is what correspondents that check will see), then revoke the identity as above (no revocation certificate exists for S/MIME, and --upload has no meaning) so this host stops signing with it.

  3. A correspondent’s key that you learnt is exposed (they told you, or their revocation never reached you): remove the cached copy, and the message in hand is then handled as for a correspondent without a key, under the encrypt stage’s ON_NO_KEY:

    pepsi-keys -c /etc/pepsi/pepsi.conf peer list --address ADDRESS
    pepsi-keys -c /etc/pepsi/pepsi.conf peer remove PEER-KEY-ID
    

    A CA whose key is exposed is disabled or removed the same way (pepsi-keys ca disable CA-ID, ca remove CA-ID), which stops every certificate that it issued from verifying here. An S/MIME certificate that its CA has listed is retired automatically once pepsi-keydisc fetches that CRL (pepsi-keydisc).

  4. The key-encryption key (secrets.d/pepsi-crypto.secret) or the pepsi-crypto role’s process: every MTA private key was readable. Rotate the KEK (pepsi-keys wrap rotate), then revoke and re-create each affected identity as in step 1; the rotation alone re-wraps keys that an attacker may already hold in the clear.

  5. The TLS key of a listener: obtain a certificate for a new key and publish its TLSA record first, as Changing the key behind a TLSA record describes, then have the old certificate revoked by its CA (certbot revoke --cert-name mx.example.org --reason keycompromise for a certbot lineage). Peers that validate by DANE stop accepting the old key once the old record is gone. Peers that validate by PKIX see the revocation only if they check; Pepsi itself checks a stapled OCSP response and refuses a revoked certificate on a validating session, but fetches no status. The web server’s key (the mx.example.org-web lineage of pepsi-httpd) is a separate key that no TLSA record pins: replace it with certbot renew --cert-name mx.example.org-web --force-renewal --new-key and revoke the old certificate; the mail key is not affected. Its certificate does name HOSTNAME, so until the revocation reaches them, senders and mail clients that authenticate the MX by PKIX alone can be deceived with it.

  6. A DKIM or ARC key, or the pepsi-dkim account: rotate at once with pepsi-dkim rotate --hard DOMAIN (Rotating DKIM keys), publish the new records that pepsi-dkim records prints, and remove the old selector’s records right away instead of waiting RETIRE_AFTER_DAYS. Mail signed before the switch then fails DKIM on the way, which costs less than letting the holder of the key sign as the domain. Do not publish an exposed key with PUBLISH_RETIRED_KEYS before its records are gone. A compromised stage running as pepsi could not read the keys, but could have had mail signed while it ran: the daemon’s log names every signature and the requesting process.

Whatever was exposed, the audit log (GET /api/v1/events; pepsi-keys and pepsi-queue write to it) shows what the key was used for after the suspected time, and A login that changes owner what a recycled login loses.

8.7. Backup and restore

Warning

A database dump alone is not a backup of Pepsi. The private keys in the database are encrypted under a key-encryption key that is kept, on purpose, outside the database, in /etc/pepsi/secrets.d/pepsi-crypto.secret. Lose that file and every user’s stored private keys are lost with it: nothing can open them again, and pepsi-setup will not generate a replacement while encrypted keys exist (see Key management too hard: end-to-end mail clients error-prone or unused). The custody boundary that this split upholds is described in The private-key custody boundary, and what a copy of the database alone does and does not yield in Somebody holding a copy of the database.

8.7.1. What to back up

What

What is lost without it

/etc/pepsi/secrets.d/, above all pepsi-crypto.secret

pepsi-crypto.secret: every stored private key (above). The other fragments: the SRS key (bounces to mail forwarded before the loss are refused), the secure-link pepper (secure messages not yet collected cannot be opened), the proof-of-origin key (payment demands for mail sent before the loss are no longer recognised as ours), and the passwords and tokens that you entered (smarthost and LMTP passwords, the LDAP bind password of key discovery, the list API password, the OAuth client credentials of the token refresher, and the mpki-<domain>.secret API keys of the managed PKI’s CA accounts, without which no certificate is ordered, renewed or revoked until you store a key again). A retired SRS secret named by [pepsi-srs] PREVIOUS_SECRET_FILE belongs in the backup too while it is configured.

The PostgreSQL database (pepsi by default)

The queue, per-user settings, whitelists, mailing lists and their archive, the key store, quotas, administrator accounts and API tokens, the audit log.

/etc/pepsi/ (the rest)

The configuration. It can be written again with the wizard, but not quickly.

/var/lib/pepsi-dkim/

Leave it out. It holds the DKIM and ARC private keys, which do not belong in a backup: a copy is one more place they can leak from. After a restore without it, the daemon generates fresh keys under new selectors and signs with them at once, which is a rotation; publish the records pepsi-setup run prints and remove the old ones. Until the new records are published, DKIM fails at receivers and DMARC rests on SPF.

/var/lib/pepsi-wallets/

With pepsi-stage-auto-pay in shared wallet mode: the wallets and the money in them. In local-user mode the wallets are in the users’ home directories.

/etc/letsencrypt/

Nothing permanent; certbot obtains new certificates. Keep it anyway if you publish DANE (TLSA) records for the current key.

/var/pepsi/tokens, /var/pepsi/token-refresh, /var/pepsi/tls

Only with an OAuth or client-certificate smarthost: the refresh token has to be authorised again.

The users’ mailboxes (Maildir, or the MDA’s store behind LMTP) are not Pepsi’s and belong in your ordinary backup of home directories.

Keep the files and the database dump together, and from the same moment: a dump newer than secrets.d may hold keys wrapped under a key-encryption key that the file backup does not have. Both hold secrets; store them encrypted and readable by root only. The database dump also holds queued messages, which are plaintext for mail that Pepsi decrypted (see Decrypted mail, keys or subjects persisted to logs and local storage).

8.7.2. Taking a backup

sudo -u pepsi-owner pg_dump --format=custom pepsi > pepsi-$(date +%F).dump
tar -C / -czpf pepsi-files-$(date +%F).tar.gz \
    etc/pepsi var/pepsi var/lib/pepsi-wallets

Run as root, so that tar can read secrets.d. A [pepsi-mpki] CREDENTIAL_DIR outside /etc/pepsi needs adding to the tar command. pg_dump takes a consistent snapshot while the services run; the files change rarely (a new key, a configuration change), so taking them just before or after the dump is enough. The database name and its owner are those in [pepsi-postgres].

The dump includes the known-compromised-key list (pepsi.compromised_key, about 240 MB). It is the one large table a restore could do without: leaving it out (--exclude-table-data=pepsi.compromised_key) keeps the dump small, and after a restore pepsi-keys blocklist update --force fetches the list again. The moduli that pepsi-keys peer screen factored (source screen) are then lost, however, until the next screen finds them again.

The dump that pepsi-setup schema --backup-dir takes before an upgrade (see Backing up before a schema upgrade) covers the database only, and only the pepsi schema, and only the two newest such dumps are kept; it does not replace this.

8.7.3. Restoring

Restore on a host running the same Pepsi release that the backup came from (upgrade afterwards, if you want to), in this order:

  1. Install Pepsi. The Debian package creates the service accounts and database roles; on a source install create them as in Installation.

  2. Stop everything: systemctl stop pepsi.target.

  3. Restore the files, secrets.d first. tar keeps owners by name, so the accounts must exist before you unpack:

    tar -C / -xzpf pepsi-files-2026-10-01.tar.gz
    
  4. Restore the database into an empty one owned by the schema owner:

    sudo -u postgres dropdb --if-exists pepsi
    sudo -u postgres createdb -O pepsi-owner pepsi
    sudo -u pepsi-owner pg_restore -d pepsi pepsi-2026-10-01.dump
    
  5. Run pepsi-setup -c /etc/pepsi/pepsi.conf run. It re-applies the role grants, writes again what lives outside both backups (the systemd drop-ins that hand the TLS certificates to the services) and has pepsi-dkim make fresh DKIM and ARC keys, since those are not in the backup. It does not replace a key or secret that already exists. Publish the DKIM records it prints (new selectors) and remove the old selectors’ records.

  6. Start the services: systemctl start pepsi.target.

If pepsi-setup says it refuses to generate a key-encryption key because identities exist, secrets.d/pepsi-crypto.secret was not restored, or is not the one that goes with this database. Do not work around it: find the right file.

8.7.4. Cloned and restored virtual machines

Every key, nonce and secret that Pepsi makes comes from the operating system’s random number generator. On a virtual machine that generator’s state is part of every snapshot: two instances started from one snapshot, or one instance restored twice, can draw the same “random” bytes until something reseeds them. Under ENABLE_PEP one host generates every user’s long-term keys, and pepsi-setup generates the key-encryption key, so a repeated state would make all of them predictable at once. Waiting for the generator to be seeded does not help: after a clone it is seeded, with the state of the original.

  • Reboot a cloned or restored instance before it generates anything, and before it starts the Pepsi services: a fresh boot reseeds the kernel’s generator, and processes started afterwards draw from it. Restoring a backup onto a newly installed host does not have this problem; restoring a whole virtual machine image does.

  • Let the hypervisor make it automatic. A hypervisor that exposes a VM generation ID (QEMU/KVM -device vmgenid, Hyper-V, VMware) lets the Linux vmgenid driver notice a clone or a restore and reseed the kernel’s generator at once. Check that the device is present (ls /sys/bus/acpi/devices/ | grep -i -e VMGENID -e QEMUVGID) and turn it on where it is not.

  • Pepsi warns about what it generated just after a change. The driver announces the change with a udev event, on which Pepsi’s udev rule (60-pepsi.rules) touches /run/pepsi-vmgenid-changed. Generating an OpenPGP or S/MIME key (pepsi-keys, the encrypt stage under ENABLE_PEP), a one-time-password secret, the key-encryption key or a new wrapping of the private keys within ten minutes of that logs a warning naming what was made. If the instance was not rebooted since the change, reboot it, then revoke and regenerate what the warnings name (After a key exposure). Without the driver, or without the rule, nothing is recorded and nothing is warned about.