10. Troubleshooting

Start with two commands; most problems show up in one of them:

pepsi-status
journalctl -u 'pepsi-*' -p warning --since '1 hour ago'

Operating Pepsi explains the logs and what pepsi-status reports. The problems below are the ones early deployments met most often.

10.1. Common problems

10.1.1. No mail arrives

  • Another MTA holds port 25. Debian installs one by default. systemctl start pepsi.target then fails, and systemctl status pepsi-ingress.socket shows Result: resources. ss -ltnp 'sport = :25' names the program; stop and disable it (see Installation, “Migrating from an existing mail server”).

  • DNS or the firewall. The domain’s MX record has to name this host and port 25 has to be open from the Internet. pepsi-setup -c /etc/pepsi/pepsi.conf run prints the records that it expects and checks the published ones.

  • Pepsi refuses the message. The sender’s server gets the reason, and so does the journal of pepsi-ingress. The usual reasons are 550 5.1.1 for an address that nothing on this host delivers to (see VERIFY_RECIPIENTS in pepsi-ingress(1)); 550 for a definite DMARC failure under DMARC_ENFORCE; 554 5.6.0 for a MIME structure that has no single correct reading, such as an empty boundary, a NUL in a MIME header, conflicting duplicate fields or an encoded-word boundary (MIME_ANOMALIES = log accepts such a message and records the anomaly instead; the journal names the class); and 452 4.3.1 when the queue or the disk is full (see MAX_QUEUE_ROWS and MIN_FREE_SPACE in pepsi.conf(5)).

  • A filter quarantined it. The message was accepted and waits at a quarantine stage, and nobody is told. pepsi-queue quarantine list shows it with the filter that sent it there and the reason; pepsi-queue quarantine release TOKEN lets it through (Quarantined mail).

10.1.2. Outgoing mail is not delivered

pepsi-status lists the messages waiting at each stage. Messages that stay paused at a relay stage are being retried; pepsi-queue list ID --state-only shows the last reply of the remote server. Common causes:

  • Outbound port 25 is blocked. Many hosting and residential providers block it. Relay through a smarthost instead (pepsi-setup --wizard).

  • The receiving server rejects or files the mail as spam. Check that the SPF, DKIM and DMARC records that pepsi-setup run prints are published, and that the host’s address has a reverse DNS name matching [pepsi-ingress] HOSTNAME.

  • Messages wait at the DKIM-signing or ARC stage. The signing stages ask pepsi-dkim for every signature and retry while it does not answer; the last error names the socket it could not connect to. Check systemctl status pepsi-dkim and its journal. A message from a domain that the daemon does not sign for is refused instead (the sender gets a bounce): run pepsi-setup run, which hands the daemon every signing domain.

  • DKIM fails at the receiver. The selector that the message names (s= in its DKIM-Signature) has no published record: pepsi-dkim status shows each key’s DNS verdict and pepsi-dkim records the records to publish (Rotating DKIM keys).

  • The next hop’s DANE record does not match its certificate. DANE = strict is the default, so a published TLSA record that matches nothing that the server presents defers the message (and eventually bounces it) rather than sending it over unauthenticated TLS. The last error names the host. The fix belongs to the receiving domain; DANE = warn in the relay stage’s section is the explicit opt-out (see The network adversary).

  • An encrypted message waits on a key lookup. If the DNS lookup of a recipient’s key failed, and the message would otherwise have gone out in cleartext, it is deferred rather than sent in clear. The sender can accept cleartext with [clear-ok] in the subject (see A key lookup that fails in the DNS).

10.1.3. A submission is refused

The mail client shows the reason. 550 5.7.1 means the MAIL FROM or From: address is not one that USERNAME_MAP (or SENDER_MAP, for a trusted network or client certificate) lets this account use. 550 5.6.0 means that the message has no usable From: (none, more than one, only a group, or a mailbox without a full address), or that the display name of its From: contains an address other than its own, which many clients would show in place of the real one. See Supported Features (“Message submission”).

If a mail client’s sign-in (SMTP AUTH) takes twenty seconds or fails with 454 4.7.0, the account has collected ACCOUNT_AUTH_FAILURES_PER_HOUR failed AUTH attempts from all sources (the journal says “AUTH for ‘…’ delayed”). Attempts from networks that the account has not signed in from then wait ACCOUNT_AUTH_DELAY, one at a time. Somebody is guessing at the account; the owner still gets in after the delay, and at once from a network they signed in from before.

10.1.4. The administration console refuses a password

A 429 (or the sign-in page answering “too many”) on an account that nobody is using means that somebody has used up its LOGIN_DAILY_LIMIT of failed sign-ins (auth.login.ceiling in the audit log); the limit refills over the day. For an account with a second factor, a missing, wrong or locked code gets exactly the same answer as a wrong password, and so does an applier that is not installed or not answering; the audit log’s auth.login.second-factor-refused says which of these happened. The local socket needs no password: pepsi-keys otp reset --console LOGIN clears a locked or lost second factor, and PATCH /api/v1/accounts/{id} sets a new password.

10.1.5. A message bounced with 5.7.30 or 5.7.10

The message was tagged REQUIRETLS by its sender (RFC 8689), and no MX host of the recipient domain, nor the configured smarthost, could take it under that requirement. 5.7.30 means that the hop does not advertise REQUIRETLS after STARTTLS. 5.7.10 means that there was no validated TLS to it: no STARTTLS, a certificate that fails DANE or PKIX, a domain whose MX hosts are validated neither by DNSSEC nor by an MTA-STS policy, or a smarthost entry with MODE = plain. Nothing on this host is misconfigured, unless DANE = off hides the validation of a DNSSEC-signed domain, or a smarthost that could offer the extension does not. The sender asked for protection that the path cannot give, and the bounce (headers only) tells them so. See Require TLS for one message (RFC 8689).

10.1.6. pepsi-ingress does not start after a configuration change

The journal names the reason. A MYNETWORKS entry outside loopback that [pepsi-ingress] SENDER_MAP does not cover is refused, because every host in that network could send as any address and have the message signed for every served domain. Add a line <network> <addresses> to the map (* allows the unrestricted behaviour, as a deliberate choice), or remove the entry. pepsi-setup -c /etc/pepsi/pepsi.conf validate reports the same refusal without restarting anything. See pepsi-ingress(1), Sender bindings.

10.1.7. sendmail or cron mail fails with “temporary failure”

pepsi-sendmail hands mail to pepsi-ingress over /run/pepsi/submission.sock and exits with status 75 (temporary failure) when nobody answers there. Check that pepsi.target is running. If /run/pepsi was deleted by hand, systemd-tmpfiles --create re-creates it with the right permissions; then restart pepsi-ingress.

10.1.8. A program exits with status 78

The database schema is not the one that the program was built with, typically in the minutes after an upgrade before pepsi-setup schema has run. The message names the fix; Upgrading has the table of cases. The services retry by themselves and come back once the schema matches.

10.1.9. “permission denied” from the database

The database roles’ grants are set by pepsi-setup. After restoring a database, or after changing roles by hand, run pepsi-setup -c /etc/pepsi/pepsi.conf run again.

10.1.10. An S/MIME recipient sees a blank message

Their client cannot read AuthEnvelopedData (AES-256-GCM). Thunderbird, for example, cannot, and shows an empty message instead of an error. Pepsi sends this format to every S/MIME recipient for whom it has no capability record. The first such message carries a plaintext notice beside the ciphertext, which asks the recipient to report exactly this problem and to reply with a signed message (What a correspondent on an unmeasured client sees). The first signed message from a correspondent records their client’s capabilities automatically; until then, configure the record by hand:

pepsi-keys -c /etc/pepsi/pepsi.conf peer capability set alice@example.org cbc

peer capability list --address alice@example.org shows what is recorded, and the message’s state.crypto.out names the container that went out. See The per-peer S/MIME capability record.

10.1.11. A served key is reported as a mismatch, or WKD as failed

pepsi-setup check, pepsi-status and pepsi-keys identity list --served fetch what the Web Key Directory and the DNS serve for each published identity (Checking what is actually served). A MISMATCH names the channel and the foreign fingerprint:

  • WKD (direct) – the domain’s apex is answered by another web server (a CMS, a CDN) that has its own /.well-known/openpgpkey/. Remove that directory there, or have the apex proxy /.well-known/openpgpkey/ to pepsi-httpd.

  • OPENPGPKEY or SMIMEA – the zone still holds a record for a rotated or revoked identity, or a record that somebody else added. Replace it with what pepsi-keys dns ADDRESS prints.

  • WKD (advanced) – openpgpkey.<domain> points at a host that is not this one.

Until it is fixed, correspondents who look the address up may encrypt to that key. A WKD channel that is FAILED with “refusing to connect” names a host that resolves to a private or loopback address; the client refuses such an address, as every key fetch does. A WKD channel that times out on a NATed host usually means that the router does not route this host’s own public address back to it. In neither case is the result what a correspondent sees, so check from outside before you change anything.

10.1.12. The resolver “does not validate” DNSSEC

The relay workers and pepsi-keydisc@dane log this warning when they start, and pepsi-status shows it in its “DNSSEC resolver” section. It means that the resolver named there (/etc/resolv.conf, [pepsi] DNS_SERVERS, or a stage’s DNS_SERVERS) answered a query for the signed ietf.org without the DNSSEC AD bit. Pepsi trusts that bit instead of validating by itself, so every DANE lookup finds nothing: outbound DANE never applies and DANE key discovery finds no key, although nothing fails. Typical causes are an ISP or DHCP resolver written into resolv.conf after setup, or systemd-resolved without DNSSEC=yes. To fix it, point resolv.conf (or DNS_SERVERS) at a validating resolver on this host, such as unbound (Installation), or name a validating resolver that you trust as a tls:// or https:// entry of [pepsi] DNS_SERVERS, and confirm with pepsi-status. Alternatively, set DANE = off where DANE is not wanted. “Could not be established” means that the probe got no answer at all. Check that the resolver is reachable; for an encrypted entry, check that its port is open and that its certificate is valid for the name after #.

10.1.13. The resolver “is not on this host”

The DANE workers log “trusts the DNSSEC AD bit from … which is not on this host and is reached over plain DNS”, and pepsi-status and pepsi-setup run name the same resolvers. The AD bit that those answers carry travels unprotected, so anyone on the path can set it on a forged TLSA, OPENPGPKEY or SMIMEA record. Either run a validating resolver on this host (unbound, systemd-resolved with DNSSEC=yes), or reach a validating resolver that you trust over an authenticated channel: [pepsi] DNS_SERVERS = tls://192.0.2.53#dns.example.net (DNS over TLS) or https://…#NAME (DNS over HTTPS), with the name that its certificate is issued for. A list may not mix plain and encrypted entries.

10.1.14. Recipients see ... as the subject of encrypted mail

[stage-encrypt] PROTECT_HEADERS = yes (the default is no) moves the real Subject: and the other RFC 5322 fields inside the encrypted and signed part, and the outer subject reads .... A client that renders protected headers (Thunderbird, for one) shows the real subject; one that does not shows the placeholder. Where most recipients’ clients are of the second kind, remove the setting or set PROTECT_HEADERS = no in the encrypt stage’s section (or per sender through pepsi-settings). The signature then no longer covers To: and Cc: for clients that ignore Pepsi’s own recipient attribute (Who a signed and encrypted message was for).

10.1.15. A message from an S/MIME sender disappeared instead of bouncing

With ON_DECRYPT_FAILURE = bounce-if-unproblematic, ON_BAD_SIGNATURE = bounce or the re-encrypt stage’s ON_NO_CLIENT_KEY = bounce, a message that arrived as AES-CBC EnvelopedData is dropped, not bounced, because the bounce would tell its sender whether a possibly modified ciphertext could be opened. The journal says so (dropped rather than bounced) and the telemetry counts decrypt-dropped. passthrough, record and plaintext deliver such a message instead (Bounces and replies to encrypted mail).

10.1.16. A user’s CA certificate does not arrive

Look at the user’s orders first:

pepsi-keys -c /etc/pepsi/pepsi.conf mpki order list --address alice@example.org
pepsi-keys -c /etc/pepsi/pepsi.conf mpki user list --domain example.org

No order at all means that the user is not wanted: under first-send they have not sent mail through the server yet (or only from outside it), under manual nobody selected them (mpki user set ADDRESS --select yes), or their selection is no. Check also that the domain is enabled (mpki domain show DOMAIN). A held order names its reason:

name-unconfirmed

The name came from the display name of the user’s first message, and the domain waits for an administrator to confirm such names. Check it in mpki user list (marked ?), then mpki user set ADDRESS --confirm, or correct it with --given-name/--surname. The next pass releases the order.

name-missing

The display name or the directory entry gave no given name and surname (a single word, an address, nothing). Enter both with mpki user set, or import the user from the directory.

daily-limit

The domain’s new enrolments of the last 24 hours reached its daily limit. Check that the orders are the ones you expect (an import of the wrong file or a script that sends as many addresses shows here), then release them one by one with mpki order release ID, or raise the limit (mpki domain set DOMAIN --daily-limit N) and release them.

no-licence

The CA account has no free subscription licence for the address: the CA answered insufficient_subscription. Pepsi cannot buy one. Add a licence in the CA’s portal (or free one by removing a departed user’s address from the subscription there), then mpki order release ID. A revocation does not free a licence by itself.

no-credential

The domain has no readable secrets.d/mpki-<domain>.secret. Store the API key with mpki credential set DOMAIN and run mpki domain check DOMAIN; the next pass releases the order.

A failed order shows the reason under last error. A refusal by the CA (HTTP 4xx with the CA’s message) is usually a setting that mpki domain check DOMAIN also reports: an API key the CA does not accept, a product that the account lacks, several organisations and no --organization-id, or a domain that is not validated in the account. A user who is still wanted gets a new order a day after the failed one, so until the cause is fixed each such user costs one failed order and one notice a day. Once it is fixed, mpki order retry ID tries again at the next pass instead of the next day. “the certificate does not name …” or “is not over the key” means that the CA returned a certificate that Pepsi refuses to install; report it to the CA. An order that stays submitted waits at the CA, and mpki order list --json shows the CA’s own status (ca_status). Pepsi keeps asking, at most once an hour, and installs the certificate once it is issued; nothing here can speed this up. Typical causes are an approval that the account requires in the CA’s portal, and a CA status of pending while the CA has not finished validating the sponsoring organisation for S/MIME. In CertCentral, GET organization/{id}/validation then lists a validation of type smime with status pending; the CA completes it, with whatever the portal asks of the organisation.

10.1.17. Messages in failed or timeout

Most errors do not end a message at all, because they are this host’s fault rather than the message’s:

  • A lost database connection: the message is paused and retried every 30 seconds until the database is back, and each retry is logged under pepsi-dispatch (“lost its database connection”).

  • Any other error that a stage reports, such as a key, template, map or trust store that cannot be read, a helper that cannot be started, a database statement that is refused, or a service that does not answer: the message stays paused at its stage with the error in state.last_error, and is retried after one minute, then at doubling intervals up to an hour, for the stage’s MAX_LIFETIME (120 hours unless configured). The stage logs each retry (“failed at stage ‘…’ (attempt N); retrying in …s”). Fix the cause and the queue drains by itself; pepsi-status shows the waiting messages meanwhile. When MAX_LIFETIME runs out the message goes to the stage’s BOUNCE_STAGE, whose DSN tells the sender that a local error persisted.

  • A stage whose configuration does not parse does not start at all: the dispatcher logs “the worker refused to start”, keeps the messages queued and tries again every few seconds. The stage’s own log line says what is wrong.

  • A worker that crashes or hangs past MAX_RUNTIME pauses the message it was working on and retries it a minute later; only the third time is the message given up on. In a delivery stage a worker killed at MAX_RUNTIME is retried until MAX_LIFETIME instead.

  • A DSN with the subject “Outcome Unknown” comes from LMTP delivery: the MDA took the message three times without answering in time, so it may hold up to three copies. Look for a slow Sieve program or an overloaded MDA, and raise DATA_TIMEOUT and that stage’s MAX_RUNTIME if it is only slow.

  • A milter that keeps deferring a message, or cannot be reached, holds it for MAX_LIFETIME as well, and then bounces it (or leaves it failed); it never drops it through the filter’s REJECT_STAGE. A milter that defers only some recipients holds just their copy, as a separate paused message at the milter stage, while the others are delivered.

A message ends in failed only when a stage found a defect in the message itself, when a stage with no BOUNCE_STAGE ran out of retries, or when the message crashed its worker three times. It ends in timeout when it made its worker hang three times. state.failure_class and state.last_error say which and why.

Delivery errors from the next server (a refusing or unreachable MX, a DNS lookup timing out) are retried by the relay stages themselves in the same way. A failed message is still often worth retrying once the cause is fixed.

pepsi.target runs pepsi-failure-bouncer --once every ten minutes (pepsi-failure-bouncer.timer). It moves every message that has been failed or timeout for [pepsi-failure-bouncer] MIN_AGE (an hour unless configured) to the stage that [pepsi-failure-bouncer] BOUNCE_STAGE names (bounce in the shipped configuration and in the wizard’s), and it logs each message with the reason. That stage sends the sender a delivery-status notification as their NOTIFY asks. A sender whose domain the message did not authenticate gets no notification under BOUNCE_UNAUTHENTICATED = drop, and the message is deleted, with a warning in the bounce stage’s log. A copy of a mailing-list post is deleted rather than bounced. So a failed message stays in the queue for about MIN_AGE; to retry it rather than bounce it, act within that time:

pepsi-queue list --status failed          # which messages, at which stage
pepsi-queue list 1234 --state-only        # why: state.last_error, state.bounce, …
pepsi-queue set-stage 1234 relay          # retry it from a stage of your choice
pepsi-failure-bouncer --once              # or bounce all of them now
pepsi-queue delete 1234                   # or discard it

The options of list go after the word list. systemctl list-timers pepsi-failure-bouncer.timer shows when the next sweep runs, and journalctl -u pepsi-failure-bouncer how many messages each one moved.

10.1.18. Keeping failed messages for diagnosis

On a development or test host a failed message is usually the thing you want to look at, and a bounce an hour later destroys it. Raise MIN_AGE, or turn the sweep off there:

systemctl mask --now pepsi-failure-bouncer.timer

mask rather than disable: pepsi.target wants the timer, so a merely disabled or stopped timer starts again with the target. Failed messages then stay in the queue until you retry, bounce or delete them as above. To turn the sweep back on:

systemctl unmask pepsi-failure-bouncer.timer
systemctl start pepsi-failure-bouncer.timer

The next sweep then bounces everything that failed in the meantime; delete what you do not want bounced first. Do not leave the timer masked on a server that handles real mail: nobody is told about a message that failed, neither the sender nor you, unless you watch problem_total (see Monitoring).

pepsi-failure-bouncer can instead run as a service that bounces each failed message as it happens. No unit is shipped for that mode; mask the timer before starting it from your own unit.

10.1.19. No core file after a crash

pepsi-httpd, pepsi-keys, pepsi-config export/import and pepsi-setup apply hold long-lived secrets (the key-encryption key and the private keys that it unwraps, the secure-link pepper, administrator password checks, an archive password). They mark themselves non-dumpable at start, and their systemd units set LimitCORE=0: a crash leaves the panic message and the backtrace in the journal (RUST_BACKTRACE=1 in a drop-in adds the frames), but no core file, and another process of the same account cannot attach a debugger. The setuid pepsi-stage-encrypt and pepsi-stage-decrypt are non-dumpable for the same reason. There is no option to turn this off; to examine such a crash, reproduce it on a test host with your own build, run under a debugger as root.

10.2. Frequently asked questions

Does Pepsi keep a log of who mailed whom?

Not by default. A delivered message is deleted from the queue and only the journal mentions it. [pepsi] MAIL_LOG turns on a per-message record; read the warning in pepsi.conf(5) first.

Where is the queue?

In PostgreSQL, table pepsi.workqueue. pepsi-queue and pepsi-status are the supported ways to look at it; plain SELECT works too.

How do I see the pipeline this host runs?

pepsi-config dump prints the effective configuration (--origin also shows which settings the database overrides), and pepsi-setup run prints, for each domain, how recipients are checked. The Wizard explains the pipelines that the wizard builds.

Can I go back to the previous release?

Not in place: the schema is migrated forward only. Restore the backup taken before the upgrade, then install the older packages; see Upgrading.

Can Pepsi run next to Postfix or Exim?

Not both on port 25. Pepsi can import the other server’s configuration; see Installation.

Why does the console show everything read-only?

The pepsi-httpd-admin package, which carries out the console’s changes, is not installed or its socket is not running; see Debian packages.

10.3. Uninstalling

Debian packages. apt remove pepsi stops the services and removes the programs; configuration, keys, the database and the service accounts stay, so a later install carries on where it left off. apt purge pepsi removes the rest of what the package installed but, on purpose, still keeps the configuration, the database, the keys, the wallets and the accounts, and prints the commands that remove them. They are, as root:

runuser -u postgres -- dropdb pepsi
rm -rf /var/pepsi /var/lib/pepsi-wallets /etc/pepsi \
       /var/lib/pepsi-dkim /var/lib/pepsi-dkim-public
for u in pepsi pepsi-ingress pepsi-httpd pepsi-owner pepsi-config \
         pepsi-helper-token-refresh pepsi-keydisc pepsi-crypto \
         pepsi-whitelist pepsi-wallets pepsi-dkim; do deluser "$u"; done
for g in pepsi-maildir pepsi-forward pepsi-token pepsi-admin \
         pepsi-telemetry; do delgroup "$g"; done

The database roles of the same names can then be dropped with dropuser. Remove pepsi-telemetry and pepsi-httpd-admin the same way if they are installed.

Source installation. Stop and disable pepsi.target, then run make uninstall with the same PREFIX/DESTDIR that you installed with. It removes the programs, the shipped data and the systemd units, not the configuration, keys or database; remove those as above.

Warning

Everything removed this way is gone for good, including the key-encryption key without which the stored private keys cannot be opened. Take a backup first (Backup and restore) if there is any chance you will want the keys, the mailing lists or their archive again.