10. Troubleshooting¶
Start with two commands; most problems show up in one of them:
pepsi-status
journalctl -u 'pepsi-*' -p warning --since '1 hour ago'
Operating Pepsi explains the logs and what pepsi-status reports. The
problems below are the ones early deployments met most often.
10.1. Common problems¶
10.1.1. No mail arrives¶
Another MTA holds port 25. Debian installs one by default.
systemctl start pepsi.targetthen fails, andsystemctl status pepsi-ingress.socketshowsResult: resources.ss -ltnp 'sport = :25'names the program; stop and disable it (see Installation, “Migrating from an existing mail server”).DNS or the firewall. The domain’s
MXrecord has to name this host and port 25 has to be open from the Internet.pepsi-setup -c /etc/pepsi/pepsi.conf runprints the records that it expects and checks the published ones.Pepsi refuses the message. The sender’s server gets the reason, and so does the journal of
pepsi-ingress. The usual reasons are550 5.1.1for an address that nothing on this host delivers to (seeVERIFY_RECIPIENTSin pepsi-ingress(1));550for a definite DMARC failure underDMARC_ENFORCE;554 5.6.0for a MIME structure that has no single correct reading, such as an empty boundary, a NUL in a MIME header, conflicting duplicate fields or an encoded-word boundary (MIME_ANOMALIES = logaccepts such a message and records the anomaly instead; the journal names the class); and452 4.3.1when the queue or the disk is full (seeMAX_QUEUE_ROWSandMIN_FREE_SPACEin pepsi.conf(5)).A filter quarantined it. The message was accepted and waits at a quarantine stage, and nobody is told.
pepsi-queue quarantine listshows it with the filter that sent it there and the reason;pepsi-queue quarantine release TOKENlets it through (Quarantined mail).
10.1.2. Outgoing mail is not delivered¶
pepsi-status lists the messages waiting at each stage. Messages that stay
paused at a relay stage are being retried; pepsi-queue list ID
--state-only shows the last reply of the remote server. Common causes:
Outbound port 25 is blocked. Many hosting and residential providers block it. Relay through a smarthost instead (
pepsi-setup --wizard).The receiving server rejects or files the mail as spam. Check that the SPF, DKIM and DMARC records that
pepsi-setup runprints are published, and that the host’s address has a reverse DNS name matching[pepsi-ingress] HOSTNAME.Messages wait at the DKIM-signing or ARC stage. The signing stages ask
pepsi-dkimfor every signature and retry while it does not answer; the last error names the socket it could not connect to. Checksystemctl status pepsi-dkimand its journal. A message from a domain that the daemon does not sign for is refused instead (the sender gets a bounce): runpepsi-setup run, which hands the daemon every signing domain.DKIM fails at the receiver. The selector that the message names (
s=in itsDKIM-Signature) has no published record:pepsi-dkim statusshows each key’s DNS verdict andpepsi-dkim recordsthe records to publish (Rotating DKIM keys).The next hop’s DANE record does not match its certificate.
DANE = strictis the default, so a publishedTLSArecord that matches nothing that the server presents defers the message (and eventually bounces it) rather than sending it over unauthenticated TLS. The last error names the host. The fix belongs to the receiving domain;DANE = warnin the relay stage’s section is the explicit opt-out (see The network adversary).An encrypted message waits on a key lookup. If the DNS lookup of a recipient’s key failed, and the message would otherwise have gone out in cleartext, it is deferred rather than sent in clear. The sender can accept cleartext with
[clear-ok]in the subject (see A key lookup that fails in the DNS).
10.1.3. A submission is refused¶
The mail client shows the reason. 550 5.7.1 means the MAIL FROM or
From: address is not one that USERNAME_MAP (or SENDER_MAP, for a
trusted network or client certificate) lets this account use. 550 5.6.0
means that the message has no usable From: (none, more than one, only a
group, or a mailbox without a full address), or that the display name of its
From: contains an address other than its own, which many clients would show
in place of the real one. See Supported Features (“Message submission”).
If a mail client’s sign-in (SMTP AUTH) takes twenty seconds or fails with
454 4.7.0, the account has collected ACCOUNT_AUTH_FAILURES_PER_HOUR
failed AUTH attempts from all sources (the journal says “AUTH for ‘…’
delayed”). Attempts from networks that the account has not signed in from then
wait ACCOUNT_AUTH_DELAY, one at a time. Somebody is
guessing at the account; the owner still gets in after the delay, and at once
from a network they signed in from before.
10.1.4. The administration console refuses a password¶
A 429 (or the sign-in page answering “too many”) on an account that nobody
is using means that somebody has used up its LOGIN_DAILY_LIMIT of failed
sign-ins (auth.login.ceiling in the audit log); the limit refills over the
day. For an account with a second factor, a missing, wrong or locked code gets
exactly the same answer as a wrong password, and so does an applier that is not
installed or not answering; the audit log’s
auth.login.second-factor-refused says which of these happened. The local socket
needs no password: pepsi-keys otp reset --console LOGIN clears a locked or
lost second factor, and PATCH /api/v1/accounts/{id} sets a new password.
10.1.5. A message bounced with 5.7.30 or 5.7.10¶
The message was tagged REQUIRETLS by its sender (RFC 8689), and no MX host
of the recipient domain, nor the configured smarthost, could take it under that
requirement. 5.7.30 means that the hop does not advertise REQUIRETLS
after STARTTLS. 5.7.10 means that there was no validated TLS to it: no
STARTTLS, a certificate that fails DANE or PKIX, a domain whose MX hosts are
validated neither by DNSSEC nor by an MTA-STS policy, or a smarthost entry with
MODE = plain. Nothing on this host is misconfigured, unless DANE = off
hides the validation of a DNSSEC-signed domain, or a smarthost that could offer
the extension does not. The sender asked for protection that the path cannot
give, and the bounce (headers only) tells them so. See
Require TLS for one message (RFC 8689).
10.1.6. pepsi-ingress does not start after a configuration change¶
The journal names the reason. A MYNETWORKS entry outside loopback that
[pepsi-ingress] SENDER_MAP does not cover is refused, because every host in
that network could send as any address and have the message signed for every
served domain. Add a line <network> <addresses> to the map (* allows the unrestricted
behaviour, as a deliberate choice), or remove the entry. pepsi-setup -c
/etc/pepsi/pepsi.conf validate reports the same refusal without restarting
anything. See pepsi-ingress(1), Sender bindings.
10.1.7. sendmail or cron mail fails with “temporary failure”¶
pepsi-sendmail hands mail to pepsi-ingress over
/run/pepsi/submission.sock and exits with status 75 (temporary failure) when
nobody answers there. Check that pepsi.target is running. If
/run/pepsi was deleted by hand, systemd-tmpfiles --create re-creates it
with the right permissions; then restart pepsi-ingress.
10.1.8. A program exits with status 78¶
The database schema is not the one that the program was built with, typically in the
minutes after an upgrade before pepsi-setup schema has run. The message
names the fix; Upgrading has the table of cases. The services retry by
themselves and come back once the schema matches.
10.1.9. “permission denied” from the database¶
The database roles’ grants are set by pepsi-setup. After restoring a
database, or after changing roles by hand, run pepsi-setup -c
/etc/pepsi/pepsi.conf run again.
10.1.10. An S/MIME recipient sees a blank message¶
Their client cannot read AuthEnvelopedData (AES-256-GCM). Thunderbird, for
example, cannot, and shows an empty message instead of an error. Pepsi sends
this format to every S/MIME recipient for whom it has no capability record. The
first such message carries a plaintext notice beside the ciphertext, which asks
the recipient to report exactly this problem and to reply with a signed message
(What a correspondent on an unmeasured client sees). The first signed message from a correspondent
records their client’s capabilities automatically; until then, configure the
record by hand:
pepsi-keys -c /etc/pepsi/pepsi.conf peer capability set alice@example.org cbc
peer capability list --address alice@example.org shows what is recorded, and
the message’s state.crypto.out names the container that went out. See
The per-peer S/MIME capability record.
10.1.11. A served key is reported as a mismatch, or WKD as failed¶
pepsi-setup check, pepsi-status and pepsi-keys identity list --served
fetch what the Web Key Directory and the DNS serve for each published identity
(Checking what is actually served). A MISMATCH names the channel and the foreign
fingerprint:
WKD (direct) – the domain’s apex is answered by another web server (a CMS, a CDN) that has its own
/.well-known/openpgpkey/. Remove that directory there, or have the apex proxy/.well-known/openpgpkey/topepsi-httpd.OPENPGPKEY or SMIMEA – the zone still holds a record for a rotated or revoked identity, or a record that somebody else added. Replace it with what
pepsi-keys dns ADDRESSprints.WKD (advanced) –
openpgpkey.<domain>points at a host that is not this one.
Until it is fixed, correspondents who look the address up may encrypt to that key. A WKD channel that is FAILED with “refusing to connect” names a host that resolves to a private or loopback address; the client refuses such an address, as every key fetch does. A WKD channel that times out on a NATed host usually means that the router does not route this host’s own public address back to it. In neither case is the result what a correspondent sees, so check from outside before you change anything.
10.1.12. The resolver “does not validate” DNSSEC¶
The relay workers and pepsi-keydisc@dane log this warning when they start,
and pepsi-status shows it in its “DNSSEC resolver” section. It means that
the resolver named there (/etc/resolv.conf, [pepsi] DNS_SERVERS, or a
stage’s DNS_SERVERS) answered a query for the signed ietf.org without
the DNSSEC AD bit. Pepsi trusts that bit instead of validating by itself, so
every DANE lookup finds nothing: outbound DANE never applies and DANE key
discovery finds no key, although nothing fails. Typical causes are an ISP or
DHCP resolver written into resolv.conf after setup, or systemd-resolved
without DNSSEC=yes. To fix it, point resolv.conf (or DNS_SERVERS)
at a validating resolver on this host, such as unbound
(Installation), or name a validating resolver that you trust as a
tls:// or https:// entry of [pepsi] DNS_SERVERS, and confirm with
pepsi-status. Alternatively, set DANE = off where DANE is not wanted.
“Could not be established” means that the probe got no answer at all. Check
that the resolver is reachable; for an encrypted entry, check that its port is
open and that its certificate is valid for the name after #.
10.1.13. The resolver “is not on this host”¶
The DANE workers log “trusts the DNSSEC AD bit from … which is not on this host
and is reached over plain DNS”, and pepsi-status and pepsi-setup run
name the same resolvers. The AD bit that those answers carry travels
unprotected, so anyone on the path can set it on a forged TLSA,
OPENPGPKEY or SMIMEA record. Either run a validating resolver on this
host (unbound, systemd-resolved with DNSSEC=yes), or reach a validating
resolver that you trust over an authenticated channel: [pepsi] DNS_SERVERS =
tls://192.0.2.53#dns.example.net (DNS over TLS) or https://…#NAME (DNS
over HTTPS), with the name that its certificate is issued for. A list may not mix
plain and encrypted entries.
10.1.14. Recipients see ... as the subject of encrypted mail¶
[stage-encrypt] PROTECT_HEADERS = yes (the default is no) moves the
real Subject: and the other RFC 5322 fields inside the encrypted and signed
part, and the outer subject reads .... A client that renders protected
headers (Thunderbird, for one) shows the real subject; one that does not shows
the placeholder. Where most recipients’ clients are of the second kind, remove
the setting or set PROTECT_HEADERS = no in the encrypt stage’s section (or
per sender through pepsi-settings). The signature then no longer covers To: and Cc:
for clients that ignore Pepsi’s own recipient attribute
(Who a signed and encrypted message was for).
10.1.15. A message from an S/MIME sender disappeared instead of bouncing¶
With ON_DECRYPT_FAILURE = bounce-if-unproblematic, ON_BAD_SIGNATURE =
bounce or the re-encrypt stage’s ON_NO_CLIENT_KEY = bounce, a message
that arrived as AES-CBC EnvelopedData is dropped, not bounced, because the
bounce would tell its sender whether a possibly modified ciphertext could be
opened. The
journal says so (dropped rather than bounced) and the telemetry counts
decrypt-dropped. passthrough, record and plaintext deliver such
a message instead (Bounces and replies to encrypted mail).
10.1.16. A user’s CA certificate does not arrive¶
Look at the user’s orders first:
pepsi-keys -c /etc/pepsi/pepsi.conf mpki order list --address alice@example.org
pepsi-keys -c /etc/pepsi/pepsi.conf mpki user list --domain example.org
No order at all means that the user is not wanted: under first-send they
have not sent mail through the server yet (or only from outside it), under
manual nobody selected them (mpki user set ADDRESS --select yes), or
their selection is no. Check also that the domain is enabled (mpki domain
show DOMAIN). A held order names its reason:
name-unconfirmedThe name came from the display name of the user’s first message, and the domain waits for an administrator to confirm such names. Check it in
mpki user list(marked?), thenmpki user set ADDRESS --confirm, or correct it with--given-name/--surname. The next pass releases the order.name-missingThe display name or the directory entry gave no given name and surname (a single word, an address, nothing). Enter both with
mpki user set, or import the user from the directory.daily-limitThe domain’s new enrolments of the last 24 hours reached its daily limit. Check that the orders are the ones you expect (an import of the wrong file or a script that sends as many addresses shows here), then release them one by one with
mpki order release ID, or raise the limit (mpki domain set DOMAIN --daily-limit N) and release them.no-licenceThe CA account has no free subscription licence for the address: the CA answered
insufficient_subscription. Pepsi cannot buy one. Add a licence in the CA’s portal (or free one by removing a departed user’s address from the subscription there), thenmpki order release ID. A revocation does not free a licence by itself.no-credentialThe domain has no readable
secrets.d/mpki-<domain>.secret. Store the API key withmpki credential set DOMAINand runmpki domain check DOMAIN; the next pass releases the order.
A failed order shows the reason under last error. A refusal by the CA
(HTTP 4xx with the CA’s message) is usually a setting that mpki domain check
DOMAIN also reports: an API key the CA does not accept, a product that the
account lacks, several organisations and no --organization-id, or a domain
that is not validated in the account. A user who is still wanted gets a new
order a day after the failed one, so until the cause is fixed each such user
costs one failed order and one notice a day. Once it is fixed, mpki order
retry ID tries again at the next pass instead of the next day. “the certificate does
not name …” or “is not over the key” means that the CA returned a certificate
that Pepsi refuses to install; report it to the CA. An order that stays
submitted waits at the CA, and mpki order list --json shows the CA’s
own status (ca_status). Pepsi keeps asking, at most once an hour, and
installs the certificate once it is issued; nothing here can speed this up.
Typical causes are an approval that the account requires in the CA’s portal,
and a CA status of pending while the CA has not finished validating the
sponsoring organisation for S/MIME. In CertCentral, GET
organization/{id}/validation then lists a validation of type smime with
status pending; the CA completes it, with whatever the portal asks of the
organisation.
10.1.17. Messages in failed or timeout¶
Most errors do not end a message at all, because they are this host’s fault rather than the message’s:
A lost database connection: the message is paused and retried every 30 seconds until the database is back, and each retry is logged under
pepsi-dispatch(“lost its database connection”).Any other error that a stage reports, such as a key, template, map or trust store that cannot be read, a helper that cannot be started, a database statement that is refused, or a service that does not answer: the message stays
pausedat its stage with the error instate.last_error, and is retried after one minute, then at doubling intervals up to an hour, for the stage’sMAX_LIFETIME(120 hours unless configured). The stage logs each retry (“failed at stage ‘…’ (attempt N); retrying in …s”). Fix the cause and the queue drains by itself;pepsi-statusshows the waiting messages meanwhile. WhenMAX_LIFETIMEruns out the message goes to the stage’sBOUNCE_STAGE, whose DSN tells the sender that a local error persisted.A stage whose configuration does not parse does not start at all: the dispatcher logs “the worker refused to start”, keeps the messages queued and tries again every few seconds. The stage’s own log line says what is wrong.
A worker that crashes or hangs past
MAX_RUNTIMEpauses the message it was working on and retries it a minute later; only the third time is the message given up on. In a delivery stage a worker killed atMAX_RUNTIMEis retried untilMAX_LIFETIMEinstead.A DSN with the subject “Outcome Unknown” comes from LMTP delivery: the MDA took the message three times without answering in time, so it may hold up to three copies. Look for a slow Sieve program or an overloaded MDA, and raise
DATA_TIMEOUTand that stage’sMAX_RUNTIMEif it is only slow.A milter that keeps deferring a message, or cannot be reached, holds it for
MAX_LIFETIMEas well, and then bounces it (or leaves itfailed); it never drops it through the filter’sREJECT_STAGE. A milter that defers only some recipients holds just their copy, as a separatepausedmessage at the milter stage, while the others are delivered.
A message ends in failed only when a stage found a defect in the message
itself, when a stage with no BOUNCE_STAGE ran out of retries, or when the
message crashed its worker three times. It ends in timeout when it made its
worker hang three times.
state.failure_class and state.last_error say which and why.
Delivery errors from the next server (a refusing or unreachable MX, a DNS lookup timing out) are retried by the relay stages themselves in the same way. A failed message is still often worth retrying once the cause is fixed.
pepsi.target runs pepsi-failure-bouncer --once every ten minutes
(pepsi-failure-bouncer.timer). It moves every message that has been
failed or timeout for [pepsi-failure-bouncer] MIN_AGE (an hour
unless configured) to the stage that [pepsi-failure-bouncer] BOUNCE_STAGE
names (bounce in the shipped configuration and in the wizard’s), and it logs
each message with the reason. That stage sends the sender a delivery-status
notification as their NOTIFY asks. A sender whose domain the message did not
authenticate gets no notification under BOUNCE_UNAUTHENTICATED = drop, and
the message is deleted, with a warning in the bounce stage’s log. A copy of a
mailing-list post is deleted rather than bounced. So a failed message stays in
the queue for about MIN_AGE; to retry it rather than bounce it, act within
that time:
pepsi-queue list --status failed # which messages, at which stage
pepsi-queue list 1234 --state-only # why: state.last_error, state.bounce, …
pepsi-queue set-stage 1234 relay # retry it from a stage of your choice
pepsi-failure-bouncer --once # or bounce all of them now
pepsi-queue delete 1234 # or discard it
The options of list go after the word list.
systemctl list-timers pepsi-failure-bouncer.timer shows when the next sweep
runs, and journalctl -u pepsi-failure-bouncer how many messages each one
moved.
10.1.18. Keeping failed messages for diagnosis¶
On a development or test host a failed message is usually the thing you want to
look at, and a bounce an hour later destroys it. Raise MIN_AGE, or turn the
sweep off there:
systemctl mask --now pepsi-failure-bouncer.timer
mask rather than disable: pepsi.target wants the timer, so a merely
disabled or stopped timer starts again with the target. Failed messages then
stay in the queue until you retry, bounce or delete them as above. To turn the
sweep back on:
systemctl unmask pepsi-failure-bouncer.timer
systemctl start pepsi-failure-bouncer.timer
The next sweep then bounces everything that failed in the meantime; delete what
you do not want bounced first. Do not leave the timer masked on a server that
handles real mail: nobody is told about a message that failed, neither the
sender nor you, unless you watch problem_total (see
Monitoring).
pepsi-failure-bouncer can instead run as a service that bounces each failed message as it happens. No unit is shipped for that mode; mask the timer before starting it from your own unit.
10.1.19. No core file after a crash¶
pepsi-httpd, pepsi-keys, pepsi-config export/import and
pepsi-setup apply hold long-lived secrets (the key-encryption key and the
private keys that it unwraps, the secure-link pepper, administrator password checks,
an archive password). They mark themselves non-dumpable at start, and their
systemd units set LimitCORE=0: a crash leaves the panic message and the
backtrace in the journal (RUST_BACKTRACE=1 in a drop-in adds the frames),
but no core file, and another process of the same account cannot attach a
debugger. The setuid pepsi-stage-encrypt and pepsi-stage-decrypt are
non-dumpable for the same reason. There is no option to turn this off; to
examine such a crash, reproduce it on a test host with your own build, run
under a debugger as root.
10.2. Frequently asked questions¶
- Does Pepsi keep a log of who mailed whom?
Not by default. A delivered message is deleted from the queue and only the journal mentions it.
[pepsi] MAIL_LOGturns on a per-message record; read the warning in pepsi.conf(5) first.- Where is the queue?
In PostgreSQL, table
pepsi.workqueue.pepsi-queueandpepsi-statusare the supported ways to look at it; plainSELECTworks too.- How do I see the pipeline this host runs?
pepsi-config dumpprints the effective configuration (--originalso shows which settings the database overrides), andpepsi-setup runprints, for each domain, how recipients are checked. The Wizard explains the pipelines that the wizard builds.- Can I go back to the previous release?
Not in place: the schema is migrated forward only. Restore the backup taken before the upgrade, then install the older packages; see Upgrading.
- Can Pepsi run next to Postfix or Exim?
Not both on port 25. Pepsi can import the other server’s configuration; see Installation.
- Why does the console show everything read-only?
The
pepsi-httpd-adminpackage, which carries out the console’s changes, is not installed or its socket is not running; see Debian packages.
10.3. Uninstalling¶
Debian packages. apt remove pepsi stops the services and removes the
programs; configuration, keys, the database and the service accounts stay, so a
later install carries on where it left off. apt purge pepsi removes the
rest of what the package installed but, on purpose, still keeps the
configuration, the database, the keys, the wallets and the accounts, and
prints the commands that remove them.
They are, as root:
runuser -u postgres -- dropdb pepsi
rm -rf /var/pepsi /var/lib/pepsi-wallets /etc/pepsi \
/var/lib/pepsi-dkim /var/lib/pepsi-dkim-public
for u in pepsi pepsi-ingress pepsi-httpd pepsi-owner pepsi-config \
pepsi-helper-token-refresh pepsi-keydisc pepsi-crypto \
pepsi-whitelist pepsi-wallets pepsi-dkim; do deluser "$u"; done
for g in pepsi-maildir pepsi-forward pepsi-token pepsi-admin \
pepsi-telemetry; do delgroup "$g"; done
The database roles of the same names can then be dropped with dropuser.
Remove pepsi-telemetry and pepsi-httpd-admin the same way if they are
installed.
Source installation. Stop and disable pepsi.target, then run make
uninstall with the same PREFIX/DESTDIR that you installed with. It removes
the programs, the shipped data and the systemd units, not the configuration,
keys or database; remove those as above.
Warning
Everything removed this way is gone for good, including the key-encryption key without which the stored private keys cannot be opened. Take a backup first (Backup and restore) if there is any chance you will want the keys, the mailing lists or their archive again.