95.1.45. pepsi-status

summarise the health of the Pepsi mail pipeline

Manual section:

1

95.1.45.1.1. Name

pepsi-status - print a human-readable health summary of the queue, stuck messages, delivery and failure statistics, and outbound TLS outcomes.

95.1.45.1.2. Synopsis

pepsi-status [GLOBAL-OPTIONS] [–json] [–limit N] [–offline]

95.1.45.1.3. Description

pepsi-status prints a one-shot, read-only health report of the Pepsi pipeline, drawn from the shared pepsi database and, for the published keys, from what the Web Key Directory and the DNS serve. It is intended to be run interactively or over SSH (for example from a monitoring host); it never modifies a row and issues no database notification, so it is safe to run against a live deployment at any time.

The tool connects to the same database as the other components, through the shared [pepsi-postgres] section; it introduces no configuration of its own (it reads [pepsi-dmarc-reports] WINDOW_DAYS for the DMARC section and [pepsi-dkim] PUBLIC_DIR for the DKIM section). When started as root it continues as the pepsi service account; see Running as root.

The report has these sections:

Queue

The current contents of the pepsi.workqueue table, broken down by stage and status, with the age of the oldest message in each bucket and a grand total. This is the live backlog: pending and running rows are work in progress, paused rows are waiting (for example for a payment deadline or a relay retry), and failed/timeout rows need attention. A line below the table counts the paused messages that a pepsi-stage-quarantine(1) stage holds: they wait for an operator (pepsi-queue(1) quarantine list), and they are not stuck.

Stuck messages

The individual messages whose status is failed or timeout, oldest first. For each, the report shows the workqueue_id, status, stage, age, envelope sender and subject. Where they are present, it also shows three reasons. The first is the structured reason why the next hop refused the message, when a relay stage recorded one (remote MTA, SMTP code, enhanced status and reply text, or our own diagnostic, from state.bounce). The second is the error text that the stage recorded when it failed or deferred the message (state.last_error, shown as error:). The third is the reason that pepsi-dispatch gave when it failed or timed out the message itself, such as a crashed or hung worker, or an unconfigured stage (state.dispatch_error, shown as dispatch:). In –json output the three are the bounce, last_error and dispatch_error fields of each problems entry. The list is capped by –limit; the true backlog size is always shown.

Delivery & failures

Cumulative pipeline counters: the total number of messages that completed and left the pipeline, the total stage executions, and per stage the number of executions it counted (the MESSAGES column, which is the denominator of the average), the average processing time, worker timeouts and worker crashes. These are lifetime totals accumulated since the schema was created. Because a successfully delivered message is deleted from the queue, there is no per-message delivery log and therefore no rolling time window: the figures are cumulative, not “in the last hour”.

Outbound TLS

Outbound TLS session outcomes over the last seven UTC report-days (today’s, in UTC, and the six before it, whatever the server’s time zone), grouped by policy domain and result type, with non-successful results marked. This data is only present when [pepsi-tlsrpt] SEND_REPORTS is enabled, and only for as long as pepsi-tlsrpt prune keeps it ([pepsi-tlsrpt] RETAIN_DAYS, also seven days by default), so a shorter retention shortens this section. The section then lists any MX-host addresses the resolver cache currently has marked as failing to connect, with the time of their last successful connection (or never).

Mailbox quotas

One line per local account that Pepsi has accounted for, oldest measurement first: usage, effective limit, how full it is, where the figure came from (fsquota/maildirsize/scan) and how long ago the mailbox was measured. Delivering to an account creates its row, so a row exists for every account that Pepsi has delivered to, whether or not it has a quota. On a deployment that does not use the feature the section still lists those accounts, with - in the LIMIT and FULL columns. It is empty, and says so, only when no local delivery has been accounted for.

LIMIT is the limit actually in force, resolved exactly as the delivery path resolves it: the account’s own limit if it has one, otherwise the site default [pepsi] MAILBOX_QUOTA, with an explicit 0 on the account meaning unlimited rather than inherit, and the kernel’s own quota tightening whichever won.

USED is the last measurement plus everything delivered since; it only ever grows, because nothing tells Pepsi when a user deletes mail over IMAP, so it is an upper bound rather than a fact. MEASURED is how long ago somebody actually looked, and it matters because a refusal at RCPT rests on a measurement being fresh: an account showing “full” against a very old measurement is one that pepsi-quota(1) reconcile should look at.

Correspondent key changes

How many correspondents’ encryption keys the Autocrypt newest-wins rule replaced in the last 30 days (rotated), and how many such replacements ACCEPT_ROTATION = expired refused (held). Then come the most recent rotations and holds, at most –limit of them, each with the date, the address, the protocol, the old and new fingerprints and whether the new key was the correspondent’s own (inbound) or introduced by a third party (gossip). A rotation is the one way in which a message that merely claims to come from a correspondent can change the key that we encrypt to them with. That is why it is shown here rather than left in the audit log. The figures are read from the key.peer.rotate and key.peer.rotate.held rows of pepsi.event_log, which the database writes together with the change itself, so the section reaches back as far as [pepsi-admin] EVENT_RETENTION keeps them. See pepsi-stage-autocrypt-learn(1).

DMARC aggregate reports

What the DMARC aggregate reports that pepsi-stage-dmarc-report(1) stored say about the last [pepsi-dmarc-reports] WINDOW_DAYS (7) days, per policy domain: how many messages the authenticated reports cover and from which organisations, and either that every source aligned (p=reject refuses none of them) or a WARNING with how many messages from how many sources did not. Then come the unaligned sources, at most –limit of them, each with its address, message count, From: domain, envelope domain, DKIM d=, disposition and reporters. A report whose mail passed neither SPF nor DMARC is counted as ignored and decides nothing.

Known-compromised-key list

When the badkeys list was last checked and which list it was, with a warning when it has never been fetched, has not been checked for three days, or the last update failed or was refused. See pepsi-keys(1) blocklist.

IP reputation

How many of the enabled IP blocklists (pepsi-reputation(1) iplists) are current, and a warning for each that ingress cannot use: one that is stale, never-loaded or no-rsync, or whose last download failed. Then the last self-check of this host against the DNS blocklists (pepsi-reputation check): how long ago it ran and how many lists could not be asked, a warning for every listing of one of this host’s addresses or domains, and a warning when it never ran or is three days old.

Greylisting

Shown when the deployment greylists (a [pepsi-greylist] section, or an ingress milter whose SOCKET is the daemon’s). Whether pepsi-greylist(1) answers a milter negotiation on its socket (asked before the switch to the service account, so that root may connect; an account outside the socket’s group is told it was not asked), then the counts of the last 24 hours: deferred, passed on retry, passed from another host (by rung and by how the host matched), never retried, known, bypassed by reason, exempt, denied, and the errors that let a recipient through, with a warning when there were any.

DKIM keys

What the signing daemon pepsi-dkim(1) reports in its public status file ([pepsi-dkim] PUBLIC_DIR/status.json; no key is read): how long ago it was written, whether rotation is on and every how many days, the DNS update method, and whether a switch waits for DNS. Then, per signing domain, the current, next and retiring keys, each with its selector, algorithm, RSA size and DNS state (visible, missing, disagree or unknown), the records to publish (a WARNING each) and the records that can be removed. A WARNING also marks a domain for which no key signs, a current key whose record is not visible (receivers fail its signatures until it is published), a rotation that has waited for the next key’s records, a daemon that signs for no domain, and a status file that is not written yet or cannot be read.

Published keys as served

For every address with a published, active end-to-end identity, what a correspondent is given: both Web Key Directory methods (openpgpkey.domain and the domain itself) and the OPENPGPKEY and SMIMEA records, fetched with the same hardened clients and [pepsi-keydiscovery] settings that key discovery uses, compared with the key store. The section says how many addresses are served exactly as stored and lists every channel of the others: MISMATCH (a fingerprint the store does not publish for the address is served; the address line then starts with WARNING), INCOMPLETE (a stored key is not served, or neither WKD method serves anything), FAILED (the channel could not be asked) and ABSENT. A DNS record that nobody published is not a fault. This is the one section that uses the network rather than the database; –offline skips it. The check runs from this host: a server that answers other askers differently is not detected.

RSA decryption keys

The active S/MIME identities whose private key Pepsi holds, which may decrypt (purpose encrypt or both), and whose certificate carries an RSA key – judged from the certificate, since an imported identity’s recorded algorithm name is free text. Inbound S/MIME to them arrives as RSA key transport and reaches an RSA private-key operation, the path that the remaining Bleichenbacher and Marvin exposure concerns; an identity with purpose = both (CRYPTO_SMIME_SHARED_KEY) is marked “signing and decryption share this key”. The section gives the count, lists the identities up to –limit, and ends with one INFO line: generating a P-256 identity with pepsi-keys identity generate makes new mail arrive as ECDH key agreement, while the RSA identity keeps decrypting what was sent to it. Holding an RSA identity is a deliberate configuration, so nothing here is a warning. (none: …) when there is no such identity.

DNSSEC resolver

The resolvers whose DNSSEC AD bit the DANE consumers trust (each relay stage with DANE not off, via its DNS_SERVERS, [pepsi] DNS_SERVERS or resolv.conf, and dane in [pepsi-keydiscovery] SOURCES), one line per resolver set with the consumers that use it. Each resolver is shown in its DNS_SERVERS spelling, which names the transport: a bare address is plain DNS, tls://… DNS over TLS, https://… DNS over HTTPS. Each set is asked for the signed ietf.org SOA: validates when the answer carries the AD bit, a warning when it does not (the resolver does not validate, so every DANE lookup finds nothing and DANE is configured but never applied) or when no answer comes. A further warning names every resolver of the set that is not on this host and is reached over plain DNS, since anyone on the path to it can forge the bit. –offline keeps the list and skips the probe. (nothing uses DANE) when no consumer is configured.

Own DNSSEC

Whether the DNSSEC chain of trust of this host’s own domains holds. The first resolver set of the DNSSEC resolver section (resolv.conf when nothing uses DANE) is asked, with the AD bit requested, for the MX records of every [pepsi-ingress] ACCEPTED_DOMAINS entry and, where an MX host publishes them, for its _25._tcp TLSA records. Each line says validated (AD bit set), insecure (no AD bit: the zone is unsigned or has no DS at the parent), bogus with the RFC 8914 Extended DNS Error the resolver gave (Signature Expired, DNSKEY Missing, DNSSEC Bogus …), servfail (a SERVFAIL naming no DNSSEC failure: bogus, or the zone’s servers unreachable) or no answer. bogus and servfail are WARNINGs: every validating resolver refuses the name, so senders behind one cannot find the MX and senders using DANE defer mail. An insecure TLSA RRset is a WARNING too, since DANE never uses it; an insecure MX is not, since signing is optional. The resolver must validate for the verdicts to mean anything (see the section above).

Then comes one line per zone that this host serves, listing the keys of its DNSKEY RRset (key tag, algorithm, KSK/ZSK, RSA size), or none (unsigned). These are the zones of the accepted domains, of [pepsi-ingress] HOSTNAME and of [pepsi-srs] SRS_DOMAIN, each found through its SOA record. A WARNING with the remedy follows for each key that uses an algorithm that RFC 8624 / RFC 9905 say not to sign with (RSAMD5, DSA, RSASHA1, DSA-NSEC3-SHA1, RSASHA1-NSEC3-SHA1, ECC-GOST; RSASHA512 is NOT RECOMMENDED), an RSA modulus below 2048 bits, a key that also signs another of these zones, and a key in the known-compromised-key list (pepsi.compromised_key). pepsi-setup check reports the same findings in its DNSKEY group. –offline skips the section.

S/MIME trust anchors

The enabled trust anchors that vouch for every mail domain: no scope set with pepsi-keys ca scope and no nameConstraints in the certificate. Each is a WARNING naming the anchor’s id and label (or subject), since a signature from any address that chains to it reads valid; the last line says how to scope it. (every enabled anchor is limited …) when there is none. In --json, unscoped_anchors.

Free-form text in the human-readable report — a remote MTA’s reply, a message subject, a host name from the resolver cache — has its control characters replaced by spaces before it is printed. All of it is chosen by somebody else, and this tool is meant to be run over SSH against a live deployment, which is precisely where a terminal escape sequence would repaint the operator’s screen. The --json output is unaffected: JSON escaping already covers it.

95.1.45.1.4. Options

–json

Emit the whole report as a single JSON object (keys queue, quarantined, problem_total, problems, global, stages, tls_report_days, tls, dns_failures, mailboxes, quota_default_bytes, key_change_days, key_changes_rotated, key_changes_held, key_changes, dmarc_window_days, dmarc, dmarc_unaligned, key_blocklist, reputation, greylist (summary with socket, probe_error, probe_denied, counts and never_retried, or an error), rsa_decryption, unscoped_anchors, dkim (path, the daemon’s status document as status and an error when it could not be read), resolvers and, unless –offline, served_keys and own_dnssec) instead of the human-readable text, for scripting and monitoring. served_keys.addresses holds one object per address with its channels, each carrying channel, verdict (ok, absent, failed, incomplete or mismatch), expected, served, unexpected and missing. resolvers holds one object per resolver set: source, users, servers, off_host, and validates (absent with –offline or when the probe got no answer, which error then explains). own_dnssec holds the servers asked and records, each with name, rtype (MX or TLSA), verdict (validated, insecure, absent, bogus, servfail or no answer) and an optional detail (the Extended DNS Error), then zones (one object per served zone: zone, names, keys) and key_findings (zone, key_tag, problem, detail).

–offline

Skip the Published keys as served and Own DNSSEC sections and the DNSSEC probe of the DNSSEC resolver section, the only parts that ask the network, for a report that touches nothing but the database and the local configuration.

–limit N

List at most N individual stuck (failed/timeout) messages, at most *N* mailbox-quota accounts, at most *N* correspondent key changes, at most *N* unaligned DMARC sources and at most *N* RSA decryption identities. Defaults to 50. The reported total backlog is unaffected by this cap.

The stuck-message section says “showing X of Y” when it is cut off; the mailbox-quota section does not, so on a host with more than N accounted accounts that list is silently the N with the oldest measurements. That is the useful end of the list, but raise –limit before concluding that an account is absent.

95.1.45.1.5. Running as root

Pepsi gives every component its own PostgreSQL role, authenticated over the local socket by the operating-system account it runs as, and root is deliberately not one of them. Rather than fail to connect and oblige you to remember sudo -u pepsi pepsi-status, the tool detects that it was started as root and becomes the unprivileged pepsi service account before it connects; the report itself stays strictly read-only.

The configuration file (and any @inline-secret@ fragment it references) is read before the switch, so a root-only configuration is still loaded normally.

If the pepsi account does not exist (an uninstalled source tree, a test rig), the identity is left untouched, a warning is logged, and the connection is attempted as the invoking user, so a setup in which root can reach the database keeps working.

95.1.45.1.6. Global Options

These options may appear before or after the other options.

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity. LOGLEVEL is one of error, warn, info, debug or trace. When the flag is absent the global [pepsi] LOG option applies, and failing that info.

-v, –verbose

Show log messages from all sources, including third-party libraries.

-h, –help

Print a usage summary and exit.

-V, –version

Print the version and exit.

95.1.45.1.7. Exit Status

0

Successful completion.

1

An error occurred: a malformed configuration file or a failed database connection or query. The reason is written to the log. A served-key check that cannot run is reported in its section, not as an exit status.

78

The database schema belongs to another Pepsi release (not installed, older, newer or modified); run pepsi-setup schema (see pepsi-setup(1)).

95.1.45.1.8. Files

When –config is not given, the first existing file from the following list is used. Every Pepsi component shares the same configuration file, so this is the same list each one searches:

  • $XDG_CONFIG_HOME/pepsi.conf

  • $HOME/.config/pepsi.conf

  • /etc/pepsi/pepsi.conf

  • /etc/pepsi.conf

95.1.45.1.9. Examples

Print the full health report:

pepsi-status -c /etc/pepsi/pepsi.conf

Collect the report as JSON for a monitoring system over SSH:

ssh mail.example.com pepsi-status --json

Show up to 200 stuck messages:

pepsi-status -c /etc/pepsi/pepsi.conf --limit 200

95.1.45.1.10. See Also

pepsi-queue(1), pepsi-tlsrpt(1), pepsi-quota(1), pepsi-dispatch(1), pepsi.conf(5), pepsi-setup(1)

95.1.45.1.11. Bugs

Report bugs to the Pepsi issue tracker.