70. pepsi-status

Summarise the health of the mail pipeline.

70.1. Role

pepsi-status is the operator’s read-only health overview of the pipeline. It connects through the shared [pepsi-postgres] section and has no section of its own. It never modifies a row or issues a notification, so it is safe to run over SSH against a live deployment. The other options that it reads include [pepsi] MAILBOX_QUOTA (the site default that an account with no row of its own inherits), [pepsi-dmarc-reports] WINDOW_DAYS and [pepsi-dkim] PUBLIC_DIR.

Three sections ask the network rather than the database: what the Web Key Directory and the DNS serve for the published identities (this section reads [pepsi-keydiscovery] for its timeouts and CA bundle), whether the resolver that the DANE consumers trust validates DNSSEC, and what it answers for the served domains. --offline skips the first and the last and lists the resolvers without probing them. When started as root, the program continues as the pepsi service account before connecting, so no sudo -u pepsi is needed. Reference: pepsi-status(1).

70.2. Features

  • Queue — the pepsi.workqueue backlog broken down by stage and status, with the age of the oldest message in each bucket.

  • Stuck messages — the individual failed/timeout messages, oldest first, with every reason recorded for them: the structured next-hop failure from state.bounce (remote MTA, SMTP code, enhanced status, reply text), the stage’s own state.last_error, and the dispatcher’s state.dispatch_error (a crashed or hung worker, an unconfigured stage); capped by --limit.

  • Delivery & failures — cumulative dispatch_stats/stage_stats counters: total messages processed, per-stage message counts, average processing time, worker timeouts and crashes. A warning reports how many running rows the dispatcher found with no worker holding them. It returns them by itself, but each one means that a write was lost, so look for database outages in its log.

  • Outbound TLS — tls_session outcomes of the last seven UTC report-days (today’s included) grouped by policy domain and result type (non-successful results marked), plus MX addresses that the resolver cache has flagged as failing to connect.

  • Mailbox quotas — every account with quota accounting, oldest measurement first (the order in which the reconcile sweep of pepsi-quota works), with its effective limit, the measured usage, the un-measured delivery estimate on top of it and the age of the measurement. An account appears here once something has been delivered to it, quota or no quota.

  • Correspondent key changes — how many correspondents’ keys the Autocrypt newest-wins rule replaced in the last 30 days and how many replacements ACCEPT_ROTATION = expired held back, with the most recent of both (address, old and new fingerprint, own key or gossip), read from the key.peer.rotate[.held] rows of pepsi.event_log. A rotation is the one way a message that merely claims to come from a correspondent can change the key that Pepsi encrypts to, which is why it is surfaced here (Keys of correspondents).

  • DMARC aggregate reports — per served domain, the verdict of the authenticated reports of the last WINDOW_DAYS that pepsi-stage-dmarc-report stored: every source aligned, so p=reject refuses none of them, or a warning naming the sources that did not (DMARC aggregate reports).

  • Known-compromised-key list — when the badkeys list was last checked, and whether the last update failed.

  • IP reputation — which of the IP blocklists that ingress consults are current (pepsi-reputation), and the last self-check of this host’s own addresses and domains against the DNS blocklists, every listing a warning.

  • DKIM keys — what pepsi-dkim(1) reports in its public status file: rotation on or off, and per signing domain the current, next and retiring keys with their DNS state, the records to publish or remove, and a rotation waiting for DNS. A current key whose record is not visible, an owed record, a delayed rotation, a domain that no key signs for and a status file that is missing are warnings. In --json it is dkim.

  • Published keys as served — both Web Key Directory methods and the OPENPGPKEY and SMIMEA records of every address with a published, active identity, fetched the way a correspondent would and compared with the key store (Checking what is actually served): how many addresses are served exactly as stored, and every channel of the others. A fingerprint that the store does not publish for the address is a warning (What the directory serves for our users). Skipped with --offline; in --json it is served_keys.

  • RSA decryption keys — the active S/MIME identities whose certificate is RSA and whose private key Pepsi holds for decryption, so that inbound S/MIME to them reaches an RSA private-key operation (Timing and microarchitectural Bleichenbacher oracles (Marvin and successors)); one with purpose = both is marked as sharing its key between signing and decryption. One INFO line says what to do: generate a P-256 identity (pepsi-keys identity generate) so that new mail uses ECDH, and keep the RSA one for decrypting old mail. In --json it is rsa_decryption.

  • DNSSEC resolver — every resolver whose AD bit a DANE consumer trusts (the relay stages with DANE on, pepsi-keydisc@dane), probed with the signed ietf.org SOA: a resolver that does not set the AD bit is a warning, because DANE is then configured but never applied, and so is one that is not on this host (the validating-resolver assumption of Security model). --offline lists the resolvers without probing; in --json it is resolvers.

  • Own DNSSEC — whether the chain of trust of the served domains holds: that resolver is asked for each accepted domain’s MX records and the _25._tcp TLSA records of its MX hosts, and says validated, insecure, bogus (with the RFC 8914 Extended DNS Error, such as Signature Expired) or servfail. bogus and servfail are warnings: validating resolvers then refuse the domain, so senders behind one cannot find its MX and senders using DANE defer mail to it. It also lists the DNSKEY keys of every zone that this host serves and warns about a deprecated algorithm, an RSA key below 2048 bits, a key shared between these zones and a known-compromised key. Skipped with --offline; in --json it is own_dnssec.

  • S/MIME trust anchors — every enabled trust anchor that vouches for every mail domain (no pepsi-keys ca scope and no nameConstraints) is a warning; in --json it is unscoped_anchors.

Text chosen by somebody else — a remote MTA’s reply, a subject, a host name — has its control characters replaced before it is printed, so a terminal escape sequence cannot repaint the operator’s screen over SSH.

--json emits the whole report as one JSON object for monitoring systems, and --limit caps the stuck-message, mailbox-quota, key-change, unaligned DMARC source and RSA decryption key lists.

70.3. A note on “recent successes”

A successfully delivered message is deleted from the queue, so there is no per-message delivery log. The delivery figures are therefore the cumulative counters accumulated since the schema was created — lifetime totals, not a rolling time window. Only the queue snapshot and the TLS section (which is keyed by report-day) reflect genuinely recent activity.

70.4. See also

pepsi-queue, pepsi-quota, pepsi-tlsrpt, pepsi-dispatch, Architecture, pepsi-status(1), pepsi.conf(5).