95.1.45. pepsi-status¶
summarise the health of the Pepsi mail pipeline
- Manual section:
1
95.1.45.1.1. Name¶
pepsi-status - print a human-readable health summary of the queue, stuck messages, delivery and failure statistics, and outbound TLS outcomes.
95.1.45.1.2. Synopsis¶
pepsi-status [GLOBAL-OPTIONS] [–json] [–limit N] [–offline]
95.1.45.1.3. Description¶
pepsi-status prints a one-shot, read-only health report of the Pepsi
pipeline, drawn from the shared pepsi database and, for the published keys,
from what the Web Key Directory and the DNS serve. It is intended to be
run interactively or over SSH (for example from a monitoring host); it never
modifies a row and issues no database notification, so it is safe to run against
a live deployment at any time.
The tool connects to the same database as the other components, through the
shared [pepsi-postgres] section; it introduces no configuration of its own
(it reads [pepsi-dmarc-reports] WINDOW_DAYS for the DMARC section and
[pepsi-dkim] PUBLIC_DIR for the DKIM section).
When started as root it continues as the pepsi service account; see
Running as root.
The report has these sections:
- Queue
The current contents of the
pepsi.workqueuetable, broken down bystageandstatus, with the age of the oldest message in each bucket and a grand total. This is the live backlog:pendingandrunningrows are work in progress,pausedrows are waiting (for example for a payment deadline or a relay retry), andfailed/timeoutrows need attention. A line below the table counts the paused messages that a pepsi-stage-quarantine(1) stage holds: they wait for an operator (pepsi-queue(1)quarantine list), and they are not stuck.- Stuck messages
The individual messages whose status is
failedortimeout, oldest first. For each, the report shows theworkqueue_id, status, stage, age, envelope sender and subject. Where they are present, it also shows three reasons. The first is the structured reason why the next hop refused the message, when a relay stage recorded one (remote MTA, SMTP code, enhanced status and reply text, or our own diagnostic, fromstate.bounce). The second is the error text that the stage recorded when it failed or deferred the message (state.last_error, shown aserror:). The third is the reason that pepsi-dispatch gave when it failed or timed out the message itself, such as a crashed or hung worker, or an unconfigured stage (state.dispatch_error, shown asdispatch:). In –json output the three are thebounce,last_erroranddispatch_errorfields of eachproblemsentry. The list is capped by –limit; the true backlog size is always shown.- Delivery & failures
Cumulative pipeline counters: the total number of messages that completed and left the pipeline, the total stage executions, and per stage the number of executions it counted (the MESSAGES column, which is the denominator of the average), the average processing time, worker timeouts and worker crashes. These are lifetime totals accumulated since the schema was created. Because a successfully delivered message is deleted from the queue, there is no per-message delivery log and therefore no rolling time window: the figures are cumulative, not “in the last hour”.
- Outbound TLS
Outbound TLS session outcomes over the last seven UTC report-days (today’s, in UTC, and the six before it, whatever the server’s time zone), grouped by policy domain and result type, with non-successful results marked. This data is only present when
[pepsi-tlsrpt] SEND_REPORTSis enabled, and only for as long aspepsi-tlsrpt prunekeeps it ([pepsi-tlsrpt] RETAIN_DAYS, also seven days by default), so a shorter retention shortens this section. The section then lists any MX-host addresses the resolver cache currently has marked as failing to connect, with the time of their last successful connection (ornever).- Mailbox quotas
One line per local account that Pepsi has accounted for, oldest measurement first: usage, effective limit, how full it is, where the figure came from (
fsquota/maildirsize/scan) and how long ago the mailbox was measured. Delivering to an account creates its row, so a row exists for every account that Pepsi has delivered to, whether or not it has a quota. On a deployment that does not use the feature the section still lists those accounts, with-in the LIMIT and FULL columns. It is empty, and says so, only when no local delivery has been accounted for.LIMIT is the limit actually in force, resolved exactly as the delivery path resolves it: the account’s own limit if it has one, otherwise the site default
[pepsi] MAILBOX_QUOTA, with an explicit0on the account meaning unlimited rather than inherit, and the kernel’s own quota tightening whichever won.USED is the last measurement plus everything delivered since; it only ever grows, because nothing tells Pepsi when a user deletes mail over IMAP, so it is an upper bound rather than a fact. MEASURED is how long ago somebody actually looked, and it matters because a refusal at
RCPTrests on a measurement being fresh: an account showing “full” against a very old measurement is one that pepsi-quota(1)reconcileshould look at.- Correspondent key changes
How many correspondents’ encryption keys the Autocrypt newest-wins rule replaced in the last 30 days (rotated), and how many such replacements
ACCEPT_ROTATION = expiredrefused (held). Then come the most recent rotations and holds, at most –limit of them, each with the date, the address, the protocol, the old and new fingerprints and whether the new key was the correspondent’s own (inbound) or introduced by a third party (gossip). A rotation is the one way in which a message that merely claims to come from a correspondent can change the key that we encrypt to them with. That is why it is shown here rather than left in the audit log. The figures are read from thekey.peer.rotateandkey.peer.rotate.heldrows ofpepsi.event_log, which the database writes together with the change itself, so the section reaches back as far as[pepsi-admin] EVENT_RETENTIONkeeps them. See pepsi-stage-autocrypt-learn(1).- DMARC aggregate reports
What the DMARC aggregate reports that pepsi-stage-dmarc-report(1) stored say about the last
[pepsi-dmarc-reports] WINDOW_DAYS(7) days, per policy domain: how many messages the authenticated reports cover and from which organisations, and either that every source aligned (p=rejectrefuses none of them) or aWARNINGwith how many messages from how many sources did not. Then come the unaligned sources, at most –limit of them, each with its address, message count,From:domain, envelope domain, DKIMd=, disposition and reporters. A report whose mail passed neither SPF nor DMARC is counted as ignored and decides nothing.- Known-compromised-key list
When the badkeys list was last checked and which list it was, with a warning when it has never been fetched, has not been checked for three days, or the last update failed or was refused. See pepsi-keys(1)
blocklist.- IP reputation
How many of the enabled IP blocklists (pepsi-reputation(1)
iplists) are current, and a warning for each that ingress cannot use: one that isstale,never-loadedorno-rsync, or whose last download failed. Then the last self-check of this host against the DNS blocklists (pepsi-reputation check): how long ago it ran and how many lists could not be asked, a warning for every listing of one of this host’s addresses or domains, and a warning when it never ran or is three days old.- Greylisting
Shown when the deployment greylists (a
[pepsi-greylist]section, or an ingress milter whoseSOCKETis the daemon’s). Whether pepsi-greylist(1) answers a milter negotiation on its socket (asked before the switch to the service account, so that root may connect; an account outside the socket’s group is told it was not asked), then the counts of the last 24 hours: deferred, passed on retry, passed from another host (by rung and by how the host matched), never retried, known, bypassed by reason, exempt, denied, and the errors that let a recipient through, with a warning when there were any.- DKIM keys
What the signing daemon pepsi-dkim(1) reports in its public status file (
[pepsi-dkim] PUBLIC_DIR/status.json; no key is read): how long ago it was written, whether rotation is on and every how many days, the DNS update method, and whether a switch waits for DNS. Then, per signing domain, thecurrent,nextandretiringkeys, each with its selector, algorithm, RSA size and DNS state (visible,missing,disagreeorunknown), the records to publish (a WARNING each) and the records that can be removed. A WARNING also marks a domain for which no key signs, a current key whose record is not visible (receivers fail its signatures until it is published), a rotation that has waited for the next key’s records, a daemon that signs for no domain, and a status file that is not written yet or cannot be read.- Published keys as served
For every address with a published, active end-to-end identity, what a correspondent is given: both Web Key Directory methods (
openpgpkey.domain and the domain itself) and theOPENPGPKEYandSMIMEArecords, fetched with the same hardened clients and[pepsi-keydiscovery]settings that key discovery uses, compared with the key store. The section says how many addresses are served exactly as stored and lists every channel of the others:MISMATCH(a fingerprint the store does not publish for the address is served; the address line then starts withWARNING),INCOMPLETE(a stored key is not served, or neither WKD method serves anything),FAILED(the channel could not be asked) andABSENT. A DNS record that nobody published is not a fault. This is the one section that uses the network rather than the database; –offline skips it. The check runs from this host: a server that answers other askers differently is not detected.- RSA decryption keys
The active S/MIME identities whose private key Pepsi holds, which may decrypt (
purposeencryptorboth), and whose certificate carries an RSA key – judged from the certificate, since an imported identity’s recorded algorithm name is free text. Inbound S/MIME to them arrives as RSA key transport and reaches an RSA private-key operation, the path that the remaining Bleichenbacher and Marvin exposure concerns; an identity withpurpose = both(CRYPTO_SMIME_SHARED_KEY) is marked “signing and decryption share this key”. The section gives the count, lists the identities up to –limit, and ends with one INFO line: generating a P-256 identity withpepsi-keys identity generatemakes new mail arrive as ECDH key agreement, while the RSA identity keeps decrypting what was sent to it. Holding an RSA identity is a deliberate configuration, so nothing here is a warning.(none: …)when there is no such identity.- DNSSEC resolver
The resolvers whose DNSSEC AD bit the DANE consumers trust (each relay stage with
DANEnotoff, via itsDNS_SERVERS,[pepsi] DNS_SERVERSorresolv.conf, anddanein[pepsi-keydiscovery] SOURCES), one line per resolver set with the consumers that use it. Each resolver is shown in itsDNS_SERVERSspelling, which names the transport: a bare address is plain DNS,tls://…DNS over TLS,https://…DNS over HTTPS. Each set is asked for the signedietf.orgSOA:validateswhen the answer carries the AD bit, a warning when it does not (the resolver does not validate, so every DANE lookup finds nothing and DANE is configured but never applied) or when no answer comes. A further warning names every resolver of the set that is not on this host and is reached over plain DNS, since anyone on the path to it can forge the bit. –offline keeps the list and skips the probe.(nothing uses DANE)when no consumer is configured.- Own DNSSEC
Whether the DNSSEC chain of trust of this host’s own domains holds. The first resolver set of the DNSSEC resolver section (
resolv.confwhen nothing uses DANE) is asked, with the AD bit requested, for theMXrecords of every[pepsi-ingress] ACCEPTED_DOMAINSentry and, where an MX host publishes them, for its_25._tcpTLSA records. Each line saysvalidated(AD bit set),insecure(no AD bit: the zone is unsigned or has no DS at the parent),boguswith the RFC 8914 Extended DNS Error the resolver gave (Signature Expired,DNSKEY Missing,DNSSEC Bogus…),servfail(a SERVFAIL naming no DNSSEC failure: bogus, or the zone’s servers unreachable) orno answer.bogusandservfailare WARNINGs: every validating resolver refuses the name, so senders behind one cannot find the MX and senders using DANE defer mail. AninsecureTLSA RRset is a WARNING too, since DANE never uses it; aninsecureMX is not, since signing is optional. The resolver must validate for the verdicts to mean anything (see the section above).Then comes one line per zone that this host serves, listing the keys of its
DNSKEYRRset (key tag, algorithm,KSK/ZSK, RSA size), ornone (unsigned). These are the zones of the accepted domains, of[pepsi-ingress] HOSTNAMEand of[pepsi-srs] SRS_DOMAIN, each found through itsSOArecord. A WARNING with the remedy follows for each key that uses an algorithm that RFC 8624 / RFC 9905 say not to sign with (RSAMD5, DSA, RSASHA1, DSA-NSEC3-SHA1, RSASHA1-NSEC3-SHA1, ECC-GOST; RSASHA512 is NOT RECOMMENDED), an RSA modulus below 2048 bits, a key that also signs another of these zones, and a key in the known-compromised-key list (pepsi.compromised_key). pepsi-setup check reports the same findings in itsDNSKEYgroup. –offline skips the section.- S/MIME trust anchors
The enabled trust anchors that vouch for every mail domain: no scope set with
pepsi-keys ca scopeand nonameConstraintsin the certificate. Each is a WARNING naming the anchor’s id and label (or subject), since a signature from any address that chains to it readsvalid; the last line says how to scope it.(every enabled anchor is limited …)when there is none. In--json,unscoped_anchors.
Free-form text in the human-readable report — a remote MTA’s reply, a message
subject, a host name from the resolver cache — has its control characters replaced
by spaces before it is printed. All of it is chosen by somebody else, and this tool
is meant to be run over SSH against a live deployment, which is precisely where a
terminal escape sequence would repaint the operator’s screen. The --json output
is unaffected: JSON escaping already covers it.
95.1.45.1.4. Options¶
- –json
Emit the whole report as a single JSON object (keys
queue,quarantined,problem_total,problems,global,stages,tls_report_days,tls,dns_failures,mailboxes,quota_default_bytes,key_change_days,key_changes_rotated,key_changes_held,key_changes,dmarc_window_days,dmarc,dmarc_unaligned,key_blocklist,reputation,greylist(summarywithsocket,probe_error,probe_denied,countsandnever_retried, or anerror),rsa_decryption,unscoped_anchors,dkim(path, the daemon’s status document asstatusand anerrorwhen it could not be read),resolversand, unless –offline,served_keysandown_dnssec) instead of the human-readable text, for scripting and monitoring.served_keys.addressesholds one object per address with itschannels, each carryingchannel,verdict(ok,absent,failed,incompleteormismatch),expected,served,unexpectedandmissing.resolversholds one object per resolver set:source,users,servers,off_host, andvalidates(absent with –offline or when the probe got no answer, whicherrorthen explains).own_dnssecholds theserversasked andrecords, each withname,rtype(MXorTLSA),verdict(validated,insecure,absent,bogus,servfailorno answer) and an optionaldetail(the Extended DNS Error), thenzones(one object per served zone:zone,names,keys) andkey_findings(zone,key_tag,problem,detail).- –offline
Skip the Published keys as served and Own DNSSEC sections and the DNSSEC probe of the DNSSEC resolver section, the only parts that ask the network, for a report that touches nothing but the database and the local configuration.
- –limit N
List at most N individual stuck (
failed/timeout) messages, at most *N* mailbox-quota accounts, at most *N* correspondent key changes, at most *N* unaligned DMARC sources and at most *N* RSA decryption identities. Defaults to50. The reported total backlog is unaffected by this cap.The stuck-message section says “showing X of Y” when it is cut off; the mailbox-quota section does not, so on a host with more than N accounted accounts that list is silently the N with the oldest measurements. That is the useful end of the list, but raise –limit before concluding that an account is absent.
95.1.45.1.5. Running as root¶
Pepsi gives every component its own PostgreSQL role, authenticated over the local
socket by the operating-system account it runs as, and root is deliberately
not one of them. Rather than fail to connect and oblige you to remember
sudo -u pepsi pepsi-status, the tool detects that it was started as root
and becomes the unprivileged pepsi service account before it connects; the
report itself stays strictly read-only.
The configuration file (and any @inline-secret@ fragment it references) is
read before the switch, so a root-only configuration is still loaded normally.
If the pepsi account does not exist (an uninstalled source tree, a test rig),
the identity is left untouched, a warning is logged, and the connection is
attempted as the invoking user, so a setup in which root can reach the
database keeps working.
95.1.45.1.6. Global Options¶
These options may appear before or after the other options.
- -c FILE, –config FILE
Read the configuration from FILE instead of searching the default locations.
- -L LOGLEVEL, –log LOGLEVEL
Set the logging verbosity. LOGLEVEL is one of
error,warn,info,debugortrace. When the flag is absent the global[pepsi] LOGoption applies, and failing thatinfo.- -v, –verbose
Show log messages from all sources, including third-party libraries.
- -h, –help
Print a usage summary and exit.
- -V, –version
Print the version and exit.
95.1.45.1.7. Exit Status¶
- 0
Successful completion.
- 1
An error occurred: a malformed configuration file or a failed database connection or query. The reason is written to the log. A served-key check that cannot run is reported in its section, not as an exit status.
- 78
The database schema belongs to another Pepsi release (not installed, older, newer or modified); run
pepsi-setup schema(see pepsi-setup(1)).
95.1.45.1.8. Files¶
When –config is not given, the first existing file from the following list is used. Every Pepsi component shares the same configuration file, so this is the same list each one searches:
$XDG_CONFIG_HOME/pepsi.conf$HOME/.config/pepsi.conf/etc/pepsi/pepsi.conf/etc/pepsi.conf
95.1.45.1.9. Examples¶
Print the full health report:
pepsi-status -c /etc/pepsi/pepsi.conf
Collect the report as JSON for a monitoring system over SSH:
ssh mail.example.com pepsi-status --json
Show up to 200 stuck messages:
pepsi-status -c /etc/pepsi/pepsi.conf --limit 200
95.1.45.1.10. See Also¶
pepsi-queue(1), pepsi-tlsrpt(1), pepsi-quota(1), pepsi-dispatch(1), pepsi.conf(5), pepsi-setup(1)
95.1.45.1.11. Bugs¶
Report bugs to the Pepsi issue tracker.