27. The message state¶
Every message in flight is one row of the pepsi.workqueue table. Alongside the
envelope and the body, each row carries a free-form JSON object that travels
with the message from ingress to its terminal stage: the state column
(PostgreSQL JSONB). The exhaustive key reference is the
pepsi.state(7) man page; this chapter is the narrative companion.
27.1. The state is the only channel¶
Stages do not talk to each other directly. The pipeline is a single-table state
machine (see Architecture): a stage loads a row, does its work, and writes
exactly one terminal result. The state object is the only place a stage
can leave information for a later stage to read. Ingress records what it learned
about the connection and the sender’s intent; the ARC stage records the verdict
of the chain that it re-evaluated; a relay stage records why a delivery failed
so that the bounce stage can explain it. None of this lives in the message
itself — it lives in state.
Because state is plain JSON with no fixed schema, the set of keys is
open-ended. A custom stage (see Extending the Pipeline) may invent its own keys; the
only contract is the merge rule below.
27.2. Merge semantics¶
state is built up incrementally rather than overwritten. Every terminal
stage helper that mutates a row merges its new keys into the existing
object, so a key written early survives unless a stage deliberately replaces
that specific key. The merge is a shallow, key-wise object merge with exactly
PostgreSQL’s state || $new semantics. The worker folds it in for the
single-row commit, and the fan-out functions (workqueue_split,
workqueue_pause, workqueue_finish_with_clones) evaluate it in SQL. That
is why the origin provenance that ingress seeds is still readable by a relay
stage at the far end of a long pipeline, and why the RFC 3461 dsn parameters
reach the bounce stage intact.
There is exactly one exception. When pepsi-stage-bounce
rewrites a message into a delivery-status notification, it replaces state
with a lone retry_since, the DSN’s own retry clock, plus the
tls_required tag of a REQUIRETLS message. The reason is that the bounce
is a brand-new null-sender message that inherits none of the original’s
provenance.
27.3. What ingress seeds¶
When pepsi-ingress accepts a message it seeds three families of
keys, plus conditional ones (dsn and spam below; tls_required for
REQUIRETLS or TLS-Required: No; and mime_anomalies,
mime_normalise and mime_normalised for ambiguous MIME, see
pepsi.state(7)):
originThe SMTP-origin provenance of the delivering connection — client IP, HELO/EHLO, the ESMTP/SMTPUTF8/
BODY=declarations, TLS parameters, the listener, reverse-DNS/iprev, and this receiver’sauthserv_id. The ARC stage needs theauthserv_idto reproduce itsAuthentication-Resultsidentity, and the relay stages consult the declarations when deciding whether to downgrade 8-bit/UTF-8 content for a given next hop. It also records how a submission authenticated (auth_methodand friends) andfrom_bound: whether theFrom:is exactly one mailbox bound to the account by a concrete, wildcard-freeUSERNAME_MAPentry (or its defaultlogin@HOSTNAME). The encrypt stage lets a submission register a key or run an e-mail key command only when it came from a trusted relay orfrom_boundis true.authThe inbound SPF/DKIM/DMARC verdicts (also reflected in the prepended
Authentication-Resultsheader). Thearcmember starts asnoneand is overwritten by the ARC stage. On mail from an unauthenticated session whoseFrom:display name names an address other than its own,auth.display_name_mismatchholds that claimed address, and ingress prependsX-Pepsi-Warning: display-name-claims-other-address(see What a security indicator means).auth.warningslists everyX-Pepsi-Warningvalue that ingress wrote: that one,from-has-bidi-or-invisible-characters(bidi controls or invisible characters inFrom:),from-domain-resembles-served-domain(aFrom:domain that looks like a served one),attachment-name-has-bidi-or-invisible-characters(such characters in an attachment’s file name),duplicate-header-<field>(differing copies ofDate,Subject,To,Cc,Reply-ToorSenderthat were accepted), and the four that also earn the[external]subject tag:from-ambiguous(aFrom:that reads more than one way or names several domains),served-domain-not-authenticated,served-sender-not-authenticated(aSender:at a served domain that nobody authenticated) anddmarc-fail-not-enforced.local_originA boolean:
truewhen the delivering session was authenticated (inMYNETWORKS, via SASLAUTH, with a matching TLS client certificate, or as a peer on a UNIX socket withAUTH_PEERCRED, the default there, whose uid resolved to a permitted login, which is how every locally submitted message arrives). It distinguishes outbound mail from a trusted sender from inbound mail off the Internet, and the per-address settings layer and the edit-settings control channel key on it.dsnThe RFC 3461 parameters that the sender requested: message-level
ret/envidand a per-recipient array ofnotify/orcptparallel to the recipient list, which also carries each recipient’s RFC 7293rrvsrequest. Every stage must preserve and honour it. The key is present only when the sender asked for something.spamSeeded as
false, and only in one case: a message addressed solely to the reserved<Postmaster>mailbox, which RFC 5321 §4.5.1 requires to stay deliverable. The language, confirm-to-send and pay-to-send gates forward any message already marked not-spam, so this is what keeps that mailbox reachable. Restricting it to the sole-recipient case is deliberate: otherwise a spammer could whitelist co-recipients by adding a<Postmaster>RCPTalongside a real victim.
27.4. What the stages write¶
As the message advances, stages merge in their own keys:
the ARC stage overwrites
auth.arcwith the verdict of the chain that it re-evaluated;any stage routing a message to its
BOUNCE_STAGE(the delivery and discard stages, and policy stages such as the milter, secretary and crypto stages) writes abounceobject, which pepsi-stage-bounce turns into the DSN: itskind(permanent/success/delay), the human-readablediagnosticandfailed_recipient, the copiednotify/orcpt/envid, and structured next-hop detail (remote_mta/smtp_code/enhanced_status/phase/reply_text);the relay stages keep retry scratch (
attempts,last_error,delay_sent);the stage scaffolding itself, when a stage’s error was not marked permanent, pauses the row with
last_error,temporary_failuresandtemporary_stage(which drive the back-off), and when it gives up recordsfailed_stageandfailure_class(permanent,retries-exhausted,crashedortimed-out), from which the bounce stage builds a DSN with a generic text (last_erroris never copied into a DSN). The dispatcher counts a message’s worker crashes and hangs instrikes;a crypto stage that parks a message on a key lookup records under
keydiscwhat it waits for. Thepepsi-keydiscservice that settles the request matches on this key to release the message;a reply typed into the The secure-link fallback portal portal is injected with a
secure_linkobject and deliberately withoutlocal_origin, which is what keeps the portal from being an open relay;pepsi-stage-srs records the envelope sender that it replaced under
srs.original, so that a DSN originated by this deployment goes to the real sender rather than to its own SRS alias. When pepsi-ingress makes a row from a remote report about a forwarded copy, it marks that row withforward_return(Forwarding addresses in delivery reports);pepsi-stage-list and the list stages after it carry the mailing-list descriptor under
list;pepsi-stage-detect-language records the detected
languagethat pepsi-stage-block-language(1) then scores;pepsi-stage-check-whitelist and pepsi-stage-anti-spam cooperate through the
spam/paidshort-circuit flags and thepay_deadline;pepsi-stage-secretary records under
secretarythe challenge that a held message waits on (the cookie, itsdeadline, thewhitelist), marks itconfirmedorabandoned, and on release setsspam = false; the timeout path removes the key again;pepsi-stage-dot-forward records a forwarded row’s chain of forwarding logins in
dot-forwardersto break~/.forwardforwarding loops;pepsi-stage-milter records under
milterwhich stage ran the filter, theverdictit returned and, when it went wrong, theerror;pepsi-stage-vacation records under
vacationthat it answered a message on the recipient’s behalf, and in which language. Nothing reads this key; it exists so that a surprising out-of-office reply is explainable from the row;pepsi-stage-encrypt records what it signed and encrypted, per recipient, under
crypto.out(withfingerprintsandskippedwhen a local recipient’s several MUA keys were, or were not, all encrypted to), and setsencryptedwhen a recipient’s message really was encrypted; pepsi-stage-auto-whitelist copies this flag into itssignature_requiredcolumn.crypto.out.key_attachedsays that the sender’s key went out as a file, andcrypto.out.client_protectedsays that the user’s own mail client had already encrypted the message, so the stage left it alone;pepsi-stage-decrypt records the mirror image under
crypto.in: whether the message arrived encrypted, what became of the ciphertext (decryption = for-client, withfor_clientandclient_fingerprint, when it was encrypted to the recipient’s own MUA key and passed on unopened;withheldwhen an unsigned plaintext from a malleable S/MIME container was not opened for the user;unauthenticatedandsanitisedwhen such a plaintext was delivered as text), which of our identities opened it, the signature verdict (withcovers_plaintext),usually_signedwhen a message without avalidsignature comes from an address whose mail recently carried one, and the ordered layer list. It setssignature_verifiedfor thevalidverdict and only for that one, and pepsi-stage-check-whitelist gates asignature_requiredrow on this flag. Two of those members exist because nothing downstream could recover them: the plaintext is committed over the arriving bytes, socrypto.in.encryptedandcrypto.in.outer_gossip(whether the arriving header block already carried anAutocrypt-Gossip:field) are only answerable here. pepsi-stage-bounce readscrypto.inbefore it clears the state, and quotes only the addressing fields of a message that arrived encrypted, to avoid returning decrypted content to the envelope sender (Bounces after gateway decryption reveal decrypted content). pepsi-stage-autocrypt-learn reads both. Their names live inpepsi_common::crypto_state, so that neither crypto stage owns the vocabulary;pepsi-stage-reencrypt records under
crypto.storewhat it did with a message that decrypt opened:outcomereencrypted(with theprotocol, the MUA-keyfingerprintsthat it sealed to, of whichfingerprintis the first, any keysskippedwith a reason,containeranddowngraded),plaintextorbounced(with areason). It merges the wholecryptoobject, carryingcrypto.inalong, and the key’s presence is what keeps it from ever sealing a row twice;the dispatcher writes
dispatch_errorwhen it forces a row tofailed/timeoutbecause a stage crashed or ran too long.
Both crypto stages live under the same crypto key with the same field names,
and both are a shallow merge, so a row carries crypto.out or
crypto.in (plus crypto.store, which the re-encryption stage writes
alongside it) and not both — which is correct, since a message is on the outbound
path or the inbound one. The full list, with the exact JSON shapes and the
producing/consuming stage for each key, is in pepsi.state(7).