23. The message state

Every message in flight is one row of the pepsi.workqueue table. Alongside the envelope and the body, each row carries a free-form JSON object — the state column (PostgreSQL JSONB) — that travels with the message from ingress to its terminal stage. The exhaustive key reference is the pepsi.state(7) man page; this chapter is the narrative companion.

23.1. The state is the only channel

Stages do not talk to each other directly. The pipeline is a single-table state machine (see Architecture): a stage loads a row, does its work, and writes exactly one terminal result. The state object is the only place a stage can leave information for a later stage to read. Ingress records what it learned about the connection and the sender’s intent; the ARC stage records the verdict of the chain it re-evaluated; a relay stage records why a delivery failed so the bounce stage can explain it. None of this lives in the message itself — it lives in state.

Because state is plain JSON with no fixed schema, the set of keys is open-ended. A custom stage (see Extending the Pipeline) may invent its own keys; the only contract is the merge rule below.

23.2. Merge semantics

state is built up incrementally rather than overwritten. Every terminal stage helper that mutates a row merges its new keys into the existing object — a shallow, key-wise object merge with exactly PostgreSQL’s state || $new semantics, folded in the worker for the single-row commit and evaluated in SQL by the fan-out functions (workqueue_split, workqueue_pause, workqueue_finish_with_clones) — so a key written early survives unless a stage deliberately replaces that specific key. That is why the origin provenance seeded by ingress is still readable by a relay stage at the far end of a long pipeline, and why the RFC 3461 dsn parameters reach the bounce stage intact.

There is exactly one exception: pepsi-stage-bounce clears state to null when it rewrites a message into a delivery-status notification, because the bounce is a brand-new null-sender message that inherits none of the original’s provenance.

23.3. What ingress seeds

When pepsi-ingress accepts a message it seeds three families of keys, plus two conditional ones (dsn and spam):

origin

The SMTP-origin provenance of the delivering connection — client IP, HELO/EHLO, the ESMTP/SMTPUTF8/BODY= declarations, TLS parameters, the listener, reverse-DNS/iprev, and this receiver’s authserv_id. The ARC stage needs the authserv_id to reproduce its Authentication-Results identity, and the relay stages consult the declarations when deciding whether to downgrade 8-bit/UTF-8 content for a given next hop. It also records how a submission authenticated (auth_method and friends) and from_bound: whether the From: is exactly one mailbox bound to the account by a concrete, wildcard-free USERNAME_MAP entry (or its default login@HOSTNAME). The encrypt stage lets a submission register a key or run an e-mail key command only when it came from a trusted relay or from_bound is true.

auth

The inbound SPF/DKIM/DMARC verdicts (also reflected in the prepended Authentication-Results header). The arc member starts as none and is overwritten by the ARC stage.

local_origin

A boolean: true when the delivering session was authenticated (in MYNETWORKS, via SASL AUTH, with a matching TLS client certificate, or — on a UNIX socket with AUTH_PEERCRED, the default there — as a peer whose uid resolved to a permitted login, which is how every locally submitted message arrives). It distinguishes outbound mail from a trusted sender from inbound mail off the Internet, and the per-address settings layer and the edit-settings control channel key on it.

dsn

The RFC 3461 parameters the sender requested: message-level ret/envid and a per-recipient array of notify/orcpt parallel to the recipient list. Every stage must preserve and honour it. Present only when the sender asked for something.

spam

Seeded as false, and only in one case: a message addressed solely to the reserved <Postmaster> mailbox, which RFC 5321 §4.5.1 requires to stay deliverable. The language, confirm-to-send and pay-to-send gates forward any message already marked not-spam, so this is what keeps that mailbox reachable. Restricting it to the sole-recipient case is deliberate: otherwise a spammer could whitelist co-recipients by adding a <Postmaster> RCPT alongside a real victim.

23.4. What the stages write

As the message advances, stages merge in their own keys:

  • the ARC stage overwrites auth.arc with the verdict of the chain it re-evaluated;

  • any stage routing a message to its BOUNCE_STAGE (the delivery and discard stages, and policy stages such as the milter, secretary and crypto stages) writes a bounce object — its kind (permanent/success/delay), the human-readable diagnostic and failed_recipient, the copied notify/orcpt/envid, and structured next-hop detail (remote_mta/smtp_code/enhanced_status/phase/reply_text) — which pepsi-stage-bounce turns into the DSN;

  • the relay stages keep retry scratch (attempts, last_error, delay_sent);

  • pepsi-stage-srs records the envelope sender it replaced under srs.original, so a DSN this deployment originates goes to the real sender rather than to its own SRS alias;

  • pepsi-stage-list and the list stages after it carry the mailing-list descriptor under list;

  • pepsi-stage-detect-language records the detected language that pepsi-stage-block-language(1) then scores;

  • pepsi-stage-check-whitelist and pepsi-stage-anti-spam cooperate through the spam/paid short-circuit flags and the pay_deadline;

  • pepsi-stage-secretary records the challenge a held message waits on under secretary (the cookie, its deadline, the whitelist), marks it confirmed or abandoned, and on release sets spam = false; the timeout path removes the key again;

  • pepsi-stage-dot-forward records a forwarded row’s chain of forwarding logins in dot-forwarders to break ~/.forward forwarding loops;

  • pepsi-stage-milter records under milter which stage ran the filter, the verdict it returned and, when it went wrong, the error;

  • pepsi-stage-vacation records under vacation that it answered a message on the recipient’s behalf, and in which language — read by nobody, so that a surprising out-of-office reply is explainable from the row;

  • pepsi-stage-encrypt records what it signed and encrypted, per recipient, under crypto.out, and sets encrypted when a recipient’s message really was encrypted — the flag pepsi-stage-auto-whitelist copies into its signature_required column. crypto.out.key_attached says the sender’s key went out as a file, and crypto.out.client_protected that the user’s own mail client had already encrypted the message, so the stage left it alone;

  • pepsi-stage-decrypt records the mirror image under crypto.in — whether the message arrived encrypted, what became of the ciphertext (decryption = for-client, with for_client and client_fingerprint, when it was encrypted to the recipient’s own MUA key and passed on unopened), which of our identities opened it, the signature verdict (with covers_plaintext) and the ordered layer list — and sets signature_verified for the valid verdict and only that one, the flag pepsi-stage-check-whitelist gates a signature_required row on. Two of those members exist because nothing downstream could recover them: the plaintext is committed over the arriving bytes, so crypto.in.encrypted and crypto.in.outer_gossip (whether the arriving header block already carried an Autocrypt-Gossip: field) are only answerable here. pepsi-stage-autocrypt-learn reads both — their names live in pepsi_common::crypto_state so neither crypto stage owns the vocabulary;

  • pepsi-stage-reencrypt records under crypto.store what it did with a message decrypt opened: outcome reencrypted (with the protocol, MUA-key fingerprint, container and downgraded), plaintext or bounced (with a reason). It merges the whole crypto object, carrying crypto.in along, and the key’s presence is what keeps it from ever sealing a row twice;

  • the dispatcher writes dispatch_error when it forces a row to failed/timeout because a stage crashed or ran too long.

Both crypto stages live under the same crypto key with the same field names, and both are a shallow merge, so a row carries crypto.out or crypto.in (plus crypto.store, which the re-encryption stage writes alongside it) and not both — which is correct, since a message is on the outbound path or the inbound one. The full list, with the exact JSON shapes and the producing/consuming stage for each key, is in pepsi.state(7).