27. The message state

Every message in flight is one row of the pepsi.workqueue table. Alongside the envelope and the body, each row carries a free-form JSON object that travels with the message from ingress to its terminal stage: the state column (PostgreSQL JSONB). The exhaustive key reference is the pepsi.state(7) man page; this chapter is the narrative companion.

27.1. The state is the only channel

Stages do not talk to each other directly. The pipeline is a single-table state machine (see Architecture): a stage loads a row, does its work, and writes exactly one terminal result. The state object is the only place a stage can leave information for a later stage to read. Ingress records what it learned about the connection and the sender’s intent; the ARC stage records the verdict of the chain that it re-evaluated; a relay stage records why a delivery failed so that the bounce stage can explain it. None of this lives in the message itself — it lives in state.

Because state is plain JSON with no fixed schema, the set of keys is open-ended. A custom stage (see Extending the Pipeline) may invent its own keys; the only contract is the merge rule below.

27.2. Merge semantics

state is built up incrementally rather than overwritten. Every terminal stage helper that mutates a row merges its new keys into the existing object, so a key written early survives unless a stage deliberately replaces that specific key. The merge is a shallow, key-wise object merge with exactly PostgreSQL’s state || $new semantics. The worker folds it in for the single-row commit, and the fan-out functions (workqueue_split, workqueue_pause, workqueue_finish_with_clones) evaluate it in SQL. That is why the origin provenance that ingress seeds is still readable by a relay stage at the far end of a long pipeline, and why the RFC 3461 dsn parameters reach the bounce stage intact.

There is exactly one exception. When pepsi-stage-bounce rewrites a message into a delivery-status notification, it replaces state with a lone retry_since, the DSN’s own retry clock, plus the tls_required tag of a REQUIRETLS message. The reason is that the bounce is a brand-new null-sender message that inherits none of the original’s provenance.

27.3. What ingress seeds

When pepsi-ingress accepts a message it seeds three families of keys, plus conditional ones (dsn and spam below; tls_required for REQUIRETLS or TLS-Required: No; and mime_anomalies, mime_normalise and mime_normalised for ambiguous MIME, see pepsi.state(7)):

origin

The SMTP-origin provenance of the delivering connection — client IP, HELO/EHLO, the ESMTP/SMTPUTF8/BODY= declarations, TLS parameters, the listener, reverse-DNS/iprev, and this receiver’s authserv_id. The ARC stage needs the authserv_id to reproduce its Authentication-Results identity, and the relay stages consult the declarations when deciding whether to downgrade 8-bit/UTF-8 content for a given next hop. It also records how a submission authenticated (auth_method and friends) and from_bound: whether the From: is exactly one mailbox bound to the account by a concrete, wildcard-free USERNAME_MAP entry (or its default login@HOSTNAME). The encrypt stage lets a submission register a key or run an e-mail key command only when it came from a trusted relay or from_bound is true.

auth

The inbound SPF/DKIM/DMARC verdicts (also reflected in the prepended Authentication-Results header). The arc member starts as none and is overwritten by the ARC stage. On mail from an unauthenticated session whose From: display name names an address other than its own, auth.display_name_mismatch holds that claimed address, and ingress prepends X-Pepsi-Warning: display-name-claims-other-address (see What a security indicator means). auth.warnings lists every X-Pepsi-Warning value that ingress wrote: that one, from-has-bidi-or-invisible-characters (bidi controls or invisible characters in From:), from-domain-resembles-served-domain (a From: domain that looks like a served one), attachment-name-has-bidi-or-invisible-characters (such characters in an attachment’s file name), duplicate-header-<field> (differing copies of Date, Subject, To, Cc, Reply-To or Sender that were accepted), and the four that also earn the [external] subject tag: from-ambiguous (a From: that reads more than one way or names several domains), served-domain-not-authenticated, served-sender-not-authenticated (a Sender: at a served domain that nobody authenticated) and dmarc-fail-not-enforced.

local_origin

A boolean: true when the delivering session was authenticated (in MYNETWORKS, via SASL AUTH, with a matching TLS client certificate, or as a peer on a UNIX socket with AUTH_PEERCRED, the default there, whose uid resolved to a permitted login, which is how every locally submitted message arrives). It distinguishes outbound mail from a trusted sender from inbound mail off the Internet, and the per-address settings layer and the edit-settings control channel key on it.

dsn

The RFC 3461 parameters that the sender requested: message-level ret/envid and a per-recipient array of notify/orcpt parallel to the recipient list, which also carries each recipient’s RFC 7293 rrvs request. Every stage must preserve and honour it. The key is present only when the sender asked for something.

spam

Seeded as false, and only in one case: a message addressed solely to the reserved <Postmaster> mailbox, which RFC 5321 §4.5.1 requires to stay deliverable. The language, confirm-to-send and pay-to-send gates forward any message already marked not-spam, so this is what keeps that mailbox reachable. Restricting it to the sole-recipient case is deliberate: otherwise a spammer could whitelist co-recipients by adding a <Postmaster> RCPT alongside a real victim.

27.4. What the stages write

As the message advances, stages merge in their own keys:

  • the ARC stage overwrites auth.arc with the verdict of the chain that it re-evaluated;

  • any stage routing a message to its BOUNCE_STAGE (the delivery and discard stages, and policy stages such as the milter, secretary and crypto stages) writes a bounce object, which pepsi-stage-bounce turns into the DSN: its kind (permanent/success/delay), the human-readable diagnostic and failed_recipient, the copied notify/orcpt/envid, and structured next-hop detail (remote_mta/smtp_code/enhanced_status/phase/reply_text);

  • the relay stages keep retry scratch (attempts, last_error, delay_sent);

  • the stage scaffolding itself, when a stage’s error was not marked permanent, pauses the row with last_error, temporary_failures and temporary_stage (which drive the back-off), and when it gives up records failed_stage and failure_class (permanent, retries-exhausted, crashed or timed-out), from which the bounce stage builds a DSN with a generic text (last_error is never copied into a DSN). The dispatcher counts a message’s worker crashes and hangs in strikes;

  • a crypto stage that parks a message on a key lookup records under keydisc what it waits for. The pepsi-keydisc service that settles the request matches on this key to release the message;

  • a reply typed into the The secure-link fallback portal portal is injected with a secure_link object and deliberately without local_origin, which is what keeps the portal from being an open relay;

  • pepsi-stage-srs records the envelope sender that it replaced under srs.original, so that a DSN originated by this deployment goes to the real sender rather than to its own SRS alias. When pepsi-ingress makes a row from a remote report about a forwarded copy, it marks that row with forward_return (Forwarding addresses in delivery reports);

  • pepsi-stage-list and the list stages after it carry the mailing-list descriptor under list;

  • pepsi-stage-detect-language records the detected language that pepsi-stage-block-language(1) then scores;

  • pepsi-stage-check-whitelist and pepsi-stage-anti-spam cooperate through the spam/paid short-circuit flags and the pay_deadline;

  • pepsi-stage-secretary records under secretary the challenge that a held message waits on (the cookie, its deadline, the whitelist), marks it confirmed or abandoned, and on release sets spam = false; the timeout path removes the key again;

  • pepsi-stage-dot-forward records a forwarded row’s chain of forwarding logins in dot-forwarders to break ~/.forward forwarding loops;

  • pepsi-stage-milter records under milter which stage ran the filter, the verdict it returned and, when it went wrong, the error;

  • pepsi-stage-vacation records under vacation that it answered a message on the recipient’s behalf, and in which language. Nothing reads this key; it exists so that a surprising out-of-office reply is explainable from the row;

  • pepsi-stage-encrypt records what it signed and encrypted, per recipient, under crypto.out (with fingerprints and skipped when a local recipient’s several MUA keys were, or were not, all encrypted to), and sets encrypted when a recipient’s message really was encrypted; pepsi-stage-auto-whitelist copies this flag into its signature_required column. crypto.out.key_attached says that the sender’s key went out as a file, and crypto.out.client_protected says that the user’s own mail client had already encrypted the message, so the stage left it alone;

  • pepsi-stage-decrypt records the mirror image under crypto.in: whether the message arrived encrypted, what became of the ciphertext (decryption = for-client, with for_client and client_fingerprint, when it was encrypted to the recipient’s own MUA key and passed on unopened; withheld when an unsigned plaintext from a malleable S/MIME container was not opened for the user; unauthenticated and sanitised when such a plaintext was delivered as text), which of our identities opened it, the signature verdict (with covers_plaintext), usually_signed when a message without a valid signature comes from an address whose mail recently carried one, and the ordered layer list. It sets signature_verified for the valid verdict and only for that one, and pepsi-stage-check-whitelist gates a signature_required row on this flag. Two of those members exist because nothing downstream could recover them: the plaintext is committed over the arriving bytes, so crypto.in.encrypted and crypto.in.outer_gossip (whether the arriving header block already carried an Autocrypt-Gossip: field) are only answerable here. pepsi-stage-bounce reads crypto.in before it clears the state, and quotes only the addressing fields of a message that arrived encrypted, to avoid returning decrypted content to the envelope sender (Bounces after gateway decryption reveal decrypted content). pepsi-stage-autocrypt-learn reads both. Their names live in pepsi_common::crypto_state, so that neither crypto stage owns the vocabulary;

  • pepsi-stage-reencrypt records under crypto.store what it did with a message that decrypt opened: outcome reencrypted (with the protocol, the MUA-key fingerprints that it sealed to, of which fingerprint is the first, any keys skipped with a reason, container and downgraded), plaintext or bounced (with a reason). It merges the whole crypto object, carrying crypto.in along, and the key’s presence is what keeps it from ever sealing a row twice;

  • the dispatcher writes dispatch_error when it forces a row to failed/timeout because a stage crashed or ran too long.

Both crypto stages live under the same crypto key with the same field names, and both are a shallow merge, so a row carries crypto.out or crypto.in (plus crypto.store, which the re-encryption stage writes alongside it) and not both — which is correct, since a message is on the outbound path or the inbound one. The full list, with the exact JSON shapes and the producing/consuming stage for each key, is in pepsi.state(7).