17. The message state

Every message in flight is one row of the pepsi.ingress table. Alongside the envelope and the body, each row carries a free-form JSON object — the state column (PostgreSQL JSONB) — that travels with the message from ingress to its terminal stage. This chapter is the narrative companion to the pepsi.state(7) man page, which is the exhaustive key reference.

17.1. The state is the only channel

Stages do not talk to each other directly. The pipeline is a single-table state machine (see Architecture): a stage loads a row, does its work, and writes exactly one terminal result. The state object is the only place a stage can leave information for a later stage to read. Ingress records what it learned about the connection and the sender’s intent; the ARC stage records the verdict of the chain it re-evaluated; a relay stage records why a delivery failed so the bounce stage can explain it. None of this lives in the message itself — it lives in state.

Because state is plain JSON with no fixed schema, the set of keys is open-ended. A custom stage (see Extending the Pipeline) may invent its own keys; the only contract is the merge rule below.

17.2. Merge semantics

state is built up incrementally rather than overwritten. Every terminal stage helper that mutates a row merges its new keys into the existing object — the SQL terminal functions evaluate state || $new — so a key written early survives the whole lifetime of the message unless a stage deliberately replaces that specific key. This is why the origin provenance seeded by ingress is still readable by a relay stage at the far end of a long pipeline, and why the RFC 3461 dsn parameters reach the bounce stage intact.

There is exactly one exception: pepsi-stage-bounce clears state to null when it rewrites a message into a delivery-status notification, because the bounce is a brand-new null-sender message that inherits none of the original’s provenance.

17.3. What ingress seeds

When pepsi-ingress accepts a message it seeds three families of keys:

origin

The SMTP-origin provenance of the delivering connection — client IP, HELO/EHLO, the ESMTP/SMTPUTF8/BODY= declarations, TLS parameters, the listener, reverse-DNS/iprev, and this receiver’s authserv_id. The ARC stage needs the authserv_id to reproduce its Authentication-Results identity, and the relay stages consult the declarations when deciding whether to downgrade 8-bit/UTF-8 content for a given next hop.

auth

The inbound SPF/DKIM/DMARC verdicts (also reflected in the prepended Authentication-Results header). The arc member starts as none and is overwritten by the ARC stage.

local_origin

A boolean: true when the delivering session was authenticated (in MYNETWORKS, via SASL AUTH, or with a matching TLS client certificate). It distinguishes outbound mail from a trusted sender from inbound mail off the Internet, and the per-address settings layer and the edit-settings control channel key on it.

dsn

The RFC 3461 parameters the sender requested: message-level ret/envid and a per-recipient array of notify/orcpt parallel to the recipient list. Every stage must preserve and honour it.

17.4. What the stages write

As the message advances, stages merge in their own keys:

  • the ARC stage overwrites auth.arc with the verdict of the chain it re-evaluated;

  • a delivery or discard stage routing a message to its BOUNCE_STAGE writes a bounce object — its kind (permanent/success/delay), the human-readable diagnostic and failed_recipient, the copied notify/orcpt/envid, and structured next-hop detail (remote_mta/smtp_code/enhanced_status/phase/reply_text) — which pepsi-stage-bounce turns into the DSN;

  • the relay stages keep retry scratch (attempts, last_error, delay_sent);

  • pepsi-stage-detect-language records the detected language that pepsi-stage-block-language(1) then scores;

  • pepsi-stage-check-whitelist and pepsi-stage-anti-spam cooperate through the spam/paid short-circuit flags and the pay_deadline;

  • pepsi-stage-dot-forward records each processed login in dot-forwarders to break ~/.forward forwarding loops;

  • pepsi-stage-vacation records under vacation that it answered a message on the recipient’s behalf, and in which language — read by nobody, so that a surprising out-of-office reply is explainable from the row;

  • pepsi-stage-encrypt records what it signed and encrypted, per recipient, under crypto.out, and sets encrypted when a recipient’s message really was encrypted — the flag pepsi-stage-auto-whitelist copies into its signature_required column;

  • pepsi-stage-decrypt records the mirror image under crypto.in — whether the message arrived encrypted, what became of the ciphertext, which of our identities opened it, the signature verdict and the ordered layer list — and sets signature_verified for the valid verdict and only that one, the flag pepsi-stage-check-whitelist gates a signature_required row on;

  • the dispatcher writes dispatch_error when it forces a row to failed/timeout because a stage crashed or ran too long.

Both crypto stages live under the same crypto key with the same field names, and both are a shallow merge, so a row carries crypto.out or crypto.in and not both — which is correct, since a message is on the outbound path or the inbound one. The full list, with the exact JSON shapes and the producing/consuming stage for each key, is in pepsi.state(7).