.. This file is part of PEPSI. Copyright (C) 2026 GNUnet e.V. PEPSI is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. PEPSI is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. ================= The message state ================= Every message in flight is one row of the ``pepsi.workqueue`` table. Alongside the envelope and the body, each row carries a free-form JSON object — the ``state`` column (PostgreSQL ``JSONB``) — that travels *with* the message from ingress to its terminal stage. The exhaustive key reference is the :manpage:`pepsi.state(7)` man page; this chapter is the narrative companion. The state is the only channel ============================= Stages do not talk to each other directly. The pipeline is a single-table state machine (see :doc:`architecture`): a stage loads a row, does its work, and writes exactly one terminal result. The ``state`` object is the **only** place a stage can leave information for a later stage to read. Ingress records what it learned about the connection and the sender's intent; the ARC stage records the verdict of the chain it re-evaluated; a relay stage records *why* a delivery failed so the bounce stage can explain it. None of this lives in the message itself — it lives in ``state``. Because ``state`` is plain JSON with no fixed schema, the set of keys is open-ended. A custom stage (see :doc:`extending`) may invent its own keys; the only contract is the merge rule below. Merge semantics =============== ``state`` is built up incrementally rather than overwritten. Every terminal stage helper that mutates a row **merges** its new keys into the existing object — a shallow, key-wise object merge with exactly PostgreSQL's ``state || $new`` semantics, folded in the worker for the single-row commit and evaluated in SQL by the fan-out functions (``workqueue_split``, ``workqueue_pause``, ``workqueue_finish_with_clones``) — so a key written early survives unless a stage deliberately replaces that specific key. That is why the ``origin`` provenance seeded by ingress is still readable by a relay stage at the far end of a long pipeline, and why the RFC 3461 ``dsn`` parameters reach the bounce stage intact. There is exactly **one** exception: :doc:`programs/pepsi-stage-bounce` clears ``state`` to ``null`` when it rewrites a message into a delivery-status notification, because the bounce is a brand-new null-sender message that inherits none of the original's provenance. What ingress seeds ================== When :doc:`programs/pepsi-ingress` accepts a message it seeds three families of keys, plus two conditional ones (``dsn`` and ``spam``): ``origin`` The SMTP-origin provenance of the delivering connection — client IP, HELO/EHLO, the ESMTP/SMTPUTF8/``BODY=`` declarations, TLS parameters, the listener, reverse-DNS/iprev, and this receiver's ``authserv_id``. The ARC stage needs the ``authserv_id`` to reproduce its ``Authentication-Results`` identity, and the relay stages consult the declarations when deciding whether to downgrade 8-bit/UTF-8 content for a given next hop. It also records how a submission authenticated (``auth_method`` and friends) and ``from_bound``: whether the ``From:`` is exactly one mailbox bound to the account by a concrete, wildcard-free ``USERNAME_MAP`` entry (or its default ``login@HOSTNAME``). The encrypt stage lets a submission register a key or run an e-mail key command only when it came from a trusted relay or ``from_bound`` is true. ``auth`` The inbound SPF/DKIM/DMARC verdicts (also reflected in the prepended ``Authentication-Results`` header). The ``arc`` member starts as ``none`` and is overwritten by the ARC stage. ``local_origin`` A boolean: ``true`` when the delivering session was authenticated (in ``MYNETWORKS``, via SASL ``AUTH``, with a matching TLS client certificate, or — on a UNIX socket with ``AUTH_PEERCRED``, the default there — as a peer whose uid resolved to a permitted login, which is how every locally submitted message arrives). It distinguishes outbound mail from a trusted sender from inbound mail off the Internet, and the per-address settings layer and the edit-settings control channel key on it. ``dsn`` The RFC 3461 parameters the sender requested: message-level ``ret``/``envid`` and a per-recipient array of ``notify``/``orcpt`` parallel to the recipient list. **Every stage must preserve and honour it.** Present only when the sender asked for something. ``spam`` Seeded as ``false``, and only in one case: a message addressed *solely* to the reserved ```` mailbox, which RFC 5321 §4.5.1 requires to stay deliverable. The language, confirm-to-send and pay-to-send gates forward any message already marked not-spam, so this is what keeps that mailbox reachable. Restricting it to the sole-recipient case is deliberate: otherwise a spammer could whitelist co-recipients by adding a ```` ``RCPT`` alongside a real victim. What the stages write ===================== As the message advances, stages merge in their own keys: * the ARC stage overwrites ``auth.arc`` with the verdict of the chain it re-evaluated; * any stage routing a message to its ``BOUNCE_STAGE`` (the delivery and discard stages, and policy stages such as the milter, secretary and crypto stages) writes a ``bounce`` object — its ``kind`` (``permanent``/``success``/``delay``), the human-readable ``diagnostic`` and ``failed_recipient``, the copied ``notify``/``orcpt``/``envid``, and structured next-hop detail (``remote_mta``/``smtp_code``/``enhanced_status``/``phase``/``reply_text``) — which :doc:`programs/pepsi-stage-bounce` turns into the DSN; * the relay stages keep retry scratch (``attempts``, ``last_error``, ``delay_sent``); * :doc:`programs/pepsi-stage-srs` records the envelope sender it replaced under ``srs.original``, so a DSN this deployment originates goes to the real sender rather than to its own SRS alias; * :doc:`programs/pepsi-stage-list` and the list stages after it carry the mailing-list descriptor under ``list``; * :doc:`programs/pepsi-stage-detect-language` records the detected ``language`` that :manpage:`pepsi-stage-block-language(1)` then scores; * :doc:`programs/pepsi-stage-check-whitelist` and :doc:`programs/pepsi-stage-anti-spam` cooperate through the ``spam``/``paid`` short-circuit flags and the ``pay_deadline``; * :doc:`programs/pepsi-stage-secretary` records the challenge a held message waits on under ``secretary`` (the cookie, its ``deadline``, the ``whitelist``), marks it ``confirmed`` or ``abandoned``, and on release sets ``spam = false``; the timeout path removes the key again; * :doc:`programs/pepsi-stage-dot-forward` records a forwarded row's chain of forwarding logins in ``dot-forwarders`` to break ``~/.forward`` forwarding loops; * :doc:`programs/pepsi-stage-milter` records under ``milter`` which stage ran the filter, the ``verdict`` it returned and, when it went wrong, the ``error``; * :doc:`programs/pepsi-stage-vacation` records under ``vacation`` that it answered a message on the recipient's behalf, and in which language — read by nobody, so that a surprising out-of-office reply is explainable from the row; * :doc:`programs/pepsi-stage-encrypt` records what it signed and encrypted, per recipient, under ``crypto.out``, and sets ``encrypted`` when a recipient's message really was encrypted — the flag :doc:`programs/pepsi-stage-auto-whitelist` copies into its ``signature_required`` column. ``crypto.out.key_attached`` says the sender's key went out as a file, and ``crypto.out.client_protected`` that the user's own mail client had already encrypted the message, so the stage left it alone; * :doc:`programs/pepsi-stage-decrypt` records the mirror image under ``crypto.in`` — whether the message arrived encrypted, what became of the ciphertext (``decryption = for-client``, with ``for_client`` and ``client_fingerprint``, when it was encrypted to the recipient's own MUA key and passed on unopened), which of our identities opened it, the signature verdict (with ``covers_plaintext``) and the ordered layer list — and sets ``signature_verified`` for the ``valid`` verdict and only that one, the flag :doc:`programs/pepsi-stage-check-whitelist` gates a ``signature_required`` row on. Two of those members exist because nothing downstream could recover them: the plaintext is committed over the arriving bytes, so ``crypto.in.encrypted`` and ``crypto.in.outer_gossip`` (whether the *arriving* header block already carried an ``Autocrypt-Gossip:`` field) are only answerable here. :doc:`programs/pepsi-stage-autocrypt-learn` reads both — their names live in ``pepsi_common::crypto_state`` so neither crypto stage owns the vocabulary; * :doc:`programs/pepsi-stage-reencrypt` records under ``crypto.store`` what it did with a message decrypt opened: ``outcome`` ``reencrypted`` (with the ``protocol``, MUA-key ``fingerprint``, ``container`` and ``downgraded``), ``plaintext`` or ``bounced`` (with a ``reason``). It merges the whole ``crypto`` object, carrying ``crypto.in`` along, and the key's presence is what keeps it from ever sealing a row twice; * the dispatcher writes ``dispatch_error`` when it forces a row to ``failed``/``timeout`` because a stage crashed or ran too long. Both crypto stages live under the same ``crypto`` key with the same field names, and both are a **shallow** merge, so a row carries ``crypto.out`` or ``crypto.in`` (plus ``crypto.store``, which the re-encryption stage writes alongside it) and not both — which is correct, since a message is on the outbound path or the inbound one. The full list, with the exact JSON shapes and the producing/consuming stage for each key, is in :manpage:`pepsi.state(7)`.