.. This file is part of PEPSI. Copyright (C) 2026 Pepsi contributors PEPSI is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. ============ Architecture ============ Pepsi's design choice is to make every post-ingress processing step an independent **stage program** driven by a **single database table**. This chapter explains that model in depth: the workspace layout, the data model, the stage contract, the dispatcher, and the message lifecycle. Workspace layout ================ Pepsi is a Cargo workspace (edition 2024). The **workspace root is itself the ``pepsi`` package** and owns every ``[[bin]]`` in ``src/bin/*.rs``; the ``pepsi-*`` member crates are **libraries** whose ``run()`` the thin binaries call. So binary ``pepsi-X`` lives in ``src/bin/pepsi-X.rs`` and delegates to crate ``pepsi-x``'s library. * ``pepsi-common`` — shared infrastructure: domain validation (``domain``), DKIM/ARC keys and signing (``keys``/``sign``/``auth``), the outbound SMTP client (``smtp``), the SRS engine (``srs``), DSN types (``dsn``), the MIME downgrade machinery (``mime``), the header/body split (``message``), and the **stage scaffolding** (``stage``). * ``pepsi-ingress``, ``pepsi-dispatch``, ``pepsi-httpd``, ``pepsi-setup``, ``pepsi-queue`` — the non-stage programs. * ``pepsi-stage-*`` — the stage programs. * ``vendor/taler-rust`` — a vendored git submodule (a *separate* workspace, excluded here) providing ``taler-common``'s config/CLI/database machinery (``taler_main``, ``ConfigSource``, ``Config``/``Section``, ``taler_common::db``). Each program defines a ``constants::CONFIG_SOURCE`` identifying it (project/component/exec), and stage programs additionally define a ``constants::PROGRAM`` — the binary name used as ``PROGRAM`` in a stage section. The unified binary ================== A development build (the default ``multibin`` Cargo feature) compiles each program to its own executable. A **release** build instead enables the ``unibin`` feature, which links most programs into one multi-call executable, ``pepsi``: a small ``main`` inspects ``argv[0]`` and dispatches to the matching program's entry point. ``make install`` places that single binary in ``$PREFIX/libexec/pepsi/pepsi`` and creates a ``$PREFIX/bin`` symlink per program name pointing at it (``pepsi-ingress -> ../libexec/pepsi/pepsi``, and so on). Operators, the dispatcher and the systemd units keep using the per-program names unchanged; only the on-disk shape differs. The folded programs' ``main``\ s live in the root package's own library (``pepsi::programs::``), so the same code backs both the standalone binaries and the unified one. The motivation mirrors why a C project prefers a shared library over statically linking the same code into every executable. Pepsi's programs share a great deal of code — the async runtime (Tokio), the SQL and TLS stacks (sqlx, rustls), the SMTP/MIME/DKIM machinery in ``pepsi-common``, the CLI/config layer, and more. When each program is its own statically-linked binary, every one carries a private copy of all that common code. Folding them into a single binary keeps **one** copy, with three intended benefits: Smaller on disk The two dozen folded programs become one ~10 MB binary plus a symlink (a few bytes) each, instead of two dozen multi-megabyte executables that each re-embed the shared code. Faster process start Pepsi is process-heavy: the dispatcher runs each stage as a *pool* of worker processes and re-spawns them (after ``MAX_MESSAGES``, a timeout or a crash; see `The dispatcher`_). Because every worker — of *every* stage — now ``exec``\ s the **same** file, that file's text is read from disk at most once and then served from the kernel page cache for every subsequent spawn. With separate binaries, the first spawn of each distinct stage pays its own cold-cache read. Less physical memory The read-only segments (code and read-only data) of an ``mmap``\ ed executable are backed by shared physical pages: every process running the file maps the *same* physical copy (this is independent of ASLR, which only randomises the virtual address, not the page-cache backing). With one binary the dispatcher, ingress, the HTTP server and all the stage workers — potentially dozens of processes — share a single resident copy of the common code. With separate binaries, each distinct program that is running keeps its own resident copy of that duplicated code, exactly the redundancy a shared library removes. Two classes of program are deliberately **kept separate**, installed as their own ``$PREFIX/bin`` executables (their ``[[bin]]`` targets build in both feature modes and are excluded from the unified binary): * **Programs that share little code with the rest.** ``pepsi-stage-detect-language`` links the ``lingua`` language models — a large dependency nothing else uses — and ``pepsi-setup`` pulls in an HTTP client and the one-off provisioning logic. Folding either in would *grow* the shared binary, and every process's resident set, with code only one program uses — working against the goal — so they stay out. ``pepsi-config`` is likewise left standalone. * **Privileged binaries.** ``pepsi-helper-maildir-writer`` (setuid-root), ``pepsi-stage-relay-to-maildir`` and ``pepsi-stage-relay-to-smarthost`` (both setgid) each carry their own permission bit. A symlink cannot hold a setuid/setgid bit — the bit must live on the target file — so folding these in would force that bit onto the shared binary, where it would apply to *every* symlinked program and collapse the privilege separation described in :doc:`installation`. .. note:: The start-time and memory wins are realised at runtime by the long-running, process-heavy services (the dispatcher and its worker pools, where many processes run the one file concurrently); a one-shot operator command invoked once — say ``pepsi-status`` over SSH — sees the disk-size win but no start-time benefit from sharing. The six standalone binaries also still each embed their own copy of the common code, so the de-duplication is large but not total — a deliberate trade for the two reasons above. The data model ============== One PostgreSQL schema, ``pepsi``, holds two things the whole system turns on: the **message queue** and, since it is the only piece of shared, writable, transactional state a deployment has, the **configuration overlay**. Everything else in the schema is a companion table to one of the two. ``pepsi.ingress`` One row per message in flight. Besides the envelope and parsed metadata, the message itself is stored **split** into a ``headers`` column and a ``body`` column (split at the first blank line; the invariant is ``raw = headers || CRLF || body``). Header-only stages never load or rewrite the body. The pipeline-control columns are: * ``stage`` — the ``[stage-]`` section of the program currently responsible for the message (``init`` for a fresh message). * ``status`` — ``pending`` / ``running`` / ``paused`` / ``failed`` / ``timeout`` (a PostgreSQL ``ENUM``). * ``state`` — free-form ``JSONB`` carried with the message (see `The state object`_). * ``timeout`` — when a ``paused`` message becomes due for retry. The authentication verdicts (``spf``/``dkim``/``dmarc``/``arc``) and the ``authserv_id`` live in ``state`` (under the ``auth`` and ``origin`` keys respectively), not in dedicated columns. ``pepsi.dns_address`` A cache of resolved MX-host A/AAAA addresses with per-address connect health, maintained by ``pepsi-stage-relay-to-internet``. A working address is preferred until its DNS TTL elapses; a host is re-resolved once all its addresses have failed or expired. ``pepsi.stage_stats`` / ``pepsi.dispatch_stats`` Cumulative pipeline statistics: per-stage message counts, total processing time, kills/timeouts and crashes, plus the global stage/message totals. The global row also carries ``messages_failed`` (messages that ended in a terminal ``failed``/``timeout`` state) and ``serialization_failures`` (transient ``40001``/``40P01`` serialization/deadlock retry events the dispatcher observed) — both expected to stay at zero under normal operation, so a rising count signals a defect rather than a load limit. ``pepsi-dispatch`` is the **only** writer: it accumulates the deltas in memory and flushes them in one transaction roughly once a minute, whenever the pipeline goes idle, and on shutdown. Stages fused into another stage's worker pass are reported back to it on that pass's worker status line rather than written by the worker, so that a table with one row per stage never has every worker of a stage contending for it. ``pepsi-httpd``'s ``/metrics`` reads them (and the live active/paused gauges straight from ``pepsi.ingress``). ``pepsi.config_override`` The administrator-managed configuration layered on top of the INI file: one row per ``(scope, section, option)`` override, where *scope* is ``global``, ``domain:`` or ``address:
``. This is what lets a running deployment be reconfigured without editing files. It is the second thing the database owns outright, next to the queue, and it works the same way the queue does: a trigger issues a ``config_changed`` ``NOTIFY``, and the components react — ``pepsi-dispatch`` retires its stage workers so their replacements read the new values. Two properties are structural rather than conventional. Sections that must exist before a database connection does — and the listener sections, which are a security boundary — are **never** read from here, so a compromised database cannot move a listening socket or retarget the database connection. And write access belongs to a dedicated ``pepsi-config`` PostgreSQL role that no mail-processing component holds: a stage worker may read the configuration and may not change it. See :doc:`configuration`. ``pepsi.settings`` The per-address override layer, and a deliberately *separate* table from ``config_override`` because account owners write it themselves by e-mail. The distinction is a privilege boundary, not duplication. Rows are inserted by the ``ingress_add`` stored function, which also issues the ``NOTIFY`` (in the same transaction, so listeners only see committed rows). A stage can spawn a *side* message — e.g. a delay DSN — with ``ingress_enqueue``, which clones a row server-side so the body never round-trips through the application; the cloned message's unique ``token`` makes it at-most-once. Three further tables form the end-to-end cryptography **key store**, which has no part in the message state machine and is managed by its own tool (:doc:`programs/pepsi-keys`; the design is in :doc:`key-management`): ``pepsi.crypto_identity`` The key pairs of the addresses we serve — public material plus the **private** half — one row per capability (``sign``, ``encrypt`` or ``both``), with its lifecycle columns (``status``, ``is_primary``, ``published``, ``expires_at``, ``private_purged_at``). ``pepsi.peer_key`` Remote correspondents' cached public keys and certificates, each with the ``source`` it came from, whether that lookup was DNSSEC-validated, the last validity verdict and a cache deadline. ``pepsi.ca_trust`` The CA certificates an inbound S/MIME chain must reach to count as trusted. Seven more tables belong to the **administrative surface** (:doc:`admin-api`) and the online setup, and are likewise outside the message state machine: ``pepsi.admin_account`` / ``pepsi.admin_session`` / ``pepsi.api_token`` Remote administrator accounts (Argon2id password hashes), their live browser sessions, and the bearer tokens automation authenticates with. Sessions are rows rather than process memory, so a restart does not log everyone out and two ``pepsi-httpd`` processes agree about who is logged in. Only the digests of a session cookie and of a token's secret half are stored: a database backup contains no usable credential. ``pepsi.event_log`` The **audit log** — one row per configuration change, key operation, login, failed login, account or token change and administrative queue action, with the principal that performed it. It is written by the API *and* by the operator command-line tools, through one helper in ``pepsi-common``, so it is complete regardless of which surface acted; a log that recorded only what happened over HTTP would invite the wrong conclusion from an absence. ``pepsi.mail_log`` The **opt-in** per-message record (``[pepsi] MAIL_LOG``, off by default). Since a delivered message's row is deleted, an ordinary deployment keeps no per-message log at all — a property of the design, not an omission. Switching this on makes the deployment keep a record of who corresponds with whom, which is why it is a deliberate act with a bounded retention. See :ref:`mail-log`. Two more belong to the **online setup** (:doc:`installation`), and carry the sharpest privilege boundary in the schema: ``pepsi.setup_task`` / ``pepsi.setup_task_log`` The **intent queue** the browser-driven setup writes and a root program drains. Setting a mail server up is privileged — writing ``/etc/pepsi/pepsi.conf``, handing each ``secrets.d`` fragment to the one account that reads it, running certbot, creating database roles, generating keys — and ``pepsi-httpd`` drops privileges before it accepts a connection. So it does not act: it writes a row describing *what should be true*, from a closed set of seven kinds with strictly validated parameters, and ``pepsi-setup apply`` decides how. Root is reached through a table, never through a socket that speaks a protocol, and every request and outcome is a durable row. The second table carries the progress lines a running task emits, so a certbot run can be watched without the applier holding an HTTP connection. A row in this table is a request to a process running as root, so the grants are the tightest in the schema: **no account that processes mail may touch it at all**, ``pepsi-httpd`` may only ``SELECT`` (it enqueues through the separate configuration connection), and only ``pepsi-config`` may ``INSERT``. Which of those wrote a row is not taken on trust: ``written_by`` is forced from ``current_user`` by a ``BEFORE INSERT`` trigger, so it is a fact about the connection rather than a claim in the payload, and the applier refuses anything else. The full trust model is in :manpage:`pepsi-setup(1)`. These carry the schema's other access boundary. The three credential tables are granted to the ``pepsi-httpd`` role **alone** — a component that could read ``admin_session`` could mint itself an administrative session — and the two logs are **append-only** for everything that processes mail: every role may ``INSERT`` (that is what makes the audit log complete) and none may ``UPDATE`` or ``DELETE``. ``pepsi-setup`` re-applies both restrictions on every run, because the blanket ``GRANT … ON ALL TABLES`` it hands the service roles would otherwise undo them, and then *verifies* them against the live server. ``crypto_identity.private_wrapped`` holds each private key sealed with AES-256-GCM under a key-encryption key that lives only in a ``secrets.d`` fragment, and ``pepsi-setup`` grants that **one column** to the ``pepsi-crypto`` role alone: every ordinary service role (``pepsi``, ``pepsi-ingress``, ``pepsi-httpd``, ``pepsi-telemetry``) has its table-level grant replaced by a column-level one that omits it. So a compromise of an unrelated stage yields the public half of every identity and nothing more, and a stolen database yields ciphertext even for that column. The schema is a numbered patch series with a re-creatable procedures file; the patch number is bumped only for the first schema change after a release tag, so pre-release work accumulates in the current (CREATE-only) ``pepsi-0001.sql``. See :doc:`installation`. The stage pipeline ================== Everything after ingress is a **state machine over a single table**. The moving parts are: * **Stage sections.** Each ``[stage-]`` names a ``PROGRAM`` and optional ``NEXT_STAGE`` / ``BOUNCE_STAGE`` (:ref:`config-pipeline`), plus the worker-pool knobs ``PARALLELISM``, ``MAX_MESSAGES`` and ``QUEUE_LIMIT`` (how many messages the dispatcher pipelines to one worker at once). Stage names are operator-chosen labels independent of crate names, so the same binary can serve several stages. * **Stage workers.** Each stage runs as a pool of persistent worker processes (``PROGRAM worker``). A worker reads ``ingress_id``\ s from standard input (pipelined up to ``QUEUE_LIMIT`` at a time, processed strictly in order), does its work per message, calls **exactly one** terminal helper, and writes one status line (``0`` on success) per message to standard output. For manual operation or tests a single id can be piped to a worker (``echo | PROGRAM worker``). * **The dispatcher.** The single long-lived coordinator that claims rows and feeds them to workers. The stage contract ================== Each worker opens the database pool once at start-up, then for every message id calls ``pepsi_common::stage::prepare_on(pool, PROGRAM, id, cfg, load)``. This: #. loads the ``running`` row in a **single** ``SELECT`` (refusing to act unless the row is ``running``), #. resolves the message's ``[stage-]`` section, and #. verifies that section's ``PROGRAM`` matches this binary. The ``load`` argument (``stage::Load``) is **fixed per stage** and selects one of three prepared statements, so the row is fetched in one round-trip pulling only the columns the stage needs of the two potentially-large ones (``headers``, ``body``); the cheap envelope/status columns are always loaded. ``Load::Metadata`` loads neither big column (envelope/``state`` only — SRS, discard, ``if``, check-whitelist); ``Load::Headers`` adds the header block but not the body (bounce); ``Load::Full`` adds both, reassembled with ``ctx.raw_message()`` for stages that transmit or hash the message (relay, ARC, DKIM-sign, detect-language). The header block, when loaded, is read via ``ctx.headers()``; both it and ``raw_message()`` error if the stage under-declared its ``Load``. Standard output is reserved for the worker status protocol, so a stage body must never write to it. The shared pool runs ``READ COMMITTED`` — at most one writer ever touches a given queue row (the single dispatcher claims serially, a worker mutates only its own claimed row), so serializable isolation buys nothing — yet the terminal helpers and the dispatcher still route their queue writes through ``pepsi_common::db::with_retry`` as cheap insurance against a transient serialization/deadlock failure. A stage that rewrites the message records its changes through the content setters (``set_headers``, ``set_mail_from``, ``merge_state``, …) rather than issuing SQL; the worker folds those changes **and** the stage transition into **one** ``UPDATE``, built from exactly the columns the stage touched (signing stages, for example, change only the ``headers`` column, leaving the body untouched). Because content changes ride the in-memory row, a fused successor sees them and they are persisted with the chain's single commit — fusion never drops an update. The program then calls exactly **one** terminal helper, whose database semantics are the contract: ``advance()`` / ``advance_to()`` Move ``stage``, **stay ``running``** (the dispatcher then requeues the row to ``pending`` for the next stage's worker pool). ``reroute(stage, state)`` Like advance, but merges ``state`` — used to hand a message to a bounce stage. ``pause(state, secs)`` Set ``paused`` and a retry ``timeout`` (the dispatcher re-queues it). ``fail(state)`` Terminal ``failed`` (left for an operator). ``finish()`` ``DELETE`` the row (delivered or dropped). ``advance_or_finish()`` / ``complete_success(...)`` The delivery-stage terminal paths: advance to ``NEXT_STAGE`` else finish; and the success-DSN gate that reroutes to ``BOUNCE_STAGE`` when ``ORIGINATE_SUCCESS_DSN`` + ``NOTIFY=SUCCESS``. The state object ================ The ``state`` ``JSONB`` is the only channel between stages. The terminal helpers **merge** new keys (``state || $new`` in SQL) so earlier data survives; the one exception is ``pepsi-stage-bounce``, which clears ``state`` because a bounce is a new null-sender message. The stock keys are: ``origin`` SMTP-origin provenance seeded by ingress: client IP, HELO/EHLO, ESMTP/SMTPUTF8 flags, the ``BODY=`` declaration, TLS parameters, listener, reverse-DNS/iprev, and the ``authserv_id`` (which the ARC stage needs to reproduce the AAR identity). Read by the ARC stage and the relay stages' downgrade decision. ``dsn`` The RFC 3461 parameters: message-level ``ret``/``envid`` plus a per-recipient array of ``notify``/``orcpt``. **Every stage must preserve it.** ``bounce`` Written by a delivery/discard stage when routing to ``BOUNCE_STAGE``, consumed by the bounce stage: ``kind`` (``permanent``/``success``/``delay``), ``diagnostic``, ``failed_recipient`` and the copied ``notify``/``orcpt``/ ``envid``. ``attempts`` / ``last_error`` / ``delay_sent`` Delivery-stage retry scratch. ``dispatch_error`` Written by the dispatcher when it forces a row to ``failed``/``timeout``. The state object has its own chapter — :doc:`state` — and the exhaustive key reference is :manpage:`pepsi.state(7)`. The dispatcher ============== ``pepsi-dispatch`` is the **single** coordinator (run exactly one per system). It ``LISTEN``\ s on the ``ingress`` channel with a periodic safety-net heartbeat, and: * **Claims in batches.** On each wake it claims ``pending`` work for every stage with spare capacity in **one** ``UPDATE … RETURNING`` (each stage up to its own remaining capacity), sets the rows ``running`` and hands each id to a worker of that stage over the worker's standard input. As the single serial claimer — the sole ``pending``→``running`` writer — it needs no ``FOR UPDATE SKIP LOCKED``. * **Scales elastically, pipelines work.** Each stage's worker pool grows on demand up to its ``PARALLELISM`` and shrinks when idle: no worker runs until a stage sees work; work is packed onto the fewest workers so a surplus goes cold and is reaped after ``WORKER_IDLE_TIMEOUT``; after ``MAX_MESSAGES`` a worker recycles its child. A worker is fed up to ``QUEUE_LIMIT`` ids at once, lifting a stage's in-flight capacity to ``QUEUE_LIMIT × PARALLELISM`` without adding processes or connections. A stage whose ``PROGRAM`` cannot be started is held off with a short spawn cooldown rather than retried in a tight loop. * **Requeues advances.** When a worker reports success and left the row ``running`` at a new stage (i.e. it *advanced*), the dispatcher requeues it to ``pending`` so the next stage's pool claims it (workers are per-stage, so a message is no longer chained inside one process). * **Requeues** ``paused`` rows once their ``timeout`` elapses (it sleeps until the earliest due time, or the heartbeat). * **Recovers.** A worker that reports a non-zero status or whose child crashes → ``failed`` (and is replaced); one that does not answer within ``MAX_RUNTIME`` is killed → ``timeout`` (reason recorded in ``state.dispatch_error``). On start-up it resets orphaned ``running`` rows to ``pending``; on ``SIGINT``/``SIGTERM`` it stops workers and resets their (and any claimed-but-unassigned) rows. The dispatcher never processes message content itself — it only runs the stage workers, which record their own outcome on the row. Message lifecycle (end to end) ============================== #. **Receive.** ``pepsi-ingress`` accepts the SMTP transaction, authenticates the message, prepends the trace and ``Authentication-Results`` headers, splits and stores it at ``stage = init``, ``status = pending``, and ``NOTIFY``\ s. #. **Dispatch.** ``pepsi-dispatch`` claims the row → ``running`` and hands it to a ``[stage-init]`` worker. #. **Stages.** Each stage advances the message (``running`` → next stage); the dispatcher requeues the advanced row to ``pending`` and its next stage's pool picks it up: e.g. ARC → SRS → DKIM-sign → deliver. #. **Deliver.** The delivery stage transmits the mail. On success it ``finish``\ es (or advances). A transient failure ``pause``\ s with backoff; the dispatcher re-queues it later. #. **Bounce.** A permanent failure ``reroute``\ s to ``BOUNCE_STAGE``; ``pepsi-stage-bounce`` builds an (unsigned) DSN, which is then signed and delivered like any other message. A bounce is never itself bounced. To add a new step anywhere in this flow — say a spam filter or an archival hook — you write a stage program and splice its section into the graph; see :doc:`extending`.