..
  This file is part of PEPSI.
  Copyright (C) 2026 Pepsi contributors

  PEPSI is free software; you can redistribute it and/or modify it under the
  terms of the GNU Affero General Public License as published by the Free
  Software Foundation; either version 3, or (at your option) any later version.

  PEPSI is distributed in the hope that it will be useful, but WITHOUT ANY
  WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
  A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
  details.

===========
Performance
===========

This chapter reports measured performance of the Pepsi pipeline: the relative
cost of each stage, the end-to-end latency of a single message on an idle system,
and the goodput sustained under load — together with the system-wide CPU
utilisation at saturation, which tells us whether that goodput is limited by
compute or by coordination. The numbers come from the three benchmark scripts
under ``tests/`` (``08-perstage-bench.sh``, ``09-latency-bench.sh`` and
``10-goodput-bench.sh``); how they obtain their figures — by snapshotting the
dispatcher's own ``pepsi.stage_stats`` and ``pepsi.dispatch_stats`` counters, by
counting delivered ``Maildir`` files, and by sampling ``/proc/stat`` and
``/proc/loadavg`` on the host — is documented in the :doc:`Benchmark Suite
<benchmarks>` chapter and is not repeated here.

.. note::

   Absolute numbers are specific to the test host and pipeline below and to a
   single point-in-time run; treat them as **orders of magnitude and ratios**,
   not guarantees. They were gathered on a deliberately modest, decade-old
   machine, so a contemporary server will be considerably faster. What is
   portable is the *shape* of the results — which stages are cheap, which scale
   with message size, and where the throughput ceiling sits.

Test environment
================

All measurements were taken on ``tripwire``, the live MTA-under-test of the
:doc:`testing` topology (inbound mail from the foreign host ``frontier``,
authorized outbound from ``enoki``, local delivery to a unix ``Maildir``). The
whole pipeline was run at log level ``warn`` for the duration (via the ``[pepsi]
LOG`` knob, which every component — the dispatcher *and* the stage workers it
spawns — honours) so per-message logging does not inflate the figures.

.. list-table:: Benchmark host (``tripwire``)
   :widths: 30 70

   * - CPU
     - Intel Core i7-920 @ 2.67 GHz (Nehalem, 2009): 1 socket, 4 cores, 1
       thread per core (no SMT)
   * - Memory
     - 11 GiB RAM
   * - Storage
     - Samsung 860 PRO SATA SSD, holding the PostgreSQL data directory
       (``/var``) and thus the write-ahead log
   * - OS / kernel
     - Debian 13 (trixie), Linux 6.12 (x86-64), glibc 2.42
   * - Database
     - PostgreSQL 15.13, ``max_connections = 100``, peer-auth over the local
       socket
   * - Pipeline
     - the deployment ``tests/01-deploy.sh`` writes: the full inbound chain
       ``arc`` → ``if`` (route) → ``decrypt`` → ``detect-language`` →
       ``block-language`` → ``check-whitelist`` → ``anti-spam`` →
       ``dot-forward`` → local ``Maildir`` delivery, its relay tail (``srs`` →
       ``dkim-sign`` → ``relay-to-smarthost``), the outbound submission chain
       (``edit-settings`` → ``auto-whitelist`` → ``if`` → ``encrypt`` →
       ``dkim-sign`` → ``relay-to-internet``) and three ``bounce`` stages; the
       ``init`` (arc) stage at ``PARALLELISM = 16`` and every other stage at the
       default ``PARALLELISM = 4``, with the default ``QUEUE_LIMIT = 4`` (worker
       pipelining on) throughout

How the pipeline spends its time
================================

Five properties of the architecture set the numbers that follow, so they are
worth stating up front. The last three are consequences of one principle: at this
message rate, the pipeline's cost is dominated not by the work a stage does but
by how often it makes the *database* coordinate — so it asks the database to
coordinate only where the information genuinely is.

* **Every stage pays a small fixed per-message database cost.** A stage worker
  loads its row in one ``SELECT`` (the row *and* its applicable per-address
  settings, in a single round-trip) and commits its outcome in one terminal
  ``UPDATE``/``DELETE`` (or one fan-out stored function). That pair of
  round-trips, plus worker scheduling, is the floor every message pays at every
  hop, independent of message size.

* **Several mechanisms remove or amortise those round-trips.** The dispatcher
  claims work for *all* stages in a single window-function ``UPDATE`` (a
  per-stage ``LATERAL … LIMIT`` stops each stage's index scan at its capacity)
  and pipelines up to ``QUEUE_LIMIT`` messages to each worker, so completing a
  message needs no prior coordinator round-trip. The load-and-advance hot path
  runs under ``READ COMMITTED`` rather than ``SERIALIZABLE`` — safe because at
  most one writer ever touches a given row (the dispatcher is the sole claimer).
  A body-free stage's worker loads and commits a *whole* pipelined batch of
  messages in one ``SELECT … = ANY`` and one arrayed ``UPDATE``. And **stage
  fusion** (``[pepsi] ALLOW_FUSION``, on by default) runs a fast body-free
  successor marked ``FUSION = yes`` — ``if``, ``srs``, ``check-whitelist``,
  ``auto-whitelist``, ``block-language``, ``discard`` — inside its predecessor's
  worker process on the already-loaded row, collapsing a run of *n* such stages
  from *n* loads + *n* advances + *n* hand-offs into a single load, the stages'
  work, and one terminal write. Fusion changes no outcome: a stage that cannot be
  fused (it needs the body, is not folded into the unified binary, or its
  successor is not marked ``FUSION``) simply falls back to a normal dispatched
  hop.

* **Hand-offs between stages are deliberately silent.** The dispatcher learns
  there is work from a ``LISTEN`` on the ``ingress`` channel, but a stage
  advancing a message does **not** notify. It does not need to: the worker
  reports each finished message on its standard output, and the coordinator
  answers that report with the same full capacity-bounded claim a notification
  would have triggered. A notification on the advance path would tell the
  dispatcher something it is about to be told anyway, over a cheaper channel.

  Silence on that path is worth the discipline it costs, because a notification
  is far from free at this rate: PostgreSQL holds a database-wide
  ``AccessExclusive`` lock from the moment a transaction queues one until it
  commits, so notifying transactions cannot group-commit and each hop would
  serialise the *whole database's* commit path. A notification is therefore
  issued only where the information is genuinely new — by writers **outside** the
  dispatcher's loop: message admission (coalesced, once per burst rather than
  once per message), ``ingress_inject``, ``ingress_resume``,
  **pepsi-keydisc**\ (1) releasing parked mail, **pepsi-failure-bouncer**\ (1)
  and the **pepsi-queue**\ (1) repair commands.

  ``POLL_INTERVAL`` remains the safety net behind all of it: a wake-up that is
  never sent, or is lost, costs latency and never a message.

* **Stage transitions do not wait for the disk.** A stage worker's connection
  runs with ``synchronous_commit`` off (``[pepsi-postgres]
  WORKER_SYNCHRONOUS_COMMIT``), so the pipeline's writes group-commit instead of
  each paying its own flush. Message *admission* is unaffected — ingress always
  commits synchronously, so a ``250`` still means the message reached the disk.
  The exposure this adds is that a machine crash can lose the last few hundred
  milliseconds of stage *transitions*, whereupon the message is found at the
  previous stage and runs through it again — which is exactly what already
  happens when a worker is killed between doing its work and committing.

* **Bookkeeping has exactly one writer.** The per-stage and global counters are
  accumulated in the dispatcher's memory and written in a single transaction —
  every ``STATS_INTERVAL``, when the pipeline goes idle, and on shutdown. No
  worker writes them, and that is deliberate: ``stage_stats`` holds one row per
  stage, so a counter written per message would put every worker of a stage in a
  queue behind that stage's single row — bookkeeping contending where the mail
  itself never does, since exactly one writer ever touches a given message. A
  stage *fused* into another stage's pass is therefore **reported** to the
  dispatcher on the status line that pass already writes, and folded into the
  counters it already keeps, so per-stage accounting stays exact with no write at
  all. The cost is the usual one for statistics: deltas not yet flushed are lost
  if the dispatcher dies.

Per-stage processing cost
=========================

The dispatcher times every stage execution, so the cost of a stage is measured
directly, with no instrumentation of the stage code. The table below is the wall
time a single message spends *inside* each stage (worker reads the row, runs the
stage logic, commits the outcome), as a function of message size. Each cell is
the mean of three messages, in **milliseconds per message**. So that each stage
is observable as its own dispatched worker, this benchmark runs with ``[pepsi]
ALLOW_FUSION = no``; the latency and goodput benchmarks below leave fusion on,
measuring the system as deployed.

Because a single message only traverses one path through the pipeline, the
benchmark drives a set of *scenarios* — one message per path — and accumulates
their per-stage timings into one matrix, so that **every stage of the deployed
pipeline** is covered. A scenario can nevertheless only reach stages that
deployment routes a message through, so a second half of the benchmark measures
the remaining stage programs **in isolation**, by injecting a message straight at
a benchmark-only section configured with that stage's own sensible defaults.
Those rows are tabulated separately below. See the :doc:`Benchmark Suite
<benchmarks>` chapter for both lists. The relay stages (``smarthost``,
``internet``, ``delaytest``, ``bench-lmtp``) are **network- or MDA-inclusive**:
their figure is the next hop's round-trip, not local CPU, and so is not
comparable to the body-processing stages.

.. list-table:: Deployed pipeline — per-stage cost (ms/message) vs. message size
   :header-rows: 1
   :widths: 24 12 12 12 12 12

   * - stage (program)
     - 10 KB
     - 100 KB
     - 1 MB
     - 10 MB
     - 20 MB
   * - ``init`` (arc)
     - 54
     - 56
     - 83
     - 428
     - 715
   * - ``route`` (if)
     - 8
     - 8
     - 8
     - 20
     - 20
   * - ``decrypt``
     - 11
     - 16
     - 63
     - 399
     - 728
   * - ``detect-language``
     - 31
     - 131
     - 1187
     - 11620
     - 22931
   * - ``block-language``
     - 7
     - 7
     - 8
     - 19
     - 18
   * - ``check-whitelist``
     - 8
     - 7
     - 8
     - 20
     - 18
   * - ``anti-spam`` (cleared)
     - 7
     - 7
     - 10
     - 44
     - 67
   * - ``anti-spam`` (pay-gated) [#optin]_
     - 119
     - 129
     - 140
     - 166
     - 200
   * - ``forward`` (dot-forward)
     - 10
     - 9
     - 14
     - 47
     - 69
   * - ``local`` (maildir)
     - 181
     - 226
     - 206
     - 767
     - 1183
   * - ``srs``
     - 8
     - 9
     - 17
     - n/a [#cap]_
     - n/a [#cap]_
   * - ``dkim-sign-relay``
     - 39
     - 47
     - 156
     - n/a [#cap]_
     - n/a [#cap]_
   * - ``smarthost`` (relay) [#net]_
     - 130
     - 168
     - 347
     - n/a [#cap]_
     - n/a [#cap]_
   * - ``edit-settings``
     - 7
     - 7
     - 19
     - 42
     - 65
   * - ``auto-whitelist``
     - 10
     - 9
     - 19
     - 19
     - 21
   * - ``delay-route`` (if)
     - 8
     - 8
     - 17
     - 17
     - 18
   * - ``encrypt``
     - 14
     - 15
     - 29
     - 65
     - 91
   * - ``dkim-sign``
     - 39
     - 47
     - 156
     - 679
     - 1251
   * - ``internet`` (relay) [#net]_
     - 172
     - 198
     - 383
     - 1366
     - 2396
   * - ``bounce``
     - 7
     - 16
     - 17
     - 17
     - 16
   * - ``bounce-language``
     - 7
     - 7
     - 7
     - 15
     - 15
   * - ``bounce-payment`` [#optin]_
     - 18
     - 17
     - 18
     - 17
     - 18
   * - ``delaytest`` (relay) [#optin]_ [#net]_
     - 6029
     - 6024
     - 6025
     - 6027
     - 6030
.. [#net] Network-inclusive: the figure is the next-hop SMTP round-trip, not
   local CPU. ``delaytest`` deliberately relays to an unrouteable TEST-NET-1
   address, so its ~6.1 s is its ``CONNECT_TIMEOUT`` — flat by construction, and
   the reason a stage that waits on a dead next hop is not free.
.. [#optin] Measured by an opt-in scenario (``payment`` / ``delay``), run
   separately from the main matrix.
.. [#cap] Not measurable on this topology: the smarthost these stages relay to
   refuses messages over 10 MB (``552 message size … exceeds server limit``), so
   no message of that size completes the path. A local-part cost, not a limit of
   Pepsi's.

The isolated half covers the stage programs this deployment routes no message
through. Each was measured on its own, injected directly at a benchmark-only
section, so these figures carry no ``pepsi-ingress`` or SMTP cost at all:

.. list-table:: Isolated stages — per-stage cost (ms/message) vs. message size
   :header-rows: 1
   :widths: 24 12 12 12 12 12

   * - stage (program)
     - 10 KB
     - 100 KB
     - 1 MB
     - 10 MB
     - 20 MB
   * - ``discard``
     - 8
     - 8
     - 7
     - 8
     - 8
   * - ``aliases``
     - 8
     - 9
     - 8
     - 8
     - 8
   * - ``route``
     - 9
     - 8
     - 8
     - 8
     - 8
   * - ``vacation``
     - 9
     - 10
     - 10
     - 9
     - 9
   * - ``autocrypt-learn``
     - 8
     - 7
     - 11
     - 34
     - 56
   * - ``vks-confirm``
     - 8
     - 7
     - 10
     - 31
     - 53
   * - ``auto-pay`` (not a demand)
     - 6
     - 7
     - 10
     - 33
     - 54
   * - ``secure-link``
     - 94
     - 93
     - 132
     - 668
     - 1559
   * - ``relay-to-lmtp`` [#mda]_
     - 185
     - 282
     - 517
     - 587
     - 1150
.. [#mda] MDA-inclusive: Dovecot receives the message over LMTP, runs the
   recipient's Sieve script and writes the mailbox, all inside this figure.

Two clear classes emerge:

* **Metadata-only stages are flat and cheap.** ``if`` (the ``route``/``delay-route``
  branchers), ``srs``, ``check-whitelist``, ``aliases``, ``route`` and
  ``discard`` declare ``Load::Metadata`` — they never touch the message body —
  and cost roughly **6–20 ms regardless of size**. At that floor the stage logic
  is negligible; the time is the fixed per-message overhead described above: the
  single load ``SELECT`` and the single terminal ``UPDATE``/``DELETE``, plus
  worker scheduling. The cheapest stages measured, ``discard`` and ``route`` at
  ~8 ms, are effectively that floor with no logic on top of it, and it is why a
  message's end-to-end cost is dominated by *how many* stages it traverses rather
  than by any one stage's work.

  This floor roughly halved (from ~12–25 ms) when stage workers stopped
  committing synchronously: at this size the per-message cost *is* the two
  round-trips, and one of them no longer waits for a disk flush.

* **Body-reading stages scale with message size**, and they are unaffected by
  the coordination changes — their cost is CPU spent on the body, not a
  round-trip. ``detect-language`` is the dominant cost and grows essentially
  **linearly with the body (~1.14 s/MB)**: 30 ms at 10 KB but 1.1 s at 1 MB and
  **23 s at 20 MB**, because the ``lingua`` detector scans the entire decoded
  body. ``dkim-sign`` (1.3 s at 20 MB), ``secure-link`` (1.6 s — it AEAD-seals the
  whole message), ``init`` (ARC verify, hash and seal, 0.7 s) and ``decrypt``
  (0.7 s) climb far more gently, and ``local`` — the setuid Maildir-writer
  handling the full message — reaches only ~1 s at 20 MB.

Three results are worth calling out beyond those classes:

* **Language detection on large messages is the single most expensive operation
  in the pipeline** — an order of magnitude above every other body-reading stage.
  Operators who do not need it should leave ``detect-language`` out of the chain,
  or place a size-based ``if`` branch in front of it so that very large messages
  bypass it.

* **The pay-to-send gate costs ~10× more when it actually gates.** ``anti-spam``
  is ~13 ms for a message ``check-whitelist`` has cleared (it short-circuits on
  ``state.spam``) but ~150 ms when it must create a Taler order, because that is a
  round-trip to the merchant backend. The figure is therefore a property of the
  merchant's latency, and whitelisting is what keeps it off the common path.

* **The stages that only *inspect* are nearly free.** ``autocrypt-learn``,
  ``vks-confirm`` and ``auto-pay`` all declare ``Load::Full`` and all sit at the
  metadata floor for ordinary mail, rising only with the cost of *loading* a
  larger body (13 → 57–64 ms from 10 KB to 20 MB). Adding them to a pipeline
  costs about one more hop, not one more scan.

End-to-end latency (idle)
=========================

With the system otherwise idle, ``09-latency-bench.sh`` injects a short (~1 KB)
message and polls for its arrival in the destination ``Maildir``, timing the
whole inject→deliver round-trip across the nine-stage inbound path. Over 50
samples:

.. list-table::
   :header-rows: 1
   :widths: 20 16 16 16 16 16

   * - metric
     - min
     - mean
     - median
     - p95
     - max
   * - latency (ms)
     - 162
     - 225
     - 195
     - 423
     - 494

All 50 samples were delivered with zero timeouts. The sum of the mean per-stage
processing time over the same run — counting only the nine stages every sample
traversed — was **148 ms**, against a **median end-to-end latency of 195 ms**. On
an idle system the latency is therefore very nearly the stage work itself:
dispatcher scheduling and the hand-off between stages add a few milliseconds per
hop, not a poll interval. The advance path is driven by the worker's status
report rather than by a notification, so ``POLL_INTERVAL`` does not gate it
either.

Run-to-run spread at this scale is tens of milliseconds, so read the stage-work
total as a band of roughly 130–150 ms rather than a point. Most of what keeps it
there is that a stage hop does not wait for the write-ahead log: with
``WORKER_SYNCHRONOUS_COMMIT`` on, each of the nine hops pays its own flush, which
on this host adds some 80 ms to the total.

Idle latency is the per-message work a single message pays on an empty queue. It
is *not* the reciprocal of the saturation throughput below — the two are limited
by different things, and the gap between them is the subject of the next section.

Saturation goodput
==================

``10-goodput-bench.sh`` ramps the offered load every five seconds and watches the
rate at which Pepsi actually *completes* messages (the global
``messages_processed`` counter, incremented only when a row leaves the pipeline)
while the queue depth grows, across three load and delivery paths that isolate
successive layers. Each ramp is preceded by a warm-up burst that is drained and
discarded, so the measured steps see live stage workers rather than the
dispatcher forking them. So that the benchmark measures the *pipeline* rather
than the SMTP front end, it raises the admission caps for the duration —
``[pepsi-ingress]
CONN_RATE_PER_SECOND``/``CONN_RATE_BURST``/``MAX_CONNECTIONS`` and the ingress
``DB_POOL_SIZE`` (to 4) — and quadruples the outbound relay's ``PARALLELISM`` (to
16); it restores the deployed configuration on exit. It also samples the host's
system-wide CPU utilisation (``/proc/stat``) and 1-minute load average over each
ramp step, so the goodput ceiling can be read against how busy the machine
actually is.

.. list-table:: Sustained goodput at saturation
   :header-rows: 1
   :widths: 14 18 14 56

   * - scenario
     - sustained
     - peak CPU
     - bottleneck
   * - A — pipeline
     - ~11–16 msg/s
     - 40 %
     - CPU and write-ahead-log flush (see below)
   * - B — network
     - ~16 msg/s
     - 47 %
     - the same, reached through the SMTP front end
   * - C — relay
     - ~0 msg/s (network-bound)
     - 6 %
     - per-message round-trips to the remote MX, not the local pipeline

The three paths are constructed to isolate successive layers: **A** injects
locally and delivers to a unix ``Maildir`` (the pure stage pipeline); **B** is
the same delivery reached over the network through the SMTP front end (A vs B
isolates connection admission and STARTTLS); **C** relays to a *real* external MX
(B vs C isolates the outbound relay — DNS/MX, TLS, remote-MTA pacing).

* **The local-delivery pipeline sustains ~14–16 msg/s, and A and B remain close
  enough to be treated as equal.** Under a force-fed queue that grows without
  bound both plateau, confirmed by the matching count of files delivered into the
  destination ``Maildir``; the small edge B shows is within the run-to-run spread.
  That the two agree is itself the result: the SMTP front end costs nothing
  measurable here, because both paths hit the same ceiling further in. Throughout
  every ramp of every scenario the dispatcher recorded **zero failed messages and
  zero serialization retries** — the ceiling is not a correctness problem, and no
  ``40001 could not serialize access`` event occurs.

* **Nothing inside the database contends.** The ramp's plateau detector is noisy
  at this scale, so the sustained figure is better read from a controlled drain:
  injecting 500 messages at the head of the inbound chain and timing the queue to
  empty gives **13.7 / 16.2 / 14.7 msg/s** over three trials (~14.9 mean), which
  is the same path scenario A exercises.

  Sampling ``pg_stat_activity`` and ``pg_locks`` once a second through such a
  drain — with the injection harness finished, so only the pipeline is running —
  shows **no ungranted lock of any kind, in every sample**. The only waits left
  are ``IO/WALSync`` and ``LWLock/WALWrite``: the write-ahead log, plus sessions
  actually executing. Host CPU sits at **67–84 %** across the four cores for the
  duration.

  That is the intended end state for a queue of this shape — the machine doing
  work and the disk taking writes — and it is what the three coordination
  properties in *How the pipeline spends its time* buy. Each of them removes a
  point at which PostgreSQL would otherwise have had to serialise something
  database-wide; without them this host plateaus at a *fraction* of the figures
  above with most of its cores idle, waiting on a lock rather than on work.

* **The outbound relay is network-bound.** Scenario C relays to a real external
  MX, so each message pays DNS/MX resolution, an outbound TLS handshake and a
  remote SMTP transaction. Over the short ramp windows essentially no completions
  register (the per-message remote latency dominates) and the host's CPU barely
  moves (~6 % peak); the relay's view of the next hop showed **no remote
  rate-limiting**. Its throughput is set by the remote round-trip latency, not by
  anything in Pepsi's pipeline, and is not comparable to the local-delivery
  figures of A and B.

.. note::

   A saturated queue only ever reports its *outermost* constraint, so a
   throughput figure on its own says nothing about what is actually limiting it,
   and a plausible story about where the time ought to go is not evidence. The
   check that settles it is ``pg_locks`` and ``pg_stat_activity`` under load:
   which sessions are blocked, on what, running which statement. That is why the
   figures above are quoted alongside their lock sample — so the claim can be
   checked rather than believed, and so re-measuring after a change means
   re-running the same check.

Roughly 15 msg/s is on the order of 1.3 million messages a day — far more than a
personal or small-group forwarder needs, on a deliberately modest decade-old
machine that is by then genuinely busy rather than blocked. The remaining
headroom lies in the per-hop coordination cost: fewer
stages on the path, and stage **fusion**, which collapses a run of fusible
body-free stages into one load, one terminal write and one round of bookkeeping
instead of *n*. Fusion is on by default and was deliberately left on for this
measurement; the per-stage table above is the one place it is disabled, so that
each stage is separately observable.

A message costs about **33 database transactions** end to end across the
nine-stage inbound path — the load and terminal write of each hop, plus
admission — which is the quantity the coordination properties above are really
about, and the one to watch when adding a stage.

Tuning notes
============

* **Parallelism and the connection budget.** Each stage worker is a process that
  holds **one** database connection, so the live connection count is
  approximately ``Σ PARALLELISM`` over busy stages, plus the dispatcher pool and
  the ``pepsi-ingress``/``pepsi-httpd`` pools. This sum must stay under
  PostgreSQL's ``max_connections``. The default ``PARALLELISM`` is **4** and the
  default per-server ``DB_POOL_SIZE`` is **1** precisely so a default install
  fits a default server. Raising ``PARALLELISM`` to push throughput, or
  ``DB_POOL_SIZE`` to admit more concurrent sessions, means re-checking that
  budget. If it is exceeded, workers that cannot get a connection report the
  database-overload status and the dispatcher requeues the message and throttles
  the busiest stage rather than losing mail — see
  :doc:`programs/pepsi-dispatch` (*Database-overload backpressure*). The remedy
  for sustained pressure is to raise ``max_connections`` or lower
  ``PARALLELISM``, not to rely on the throttle.

* **Fewer hops beats more workers.** A message pays one statistics upsert — the
  currently binding contention — per stage it advances through, so the goodput
  ceiling still moves with path length. Adding ``PARALLELISM`` puts more workers
  behind the same row lock; it does not raise the ceiling. Shortening the path
  does.

* **Leave stage fusion on.** ``[pepsi] ALLOW_FUSION`` (on by default) runs a
  fusible body-free successor inside its predecessor's worker on the
  already-loaded row, so a run of *n* such stages costs one load, one terminal
  write and one round of bookkeeping instead of *n*. It remains the single most
  valuable throughput mechanism in the system, and the only reason to turn it off
  is to measure stages separately, as the per-stage table above does. Marking a
  cheap metadata-only stage ``FUSION = yes`` is worth more than any pool knob.

* **Pipeline depth before parallelism.** For the same reason, reach for a stage's
  ``QUEUE_LIMIT`` (messages pipelined to one worker) before its ``PARALLELISM``.
  ``QUEUE_LIMIT`` lifts a stage's in-flight capacity to ``QUEUE_LIMIT ×
  PARALLELISM`` and removes a coordinator round-trip per message **without**
  adding worker processes or database connections, so it does not enter the
  budget above; and a body-free stage's worker commits a whole pipelined batch in
  one arrayed ``UPDATE``, which is one notification for the batch. It **defaults
  to 4** — pipelining is on out of the box — so the knob is often about
  *lowering* it: set ``QUEUE_LIMIT = 1`` for stages whose per-message work is long
  and uneven, chiefly the network relays, where a pipelined message would
  otherwise wait behind a slow sibling.

* **Drop work you do not need.** The cheapest stage is one that is not in the
  chain — cheapest twice over, since it costs neither its own work nor a hop's
  share of the coordination. ``detect-language`` in particular is worth omitting,
  or guarding with a size branch, unless language filtering is actually used.

* **Leave ``WORKER_SYNCHRONOUS_COMMIT`` off** unless a stage in your pipeline has
  an external side effect that must not be repeated even across a power loss.
  Turning it on costs roughly a write-ahead-log flush per stage hop; on this host
  that is about 10 ms a hop, so it adds some 80 ms to the ~130–150 ms of stage
  work a message pays across the nine-stage path.

* **Front-end caps are separate from pipeline throughput.** A high-volume single
  sender can hit ``tripwire``'s per-client connection limit
  (``CONN_RATE_PER_SECOND`` / ``CONN_RATE_BURST`` / ``MAX_CONNECTIONS`` in
  ``[pepsi-ingress]``) long before the pipeline saturates; that is an admission
  knob, tuned independently of the delivery-side numbers above. The goodput
  benchmark raises these caps deliberately so it can measure the pipeline rather
  than the front end.

Reproducing
===========

The benchmarks are not part of ``make integrationtests`` (they are slow and
deliberately load the MTA). Run them against a deployed pipeline with::

   tests/08-perstage-bench.sh tests/test-accounts.ini
   tests/09-latency-bench.sh  tests/test-accounts.ini
   tests/10-goodput-bench.sh  tests/test-accounts.ini
   # or all three in order:
   make benchmarks ACCOUNTS=tests/test-accounts.ini

Each script lowers the dispatcher's ``STATS_INTERVAL``/``POLL_INTERVAL`` for the
duration of the run so the counters flush promptly, and **restores the deployed
configuration on exit** (including the admission caps the goodput run raises). See
the :doc:`Benchmark Suite <benchmarks>` chapter for the full methodology and the
available tunables (message sizes, sample counts, ramp parameters, admission
overrides).