21. Performance

This chapter reports measured performance of the Pepsi pipeline: the relative cost of each stage, the end-to-end latency of a single message on an idle system, and the goodput sustained under load — together with the system-wide CPU utilisation at saturation, which tells us whether that goodput is limited by compute or by coordination. The numbers come from the three benchmark scripts under tests/ (08-perstage-bench.sh, 09-latency-bench.sh and 10-goodput-bench.sh); how they obtain their figures — by snapshotting the dispatcher’s own pepsi.stage_stats and pepsi.dispatch_stats counters, by counting delivered Maildir files, and by sampling /proc/stat and /proc/loadavg on the host — is documented in the Benchmark Suite chapter and is not repeated here.

Note

Absolute numbers are specific to the test host and pipeline below and to a single point-in-time run; treat them as orders of magnitude and ratios, not guarantees. They were gathered on a deliberately modest, decade-old machine, so a contemporary server will be considerably faster. What is portable is the shape of the results — which stages are cheap, which scale with message size, and where the throughput ceiling sits.

21.1. Test environment

All measurements were taken on tripwire, the live MTA-under-test of the Test Suite topology (inbound mail from the foreign host frontier, authorized outbound from enoki, local delivery to a unix Maildir). The whole pipeline was run at log level warn for the duration (via the [pepsi] LOG knob, which every component — the dispatcher and the stage workers it spawns — honours) so per-message logging does not inflate the figures.

Benchmark host (tripwire)

CPU	Intel Core i7-920 @ 2.67 GHz (Nehalem, 2009): 1 socket, 4 cores, 1 thread per core (no SMT)
Memory	11 GiB RAM
Storage	Samsung 860 PRO SATA SSD, holding the PostgreSQL data directory (/var) and thus the write-ahead log
OS / kernel	Debian 13 (trixie), Linux 6.12 (x86-64), glibc 2.42
Database	PostgreSQL 15.13, max_connections = 100, peer-auth over the local socket
Pipeline	the deployment tests/01-deploy.sh writes: the full inbound chain arc → if (route) → decrypt → detect-language → block-language → check-whitelist → anti-spam → dot-forward → local Maildir delivery, its relay tail (srs → dkim-sign → relay-to-smarthost), the outbound submission chain (edit-settings → auto-whitelist → if → encrypt → dkim-sign → relay-to-internet) and three bounce stages; the init (arc) stage at PARALLELISM = 16 and every other stage at the default PARALLELISM = 4, with the default QUEUE_LIMIT = 4 (worker pipelining on) throughout

21.2. How the pipeline spends its time

Five properties of the architecture set the numbers that follow, so they are worth stating up front. The last three are consequences of one principle: at this message rate, the pipeline’s cost is dominated not by the work a stage does but by how often it makes the database coordinate — so it asks the database to coordinate only where the information genuinely is.

• Every stage pays a small fixed per-message database cost. A stage worker loads its row in one SELECT (the row and its applicable per-address settings, in a single round-trip) and commits its outcome in one terminal UPDATE/DELETE (or one fan-out stored function). That pair of round-trips, plus worker scheduling, is the floor every message pays at every hop, independent of message size.

• Several mechanisms remove or amortise those round-trips. The dispatcher claims work for all stages in a single window-function UPDATE (a per-stage LATERAL … LIMIT stops each stage’s index scan at its capacity) and pipelines up to QUEUE_LIMIT messages to each worker, so completing a message needs no prior coordinator round-trip. The load-and-advance hot path runs under READ COMMITTED rather than SERIALIZABLE — safe because at most one writer ever touches a given row (the dispatcher is the sole claimer). A body-free stage’s worker loads and commits a whole pipelined batch of messages in one SELECT … = ANY and one arrayed UPDATE. And stage fusion ([pepsi] ALLOW_FUSION, on by default) runs a fast body-free successor marked FUSION = yes — if, srs, check-whitelist, auto-whitelist, block-language, discard — inside its predecessor’s worker process on the already-loaded row, collapsing a run of n such stages from n loads + n advances + n hand-offs into a single load, the stages’ work, and one terminal write. Fusion changes no outcome: a stage that cannot be fused (it needs the body, is not folded into the unified binary, or its successor is not marked FUSION) simply falls back to a normal dispatched hop.

• Hand-offs between stages are deliberately silent. The dispatcher learns there is work from a LISTEN on the ingress channel, but a stage advancing a message does not notify. It does not need to: the worker reports each finished message on its standard output, and the coordinator answers that report with the same full capacity-bounded claim a notification would have triggered. A notification on the advance path would tell the dispatcher something it is about to be told anyway, over a cheaper channel.

  Silence on that path is worth the discipline it costs, because a notification is far from free at this rate: PostgreSQL holds a database-wide AccessExclusive lock from the moment a transaction queues one until it commits, so notifying transactions cannot group-commit and each hop would serialise the whole database’s commit path. A notification is therefore issued only where the information is genuinely new — by writers outside the dispatcher’s loop: message admission (coalesced, once per burst rather than once per message), ingress_inject, ingress_resume, pepsi-keydisc(1) releasing parked mail, pepsi-failure-bouncer(1) and the pepsi-queue(1) repair commands.

  POLL_INTERVAL remains the safety net behind all of it: a wake-up that is never sent, or is lost, costs latency and never a message.

• Stage transitions do not wait for the disk. A stage worker’s connection runs with synchronous_commit off ([pepsi-postgres] WORKER_SYNCHRONOUS_COMMIT), so the pipeline’s writes group-commit instead of each paying its own flush. Message admission is unaffected — ingress always commits synchronously, so a 250 still means the message reached the disk. The exposure this adds is that a machine crash can lose the last few hundred milliseconds of stage transitions, whereupon the message is found at the previous stage and runs through it again — which is exactly what already happens when a worker is killed between doing its work and committing.

• Bookkeeping has exactly one writer. The per-stage and global counters are accumulated in the dispatcher’s memory and written in a single transaction — every STATS_INTERVAL, when the pipeline goes idle, and on shutdown. No worker writes them, and that is deliberate: stage_stats holds one row per stage, so a counter written per message would put every worker of a stage in a queue behind that stage’s single row — bookkeeping contending where the mail itself never does, since exactly one writer ever touches a given message. A stage fused into another stage’s pass is therefore reported to the dispatcher on the status line that pass already writes, and folded into the counters it already keeps, so per-stage accounting stays exact with no write at all. The cost is the usual one for statistics: deltas not yet flushed are lost if the dispatcher dies.

21.3. Per-stage processing cost

The dispatcher times every stage execution, so the cost of a stage is measured directly, with no instrumentation of the stage code. The table below is the wall time a single message spends inside each stage (worker reads the row, runs the stage logic, commits the outcome), as a function of message size. Each cell is the mean of three messages, in milliseconds per message. So that each stage is observable as its own dispatched worker, this benchmark runs with [pepsi] ALLOW_FUSION = no; the latency and goodput benchmarks below leave fusion on, measuring the system as deployed.

Because a single message only traverses one path through the pipeline, the benchmark drives a set of scenarios — one message per path — and accumulates their per-stage timings into one matrix, so that every stage of the deployed pipeline is covered. A scenario can nevertheless only reach stages that deployment routes a message through, so a second half of the benchmark measures the remaining stage programs in isolation, by injecting a message straight at a benchmark-only section configured with that stage’s own sensible defaults. Those rows are tabulated separately below. See the Benchmark Suite chapter for both lists. The relay stages (smarthost, internet, delaytest, bench-lmtp) are network- or MDA-inclusive: their figure is the next hop’s round-trip, not local CPU, and so is not comparable to the body-processing stages.

Deployed pipeline — per-stage cost (ms/message) vs. message size

stage (program)	10 KB	100 KB	1 MB	10 MB	20 MB
init (arc)	54	56	83	428	715
route (if)	8	8	8	20	20
decrypt	11	16	63	399	728
detect-language	31	131	1187	11620	22931
block-language	7	7	8	19	18
check-whitelist	8	7	8	20	18
anti-spam (cleared)	7	7	10	44	67
anti-spam (pay-gated) [2]	119	129	140	166	200
forward (dot-forward)	10	9	14	47	69
local (maildir)	181	226	206	767	1183
srs	8	9	17	n/a [3]	n/a [3]
dkim-sign-relay	39	47	156	n/a [3]	n/a [3]
smarthost (relay) [1]	130	168	347	n/a [3]	n/a [3]
edit-settings	7	7	19	42	65
auto-whitelist	10	9	19	19	21
delay-route (if)	8	8	17	17	18
encrypt	14	15	29	65	91
dkim-sign	39	47	156	679	1251
internet (relay) [1]	172	198	383	1366	2396
bounce	7	16	17	17	16
bounce-language	7	7	7	15	15
bounce-payment [2]	18	17	18	17	18
delaytest (relay) [2] [1]	6029	6024	6025	6027	6030

[1] (1,2,3)

Network-inclusive: the figure is the next-hop SMTP round-trip, not local CPU. delaytest deliberately relays to an unrouteable TEST-NET-1 address, so its ~6.1 s is its CONNECT_TIMEOUT — flat by construction, and the reason a stage that waits on a dead next hop is not free.

[2] (1,2,3)

Measured by an opt-in scenario (payment / delay), run separately from the main matrix.

[3] (1,2,3,4,5,6)

Not measurable on this topology: the smarthost these stages relay to refuses messages over 10 MB (552 message size … exceeds server limit), so no message of that size completes the path. A local-part cost, not a limit of Pepsi’s.

The isolated half covers the stage programs this deployment routes no message through. Each was measured on its own, injected directly at a benchmark-only section, so these figures carry no pepsi-ingress or SMTP cost at all:

Isolated stages — per-stage cost (ms/message) vs. message size

stage (program)	10 KB	100 KB	1 MB	10 MB	20 MB
discard	8	8	7	8	8
aliases	8	9	8	8	8
route	9	8	8	8	8
vacation	9	10	10	9	9
autocrypt-learn	8	7	11	34	56
vks-confirm	8	7	10	31	53
auto-pay (not a demand)	6	7	10	33	54
secure-link	94	93	132	668	1559
relay-to-lmtp [4]	185	282	517	587	1150

[4]

MDA-inclusive: Dovecot receives the message over LMTP, runs the recipient’s Sieve script and writes the mailbox, all inside this figure.

Two clear classes emerge:

• Metadata-only stages are flat and cheap. if (the route/delay-route branchers), srs, check-whitelist, aliases, route and discard declare Load::Metadata — they never touch the message body — and cost roughly 6–20 ms regardless of size. At that floor the stage logic is negligible; the time is the fixed per-message overhead described above: the single load SELECT and the single terminal UPDATE/DELETE, plus worker scheduling. The cheapest stages measured, discard and route at ~8 ms, are effectively that floor with no logic on top of it, and it is why a message’s end-to-end cost is dominated by how many stages it traverses rather than by any one stage’s work.

  This floor roughly halved (from ~12–25 ms) when stage workers stopped committing synchronously: at this size the per-message cost is the two round-trips, and one of them no longer waits for a disk flush.

• Body-reading stages scale with message size, and they are unaffected by the coordination changes — their cost is CPU spent on the body, not a round-trip. detect-language is the dominant cost and grows essentially linearly with the body (~1.14 s/MB): 30 ms at 10 KB but 1.1 s at 1 MB and 23 s at 20 MB, because the lingua detector scans the entire decoded body. dkim-sign (1.3 s at 20 MB), secure-link (1.6 s — it AEAD-seals the whole message), init (ARC verify, hash and seal, 0.7 s) and decrypt (0.7 s) climb far more gently, and local — the setuid Maildir-writer handling the full message — reaches only ~1 s at 20 MB.

Three results are worth calling out beyond those classes:

• Language detection on large messages is the single most expensive operation in the pipeline — an order of magnitude above every other body-reading stage. Operators who do not need it should leave detect-language out of the chain, or place a size-based if branch in front of it so that very large messages bypass it.

• The pay-to-send gate costs ~10× more when it actually gates. anti-spam is ~13 ms for a message check-whitelist has cleared (it short-circuits on state.spam) but ~150 ms when it must create a Taler order, because that is a round-trip to the merchant backend. The figure is therefore a property of the merchant’s latency, and whitelisting is what keeps it off the common path.

• The stages that only *inspect* are nearly free. autocrypt-learn, vks-confirm and auto-pay all declare Load::Full and all sit at the metadata floor for ordinary mail, rising only with the cost of loading a larger body (13 → 57–64 ms from 10 KB to 20 MB). Adding them to a pipeline costs about one more hop, not one more scan.

21.4. End-to-end latency (idle)

With the system otherwise idle, 09-latency-bench.sh injects a short (~1 KB) message and polls for its arrival in the destination Maildir, timing the whole inject→deliver round-trip across the nine-stage inbound path. Over 50 samples:

metric	min	mean	median	p95	max
latency (ms)	162	225	195	423	494

All 50 samples were delivered with zero timeouts. The sum of the mean per-stage processing time over the same run — counting only the nine stages every sample traversed — was 148 ms, against a median end-to-end latency of 195 ms. On an idle system the latency is therefore very nearly the stage work itself: dispatcher scheduling and the hand-off between stages add a few milliseconds per hop, not a poll interval. The advance path is driven by the worker’s status report rather than by a notification, so POLL_INTERVAL does not gate it either.

Run-to-run spread at this scale is tens of milliseconds, so read the stage-work total as a band of roughly 130–150 ms rather than a point. Most of what keeps it there is that a stage hop does not wait for the write-ahead log: with WORKER_SYNCHRONOUS_COMMIT on, each of the nine hops pays its own flush, which on this host adds some 80 ms to the total.

Idle latency is the per-message work a single message pays on an empty queue. It is not the reciprocal of the saturation throughput below — the two are limited by different things, and the gap between them is the subject of the next section.

21.5. Saturation goodput

10-goodput-bench.sh ramps the offered load every five seconds and watches the rate at which Pepsi actually completes messages (the global messages_processed counter, incremented only when a row leaves the pipeline) while the queue depth grows, across three load and delivery paths that isolate successive layers. Each ramp is preceded by a warm-up burst that is drained and discarded, so the measured steps see live stage workers rather than the dispatcher forking them. So that the benchmark measures the pipeline rather than the SMTP front end, it raises the admission caps for the duration — [pepsi-ingress] CONN_RATE_PER_SECOND/CONN_RATE_BURST/MAX_CONNECTIONS and the ingress DB_POOL_SIZE (to 4) — and quadruples the outbound relay’s PARALLELISM (to 16); it restores the deployed configuration on exit. It also samples the host’s system-wide CPU utilisation (/proc/stat) and 1-minute load average over each ramp step, so the goodput ceiling can be read against how busy the machine actually is.

Sustained goodput at saturation

scenario	sustained	peak CPU	bottleneck
A — pipeline	~11–16 msg/s	40 %	CPU and write-ahead-log flush (see below)
B — network	~16 msg/s	47 %	the same, reached through the SMTP front end
C — relay	~0 msg/s (network-bound)	6 %	per-message round-trips to the remote MX, not the local pipeline

The three paths are constructed to isolate successive layers: A injects locally and delivers to a unix Maildir (the pure stage pipeline); B is the same delivery reached over the network through the SMTP front end (A vs B isolates connection admission and STARTTLS); C relays to a real external MX (B vs C isolates the outbound relay — DNS/MX, TLS, remote-MTA pacing).

• The local-delivery pipeline sustains ~14–16 msg/s, and A and B remain close enough to be treated as equal. Under a force-fed queue that grows without bound both plateau, confirmed by the matching count of files delivered into the destination Maildir; the small edge B shows is within the run-to-run spread. That the two agree is itself the result: the SMTP front end costs nothing measurable here, because both paths hit the same ceiling further in. Throughout every ramp of every scenario the dispatcher recorded zero failed messages and zero serialization retries — the ceiling is not a correctness problem, and no 40001 could not serialize access event occurs.

• Nothing inside the database contends. The ramp’s plateau detector is noisy at this scale, so the sustained figure is better read from a controlled drain: injecting 500 messages at the head of the inbound chain and timing the queue to empty gives 13.7 / 16.2 / 14.7 msg/s over three trials (~14.9 mean), which is the same path scenario A exercises.

  Sampling pg_stat_activity and pg_locks once a second through such a drain — with the injection harness finished, so only the pipeline is running — shows no ungranted lock of any kind, in every sample. The only waits left are IO/WALSync and LWLock/WALWrite: the write-ahead log, plus sessions actually executing. Host CPU sits at 67–84 % across the four cores for the duration.

  That is the intended end state for a queue of this shape — the machine doing work and the disk taking writes — and it is what the three coordination properties in How the pipeline spends its time buy. Each of them removes a point at which PostgreSQL would otherwise have had to serialise something database-wide; without them this host plateaus at a fraction of the figures above with most of its cores idle, waiting on a lock rather than on work.

• The outbound relay is network-bound. Scenario C relays to a real external MX, so each message pays DNS/MX resolution, an outbound TLS handshake and a remote SMTP transaction. Over the short ramp windows essentially no completions register (the per-message remote latency dominates) and the host’s CPU barely moves (~6 % peak); the relay’s view of the next hop showed no remote rate-limiting. Its throughput is set by the remote round-trip latency, not by anything in Pepsi’s pipeline, and is not comparable to the local-delivery figures of A and B.

Note

A saturated queue only ever reports its outermost constraint, so a throughput figure on its own says nothing about what is actually limiting it, and a plausible story about where the time ought to go is not evidence. The check that settles it is pg_locks and pg_stat_activity under load: which sessions are blocked, on what, running which statement. That is why the figures above are quoted alongside their lock sample — so the claim can be checked rather than believed, and so re-measuring after a change means re-running the same check.

Roughly 15 msg/s is on the order of 1.3 million messages a day — far more than a personal or small-group forwarder needs, on a deliberately modest decade-old machine that is by then genuinely busy rather than blocked. The remaining headroom lies in the per-hop coordination cost: fewer stages on the path, and stage fusion, which collapses a run of fusible body-free stages into one load, one terminal write and one round of bookkeeping instead of n. Fusion is on by default and was deliberately left on for this measurement; the per-stage table above is the one place it is disabled, so that each stage is separately observable.

A message costs about 33 database transactions end to end across the nine-stage inbound path — the load and terminal write of each hop, plus admission — which is the quantity the coordination properties above are really about, and the one to watch when adding a stage.

21.6. Tuning notes

• Parallelism and the connection budget. Each stage worker is a process that holds one database connection, so the live connection count is approximately Σ PARALLELISM over busy stages, plus the dispatcher pool and the pepsi-ingress/pepsi-httpd pools. This sum must stay under PostgreSQL’s max_connections. The default PARALLELISM is 4 and the default per-server DB_POOL_SIZE is 1 precisely so a default install fits a default server. Raising PARALLELISM to push throughput, or DB_POOL_SIZE to admit more concurrent sessions, means re-checking that budget. If it is exceeded, workers that cannot get a connection report the database-overload status and the dispatcher requeues the message and throttles the busiest stage rather than losing mail — see pepsi-dispatch (Database-overload backpressure). The remedy for sustained pressure is to raise max_connections or lower PARALLELISM, not to rely on the throttle.

• Fewer hops beats more workers. A message pays one statistics upsert — the currently binding contention — per stage it advances through, so the goodput ceiling still moves with path length. Adding PARALLELISM puts more workers behind the same row lock; it does not raise the ceiling. Shortening the path does.

• Leave stage fusion on. [pepsi] ALLOW_FUSION (on by default) runs a fusible body-free successor inside its predecessor’s worker on the already-loaded row, so a run of n such stages costs one load, one terminal write and one round of bookkeeping instead of n. It remains the single most valuable throughput mechanism in the system, and the only reason to turn it off is to measure stages separately, as the per-stage table above does. Marking a cheap metadata-only stage FUSION = yes is worth more than any pool knob.

• Pipeline depth before parallelism. For the same reason, reach for a stage’s QUEUE_LIMIT (messages pipelined to one worker) before its PARALLELISM. QUEUE_LIMIT lifts a stage’s in-flight capacity to QUEUE_LIMIT × PARALLELISM and removes a coordinator round-trip per message without adding worker processes or database connections, so it does not enter the budget above; and a body-free stage’s worker commits a whole pipelined batch in one arrayed UPDATE, which is one notification for the batch. It defaults to 4 — pipelining is on out of the box — so the knob is often about lowering it: set QUEUE_LIMIT = 1 for stages whose per-message work is long and uneven, chiefly the network relays, where a pipelined message would otherwise wait behind a slow sibling.

• Drop work you do not need. The cheapest stage is one that is not in the chain — cheapest twice over, since it costs neither its own work nor a hop’s share of the coordination. detect-language in particular is worth omitting, or guarding with a size branch, unless language filtering is actually used.

• Leave ``WORKER_SYNCHRONOUS_COMMIT`` off unless a stage in your pipeline has an external side effect that must not be repeated even across a power loss. Turning it on costs roughly a write-ahead-log flush per stage hop; on this host that is about 10 ms a hop, so it adds some 80 ms to the ~130–150 ms of stage work a message pays across the nine-stage path.

• Front-end caps are separate from pipeline throughput. A high-volume single sender can hit tripwire’s per-client connection limit (CONN_RATE_PER_SECOND / CONN_RATE_BURST / MAX_CONNECTIONS in [pepsi-ingress]) long before the pipeline saturates; that is an admission knob, tuned independently of the delivery-side numbers above. The goodput benchmark raises these caps deliberately so it can measure the pipeline rather than the front end.

21.7. Reproducing

The benchmarks are not part of make integrationtests (they are slow and deliberately load the MTA). Run them against a deployed pipeline with:

tests/08-perstage-bench.sh tests/test-accounts.ini
tests/09-latency-bench.sh  tests/test-accounts.ini
tests/10-goodput-bench.sh  tests/test-accounts.ini
# or all three in order:
make benchmarks ACCOUNTS=tests/test-accounts.ini

Each script lowers the dispatcher’s STATS_INTERVAL/POLL_INTERVAL for the duration of the run so the counters flush promptly, and restores the deployed configuration on exit (including the admission caps the goodput run raises). See the Benchmark Suite chapter for the full methodology and the available tunables (message sizes, sample counts, ramp parameters, admission overrides).