20. Benchmark Suite

The tests/NN-*-bench.sh scripts measure how fast the live Pepsi pipeline processes mail. They reuse the same hosts, accounts and helpers as the correctness suite (tests/test-accounts.ini, tests/lib.sh) plus a shared tests/bench-lib.sh, and are driven the same way:

tests/08-perstage-bench.sh tests/test-accounts.ini
tests/09-latency-bench.sh  tests/test-accounts.ini
tests/10-goodput-bench.sh  tests/test-accounts.ini
# or all three in order:
make benchmarks ACCOUNTS=tests/test-accounts.ini

They are not part of make integrationtests (they are slow and deliberately load the MTA). Each one runs preflight_common first, so it SKIPs cleanly if the live hosts/DB/services are unreachable. Run tests/01-deploy.sh first so the pipeline — including the DAVE local-Maildir delivery target — exists.

This chapter describes the method; for a worked set of measured results (per-stage cost, idle latency and saturation goodput, with the test host’s hardware), see the Performance chapter.

20.1. How the numbers are obtained

• Per-stage cost comes from the dispatcher’s own timing: pepsi-dispatch records every stage execution into pepsi.stage_stats(stage, messages, duration_us). Snapshotting that table before/after a message gives exact microseconds-per-message per stage — for a stage reached by an SMTP-sent message and for one injected straight at it alike, since both run as ordinary dispatched workers. The benchmarks lower [pepsi-dispatch] STATS_INTERVAL (and POLL_INTERVAL) so the counters flush promptly, and also set [pepsi] LOG to a quiet level (warn by default, overridable via BENCH_LOG_LEVEL) so per-message logging does not inflate the timings — that knob is read by every component, so it covers the dispatcher and the stage workers it spawns (which take no log flag of their own). Both pepsi-dispatch and pepsi-ingress are restarted so the new level takes effect, and the benchmarks restore the deployed config on exit (a one-shot pepsi.conf.bench.bak backup on the ADMIN host, replayed by a trap … EXIT).

• Goodput is the delta of the global pepsi.dispatch_stats.messages_processed counter over each 5-second window.

• Delivery is cross-checked by counting files in the target Maildir/new.

To keep the inbound pay-to-send (anti-spam) stage from gating the load, the benchmarks pre-whitelist the sender (bench_whitelist_sender), so check-whitelist sets state.spam=false and anti-spam short-circuits.

20.2. The three benchmarks

1. ``08-perstage-bench.sh`` — per-stage cost vs message size. Prints one stage × size matrix of µs/msg covering every stage Pepsi has, filled in by two halves that answer two different questions: the scenarios time the stages of the deployed pipeline in situ, and the isolated stages time every stage program the deployed pipeline routes no message through.

   The scenarios. A single message only traverses one path, so it can only time the stages on that path; covering the whole bidirectional, branching pipeline therefore takes several messages — one per path. The benchmark drives a set of SCENARIOS, each a message whose journey reveals a different group of stages, and accumulates their per-stage costs into the matrix:

   scenario	message	stages it newly reveals
   inbound-local	CAROL → DAVE (foreign inbound → local Maildir)	init (arc), route (if), decrypt, detect-language, block-language, check-whitelist, anti-spam (short-circuit), forward (dot-forward), local (maildir)
   inbound-relay	CAROL → ALICE (inbound → onward relay)	srs, dkim-sign-relay, smarthost (the non-local recipient is peeled by local onto that tail)
   outbound	ALICE → CAROL (local-origin submission → direct relay)	edit-settings, auto-whitelist, delay-route (if), encrypt, dkim-sign, internet (relay-to-internet)
   bounce	CAROL → dave-broken (local delivery fails)	bounce (the maildir helper fails → a bounce SideClone routes to the bounce stage → dkim-sign)
   bounce-language	CAROL → DAVE, French body (blacklisted language)	bounce-language (block-language bounces an fr message)
   payment (opt-in)	CAROL (not whitelisted) → DAVE (pay-to-send gate)	bounce-payment (shortens PAYMENT_DEADLINE for the run)
   delay (opt-in)	ALICE → blackhole.invalid, ENVID=PEPSIDELAY (DELAY DSN)	delaytest (relay-to-smarthost to an unrouteable MTA)

   The default SCENARIOS are the five fast ones; append payment and/or delay to also time the last two deployed stages (they are slower — merchant- and retry-bound). The payment/delay scenarios and the two opt-in stages need a sender that is not local-origin and the broken local target, all of which 01-deploy.sh provisions. Setting SCENARIOS="" runs the isolated half alone.

   The isolated stages. pepsi-ingress always admits a message at init (pepsi_common::stage::INITIAL_STAGE), so a scenario can only ever reach the stages the deployed pipeline routes it through. Every stage program that is not wired into that pipeline is therefore unreachable by any message, however constructed — which is why a third of the tree used to go unmeasured.

   So the run has a second half. It appends benchmark-only [stage-bench-*] sections to the deployed configuration — each with that stage’s own sensible defaults, and every edge pointing at one shared bench-discard sink (pepsi-stage-discard), so nothing an isolated stage advances can escape into the live pipeline or onto the network — and then injects a sized message straight at each of them with the pepsi.ingress_inject SQL function, which is exactly what a stage that originates mail does (StageContext::enqueue_new). ingress_inject notifies the ingress channel, so the dispatcher picks the row up at once, the stage runs as an ordinary worker, and its cost lands in pepsi.stage_stats like any other. Injecting rather than sending also takes pepsi-ingress, SMTP and the admission limits out of the measurement, so an isolated figure is the stage’s own cost and nothing else.

   isolated stage	program	what is configured, and what it therefore measures
   bench-discard	pepsi-stage-discard	DISPOSITION = success, BOUNCE = no — the shared sink, so its own row is also the pipeline’s bare per-message floor
   bench-aliases	pepsi-stage-aliases	a three-entry ALIASES map (a two-target expansion, a transitive hop, an @domain catch-all) and a recipient that expands
   bench-route	pepsi-stage-route	one explicit ROUTES carve-out plus MANAGED_STAGE; the recipient is at a managed domain, so the managed branch is taken
   bench-vacation	pepsi-stage-vacation	VACATION_RANGES covering today (in the server’s timezone) and SUPPRESS_DAYS = 0, so every message really composes and injects a notice rather than taking the “nobody is away” exit
   bench-autocrypt-learn	pepsi-stage-autocrypt-learn	defaults, on a message with no Autocrypt: header — the pass over a message with nothing to learn, which is what all but a sliver of real inbound mail is
   bench-vks-confirm	pepsi-stage-vks-confirm	a VKS_HOST the bench sender is not, so the message passes through and no link is followed (no network I/O)
   bench-auto-pay	pepsi-stage-auto-pay	the wallet configured as the deployed [stage-pay], on a message that is not a payment demand — again the common inbound case. The paying half needs a live merchant and wallet; 12/13 drive that.
   bench-secure-link	pepsi-stage-secure-link	defaults; terminal. The global [pepsi-secure-link] NOTIFY_STAGE is pointed at the sink for the run, so timing the stage puts no mail on the wire.
   bench-lmtp	pepsi-stage-relay-to-lmtp	the local Dovecot socket and the throwaway mailbox 11-lmtp-test.sh provisions; skipped unless both are present
   bench-milter	pepsi-stage-milter	skipped unless a milter is named in BENCH_MILTER_ADDR (e.g. inet:8890@127.0.0.1)

   An isolated stage whose program is not installed, or whose external dependency is missing, is dropped before the sections are written; and if the dispatcher does not come back up with them, the append is rolled back and the isolated half is abandoned rather than leaving the host without its coordinator. ISO_STAGES="" skips the half entirely.

   Deliberately not isolated: pepsi-stage-encrypt, -decrypt, -dot-forward, -srs, -dkim-sign and -if — the deployed pipeline routes through all of them, so the scenarios already time them in situ, which is the more honest figure.

   Reading the matrix. Body-reading stages (arc verify, decrypt, detect-language, encrypt, dkim-sign, secure-link, relay/maildir) climb with size; metadata-only stages (if-branches, srs, whitelist, aliases, route) stay flat. The relay stages (smarthost, internet, delaytest, bench-lmtp) are network- or MDA-inclusive — they hold the outbound transaction open, so their figure is the next hop’s round-trip, not local CPU; the benchmark prints an explicit note when such a stage ran more times than messages were injected, i.e. when the next hop deferred and the figure averages retries in.

2. ``09-latency-bench.sh`` — idle end-to-end latency. Runs the whole inject→observe loop on the ADMIN host (one clock, no ssh-per-poll) for N short messages and reports min/mean/median/p95/max, plus the sum of mean per-stage processing time as a floor.

3. ``10-goodput-bench.sh`` — saturation goodput, three load paths. Ramps the offered load every 5 s until the processed rate plateaus while the queue grows, for: A pure pipeline (local inject → local Maildir), B network + front-end (remote sender → local Maildir), C outbound relay (authorized enoki origin → relay-to-internet → external MX). It prints a per-step table and a knee comparison (A vs B isolates the network/SMTP front-end; B vs C isolates the outbound relay).

20.3. Rate-limit reporting

The parallel load generator records the SMTP reply of every rejection, so tripwire’s own front-end throttle ([pepsi-ingress] CONN_RATE_PER_SECOND / CONN_RATE_BURST) shows up as 421s and is reported with the knob to raise.

For the outbound path (scenario C), which touches MTAs Pepsi does not control, the script reads Pepsi’s view of the next hop — state.bounce SMTP codes/text, pepsi.tls_session, the relay journal — and, if it sees 4xx / “rate”/”too many”/”try again”/”throttle” deferrals, reports external rate-limiting with concrete fixes:

• On the remote MTA (Postfix): raise smtpd_client_connection_rate_limit, smtpd_client_message_rate_limit, the *_connection_count_limit, widen anvil_rate_time_unit, or exempt tripwire’s IP via smtpd_client_event_limit_exceptions / a dedicated transport.

• On the Pepsi side (to stay under a limit you cannot change): lower [stage-internet] PARALLELISM and pace retries via RETRY_INITIAL / RETRY_MAX_INTERVAL / RETRY_FACTOR.

20.4. Useful knobs

script	knobs
all	BENCH_STATS_INTERVAL, BENCH_POLL_INTERVAL, BENCH_LOG_LEVEL, MSGS_PER_CONN
08	SIZES, REPEAT, SCENARIOS, PERMSG_TIMEOUT, BOUNCE_TIMEOUT_S, PAY_DEADLINE_S, PAY_TIMEOUT_S; and for the isolated half ISO_STAGES, ISO_SIZES, ISO_REPEAT, ISO_TIMEOUT, ISO_LOCAL_RCPT, ISO_SENDER, ISO_LMTP_USER, BENCH_MILTER_ADDR
09	N, WARMUP, MSG_BYTES, LATENCY_MAX_MS
10	RATE_UNIT, PAR_UNIT, PAR_MAX, MAX_STEPS, STEP_SECS, MSG_BYTES, STALL_STEPS, SAT_QUEUE, RUN_A/RUN_B/RUN_C