19. Test Suite

Pepsi ships an end-to-end test suite under tests/ that exercises a live deployment over SMTP, asserting the observable behaviour of the whole pipeline rather than any single function in isolation. It is deliberately a black-box suite: messages are sent with the ordinary mail command or by raw SMTP, and the assertions read the recipients’ mailboxes and the pepsi.ingress queue.

This complements — it does not replace — the in-tree unit tests (cargo test --workspace), which cover the pure logic (SRS, MIME downgrade, DSN construction, language scoring, DNS classification, …) without a network.

Two things are covered in between, by integration tests that need neither a network nor a database. The migration suites (pepsi-setup/tests/import_*.rs and tests/cli_import.rs) run each MTA importer against a fixture /etc tree and then run the real pepsi-setup binary end to end, up to a --wizard --import that must produce a configuration passing Pepsi’s own validation; see the extending chapter for their layout. The dispatcher lifecycle suite (pepsi-test-stages) needs a PostgreSQL and is run by make check.

19.1. The documentation builds are checks too

make docs runs all of them for the source language, and is the one documentation target in the general quality gate. It is an alias for make docs-en; make docs-de and make docs-fr build the same artifacts from the translated catalogs and make docs-all builds every language. Four separate builds sit under it, and passing one says little about the others:

make docs-html-en

Sphinx HTML. Catches broken cross-references, unknown roles and malformed tables.

make man

The section-1/5 man pages through docs/rst2man.py. Catches title over/underlines that do not match the title length — a docutils error the HTML build tolerates. Source language only: the roff pages are rendered straight from docs/*.rst and never go through Sphinx, so they have no translation catalog.

make info

The GNU info manual, through Sphinx’s texinfo builder and makeinfo. Source language only, for the same reason install-info registers one pepsi.info in one info dir index.

make docs-pdf-en

LaTeX. The one most easily forgotten, and the one that fails hardest. pdflatex has no glyph for an arbitrary Unicode character and does not degrade when it meets one: it aborts with “Unicode character … not set up for use with LaTeX” and produces no PDF at all. HTML and man render the same character without complaint, so a new arrow, box-drawing character or Greek letter in the prose passes both and breaks only this build.

Every non-ASCII character used in the manual must be declared in latex_elements["preamble"] in docs/manual/conf.py. The box-drawing set used by the ASCII-art pipeline diagrams is mapped to plain ASCII on purpose: those diagrams sit inside code-block:: text, so the replacement is expanded in a verbatim environment where math mode is unavailable.

The second failure mode is a table that does not fit on a page. Sphinx renders a list-table into LaTeX’s longtable — which splits across pages and repeats the header — only past a row-count threshold; below it the table becomes a tabular, which cannot break at all. The heuristic counts rows, but what overflows a page is height, and those diverge badly for a wide table whose cells wrap: the 113-row RFC summary table splits happily while the 17-row, seven-column comparison table in Introduction did not, and ran 397 pt off the bottom of the page.

The fix is to say so explicitly rather than rely on the heuristic:

.. list-table::
   :header-rows: 1
   :widths: 20 14 14 14 14 14 14
   :class: longtable

Unlike a missing character this does not fail the build — it is an Overfull \vbox warning and a PDF with content off the page — so grep the log for Overfull \vbox and treat anything beyond a point or two as a real defect. Sub-point overshoots are ordinary typographic rounding.

When reading its output, ignore the Hyper reference … undefined warnings from the first passes — latexmk resolves forward references on later ones. The final pass, in docs/manual/_build/pdf/en/latex/pepsi.log, is the one that must be clean.

The translations are checked the same way, and the check is not redundant: a character a translator introduces — a guillemet, a German quotation mark, a non-breaking space — fails the same pdflatex build in the same way, and no English-only run would ever meet it. make docs-all is therefore a release gate rather than a per-commit one; each language’s PDF lands in docs/manual/_build/pdf/<lang>/latex/pepsi.pdf, and a German or French build additionally needs that language’s babel support installed (Debian: texlive-lang-german, texlive-lang-french).

Keeping the translations current is make update-po, which re-extracts the message templates and merges them into docs/manual/locale/<lang>/LC_MESSAGES so that changed paragraphs come back marked fuzzy and new ones untranslated. Only the .po files are under version control; Sphinx compiles them at build time.

19.2. Topology

The suite drives three real mail accounts across three hosts, described in a small INI file (tests/test-accounts.ini):

Role	Account (ssh target = e-mail)	Function
ADMIN	root@<mx-host>	Runs Pepsi (ingress, dispatch, httpd), PostgreSQL and systemd; also the smarthost the served domain relays through.
ALICE / BOB	alice@<served> / bob@<served>	Local accounts on the served domain. Their MTA relays through ADMIN and is trusted by it (MYNETWORKS), so their mail is local origin (the outbound path).
CAROL	test@<other>	An account on an independent host. Her mail reaches ADMIN’s public MX on port 25 as a foreign MTA — the inbound path.
DAVE	dave@<mx-host> (optional)	A local unix user on the ADMIN host that receives mail by direct local delivery into its Maildir (the local-delivery path; no onward relay). Unlike the others it is never ssh’d into directly — its mailbox is read as root over the ADMIN connection. Only 07-maildir-test.sh needs it; 01-deploy.sh provisions the dave/dave-broken users and the pepsi-maildir group when it is set.

Every host is reached over passwordless ssh; the harness installs nothing. ALICE/BOB use an mbox (/var/mail/<user>); CAROL and DAVE use a Maildir. The mailbox helpers are format-agnostic and locate a message by a run-unique token across both layouts.

The same file carries one non-mail account, in a [demobank] section:

[demobank]
URL  = https://bank.demo.taler.net
USER = pepsi-ci
PASS = pepsi-pass

This is a bank account at the public GNU Taler demo bank, whose currency (KUDOS) is play money. 12-wallet-topup-test.sh needs it because a manual wallet top-up only tells the user where to wire money — nothing credits the wallet until that bank transfer actually happens. The harness therefore plays the user’s bank: it logs in and makes the transfer the e-mailed instructions ask for, which is the only way the test can assert that the withdrawal completes rather than merely that instructions were sent. Register an account at https://bank.demo.taler.net/ (a new one starts with KUDOS:100 and may go KUDOS:500 into debt — roughly 120 runs of that case); DEMOBANK_USER, DEMOBANK_PASS and DEMOBANK_URL in the environment override the file. No other script uses it, and with the section absent only that one credit assertion skips.

19.3. The scripts

tests/lib.sh

Shared library sourced by every numbered script. It parses the accounts INI, provides the ssh/dependency/database/service probes used by the common pre-flight, the result counters and coloured ok/no/skip output, the ADMIN-host SQL and reset helpers, the mailbox send/receive helpers, the Taler wallet helpers, the RFC 3464 DSN assertion, and — for the extended scripts — send_smtp (a raw-SMTP sender that can set the ESMTP and DSN parameters the mail command cannot) and hop_caps (an EHLO capability probe). It is not executable on its own.

01-deploy.sh

Provisions a complete Pepsi deployment on the ADMIN host from a clean checkout: service users, PostgreSQL role/database, the configuration file (including the full stage pipeline), pepsi-setup (TLS, schema, DKIM keys, DNS records), the demo payment merchant, and the systemd units. It only verifies required tools and bails with a precise list if any are missing — it never installs OS dependencies.

02-pipeline-test.sh

The core happy-path suite (T1–T6): the pay-to-send anti-spam stage (unpaid → bounce, paid → delivered, paying centrally from the runner’s Taler wallet), the language filter (blacklisted French → bounce), the outbound signing/relay path (DKIM + ARC verified at the recipient), whitelist learning with SRS envelope rewriting, and a second local user.

03-dsn-test.sh

RFC 3461 DSN extensions and bounce variants, using send_smtp: NOTIFY=NEVER suppresses the DSN (D1); ENVID/ORCPT are echoed back (D2); RET=HDRS returns headers only (D3); NOTIFY=SUCCESS with ORIGINATE_SUCCESS_DSN yields a positive DSN (D4); an NXDOMAIN recipient routes to the default bounce (D5); an unreachable relay produces a DELAY DSN (D6); and a null-sender bounce to an SRS0=… address is reverse-decoded by ingress back to the original sender (D7).

04-settings-test.sh

The per-address override layer: a pepsi-settings row changes one recipient’s outcome but not another’s (S1); an account edits its own overrides by e-mail and is acknowledged (S2); an edit to a non-EDITABLE_STAGES section is rejected and stores nothing (S3); and one DATA with divergent per-recipient overrides is split into separate ingress rows (S4).

05-mime-test.sh

8BITMIME / SMTPUTF8 handling: an 8-bit body and a UTF-8 Subject survive the relay intact when the delivery hop advertises the extension, are downgraded (quoted-printable/base64, RFC 2047) when it does not, and a non-ASCII recipient address that the hop cannot represent is permanently failed. The script probes the hop’s EHLO first and takes the matching branch.

06-cli-test.sh

The operator CLIs, run directly over ssh: pepsi-config (get/dump/pathsub), pepsi-whitelist (add/list/remove and the dkim/signature flags), pepsi-settings (set/get/list/unset/remove), pepsi-tlsrpt (report --dry-run / prune), and the pepsi-httpd /metrics endpoint.

07-maildir-test.sh

Direct (local Maildir) delivery via the pepsi-stage-relay-to-maildir stage, which drives the setuid-root pepsi-helper-maildir-writer. A message to the local DAVE user lands in its Maildir with the prepended Return-Path/Delivered-To/Received headers (L1); a single message to one local and one remote recipient is split, delivering dave locally while alice is relayed onward (L2); and a message to a deliberately undeliverable account (dave-broken, no usable home) gives up past MAX_LIFETIME and returns a failure DSN to the sender (L3). Because [stage-local] sits after the inbound filters (… → anti-spam → local → srs), the sender is whitelisted first to clear the pay-to-send gate. The whole script skips cleanly when DAVE is unset, the deployment predates [stage-local], or the target users are missing.

12-wallet-topup-test.sh

The by-e-mail wallet self-service half of pepsi-stage-auto-pay (the [stage-wallet] section): a local account tops up its own per-user GNU Taler wallet by sending pepsi-wallet@<served-domain> a Subject: command. All three methods are exercised end-to-end against the demo exchange (KUDOS), with the funded local taler-wallet-cli as the genuine peer: withdraw (manual bank-wire instructions, the wire then actually made from the [demobank] account), pull (a taler://pay-pull/ URI the harness settles) and push (the harness pushes; the user accepts the taler://pay-push/ URI). Each case asserts the control message is consumed and the wallet balance grows.

13-auto-pay-test.sh

The demand-paying half: pepsi-stage-auto-pay settles a returning pay-to-send demand from the sender’s own wallet. The demand is produced genuinely — a local user sends to a paywalled local recipient (DAVE), the outbound is internet-relayed (stamping a real Pepsi-Origin), loops back into this host’s inbound MX where pepsi-stage-anti-spam holds it and e-mails the sender a real demand. That demand is replayed through the auto-pay worker. The two cases use separate senders (hence separate per-user wallets/orders): BOB’s wallet is left empty, so auto-pay forwards the demand and charges nothing (A1); ALICE’s wallet is pre-funded ahead of time (so the slow withdrawal is out of the paywall’s short 30s payment window), so auto-pay settles the demand within it — the order flips to paid, the wallet is debited, and the held message is released and delivered to DAVE (A2).

14-multi-recipient-test.sh

Recipient fan-out, the failure mode where a message addressed to several people reaches only some of them. Both cases assert positive delivery to every intended recipient. T1 sends one envelope to two remote recipients and requires both to arrive — a relay that takes rcpt_to.first() and then terminates the row passes nothing else on. T2 covers the ~/.forward fan-out: one envelope addressed to a local user who forwards (DAVE) and a second plain recipient, where the forwarded copy and the passthrough must each arrive exactly once — a stage that reduces the database row server-side without mutating the in-memory message leaves a fused successor working from the stale recipient list. T2 needs the optional DAVE account and a deployed ~/.forward stage.

16-whitelist-import-test.sh

The setuid pepsi-whitelist and its mailbox importer, against the installed deployment — everything about that program that only exists once it is installed. It checks the installation itself (mode 4755 owned pepsi-whitelist, the scan helper carrying no privilege bits, the table-scoped database role, the hoster list); the namespace rule, by creating two throwaway accounts and confirming one may use <login>/… and nothing else, that list does not disclose the operator’s shared whitelist, and that -c is refused; the import itself, which must yield exactly the recipients of the mail that user sent and ignore received mail; the wildcard decision, where a busy domain collapses to one *@domain row while gmail.com with just as many correspondents does not; the privilege drop, by substituting a helper that reports its own credentials and asserting the caller’s uid appears in all three id slots; --user seeding another account’s own namespace; and finally that pepsi-stage-check-whitelist consults the per-user name its {localpart} placeholder expands to. Individual checks skip when the deployment predates the feature.

Beside the numbered scripts, tests/whitelist-suid.sh covers the same setuid contract without SSH or a deployment: run as root on any host with a local PostgreSQL, it creates a throwaway database, installs the binary setuid into a suid-honouring directory and runs the same namespace/privilege-drop assertions. make check runs it (and the other *-suid.sh scripts) best-effort — they SKIP rather than fail when they cannot get root, a database or a suid-capable filesystem.

19.4. Running

# One-time, on the MX host (from a clean checkout):
ssh root@<mx-host> 'bash -s' < tests/01-deploy.sh

# Then, repeatedly, from the test runner:
tests/02-pipeline-test.sh tests/test-accounts.ini
tests/03-dsn-test.sh      tests/test-accounts.ini
tests/04-settings-test.sh tests/test-accounts.ini
tests/05-mime-test.sh     tests/test-accounts.ini
tests/06-cli-test.sh      tests/test-accounts.ini
tests/07-maildir-test.sh  tests/test-accounts.ini   # needs the optional DAVE account
tests/12-wallet-topup-test.sh tests/test-accounts.ini   # needs taler-wallet-cli on the MX host
tests/13-auto-pay-test.sh     tests/test-accounts.ini   # needs DAVE + taler-wallet-cli
tests/16-whitelist-import-test.sh tests/test-accounts.ini   # needs root on the MX host

Each script is self-managing: it runs a pre-flight (ssh reachability with the real error on failure, required remote tools, a database probe, and the three services’ state with the journal tail of any that is down), ensures a short PAYMENT_DEADLINE and brisk POLL_INTERVAL for fast turnaround, and resets the whitelist/queue/settings it depends on. A missing assumption fails the pre-flight with an actionable message; a message that does not arrive triggers a diagnostics dump (the live ingress queue plus the dispatcher journal). Tunables such as PAYMENT_DEADLINE_SECS, DELIVER_TIMEOUT and WITHDRAW_AMOUNT are environment overrides. Each script exits 0 only when no case failed (skips are allowed).

19.5. Skips and gated cases

A case skips (rather than fails) when a precondition it cannot satisfy is absent, always with a one-line reason. The common ones:

• No usable Taler wallet. The paid leg of 02 skips if the local taler-wallet-cli is missing or incompatible with the demo exchange. All payments are made centrally from the runner’s wallet (in production each sender would pay for their own message).

• Deployment predates a feature. 03/04 skip the cases that need pipeline features added later to 01-deploy.sh (ORIGINATE_SUCCESS_DSN, a [stage-internet] BOUNCE_STAGE, the [stage-delaytest] blackhole relay, the [stage-edit-settings] stage) when run against an older deployment — re-run 01-deploy.sh to enable them. 06 skips the pepsi-tlsrpt case unless a [pepsi-tlsrpt] section is configured. 07 skips entirely when the optional DAVE account is unset, the deployment predates [stage-local], or the dave/dave-broken target users are missing on the ADMIN host. 14’s T2 skips when DAVE is unset or the deployment has no ~/.forward stage.

• Hop capability. 05 takes its pass-through, downgrade or unrepresentable-address branch according to whether the delivery hop advertises 8BITMIME / SMTPUTF8; the cases that require the hop to lack an extension skip when it advertises it.

• Wallet / auto-pay not deployed. 12/13 skip entirely when the [stage-wallet]/[stage-pay] sections, taler-wallet-cli or the pepsi-wallets account are absent (re-run 01-deploy.sh). 12’s withdraw credit leg skips without a [demobank] account (see above); its pull/push cases skip without a usable local wallet. 13 skips when DAVE is unset, the genuine demand is not produced (paywalled hop or merchant down), or — for A2 — the per-user wallet cannot be funded.

19.6. Constraints

Testing must never send real mail to third-party MX hosts: the suite uses only the operators’ own domains and reserved/.invalid names (e.g. the DELAY test relays to an unrouteable 192.0.2.0/24 TEST-NET-1 address via a recipient in blackhole.invalid). The development uid cannot bind the privileged ports 25/53, so the privileged listeners are exercised only on the deployed host.