29. Test Suite¶
Pepsi ships an end-to-end test suite under tests/ that exercises a live
deployment over SMTP, asserting the observable behaviour of the whole pipeline
rather than any single function in isolation. It is deliberately a black-box
suite: messages are sent with the ordinary mail command or by raw SMTP, and
the assertions read the recipients’ mailboxes and the pepsi.workqueue table.
This suite complements the in-tree unit tests (cargo test --workspace) and
does not replace them. The unit tests cover the pure logic (SRS, MIME
downgrade, DSN construction, language scoring, DNS classification, …) without
a network.
Two things are covered in between, by integration tests that need neither a
network nor a database. The migration suites
(pepsi-setup/tests/import_*.rs and tests/cli_import.rs) run each MTA
importer against a fixture /etc tree and then run the real pepsi-setup
binary end to end, up to a --wizard --import that must produce a
configuration passing Pepsi’s own validation; see
the extending chapter for their layout. The
dispatcher lifecycle suite (pepsi-test-stages) needs a PostgreSQL and is
run by make check.
Each crate links its integration tests into one test binary
(tests/main.rs declares every other file under tests/ as a module):
stages for pepsi-test-stages, crypto, setup, common,
httpd and keydisc for the crates of those names, and workspace for
the files in the top-level tests/. A binary per file cost a link of several
seconds and up to 500 MB of disk each. To run one file’s tests, filter on its
module name, e.g. cargo test -p pepsi-test-stages keystore:: or cargo test
-p pepsi-crypto --test crypto sweep::. The two pepsi-common telemetry
tests (telemetry_off, telemetry_short_lived) stay binaries of their own,
because each installs the process-global telemetry sink.
29.1. make check: the no-remote-accounts gate¶
make check is the entry point for everything that needs no live mail
accounts. In order, it runs:
make check-telemetry-socket,make check-systemd-units,make check-versionandmake check-extensions—tests/check-telemetry-socket.sh,tests/check-systemd-units.sh,tests/check-version.shandtests/check-extensions.sh. They are consistency checks over the systemd units, the Debian packaging, the version that every manifest anddebian/changelogcarry, and the two mail-client extensions against the header fields that the gateway emits; they need nothing installed and never skip.cargo test --workspace --exclude pepsi-test-stages --locked— every database-free test in the tree. (Plaincargo test --workspacewould includepepsi-test-stagesand so need a database; prefermake check, which gates that suite instead of failing on it.)tests/check-lifecycle.sh— the dispatcher-lifecycle suite, best effort.tests/maildir-writer-suid.sh,tests/dot-forward-suid.shandtests/whitelist-suid.sh— the setuid/setgid helper contracts, best effort.tests/interop-auth.sh— a cross-implementation DKIM/ARC check, best effort. Pepsi’s own round-trip tests sign and verify with the samemail-authlibrary, so they cannot catch a transform that is consistently wrong in it; this script grades Pepsi-signed messages (all fourc=combinations, plus a tampered copy that must be rejected, plus an ARC seal) with two independent implementations, with no DNS involved. dkimpy gets the public keys through itsdnsfunc(tests/interop-verify.py) and grades DKIM and the ARC chain; OpenDKIM reads them from aTestPublicKeysfile in theopendkimfilter’s-ttest mode and grades DKIM (it has no ARC verifier;opendkim-testmsgcannot be used, because it verifies only against live DNS). EveryDKIM-Signatureis graded, the RSA and the Ed25519 one — dkimpy needs PyNaCl for the latter. Each verifier SKIPs its own checks when absent, and the script SKIPs as a whole with neither. On Debian:apt install python3-dkim python3-nacl opendkim(the filter lives in/usr/sbin; setOPENDKIMif it is elsewhere, and disable theopendkimservice the package enables — the test never talks to it).cargo deny check— advisories, bans, licences and sources. This one is not best-effort: it is skipped only whencargo-denyis genuinely not installed, and a real finding fails the target. Readdeny.tomlbefore trusting the gate: itsignorelist is not empty.cargo vet --locked— the dependency audit record,supply-chain/: every third-party crate version inCargo.lockmust be covered by an audit (the project’s, or one of the imported public sets pinned inimports.lock) or by an exemption, so a crate added or updated without a recorded decision fails here.--lockednever fetches, so the gate runs offline. Skipped only whencargo-vetis not installed (cargo install --locked cargo-vet); a real finding fails the target. What to do when it fails is in The supply chain.make docs-linkcheck— every internal cross-reference of the manual must resolve, in every language (a Sphinxdummybuild with warnings as errors). Skipped only whensphinx-buildis not installed.
The Thunderbird S/MIME interoperability gate is part of the workspace tests: it
runs wherever Thunderbird is installed, starts a real (headless,
Marionette-driven) Thunderbird and costs about fifteen seconds. make
interop-thunderbird runs it alone. Its finding, that Thunderbird cannot read
AuthEnvelopedData, is why Pepsi keeps a per-peer S/MIME capability record
and sends unauthenticated AES-CBC only to a correspondent recorded as unable to
read AEAD (see The per-peer S/MIME capability record and
EFAIL: CBC/CFB malleability gadgets exfiltrate decrypted mail).
29.1.1. When a test may skip¶
A test skips for three reasons only: its permissions are truly insufficient, a dependency is really unavailable, or it is the wrong platform. Anything else is a failure. A tool that is installed and then refuses to work is a failure, and so is a fixture that cannot be built: gpg that will not import a key has failed the test, and only gpg that is not installed skips it.
The Rust tests record a skip with pepsi_common::skip_test! or
pepsi_common::require_tool! (pepsi-common/src/test_skip.rs). The line
SKIP <test>: <reason> goes straight to standard error, past the output
capture of the test harness, so a plain cargo test shows it beside the
SKIP lines of the shell suites and of make check.
With PEPSI_TEST_NO_SKIP=1 in the environment every skip is a failure: in the
Rust tests, in tests/check-lifecycle.sh, tests/interop-auth.sh and the
three *-suid.sh suites, and in the gates of make check. The one exception
is the suites’ “not running as root” line, because make check runs them
unprivileged by design. Set the variable when every dependency is installed and
you want to know that nothing was left out:
$ PEPSI_TEST_NO_SKIP=1 PEPSI_X509_LIMBO=$PWD/limbo.json cargo test --workspace
These are the skips that remain possible, and what each one lacks:
Reason |
What is missing |
Tests |
|---|---|---|
Dependency |
|
the interoperability tests, and every test that makes a certificate |
Dependency |
|
|
Dependency |
no public DNS resolver answers |
|
Dependency |
the host has no IPv6 loopback address |
the last case of |
Dependency |
no PostgreSQL, or none on TCP |
|
Dependency |
a verifier, or a tool such as |
|
Permission |
the suite does not run as root |
the |
Permission |
the host grants no user and network namespace |
the namespace tests of |
Permission |
the test role may not |
the grant tests (see below), and all of
|
Permission |
the test runs as root, or under a uid outside the regular range of
|
|
Platform |
there is no Linux |
|
No test is #[ignore]d. The tests with live in their name assert what
is true on any host and print the rest, so with --nocapture they also serve
to look at a deployment. batch_bench is an ordinary test at a small size;
BENCH_N and BENCH_REPS give it the size of a benchmark.
29.1.2. Continuous integration¶
Several of the steps above skip when a tool or a database is missing. That is
the right default on a developer’s machine and exactly wrong for an automated
run, where a skipped gate reported green looks like a passed one.
contrib/ci/ holds jobs in the layout that GNU Taler’s CI worker runs (the
same one that vendor/taler-rust/contrib/ci/ uses): contrib/ci/ci.sh
<job> builds the image from contrib/ci/Containerfile with podman and runs
contrib/ci/jobs/<job>/job.sh in it with the source tree mounted;
contrib/ci/run-all-jobs.sh runs every job in order. The jobs are:
1-gatesThe compile-time gates:
RUSTFLAGS="-D warnings" cargo build,cargo clippy … -D warningsandcargo fmt --check.2-checkmake checkwith every best-effort gate armed. The image carries dkimpy, PyNaCl, OpenDKIM, gpg, OpenSSL, BIND’snamed, Thunderbird, the x509-limbo corpus, cargo-deny, cargo-vet and Sphinx. The job starts PostgreSQL, setsPEPSI_TEST_NO_SKIP=1(see When a test may skip), and fails if any line of the transcript reports a SKIP.make checkruns as an unprivileged user on a private copy of the tree, as a developer runs it; otherwise the lifecycle suite’spepsi-setup runwould switch to packaged system accounts that the container does not have. The three setuid/setgid suites, which need root, are then run again as root; their “not running as root” skip in the first pass is the one skip accepted.CI_ALLOWED_SKIPS(an extended regex) names any further skip accepted on purpose.3-docsmake docs-all: every manual language including its PDF, which is the build that fails on an undeclared Unicode character.4-upgradeThe schema-upgrade test,
tests/check-upgrade.sh, against the previous release tag (a no-op until one exists): it builds that release’spepsi-setupandpepsi-status, installs its schema and loads its frozen fixture (tests/upgrade/), upgrades with this tree taking a backup, and checks that every row survives, the backup restores, this tree’s dispatcher drains the old queue, and the old release refuses the upgraded schema with exit status 78. The script runs outside CI too, given a previous release’s binaries and SQL directory.5-reproducibleThe reproducible-build check: two clones of the commit under test, at different paths and with different crate-cache directories, each
make buildandmake dist, and the job fails on any byte that differs between the two sets of release binaries or the two tarballs (Reproducible builds). It testsHEAD, not the working tree, since a release is a commit andmake distwants a clean checkout. The script runs outside podman too:CI_REPOnames the checkout to clone (default/workdir) andCI_WORKthe scratch directory.6-crypto-opensslThe OpenPGP crypto backend a packager may substitute: every program built with
--features crypto-opensslinstead of Nettle, and thepepsi-cryptosuite run on it. OpenSSL 3.5 is the backend with the RFC 9980 post-quantum composite algorithms, so the suite there also signs and encrypts a message with ML-DSA-65+Ed25519 and ML-KEM-768+X25519 keys through Pepsi’s own code (pepsi-crypto/tests/postquantum.rs); on Nettle the same file checks that the backend still offers none of them.
The Rust toolchain in the image is pinned rather than tracking stable, so a
new clippy lint cannot turn an unrelated commit red: rust-toolchain.toml
names the version, ci.sh passes it to the image build, and
check-version keeps the Containerfile’s own default equal to it; bump
it there, deliberately. The jobs build into target-ci/ so they do not
invalidate a developer’s target/. Run a job locally from the top of the
tree, with the vendor/taler-rust submodule initialised, e.g.
contrib/ci/ci.sh 2-check.
29.1.3. The lifecycle suite and its database¶
pepsi-test-stages spawns the real pepsi-dispatch binary against a real
PostgreSQL and asserts on pepsi.workqueue row states, so it needs a reachable
server. It has two database modes, chosen by PEPSI_TEST_DATABASE:
- unset — the default, and the development workflow
Each test
CREATEs andDROPs its own throwaway database, so the suite runs in parallel. This is what a barecargo test -p pepsi-test-stagesdoes; the connecting role must be able toCREATE DATABASE/DROP DATABASE.- set (e.g.
pepsicheck) Every test shares that one pre-existing database — never created, never dropped, its content preserved — resetting only
pepsi.workqueueon setup. The suite then must run with--test-threads=1.
tests/check-lifecycle.sh is the second mode automated: it probes for a
pepsicheck database (creating it if absent), resets the schema with
pepsi-setup run --reset and runs the suite serially against it, leaving the
database in place. It runs a plain run before the reset, so that a freshly
created database has the _v schema that the drop step deletes from. If
neither the probe nor the
creation works it skips: exit 0, which is not a pass and not a failure.
Four environment variables tune all of this:
PEPSI_TEST_DATABASESelects shared mode and names the database. Unset means throwaway mode.
PEPSI_TEST_PGHOSTThe socket directory for the local peer-authenticated connection. Default
/var/run/postgresql.PEPSI_TEST_PGUSERThe login/peer role to connect as. Default
$USER.PEPSI_TEST_DISPATCH_LOGSet to
1to see the spawned dispatcher’s own logs, which the suite silences by default.
(tests/check-lifecycle.sh additionally reads PEPSI_CHECK_DB to rename the
shared database it manages, default pepsicheck.)
A single test or module is cargo test -p <crate> <name::filter>, e.g.
cargo test -p pepsi-common srs::.
29.1.3.1. The privilege-boundary tests¶
Eight test files assert the database grants of The privilege split and
An attacker who has the database against a real server — service_role_grants,
secure_link_grants, admin_grants, whitelist_grants, setup_task
and the grant cases in keystore, keydisc_lifecycle and crl_cache.
Each runs the production provision_roles and then SET ROLEs into each
account to ask PostgreSQL what it may do. These grants are what keeps the
database from being one store that every process can read (see
Mail appliances store secrets world-readable or unprotected).
They skip when any audited role is a superuser, because a superuser
bypasses every grant and nothing about the boundary can be learnt. The audited
roles have fixed names and are global to the cluster, so a test cannot make
itself a non-superuser pepsi. In CI none of them is a superuser, so the tests
run there. On a development machine the skip is the common case: the account
running the tests is pepsi, which is also the pipeline’s own role, and it is
usually the cluster’s superuser too. To exercise them, run the suite with pepsi
temporarily not a superuser but able to switch into the other roles, from a
second superuser session, and restore it afterwards:
$ psql -U some_superuser -c "ALTER ROLE pepsi NOSUPERUSER CREATEDB CREATEROLE" \
-c 'GRANT "pepsi-ingress", "pepsi-httpd", "pepsi-telemetry", "pepsi-whitelist",
"pepsi-crypto", "pepsi-keydisc", "pepsi-config"
TO pepsi WITH INHERIT FALSE, SET TRUE'
$ cargo test -p pepsi-test-stages service_role_grants:: -- --nocapture
$ psql -U some_superuser \
-c 'REVOKE "pepsi-ingress", "pepsi-httpd", "pepsi-telemetry", "pepsi-whitelist",
"pepsi-crypto", "pepsi-keydisc", "pepsi-config" FROM pepsi' \
-c "ALTER ROLE pepsi SUPERUSER"
INHERIT FALSE matters: it lets pepsi become each role without
acquiring its privileges, so the checks on pepsi itself stay meaningful.
The administrative API’s end-to-end tests (admin_api) include one more
role-dependent case: the successful CONFIG_DB write connects with
options[role]=pepsi-config. The test creates the role when it may, and it
skips when the test role cannot SET ROLE "pepsi-config" after that. The
same grant above makes it run. None of the expiry tests there waits for a window to elapse (session
idle and absolute expiry, token expires_at and disabled, the
pepsi-httpd prune job): each one backdates the timestamp that the database
compares against now(), and the enforcing query cannot tell this apart from
time passing.
29.2. The documentation builds are checks too¶
make docs runs all of them for the source language, and is the one
documentation target in the general quality gate. It is an alias for
make docs-en; make docs-de and make docs-fr build the same artifacts
from the translated catalogs and make docs-all builds every language. Four
separate builds sit under it, and passing one says little about the others:
make docs-html-enSphinx HTML. Catches broken cross-references, unknown roles and malformed tables.
make manThe section-1/5 man pages through
docs/rst2man.py. Catches title over/underlines that do not match the title length — a docutils error the HTML build tolerates. Source language only: the roff pages are rendered straight fromdocs/*.rstand never go through Sphinx, so they have no translation catalog.make infoThe GNU info manual, through Sphinx’s texinfo builder and
makeinfo. Source language only, for the same reasoninstall-inforegisters onepepsi.infoin one infodirindex.make docs-pdf-en,make docs-pdf-de,make docs-pdf-frLaTeX, one PDF per language, each written to
docs/manual/_build/pdf/<lang>/latex/pepsi.pdf;make docs-pdf-allbuilds all three (and nothing else),make docs-pdfis the English one. The German and French builds need babel’s language support (texlive-lang-german,texlive-lang-frenchon Debian). The one most easily forgotten, and the one that fails hardest. pdflatex has no glyph for an arbitrary Unicode character and does not degrade when it meets one: it aborts with “Unicode character … not set up for use with LaTeX” and produces no PDF at all. HTML and man render the same character without complaint, so a new arrow, box-drawing character or Greek letter in the prose passes both and breaks only this build.Every non-ASCII character used in the manual must be declared in
latex_elements["preamble"]indocs/manual/conf.py. The box-drawing set used by the ASCII-art pipeline diagrams is mapped to plain ASCII on purpose: those diagrams sit insidecode-block:: text, so the replacement is expanded in a verbatim environment where math mode is unavailable.The second failure mode is a table that does not fit on a page. Only past a row-count threshold does Sphinx render a
list-tableinto LaTeX’slongtable, which splits across pages and repeats the header; below it the table becomes atabular, which cannot break at all. The heuristic counts rows, but what overflows a page is height, and those diverge badly for a wide table whose cells wrap: the 113-row RFC summary table splits happily while the 17-row, seven-column comparison table in Introduction does not, and runs 397 pt off the bottom of the page.The fix is to say so explicitly rather than rely on the heuristic:
.. list-table:: :header-rows: 1 :widths: 20 14 14 14 14 14 14 :class: longtable
Unlike a missing character this does not fail the build — it is an
Overfull \vboxwarning and a PDF with content off the page — so grep the log forOverfull \vboxand treat anything beyond a point or two as a real defect. Sub-point overshoots are ordinary typographic rounding.When reading its output, ignore the
Hyper reference … undefinedwarnings from the first passes — latexmk resolves forward references on later ones. The final pass, indocs/manual/_build/pdf/en/latex/pepsi.log, is the one that must be clean.
The translations are checked the same way, and the check is not redundant: the
same pdflatex build fails in the same way on a character that a translator
introduces (a guillemet, a German quotation mark, a non-breaking space), and no
English-only run would ever meet it. make docs-all is therefore a release
gate rather than a per-commit one; each language’s PDF lands in
docs/manual/_build/pdf/<lang>/latex/pepsi.pdf, and a German or French build
additionally needs that language’s babel support installed (Debian:
texlive-lang-german, texlive-lang-french).
Keeping the translations current is make update-po, which re-extracts the
message templates and merges them into docs/manual/locale/<lang>/LC_MESSAGES
so that changed paragraphs come back marked fuzzy and new ones untranslated.
Only the .po files are under version control; Sphinx compiles them at build
time.
29.3. Topology¶
The suite drives three real mail accounts across three hosts, described in a
small INI file, tests/test-accounts.ini. That file is not part of the
source tree: create it from the template with cp tests/test-accounts.ini.sample
tests/test-accounts.ini and replace its placeholder accounts with hosts you
control:
Role |
Account (ssh target = e-mail) |
Function |
|---|---|---|
|
|
Runs Pepsi (ingress, dispatch, httpd), PostgreSQL and systemd; also the smarthost the served domain relays through. |
|
|
Local accounts on the served domain. Their MTA relays through ADMIN and
is trusted by it ( |
|
|
An account on an independent host. Her mail reaches ADMIN’s public MX on port 25 as a foreign MTA — the inbound path. |
|
|
A local unix user on the ADMIN host that receives mail by direct
local delivery into its Maildir (the local-delivery path; no onward
relay). Unlike the others it is never ssh’d into directly — its mailbox is
read as root over the ADMIN connection. Only |
Every host is reached over passwordless ssh; the harness installs nothing.
ALICE/BOB use an mbox (/var/mail/<user>); CAROL and DAVE use a
Maildir. The mailbox helpers are format-agnostic and locate a message by a
run-unique token across both layouts.
Note
The other two hosts have admission policies of their own, and they bind long before Pepsi does. They are invisible in the correctness suite, which sends a handful of messages, and decisive in the benchmarks, which send hundreds of thousands. Two in particular:
CAROL’s MX is the receiving end of every outbound test. When it is itself a Pepsi deployment,[pepsi-ingress] CONN_RATE_PER_SECONDandCONN_RATE_BURSTthere default to 1 and 1 — one new connection a second from a given source range — so a relay opening a connection per message is answered421 4.7.0 Too many connections from your addressand defers. Nothing is lost; nothing is measured either. Raise them on that host before believing any outbound figure.Once they are raised, what remains is that host’s own speed. If the
CAROLhost is much smaller than the MTA under test, every outbound measurement is that host’s figure, not Pepsi’s; the Performance chapter says how to read such figures.ALICE/BOB’s MTA is the smarthost the relay tail ends at, and Postfix’smessage_size_limitdefaults to ten million bytes. That is why the per-stage sweep’s larger sizes cannot traverse the smarthost path at all, which the sweep reports as the next hop’s own552rather than guessing.
The same file carries one non-mail account, in a [demobank] section:
[demobank]
URL = https://bank.demo.taler.net
USER = pepsi-ci
PASS = pepsi-pass
This is a bank account at the public GNU Taler demo bank, whose currency
(KUDOS) is play money. 12-wallet-topup-test.sh needs it because a
manual wallet top-up only tells the user where to wire money — nothing credits
the wallet until that bank transfer actually happens. The harness therefore
plays the user’s bank: it logs in and makes the transfer that the e-mailed
instructions ask for, which is the only way the test can assert that the
withdrawal completes rather than merely that instructions were sent. Register an
account at https://bank.demo.taler.net/ (a new one starts with KUDOS:100 and
may go KUDOS:500 into debt — roughly 120 runs of that case);
DEMOBANK_USER, DEMOBANK_PASS and DEMOBANK_URL in the environment
override the file. No other script uses it, and with the section absent only
that one credit assertion skips, as a missing dependency.
No script names a host of its own: every host, service and per-host setting comes
from this file, and tests/test_layout.rs fails if a script names another host or
reads a setting the sample does not list. Three more sections complete it:
[taler]
MERCHANT_URL = https://backend.demo.taler.net/instances/sandbox
MERCHANT_TOKEN = secret-token:sandbox
EXCHANGE_URL = https://exchange.demo.taler.net/
[deployment]
BEHIND_FRONT = 127.0.0.1:9443
[lmtp-interop]
BINDIR = /usr/local/bin
SQLDIR = /usr/local/share/pepsi/sql
[taler] is required and names the merchant instance and the exchange, here the
public demo services. [deployment] and [lmtp-interop] are optional and
described with the scripts that read them.
29.4. The scripts¶
tests/lib.shShared library sourced by every numbered script. It parses the accounts INI, provides the
ssh/dependency/database/service probes used by the common pre-flight, the result counters and colouredok/nooutput, the three skip helpers (see When a case may skip), the ADMIN-host SQL and reset helpers, the mailbox send/receive helpers, the Taler wallet helpers, the RFC 3464 DSN assertion, and — for the extended scripts —send_smtp(a raw-SMTP sender that can set the ESMTP and DSN parameters themailcommand cannot) andhop_caps(an EHLO capability probe). It is not executable on its own.01-deploy.shProvisions a complete Pepsi deployment on the ADMIN host from a clean checkout: service users, PostgreSQL role/database, the configuration file (including the full stage pipeline),
pepsi-setup(TLS, schema, DKIM keys, DNS records), the demo payment merchant, and the systemd units. It only verifies required tools and bails with a precise list if any are missing — it never installs OS dependencies.02-pipeline-test.shThe core happy-path suite (T1–T6): the pay-to-send anti-spam stage (unpaid → bounce, paid → delivered, paying centrally from the runner’s Taler wallet), the language filter (blacklisted French → bounce), the outbound signing/relay path (DKIM + ARC verified at the recipient), whitelist learning with SRS envelope rewriting, and a second local user.
03-dsn-test.shRFC 3461 DSN extensions and bounce variants, using
send_smtp:NOTIFY=NEVERsuppresses the DSN (D1);ENVID/ORCPTare echoed back (D2);RET=HDRSreturns headers only (D3);NOTIFY=SUCCESSwithORIGINATE_SUCCESS_DSNyields a positive DSN (D4); an NXDOMAIN recipient routes to the default bounce (D5); an unreachable relay produces a DELAY DSN (D6); and a null-sender bounce to anSRS0=…address is reverse-decoded by ingress back to the original sender (D7).04-settings-test.shThe per-address override layer: a
pepsi-settingsrow changes one recipient’s outcome but not another’s (S1); an account edits its own overrides by e-mail and is acknowledged (S2); an edit to a non-EDITABLE_STAGESsection is rejected and stores nothing (S3); and oneDATAwith divergent per-recipient overrides is split into separate workqueue rows (S4).05-mime-test.sh8BITMIME / SMTPUTF8 handling: an 8-bit body and a UTF-8
Subjectsurvive the relay intact when the delivery hop advertises the extension, are downgraded (quoted-printable/base64, RFC 2047) when it does not, and a non-ASCII recipient address that the hop cannot represent is permanently failed. The script probes the hop’s EHLO first and takes the matching branch.06-cli-test.shThe operator CLIs, run directly over
ssh:pepsi-config(get/dump/pathsub),pepsi-whitelist(add/list/remove and the dkim/signature flags),pepsi-settings(set/get/list/unset/remove),pepsi-tlsrpt(report--dry-run/ prune), and thepepsi-httpd/metricsendpoint.07-maildir-test.shDirect (local Maildir) delivery via the
pepsi-stage-relay-to-maildirstage, which drives the setuid-rootpepsi-helper-maildir-writer. A message to the localDAVEuser lands in its Maildir with the prependedReturn-Path/Delivered-To/Receivedheaders (L1); a single message to one local and one remote recipient is split, deliveringdavelocally whilealiceis relayed onward (L2); and a message to a deliberately undeliverable account (dave-broken, no usable home) gives up pastMAX_LIFETIMEand returns a failure DSN to the sender (L3). Because[stage-local]sits after the inbound filters (… → anti-spam → local → srs), the sender is whitelisted first to clear the pay-to-send gate. The whole script fails when the deployment has no[stage-local]or the target users are missing.11-lmtp-test.shLMTP local delivery (
pepsi-stage-relay-to-lmtp) against a real Dovecot on the ADMIN host, so that the recipient’s Sieve script runs as the MDA files the mail: a message whoseSubjectcarries the Sieve trigger isfileinto :create-d into a Pepsi folder (S1, which proves Sieve ran, not merely that LMTP delivered), and a message to an unknown user is rejected per-recipient by Dovecot and routed onward toNEXT_STAGE— the stage never builds a DSN itself (S2). The script deploys and configures Dovecot itself, idempotently, and wires a throwaway[stage-lmtp]section intopepsi.conf; it drives the stage’sworkerdirectly rather than putting LMTP into the live pipeline, so it disturbs no other script. It skips only when Dovecot cannot be installed; a missing stage binary is a failure.12-wallet-topup-test.shThe by-e-mail wallet self-service half of
pepsi-stage-auto-pay(the[stage-wallet]section): a local account tops up its own per-user GNU Taler wallet by sendingpepsi-wallet@<served-domain>aSubject:command. All three methods are exercised end-to-end against the demo exchange (KUDOS), with the funded localtaler-wallet-clias the genuine peer:withdraw(manual bank-wire instructions, the wire then actually made from the[demobank]account),pull(ataler://pay-pull/URI the harness settles) andpush(the harness pushes; the user accepts thetaler://pay-push/URI). Each case asserts the control message is consumed and the wallet balance grows.13-auto-pay-test.shThe demand-paying half:
pepsi-stage-auto-paysettles a returning pay-to-send demand from the sender’s own wallet. The demand is produced genuinely: a local user sends to a paywalled local recipient (DAVE), and the outbound message is relayed over the internet (stamping a realPepsi-Origin) and loops back into this host’s inbound MX, wherepepsi-stage-anti-spamholds it and e-mails the sender a real demand. That demand is replayed through the auto-payworker. The two cases use separate senders (hence separate per-user wallets/orders):BOB’s wallet is left empty, so auto-pay forwards the demand and charges nothing (A1);ALICE’s wallet is pre-funded ahead of time (so the slow withdrawal is out of the paywall’s short 30s payment window), so auto-pay settles the demand within it — the order flips to paid, the wallet is debited, and the held message is released and delivered toDAVE(A2).14-multi-recipient-test.shRecipient fan-out, the failure mode where a message addressed to several people reaches only some of them. Both cases assert positive delivery to every intended recipient. T1 sends one envelope to two remote recipients and requires both to arrive — a relay that takes
rcpt_to.first()and then terminates the row passes nothing else on. T2 covers the~/.forwardfan-out: one envelope addressed to a local user who forwards (DAVE) and a second plain recipient, where the forwarded copy and the passthrough must each arrive exactly once — a stage that reduces the database row server-side without mutating the in-memory message leaves a fused successor working from the stale recipient list. T2 needs theDAVEaccount and the~/.forwardstage that01-deploy.shsets up.15-debian-deploy-test.shThe packaged artefact and the wizard — the real operator install path, where
01-deploy.shbuilds from a checkout and hand-writes the configuration. It builds the.deblocally, installs it on the ADMIN host (so thepostinstcreates the accounts, roles anddpkg-statoverridebits), then runspepsi-setup --wizard --answersfor a range of scenarios, each followed bypepsi-setup runand real probe mail through the pipeline the wizard generated. It rewrites the deployed host, so it does nothing but print its plan unlessPEPSI_DEPLOY_CONFIRM=yesis set.16-whitelist-import-test.shThe setuid
pepsi-whitelistand its mailbox importer, against the installed deployment — everything about that program that only exists once it is installed. It checks the installation itself (mode4755ownedpepsi-whitelist, the scan helper carrying no privilege bits, the table-scoped database role, the hoster list); the namespace rule, by creating two throwaway accounts and confirming one may use<login>/…and nothing else, thatlistdoes not disclose the operator’s shared whitelist, and that-cis refused; the import itself, which must yield exactly the recipients of the mail that user sent and ignore received mail; the wildcard decision, where a busy domain collapses to one*@domainrow whilegmail.comwith just as many correspondents does not; the privilege drop, by substituting a helper that reports its own credentials and asserting the caller’s uid appears in all three id slots;--userseeding another account’s own namespace; and finally thatpepsi-stage-check-whitelistconsults the per-user name to which its{localpart}placeholder expands. Every one of these checks fails when the deployment lacks the feature.17-crypto-test.shEnd-to-end cryptography: the parts of
pepsi-stage-encrypt,pepsi-stage-decryptandpepsi-keysthat only exist once the binaries are installed setuid and the key store holds a key-encryption key. It asserts the installation and the custody boundary (4750 pepsi-crypto:pepsi; thepepsirole cannot readcrypto_identity.private_wrapped), identity generation for both protocols, and interoperability rather than self-consistency: Pepsi’s ciphertext is read by a stockgpgandgpg’s ciphertext is read by Pepsi. It also covers the fail-open inbound verdicts, the forgedX-Pepsi-Cryptoheader, andENCRYPT = requiredbouncing rather than ever sending cleartext.18-wkd-test.shThe mirror image of
17: how other people find our keys.pepsi-httpdserves the Web Key Directory for every accepted domain, out of the key store, honouring each identity’spublishedflag — the policy file, the direct and advanced paths, that the bytes returned import into a stockgpg, thatgpg --locate-external-keyfinds them over real DNS and TLS, and that an unpublished identity and a domain we do not serve are both404.19-secure-link-test.shThe far end of
[stage-encrypt] ON_NO_KEY = secure-link: the cleartext is taken off the wire, the recipient is mailed a link and the sender a PIN. It covers the routing, the PIN going to the sender, the portal’s disclosure behaviour, the lockout afterMAX_ATTEMPTS, the successful read, the read receipt, and that the database alone holds ciphertext and no PIN.20-admin-api-test.shThe administrative API, over the UNIX-socket
ADMIN = yeslistener01-deploy.shconfigures: that all three mechanisms authenticate (SO_PEERCRED, a bearer token, a password session), that a token is confined to its scopes and a missing scope is a403, that every read-only endpoint answers JSON, and thatGET /api/v1/configdiscloses no secret.21-online-setup-test.shThat the two front ends of one setup interview — the terminal
pepsi-setup --wizardand the browser one under/api/v1/setup— do not drift: same questions, identifiers, kinds, order, defaults and conditions; answers round-tripping unchanged; the same values refused by both; and the same answers rendering to the same configuration. The last case rewrites the deployed configuration (restoring it afterwards) and is gated onPEPSI_DEPLOY_CONFIRM=yes; the read-only ones always run.22-list-api-test.shThe mailing-list compatibility gate, and the one script here that is not ours. It runs
mailmanclient3.3.5’s own suite (344 doctest statements inusing.rstplus 36 test functions) and Postorius 1.3.13’smailman_api_tests(30 files, 248 functions) against the deployed server, with only the fixture that starts GNU Mailman Core replaced — those two replacements are incontrib/compat/, versioned beside the pins. Ahead of them it checks the things whose failure would otherwise read as 250 red Python tracebacks: that the API answers on the loopbackLIST_API = yeslistener and refuses without the credential, that the same paths are a byte-identical404on the public listener, that the test-only reset hook works, and a smoke test of the resource tree including the400-versus-404asymmetry on/configand the decimal-versus-hexmember_idsplit between/3.0/and/3.1/. A suite that cannot be installed fails the run, because a silently skipped gate is the other way this test can lie; only PyPI not answering is a stated skip. A suite passes withN passedand any number ofxfailed: each expected failure is a declared difference from GNU Mailman, marked strict by the fixture, so one that starts passing fails the gate; a skip, an error or an unexpected pass fails it.contrib/compat/run-local.shruns the same two suites against a build tree with no deployment (see The GNU Mailman 3 REST API).24-list-post-test.sh…27-list-archive-test.shThe four halves of the mailing-list subsystem over real mail: a post reaching every member with its own one-click unsubscribe URI;
-join,-leaveand-confirmdriving the subscription workflows; a bounce being attributed to a member, scored, and turned into a warning and then a removal; and the archive ingesting, threading, searching and purging what was posted.Script 24 also carries the digest phase: posts accumulate, an issue goes out in each format to the members who asked for that format, the volume and number are what the rule says, and a member who switched digests off that morning still gets the final issue. It is a phase of that script rather than a new one, because the deployment suite is already long.
28-list-web-test.shThe public mailing-list pages over HTTP: that every route is a byte-identical
404on a listener withoutLISTS = yes; that an advertised list is on the index and an unadvertised one is not and is still reachable by URL; that the subscribe form mints exactly one pending token, subscribes nobody by itself, re-uses a live token rather than minting a second, and answers identically for a known and an unknown address; that a confirmationGETcompletes nothing and only thePOSTdoes; that a post is findable at its HyperKitty URL and by search with its body escaped rather than rendered; that a private archive is a404and not a403; and that the RFC 8058 one-click target removes the member with a singlePOSTeven whenunsubscription_policyismoderate, is idempotent, and removes nobody on a tampered token.Its last phase covers the member tier and the owner console: registering an account and being sent exactly one verification token (a second registration re-using it rather than minting a second), verifying and setting a password, being made an owner from the command line, signing in, and then the test that matters most — an owner changes a setting in the browser and the REST API reads the same value back, which is what proves the two surfaces write one row. It also asserts that a stranger asking for the console gets the same answer as for a list that does not exist, and that a settings post without a CSRF token is refused.
Beside the numbered scripts, tests/whitelist-suid.sh covers the same setuid
contract without SSH or a deployment: run as root on any host with a local
PostgreSQL, it creates a throwaway database, installs the binary setuid into a
suid-honouring directory and runs the same namespace/privilege-drop assertions.
make check runs it (and the other *-suid.sh scripts) best-effort — they
SKIP rather than fail when they cannot get root, a database or a suid-capable
filesystem. As root, a fixture that cannot be built is a failure.
29-list-import-test.shThe migration tooling. A real protocol-0
config.pckimports and the conversions that are not copies are asserted through the database rather than through the converter’s return value — thearchive_policycollapse, the DMARC precedence rule, the ninety days of grace period, thesubject_prefix’s trailing space, theDisableMimebit producing a plaintext digest member and a bounce-disabled member arriving disabled. The import exits non-zero, because the fixture deliberately carries a ban nobody can compile, and a second run changes nothing and says so.Its middle phase is the one worth the most:
pepsi-archive exportfollowed byimportinto a second list, comparing message count, everyMessage-IDand the thread count. That tests the two halves against each other rather than against our expectations, and a re-import adds nothing — resumption keys on content, not on a timestamp.The last phase is the cutover mailing: the per-domain histogram, a rated run, and a second run that skips what the first invited.
What it does not do is stand up a real upstream Mailman 3 to import over REST; that needs a Mailman installation on the deployment host. The gap is named in the script rather than left to be discovered.
30-secretary-test.shConfirm-to-send (
pepsi-stage-secretary). The deployment01-deploy.shbuilds has no secretary in its inbound chain, so the script splices one in aftercheck-whitelistfor its own duration and restores the configuration on exit. It asserts that an unknown sender is held and sent a null-sender challenge (X1); that an automatic reply to the challenge releases nothing and whitelists nobody (X2); that a genuine reply delivers the held message and whitelists the sender (X3), whose next message then passes without a second challenge (X4); that a bounced challenge sends the held mail down the timeout path at once (X5); and that with no reply the held mail is bounced afterHOLD_TIME, a late reply to the expired cookie doing nothing (X6).31-lmtp-interop-test.shRuns the real dispatcher and
pepsi-stage-relay-to-lmtpon the ADMIN host against eleven LMTP servers in containers (Dovecot, Cyrus, Stalwart, Apache James, maddy, Citadel, WildDuck, Mailman, aiosmtpd, go-smtp and Postfix’ssmtp-sink), in a database and configuration of its own, so the live instance is untouched. Each server gets a baseline and, throughtests/lmtp-fault-proxy.py, late, split, malformed and cut replies, a freeze across the end of DATA, andkill -9with a restart, both after the first reply and between the end of DATA and the replies. What Pepsi reports is compared with what each mailbox holds, and a big message is compared byte for byte, both as it went over the wire and as the server stored it. Citadel’s IMAP server puts three empty lines before every text and Stalwart’s leaves out the final line ending, and the comparison allows for exactly that. Stalwart answers250once a message is in its queue, so after a restart it has up to a minute to file what it acknowledged before the test calls that mail lost.Two servers are judged differently because of what they are.
smtp-sinkdiscards mail and accepts every address, so the test expects Pepsi to report an unknown user as delivered and bounce nobody, and the proxy’s ledger stands in for the mailbox. James runs on its in-memory image, which forgets mail and accounts when killed: the test creates the accounts again while the proxy refuses connections, and counts what James acknowledged before the restart by the ledger. A slow Sieve program next to a full mailbox,kill -9during that program andMAX_RUNTIMEbelowDATA_TIMEOUTrun on Dovecot alone, because they need a delivery that is slow by itself.The script skips only when it cannot reach the ADMIN host or a tool is missing there; a server that does not start is a failure.
BINDIRandSQLDIRin[lmtp-interop]select another build, andPEPSI_BINDIRandPEPSI_SQLDIRoverride them for one run.LMTP_SERVERSandLMTP_ONLYnarrow a run.
29.5. Running¶
# One-time: deploy this checkout onto the MX host (over SSH):
tests/01-deploy.sh tests/test-accounts.ini
# Then, repeatedly, from the test runner:
tests/02-pipeline-test.sh tests/test-accounts.ini
tests/03-dsn-test.sh tests/test-accounts.ini
tests/04-settings-test.sh tests/test-accounts.ini
tests/05-mime-test.sh tests/test-accounts.ini
tests/06-cli-test.sh tests/test-accounts.ini
tests/07-maildir-test.sh tests/test-accounts.ini # delivers to the DAVE account
tests/11-lmtp-test.sh tests/test-accounts.ini # deploys Dovecot on the MX host
tests/31-lmtp-interop-test.sh tests/test-accounts.ini # containers on the MX host
tests/12-wallet-topup-test.sh tests/test-accounts.ini # needs taler-wallet-cli on the MX host
tests/13-auto-pay-test.sh tests/test-accounts.ini # needs DAVE + taler-wallet-cli
tests/14-multi-recipient-test.sh tests/test-accounts.ini # T2 needs DAVE + a ~/.forward stage
tests/16-whitelist-import-test.sh tests/test-accounts.ini # needs root on the MX host
tests/17-crypto-test.sh tests/test-accounts.ini # needs the setuid crypto stages + gpg
tests/18-wkd-test.sh tests/test-accounts.ini # needs pepsi-httpd + gpg
tests/19-secure-link-test.sh tests/test-accounts.ini
tests/20-admin-api-test.sh tests/test-accounts.ini # needs the ADMIN = yes listener
tests/22-list-api-test.sh tests/test-accounts.ini # needs the LIST_API = yes listener + PyPI
tests/24-list-post-test.sh tests/test-accounts.ini
tests/25-list-command-test.sh tests/test-accounts.ini
tests/26-list-bounce-test.sh tests/test-accounts.ini # needs the DAVE account for the owner notices
tests/27-list-archive-test.sh tests/test-accounts.ini
tests/28-list-web-test.sh tests/test-accounts.ini # needs the LISTS = yes listener
tests/29-list-import-test.sh tests/test-accounts.ini # the importers and the cutover
tests/30-secretary-test.sh tests/test-accounts.ini # splices in a secretary stage
# These two rewrite the deployed host. Without the confirmation 15 only plans
# and 21 leaves out O5, and each reports what did not run as a permission skip:
PEPSI_DEPLOY_CONFIRM=yes tests/15-debian-deploy-test.sh tests/test-accounts.ini
PEPSI_DEPLOY_CONFIRM=yes tests/21-online-setup-test.sh tests/test-accounts.ini
make integrationtests ACCOUNTS=tests/test-accounts.ini runs the whole set:
the target’s INTEGRATION_SH is tests/[0-9][0-9]-*.sh minus the
tests/[0-9][0-9]-*-bench.sh scripts (BENCH_SH, which have their own
make benchmarks target — see Benchmark Suite), so a script is picked up
because it exists, not because it is listed anywhere; this includes any script
that this chapter does not mention. A failed 01-deploy.sh stops the run,
because nothing after it would mean anything; after any other failed script the
remaining scripts still run, and the target fails at the end. It finishes by
listing the failed scripts and every skip with its reason. It depends on
interop-thunderbird, so the S/MIME interoperability gate runs first.
ACCOUNTS defaults to tests/test-accounts.ini.
On a host whose web server owns port 443 for other sites too, set
BEHIND_FRONT = HOST:PORT in [deployment], which 01-deploy.sh and
15-debian-deploy-test.sh read. The public HTTPS listener then uses that address, and
the web server stays up and unchanged. 28-list-web-test.sh talks to the
listener directly, since such a server may forward only some of Pepsi’s paths.
The target sets PEPSI_DEPLOY_CONFIRM=yes, the suite’s only opt-in, because
its first script already rewrites the ADMIN host: every case runs.
make integrationtests PEPSI_DEPLOY_CONFIRM=no leaves out the two destructive
cases.
Each script is self-managing: it runs a pre-flight (ssh reachability with the
real error on failure, required remote tools, a database probe, and the three
services’ state with the journal tail of any that is down), ensures a short
PAYMENT_DEADLINE and brisk POLL_INTERVAL for fast turnaround, and resets
the whitelist/queue/settings it depends on. A missing assumption fails the
pre-flight with an actionable message; a message that does not arrive triggers a
diagnostics dump (the live workqueue plus the dispatcher journal). Tunables
such as PAYMENT_DEADLINE_SECS, DELIVER_TIMEOUT and WITHDRAW_AMOUNT are
environment overrides. Each script exits 0 only when no case failed. A skip
does not fail a script, and the last line of its summary lists every skip with
its reason. The summary also names every failure again. tests/lib.sh records
each result in a file, so a check that runs in a pipeline or inside $( )
counts like any other.
29.6. When a case may skip¶
01-deploy.sh sets up everything that the numbered scripts need: every stage,
option, account and fixture, and the small tools (curl, gpg, dig, the
Python venv module) through apt-get. So on a host with root access a
script normally skips nothing. A case that cannot run says why, and only three
reasons count. Each has its own helper in tests/lib.sh, and the printed line
carries the category:
skip_permThe permissions are truly insufficient: the ADMIN account is not root, or
PEPSI_DEPLOY_CONFIRM=yesis not set for a case that rewrites the deployed host (15, and O5 of21).skip_depA dependency is really unavailable. The cases in the suite are:
The demo exchange, the demo bank, the merchant backend or PyPI does not answer. A wallet, payment or
pipstep that fails while its service answers is a failure.tests/lib.shmakes this decision inno_unless_taler_downandno_unless_merchant_down, and22inno_unless_pypi_down.12’swithdrawcredit leg without a[demobank]account (see above).The runner’s
taler-wallet-cliis missing, is incompatible with the demo exchange, or cannot withdraw from it;taler-wallet-clicould not be installed on the ADMIN host fromdeb.taler.net. All payments are made centrally from the runner’s wallet (in production each sender would pay for their own message).The DKIM selector is not published in DNS, which is the DNS administrator’s step after
01-deploy.sh:02then cannot expectdkim=passat the recipient.A tool is missing and
apt-getcould not install it (gpg,curl, Dovecot,python3-venv).In the benchmark
08, a next hop that refused a message size: the larger sizes of that scenario. The refusal itself is reported as a failure.
skip_platformThis is the wrong platform for the test. No numbered script needs this today.
Everything else is a failure:
A step that depends on an earlier one that failed reports
not verifiedas a failure of its own, so an unverified path never reads as a pass.A stage, option, account or binary that
01-deploy.shprovides but the host lacks means the deployment is broken, not that the case does not apply.A plain
skipis a failure too, so a skip cannot be written without choosing one of the three reasons.
A few cases bring their own fixture, so that they run whatever the host offers:
M3 of
05needs a next hop without SMTPUTF8, and the real hop usually has it. The script startstests/smtp-fixture-hop.pyon the ADMIN host’s loopback, and a message with theENVIDsentinel of the test relay reaches it through[pepsi-stage-relay-to-smarthost-mta-nosmtputf8]. M1 and M2 take the pass-through or the downgrade branch according to what the real hop advertises;pepsi-test-stages’rfc2231_downgrade_lifecyclecovers the downgrades whatever that hop offers, through the real smarthost stage and a mock server that advertises neither extension.The cutover phase of
29seeds three accounts with no password in a reserved domain, so that somebody always needs an invitation.21writes through[pepsi-admin] CONFIG_DBand the privileged applier. Both are off by default, and01-deploy.shenables them: a fragment readable only bypepsi-httpdholds thepepsi-configpassword, andpepsi-setup-apply.socketis the applier’s doorbell.08deploys Dovecot the way11does forbench-lmtp. It runspepsi-milter-test-server, which01-deploy.shbuilds, forbench-milter, unlessBENCH_MILTER_ADDRnames another milter.
DAVE may be left out of the accounts file. The scripts then use
dave@<ADMIN host>, the account that 01-deploy.sh creates.
29.7. Fuzzing the parsers¶
Three surfaces parse bytes that an attacker chose: the MIME walk
(pepsi_crypto::classify, which runs on every inbound message before any
authentication), our own hand-written S/MIME CMS opener, and the OpenPGP reader
including its compressed-packet path. Beside them are the inbound authentication
parser, the DKIM public keys that it fetches from senders’ DNS, X.509 path
validation, and the mailing-list subsystem’s parsers of bounces, posts, mboxes
and imported Mailman configurations. None of them may panic, abort or run away,
and that claim is kept two ways. We do this to avoid parser defects that hostile
input can turn into a crash or worse (see
Memory corruption in ASN.1, CMS, certificate and OpenPGP parsers and Invalid, expired or old signatures reported as valid).
make fuzz runs one libFuzzer target at a time:
make fuzz # FUZZ_TARGET=parse, 300 s
make fuzz FUZZ_TARGET=cms FUZZ_TIME=3600
make fuzz FUZZ_TARGET=pgp FUZZ_TIME=0 # until interrupted
The twelve targets under fuzz/fuzz_targets/ are parse (the inbound
authentication parser), classify (the MIME walk), cms (the CMS opener),
pgp (the OpenPGP reader), dkimkey (a DKIM key record and the DER of its
p= key, which AWS-LC parses), x509 (path validation, structure-aware:
see below), bounceparse (RFC 3464 extraction and the
heuristic bounce detectors), mimebuild (the parse-modify-rebuild path of the
mailing-list posting stage, asserting the round trip as well as the absence of
panics), pickle21 (the Mailman 2.1 config.pck reader, asserting that
nothing is ever constructed), mboxsplit (the mbox splitter the archive
importer uses), dkim_request (a request line to the DKIM signing daemon and
the policy check its data goes through) and webauthn (the WebAuthn CBOR
reader and the registration and assertion checks built on it). FUZZ_TIME bounds a run in seconds (0
means run until interrupted) and FUZZ_FLAGS passes anything else through to
libFuzzer. Seed corpora live in fuzz/corpus/<target>/, which is not tracked:
PEPSI_CRYPTO_WRITE_FUZZ_CORPUS=1 cargo test -p pepsi-crypto --test crypto sweep::
writes the seeds (PEPSI_CRYPTO_WRITE_FUZZ_CORPUS=1 cargo test -p pepsi-common
dkim_key_never_panics those of dkimkey), so run it once in a fresh
checkout before fuzzing; findings land in fuzz/artifacts/<target>/.
x509 does not mutate bytes. A byte mutator rarely turns DER into a
well-formed certificate, let alone a chain in which a constrained CA sits above
a leaf naming something of the same form, so the matchers behind
nameConstraints were reached only through fixed paths. This target fuzzes a
description of a chain instead (with the arbitrary crate): a root, an
optional intermediate and a leaf, really signed; permitted and excluded subtrees
of rfc822Name (mailbox, host, .domain), dNSName and directoryName on the
root and the intermediate; and leaf names of every form, the RFC 9598
SmtpUTF8Mailbox and a DN emailAddress included. Besides the absence of
panics it asserts that a name inside an excluded subtree never validates,
judged by a deliberately simple oracle of its own.
cargo-fuzz instruments the Rust code. The C libraries below it parse hostile
input too: AWS-LC every DKIM key and S/MIME RSA operation, and Nettle (sequoia’s
backend) every OpenPGP key and message. Their crates’ build scripts compile
them, so make fuzz builds them with AddressSanitizer as well
(FUZZ_ASAN_C=yes, the default). AWS-LC is built through aws-lc-sys’s
AWS_LC_SYS_SANITIZER=asan, with clang as the C compiler and
fuzz/asan-rustc.sh as RUSTC_WRAPPER (the proc-macros link the
instrumented AWS-LC too, so rustc is given the ASan runtime to load them). This
needs clang and cmake; without them the target SKIPs with a hint, and
FUZZ_ASAN_C=no fuzzes with the C code uninstrumented. Nettle is
instrumented when FUZZ_ASAN_NETTLE=1: fuzz/build-nettle-asan.sh
downloads the installed version’s release,
checks it against a pinned SHA-256, builds the static libraries with
-fsanitize=address and without assembly (so every routine is
instrumented C) and points PKG_CONFIG_PATH at it:
make fuzz FUZZ_TARGET=dkimkey FUZZ_TIME=3600
make fuzz FUZZ_TARGET=pgp FUZZ_ASAN_NETTLE=1
For a many-core campaign, fuzz/campaign.sh starts several targets at once
under AddressSanitizer, each in libFuzzer’s fork mode with its own share of the
cores, and keeps one corpus directory per target that successive runs build
on:
cd fuzz && for t in parse classify cms pgp dkimkey x509; do
AWS_LC_SYS_SANITIZER=asan CC=clang CXX=clang++ \
RUSTC_WRAPPER=$PWD/asan-rustc.sh PKG_CONFIG_PATH=$(./build-nettle-asan.sh) \
cargo +nightly fuzz build -O --debug-assertions $t; done
fuzz/campaign.sh p1 3600 "parse:24 classify:80 cms:90 pgp:54"
The first such campaign — eight hours on 256 cores, about four billion
executions — found no crash, leak or hang. A shorter one with AWS-LC and
Nettle instrumented as well (cms, pgp, dkimkey and x509,
fifteen minutes with three workers each, about 82 million executions) found
nothing either.
fuzz/ is a separate Cargo workspace, excluded from the root one, because
libfuzzer-sys needs the nightly toolchain and a sanitizer runtime. So the
stable quality gates never build it, and make fuzz is a deliberate,
expensive, opt-in target.
Without cargo-fuzz or a nightly toolchain it SKIPs with the
install hint and exits 0, so running it on a stable-only box is harmless.
The always-on half is stable and needs no nightly: pepsi-crypto/tests/sweep.rs
is a deterministic mutation sweep over the same three crypto surfaces, input for
input, and pepsi-common’s parse_verify_never_panics and
dkim_key_never_panics cover the authentication parser and the DKIM key
path. They run under plain cargo test — that is, under make check — so a
regression that a fuzzing session would find is usually caught by the ordinary
gate first.
29.8. X.509 path validation against x509-limbo¶
The S/MIME validator’s conformance is pinned in the ordinary suite by NIST PKITS
4.6 and 4.8–4.13 (pepsi-crypto/tests/pkits.rs) and a name-constraint corpus
of our own. x509-limbo, the vector suite other
X.509 implementations are compared on, needs a file of forty megabytes that is
not in the tree. The CI image holds it. Elsewhere, fetch the pinned release and
point the test at it:
curl -LO https://raw.githubusercontent.com/C2SP/x509-limbo/554528a9b0c0d95e071f55de018326f0b65a8364/limbo.json
PEPSI_X509_LIMBO=$PWD/limbo.json cargo test -p pepsi-crypto --test crypto limbo:: -- --nocapture
Without PEPSI_X509_LIMBO the test skips. With it, the file must be the
pinned one (its SHA-256 is checked), because the list
of expected divergences in pepsi-crypto/tests/limbo.rs speaks about that
version. The test runs the cases that map onto an S/MIME validator — the path,
signatures, validity, basic and name constraints, key usage of the CAs, the
policy tree, CRLs — and leaves out what does not: the expected peer DNS name or
IP address of a case that is to fail (Pepsi binds an rfc822Name, never a host),
TLS extendedKeyUsage, ML-DSA, the Web PKI’s own profile rules and a
caller-chosen chain depth. Every result is limbo’s; a disagreement not on the
commented divergence list fails the test, and so does a listed case that has
started to agree.
29.9. Constraints¶
Testing must never send real mail to third-party MX hosts: the suite uses only
the operators’ own domains and reserved/.invalid names (e.g. the DELAY test
relays to an unrouteable 192.0.2.0/24 TEST-NET-1 address via a recipient in
blackhole.invalid). The development uid cannot bind the privileged ports
25/53, so the privileged listeners are exercised only on the deployed host.