3. Installation

Pepsi can be installed two ways. Building from source out of the Cargo workspace gives you the binaries and leaves it to you to create the service accounts, set the privileged bits and lay out the runtime directories. The prebuilt Debian package does all of that for you — its postinst creates the accounts, groups, directories, privileged-binary overrides and the database — so only the site-specific bootstrap (editing the config and running pepsi-setup) is left to do.

Pick the tab for the method you are using. Everything after it — provisioning the schema/keys/DNS, running the services, the privilege model and the schema lifecycle — is the same for both.

3.1. Prerequisites

  • PostgreSQL — one database, into which Pepsi installs the single pepsi schema. All components connect to the same database. (The Debian package Depends on postgresql and pulls it in for you.)

  • PostgreSQL’s postgresql-contrib — optional, and only for mailing lists. It carries the pg_trgm extension, which the mailing-list archive uses for substring and fuzzy search: finding a hostname, a config key, a line from a traceback, or a name the searcher has misspelled. A stemming dictionary throws exactly that material away, so full-text search cannot answer those queries at all.

    A site without it installs and runs normally. The extension is created in a guarded block, so its absence is a warning rather than a failed install; the archive then has full-text search only, and pepsi-list check says which mode the site is in. Installing it later and re-running pepsi-setup run enables it, after which the existing archive needs pepsi-archive reindex to populate the column the index reads.

    The Debian package Recommends it, so a default apt install pulls it in and a site that hosts no lists can remove it.

  • For real mail flow: public DNS you control for each served domain (to publish DKIM, SPF and MTA-STS records), and the ability to bind the privileged SMTP port 25 (and 465/587 for submission). A development build can run entirely on unprivileged ports and a throwaway database.

3.2. Installing Pepsi

Build the workspace, place the binaries and SQL with make install, and create the system accounts and privileges yourself (the same ones the package would create — see Users, groups and privileges below).

Additional prerequisites

  • A Rust toolchain with rustc 1.93 or later (the floor is set by the vendored taler-common; ./configure warns about an older one), plus the native build dependencies of the crypto and Kerberos crates: pkg-config, cmake, libclang, nettle, GMP and the Kerberos development headers (on Debian: libclang-dev, nettle-dev, libgmp-dev, libkrb5-dev).

  • The vendored GNU Taler Rust submodule, fetched before the first build:

    git submodule update --init --recursive

Building

Pepsi is a Cargo workspace whose root package (pepsi) owns every binary under src/bin/; the pepsi-* member crates are libraries the thin binaries call into. A development build produces one executable per program:

cargo build --workspace

A release build instead links most programs into a single multi-call binary, pepsi, which selects the program to run from the name it is invoked under (argv[0]). This is selected by the unibin Cargo feature (which the make build target enables for you); the default multibin feature keeps the per-program binaries used for development and the test suite:

make build   # == cargo build --release --locked \
             #      --no-default-features --features unibin,crypto-nettle

Either way the per-program names are what you run; the release layout installs them as symlinks to the one binary (see Installing). Some programs are not folded in and stay standalone; the authoritative list is the Makefile’s STANDALONE_BINARIES. Two reasons put a program there:

  • it carries its own setuid or setgid bit, which one shared binary cannot do per program — the local-delivery, ~/.forward and wallet helpers and their calling stages, the smarthost relay, pepsi-quota, pepsi-whitelist, and pepsi-stage-encrypt and pepsi-stage-decrypt (both setuid pepsi-crypto, so they can reach the private key material) — or a site may give it one (pepsi-keys);

  • it is deployed or invoked separately — pepsi-setup, pepsi-config, pepsi-stage-detect-language (its language models are large; that binary is itself multi-call, and a second $PATH symlink, pepsi-detect-language, runs the offline diagnostic tool out of it), pepsi-telemetry (which ships in its own package and runs on a separate host) and pepsi-helper-mailbox-scan (which pepsi-whitelist executes by name).

The programs this chapter refers to are listed below. This is a selection, not an inventory: the authoritative lists are the Makefile’s FOLDED_BINARIES and STANDALONE_BINARIES (BINARIES is derived from those two and is not itself editable).

  • pepsi-ingress — inbound SMTP server

  • pepsi-dispatch — stage dispatcher

  • pepsi-stage-arc, pepsi-stage-srs, pepsi-stage-encrypt, pepsi-stage-decrypt, pepsi-stage-dkim-sign, pepsi-stage-bounce, pepsi-stage-relay-to-internet, pepsi-stage-relay-to-smarthost, pepsi-stage-discard — the stages

  • pepsi-setup — provisioning/bootstrap

  • pepsi-queue — queue inspection/repair

  • pepsi-status — pipeline health summary

  • pepsi-sendmail — local submission (installed as /usr/sbin/sendmail by the Debian package)

The quality gates the project keeps green are:

RUSTFLAGS="-D warnings" cargo build --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check
cargo test --workspace
cargo deny check
make docs

cargo test --workspace needs a reachable PostgreSQL, because the dispatcher-lifecycle crate spawns the real dispatcher against a database. make check is the entry point that gates it – skipping that suite rather than failing when no database is usable – and it also runs cargo deny check; see Other make targets below.

Installing

A bare build only produces binaries; Pepsi also ships SQL migrations and a sample configuration that must be placed on disk. The Makefile wraps both — run it as root so it can also apply the privileged-binary bits (it prints the exact chown/chmod to run by hand otherwise):

make install PREFIX=/usr/local SYSCONFDIR=/etc DESTDIR=

Better still, run ./configure first and let it resolve the directory variables into config.mk (which the Makefile reads before its own defaults, so what configure decided wins). Every directory derives from --prefix, so naming a prefix and nothing else relocates the whole installation — binaries, SQL, config.d defaults and message templates — below it:

./configure --prefix=/opt/pepsi     # templates at /opt/pepsi/share/pepsi/templates
make && make install

--sysconfdir is the one exception, because a configuration file has to be where the system looks for it: it is /etc when no --prefix was given (and when --prefix=/usr was, since /usr/etc exists on no system), and $(PREFIX)/etc otherwise. Pass --sysconfdir explicitly to override either way.

./configure --help lists everything it accepts. Besides the GNU directory variables (--prefix, --exec-prefix, --bindir, --sbindir, --libexecdir, --sysconfdir, --datarootdir, --datadir, --mandir, --infodir, --docdir) it takes:

--with-templatedir=DIR

Where the message templates go; must match [pepsi] TEMPLATE_DIR. Defaults to $(DATADIR)/templates.

--with-systemdunitdir=DIR / --with-tmpfilesdir=DIR / --with-sendmail-libdir=DIR

The unit files, the tmpfiles.d snippet, and where the legacy /usr/lib/sendmail link goes.

--with-maildir-group=NAME, --with-forward-group=NAME, --with-wallet-group=NAME, --with-token-group=NAME, --with-dispatch-group=NAME, --with-crypto-user=NAME, --with-whitelist-user=NAME

The site names of the accounts and groups that gate the privileged binaries (see Privileged binaries below). They default to pepsi-maildir, pepsi-forward, pepsi-wallets, pepsi-token, pepsi, pepsi-crypto and pepsi-whitelist.

--enable-docs / --disable-docs

Whether the man pages and the info manual are rendered from source or taken pre-built (what a release tarball ships). The default is auto: on when sphinx-build, makeinfo and Python with docutils are all present.

--enable-mta-links / --disable-mta-links

Whether make install claims /usr/sbin/sendmail and the rest of the Debian mail-transport-agent interface. Off by default, so a source install cannot silently displace the system’s existing MTA; the Debian package passes INSTALL_MTA_LINKS=yes because its Conflicts: mail-transport-agent already guarantees exclusivity.

--with-crypto-backend=FEATURE

The sequoia crypto backend Cargo feature (default crypto-nettle).

CARGO=, INSTALL=, SPHINXBUILD=, MAKEINFO=, INSTALL_INFO=, RST2MAN= and PYTHON= are accepted as VAR=VALUE arguments or from the environment. Options this build has no use for (--libdir, --localstatedir, --build, --host, …) are accepted and ignored, so a generic packaging driver can call the script unchanged; only a missing cargo is a hard error.

make install performs the following steps:

install-bin

Builds the release (unified) binary and installs it to $(DESTDIR)$(PREFIX)/libexec/pepsi/pepsi, then creates one $(DESTDIR)$(PREFIX)/bin symlink per folded program name pointing at it (e.g. pepsi-ingress -> ../libexec/pepsi/pepsi). The seventeen standalone programs named in the Makefile’s STANDALONE_BINARIES are installed as their own $(DESTDIR)$(PREFIX)/bin executables. Because each symlink still lives in $(PREFIX)/bin (on $PATH), the runtime DATADIR derivation below is unaffected.

install-data

Copies the SQL files from pepsi-setup/db/ to $(DESTDIR)$(DATADIR)/sql (where DATADIR = $(PREFIX)/share/pepsi). pepsi-setup reads these when installing the schema; the default SQL_DIR is ${DATADIR}/sql. Also installs contrib/hosters.txt as $(DATADIR)/hosters.txt, the public-hoster list pepsi-whitelist import --auto-wildcard never proposes a wildcard for.

install-templates

Copies the message templates (<name>.<lang>.body — the anti-spam payment request, the bounce bodies, the mailing-list notices and the other automatic replies) to $(DESTDIR)$(TEMPLATEDIR), which defaults to $(DATADIR)/templates. The binaries’ [pepsi] TEMPLATE_DIR default is ${DATADIR}/templates and resolves through the same runtime derivation, so the two ends agree under any prefix without the configuration naming a path.

install-config

Installs the reference pepsi.conf.sample under $(DESTDIR)$(SYSCONFDIR)/pepsi/. It never creates pepsi.conf: the sample wires up optional stages that need further setup and is not a working configuration, so the live file comes from pepsi-setup --wizard (an existing one is never touched).

install-config-d

Copies contrib/config.d/*.conf to $(DESTDIR)$(DATADIR)/config.d. These are packaged defaults, not conffiles, and they are parsed before pepsi.conf — see Configuration. Two ship with Pepsi: the per-language vacation notice texts and the S/MIME container setting (CRYPTO_ALLOW_DOWNGRADE).

install-man / install-info

Installs the rendered roff man pages under $(DESTDIR)$(MANDIR)/manN and pepsi.info under $(DESTDIR)$(INFODIR) (registered with install-info). With BUILD_DOCS=no — what ./configure --disable-docs sets, and what a release tarball uses — pre-built pages are installed instead of being rendered, and each step is skipped with a message if none is present.

install-doc

Installs README.md, NEWS, SECURITY.md, AUTHORS, COPYING and the licence texts from LICENSES/ under $(DESTDIR)$(DOCDIR), which defaults to $(DATAROOTDIR)/doc/pepsi.

install-systemd

Installs the systemd units from debian/systemd/ into $(DESTDIR)$(SYSTEMDUNITDIR) (rewriting /usr/bin and /etc/pepsi to the configured BINDIR/SYSCONFDIR) plus the tmpfiles.d snippet that gives /run/pepsi its owner and ACL – which is where that directory’s permissions come from entirely, since no unit may declare it as a RuntimeDirectory (see No unit may claim it as a RuntimeDirectory). Everything ships disabled. make install INSTALL_ADMIN_UNITS=no omits the two pepsi-setup-apply units, which is the source-install equivalent of not installing the pepsi-httpd-admin package.

In addition, install depends on the privileged-bit targets (install-helper, install-maildir-stage, install-forward-*, install-auto-pay-*, install-smarthost-stage, install-whitelist-suid, install-crypto-suid, install-quota-tool) described under Privileged binaries below; each prints the chown/ chmod to run by hand when it is not run as root, or when the group it needs does not exist.

Override PREFIX, SYSCONFDIR and DESTDIR to suit the target system. make uninstall removes everything make install placed (always including both pepsi-setup-apply units, whatever INSTALL_ADMIN_UNITS says) but leaves the live pepsi.conf in place.

Other make targets

make check

The test entry point that needs no remote accounts: the file-inspection cross-checks of the systemd units and the telemetry socket path, the workspace tests (minus the dispatcher-lifecycle crate, which needs PostgreSQL), then – best effort, skipped rather than failed when the prerequisite is absent – the lifecycle suite against a local pepsicheck database, the setuid-helper contract tests, the authentication interoperability script, cargo deny check and the manual’s cross-reference check.

make integrationtests / make benchmarks

The live-pipeline scripts under tests/, which need passwordless SSH to the hosts named in ACCOUNTS (default tests/test-accounts.ini, created from tests/test-accounts.ini.sample); integrationtests first runs the Thunderbird interoperability gate. The benchmarks are a separate target because they deliberately load the MTA.

make docs

The manual for the source language – an alias for make docs-en, which builds its HTML and its PDF, plus the man pages and the info manual (both source-language only). make docs-<lang> builds one other language and make docs-all builds every language in DOC_LANGUAGES (en de fr); make html builds every language’s HTML alone.

make dist / make distcheck

Build the release tarball pepsi-$(VERSION).tar.gz (every tracked file including the vendored submodule, plus configure and pre-built documentation), and verify it by unpacking it into a clean tree and running configure, make, make check, make install and make uninstall against a throwaway prefix. The tarball is copied from the working tree, so make dist refuses to run while a tracked file differs from HEAD or the submodule is not at the commit HEAD records (untracked files do not count); make dist DIST_ALLOW_DIRTY=yes builds a development tarball anyway.

make clean / make distclean

clean runs cargo clean and removes the rendered documentation; distclean additionally removes config.mk, config.status and anything make dist generated.

Note

DATADIR must match what the binaries compute at runtime: taler-common derives ${DATADIR} as <install-prefix>/share/pepsi from the binary’s own location. When running from a source checkout (not installed), set [pepsi-postgres] SQL_DIR explicitly so pepsi-setup finds the migrations.

Preparing the system

The Debian package’s postinst does the steps below automatically; on a source install you do them yourself, using the same account, group, mode and directory layout documented under Users, groups and privileges and Directories and secrets below. Create the accounts and groups before make install (so it can apply the privileged bits), and the database roles before pepsi-setup (which installs the schema):

  1. Create the system accounts, each --system with /usr/sbin/nologin and home /var/pepsi and a private group of the same name, plus the gating groups. debian/pepsi.postinst is the authoritative list; this loop is it, translated:

    for u in pepsi pepsi-ingress pepsi-httpd pepsi-owner \
         pepsi-helper-token-refresh pepsi-whitelist pepsi-crypto \
         pepsi-keydisc pepsi-config; do
    addgroup --system "$u"
    adduser --system --home /var/pepsi --no-create-home \
            --ingroup "$u" --shell /usr/sbin/nologin \
            --disabled-password "$u"
done
for g in pepsi-maildir pepsi-forward pepsi-token pepsi-admin \
         pepsi-telemetry; do
    addgroup --system "$g"
done
# The shared-wallet account owns its wallet databases, so unlike the
# service accounts it gets a real home.
addgroup --system pepsi-wallets
adduser --system --home /var/lib/pepsi-wallets --ingroup pepsi-wallets \
        --shell /usr/sbin/nologin --disabled-password pepsi-wallets
install -d -o pepsi-wallets -g pepsi-wallets -m 0700 /var/lib/pepsi-wallets
# Memberships: telemetry producers, and the SRS secret fragment.
adduser pepsi pepsi-telemetry
adduser pepsi-ingress pepsi-telemetry
adduser pepsi pepsi-ingress

    Each of the last four accounts exists because something refuses to work without it: pepsi-whitelist is the account the whitelist CLI is setuid to, pepsi-crypto is the only database role granted crypto_identity.private_wrapped and the owner of the key-encryption fragment, pepsi-keydisc is the narrowest role in the deployment (it parses key material fetched from the open internet), and pepsi-config is the only role that may write the configuration overlay. Skip pepsi-crypto and install-crypto-suid merely warns — leaving the two crypto stages unprivileged and unable to open a single private key.

  2. Create the nine PostgreSQL login roles and the database (peer auth, as the postgres superuser) — the same set debian/pepsi.postinst creates, every account that authenticates as itself over the peer-auth socket. pepsi-telemetry is among them even on a node that serves no telemetry: pepsi-setup run grants it on every host, and it does its database work as pepsi-owner, which cannot create a missing role:

    for r in pepsi-owner pepsi pepsi-ingress pepsi-httpd \
         pepsi-whitelist pepsi-config pepsi-crypto pepsi-keydisc \
         pepsi-telemetry; do
    createuser "$r"          # createuser takes one role name at a time
done
createdb -O pepsi-owner pepsi

  3. Run make install as root (above) so the setuid/setgid bits are applied; otherwise apply them by hand per Privileged binaries below. Each install-* target only warns when the group it needs is missing, so an account or group skipped above leaves the corresponding program silently unprivileged.

  4. Create the runtime directories with the ownership and modes listed under Directories and secrets. pepsi-setup creates /var/pepsi/keys for you; create the smarthost-secret directories only if you configure a smarthost using them.

The package is the simpler path: it ships the binaries (installed to /usr/bin), the SQL schema, the message templates, the man pages, the GNU info manual and socket-activated systemd units, and its postinst provisions the whole system for you.

Installing the package

Install the .deb with apt so its dependencies (including postgresql) are pulled in:

apt install ./pepsi_*.deb

On configure the postinst automatically:

  • creates the ten service accounts and the six gating groups — pepsi-maildir, pepsi-forward, pepsi-token, pepsi-admin, pepsi-telemetry and pepsi-wallets — with the memberships between them (the same identities the source install makes by hand);

  • creates the /var/pepsi runtime directories — keys, tokens, token-refresh, tls, krb5 — and /etc/pepsi/secrets and /etc/pepsi/secrets.d with the ownership and modes documented under Directories and secrets below;

  • applies the privileged-binary bits via dpkg-statoverride (the three setuid-root helpers, the setgid programs that may exec them — the two delivery stages, the wallet stage and pepsi-quota — the setgid smarthost relay, the setuid pepsi-whitelist and the two setuid pepsi-crypto stages — see Privileged binaries below); and

  • best-effort creates the nine PostgreSQL login roles (pepsi-owner, pepsi, pepsi-ingress, pepsi-httpd, pepsi-whitelist, pepsi-config, pepsi-crypto, pepsi-keydisc, pepsi-telemetry) and the pepsi database (owned by pepsi-owner) when a local cluster is reachable — printing the roles to create by hand if it is not.

The service is shipped disabled: the package installs but does not start Pepsi, leaving the site-specific bootstrap to you.

Note

pepsi is one of four binary packages built from this source: the language-detection stage and the telemetry collector are split out for size and deployment reasons, and pepsi-httpd-admin — which pepsi Recommends, so apt installs it by default — carries the only mechanism by which the browser console can cause a privileged change on this host. Removing that one package is a supported hardening step for a deployment administered from a terminal. Debian packages describes each package and that lever in full.

After the package is installed

On first installation the package copies a placeholder configuration to /etc/pepsi/pepsi.conf; later upgrades never touch that file (it is not a dpkg conffile, so an upgrade never stops to ask about it). Edit it, then run the bootstrap exactly as in the shared steps below — for example:

# 1. edit /etc/pepsi/pepsi.conf (HOSTNAME, ACCEPTED_DOMAINS, domains, TLS)
# 2. install the schema + DKIM keys and print the DNS records:
pepsi-setup --no-certbot run        # or:  pepsi-setup --wizard
# 3. publish the printed DNS records, then start the pipeline:
systemctl enable --now pepsi.target

Because the units are socket-activated, pepsi.target brings up pepsi-ingress, pepsi-httpd and pepsi-dispatch together; you do not invoke the serve subcommands by hand.

3.3. Migrating from an existing mail server

If this host already runs Postfix, Exim, Sendmail, qmail or Stalwart, you do not have to re-derive its configuration by hand. With no pepsi.conf present, pepsi-setup --wizard detects what is installed and offers to import it: the hostname, accepted domains, smarthost (with the relay credentials read from that server’s own password file), local-delivery method, submission SASL socket, trusted networks, message size limit, queue lifetime and TLS certificate paths all arrive as the interview’s pre-filled answers, which you then confirm or correct one by one.

To see what a migration would produce before running the wizard — nothing is written:

pepsi-setup import

Routing tables are converted too. /etc/aliases, /etc/postfix/virtual, Sendmail’s virtusertable, qmail’s .qmail-* files and their equivalents become a Pepsi alias map beside the configuration (see pepsi-stage-aliases), with bare alias names qualified in the style you choose. Since Pepsi alias keys must carry a domain, postmaster: root becomes postmaster@example.org  root@example.org for each accepted domain by default.

Everything that does not carry over is written to import-report.txt beside the configuration, split into what failed to parse, what Pepsi has no equivalent for (header_checks, an alias delivering to a command), what was approximated (mbox delivery becoming Maildir) and what the importer did not recognise. An import never aborts setup — an unreadable file or an unknown directive produces a report entry, not a failure.

Two things are deliberately not migrated: queued mail (let the old server’s queue drain before switching the MX over) and DKIM keys (pepsi-setup run generates fresh ones and prints the records to publish). If the old MTA still holds port 25 when the wizard runs, the report says so — stop and disable it before starting pepsi-ingress.

See pepsi-setup for the per-server detail, the --alias-style choices and the --expert options that expose the settings the interview does not otherwise ask about.

3.3.1. Mail filters already on the host

Independently of any migration, the wizard scans the host for milter daemons — clamav-milter, rspamd, spamass-milter, milter-greylist, milter-regex, mimedefang, amavisd-milter — and offers each one it finds as a stage. Finding one means three things in turn: the package is installed, a socket can be read from its own configuration (or its unit, or the paths its distribution ships), and a real milter option negotiation completes against that socket. Nothing is offered on the strength of an installed package alone, and the negotiation is also how the generated ALLOW_ACTIONS comes to grant exactly what that filter asks for rather than a guess.

Accepted filters are placed by what they do — policy and greylisting, then virus, then spam, then general-purpose frameworks — after decryption and before the whitelist and paywall gates. Because Pepsi filters after accepting a message, a filter’s REJECT is routed to a generated discard stage rather than bounced, so forged senders get no backscatter; see pepsi-stage-milter. Filters whose job Pepsi already does (opendkim, openarc, opendmarc, SPF daemons, postsrsd) are named and skipped rather than offered.

3.4. Provisioning the database, keys and DNS

With a configuration file in place (see Configuration), run the bootstrap tool once:

pepsi-setup -c /etc/pepsi/pepsi.conf run > pepsi-dns.zone

This validates the whole configuration, installs the pepsi schema, generates per-domain RSA-2048 and Ed25519 DKIM keys under KEY_DIR (only if absent), and prints the DNS records to publish (DKIM public keys, an SPF policy from the configured PUBLIC_IP addresses, and an MTA-STS record). The operation is idempotent and may be re-run at any time; upgrading the schema to a new release is pepsi-setup schema (see Upgrading). run --reset drops and recreates the schema (destroying all queued mail; keys on disk are kept).

Publish the printed DNS records, then verify what is live against what Pepsi expects:

pepsi-setup -c /etc/pepsi/pepsi.conf check

See pepsi-setup for the full provisioning workflow.

3.5. Setting up in a browser

Everything above can also be done from a browser, against the same code. The terminal path is the shortest route on a machine you have a shell on; the browser path is for the ones you do not, and for operators who would rather see the DNS records as a red/green checklist than as a zone file.

3.5.1. The split, and why

Setup is privileged: it writes /etc/pepsi/pepsi.conf, hands each secrets.d fragment to the single account that reads it, runs certbot, creates database roles and generates key material — all as root. pepsi-httpd, which serves the interface, drops privileges before it accepts a connection and must never regain them.

So it does not act. It writes an intent row saying what should be true, and a separate root program — pepsi-setup apply, started on demand and gone again when it has nothing to do — decides how, does it, and records what happened. Root is never on the network. Read “The applier’s trust model” in :doc:`programs/pepsi-setup` before enabling this: it states exactly what the applier will and will not do, including the things it deliberately cannot (there is no restart or shutdown task, and install-schema refuses reset).

3.5.2. Enabling it

The Debian packages ship a minimal but complete configuration, an administrative listener on a UNIX socket, and — in pepsi-httpd-admin, which pepsi Recommends and apt therefore installs by default — the applier’s socket unit (see Debian packages). Three steps:

# systemctl enable --now pepsi-httpd.socket pepsi-setup-apply.socket
# pepsi-setup -c /etc/pepsi/pepsi.conf bootstrap

The second prints a one-time token and the curl command that turns it into the first administrator account. On the machine itself you can skip it entirely: a member of the pepsi-admin group (and root) is identified by SO_PEERCRED over /run/pepsi/admin.sock and needs no credential at all.

One more setting is needed before anything privileged can be asked for: [pepsi-admin] CONFIG_DB must name a connection that authenticates as the pepsi-config database role. That is the boundary: only that role may enqueue setup work, and pepsi-httpd’s own account is not it, so a compromise of the web tier is not a compromise of the pipeline definition. Until it is set the setup endpoints answer 503 and say so.

3.5.3. Driving it

The interview is data. GET /api/v1/setup/questions returns the ordered steps and, for each question, its shape, default, help text and the condition under which it is asked — the same model pepsi-setup questions prints and the same one the terminal wizard is described by, asserted equal by the test suite. Answers are staged with PUT /api/v1/setup/answers into draft rows that no running process can see, so an interrupted session cannot half-configure anything.

POST /api/v1/setup/tasks then asks for the privileged half — writing the configuration, obtaining certificates, installing the schema, generating keys — and GET /api/v1/setup/tasks/{id}?since=<seq> streams what the applier is doing, line by line.

3.5.4. What will always need a text editor

Some settings are read from the configuration file only, and no interface will change that: [pepsi], [pepsi-postgres], [paths], and the HTTP(S) and ingress listener sections. They have to work before a database connection exists, or they are a security boundary — a database that could move a listening socket or relax the crypto policy would make a database compromise a compromise of the MTA’s own TLS and ports.

So adding a submission port, or moving a listener, is a text-editor-and-restart operation. So is any change the console marks restart rather than hot: there is no restart button, because a button that restarts a service on request would hand the web tier’s privileges to whoever reaches it.

3.5.5. Scripting it

The same answer model drives an unattended install:

pepsi-setup --wizard --answers answers.json -c /etc/pepsi/pepsi.conf

pepsi-setup questions prints the schema that file is written against. This is not a third implementation of the interview: it runs the interactive one with every prompt answered from the file, so the branches and the per-field validation are the same ones a human sees.

3.6. Running the services

Two long-lived processes need to run continuously:

  • pepsi-ingress serve — accepts mail (typically under a service manager, optionally with systemd socket activation for the privileged ports).

  • pepsi-dispatch serve — runs exactly one instance per system; it is the only process that starts stage programs.

The stage programs themselves are spawned by the dispatcher as persistent PROGRAM worker processes that read message ids on standard input; you do not run them by hand in production (for debugging you can pipe an id to worker, e.g. echo 42 | pepsi-stage-srs -c /etc/pepsi/pepsi.conf worker).

3.7. Upgrading

From the first release, 0.0.0, on, every release upgrades the database of any earlier release in place: the queue, the key store, the settings and everything else in it are kept, and mail waiting in the queue is delivered by the new release. Downgrades are not supported (see below).

The procedure:

  1. Read NEWS for the release. It lists configuration options that changed, and anything else that needs doing by hand.

  2. Take a backup (Backup and restore), or at least enable the automatic dump described below.

  3. Install the new release. The Debian packages do the rest: they upgrade the schema and restart the services that were running. From source: make install, then pepsi-setup -c /etc/pepsi/pepsi.conf schema and systemctl try-restart pepsi.target.

  4. Check with pepsi-status that the queue drains, and look at the journal for warnings (Logs).

The rest of this section explains what happens during step 3.

Every Pepsi program checks, as it connects to the database, that the schema was built from exactly the SQL files the program itself was built with. The installer records the SHA-256 and release of each file it applies in pepsi.schema_file, and a program that finds anything else – an older schema, a newer one, or one built from different files – stops at once with exit status 78 and a message naming the fix, rather than failing at its first missing column in the middle of a message. The dispatcher treats a stage worker that stops this way as “cannot run yet”, not as a crash: the message it was handed is put back in the queue, never failed.

Upgrading the schema is its own step:

pepsi-setup -c /etc/pepsi/pepsi.conf schema

It installs the patches this release adds, replaces the stored functions, re-applies the role grants and does nothing else – no certificates, keys or DNS – so it needs only the [pepsi-postgres] section. On a telemetry collector, give it the collector’s file, /etc/pepsi-telemetry/pepsi-telemetry.conf.

Debian packages run it automatically when they are upgraded, before the services are restarted; a new installation is left to pepsi-setup run. If it fails (the database is down, say), the package still installs, a notice says so, and the services refuse to start and are retried by systemd – backing off to once a minute, without a start limit – until the schema matches. Once the cause is fixed, running the command above is all it takes; the services come back by themselves.

From source, after make install:

pepsi-setup -c /etc/pepsi/pepsi.conf schema
systemctl try-restart pepsi.target

What the refusals mean:

The message says

What to do

no pepsi schema

A new installation: pepsi-setup run (or pepsi-setup schema on a collector host).

older than this program

pepsi-setup -c FILE schema.

newer than this program

A downgrade, which is not supported. Install the newer release again, or restore the database backup taken before the upgrade (below).

built from a different copy of a patch

A released patch never changes, so this is a bug; please report it. The one exception is a database built from a development snapshot (a Git checkout between releases), where the newest patch is still being edited: drain it and re-create it with pepsi-setup run --reset.

stored functions from a different build

Development builds only: pepsi-setup -c FILE schema.

3.7.1. Downgrades

Not supported. pepsi-setup refuses to install an older release’s SQL over a newer schema (exit status 78, nothing changed), and the older release’s programs refuse to run on it. Going back to an older release means restoring a database backup taken before the upgrade, then installing the older packages.

3.7.2. Backing up before a schema upgrade

No backup is taken unless you enable it. When enabled, the backup is taken only when an upgrade actually changes the schema: pg_dump saves the pepsi schema (with the extensions it uses) as pepsi-<previous version>-<UTC time>.dump, readable by root only. If the dump fails, the schema is not upgraded. Backups are never deleted automatically.

With the Debian packages it is a debconf setting, asked only at priority low:

dpkg-reconfigure -plow pepsi

Answer yes, and confirm or change the directory (default /var/backups/pepsi). Unattended, or before the first upgrade:

echo "pepsi pepsi/schema-backup boolean true" | debconf-set-selections
echo "pepsi pepsi/schema-backup-dir string /var/backups/pepsi" | debconf-set-selections

On a telemetry collector host the questions belong to pepsi-telemetry (pepsi-telemetry/schema-backup, pepsi-telemetry/schema-backup-dir). When both packages are installed and use the same database, whichever upgrades it first takes the backup.

From source, pass the directory to the upgrade step:

pepsi-setup -c /etc/pepsi/pepsi.conf schema --backup-dir /var/backups/pepsi

To restore one, stop the services, re-create the database empty and owned by the schema owner, and restore into it as that owner:

systemctl stop pepsi.target
sudo -u postgres dropdb pepsi
sudo -u postgres createdb -O pepsi-owner pepsi
sudo -u pepsi-owner pg_restore -d pepsi /var/backups/pepsi/pepsi-0.0.0-20261001T000000Z.dump

Then install the release the backup came from and start the services again. Everything the database learnt after the backup – mail accepted since, settings changed since – is lost. The stored private keys in the dump are encrypted under the key-encryption key in secrets.d, which is not in the database: keep that file, and back it up separately (see Backup and restore).

3.8. Users, groups and privileges

Pepsi never runs as root. The long-lived daemons may be started as root so they can bind privileged ports and read root-only TLS keys, but each one drops to its own unprivileged service account before serving a single connection; if the drop cannot be completed the process refuses to start rather than risk running as root (see pepsi-common::privdrop). Local delivery — which must write into arbitrary users’ mailboxes — is the one operation that needs more than the service user can give, and it is confined to a single narrowly-scoped setuid helper rather than handed to the pipeline at large.

Under the shipped systemd units the daemons are never root at all: systemd binds the privileged sockets (socket activation) and starts each process as its service account directly. That leaves the TLS keys, which certbot keeps readable by root only — so pepsi-setup has systemd read those too, writing a LoadCredential= drop-in per unit: systemd opens each certificate and key as root when it starts the unit and hands the service a private copy under $CREDENTIALS_DIRECTORY, which is where the server looks first. Re-run pepsi-setup run whenever you add, move or remove a certificate, so the drop-in is regenerated. See pepsi-httpd.

The browser setup described above is the one path by which something other than an operator at a terminal can cause a root process to run: pepsi-httpd rings a doorbell socket and systemd starts a short-lived root pepsi-setup apply. That path is the easiest privilege in Pepsi to take away. Its two systemd units are a separate Debian package, pepsi-httpd-admin, whose source-install equivalent is make install INSTALL_ADMIN_UNITS=no; without them nothing drains pepsi.setup_task and no HTTP request can change anything under /etc/pepsi, while pepsi-setup on a terminal keeps working unchanged. Putting the package back does not retroactively act on what was asked for while it was away: installing it empties that queue before its units are armed. See The split as a hardening lever.

The accounts and groups below are created automatically by the Debian package’s postinst; on a make install you create them yourself (the same names). PostgreSQL access is by peer authentication over the local socket: each service account logs in as a same-named database role, so no passwords are stored anywhere. A remote database keeps the same model: one login role per account, each authenticated by a client certificate or password file of its own that only that account can read, named per role with the {role} placeholder in [pepsi-postgres] CONFIG (see CONFIG in pepsi.conf(5)). The configuration file never holds a password.

3.8.1. Service accounts

These are system accounts (--system, /usr/sbin/nologin, home /var/pepsi), each with a private primary group of the same name. The first three are long-lived daemons that drop to their account after binding; the rest are not daemons in the pipeline.

Account

Runs

Purpose / access

pepsi-ingress

pepsi-ingress

Inbound SMTP server. Socket-activated under systemd (never root, TLS material arrives as a service credential); started by hand it binds 25/465/587 and reads the TLS keys as root, then drops. DB role pepsi-ingress.

pepsi-httpd

pepsi-httpd

HTTP/HTTPS server (MTA-STS policy + /metrics). Socket-activated under systemd, exactly like pepsi-ingress; started by hand it binds 80/443 as root, then drops. DB role pepsi-httpd.

pepsi

pepsi-dispatch and every stage worker it spawns

The pipeline’s working identity. Reads the DKIM keys (via group pepsi) to sign. DB role pepsi (owns the queue rows). This is the account that gains the two delivery group-identities below transiently.

pepsi-owner

pepsi-setup (transiently)

Not a runtime service. Owns the pepsi database, schema, tables and functions. When pepsi-setup runs as root it temporarily assumes this identity (seteuid) so the objects it creates are owned by the peer-authenticated pepsi-owner role, then returns to root for the filesystem work (key generation, ownership fix-ups).

pepsi-helper-token-refresh

pepsi-helper-token-refresh service

Only present when a smarthost uses AUTH = oauth. Refreshes OAuth access tokens and writes the token files the smarthost relay reads. Has no database role.

pepsi-crypto

pepsi-keys (transiently), and the end-to-end crypto stages

Custodian of private end-to-end key material. Owns secrets.d/pepsi-crypto.secret, the key-encryption key that opens pepsi.crypto_identity.private_wrapped, and is the only database role granted that column — every other role, pepsi included, gets a column-level grant that omits it. pepsi-keys assumes this identity for the duration of a database call and gives it straight back. See Key management.

pepsi-whitelist

pepsi-whitelist (transiently)

Account the setuid whitelist CLI assumes so ordinary users can maintain their own sender whitelists. Its database role is granted SELECT/INSERT/DELETE on pepsi.whitelist and nothing else.

pepsi-keydisc

pepsi-keydisc@<method> services

Key discovery (WKD, DANE, VKS, LDAP). Parses key material fetched from the open internet, so its database role is the narrowest in the deployment. See Key management.

pepsi-config

pepsi-config (as run by an operator)

The only database role that may write the pepsi.config_override overlay: a component that processes mail must not be able to rewrite the pipeline it runs in.

The nine PostgreSQL login roles — pepsi-owner, pepsi, pepsi-ingress, pepsi-httpd, pepsi-telemetry, plus the narrow pepsi-whitelist, pepsi-config, pepsi-crypto and pepsi-keydisc roles — are created by the package postinst (or, on a manual install, by you) before pepsi-setup installs the schema; pepsi-setup then grants each what it needs.

3.8.2. Groups gating shared resources

Four groups exist purely to gate access to privileged binaries and secret files. The pepsi user is deliberately not a permanent member of any of them: the setgid bit on the relevant stage binary grants the worker the group identity only for the lifetime of that process.

pepsi-maildir

Gates the local-delivery helper. pepsi-helper-maildir-writer is setuid root and mode 4750 (root:pepsi-maildir), so only this group may execute it. pepsi-stage-relay-to-maildir is setgid this group (mode 2550, pepsi:pepsi-maildir — owner-execute only, so no other local user can reach the setgid bit); when the pepsi worker execs the stage it gains egid=pepsi-maildir, exactly the right to launch the helper — which then setuids to the recipient and writes ~/Maildir/new/. The helper is the only code that ever touches a third party’s mailbox.

pepsi-forward

Gates the ~/.forward helper, in exactly the shape of the maildir pair above: pepsi-helper-dot-forward is setuid root and mode 4750 (root:pepsi-forward), and pepsi-stage-dot-forward is setgid this group (mode 2550, pepsi:pepsi-forward). Only needed if you run the ~/.forward stage; without the group, make install’s install-forward-helper / install-forward-stage targets only print a warning and leave the binaries unprivileged.

pepsi-wallets

Gates the wallet helper the same way: pepsi-helper-auto-pay is 4750 root:pepsi-wallets and pepsi-stage-auto-pay is setgid this group (mode 2550, pepsi:pepsi-wallets). A system account of the same name owns the shared wallet databases under /var/lib/pepsi-wallets when WALLET_MODE = shared. Only needed if you run the auto-pay stage; install-auto-pay-helper / install-auto-pay-stage degrade to a warning otherwise.

pepsi-token

Gates the secret material the smarthost relay reads: OAuth token files, the mutual-TLS client cert/key, and the Kerberos credential cache. pepsi-stage-relay-to-smarthost is setgid this group (mode 2550, pepsi:pepsi-token); the directories holding those secrets are setgid pepsi-token so files dropped into them inherit the group, and the relay can read them without the pepsi user being a member.

3.8.3. Privileged binaries

These bits are set by make install when run as root (it prints the exact chown/chmod to run by hand otherwise), and by the Debian package via dpkg-statoverride, which is the authoritative list (debian/pepsi.postinst):

Binary

Mode / owner

Why

pepsi-helper-maildir-writer

4750 root:pepsi-maildir

Setuid root so it can write into any local user’s mailbox; group-locked so only pepsi-maildir may run it.

pepsi-stage-relay-to-maildir

2550 pepsi:pepsi-maildir

Setgid so the pepsi worker may exec the helper above. Owner-execute and not world-execute: the setgid bit is the whole gate, so every other local user must not be able to reach it. pepsi cannot get the exec right from the group (it is deliberately not a member), so it gets it by owning the file. pepsi-quota, pepsi-stage-dot-forward and pepsi-stage-auto-pay carry the same shape for their own groups.

pepsi-quota

2550 pepsi:pepsi-maildir

Same group and same reason: measure/reconcile run the helper above, and the bit is what lets an unattended sweep do it without root.

pepsi-helper-dot-forward

4750 root:pepsi-forward

Setuid root so it can run a user’s ~/.forward as that user; group-locked to its stage.

pepsi-stage-dot-forward

2550 pepsi:pepsi-forward

Setgid so the pepsi worker may exec the helper above.

pepsi-helper-auto-pay

4750 root:pepsi-wallets

Setuid root so it can drop to the wallet-owning account (or the recipient’s own uid) to run taler-wallet-cli.

pepsi-stage-auto-pay

2550 pepsi:pepsi-wallets

Setgid so the pepsi worker may exec the helper above.

pepsi-stage-relay-to-smarthost

2550 pepsi:pepsi-token

Setgid so the pepsi worker may read the pepsi-token secrets; not world-executable, since the stage accepts -c and would otherwise send those secrets to any server a local user names.

pepsi-whitelist

4755 pepsi-whitelist:pepsi-whitelist

Setuid, and world-executable on purpose: any local user may manage their own <login>/… namespace, and the rule stopping them touching anyone else’s is inside the program, not the file mode.

pepsi-stage-encrypt, pepsi-stage-decrypt

4750 pepsi-crypto:pepsi

Setuid, not setgid: PostgreSQL peer authentication keys off the effective uid, and pepsi-crypto is the only role granted crypto_identity.private_wrapped. Mode 4750 with group pepsi because these are not user commands — only the dispatcher’s account and root may run them.

Every other binary is installed unprivileged, including pepsi-keys, which is standalone for the same reasons but ships 0755: one binary able to open every private key in a deployment is a much larger prize than pepsi-whitelist’s single table, so making it setuid pepsi-crypto is a per-site decision rather than the default.

3.8.4. Directories and secrets

pepsi-setup and the package create the following under /var/pepsi (the service home) and /etc/pepsi. The setgid (2750) directories cause new files to inherit the directory’s group, so token/TLS/Kerberos material is group-readable by the relay without any per-file chgrp.

Path

Owner / mode

Contents

/var/pepsi

pepsi-owner:pepsi

Service home.

/var/pepsi/keys

pepsi-owner:pepsi 2750

Per-domain DKIM private keys: written by pepsi-setup, read by the pepsi dispatcher (group pepsi) to sign.

/var/pepsi/tokens

pepsi-helper-token-refresh:pepsi-token 2750

OAuth access-token files: written by the refresher, read by the smarthost relay via group pepsi-token.

/var/pepsi/token-refresh

pepsi-helper-token-refresh 0700

Private rotated refresh tokens; never group-readable.

/etc/pepsi/secrets

root:pepsi-helper-token-refresh 0750

The @inline-secret@ token-refresh credentials; only the refresher may read them.

/etc/pepsi/secrets.d

root:root 0751

The fragments pepsi-setup generates: the SRS key, smarthost/LMTP passwords, merchant and resume tokens, the proof-of-origin key and the pepsi-crypto key-encryption key. World-searchable (not readable) so each service can reach the one fragment it may open; the fragments themselves are 0640 owned by their single reader account. pepsi-setup creates the directory if it is absent.

/var/pepsi/tls

root:pepsi-token 2750

Mutual-TLS client cert + key for the smarthost relay (TLS_CLIENT_CERT / TLS_CLIENT_KEY).

/var/pepsi/krb5

root:pepsi-token 2750

Kerberos credential cache for AUTH = gssapi smarthosts (point KRB5CCNAME at a FILE: cache here; an external keytab refresher populates it).

/run/pepsi

root:root 0775 + ACL

The local submission, administrative, HTTP and setup-applier sockets. Created by the tmpfiles.d snippet, whose ACL grants pepsi-ingress and pepsi-httpd write access by name.

/var/lib/pepsi-wallets

pepsi-wallets:pepsi-wallets 0700

Home of the pepsi-wallets account and the wallet databases pepsi-helper-auto-pay creates under it (wallets/<key>.sqlite3, mode 0700). Only with pepsi-stage-auto-pay in WALLET_MODE = shared.

The /var/pepsi/tokens, /var/pepsi/tls and /var/pepsi/krb5 directories (and the pepsi-helper-token-refresh account) only matter when you configure a smarthost with the corresponding AUTH mechanism; a default direct-to-MX or local-delivery deployment uses none of them.

3.9. Migrating from GNU Mailman

An existing GNU Mailman installation — 2.1 or 3 — moves onto Pepsi with pepsi-list import and pepsi-archive import. Read the table below before you start, not afterwards.

The behavioural differences are a different table, and it is not duplicated here: Declared differences from GNU Mailman 3 in Mailing lists collects every place Pepsi deliberately does something other than what upstream does — one-click unsubscribe, the always-per-recipient fan-out, the archive’s obscured sender addresses, the purge tombstone, the renamed X-Mailman-* headers, and what is simply absent (no NNTP gateway, no pluggable archivers, no plugin API). Two copies of that table would diverge, and the copy a migrating operator happened to read would be the stale one. This section is only about what the importer leaves behind.

3.9.1. What does not come across, and why

Why

Passwords

Neither Mailman core’s sha512_crypt hashes nor mailman-web’s Django pbkdf2_sha256 ones are imported. Upstream’s own importer has the code and it is commented out. This is why the importer never has to read the Django database at all, and why there is no legacy hash verifier anywhere in Pepsi. Verified addresses stay verified, so nobody re-proves an address — they only choose a password, which is what pepsi-list invite is for.

Bounce scores

Upstream does not import them either: config.pck’s bounce_info and Mailman 3’s bounce events are both discarded, so a migrated member starts at score zero on both systems.

In-flight tokens

A pending confirmation or moderation token is a promise about a workflow that will not exist after the cutover. Anybody mid-subscription asks again.

Topics (2.1)

No Mailman 3 equivalent.

Template URIs we refuse to fetch

A file: override would read the server’s own disk, so it is not imported. The importer reports every such URI at import time rather than letting the first notice that needs it fail months later, and the list falls back to the built-in template.

Anything else the importer did not understand is in its report. It exits non-zero when something was lost, which is the point: a migration that half worked and exited zero is a migration nobody checks. Nothing is rolled back, and re-running an import is safe — every write is keyed on the list id, the address or the Message-ID, so a second run is a no-op.

3.9.3. When our pickle reader is stricter than theirs

pepsi-list import mailman21 reads config.pck with its own parser, which constructs nothing: no module is looked up and no callable is called, because a config.pck comes from somebody else’s server and Python’s pickle.load on untrusted input is arbitrary code execution.

The price of that is strictness. If the reader refuses a file — an opcode it does not implement, a protocol newer than Python 2 could write, an integer too wide for 64 bits — it says so and names the escape hatch:

python2 contrib/mm21-export.py /var/lib/mailman/lists/announce/config.pck \
    > /root/announce.json
pepsi-list import mailman21 --json /root/announce.json --list announce

That script runs on the old server, with the Python that server already trusted, and dumps the same dictionary as JSON. Both routes then go through the same mapping, so they cannot disagree about what the file means. A migration must never be blocked by our parser being stricter than the one that wrote the file.

3.9.4. Header renames a user will notice

Pepsi emits X-Pepsi-List-* where Mailman emitted X-Mailman-*; the mapping table is in Mailing lists. This is a migrating user’s real cost, because procmail and Sieve rules are written against those names, and it is worth telling them before the switch rather than after.