4. Installation

Pepsi can be installed two ways. Building from source out of the Cargo workspace gives you the binaries and leaves it to you to create the service accounts, set the privileged bits and lay out the runtime directories. The prebuilt Debian package does all of that for you: its postinst creates the accounts, groups, directories, privileged-binary overrides and the database. Only the site-specific bootstrap (editing the configuration and running pepsi-setup) is then left to do.

Pick the tab for the method that you are using. Everything after it is the same for both: provisioning the schema, keys and DNS, running the services, the privilege model and the schema lifecycle.

4.1. Prerequisites

  • PostgreSQL — one database, into which Pepsi installs the single pepsi schema. All components connect to the same database. (The Debian package Depends on postgresql and pulls it in for you.)

  • PostgreSQL’s postgresql-contrib — optional, and only for mailing lists. It carries the pg_trgm extension, which the mailing-list archive uses for substring and fuzzy search: finding a hostname, a config key, a line from a traceback, or a name that the searcher has misspelled. A stemming dictionary throws exactly that material away, so full-text search cannot answer those queries at all.

    A site without it installs and runs normally. The extension is created in a guarded block, so its absence is a warning rather than a failed install; the archive then has full-text search only, and pepsi-list check says which mode the site is in. Installing it later and re-running pepsi-setup run enables it, after which the existing archive needs pepsi-archive reindex to populate the column that the index reads.

    The Debian package Recommends it, so a default apt install pulls it in and a site that hosts no lists can remove it.

  • For real mail flow: public DNS you control for each served domain (to publish DKIM, SPF and MTA-STS records), and the ability to bind the privileged SMTP port 25 (and 465/587 for submission). A development build can run entirely on unprivileged ports and a throwaway database.

  • A DNSSEC-validating resolver, ideally your own on this host. DNS setup is outside Pepsi’s scope: Pepsi does not validate DNSSEC itself but trusts the AD bit of the resolver that it is given (/etc/resolv.conf, [pepsi] DNS_SERVERS, or a stage’s DNS_SERVERS), and DANE and DNSSEC-gated key discovery rest on that bit. Run your own validating recursive resolver on the host, for example unbound (apt install unbound, then nameserver 127.0.0.1 in resolv.conf), or systemd-resolved with DNSSEC=yes. Where that is not possible, name a validating resolver that you trust in [pepsi] DNS_SERVERS as DNS over TLS (tls://192.0.2.53#dns.example.net) or DNS over HTTPS (https://192.0.2.53/dns-query#dns.example.net): the channel is then authenticated by the resolver’s certificate, so nobody on the path can forge the AD bit, and every Pepsi program uses it (see pepsi.conf(5)). A resolver reached over plain DNS across a network is a trust you place in its operator and in the path to it; pepsi-setup names every such resolver by IP address when it runs, and refuses nothing. Either way the resolver’s operator decides what counts as validated. pepsi-status shows, in its “DNSSEC resolver” section, whether the resolver in use validates; the DANE workers log a warning at start when it does not.

4.2. Installing Pepsi

Build the workspace, place the binaries and SQL with make install, and create the system accounts and privileges yourself (the same ones that the package would create; see Users, groups and privileges below).

Additional prerequisites

  • A Rust toolchain with rustc 1.95 or later (the floor is set by the vendored taler-common; ./configure warns about an older one), plus the native build dependencies of the crypto and Kerberos crates: pkg-config, cmake, libclang, nettle, GMP and the Kerberos development headers (on Debian: libclang-dev, nettle-dev, libgmp-dev, libkrb5-dev).

  • The vendored GNU Taler Rust submodule, fetched before the first build:

    git submodule update --init --recursive
    

Building

Pepsi is a Cargo workspace whose root package (pepsi) owns every binary under src/bin/; the pepsi-* member crates are libraries that the thin binaries call into. A development build produces one executable per program:

cargo build --workspace

A release build instead links most programs into a single multi-call binary, pepsi, which selects the program to run from the name under which it is invoked (argv[0]). This is selected by the unibin Cargo feature (which the make build target enables for you); the default multibin feature keeps the per-program binaries used for development and the test suite:

make build   # == cargo build --release --locked \
             #      --no-default-features --features unibin,crypto-nettle

Either way the per-program names are what you run; the release layout installs them as symlinks to the one binary (see Installing). Some programs are not folded in and stay standalone; the authoritative list is the Makefile’s STANDALONE_BINARIES. Two reasons put a program there:

  • it carries its own setuid or setgid bit, which one shared binary cannot do per program, or a site may give it one (pepsi-keys). This applies to the local-delivery, ~/.forward and wallet helpers and their calling stages, the smarthost relay, pepsi-quota, pepsi-whitelist, and pepsi-stage-encrypt and pepsi-stage-decrypt (both setuid pepsi-crypto, so that they can reach the private key material);

  • it is deployed or invoked separately: pepsi-setup, pepsi-config, pepsi-stage-detect-language (its language models are large; that binary is itself multi-call, and a second $PATH symlink, pepsi-detect-language, runs the offline diagnostic tool out of it), pepsi-telemetry (which ships in its own package and runs on a separate host) and pepsi-helper-mailbox-scan (which pepsi-whitelist executes by name).

The programs this chapter refers to are listed below. This is a selection, not an inventory: the authoritative lists are the Makefile’s FOLDED_BINARIES and STANDALONE_BINARIES (BINARIES is derived from those two and is not itself editable).

  • pepsi-ingress — inbound SMTP server

  • pepsi-dispatch — stage dispatcher

  • pepsi-stage-arc, pepsi-stage-srs, pepsi-stage-encrypt, pepsi-stage-decrypt, pepsi-stage-dkim-sign, pepsi-stage-bounce, pepsi-stage-relay-to-internet, pepsi-stage-relay-to-smarthost, pepsi-stage-discard — the stages

  • pepsi-setup — provisioning/bootstrap

  • pepsi-queue — queue inspection/repair

  • pepsi-status — pipeline health summary

  • pepsi-reputation — IP blocklists for inbound mail

  • pepsi-sendmail — local submission (installed as /usr/sbin/sendmail by the Debian package)

The quality gates the project keeps green are:

RUSTFLAGS="-D warnings" cargo build --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check
cargo test --workspace
cargo deny check
cargo vet --locked
make docs

cargo test --workspace needs a reachable PostgreSQL, because the dispatcher-lifecycle crate spawns the real dispatcher against a database. make check is the entry point that gates it: it skips that suite, rather than failing, when no database is usable. It also runs cargo deny check and cargo vet --locked (the dependency audit record, supply-chain/; see The supply chain); see Other make targets below.

Installing

A bare build only produces binaries; Pepsi also ships SQL migrations and a sample configuration that must be placed on disk. The Makefile wraps both. Run it as root, so that it can also apply the privileged-binary bits; otherwise it prints the exact chown/chmod to run by hand:

make install PREFIX=/usr/local SYSCONFDIR=/etc DESTDIR=

Better still, run ./configure first and let it resolve the directory variables into config.mk (which the Makefile reads before its own defaults, so what configure decided wins). Every directory derives from --prefix, so naming a prefix and nothing else relocates the whole installation below it: binaries, SQL, config.d defaults and message templates:

./configure --prefix=/opt/pepsi     # templates at /opt/pepsi/share/pepsi/templates
make && make install

--sysconfdir is the one exception, because a configuration file has to be where the system looks for it: it is /etc when no --prefix was given (and when --prefix=/usr was, since /usr/etc exists on no system), and $(PREFIX)/etc otherwise. Pass --sysconfdir explicitly to override either way.

./configure --help lists everything it accepts. Besides the GNU directory variables (--prefix, --exec-prefix, --bindir, --sbindir, --libexecdir, --sysconfdir, --datarootdir, --datadir, --mandir, --infodir, --docdir) it takes:

--with-templatedir=DIR

Where the message templates go; must match [pepsi] TEMPLATE_DIR. Defaults to $(DATADIR)/templates.

--with-systemdunitdir=DIR / --with-tmpfilesdir=DIR / --with-udevrulesdir=DIR / --with-sendmail-libdir=DIR

The unit files, the tmpfiles.d snippet, the udev rule that records a virtual machine’s generation-ID change (see Cloned and restored virtual machines), and where the legacy /usr/lib/sendmail link goes.

--with-maildir-group=NAME, --with-forward-group=NAME, --with-wallet-group=NAME, --with-token-group=NAME, --with-dispatch-group=NAME, --with-crypto-user=NAME, --with-whitelist-user=NAME

The site names of the accounts and groups that gate the privileged binaries (see Privileged binaries below). They default to pepsi-maildir, pepsi-forward, pepsi-wallets, pepsi-token, pepsi, pepsi-crypto and pepsi-whitelist.

--enable-docs / --disable-docs

Whether the man pages and the info manual are rendered from source or taken pre-built (what a release tarball ships). The default is auto: on when sphinx-build, makeinfo and Python with docutils are all present.

--enable-mta-links / --disable-mta-links

Whether make install claims /usr/sbin/sendmail and the rest of the Debian mail-transport-agent interface. Off by default, so a source install cannot silently displace the system’s existing MTA; the Debian package passes INSTALL_MTA_LINKS=yes because its Conflicts: mail-transport-agent already guarantees exclusivity.

--with-taler-wallet-cli=PATH

The only wallet program the setuid pepsi-helper-auto-pay will run, fixed at build time (default /usr/bin/taler-wallet-cli, where the GNU Taler packages install it). It is never taken from the command line, the environment or the configuration, because the helper’s caller is less privileged than every account the helper can become.

--with-crypto-backend=FEATURE

The cryptographic library behind OpenPGP (Sequoia’s backend), selected by Cargo feature. Exactly one is built in; FEATURE is one of:

crypto-nettle (the default)

Nettle, with its hogweed public-key half and GMP, linked dynamically from the system (Debian: nettle-dev, libgmp-dev). Nettle has no post-quantum algorithms. Only Nettle 3 will do: the bindings do not build against Nettle 4, which dropped nettle/pgp.h, so on a system that ships Nettle 4 (Homebrew, for one) choose crypto-openssl.

crypto-openssl

OpenSSL’s libcrypto, linked dynamically from the system (Debian: libssl-dev). This is the backend to choose for post-quantum OpenPGP: only it, and only with OpenSSL 3.5 or later, can decrypt messages to ML-KEM-768+X25519 keys and handle ML-DSA-65+Ed25519 signatures (see Traditional public-key algorithms broken by quantum computers).

Any other value is refused. The choice affects OpenPGP only: TLS, S/MIME, DKIM and ARC use AWS-LC with either backend, and no program links OpenSSL for TLS (OpenPGP). Without configure, the same choice is make build CRYPTO_BACKEND=crypto-openssl.

--jobs=N

How many compiler jobs cargo runs at once. Cargo on its own runs one job per CPU, whatever the memory, and a machine with many cores and little RAM then runs out of memory near the end of the build. So by default (auto), contrib/build-jobs.sh decides just before each cargo command. It allows one job per CPU, but only as many as the memory that is free at that moment can hold, and it leaves a wide margin for other programs and for future growth of the build. The largest job of the release build needs about 5 GB and each further one up to 2.7 GB, so the release builds with one job when 16 GB are free, with two when 18 GB are free and with four when 32 GB are free; the test build (make check) needs a little less per job. N fixes the number instead. Without configure, CARGO_BUILD_JOBS=N in the environment or on the make command line does the same. make -jN does not raise the number but caps it at N, and the Debian build takes that cap from DEB_BUILD_OPTIONS=parallel=N.

CARGO=, INSTALL=, SPHINXBUILD=, MAKEINFO=, INSTALL_INFO=, RST2MAN= and PYTHON= are accepted as VAR=VALUE arguments or from the environment. Options that this build has no use for (--libdir, --localstatedir, --build, --host, …) are accepted and ignored, so a generic packaging driver can call the script unchanged; only a missing cargo is a hard error.

make install performs the following steps:

install-bin

Builds the release (unified) binary and installs it to $(DESTDIR)$(PREFIX)/libexec/pepsi/pepsi, then creates one $(DESTDIR)$(PREFIX)/bin symlink per folded program name pointing at it (e.g. pepsi-ingress -> ../libexec/pepsi/pepsi). The seventeen standalone programs named in the Makefile’s STANDALONE_BINARIES are installed as their own $(DESTDIR)$(PREFIX)/bin executables. Because each symlink still lives in $(PREFIX)/bin (on $PATH), the runtime DATADIR derivation below is unaffected.

install-data

Copies the SQL files from pepsi-setup/db/ to $(DESTDIR)$(DATADIR)/sql (where DATADIR = $(PREFIX)/share/pepsi). pepsi-setup reads these when installing the schema; the default SQL_DIR is ${DATADIR}/sql. Also installs contrib/hosters.txt as $(DATADIR)/hosters.txt, the list of public hosters for which pepsi-whitelist import --auto-wildcard never proposes a wildcard. (The list of known compromised keys, against which every peer key is checked, is not installed: each deployment fetches it into its database, see pepsi-keys blocklist update.)

install-templates

Copies the message templates (<name>.<lang>.body — the anti-spam payment request, the bounce bodies, the mailing-list notices and the other automatic replies) to $(DESTDIR)$(TEMPLATEDIR), which defaults to $(DATADIR)/templates. The binaries’ [pepsi] TEMPLATE_DIR default is ${DATADIR}/templates and resolves through the same runtime derivation, so the two ends agree under any prefix without the configuration naming a path.

install-config

Installs the reference pepsi.conf.sample under $(DESTDIR)$(SYSCONFDIR)/pepsi/. It never creates pepsi.conf: the sample wires up optional stages that need further setup and is not a working configuration, so the live file comes from pepsi-setup --wizard (an existing one is never touched).

install-config-d

Copies contrib/config.d/*.conf to $(DESTDIR)$(DATADIR)/config.d. These are packaged defaults, not conffiles, and they are parsed before pepsi.conf — see Configuration. One ships with Pepsi: the per-language vacation notice texts. If that directory holds a thunderbird.conf, the target deletes it: every file there is parsed, and that file sets CRYPTO_ALLOW_DOWNGRADE = yes for every peer, where the S/MIME capability record decides per peer.

install-man / install-info

Installs the rendered roff man pages under $(DESTDIR)$(MANDIR)/manN and pepsi.info under $(DESTDIR)$(INFODIR) (registered with install-info). With BUILD_DOCS=no — what ./configure --disable-docs sets, and what a release tarball uses — pre-built pages are installed instead of being rendered, and each step is skipped with a message if none is present.

install-doc

Installs README.md, NEWS, SECURITY.md, AUTHORS, COPYING and the licence texts from LICENSES/ under $(DESTDIR)$(DOCDIR), which defaults to $(DATAROOTDIR)/doc/pepsi.

install-systemd

Installs the systemd units from debian/systemd/ into $(DESTDIR)$(SYSTEMDUNITDIR) (rewriting /usr/bin and /etc/pepsi to the configured BINDIR/SYSCONFDIR) plus the tmpfiles.d snippet that gives /run/pepsi its owner and ACL. That snippet is the only source of the directory’s permissions, since no unit may declare it as a RuntimeDirectory (see No unit may claim it as a RuntimeDirectory). Everything ships disabled. make install INSTALL_ADMIN_UNITS=no omits the two pepsi-setup-apply units, which is the source-install equivalent of not installing the pepsi-httpd-admin package.

In addition, install depends on the privileged-bit targets (install-helper, install-maildir-stage, install-forward-*, install-auto-pay-*, install-smarthost-stage, install-whitelist-suid, install-crypto-suid, install-quota-tool) described under Privileged binaries below; each prints the chown/ chmod to run by hand when it is not run as root, or when the group it needs does not exist.

Override PREFIX, SYSCONFDIR and DESTDIR to suit the target system. make uninstall removes everything that make install placed (always including both pepsi-setup-apply units, whatever INSTALL_ADMIN_UNITS says) but leaves the live pepsi.conf in place.

Other make targets

make check

The test entry point that needs no remote accounts. It runs the file-inspection cross-checks of the systemd units, the telemetry socket path, the version numbers and the two mail-client extensions, and the workspace tests (minus the dispatcher-lifecycle crate, which needs PostgreSQL). It then runs, best effort, the lifecycle suite against a local pepsicheck database, the setuid-helper contract tests, the authentication interoperability script, cargo deny check, cargo vet --locked and the manual’s cross-reference check; each of these is skipped rather than failed when its prerequisite is absent.

make integrationtests / make benchmarks

The live-pipeline scripts under tests/, which need passwordless SSH to the hosts named in ACCOUNTS (default tests/test-accounts.ini, created from tests/test-accounts.ini.sample); integrationtests first runs the Thunderbird interoperability gate. The benchmarks are a separate target because they deliberately load the MTA.

make docs

The manual for the source language. It is an alias for make docs-en, which builds its HTML and its PDF, plus the man pages and the info manual (both source-language only). make docs-<lang> builds one other language and make docs-all builds every language in DOC_LANGUAGES (en de fr); make html builds every language’s HTML alone.

make dist / make distcheck

Build the release tarball pepsi-$(VERSION).tar.gz (every tracked file including the vendored submodule, plus configure and pre-built documentation), and verify it by unpacking it into a clean tree and running configure, make, make check, make install and make uninstall against a throwaway prefix. The tarball is copied from the working tree, so make dist refuses to run while a tracked file differs from HEAD or the submodule is not at the commit that HEAD records (untracked files do not count); make dist DIST_ALLOW_DIRTY=yes builds a development tarball anyway. The tarball is reproducible: entries sorted, every mtime the commit’s date (SOURCE_DATE_EPOCH), numeric uid 0, no gzip timestamp.

make dist-sign

Sign an existing release tarball: a detached, ASCII-armoured OpenPGP signature pepsi-$(VERSION).tar.gz.asc with gpg’s default key or DIST_SIGN_KEYID, verified before the target succeeds. See Verifying a download for the user’s side and Making a release for the keys.

make deb

Build the Debian packages into target/debian/ (Debian packages), unsigned; make deb DEB_SIGN_KEYID=<fingerprint> signs the .changes and .buildinfo files with that key.

make clean / make distclean

clean runs cargo clean and removes the rendered documentation; distclean additionally removes config.mk, config.status and anything that make dist generated.

Note

DATADIR must match what the binaries compute at runtime: taler-common derives ${DATADIR} as <install-prefix>/share/pepsi from the binary’s own location. When running from a source checkout (not installed), set [pepsi-postgres] SQL_DIR explicitly so pepsi-setup finds the migrations.

Preparing the system

The Debian package’s postinst does the steps below automatically; on a source install you do them yourself, using the same account, group, mode and directory layout documented under Users, groups and privileges and Directories and secrets below. Create the accounts and groups before make install (so it can apply the privileged bits), and the database roles before pepsi-setup (which installs the schema):

  1. Create the system accounts, each --system with /usr/sbin/nologin and home /var/pepsi and a private group of the same name, plus the gating groups. debian/pepsi.postinst is the authoritative list; this loop is it, translated:

    for u in pepsi pepsi-ingress pepsi-httpd pepsi-owner \
             pepsi-helper-token-refresh pepsi-whitelist pepsi-crypto \
             pepsi-keydisc pepsi-config pepsi-greylist; do
        addgroup --system "$u"
        adduser --system --home /var/pepsi --no-create-home \
                --ingroup "$u" --shell /usr/sbin/nologin \
                --disabled-password "$u"
    done
    for g in pepsi-maildir pepsi-forward pepsi-token pepsi-admin \
             pepsi-telemetry; do
        addgroup --system "$g"
    done
    # The shared-wallet account owns its wallet databases, so unlike the
    # service accounts it gets a real home.
    addgroup --system pepsi-wallets
    adduser --system --home /var/lib/pepsi-wallets --ingroup pepsi-wallets \
            --shell /usr/sbin/nologin --disabled-password pepsi-wallets
    install -d -o pepsi-wallets -g pepsi-wallets -m 0700 /var/lib/pepsi-wallets
    # Memberships: telemetry producers, and the SRS secret fragment.
    for u in pepsi pepsi-ingress pepsi-httpd pepsi-keydisc pepsi-greylist; do
        adduser "$u" pepsi-telemetry
    done
    adduser pepsi pepsi-ingress
    # The greylisting milter's socket belongs to group pepsi-greylist.
    adduser pepsi-ingress pepsi-greylist
    # The DKIM signing daemon: its own home holds the signing keys, and
    # membership of pepsi lets it hand its socket to the stage workers.
    addgroup --system pepsi-dkim
    adduser --system --home /var/lib/pepsi-dkim --no-create-home \
            --ingroup pepsi-dkim --shell /usr/sbin/nologin \
            --disabled-password pepsi-dkim
    adduser pepsi-dkim pepsi
    install -d -o pepsi-dkim -g pepsi-dkim -m 0700 /var/lib/pepsi-dkim
    install -d -o pepsi-dkim -g pepsi-dkim -m 0755 /var/lib/pepsi-dkim-public
    

    Each of the last five accounts exists because something refuses to work without it: pepsi-whitelist is the account that the whitelist CLI is setuid to, pepsi-crypto is the only database role granted crypto_identity.private_wrapped and the owner of the key-encryption fragment, pepsi-keydisc is the narrowest role in the deployment (it parses key material fetched from the open internet), and pepsi-config is the only role that may write the configuration overlay, and pepsi-greylist runs the greylisting milter, whose socket’s group it is. If you skip pepsi-crypto, install-crypto-suid merely warns, and the two crypto stages are left unprivileged and unable to open a single private key.

  2. Create the ten PostgreSQL login roles and the database (peer authentication, as the postgres superuser). This is the same set that debian/pepsi.postinst creates: every account that authenticates as itself over the peer-authentication socket. pepsi-telemetry is among them even on a node that serves no telemetry, because pepsi-setup run grants rights to it on every host, and pepsi-setup does its database work as pepsi-owner, which cannot create a missing role:

    for r in pepsi-owner pepsi pepsi-ingress pepsi-httpd \
             pepsi-whitelist pepsi-config pepsi-crypto pepsi-keydisc \
             pepsi-telemetry pepsi-greylist; do
        createuser "$r"          # createuser takes one role name at a time
    done
    createdb -O pepsi-owner pepsi
    
  3. Run make install as root (above) so that the setuid/setgid bits are applied; otherwise apply them by hand per Privileged binaries below. Each install-* target only warns when the group it needs is missing, so an account or group skipped above leaves the corresponding program silently unprivileged.

  4. Create the runtime directories with the ownership and modes listed under Directories and secrets. Create the smarthost-secret directories only if you configure a smarthost that uses them.

The package is the simpler path: it ships the binaries (installed to /usr/bin), the SQL schema, the message templates, the man pages, the GNU info manual and socket-activated systemd units, and its postinst provisions the whole system for you.

Building the package

From a checkout (with the vendor/taler-rust submodule initialised) and the build dependencies listed in debian/control installed, run:

make deb

It copies the working tree (including uncommitted changes and new files that git does not ignore) into target/debian/pepsi-<version>/ and runs dpkg-buildpackage -b -us -uc there (override with DEB_BUILD_FLAGS, e.g. DEB_BUILD_FLAGS= to build and sign the source package too). That compiles the release build and writes the .deb files to target/debian/ (DEB_BUILDDIR chooses another directory): pepsi, pepsi-httpd-admin, pepsi-telemetry and pepsi-stage-detect-language. dpkg-buildpackage always writes its results next to the source directory, which is why the build does not run in the checkout itself.

Installing the package

Install the .deb with apt so that its dependencies (including postgresql) are pulled in:

apt install ./pepsi_*.deb

On configure the postinst automatically:

  • creates the eleven service accounts and the six gating groups (pepsi-maildir, pepsi-forward, pepsi-token, pepsi-admin, pepsi-telemetry and pepsi-wallets), with the memberships between them; these are the same identities that the source install creates by hand;

  • creates the /var/pepsi runtime directories (keys, tokens, token-refresh, tls, krb5) and /etc/pepsi/secrets and /etc/pepsi/secrets.d with the ownership and modes documented under Directories and secrets below;

  • applies the privileged-binary bits via dpkg-statoverride: to the three setuid-root helpers, to the setgid programs that may exec them (the two delivery stages, the wallet stage and pepsi-quota), to the setgid smarthost relay, to the setuid pepsi-whitelist and to the two setuid pepsi-crypto stages (see Privileged binaries below); and

  • best-effort creates the ten PostgreSQL login roles (pepsi-owner, pepsi, pepsi-ingress, pepsi-httpd, pepsi-whitelist, pepsi-config, pepsi-crypto, pepsi-keydisc, pepsi-telemetry, pepsi-greylist) and the pepsi database (owned by pepsi-owner) when a local cluster is reachable, and prints the roles to create by hand if it is not.

The service is shipped disabled: the package installs but does not start Pepsi, leaving the site-specific bootstrap to you.

Note

pepsi is one of four binary packages built from this source: the language-detection stage and the telemetry collector are split out for size and deployment reasons. pepsi-httpd-admin, which pepsi Recommends and apt therefore installs by default, carries the only mechanism by which the browser console can cause a privileged change on this host. Removing that one package is a supported hardening step for a deployment administered from a terminal. Debian packages describes each package and that lever in full.

After the package is installed

On first installation the package copies a placeholder configuration to /etc/pepsi/pepsi.conf; later upgrades never touch that file (it is not a dpkg conffile, so an upgrade never stops to ask about it). Edit it, then run the bootstrap exactly as in the shared steps below, for example:

# 1. edit /etc/pepsi/pepsi.conf (HOSTNAME, ACCEPTED_DOMAINS, domains, TLS)
# 2. install the schema, have pepsi-dkim make the DKIM keys and
#    print the DNS records:
pepsi-setup --no-certbot run        # or:  pepsi-setup --wizard
# 3. publish the printed DNS records, then start the pipeline:
systemctl enable --now pepsi.target

Because the units are socket-activated, pepsi.target brings up pepsi-ingress, pepsi-httpd, pepsi-dkim and pepsi-dispatch together; you do not invoke the serve subcommands by hand.

macOS is not yet a supported deployment target: the package has not been installed as root on a test system yet, and docs/ROADMAP.md lists what still differs from Linux. The package installs the programs under /usr/local (the system volume’s /usr is read-only) and the configuration in /etc/pepsi, as on Linux. It installs launchd jobs in /Library/LaunchDaemons (net.taler.pepsi.*) instead of systemd units, one job for each unit, and its scripts do what the Debian postinst does.

Building the package

On a Mac with the Xcode command line tools, a Rust toolchain and the native libraries listed under From source (Nettle 3, not 4, or CRYPTO_BACKEND=crypto-openssl), run:

make macos-pkg

This compiles the release build, stages it with the macOS layout and writes target/macos/pepsi-<version>-<arch>.pkg with Apple’s pkgbuild and productbuild. To sign the package, name a “Developer ID Installer” certificate from your keychain in MACOS_PKG_SIGN_IDENTITY; an unsigned package installs with:

sudo installer -pkg pepsi-<version>-<arch>.pkg -target /

What the installation does

  • It creates the same service accounts, groups and memberships as the Debian package, as hidden system accounts with ids from 400 up.

  • It creates the directories under /var/pepsi, /var/lib and /etc/pepsi with the same owners and modes.

  • It sets the same setuid and setgid bits on the privileged programs.

  • It adds /run to /etc/synthetic.conf, which makes /run a link to /private/var/run. If /run does not appear at once, it appears after the next reboot.

  • On an upgrade, it stops the jobs that are running, upgrades the database schema and starts the same jobs again.

It does not create the PostgreSQL roles and the database, because macOS has no system PostgreSQL. Create them as described under From source, in a cluster that uses peer authentication for local connections.

After the package is installed

sudo pepsi-setup --wizard
# publish the printed DNS records, then start the pipeline:
sudo /usr/local/libexec/pepsi/pepsi-launchd enable

pepsi-launchd stands in for systemctl and pepsi.target. Without a job name, enable, disable, start, stop, restart and status act on the jobs that pepsi.target pulls in; the optional telemetry-client and helper-token-refresh jobs are named explicitly (pepsi-launchd enable telemetry-client). Each job writes its log to /var/log/pepsi/<job>.log. To remove the package, run sudo /usr/local/libexec/pepsi/pepsi-launchd uninstall. Like apt remove, this keeps the configuration, the accounts, the database and the data under /var.

Three things work differently from Linux:

  • Without socket activation, pepsi-ingress and pepsi-httpd bind their ports themselves (SERVE = tcp, which the wizard writes on a host without systemd). Both start as root, read their TLS certificates and keys, and then drop to their accounts. pepsi-setup restarts their launchd jobs when their TLS files change, and the certbot deploy hook restarts them after a renewal.

  • /etc/pepsi/postgres.env is read only by the jobs that run as root. Give each account its own credentials through the {role} placeholder in [pepsi-postgres] CONFIG instead.

  • The setup applier (pepsi-httpd-admin on Debian) has no launchd job, so the browser console’s configuration pages are read-only; administer the deployment with pepsi-setup in a terminal.

4.3. Verifying a download

Release tarballs are signed with the release manager’s OpenPGP key: Christian Grothoff’s or Florian Dold’s, the same keys that sign GNU Taler’s releases on ftp.gnu.org. The signature is the detached .asc file published next to the tarball; verify it before unpacking:

gpg --verify pepsi-X.Y.Z.tar.gz.asc pepsi-X.Y.Z.tar.gz

gpg must hold the signing key, and the check is worth exactly as much as your reason to believe that the key is the release manager’s. Christian Grothoff’s key is published at https://grothoff.org/christian/grothoff.asc with the fingerprint D842 3BCB 326C 7907 0339 29C7 939E 6BE1 E29F C3CC (the same key that Reporting a vulnerability names); both keys are in the GNU keyring, https://ftp.gnu.org/gnu/gnu-keyring.gpg, which gpg --keyring gnu-keyring.gpg --verify … uses without importing. A good signature says that the tarball is the one that the release manager signed; whether it is also what the source produces is the subject of Reproducible builds.

Debian packages on deb.taler.net are signed with the taler-systems release key, which signs every package there, and apt verifies them against the keyring that you install once:

wget -O /etc/apt/keyrings/taler-systems.gpg https://taler.net/taler-systems.gpg.key

named by signed-by= in the repository’s sources entry. A package built from the source tree with make deb DEB_SIGN_KEYID=… carries a signed .changes file that dscverify and debsign --verify check; a package built without a key is unsigned, which is what a local build is.

4.4. Reproducible builds

The release build is reproducible: the same commit, built with the pinned toolchain from the same Cargo.lock, gives byte-identical binaries and the same tarball wherever and whenever it is built. A published binary can therefore be checked against the source by rebuilding it, and this is the one check that catches a compromised build host or release manager’s machine. The CI job 5-reproducible performs that check on every run, building twice from two clones at different paths with different crate caches (Continuous integration).

What the recipe pins, and where:

  • the toolchain — rust-toolchain.toml names the exact rustc version; rustup uses (and installs) that one in this tree. A toolchain not managed by rustup ignores the file, and a different compiler gives a different binary, so use rustup to reproduce a release;

  • the crates — Cargo.lock with --locked, which every make target passes;

  • the paths — make build maps the checkout, the target directory (CARGO_TARGET_DIR, or target/ in the checkout) and the crate cache (CARGO_HOME, default ~/.cargo) to the fixed names /build, /target and /cargo, each under its own and its canonical path, for rustc (--remap-path-prefix) and for the C and C++ compilers that the native crates run (-ffile-prefix-map). So neither the location of the tree nor that of the crate cache reaches the binary. strip = true removes the symbol tables. A symbolic link inside the crate cache is not covered: keep it a real tree;

  • the time — SOURCE_DATE_EPOCH is exported as the commit’s date and is what the tarball’s mtimes and the documentation’s dates use; pass SOURCE_DATE_EPOCH=<seconds> to override it in a tree that is not a git checkout;

  • the tarball — make dist writes it with tar --sort=name --mtime=@$SOURCE_DATE_EPOCH --owner=0 --group=0 --numeric-owner and gzip -n.

To reproduce a release, check out its tag with the submodule, build, and compare:

git clone --recurse-submodules https://git.taler.net/pepsi.git
cd pepsi && git checkout vX.Y.Z
make build            # the multi-call binary and the standalone programs
make dist             # the tarball
sha256sum target/release/pepsi target/release/pepsi-setup pepsi-X.Y.Z.tar.gz

Then compare the sums with those of the published artefacts. The binaries inside a .deb (unpacked with dpkg-deb -x) are the files that make build produces, but built with the toolchain of the package build, which is Debian’s own rustc and ignores rust-toolchain.toml. So reproduce a package with the rustc that its .buildinfo file names. A different RUSTFLAGS or CFLAGS in the environment, a different crate cache content (a crate whose checksum differs fails --locked anyway), or another toolchain gives another binary; diffoscope shows where two builds differ.

4.5. Migrating from an existing mail server

If this host already runs Postfix, Exim, Sendmail, qmail or Stalwart, you do not have to re-derive its configuration by hand. With no pepsi.conf present, pepsi-setup --wizard detects what is installed and offers to import it: the hostname, accepted domains, smarthost (with the relay credentials read from that server’s own password file), local-delivery method, submission SASL socket, message size limit, queue lifetime and TLS certificate paths all arrive as the interview’s pre-filled answers, which you then confirm or correct one by one. The old server’s trusted networks are listed in the migration report instead: Pepsi lets a network relay only as the addresses that [pepsi-ingress] SENDER_MAP binds it to, so each network needs a binding that you write by hand.

To see what a migration would produce before running the wizard, run the following; it writes nothing:

pepsi-setup import

Routing tables are converted too. /etc/aliases, /etc/postfix/virtual, Sendmail’s virtusertable, qmail’s .qmail-* files and their equivalents become a Pepsi alias map beside the configuration (see pepsi-stage-aliases), with bare alias names qualified in the style you choose. Since Pepsi alias keys must carry a domain, postmaster: root becomes postmaster@example.org  root@example.org for each accepted domain by default.

Everything that does not carry over is written to import-report.txt beside the configuration, split into what failed to parse, what Pepsi has no equivalent for (header_checks, an alias delivering to a command), what was approximated (mbox delivery becoming Maildir) and what the importer did not recognise. An import never aborts setup: an unreadable file or an unknown directive produces a report entry, not a failure.

Two things are deliberately not migrated: queued mail (let the old server’s queue drain before switching the MX over) and DKIM keys (pepsi-setup run generates fresh ones and prints the records to publish). If the old MTA still holds port 25 when the wizard runs, the report says so; stop and disable it before you start pepsi-ingress.

See pepsi-setup for the per-server detail, the --alias-style choices and the --expert options that expose the settings that the interview does not otherwise ask about.

4.5.1. Mail filters already on the host

Independently of any migration, the wizard scans the host for milter daemons (clamav-milter, rspamd, spamass-milter, milter-greylist, milter-regex, mimedefang, amavisd-milter) and offers each one that it finds. Finding one means three things in turn: the package is installed, a socket can be read from its own configuration (or its unit, or the paths that its distribution ships), and a real milter option negotiation completes against that socket. Nothing is offered on the strength of an installed package alone, and the negotiation is also how the generated ALLOW_ACTIONS comes to grant exactly what that filter asks for rather than a guess.

pepsi-ingress consults every accepted filter during the SMTP session, after its own checks, as an MTA would: a filter that refuses mail answers the sending server with its own 4xx or 5xx, so nothing that was accepted has to be bounced later. The filters are consulted in an order that depends on what they do: policy and greylisting, then virus, then spam, then general-purpose frameworks. Milters that an MTA import found (Postfix smtpd_milters, sendmail INPUT_MAIL_FILTER) are carried over the same way, with the failure action, timeouts and macros that the old MTA used. The wizard also writes [stage-quarantine], which holds a message that a filter asks to quarantine until you release it. See Milters in the SMTP session for the generated sections and pepsi-ingress for how the session uses them.

Two kinds of filter run as pipeline stages of pepsi-stage-milter instead, after the message was accepted: a filter that ran on locally submitted mail (Postfix non_smtpd_milters), and, with --expert, a filter that you place after decryption so that it sees plaintext. Filters whose job Pepsi already does (opendkim, openarc, opendmarc, SPF daemons, postsrsd) are named and skipped rather than offered.

The wizard also asks whether to greylist with Pepsi’s own greylisting milter, pepsi-greylist. The answer defaults to no, because greylisting delays the first message from every new correspondent. pepsi-setup check negotiates with every configured milter and reports one that does not answer, or that asks for an action that its ALLOW_ACTIONS does not grant.

4.6. Provisioning the database, keys and DNS

With a configuration file in place (see Configuration), run the bootstrap tool once:

pepsi-setup -c /etc/pepsi/pepsi.conf run > pepsi-dns.zone

This validates the whole configuration, installs the pepsi schema, has the signing daemon pepsi-dkim generate the DKIM keys for each domain (RSA and Ed25519, for every domain that has none), and prints the DNS records to publish. These are the DKIM public keys, an SPF policy from the configured PUBLIC_IP addresses, an MTA-STS record and, for each TLS ingress listener, a DANE TLSA record. That record pins the mail certificate’s public key, which is why certbot certificates are obtained with --reuse-key (see Changing the key behind a TLSA record). The mail listeners and pepsi-httpd’s HTTPS listener get separate certbot lineages, and so separate keys: /etc/letsencrypt/live/<HOSTNAME>/ for mail and /etc/letsencrypt/live/<HOSTNAME>-web/ for the web server. The web server therefore never holds the key that the TLSA record pins. (An installation whose HTTPS listener still uses the mail certificate is moved to its own at the next pepsi-setup run, once that certificate has been obtained.) The operation is idempotent and may be re-run at any time; upgrading the schema to a new release is pepsi-setup schema (see Upgrading). run --reset drops and recreates the schema; this destroys all queued mail but keeps the DKIM keys, which are not in the database.

Publish the printed DNS records, then verify what is live against what Pepsi expects:

pepsi-setup -c /etc/pepsi/pepsi.conf check

See pepsi-setup for the full provisioning workflow.

4.7. Setting up in a browser

Everything above can also be done from a browser, against the same code. The terminal path is the shortest route on a machine where you have a shell; the browser path is for machines where you do not, and for operators who would rather see the DNS records as a red/green checklist than as a zone file.

4.7.1. The split, and why

Setup is privileged: it writes /etc/pepsi/pepsi.conf, hands each secrets.d fragment to the single account that reads it, runs certbot, creates database roles and generates key material — all as root. pepsi-httpd, which serves the interface, drops privileges before it accepts a connection and must never regain them.

So it does not act. It writes an intent row that says what should be true. A separate root program, pepsi-setup apply, decides how, does it, and records what happened; it is started on demand and exits again when it has nothing to do. Root is never on the network. Read “The applier’s trust model” in :doc:`programs/pepsi-setup` before enabling this: it states exactly what the applier will and will not do, including the things that it deliberately cannot do (there is no restart or shutdown task, and install-schema refuses reset).

4.7.2. Enabling it

The Debian packages ship a minimal but complete configuration, an administrative listener on a UNIX socket, and the applier’s socket unit. The socket unit is in pepsi-httpd-admin, which pepsi Recommends and apt therefore installs by default (see Debian packages). Two commands enable it:

# systemctl enable --now pepsi-httpd.socket pepsi-setup-apply.socket
# pepsi-setup -c /etc/pepsi/pepsi.conf bootstrap

The second prints a one-time token and the curl command that turns it into the first administrator account. On the machine itself you can skip it entirely: a member of the pepsi-admin group (and root) is identified by SO_PEERCRED over /run/pepsi/admin.sock and needs no credential at all.

One more setting is needed before anything privileged can be asked for: [pepsi-admin] CONFIG_DB must name a connection that authenticates as the pepsi-config database role. That is the boundary: only that role may enqueue setup work, and pepsi-httpd’s own account is not it, so a compromise of the web tier is not a compromise of the pipeline definition. Until it is set the setup endpoints answer 503 and say so.

4.7.3. Driving it

The interview is data. GET /api/v1/setup/questions returns the ordered steps and, for each question, its shape, default, help text and the condition under which it is asked. This is the same model that pepsi-setup questions prints and that describes the terminal wizard; the test suite asserts that they are equal. Answers are staged with PUT /api/v1/setup/answers into draft rows that no running process can see, so an interrupted session cannot half-configure anything.

POST /api/v1/setup/tasks then asks for the privileged half: writing the configuration, obtaining certificates, installing the schema and generating keys. GET /api/v1/setup/tasks/{id}?since=<seq> streams what the applier is doing, line by line.

4.7.4. What will always need a text editor

Some settings are read from the configuration file only, and no interface will change that: [pepsi], [pepsi-postgres], [paths], and the HTTP(S) and ingress listener sections. They have to work before a database connection exists, or they are a security boundary: if the database could move a listening socket or relax the crypto policy, a database compromise would become a compromise of the MTA’s own TLS and ports.

So adding a submission port, or moving a listener, is a text-editor-and-restart operation. So is any change the console marks restart rather than hot: there is no restart button, because a button that restarts a service on request would hand the web tier’s privileges to whoever reaches it.

4.7.5. Scripting it

The same answer model drives an unattended install:

pepsi-setup --wizard --answers answers.json -c /etc/pepsi/pepsi.conf

pepsi-setup questions prints the schema against which that file is written. This is not a third implementation of the interview: it runs the interactive one with every prompt answered from the file, so the branches and the per-field validation are the same ones that a human sees.

4.8. Running the services

These long-lived processes run continuously (pepsi.target starts them all under the shipped units):

  • pepsi-ingress serve — accepts mail (typically under a service manager, optionally with systemd socket activation for the privileged ports).

  • pepsi-dispatch serve — runs exactly one instance per system; it is the only process that starts stage programs.

  • pepsi-dkim serve — holds the DKIM and ARC private keys and signs for the signing stages; the dispatcher’s unit pulls it in.

  • pepsi-httpd serve — the MTA-STS policy, the Web Key Directory, the secure-link portal and, on administrative listeners, the API and console.

  • pepsi-keydisc serve METHOD — one instance per key-discovery method (pepsi-keydisc@dane, @wkd-advanced, @wkd-direct and @vks by default), answering the crypto stages’ key requests.

One more runs only where greylisting is turned on, from its own unit outside pepsi.target:

  • pepsi-greylist serve — the greylisting milter that pepsi-ingress (or a Postfix in front of it) asks at RCPT.

The timers that pepsi.target also wants (failure bouncing, queue and log garbage collection, key refresh and retirement, quota reconciliation, the mailing-list tasks and digests) run one-shot jobs; systemctl list-timers 'pepsi-*' lists them.

The stage programs themselves are spawned by the dispatcher as persistent PROGRAM worker processes that read message ids on standard input; you do not run them by hand in production (for debugging you can pipe an id to worker, e.g. echo 42 | pepsi-stage-srs -c /etc/pepsi/pepsi.conf worker).

4.9. Upgrading

From the first release, 0.0.0, on, every release upgrades the database of any earlier release in place: the queue, the key store, the settings and everything else in it are kept, and mail waiting in the queue is delivered by the new release. Downgrades are not supported (see below).

The procedure:

  1. Read NEWS for the release. It lists configuration options that changed, and anything else that needs doing by hand.

  2. Take a backup (Backup and restore), or at least enable the automatic dump described below.

  3. Install the new release. The Debian packages do the rest: they upgrade the schema and restart the services that were running. From source: make install, then pepsi-setup -c /etc/pepsi/pepsi.conf schema and systemctl try-restart pepsi.target.

  4. Check with pepsi-status that the queue drains, and look at the journal for warnings (Logs).

A release may also refuse a configuration that an earlier one accepted, when the earlier behaviour was a security weakness (NEWS says which); the services then do not start. Run pepsi-setup -c /etc/pepsi/pepsi.conf validate before restarting them: it applies the same checks and names what to change.

The rest of this section explains what happens during step 3.

Every Pepsi program checks, as it connects to the database, that the schema was built from exactly the SQL files that the program itself was built with. The installer records the SHA-256 and release of each file that it applies in pepsi.schema_file. A program that finds anything else (an older schema, a newer one, or one built from different files) stops at once with exit status 78 and a message naming the fix, rather than failing at its first missing column in the middle of a message. The dispatcher treats a stage worker that stops this way as “cannot run yet”, not as a crash: the message it was handed is put back in the queue, never failed.

Upgrading the schema is its own step:

pepsi-setup -c /etc/pepsi/pepsi.conf schema

It installs the patches that this release adds, replaces the stored functions and re-applies the role grants. It does nothing else (no certificates, keys or DNS), so it needs only the [pepsi-postgres] section. On a telemetry collector, give it the collector’s file, /etc/pepsi-telemetry/pepsi-telemetry.conf.

Debian packages run it automatically when they are upgraded, before the services are restarted; a new installation is left to pepsi-setup run. If it fails (the database is down, say), the package still installs, a notice says so, and the services refuse to start. systemd retries them until the schema matches, backing off to once a minute, without a start limit. Once the cause is fixed, running the command above is all it takes; the services come back by themselves.

From source, after make install:

pepsi-setup -c /etc/pepsi/pepsi.conf schema
systemctl try-restart pepsi.target

What the refusals mean:

The message says

What to do

no pepsi schema

A new installation: pepsi-setup run (or pepsi-setup schema on a collector host).

older than this program

pepsi-setup -c FILE schema.

newer than this program

A downgrade, which is not supported. Install the newer release again, or restore the database backup taken before the upgrade (below).

built from a different copy of a patch

A released patch never changes, so this is a bug; please report it. The one exception is a database built from a development snapshot (a Git checkout between releases), where the newest patch is still being edited: drain it and re-create it with pepsi-setup run --reset.

stored functions from a different build

Development builds only: pepsi-setup -c FILE schema.

4.9.1. Upgrading from 0.0.0: the DKIM keys

Release 0.0.0 keeps its DKIM and ARC keys as files under [pepsi] KEY_DIR, readable by the pipeline. Later releases hold them in the signing daemon pepsi-dkim, under its own account and in its own directory. The daemon does not take over the 0.0.0 key files: it generates fresh keys under new selectors (pepsi-YYYYMM and pepsi-YYYYMM-ed25519) and signs with them at once. It signs only for the domains that pepsi-setup run hands it; until that has happened, outbound mail waits at the signing stage and is retried, so run it right after installing:

  1. Install the new release as above.

  2. Run pepsi-setup -c /etc/pepsi/pepsi.conf run. It hands the signing domains to pepsi-dkim, which makes the keys, deletes the old key files under KEY_DIR and prints the new DKIM records, with the old pepsi and pepsi-ed25519 records as removable comments.

  3. Publish the new records. Until they are published, receivers cannot verify the signatures and DMARC rests on SPF alone; pepsi-setup check, pepsi-dkim status and the daemon’s mail to the postmaster say what is still missing.

  4. Remove the old pepsi._domainkey and pepsi-ed25519._domainkey records once mail signed with the old keys has been delivered, a week after the upgrade at most.

Leave /var/lib/pepsi-dkim out of backups from then on (Backup and restore). Rotation stays off unless you turn it on (Rotating DKIM keys).

4.9.2. Downgrades

Not supported. pepsi-setup refuses to install an older release’s SQL over a newer schema (exit status 78, nothing changed), and the older release’s programs refuse to run on it. Going back to an older release means restoring a database backup taken before the upgrade, then installing the older packages. No Pepsi interface can install an older release, so a rollback to a vulnerable version needs root acting deliberately (see Signed firmware updates accepted without version check).

4.9.3. Backing up before a schema upgrade

No backup is taken unless you enable it. When enabled, the backup is taken only when an upgrade actually changes the schema: pg_dump saves the pepsi schema (with the extensions it uses) as pepsi-<previous version>-<UTC time>.dump, readable by root only. If the dump fails, the schema is not upgraded. After each new dump, only the two newest are kept in the directory (pepsi-setup schema --backup-keep; 0 keeps them all): a dump holds the whole queue, including the plaintext of decrypted mail waiting in it, so old dumps are copies of mail that would otherwise have left the host long ago. Copy a dump elsewhere if it has to outlive two upgrades.

With the Debian packages it is a debconf setting, asked only at priority low:

dpkg-reconfigure -plow pepsi

Answer yes, and confirm or change the directory (default /var/backups/pepsi). Unattended, or before the first upgrade:

echo "pepsi pepsi/schema-backup boolean true" | debconf-set-selections
echo "pepsi pepsi/schema-backup-dir string /var/backups/pepsi" | debconf-set-selections

On a telemetry collector host the questions belong to pepsi-telemetry (pepsi-telemetry/schema-backup, pepsi-telemetry/schema-backup-dir). When both packages are installed and use the same database, whichever upgrades it first takes the backup.

From source, pass the directory to the upgrade step:

pepsi-setup -c /etc/pepsi/pepsi.conf schema --backup-dir /var/backups/pepsi

To restore one, stop the services, re-create the database empty and owned by the schema owner, and restore into it as that owner:

systemctl stop pepsi.target
sudo -u postgres dropdb pepsi
sudo -u postgres createdb -O pepsi-owner pepsi
sudo -u pepsi-owner pg_restore -d pepsi /var/backups/pepsi/pepsi-0.0.0-20261001T000000Z.dump

Then install the release that the backup came from and start the services again. Everything that the database learnt after the backup is lost, such as mail accepted since then and settings changed since then. The stored private keys in the dump are encrypted under the key-encryption key in secrets.d, which is not in the database: keep that file, and back it up separately (see Backup and restore). The keys are sealed with an AEAD, so a modified blob in a dump never opens (see Unauthenticated private-key encryption lets the storage holder maul keys); without the file they are lost (see Key management too hard: end-to-end mail clients error-prone or unused).

4.10. Users, groups and privileges

Pepsi never runs as root. The long-lived daemons may be started as root so that they can bind privileged ports and read root-only TLS keys, but each one drops to its own unprivileged service account before serving a single connection; if the drop cannot be completed the process refuses to start rather than risk running as root (see pepsi-common::privdrop). Acting as another local user, such as writing into their mailbox, running their ~/.forward or using their wallet, is the one kind of operation that needs more rights than the service user has. Each such operation is confined to a narrowly scoped setuid helper rather than handed to the pipeline at large. The privilege split describes the split as a security mechanism, and Trust boundaries the boundaries that it upholds.

Under the shipped systemd units the daemons are never root at all: systemd binds the privileged sockets (socket activation) and starts each process as its service account directly. That leaves the TLS keys, which certbot keeps readable by root only. pepsi-setup therefore has systemd read those too, by writing a LoadCredential= drop-in per unit: systemd opens each certificate and key as root when it starts the unit, and hands the service a private copy under $CREDENTIALS_DIRECTORY, which is where the server looks first. Re-run pepsi-setup run whenever you add, move or remove a certificate, so that the drop-in is regenerated. See pepsi-httpd.

The browser setup described above is the one path by which something other than an operator at a terminal can cause a root process to run: pepsi-httpd rings a doorbell socket and systemd starts a short-lived root pepsi-setup apply. That path is the easiest privilege in Pepsi to take away. Its two systemd units are a separate Debian package, pepsi-httpd-admin, whose source-install equivalent is make install INSTALL_ADMIN_UNITS=no. Without them, nothing drains pepsi.setup_task and no HTTP request can change anything under /etc/pepsi, while pepsi-setup on a terminal keeps working unchanged. Putting the package back does not act on what was requested while it was absent: installing it empties that queue before its units are armed. See The split as a hardening lever.

The accounts and groups below are created automatically by the Debian package’s postinst; on a make install you create them yourself (the same names). PostgreSQL access is by peer authentication over the local socket: each service account logs in as a same-named database role, so no passwords are stored anywhere. A remote database keeps the same model: one login role per account, each authenticated by a client certificate or password file of its own that only that account can read, named per role with the {role} placeholder in [pepsi-postgres] CONFIG (see CONFIG in pepsi.conf(5)). The configuration file never holds a password.

4.10.1. Service accounts

These are system accounts (--system, /usr/sbin/nologin, home /var/pepsi), each with a private primary group of the same name. The first three are long-lived daemons that drop to their account after binding; the rest are not daemons in the pipeline.

Account

Runs

Purpose / access

pepsi-ingress

pepsi-ingress

Inbound SMTP server. Socket-activated under systemd (never root, TLS material arrives as a service credential); started by hand it binds 25/465/587 and reads the TLS keys as root, then drops. DB role pepsi-ingress.

pepsi-httpd

pepsi-httpd

HTTP/HTTPS server (MTA-STS policy, Web Key Directory, secure-link portal and, on administrative listeners, /metrics, the API and the console). Socket-activated under systemd, exactly like pepsi-ingress; started by hand it binds 80/443 as root, then drops. DB role pepsi-httpd.

pepsi

pepsi-dispatch and every stage worker it spawns

The pipeline’s working identity. Has messages signed by pepsi-dkim through its socket (group pepsi), but cannot read a signing key. DB role pepsi (owns the queue rows). This is the account that gains the delivery, wallet and token group-identities below transiently. Because any stage may rewrite a message after ingress checked its From:, a signature is only as strong as this account (A compromised component).

pepsi-owner

pepsi-setup (transiently)

Not a runtime service. Owns the pepsi database, schema, tables and functions. When pepsi-setup runs as root it temporarily assumes this identity (seteuid) so that the objects it creates are owned by the peer-authenticated pepsi-owner role, then returns to root for the filesystem work (key generation, ownership fix-ups).

pepsi-helper-token-refresh

pepsi-helper-token-refresh service

Only used when a smarthost uses AUTH = oauth (the Debian package creates it regardless; a make install needs it only then). Refreshes OAuth access tokens and writes the token files that the smarthost relay reads. Has no database role.

pepsi-crypto

pepsi-keys (transiently), and the end-to-end crypto stages

Custodian of private end-to-end key material. Owns secrets.d/pepsi-crypto.secret, the key-encryption key that opens pepsi.crypto_identity.private_wrapped, and is the only database role granted that column; every other role, pepsi included, gets a column-level grant that omits it. pepsi-keys assumes this identity for the duration of a database call and gives it straight back. See Key management. This limits who can read private keys; the crypto stages that run as this account still hold all of them (see Mail appliances store secrets world-readable or unprotected).

pepsi-whitelist

pepsi-whitelist (transiently)

Account that the setuid whitelist CLI assumes, so that ordinary users can maintain their own sender whitelists. Its database role is granted SELECT/INSERT/DELETE on pepsi.whitelist and nothing else.

pepsi-keydisc

pepsi-keydisc@<method> services

Key discovery (WKD, DANE, VKS, LDAP). Parses key material fetched from the open internet, so its database role is the narrowest in the deployment. See Key management.

pepsi-config

pepsi-config (as run by an operator)

The only database role that may write the pepsi.config_override overlay: a component that processes mail must not be able to rewrite the pipeline it runs in.

pepsi-greylist

pepsi-greylist service and pepsi-greylist-prune timer

The greylisting milter. Its group is the group of its socket /run/pepsi-greylist/milter.sock (0660): an MTA that is a member may ask it (pepsi-ingress is one), and the group grants nothing else. Its database role reaches the greylist_* tables, reads whitelist and appends to event_log.

pepsi-dkim

pepsi-dkim service

Custodian of the DKIM and ARC private keys, 0600 files in its home /var/lib/pepsi-dkim (0700). Signs for the stages over /run/pepsi-dkim/sign.sock, which it hands to group pepsi (it is a member for that purpose); owns secrets.d/pepsi-dkim.secret (the TSIG key of DNS_UPDATE = rfc2136). Has no database role.

The package postinst (or, on a manual install, you) creates the ten PostgreSQL login roles before pepsi-setup installs the schema: pepsi-owner, pepsi, pepsi-ingress, pepsi-httpd, pepsi-telemetry, plus the narrow pepsi-whitelist, pepsi-config, pepsi-crypto, pepsi-keydisc and pepsi-greylist roles. pepsi-setup then grants each role what it needs.

4.10.2. Groups gating shared resources

Four groups exist purely to gate access to privileged binaries and secret files. The pepsi user is deliberately not a permanent member of any of them: the setgid bit on the relevant stage binary grants the worker the group identity only for the lifetime of that process.

pepsi-maildir

Gates the local-delivery helper. pepsi-helper-maildir-writer is setuid root and mode 4750 (root:pepsi-maildir), so only this group may execute it. pepsi-stage-relay-to-maildir is setgid this group (mode 2550, pepsi:pepsi-maildir; owner-execute only, so no other local user can reach the setgid bit). When the pepsi worker execs the stage, it gains egid=pepsi-maildir, which is exactly the right to launch the helper. The helper then setuids to the recipient and writes ~/Maildir/new/. The helper is the only code that ever touches a third party’s mailbox.

pepsi-forward

Gates the ~/.forward helper, in exactly the shape of the maildir pair above: pepsi-helper-dot-forward is setuid root and mode 4750 (root:pepsi-forward), and pepsi-stage-dot-forward is setgid this group (mode 2550, pepsi:pepsi-forward). Only needed if you run the ~/.forward stage; without the group, make install’s install-forward-helper / install-forward-stage targets only print a warning and leave the binaries unprivileged.

pepsi-wallets

Gates the wallet helper the same way: pepsi-helper-auto-pay is 4750 root:pepsi-wallets and pepsi-stage-auto-pay is setgid this group (mode 2550, pepsi:pepsi-wallets). A system account of the same name owns the shared wallet databases under /var/lib/pepsi-wallets when WALLET_MODE = shared. Only needed if you run the auto-pay stage; install-auto-pay-helper / install-auto-pay-stage degrade to a warning otherwise.

pepsi-token

Gates the secret material the smarthost relay reads: OAuth token files, the mutual-TLS client cert/key, and the Kerberos credential cache. pepsi-stage-relay-to-smarthost is setgid this group (mode 2550, pepsi:pepsi-token); the directories holding those secrets are setgid pepsi-token so that files dropped into them inherit the group, and the relay can read them without the pepsi user being a member.

4.10.3. Privileged binaries

These bits are set by make install when run as root (it prints the exact chown/chmod to run by hand otherwise), and by the Debian package via dpkg-statoverride, which is the authoritative list (debian/pepsi.postinst):

Binary

Mode / owner

Why

pepsi-helper-maildir-writer

4750 root:pepsi-maildir

Setuid root so it can write into any local user’s mailbox; group-locked so only pepsi-maildir may run it.

pepsi-stage-relay-to-maildir

2550 pepsi:pepsi-maildir

Setgid so the pepsi worker may exec the helper above. Owner-execute and not world-execute: the setgid bit is the whole gate, so every other local user must not be able to reach it. pepsi cannot get the exec right from the group (it is deliberately not a member), so it gets it by owning the file. pepsi-quota, pepsi-stage-dot-forward and pepsi-stage-auto-pay carry the same shape for their own groups.

pepsi-quota

2550 pepsi:pepsi-maildir

Same group and same reason: measure/reconcile run the helper above, and the bit is what lets an unattended sweep do it without root.

pepsi-helper-dot-forward

4750 root:pepsi-forward

Setuid root so it can run a user’s ~/.forward as that user; group-locked to its stage.

pepsi-stage-dot-forward

2550 pepsi:pepsi-forward

Setgid so the pepsi worker may exec the helper above.

pepsi-helper-auto-pay

4750 root:pepsi-wallets

Setuid root so it can drop to the wallet-owning account (or the recipient’s own uid) to run taler-wallet-cli: only the one fixed at build time (--with-taler-wallet-cli), never a path that a caller names.

pepsi-stage-auto-pay

2550 pepsi:pepsi-wallets

Setgid so the pepsi worker may exec the helper above.

pepsi-stage-relay-to-smarthost

2550 pepsi:pepsi-token

Setgid so the pepsi worker may read the pepsi-token secrets; not world-executable, since the stage accepts -c and would otherwise send those secrets to any server that a local user names.

pepsi-whitelist

4755 pepsi-whitelist:pepsi-whitelist

Setuid, and world-executable on purpose: any local user may manage their own <login>/… namespace, and the rule that stops them from touching anyone else’s is inside the program, not the file mode.

pepsi-stage-encrypt, pepsi-stage-decrypt

4750 pepsi-crypto:pepsi

Setuid, not setgid: PostgreSQL peer authentication keys off the effective uid, and pepsi-crypto is the only role granted crypto_identity.private_wrapped. Mode 4750 with group pepsi because these are not user commands — only the dispatcher’s account and root may run them.

Every other binary is installed unprivileged, including pepsi-keys, which is standalone for the same reasons but ships 0755: one binary able to open every private key in a deployment is a much larger prize than pepsi-whitelist’s single table, so making it setuid pepsi-crypto is a per-site decision rather than the default.

4.10.4. Directories and secrets

pepsi-setup and the package create the following under /var/pepsi (the service home) and /etc/pepsi. The setgid (2750) directories cause new files to inherit the directory’s group, so token/TLS/Kerberos material is group-readable by the relay without any per-file chgrp.

Path

Owner / mode

Contents

/var/pepsi

pepsi-owner:pepsi

Service home.

/var/lib/pepsi-dkim

pepsi-dkim 0700

The DKIM and ARC private keys (keys/<domain>/<selector>.key, 0600) and the signing daemon’s state. Left out of backups on purpose (Backup and restore).

/var/lib/pepsi-dkim-public

pepsi-dkim 0755

The daemon’s public status.json (selectors, records, DNS changes owed) and, with PUBLISH_RETIRED_KEYS, retired keys.

/run/pepsi-dkim

pepsi-dkim 0711

The signing socket sign.sock (0660), the unit’s own RuntimeDirectory.

/run/pepsi-greylist

pepsi-greylist 0711

The greylisting milter’s socket milter.sock (0660, group pepsi-greylist), the unit’s own RuntimeDirectory.

/var/pepsi/tokens

pepsi-helper-token-refresh:pepsi-token 2750

OAuth access-token files: written by the refresher, read by the smarthost relay via group pepsi-token.

/var/pepsi/token-refresh

pepsi-helper-token-refresh 0700

Private rotated refresh tokens; never group-readable.

/etc/pepsi/secrets

root:pepsi-helper-token-refresh 0750

The @inline-secret@ token-refresh credentials; only the refresher may read them.

/etc/pepsi/secrets.d

root:root 0751

The fragments pepsi-setup generates: the SRS key, smarthost/LMTP passwords, merchant and resume tokens, the proof-of-origin key and the pepsi-crypto key-encryption key. World-searchable (not readable) so that each service can reach the one fragment that it may open; the fragments themselves are 0640 owned by their single reader account. pepsi-setup creates the directory if it is absent. A credential added by hand to pepsi.conf instead of a fragment is not protected this way (see Mail appliances store secrets world-readable or unprotected).

/var/pepsi/tls

root:pepsi-token 2750

Mutual-TLS client cert + key for the smarthost relay (TLS_CLIENT_CERT / TLS_CLIENT_KEY).

/var/pepsi/krb5

root:pepsi-token 2750

Kerberos credential cache for AUTH = gssapi smarthosts (point KRB5CCNAME at a FILE: cache here; an external keytab refresher populates it).

/run/pepsi

root:root 0775 + ACL

The local submission, administrative, HTTP and setup-applier sockets. Created by the tmpfiles.d snippet, whose ACL grants pepsi-ingress and pepsi-httpd write access by name.

/var/lib/pepsi-wallets

pepsi-wallets:pepsi-wallets 0700

Home of the pepsi-wallets account and the wallet databases that pepsi-helper-auto-pay creates under it (wallets/<key>.sqlite3, mode 0700). Only with pepsi-stage-auto-pay in WALLET_MODE = shared.

The /var/pepsi/tokens, /var/pepsi/tls and /var/pepsi/krb5 directories (and the pepsi-helper-token-refresh account) only matter when you configure a smarthost with the corresponding AUTH mechanism; a default direct-to-MX or local-delivery deployment uses none of them.

4.11. Migrating from GNU Mailman

An existing GNU Mailman installation — 2.1 or 3 — moves onto Pepsi with pepsi-list import and pepsi-archive import. Read the table below before you start, not afterwards.

The behavioural differences are a different table, and it is not duplicated here: Declared differences from GNU Mailman 3 in Mailing lists collects every place where Pepsi deliberately does something other than what upstream does. These are one-click unsubscribe, the always-per-recipient fan-out, the archive’s obscured sender addresses, the purge tombstone, the renamed X-Mailman-* headers, and what is simply absent (no NNTP gateway, no pluggable archivers, no plugin API). Two copies of that table would diverge, and the copy that a migrating operator happened to read would be the stale one. This section is only about what the importer leaves behind.

4.11.1. What does not come across, and why

Why

Passwords

Neither Mailman core’s sha512_crypt hashes nor mailman-web’s Django pbkdf2_sha256 ones are imported. Upstream’s own importer has the code and it is commented out. This is why the importer never has to read the Django database at all, and why there is no legacy hash verifier anywhere in Pepsi. Verified addresses stay verified, so nobody re-proves an address; members only choose a password, which is what pepsi-list invite is for.

Bounce scores

Upstream does not import them either: config.pck’s bounce_info and Mailman 3’s bounce events are both discarded, so a migrated member starts at score zero on both systems.

In-flight tokens

A pending confirmation or moderation token is a promise about a workflow that will not exist after the cutover. Anybody mid-subscription asks again.

Topics (2.1)

No Mailman 3 equivalent.

Template URIs we refuse to fetch

A file: override would read the server’s own disk, so it is not imported. The importer reports every such URI at import time rather than letting the first notice that needs it fail months later, and the list falls back to the built-in template.

One Pepsi setting is decided by the import rather than carried over: a list the importer creates gets hold_dmarc_failures = no, so it keeps distributing posts that fail DMARC as GNU Mailman did; the report says so per list. Turn it on with pepsi-list list set-ext <list> hold_dmarc_failures yes once the list’s posters are known to pass (Settings Pepsi invented).

Anything else the importer did not understand is in its report. It exits non-zero when something was lost, which is the point: a migration that half worked and exited zero is a migration nobody checks. Nothing is rolled back, and re-running an import is safe: every write is keyed on the list id, the address or the Message-ID, so a second run is a no-op.

4.11.3. When our pickle reader is stricter than theirs

pepsi-list import mailman21 reads config.pck with its own parser, which constructs nothing: no module is looked up and no callable is called, because a config.pck comes from somebody else’s server and Python’s pickle.load on untrusted input is arbitrary code execution.

The price of that is strictness. The reader may refuse a file, for example because of an opcode that it does not implement, a protocol newer than Python 2 could write, or an integer too wide for 64 bits. It then says so and names the escape hatch:

python2 contrib/mm21-export.py /var/lib/mailman/lists/announce/config.pck \
    > /root/announce.json
pepsi-list import mailman21 --json /root/announce.json --list announce

That script runs on the old server, with the Python that server already trusted, and dumps the same dictionary as JSON. Both routes then go through the same mapping, so they cannot disagree about what the file means. A migration must never be blocked by our parser being stricter than the one that wrote the file.

4.11.4. Header renames a user will notice

Pepsi emits X-Pepsi-List-* where Mailman emitted X-Mailman-*; the mapping table is in Mailing lists. This is a migrating user’s real cost, because procmail and Sieve rules are written against those names, and it is worth telling them before the switch rather than after.