5. Debian packages

One source package builds four binary packages: the mail server and three splits off it. Two of those splits are a matter of ordinary packaging judgement: a stage that nobody should have to download unless they use it, and a collector that belongs on somebody else’s machine. The third is a security control: it contains the one mechanism by which an HTTP request can cause a privileged change on the host, and nothing else, so removing it takes away the mechanism rather than the button that reaches it.

Installation covers installing and bootstrapping a deployment; this chapter covers what each package contains and how to use the split.

5.1. The package set

Package

Arch

What it carries

pepsi

any

The mail server: every program except the two split out below, the SQL schema, the templates, the packaged configuration defaults, the man pages, the info manual, the placeholder copied to /etc/pepsi/pepsi.conf on first installation (not a conffile, so upgrades leave the configuration alone), the tmpfiles.d snippet, and every systemd unit except the applier’s two and the collector’s two.

pepsi-httpd-admin

all

pepsi-setup-apply.socket and pepsi-setup-apply.service. Nothing else — no binary, no configuration, no ELF object at all.

pepsi-stage-detect-language

any

pepsi-stage-detect-language, the pepsi-detect-language diagnostic tool (the same executable under a second name), and both man pages.

pepsi-telemetry

any

pepsi-telemetry, its systemd units (pepsi-telemetry.service and pepsi-telemetry.socket — the socket is what the shipped SERVE = systemd listener binds), its own configuration file /etc/pepsi-telemetry/pepsi-telemetry.conf, its man page, nginx and Apache site files (installed disabled, under their canonical site names, ready to enable with a symlink), and update-feature-stability.sh, the script that turns a collector report into the manual’s feature-stability table.

5.1.1. pepsi

The base package, and on a terminal-administered host the only one you need. It installs the release layout described under Installing in Installation: the multi-call binary at /usr/libexec/pepsi/pepsi with one /usr/bin symlink per folded program, and the standalone programs as real executables of their own. The standalone programs are the setuid and setgid ones, plus those invoked separately, such as pepsi-setup. Alongside them go the SQL patch files that pepsi-setup reads, the public-hoster list, the message templates, the config.d defaults that are parsed before pepsi.conf, all of the man pages, and the GNU info manual.

It also claims Debian’s mail-transport-agent interface (Provides/Conflicts/Replaces: mail-transport-agent), so /usr/sbin/sendmail, mailq and newaliases become Pepsi’s, and no second MTA can be installed beside it. It Depends on postgresql, adduser and acl, and its postinst creates the service accounts, the gating groups, the runtime directories, the dpkg-statoverride entries for the privileged binaries and (best effort) the database roles. See Users, groups and privileges in Installation for what those identities are and why each exists.

It only Suggests rsync, which fetches the IP blocklists that are distributed no other way (pepsi-reputation). None of those is enabled by default; on a host without rsync they are not used, and pepsi-setup warns about any that is enabled.

Everything it ships is installed disabled. To bring the pipeline up, run systemctl enable --now pepsi.target after a configuration file exists and pepsi-setup has run. This statement is about installation only; see Upgrades, removal and service state for what a later apt upgrade or apt remove does to a pipeline that is already running.

Important

pepsi-setup itself is in this package, not in pepsi-httpd-admin: a host with no administrative package installed must still be fully configurable from a terminal.

5.1.2. pepsi-httpd-admin

The package contains two systemd unit files and nothing else. That is also why it is Architecture: all and needs no ${shlibs:Depends}.

Those two files are the privileged half of the browser console, and the rest of this chapter is about them. The dependency direction matters: pepsi-httpd-admin Depends on pepsi (it is useless alone), while pepsi merely Recommends pepsi-httpd-admin. A default installation therefore has it, so the console’s setup interview works out of the box, and apt remove pepsi-httpd-admin succeeds without removing the mail server too. With a Depends, the package could not be removed on its own; with a Suggests, the console of every default installation would silently be unable to apply anything.

The units ship disabled like all the others, and the package’s postinst prints that at install time rather than leaving an operator to guess from a package name what has just become possible. It also empties the setup task queue as it installs them. Installing it clears the queue first explains why a newly installed applier must not execute everything that was requested while it was absent.

5.1.3. pepsi-stage-detect-language

The language-detection stage, split off because it embeds sizable statistical models for recognising languages, and most pipelines never load it. pepsi Suggests it; the stage Enhances pepsi and Depends on the exact same binary version. Validation includes a check written for exactly this split: when the configured pipeline uses the stage and the binary is not on $PATH, pepsi-setup warns and names the apt install line. The missing package is thus found before start-up, not when the dispatcher fails to spawn a worker.

The package also carries pepsi-detect-language, the offline tool for diagnosis and tuning. It is a symlink on $PATH to the very same executable; it classifies a message read from a file and prints only what the stage would have recorded. It is here rather than in pepsi because it is this binary: it shares the whole classification path and the language models, so shipping it separately would duplicate exactly the bulk that this split exists to avoid. See pepsi-stage-detect-language and pepsi-detect-language.

5.1.4. pepsi-telemetry

The central collector for Pepsi’s anonymous, opt-in feature telemetry. It is a small HTTP server, intended to run on one host behind an existing nginx or Apache. The package therefore also ships a complete site file for each of the two, installed disabled under its canonical name in sites-available/. Each file already points at the socket that the shipped units bind, so enabling one takes only the symlink (a2ensite or ln -s into sites-enabled/). The package Recommends a front server and certbot. Both site files are conffiles: change the server_name for a private collector, and the change survives upgrades.

Ordinary Pepsi nodes never need this package. What they run is the telemetry client, which is part of the pepsi package and governed by [pepsi] SHARE_TELEMETRY. That switch is opt-in and defaults to no. Until an operator sets it to yes, the daemon (if started at all) stays dormant, with no socket and nothing submitted, and the in-process calls are no-ops. Shipping the client’s unit in the main package therefore costs a deployment nothing that it did not ask for. Note that the collector package still Depends on pepsi at the same binary version: “runs on its own host” is a statement about where you deploy it, not a claim that it is self-contained. See pepsi-telemetry.

5.2. Where the packages come from, and how to check them

The packages are published on deb.taler.net, signed with the taler-systems release key like every package there. apt verifies every download against the keyring that signed-by= names in the sources entry; you install that keyring once (Verifying a download). A package built from the source tree with make deb is unsigned unless DEB_SIGN_KEYID names a key. The binaries in a package are reproducible from the source and the toolchain that the .buildinfo file names (Reproducible builds).

5.3. Upgrades, removal and service state

Everything ships disabled, and that promise is about installation. An upgrade must not change what is running: a deployment that had the pipeline up keeps it up, and one that had it down keeps it down. Neither state is recorded anywhere: systemctl try-restart reads it off the running system, one unit at a time, restarting a unit that is active and doing nothing to one that is not. So a socket-activated service sitting idle stays idle, and a first installation still starts nothing, because the restart runs only when dpkg has passed a previously configured version.

Note

That behaviour depends on one dh_installsystemd flag, and the flag has a trap in it. --restart-after-upgrade is the tool’s default, but only while --no-start is absent. Pepsi passes --no-start precisely because a fresh installation must start nothing, and that silently cancels the default. Without both flags, an upgrade stops every unit before the unpack, nothing starts them again, and systemd is not even reloaded. make check fails if a call ever loses the flag.

Removal is the other half. apt remove stops the units, and each package’s prerm does the stopping. The same --no-start also suppresses the snippet that debhelper uses to stop units on removal. Without a replacement, removal would leave pepsi-dispatch running against deleted binaries. It would also leave pepsi-setup-apply.socket listening after its package had gone, although the absence of that package is supposed to be the security boundary. The prerm also records which of its units were running.

That record is what lets a pipeline come back after remove-then-reinstall. Reinstalling restores nothing by itself: enablement survives a removal untouched (the /etc/systemd/system/*.wants symlinks belong to no package and only purge clears them), so without the record an operator is left with a pipeline that is enabled, will come up at the next boot, and is down until then for no visible reason. The marker lives at /var/lib/pepsi/<package>.was-active, deliberately not under /tmp: the interval between removal and reinstallation can contain a reboot, a power cut or a fortnight. It is written to a temporary name in the same directory and renamed into place, so an interruption leaves a whole marker or none. purge deletes it.

Enablement itself is never recorded and never replayed. There is nothing to remember, since dpkg does not touch those symlinks. A mechanism that disabled a target in order to re-enable it later would open exactly the window that it seems to close: an upgrade interrupted in the middle would leave the deployment disabled, with its only trace in a file.

5.4. The split as a hardening lever

5.4.1. Why there is a removable half at all

Setup is privileged work: writing /etc/pepsi/pepsi.conf, handing each secrets.d fragment to the single account that reads it, running certbot, creating database roles, generating key material. pepsi-httpd, which serves the browser console, does none of it. It drops to the pepsi-httpd account before it accepts a connection and must never be able to regain what it gave up.

So it does not act. It records an intent row in pepsi.setup_task describing what should be true, and then connects to a UNIX socket and closes it. That connection is the entire message: nothing is written to it and nothing is ever read from it, so it cannot become a command channel (see Mail appliances store secrets world-readable or unprotected). The connection only lets systemd’s socket activation start pepsi-setup apply as root. The applier takes its work from the database, where every row is validated against a closed task list and audited, and exits again when there is no work left. Root is reached through a database table, never through a socket that speaks a protocol. The full trust model, including the things that the applier deliberately cannot do, is The applier’s trust model in pepsi-setup; read it before you enable any of this.

This design is what makes the capability separable. The socket unit (the doorbell) and the service unit of the applier are the whole mechanism. They are two files, and they are the two files that pepsi-httpd-admin contains.

5.4.2. What removing it actually removes

apt remove pepsi-httpd-admin stops both units and deletes their files. From that moment nothing drains pepsi.setup_task on its own. An INSERT into that table is inert: it sits there, and nothing will act on it unless a human with a root shell chooses to (pepsi-setup apply --once still works, because the package carries the units, not the program).

The loop is broken at the mechanism, not at the user interface. It does not matter what happens to the web tier afterwards, because none of the following gives anyone the ability to start a root process: a hostile administrator with a valid session, a stolen bearer token, or a bug in pepsi-httpd itself that allows remote code execution. The web tier never had that ability; it only ever had a doorbell, and the doorbell is gone (see Command injection and XXE in mail gateway management interfaces).

Two things are pointedly not removed, and confusing them with the applier is the easiest mistake to make here:

  • Configuration overrides still work. /api/v1/config and the console’s configuration pages write pepsi.config_override, the database overlay that every component reads at run time, not a file in /etc. They keep their own gate, [pepsi-admin] CONFIG_DB, and are unaffected. The applier has nothing to do with them, and widening the read-only rule to cover them would remove a capability that this split was never about. See Configuration in the database in Configuration. Note that the overlay can rewire the pipeline and choose which pepsi-stage-* program a stage runs, although it cannot name any other executable. A principal that may write it can therefore still change how mail is processed, and this split does not limit that (see Command injection and XXE in mail gateway management interfaces).

  • Nothing about mail flow changes. pepsi-ingress, pepsi-dispatch, every stage and pepsi-httpd’s MTA-STS and metrics surfaces are untouched. This is not a reduced-functionality build of the mail server.

Two console features go with it: end users signing in by e-mail (Signing in by e-mail) and password sign-in to a console account that has a second factor. The web tier cannot read the second-factor secrets. Every enrolment confirmation is therefore a confirm-enrolment task, and every such sign-in a confirm-login task, which the applier checks (A second factor for key management). Without the applier, enrolment is refused at the first step rather than mailing a link that could never be completed, and such an account cannot sign in by password (the local socket still works).

5.4.3. How pepsi-httpd notices

A console that offers forms on a host with no applier is the worst outcome of the split: an operator fills in an interview, presses apply, sees a pending row and waits for a change that is never going to happen. So before every mutation, the server asks whether a privileged change can actually happen from here.

It has already dropped privileges, so it cannot ask dpkg and cannot talk to systemd’s private bus, and it must not gain a way to do so. What it can do is look at two files, which together answer the questions installed? and armed?:

  1. pepsi-setup-apply.socket exists as a unit file in one of the directories systemd searches (/etc/systemd/system, /run/systemd/system, /usr/local/lib/systemd/system, /usr/lib/systemd/system, /lib/systemd/system); and

  2. [pepsi-admin] APPLY_SOCKET exists, is a socket, and is writable by this account — connecting to a UNIX socket needs write permission, and a doorbell that cannot be rung is not a capability.

This takes two stats and an access, and needs no privilege. The probe deliberately does not connect: connecting is the doorbell ring, and would start a root process on every page render.

Neither test is redundant, and neither should be optimised away later:

  • Without the unit-file test, a stale socket node reads as a live doorbell. Stale nodes do occur in practice: systemd’s RemoveOnStop= defaults to off (the shipped unit sets it; a hand-edited one may not), and a package whose files are deleted rather than stopped leaves the node behind outright. That is precisely the scenario that this feature exists for, and getting it wrong would fail open.

  • Without the socket test, a freshly installed package whose socket has never been enabled would read as ready, and every task would sit pending for ever.

The check fails closed. A path that cannot be examined, a path that is not a socket, a permission error, an unreadable parent directory — every uncertainty reads as “no applier” and the console goes read-only. The resulting failure is that an operator is told to install something that is already installed. A failure in the other direction would leave an operator believing that a change was made. Only the first can be fixed by reading the message, and the messages are written to be read: they distinguish not installed, installed but not started and there is a stale node here, and each names its own remedy.

5.4.4. What read-only looks like

Everything is still rendered. Every setting the setup interview asks about is shown, with its staged answer or its default; every control that would change one is disabled, under a banner naming the missing package. Hiding the pages would be the easy reading of “read-only”, and the wrong one: an operator who deliberately removed the applier still has to be able to see what the deployment is configured to do.

The API refuses the corresponding mutations with a status and a code of their own rather than a generic failure:

HTTP/1.1 503 Service Unavailable
{"code":   "setup_applier_unavailable",
 "hint":   "the administrative package 'pepsi-httpd-admin' is not installed …",
 "detail": {"package": "pepsi-httpd-admin", "unit": "pepsi-setup-apply.socket"}}

It applies to POST /api/v1/setup/tasks, /setup/preflight, /setup/dns-check, /setup/certificates, PUT /api/v1/setup/answers, and the self-service key requests POST /api/v1/identities, /api/v1/identities/client and a revoking PATCH /api/v1/identities/{id}, and to the console’s own form posts — POST /ui/setup/{step}, /ui/setup/tasks, /ui/domains/check, /ui/identities/request/{generate,register} and /ui/identities/{id}/revoke. They all reach the queue through one enqueuing function that asks first, so the refusal is a single decision rather than one per surface. The disabled control on the page is an addition to that check, not a replacement for it. Answer staging is included on purpose: a draft answer’s only purpose is to be applied, and a wizard that collects six steps of answers on a host where nothing can apply them is exactly the silent no-op that the design avoids. DELETE /api/v1/setup/answers is not gated: discarding drafts cannot cause a privileged change, and a read-only console still has to be able to clear a half-finished interview.

The code is distinct from setup_write_unavailable (which means there is no CONFIG_DB connection) because the two have different remedies, and the applier is reported first when both apply: configuring a database role would not have helped.

5.4.5. What still works with the package gone

All of it, from a terminal. pepsi-setup is in the pepsi package, so:

pepsi-setup -c /etc/pepsi/pepsi.conf run
pepsi-setup -c /etc/pepsi/pepsi.conf --wizard
pepsi-setup -c /etc/pepsi/pepsi.conf check

behave exactly as they do with the package installed, as does editing pepsi.conf by hand, generating keys, running certbot yourself and every other administrative task. The applier subcommand is still there too: what the package carries is the units, not the program.

What is removed is one thing only: the web tier’s ability to start it.

5.4.6. Arming it is a separate, deliberate step

Installing pepsi-httpd-admin makes the capability available. It does not turn it on. Like every other Pepsi unit the two ship disabled, so a host that has the package but has never run

# systemctl enable --now pepsi-setup-apply.socket

has no doorbell, and the console stays read-only and says which command would change that. Installing a package and arming a privilege stay two separate decisions.

5.4.7. Installing it clears the queue first

Removing the applier breaks the loop at the mechanism, and rows can still reach pepsi.setup_task while it is gone: an operator with a root shell can enqueue one by hand, and a console that is only told it is read-only is one release away from a path that forgets to ask. Nothing expires such a row. A request is durable, and the queue is drained in the order it was written.

Taken together, this means that an applier installed months later would execute every one of those rows at the first ring of the doorbell, against a deployment that has changed in the meantime. It might write a configuration from a set of answers that nobody remembers, or obtain a certificate for a host that has been renamed.

So installing pepsi-httpd-admin deletes the queue before its units are armed. The postinst runs

# pepsi-setup -c /etc/pepsi/pepsi.conf apply --clear

which removes every row without executing any of them and says so, naming the reason, so an operator who had legitimately queued something knows what became of it and can ask again. What was asked for is not lost either way: each request, refusal, success and failure is an independent durable entry in the audit log, which the reset does not touch. The same command is available by hand at any time — see pepsi-setup.

It is best effort, as a maintainer script must be. At install time there may be no configuration file, no schema, no database and no PostgreSQL running; each of those is reported and skipped, and none of them makes the installation fail. (A database with no Pepsi schema is not even a skip: it has never held a task, so it is an empty queue.) The reset runs on every configure, not only on a first installation, because dpkg reports a re-installation after apt remove as an upgrade, and remove-then-reinstall is exactly the case that this reset exists for.

5.4.8. The same lever from source

A make install deployment gets the identical control:

make install INSTALL_ADMIN_UNITS=no

which installs everything except pepsi-setup-apply.service and pepsi-setup-apply.socket, and prints what it left out. make uninstall always removes both sets regardless of the variable’s value on that invocation. If a privileged unit survived an uninstall because a make variable changed, that would be exactly the failure that this split is about.

5.4.9. Choosing a posture

The probe, the configuration option and the package set combine, so the question “can this console change this machine?” has more than two answers. [pepsi-admin] APPLIER is the override, documented in full in pepsi.conf(5):

Posture

How

Result

Administered from a browser

Package installed, socket enabled, APPLIER = auto (the default)

The console can apply setup tasks.

Installed but not armed

Package installed, socket never enabled

Read-only; the refusal names systemctl enable --now pepsi-setup-apply.socket.

Administered from a terminal

apt remove pepsi-httpd-admin (or INSTALL_ADMIN_UNITS=no)

Read-only; the refusal names the package, and nothing drains the task queue unattended.

Read-only by policy

[pepsi-admin] APPLIER = no, package left installed

Read-only by configuration rather than by which packages are installed, and the refusal says so. Reversible by editing one option — a policy, not a boundary.

Applied some other way

pepsi-setup apply --once from cron, or a hand-written unit, plus [pepsi-admin] APPLIER = yes

The console offers the forms. yes is a promise the operator makes; nothing verifies it.

Two of those rows deserve their difference spelled out. APPLIER = no and removing the package both produce a read-only console, and they are not the same defence. APPLIER = no is a line in a configuration file that pepsi-httpd consults; anyone who can change that file, or who can make pepsi-httpd read a different one, undoes it. Removing the package deletes the units that would have run the root process, and no amount of influence over pepsi-httpd puts them back. Use the option to state an intention; remove the package to enforce it.

The last row is a promise, not a check: with APPLIER = yes the console will happily queue tasks whether or not anything drains them. It exists for the deployments that the probe genuinely cannot see: a cron job that drains the queue, a host without systemd, or unit files installed somewhere that systemd does not search. It is the one place in this mechanism where the code takes an operator’s word for it.

There is one hand-made state the packaging cannot produce: the socket present and writable while the service unit is masked or broken. The probe reads that as available, the task is enqueued, and it stays pending. The task page shows that status rather than pretending otherwise. Both units ship in one package, so reaching this state takes deliberate effort.

5.5. See also

Installation — installing either way, and the accounts, groups and privileged bits that the pepsi package’s postinst creates. Upgrades, removal and service state — what upgrading, removing and reinstalling do to the services that were running. Threat model (A compromised component) and Security model — the threat model in which this split is a mitigation, in particular what an attacker who compromises pepsi-httpd does and does not get. The administration console and The administrative API — the console and API surfaces that go read-only. pepsi-setup — the applier’s trust model, its closed task list and its admission gates. pepsi-httpd — the probe, the gated endpoints and [pepsi-admin] in reference form.