5. Debian packages¶
One source package builds four binary packages: the mail server and three splits off it. Two of those splits are a matter of ordinary packaging judgement: a stage that nobody should have to download unless they use it, and a collector that belongs on somebody else’s machine. The third is a security control: it contains the one mechanism by which an HTTP request can cause a privileged change on the host, and nothing else, so removing it takes away the mechanism rather than the button that reaches it.
Installation covers installing and bootstrapping a deployment; this chapter covers what each package contains and how to use the split.
5.1. The package set¶
Package |
Arch |
What it carries |
|---|---|---|
|
|
The mail server: every program except the two split out below, the SQL
schema, the templates, the packaged configuration defaults, the man
pages, the info manual, the placeholder copied to
|
|
|
|
|
|
|
|
|
|
5.1.1. pepsi¶
The base package, and on a terminal-administered host the only one you need.
It installs the release layout described under Installing in
Installation: the multi-call binary at /usr/libexec/pepsi/pepsi with
one /usr/bin symlink per folded program, and the standalone programs as real
executables of their own. The standalone programs are the setuid and setgid
ones, plus those invoked separately, such as pepsi-setup. Alongside them go
the SQL patch files that pepsi-setup reads, the public-hoster list, the message templates, the
config.d defaults that are parsed before pepsi.conf, all of the man
pages, and the GNU info manual.
It also claims Debian’s mail-transport-agent interface
(Provides/Conflicts/Replaces: mail-transport-agent), so
/usr/sbin/sendmail, mailq and newaliases become Pepsi’s, and no
second MTA can be installed beside it. It Depends on postgresql,
adduser and acl, and its postinst creates the service accounts, the
gating groups, the runtime directories, the dpkg-statoverride entries for
the privileged binaries and (best effort) the database roles. See Users,
groups and privileges in Installation for what those identities are and
why each exists.
It only Suggests rsync, which fetches the IP blocklists that are
distributed no other way (pepsi-reputation). None of those is
enabled by default; on a host without rsync they are not used, and
pepsi-setup warns about any that is enabled.
Everything it ships is installed disabled. To bring the pipeline up, run
systemctl enable --now pepsi.target after a configuration file exists and
pepsi-setup has run. This statement is about installation only; see
Upgrades, removal and service state for what a later apt upgrade or apt remove does
to a pipeline that is already running.
Important
pepsi-setup itself is in this package, not in
pepsi-httpd-admin: a host with no administrative package installed must
still be fully configurable from a terminal.
5.1.2. pepsi-httpd-admin¶
The package contains two systemd unit files and nothing else. That is also why
it is Architecture: all and needs no ${shlibs:Depends}.
Those two files are the privileged half of the browser console, and the rest of
this chapter is about them. The dependency direction matters:
pepsi-httpd-admin Depends on pepsi (it is useless alone), while
pepsi merely Recommends pepsi-httpd-admin. A default installation
therefore has it, so the console’s setup interview works out of the box, and
apt remove pepsi-httpd-admin succeeds without removing the mail server too.
With a Depends, the package could not be removed on its own; with a
Suggests, the console of every default installation would silently be
unable to apply anything.
The units ship disabled like all the others, and the package’s postinst
prints that at install time rather than leaving an operator to guess from a
package name what has just become possible. It also empties the setup task
queue as it installs them. Installing it clears the queue first explains why
a newly installed applier must not execute everything that was requested while
it was absent.
5.1.3. pepsi-stage-detect-language¶
The language-detection stage, split off because it embeds sizable statistical
models for recognising languages, and most pipelines never load it. pepsi
Suggests it; the stage Enhances pepsi and Depends on the exact
same binary version. Validation includes a check written for exactly this
split: when the configured pipeline uses the stage and the binary is not on
$PATH, pepsi-setup warns and names the apt install line. The missing
package is thus found before start-up, not when the dispatcher fails to spawn a
worker.
The package also carries pepsi-detect-language, the offline tool for
diagnosis and tuning. It is a symlink on $PATH to the very same executable;
it classifies a message read from a file and prints only what the stage would
have recorded. It is here rather than in pepsi because it is this binary:
it shares the whole classification path and the language models, so shipping it
separately would duplicate exactly the bulk that this split exists to avoid. See
pepsi-stage-detect-language and
pepsi-detect-language.
5.1.4. pepsi-telemetry¶
The central collector for Pepsi’s anonymous, opt-in feature telemetry. It is a
small HTTP server, intended to run on one host behind an existing nginx or
Apache. The package therefore also ships a complete site file for each of the
two, installed disabled under its canonical name in sites-available/. Each
file already points at the socket that the shipped units bind, so enabling one
takes only the symlink (a2ensite or ln -s into sites-enabled/). The
package Recommends a front server and certbot. Both site files are
conffiles: change the server_name for a private collector, and the
change survives upgrades.
Ordinary Pepsi nodes never need this package. What they run is the telemetry
client, which is part of the pepsi package and governed by [pepsi]
SHARE_TELEMETRY. That switch is opt-in and defaults to no. Until an
operator sets it to yes, the daemon (if started at all) stays dormant, with
no socket and nothing submitted, and the in-process calls are no-ops. Shipping
the client’s unit in the main package therefore costs a deployment nothing that
it did not ask for. Note that the collector package still Depends on
pepsi at the same binary version: “runs on its own host” is a statement
about where you deploy it, not a claim that it is self-contained. See pepsi-telemetry.
5.2. Where the packages come from, and how to check them¶
The packages are published on deb.taler.net, signed with the taler-systems
release key like every package there. apt verifies every download against
the keyring that signed-by= names in the sources entry; you install that
keyring once (Verifying a download). A package built from the source tree
with make deb is unsigned unless DEB_SIGN_KEYID names a key. The
binaries in a package are reproducible from the source and the toolchain that
the .buildinfo file names (Reproducible builds).
5.3. Upgrades, removal and service state¶
Everything ships disabled, and that promise is about installation. An
upgrade must not change what is running: a deployment that had the pipeline
up keeps it up, and one that had it down keeps it down. Neither state is
recorded anywhere: systemctl try-restart reads it off the running system,
one unit at a time, restarting a unit that is active and doing nothing to one
that is not. So a socket-activated service sitting idle stays idle, and a first
installation still starts nothing, because the restart runs only when dpkg has
passed a previously configured version.
Note
That behaviour depends on one dh_installsystemd flag, and the flag has a
trap in it. --restart-after-upgrade is the tool’s default, but only while
--no-start is absent. Pepsi passes --no-start precisely because a
fresh installation must start nothing, and that silently cancels the
default. Without both flags, an upgrade stops every unit before the unpack,
nothing starts them again, and systemd is not even reloaded. make
check fails if a call ever loses the flag.
Removal is the other half. apt remove stops the units, and each
package’s prerm does the stopping. The same --no-start also suppresses
the snippet that debhelper uses to stop units on removal. Without a
replacement, removal would leave pepsi-dispatch running against deleted
binaries. It would also leave pepsi-setup-apply.socket listening after its
package had gone, although the absence of that package is supposed to be the
security boundary. The prerm also records which
of its units were running.
That record is what lets a pipeline come back after remove-then-reinstall. Reinstalling
restores nothing by itself: enablement survives a removal untouched (the
/etc/systemd/system/*.wants symlinks belong to no package and only purge
clears them), so without the record an operator is left with a pipeline that is
enabled, will come up at the next boot, and is down until then for no visible
reason. The marker lives at /var/lib/pepsi/<package>.was-active, deliberately
not under /tmp: the interval between removal and reinstallation can contain a
reboot, a power cut or a fortnight. It is written to a temporary name in the same
directory and renamed into place, so an interruption leaves a whole marker or
none. purge deletes it.
Enablement itself is never recorded and never replayed. There is nothing to remember, since dpkg does not touch those symlinks. A mechanism that disabled a target in order to re-enable it later would open exactly the window that it seems to close: an upgrade interrupted in the middle would leave the deployment disabled, with its only trace in a file.
5.4. The split as a hardening lever¶
5.4.1. Why there is a removable half at all¶
Setup is privileged work: writing /etc/pepsi/pepsi.conf, handing each
secrets.d fragment to the single account that reads it, running certbot,
creating database roles, generating key material. pepsi-httpd, which serves
the browser console, does none of it. It drops to the pepsi-httpd account
before it accepts a connection and must never be able to regain what it gave
up.
So it does not act. It records an intent row in pepsi.setup_task
describing what should be true, and then connects to a UNIX socket and closes
it. That connection is the entire message: nothing is written to it and nothing
is ever read from it, so it cannot become a command channel (see
Mail appliances store secrets world-readable or unprotected). The connection only lets
systemd’s socket activation start pepsi-setup apply as root. The applier
takes its work from the database, where every row is validated against a closed
task list and audited, and exits again when there is no work left. Root is
reached through a database table, never through a socket that speaks a
protocol. The full trust model, including the things that the applier
deliberately cannot do, is The applier’s trust model in
pepsi-setup; read it before you enable any of this.
This design is what makes the capability separable. The socket unit (the
doorbell) and the service unit of the applier are the whole mechanism. They are
two files, and they are the two files that pepsi-httpd-admin contains.
5.4.2. What removing it actually removes¶
apt remove pepsi-httpd-admin stops both units and deletes their files. From
that moment nothing drains pepsi.setup_task on its own. An
INSERT into that table is inert: it sits there, and nothing will act on it
unless a human with a root shell chooses to (pepsi-setup apply --once still
works, because the package carries the units, not the program).
The loop is broken at the mechanism, not at the user interface. It does not
matter what happens to the web tier afterwards, because none of the following
gives anyone the ability to start a root process: a hostile administrator with
a valid session, a stolen bearer token, or a bug in pepsi-httpd itself that
allows remote code execution. The web tier never had that ability; it only ever had a doorbell,
and the doorbell is gone (see Command injection and XXE in mail gateway management interfaces).
Two things are pointedly not removed, and confusing them with the applier is the easiest mistake to make here:
Configuration overrides still work.
/api/v1/configand the console’s configuration pages writepepsi.config_override, the database overlay that every component reads at run time, not a file in/etc. They keep their own gate,[pepsi-admin] CONFIG_DB, and are unaffected. The applier has nothing to do with them, and widening the read-only rule to cover them would remove a capability that this split was never about. See Configuration in the database in Configuration. Note that the overlay can rewire the pipeline and choose whichpepsi-stage-*program a stage runs, although it cannot name any other executable. A principal that may write it can therefore still change how mail is processed, and this split does not limit that (see Command injection and XXE in mail gateway management interfaces).Nothing about mail flow changes.
pepsi-ingress,pepsi-dispatch, every stage andpepsi-httpd’s MTA-STS and metrics surfaces are untouched. This is not a reduced-functionality build of the mail server.
Two console features go with it: end users signing in by e-mail
(Signing in by e-mail) and password sign-in to a console account that has a
second factor. The web tier cannot read the second-factor secrets. Every
enrolment confirmation is therefore a confirm-enrolment task, and every such
sign-in a confirm-login task, which the applier checks (A second factor for key management).
Without the applier, enrolment is refused at the first step rather than mailing a link that
could never be completed, and such an account cannot sign in by password (the
local socket still works).
5.4.3. How pepsi-httpd notices¶
A console that offers forms on a host with no applier is the worst outcome of
the split: an operator fills in an interview, presses apply, sees a pending
row and waits for a change that is never going to happen. So before every
mutation, the server asks whether a privileged change can actually happen from
here.
It has already dropped privileges, so it cannot ask dpkg and cannot talk to
systemd’s private bus, and it must not gain a way to do so. What it can do is
look at two files, which together answer the questions installed? and
armed?:
pepsi-setup-apply.socketexists as a unit file in one of the directories systemd searches (/etc/systemd/system,/run/systemd/system,/usr/local/lib/systemd/system,/usr/lib/systemd/system,/lib/systemd/system); and[pepsi-admin] APPLY_SOCKETexists, is a socket, and is writable by this account — connecting to a UNIX socket needs write permission, and a doorbell that cannot be rung is not a capability.
This takes two stats and an access, and needs no privilege. The probe
deliberately does not connect: connecting is the doorbell ring, and would
start a root process on every page render.
Neither test is redundant, and neither should be optimised away later:
Without the unit-file test, a stale socket node reads as a live doorbell. Stale nodes do occur in practice: systemd’s
RemoveOnStop=defaults to off (the shipped unit sets it; a hand-edited one may not), and a package whose files are deleted rather than stopped leaves the node behind outright. That is precisely the scenario that this feature exists for, and getting it wrong would fail open.Without the socket test, a freshly installed package whose socket has never been enabled would read as ready, and every task would sit
pendingfor ever.
The check fails closed. A path that cannot be examined, a path that is not a socket, a permission error, an unreadable parent directory — every uncertainty reads as “no applier” and the console goes read-only. The resulting failure is that an operator is told to install something that is already installed. A failure in the other direction would leave an operator believing that a change was made. Only the first can be fixed by reading the message, and the messages are written to be read: they distinguish not installed, installed but not started and there is a stale node here, and each names its own remedy.
5.4.4. What read-only looks like¶
Everything is still rendered. Every setting the setup interview asks about is shown, with its staged answer or its default; every control that would change one is disabled, under a banner naming the missing package. Hiding the pages would be the easy reading of “read-only”, and the wrong one: an operator who deliberately removed the applier still has to be able to see what the deployment is configured to do.
The API refuses the corresponding mutations with a status and a code of their own rather than a generic failure:
HTTP/1.1 503 Service Unavailable
{"code": "setup_applier_unavailable",
"hint": "the administrative package 'pepsi-httpd-admin' is not installed …",
"detail": {"package": "pepsi-httpd-admin", "unit": "pepsi-setup-apply.socket"}}
It applies to POST /api/v1/setup/tasks, /setup/preflight,
/setup/dns-check, /setup/certificates, PUT
/api/v1/setup/answers, and the self-service key requests POST
/api/v1/identities, /api/v1/identities/client and a revoking PATCH
/api/v1/identities/{id}, and to the console’s own form posts — POST
/ui/setup/{step}, /ui/setup/tasks, /ui/domains/check,
/ui/identities/request/{generate,register} and
/ui/identities/{id}/revoke. They all
reach the queue through one enqueuing function that asks first, so the refusal
is a single decision rather than one per surface. The disabled control on the
page is an addition to that check, not a replacement for it. Answer staging
is included on purpose: a draft answer’s only purpose is to be applied, and a
wizard that collects six steps of answers on a host where nothing can apply
them is exactly the silent no-op that the design avoids. DELETE
/api/v1/setup/answers is not gated: discarding drafts cannot cause a privileged change, and a read-only console
still has to be able to clear a half-finished interview.
The code is distinct from setup_write_unavailable (which means there is no
CONFIG_DB connection) because the two have different remedies, and the
applier is reported first when both apply: configuring a database role would
not have helped.
5.4.5. What still works with the package gone¶
All of it, from a terminal. pepsi-setup is in the pepsi package, so:
pepsi-setup -c /etc/pepsi/pepsi.conf run
pepsi-setup -c /etc/pepsi/pepsi.conf --wizard
pepsi-setup -c /etc/pepsi/pepsi.conf check
behave exactly as they do with the package installed, as does editing
pepsi.conf by hand, generating keys, running certbot yourself and every
other administrative task. The applier subcommand is still there too: what the
package carries is the units, not the program.
What is removed is one thing only: the web tier’s ability to start it.
5.4.6. Arming it is a separate, deliberate step¶
Installing pepsi-httpd-admin makes the capability available. It does not
turn it on. Like every other Pepsi unit the two ship disabled, so a host that
has the package but has never run
# systemctl enable --now pepsi-setup-apply.socket
has no doorbell, and the console stays read-only and says which command would change that. Installing a package and arming a privilege stay two separate decisions.
5.4.7. Installing it clears the queue first¶
Removing the applier breaks the loop at the mechanism, and rows can still reach
pepsi.setup_task while it is gone: an operator with a root shell can enqueue
one by hand, and a console that is only told it is read-only is one release
away from a path that forgets to ask. Nothing expires such a row. A request is
durable, and the queue is drained in the order it was written.
Taken together, this means that an applier installed months later would execute every one of those rows at the first ring of the doorbell, against a deployment that has changed in the meantime. It might write a configuration from a set of answers that nobody remembers, or obtain a certificate for a host that has been renamed.
So installing pepsi-httpd-admin deletes the queue before its units are armed.
The postinst runs
# pepsi-setup -c /etc/pepsi/pepsi.conf apply --clear
which removes every row without executing any of them and says so, naming the reason, so an operator who had legitimately queued something knows what became of it and can ask again. What was asked for is not lost either way: each request, refusal, success and failure is an independent durable entry in the audit log, which the reset does not touch. The same command is available by hand at any time — see pepsi-setup.
It is best effort, as a maintainer script must be. At install time there may be
no configuration file, no schema, no database and no PostgreSQL running; each of
those is reported and skipped, and none of them makes the installation fail. (A database with no
Pepsi schema is not even a skip: it has never held a task, so it is an empty
queue.) The reset runs on every configure, not only on a first installation,
because dpkg reports a re-installation after apt remove as an upgrade, and
remove-then-reinstall is exactly the case that this reset exists for.
5.4.8. The same lever from source¶
A make install deployment gets the identical control:
make install INSTALL_ADMIN_UNITS=no
which installs everything except pepsi-setup-apply.service and
pepsi-setup-apply.socket, and prints what it left out. make uninstall
always removes both sets regardless of the variable’s value on that
invocation. If a privileged unit survived an uninstall because a make variable
changed, that would be exactly the failure that this split is about.
5.4.9. Choosing a posture¶
The probe, the configuration option and the package set combine, so the
question “can this console change this machine?” has more than two answers.
[pepsi-admin] APPLIER is the override, documented in full in
pepsi.conf(5):
Posture |
How |
Result |
|---|---|---|
Administered from a browser |
Package installed, socket enabled, |
The console can apply setup tasks. |
Installed but not armed |
Package installed, socket never enabled |
Read-only; the refusal names |
Administered from a terminal |
|
Read-only; the refusal names the package, and nothing drains the task queue unattended. |
Read-only by policy |
|
Read-only by configuration rather than by which packages are installed, and the refusal says so. Reversible by editing one option — a policy, not a boundary. |
Applied some other way |
|
The console offers the forms. |
Two of those rows deserve their difference spelled out. APPLIER = no and
removing the package both produce a read-only console, and they are not the
same defence. APPLIER = no is a line in a configuration file that
pepsi-httpd consults; anyone who can change that file, or who can make
pepsi-httpd read a different one, undoes it. Removing the package deletes
the units that would have run the root process, and no amount of influence over
pepsi-httpd puts them back. Use the option to state an intention; remove the
package to enforce it.
The last row is a promise, not a check: with APPLIER = yes the console will
happily queue tasks whether or not anything drains them. It exists for the
deployments that the probe genuinely cannot see: a cron job that drains the
queue, a host without systemd, or unit files installed somewhere that systemd
does not search. It is the one place in this mechanism where the code takes an
operator’s word for it.
There is one hand-made state the packaging cannot produce: the socket present
and writable while the service unit is masked or broken. The probe reads that
as available, the task is enqueued, and it stays pending. The task page
shows that status rather than pretending otherwise. Both units ship in one
package, so reaching this state takes deliberate effort.
5.5. See also¶
Installation — installing either way, and the accounts, groups and
privileged bits that the pepsi package’s postinst creates.
Upgrades, removal and service state — what upgrading, removing and reinstalling do to the
services that were running.
Threat model (A compromised component) and Security model — the
threat model in which this split is a mitigation, in particular what an attacker who
compromises pepsi-httpd does and does not get.
The administration console and The administrative API — the console and API surfaces that go
read-only. pepsi-setup — the applier’s trust model, its closed
task list and its admission gates. pepsi-httpd — the probe, the
gated endpoints and [pepsi-admin] in reference form.