4. Debian packages¶

One source package builds four binary packages. Three of the splits are ordinary packaging judgement — a stage nobody should have to download unless they use it, a collector that belongs on somebody else’s machine. The fourth is a security control, and it is why this chapter exists: it contains the one mechanism by which an HTTP request can cause a privileged change on the host, and nothing else. Remove it and the removal means something, because what is gone is the mechanism rather than the button that reaches it.

Installation covers installing and bootstrapping a deployment. This chapter is about what each package contains, why the line was drawn where it was, and how to use the split deliberately.

4.1. The package set ¶

Package	Arch	What it carries
`pepsi`	`any`	The mail server: every program except the two split out below, the SQL schema, the templates, the packaged configuration defaults, the man pages, the info manual, `/etc/pepsi/pepsi.conf`, the `tmpfiles.d` snippet, and every systemd unit except the applier’s two and the collector’s one.
`pepsi-httpd-admin`	`all`	`pepsi-setup-apply.socket` and `pepsi-setup-apply.service`. Nothing else — no binary, no configuration, no ELF object at all.
`pepsi-stage-detect-language`	`any`	`pepsi-stage-detect-language`, the `pepsi-detect-language` diagnostic tool (the same executable under a second name), and both man pages.
`pepsi-telemetry`	`any`	`pepsi-telemetry`, its systemd unit, its man page, nginx and Apache site files, and the report-formatting helper script.

4.1.1. `pepsi`¶

The base package, and on a terminal-administered host the only one you need. It installs the release layout described under Installing in Installation: the multi-call binary at /usr/libexec/pepsi/pepsi with one /usr/bin symlink per folded program, and the standalone programs — the setuid and setgid ones, plus the separately-invoked ones such as pepsi-setup — as real executables of their own. Alongside them go the SQL patch files pepsi-setup reads, the public-hoster list, the message templates, the config.d defaults that are parsed before pepsi.conf, all of the man pages, and the GNU info manual.

It also claims Debian’s mail-transport-agent interface — Provides/Conflicts/Replaces: mail-transport-agent — so /usr/sbin/sendmail, mailq and newaliases become Pepsi’s, and no second MTA can be installed beside it. It Depends on postgresql, adduser and acl, and its postinst creates the service accounts, the gating groups, the runtime directories, the dpkg-statoverride entries for the privileged binaries and (best effort) the database roles. See Users, groups and privileges in Installation for what those identities are and why each exists.

Everything it ships is installed disabled. Bringing the pipeline up is systemctl enable --now pepsi.target, after a configuration file exists and pepsi-setup has run.

Important

pepsi-setup itself is in this package, not in pepsi-httpd-admin. That is deliberate and load-bearing: a host with no administrative package installed must still be fully configurable from a terminal.

4.1.2. `pepsi-httpd-admin`¶

Two systemd unit files. That is the entire payload — which is also why it is Architecture: all and needs no ${shlibs:Depends}.

Those two files are the privileged half of the browser console, and the rest of this chapter is about them. The dependency direction is the point: pepsi-httpd-admin Depends on pepsi (it is useless alone), while pepsi merely Recommends pepsi-httpd-admin. A default installation therefore has it — the console’s setup interview works out of the box — and apt remove pepsi-httpd-admin succeeds without taking the mail server with it. A Depends would make the lever un-pullable; a Suggests would turn every default installation into one whose console silently cannot apply anything.

The units ship disabled like all the others, and the package’s postinst prints that at install time rather than leaving an operator to infer from a package name what has just become possible. It also empties the setup task queue as it installs them — see Installing it clears the queue first for why arriving must not mean executing everything asked for while it was away.

4.1.3. `pepsi-stage-detect-language`¶

The language-detection stage, split off because it embeds sizable statistical language-recognition models and most pipelines never load it. pepsi Suggests it; the stage Enhances pepsi and Depends on the exact same binary version. Validation carries a check written for exactly this split: when the configured pipeline uses the stage and the binary is not on $PATH, pepsi-setup warns and names the apt install line — so the omission is diagnosed before start-up rather than discovered when the dispatcher fails to spawn a worker.

The package also carries pepsi-detect-language, the offline diagnostic and tuning tool — a $PATH symlink to the very same executable, which classifies a message read from a file and prints only what the stage would have recorded. It is here rather than in pepsi because it is this binary: it shares the whole classification path and the language models, so shipping it separately would duplicate exactly the bulk this split exists to avoid. See pepsi-stage-detect-language and pepsi-detect-language.

4.1.4. `pepsi-telemetry`¶

The central collector for Pepsi’s anonymous, opt-in feature telemetry — a small HTTP server intended to run on one host behind an existing nginx or Apache, which is why the package also ships a site file for each (in sites-available/, disabled) and Recommends a front server and certbot. Ordinary Pepsi nodes never need it; what they run is the telemetry client, which is part of the pepsi package and governed by [pepsi] SHARE_TELEMETRY. That switch is opt-in and defaults to no: until an operator sets it to yes, the daemon exits at once and the in-process calls are no-ops, so the client’s unit shipping in the main package costs a deployment nothing it did not ask for. Note that the collector package still Depends on pepsi at the same binary version: “runs on its own host” is a statement about where you deploy it, not about it being self-contained. See pepsi-telemetry.

4.2. The split as a hardening lever ¶

Everything above is inventory. This is the part that changes what an attacker can do.

4.2.1. Why there is a removable half at all ¶

Setup is privileged work: writing /etc/pepsi/pepsi.conf, handing each secrets.d fragment to the single account that reads it, running certbot, creating database roles, generating key material. pepsi-httpd, which serves the browser console, does none of it. It drops to the pepsi-httpd account before it accepts a connection and must never be able to regain what it gave up.

So it does not act. It records an intent row in pepsi.setup_task describing what should be true, and then connects to a UNIX socket and closes it. That connection is the entire message: nothing is written to it and nothing is ever read from it, so it cannot become a command channel. What it does is let systemd’s socket activation start pepsi-setup apply as root, which takes its work from the database — where every row is validated against a closed task list and audited — and exits again when there is none. Root is reached through a database table, never through a socket that speaks a protocol. The full trust model, including the things the applier deliberately cannot do, is The applier’s trust model in pepsi-setup, and it is required reading before enabling any of this.

That shape is what makes the capability separable. The doorbell socket unit and the applier service unit are the whole mechanism, they are two files, and they are the two files pepsi-httpd-admin contains.

4.2.2. What removing it actually removes ¶

apt remove pepsi-httpd-admin stops and disables both units and deletes them. From that moment nothing drains pepsi.setup_task on its own. An INSERT into that table is inert: it sits there, and nothing will act on it unless a human with a root shell chooses to (pepsi-setup apply --once still works — the package carries the units, not the program).

That is the distinction worth being precise about. The loop is broken at the mechanism, not at the user interface. It does not matter what happens to the web tier afterwards — a hostile administrator with a valid session, a stolen bearer token, a remote-code-execution bug in pepsi-httpd itself — because none of those confer the ability to start a root process. The web tier never had that ability; it only ever had a doorbell, and the doorbell is gone.

Two things are pointedly not removed, and confusing them with the applier is the easiest mistake to make here:

Configuration overrides still work. /api/v1/config and the console’s configuration pages write pepsi.config_override, the database overlay every component reads at run time — not a file in /etc. They keep their own gate, [pepsi-admin] CONFIG_DB, and are unaffected. The applier has nothing to do with them, and widening the read-only rule to cover them would remove a capability this split was never about. See Configuration in the database in Configuration.
Nothing about mail flow changes. pepsi-ingress, pepsi-dispatch, every stage and pepsi-httpd’s MTA-STS and metrics surfaces are untouched. This is not a reduced-functionality build of the mail server.

4.2.3. How `pepsi-httpd` notices ¶

A console that went on offering forms after the applier was removed would be the worst outcome of the split: an operator fills in an interview, presses apply, sees a pending row and waits for a change that is never going to happen. So the server asks, before every mutation, whether a privileged change can actually happen from here.

It has already dropped privileges, so it cannot ask dpkg and cannot talk to systemd’s private bus — and must not gain a way to. What it can do is look at two files, which between them answer installed and armed:

pepsi-setup-apply.socket exists as a unit file in one of the directories systemd searches (/etc/systemd/system, /run/systemd/system, /usr/local/lib/systemd/system, /usr/lib/systemd/system, /lib/systemd/system); and
[pepsi-admin] APPLY_SOCKET exists, is a socket, and is writable by this account — connecting to a UNIX socket needs write permission, and a doorbell that cannot be rung is not a capability.

Two stats and an access, no privilege required. The probe deliberately does not connect: connecting is the doorbell ring, and would start a root process on every page render.

Neither test is redundant, which is the detail most likely to be optimised away by someone reading the code later:

Without the unit-file test, a stale socket node reads as a live doorbell. Stale nodes are reachable in practice: systemd’s RemoveOnStop= defaults to off (the shipped unit sets it; an older or hand-edited one may not), and a package whose files are deleted rather than stopped leaves the node behind outright. That is precisely the scenario this feature exists for, and getting it wrong would fail open.
Without the socket test, a freshly installed package whose socket has never been enabled would read as ready, and every task would sit pending for ever.

The check fails closed. A path that cannot be examined, a path that is not a socket, a permission error, an unreadable parent directory — every uncertainty reads as “no applier” and the console goes read-only. The failure that produces is an operator being told to install something that is already installed; the failure in the other direction is an operator believing a change was made. Only the first is recoverable by reading the message, and the messages are written to be read: they distinguish not installed from installed but not started from there is a stale node here, and each names its own remedy.

4.2.4. What read-only looks like ¶

Everything is still rendered. Every setting the setup interview asks about is shown, with its staged answer or its default; every control that would change one is disabled, under a banner naming the missing package. Hiding the pages would be the easy reading of “read-only” and the wrong one — an operator who deliberately removed the applier still has to be able to see what the deployment is configured to do.

The API refuses the corresponding mutations with a status and a code of their own rather than a generic failure:

HTTP/1.1 503 Service Unavailable
{"code":   "setup_applier_unavailable",
 "hint":   "the administrative package 'pepsi-httpd-admin' is not installed …",
 "detail": {"package": "pepsi-httpd-admin", "unit": "pepsi-setup-apply.socket"}}

It applies to POST /api/v1/setup/tasks, /setup/preflight, /setup/dns-check, /setup/certificates and PUT /api/v1/setup/answers, and to the console’s own form posts — POST /ui/setup/{step}, /ui/setup/tasks and /ui/domains/check. They all reach the queue through one enqueuing function that asks first, so the refusal is a single decision rather than one per surface, and the disabled control on the page sits over that check rather than in place of it. Answer staging is included on purpose: a draft answer’s only purpose is to be applied, and a wizard that banks six steps of them on a host where nothing can apply them is exactly the silent no-op the design is avoiding. DELETE /api/v1/setup/answers is not gated — discarding drafts cannot cause a privileged change, and a read-only console still has to be able to clear a half-finished interview.

The code is distinct from setup_write_unavailable (which means there is no CONFIG_DB connection) because the two have different remedies, and the applier is reported first when both apply: configuring a database role would not have helped.

4.2.5. What still works with the package gone ¶

All of it, from a terminal. pepsi-setup is in the pepsi package, so:

pepsi-setup -c /etc/pepsi/pepsi.conf run
pepsi-setup -c /etc/pepsi/pepsi.conf --wizard
pepsi-setup -c /etc/pepsi/pepsi.conf check

behave exactly as they always did, as does editing pepsi.conf by hand, generating keys, running certbot yourself and every other administrative task. The applier subcommand is still there too, as noted above: what the package carries is the units, not the program.

What is removed is one thing only: the web tier’s ability to start it.

4.2.6. Arming it is a separate, deliberate step ¶

Installing pepsi-httpd-admin makes the capability available. It does not turn it on. Like every other Pepsi unit the two ship disabled, so a host that has the package but has never run

# systemctl enable --now pepsi-setup-apply.socket

has no doorbell, and the console stays read-only and says which command would change that. Installing a package and arming a privilege are two decisions, and Pepsi does not let one imply the other.

4.2.7. Installing it clears the queue first ¶

Removing the applier breaks the loop at the mechanism, and rows can still reach pepsi.setup_task while it is gone: an operator with a root shell can enqueue one by hand, and a console that is only told it is read-only is one release away from a path that forgets to ask. Nothing expires such a row. A request is durable, and the queue is drained in the order it was written.

Put those together and an applier that turns up months later would execute every one of them at the first doorbell ring, against a deployment that has moved on — a configuration written from an answer set nobody remembers, a certificate obtained for a host that has been renamed. Consent does not keep.

So installing pepsi-httpd-admin deletes the queue before its units are armed. The postinst runs

# pepsi-setup -c /etc/pepsi/pepsi.conf apply --clear

which removes every row without executing any of them and says so, naming the reason, so an operator who had legitimately queued something knows what became of it and can ask again. What was asked for is not lost either way: each request, refusal, success and failure is an independent durable entry in the audit log, which the reset does not touch. The same command is available by hand at any time — see pepsi-setup.

It is best effort, as a maintainer script must be. At install time there may be no configuration file, no schema, no database and no PostgreSQL running; each of those is reported and skipped, never a failed installation. (A database with no Pepsi schema is not even a skip: it has never held a task, so it is an empty queue.) The reset runs on every configure, not only on a first installation, because dpkg reports a re-installation after apt remove as an upgrade — and remove-then-reinstall is exactly the case this exists for.

4.2.8. The same lever from source ¶

A make install deployment gets the identical control:

make install INSTALL_ADMIN_UNITS=no

which installs everything except pepsi-setup-apply.service and pepsi-setup-apply.socket, and prints what it left out. make uninstall always removes both sets regardless of the variable’s value on that invocation: a privileged unit surviving an uninstall because a make variable changed is exactly the failure this split is about.

4.2.9. Choosing a posture ¶

The probe, the configuration option and the package set compose, so the question “can this console change this machine?” has more than two answers. [pepsi-admin] APPLIER is the override, documented in full in pepsi.conf(5):

Posture	How	Result
Administered from a browser	Package installed, socket enabled, `APPLIER = auto` (the default)	The console can apply setup tasks.
Installed but not armed	Package installed, socket never enabled	Read-only; the refusal names `systemctl enable --now pepsi-setup-apply.socket`.
Administered from a terminal	`apt remove pepsi-httpd-admin` (or `INSTALL_ADMIN_UNITS=no`)	Read-only; the refusal names the package, and nothing drains the task queue unattended.
Read-only by policy	`[pepsi-admin] APPLIER = no`, package left installed	Read-only by configuration rather than by which packages are installed, and the refusal says so. Reversible by editing one option — a policy, not a boundary.
Applied some other way	`pepsi-setup apply --once` from cron, or a hand-written unit, plus `[pepsi-admin] APPLIER = yes`	The console offers the forms. `yes` is a promise the operator makes; nothing verifies it.

Two of those rows deserve their difference spelled out. APPLIER = no and removing the package both produce a read-only console, and they are not the same defence. APPLIER = no is a line in a configuration file that pepsi-httpd consults; anyone who can change that file, or who can make pepsi-httpd read a different one, undoes it. Removing the package deletes the units that would have run the root process, and no amount of influence over pepsi-httpd puts them back. Use the option to state an intention; remove the package to enforce it.

The last row is a promise, not a check: with APPLIER = yes the console will happily queue tasks whether or not anything drains them. It exists for the deployments the probe genuinely cannot see — a cron drainer, a host without systemd, unit files installed somewhere systemd does not search — and it is the one place in this mechanism where the code takes an operator’s word for it.

There is one hand-made state the packaging cannot produce: the socket present and writable while the service unit is masked or broken. The probe reads that as available, the task is enqueued, and it stays pending. The task page shows that status rather than pretending otherwise. Both units ship in one package, so reaching this state takes deliberate effort.

4.3. See also ¶

Installation — installing either way, and the accounts, groups and privileged bits the pepsi package’s postinst creates. Security model — the threat model this split is a mitigation in, in particular what an attacker who compromises pepsi-httpd does and does not get. The administration console and The administrative API — the console and API surfaces that go read-only. pepsi-setup — the applier’s trust model, its closed task list and its admission gates. pepsi-httpd — the probe, the gated endpoints and [pepsi-admin] in reference form.