85.1.3. pepsi-httpd

serve MTA-STS policy, Web Key Directory, the Outlook add-in and metrics

Manual section:

1

85.1.3.1.1. Name

pepsi-httpd - HTTP/HTTPS server for MTA-STS, the Web Key Directory, the Outlook add-in and metrics.

85.1.3.1.2. Synopsis

pepsi-httpd [GLOBAL-OPTIONS] serve

pepsi-httpd [GLOBAL-OPTIONS] prune

85.1.3.1.3. Description

pepsi-httpd is the Pepsi HTTP/HTTPS server. It binds one or more listeners and routes each request through a generic dispatch table, matching by HTTP method and URL shape: a route’s path is a list of segments, each of which is a literal, a named capture of one segment, or a capture of everything that remains. Routes are tried in registration order and the first whose method and path match wins. Further endpoints are added as single table entries.

Each [pepsi-httpd-listener-<name>] section binds one socket — SERVE = tcp (BIND_TO/PORT, default port 443), unix (UNIXPATH) or systemd (socket activation) — with MODE = plain or tls. A TLS listener selects its certificate per connection from the client’s SNI host name: every [pepsi-httpd-cert-<name>] section lists one or more SNI host names with a TLS_CERT/TLS_KEY pair, and the listener’s own TLS_CERT/TLS_KEY (if any) is the fallback for connections that present no matching SNI.

When another web server already owns ports 80/443, pepsi-setup(1) instead configures a reverse-proxy deployment. The listener stays socket-activated (SERVE = systemd) but plain-HTTP (MODE = plain); the switch happens in the pepsi-httpd.socket unit instead: pepsi-setup drops a drop-in override (/etc/systemd/system/pepsi-httpd.socket.d/10-reverse-proxy.conf) that resets the shipped ListenStream=443 and rebinds it to a UNIX socket at /run/pepsi/httpd.sock (mode 0660, owned by the front server’s own group — www-data where that group exists, else nginx/apache/httpd), so systemd passes that socket’s fd to pepsi-httpd exactly as it would the port-443 socket. The front server terminates TLS and forwards the mta-sts.<domain> requests to it; in this mode pepsi-httpd holds no certificates of its own. Deleting the drop-in and running systemctl daemon-reload restores the direct :443 bind. See pepsi-setup(1) and --no-reverse-proxy.

85.1.3.1.4. Endpoints

GET /.well-known/mta-sts.txt

Return the MTA-STS policy (RFC 8461), built from the [pepsi] MTA_STS_* options and the [pepsi-ingress] HOSTNAME (the mx). The policy is served only when the request Host is mta-sts.<domain> for a domain in ACCEPTED_DOMAINS; any other host (or MTA_STS_MODE = none) yields 404. Serving it here saves hosting the policy file by hand; pepsi-setup(1) prints the _mta-sts TXT record to publish in DNS.

GET /mail/config-v1.1.xml

GET /.well-known/autoconfig/mail/config-v1.1.xml

Return the mail account autoconfiguration document (draft-ietf-mailmaint-autoconfig): the XML a mail client fetches, knowing only the user’s address, to discover which servers to use, on which ports, under which transport security and with which authentication.

These are the first two rungs of the fallback chain the draft defines, and the Host selects between them: the first is served when the request Host is autoconfig.<domain> (the required rung, and the one clients try first), the second when it is <domain> itself. In both cases DOMAIN must be in ACCEPTED_DOMAINS; any other host yields 404. The draft’s optional ?emailaddress= parameter is accepted and ignored — the document names %EMAILADDRESS%, the placeholder the client substitutes, so no request-controlled text is ever interpolated into the response.

The response is text/xml; charset=utf-8, cacheable for an hour, and — as the draft requires — public: it carries no authentication, because a client has to read it before it can know how to authenticate.

The document is built from [pepsi-autoconfig], and the endpoint answers 404 until that section names at least one incoming server (IMAP_HOST or POP3_HOST). The outgoing server is derived from the [pepsi-ingress-listener-*] flagged SUBMISSION = yes unless SMTP_HOST overrides it. Publishing requires an autoconfig.<domain> DNS record and a certificate covering it; see pepsi.conf(5).

GET /.well-known/openpgpkey/hu/HASH

GET /.well-known/openpgpkey/policy

GET /.well-known/openpgpkey/DOMAIN/hu/HASH

GET /.well-known/openpgpkey/DOMAIN/policy

The Web Key Directory (draft-koch-openpgp-webkey-service), in both its direct form (served under the domain itself) and its advanced form (served under openpgpkey.<domain>, with the domain repeated in the path). HASH is the z-base-32 SHA-1 of the lower-cased local part, which is stored and indexed alongside the domain, so a request costs one indexed lookup.

A key is returned only for a domain in ACCEPTED_DOMAINS and only from an identity that is published, active and OpenPGP. The response is the binary transferable public key — not armor — with Content-Type: application/octet-stream and Access-Control-Allow-Origin: * as the specification requires. An unknown hash yields 404 with an empty body.

The policy file is a 200 with a zero-length body. It carries no flags, but it must exist: GnuPG treats its absence as “this domain runs no Web Key Directory” and gives up before asking for a key.

The ?l=<local-part> parameter is read and ignored. Honouring it would turn the endpoint into a lookup by caller-supplied address rather than by a hash the caller already had to know.

The endpoint serves only this deployment’s own published identities (crypto_identity). It never serves a cached correspondent key (peer_key): that material was never verified by us and is not ours to publish under our own name. There is no option to change this.

There is deliberately no rate limiting and no uniform-404 shaping here. A Web Key Directory is a public oracle by construction, the hash covers only the local part (so it answers guesses rather than enumerating), and the addresses are on the outside of every message the domain sends. See pepsi-keys(1) and the manual’s key-management chapter.

For the advanced form, openpgpkey.<domain> must resolve to this host and the served certificate must cover the name; it is an SNI name on the same HTTPS listener, not a second listener. pepsi-setup(1) adds it to the certbot request and reports any that does not resolve.

GET /addin/manifest.xml

GET /addin/taskpane.html

The Outlook add-in: a manifest and a one-page task pane that let a user of a mail system behind this gateway ask for a message to be signed or encrypted. The pane sets X-Pepsi-Sign / X-Pepsi-Encrypt on the message being composed; pepsi-stage-encrypt(1) reads them and strips them before the message goes on the wire.

Both routes yield 404 unless [pepsi-httpd] ADDIN is set. Two placeholders are substituted per request: this gateway’s public https:// origin (from ADDIN_URL, else the request’s Host; the scheme is always https, because Outlook refuses to load add-in resources over plain HTTP – a configured http:// origin is refused at start-up unless its host is loopback, which is allowed for local testing) and a manifest identifier derived from that origin — stable for a deployment and distinct between deployments, so two gateways sideloaded into one Exchange organisation do not collide. A Host containing anything other than what a host name and port may contain is refused with 400 rather than escaped.

Distribution is by sideloading: the Exchange administrator points “Integrated apps” at the manifest URL. See the manual’s “Microsoft Exchange as a gateway” chapter.

GET /metrics

Pipeline statistics in the Prometheus text exposition format. Live gauges (pepsi_stage_active_messages, pepsi_pause_backlog) are read straight from the queue; counters (pepsi_stage_timeouts_total, pepsi_stage_crashes_total, pepsi_stage_messages_total, pepsi_stage_duration_seconds_total and the global pepsi_stages_executed_total / pepsi_messages_processed_total) come from the statistics tables that pepsi-dispatch(1) updates.

The endpoint is administrative: queue depths, per-stage crash and timeout counts, lifetime totals and every operator-chosen stage name together describe how much mail this deployment carries and how its pipeline is built. It is therefore served only on a listener flagged ADMIN = yes that may actually carry the administrative surface, and answers a plain 404 — what any unknown path gets — everywhere else. Unlike /api/v1 it asks for no credential, because a Prometheus scraper has none to present: the listener flag is the whole access control, so flag a listener only the monitoring system can reach.

Point the scraper at the administrative listener, or add ADMIN = yes to the listener it already scrapes (a plaintext one must be loopback or a UNIX socket — see The administrative surface below). Binding pepsi-httpd itself to a private address is not an alternative: the same process has to answer mta-sts.<domain> (RFC 8461) and openpgpkey.<domain> from the public internet, and there is no per-route listener binding. The first scrape refused because its listener is not flagged is logged with the listener’s name.

POST /resume

Release a paused message back into the pipeline: the matching row is set from paused to pending and the dispatcher is notified so the responsible stage runs again. This is the target of the GNU Taler pepsi-resume payment webhook (see pepsi-setup(1) and [pepsi-payments]), called when an order is paid.

The request must carry Authorization: Bearer <token> matching the configured RESUME_AUTHORIZATION_TOKEN (compared in constant time), and a JSON body {"message_id": "<token>"} naming the message’s external token (the merchant order_id). Responses: 200 when the message exists (a paused message was resumed, or it was already non-paused — the call is idempotent), 404 when the token is unknown, 401 on a failed authorization, 400 on a malformed body, 413 on a body over 4 KiB. The endpoint is disabled (404) when RESUME_AUTHORIZATION_TOKEN is not configured.

/api/v1/…

The administrative API — the queue, the health summary, the configuration, the key store, the audit log. Served only on a listener flagged ADMIN = yes, and only to an authenticated principal. See The administrative surface below and the manual’s “The administrative API” chapter for the endpoint reference, the scope table and the error shape.

/ui, /ui/…

The administration console: a server-rendered browser front end for the queue, the key store, the configuration and the logs. Served on the same ADMIN = yes listeners as /api/v1, under the same authentication, the same scope checks and the same audit log — it is a client of the API and adds no capability to it. The pages are

/ui (dashboard), /ui/login, /ui/logout, /ui/queue, /ui/queue/ID, /ui/queue/ID/{requeue,bounce,cancel}, /ui/identities, /ui/identities/ID, /ui/identities/ID/{publish,unpublish,vks,primary,revoke,delete}, /ui/identities/request/{generate,register}, /ui/peers, /ui/peers/ID/delete, /ui/ca-trust, /ui/ca-trust/ID/delete, /ui/config, /ui/config/SECTION, /ui/logs, /ui/logs/{mail,tls,dns}, /ui/domains, /ui/domains/check, /ui/secure, /ui/secure/TOKEN, /ui/secure/TOKEN/ACTION, /ui/setup, /ui/setup/STEP (GET and POST), /ui/setup/tasks (GET and POST), /ui/setup/tasks/ID and the single stylesheet /ui/static/console.css.

Waiting for a setup task. GET /api/v1/setup/tasks/ID takes ?wait=SECONDS (at most 60, combinable with ?since=SEQ) and answers as soon as the task has ended or has a progress line after SEQ, or when the time is up; the console’s task page waits the same way for up to 20 seconds per reload. Neither polls the database: the server keeps one LISTEN connection on the setup_task_done and setup_task_progress channels, which the applier’s writes notify with the task id, and wakes the requests waiting on that task. The connection is separate from the request pool and reconnects on its own with a growing delay; while it is down a wait returns at once rather than hanging, and the console page falls back to reloading every five seconds.

The telemetry switch. The interview’s SHARE_TELEMETRY question can only be answered yes while pepsi-telemetry-client(1) is running (dormant while telemetry is off) and [pepsi] SYSTEM_ID is configured, because this server never generates an identifier and a yes with no daemon to act on it would do nothing. It learns both from the daemon’s liveness row in pepsi.telemetry_client, which it may read and not write, and which carries a boolean for the identifier, never its value. Otherwise the checkbox is rendered disabled with directions (start the unit; add a SYSTEM_ID), GET /api/v1/setup/questions marks the question "disabled": true with a disabled_reason, and a yes through PUT /api/v1/setup/answers or a write-config task is refused with 409 telemetry_client_not_ready. GET /api/v1/setup/telemetry reports the same state (sharing, can_enable, reason, client). Switching telemetry off is never refused. The applier’s write-config notifies the daemon after writing the file, in either direction.

A reverse-proxy path allowlist has to include the /ui/setup* and /ui/secure* families too: the first is the whole browser setup interview, the second the secure-link administration pages.

The console ships no JavaScript and loads nothing from another host; every action is a form, every destructive one is confirmed, and every mutating form carries the session’s CSRF token as a hidden field (a mutating request whose Origin names another host is refused outright). It performs no service control — there is no restart, shutdown or backup button, deliberately. See the manual’s “The administration console” chapter.

/3.0/…, /3.1/…

The GNU Mailman 3 REST API — upstream version 3.3.10, reimplemented so that software written for GNU Mailman 3 (mailmanclient, Postorius, HyperKitty) works against Pepsi unchanged. Served only on a listener flagged LIST_API = yes, under the same binding rule as ADMIN (loopback plaintext, a UNIX socket, or TLS), and authenticated by HTTP Basic against the single [pepsi-list] API_USER/API_PASS pair. This is not /api/v1: different audience, different collection envelope, different error shape, no scopes. See the manual’s “The GNU Mailman 3 REST API” chapter.

LIST_API_TESTING = yes additionally arms GET /3.1/reserved/reset, which truncates every mailing-list and archive table. It exists for the compatibility test suites and must not be set on a deployment carrying real mail.

GET /favicon.ico

GET /favicon-VERSION.svg

The Pepsi badge as the browser icon, served on every listener whatever its flags: an icon says nothing about the deployment. The console, its login page and the mailing-list pages link the SVG, whose name carries the start of its SHA-256; since a new icon is a new URL, it is sent Cache-Control: public, max-age=31536000, immutable. /favicon.ico (16, 32 and 48 px) is what a browser fetches by itself for a page that links no icon; its URL cannot change, so it is cacheable for a week. Both carry an ETag and answer a matching If-None-Match with 304. The files are compiled into the binary.

/lists, /lists/…, /archives/list/…, /robots.txt

The public mailing-list web interface: the list index, a list’s information page, the subscribe and unsubscribe forms, the confirmation pages, the RFC 8058 one-click unsubscribe target, the archive (on HyperKitty’s URL shapes, so a migrated deployment’s Archived-At: links keep working) and its search. Served only on a listener flagged LISTS = yes, to anybody, with no credential.

This is the one flag whose deployment advice is the opposite of the other two. ADMIN is for the operator and LIST_API carries one shared password, so both refuse to be served where their credentials would cross a cleartext network; this surface exists to be reached from the open internet, so that restriction is deliberately not applied to it. A plaintext LISTS listener is the operator’s choice and gets a log line rather than a refusal — but a subscribe form carries somebody’s address, so put it on a TLS listener.

These pages ship no JavaScript, and their Content-Security-Policy has no script-src at all as a result. /robots.txt allows the archive and disallows search, which is the one route that runs a full-text query per request; search is additionally rate-limited on its own budget, because a crawler that ignores the file still has to be survivable.

Both budgets are requests per minute per source address (an IPv6 client per /64), read from the [pepsi-list] section:

WEB_RATE_LIMIT

(integer, optional) Every route of this interface and of the member tier below. Default 600.

WEB_SEARCH_RATE_LIMIT

(integer, optional) The search route, on top of the above. Default 30.

A value below 1 is raised to 1: neither can be switched off. Behind a reverse proxy every request arrives over the UNIX socket with no client address, so all clients share one budget; raise the limits there and let the proxy limit per client.

The same flag also serves the member tier — /lists/register, /lists/verify/<token>, /lists/sign-in, /lists/sign-out, /lists/reset and /lists/me — and the owner and moderator console at /lists/<list-id>/admin.... Neither is a second flag: an account system nobody can reach is not a deployment choice anybody would make.

Those pages use pepsi.list_session and the cookies pepsi_list_session/pepsi_list_csrf, which are not the operator console’s pepsi_session/pepsi_csrf and grant nothing on it — nor the reverse. Authority over a list is a roster query, not an account flag: a list_member row with role = owner or moderator for that list, plus list_user.is_server_owner as the only global authority in the tier. A caller without it gets the ordinary not-found page, because on a public surface a refusal that distinguishes itself has disclosed that the thing exists. See the manual’s “Two account systems” section.

85.1.3.1.5. The administrative surface

A listener carrying ADMIN = yes additionally serves /api/v1, the /ui console and /metrics. On every other listener those routes answer a plain 404, byte-for-byte what any unknown path gets, so publishing the public listener does not publish administration and a public listener cannot be probed for whether administration is enabled on this deployment.

/api/v1 and /ui additionally authenticate and authorise every request, as below. /metrics does not: a scraper has no credential to present, so the listener flag is all that stands in front of it.

pepsi-httpd refuses to serve them on a flagged listener that would carry them in cleartext off this host: plaintext TCP is accepted only on a loopback address, while TLS listeners and UNIX sockets always qualify. A socket-activated listener (SERVE = systemd) never qualifies without TLS, whatever its unit binds — including a ListenStream= on loopback or on a UNIX path — because the inherited descriptor’s address is not visible from here and treating it as permitted would let the strictest case through the loosest check. Since the shipped deployment is socket activated, administration wants a listener of its own (a UNIX socket, or a TLS one). A listener that fails the test logs a warning at start-up and serves the public endpoints only; pepsi-setup(1) reports the same at configuration time.

Three authentication mechanisms resolve to one identity model, tried in this order — an explicitly presented credential always wins, so a deliberately narrow token is never silently widened:

1. Authorization: Bearer pepsi_<id>.<secret> — a row in pepsi.api_token. Only the digest of the secret half is stored.

2. A session cookie from POST /api/v1/auth/login — or from the console’s own POST /ui/login form, which mints the same session — with a CSRF token required on every mutating request. Sessions have a 30-minute idle timeout and a 12-hour hard lifetime by default ([pepsi-admin] SESSION_IDLE and SESSION_LIFETIME).

3. SO_PEERCRED on a UNIX-socket listener: root, or a member of [pepsi-admin] ADMIN_GROUP, is an administrator with no credential at all. This is what makes first-run bootstrap work and what the shipped configuration relies on.

PAM is deliberately not supported: it would put a privileged authentication path and system-account semantics inside the mail server’s web tier.

Authorisation is by scope, declared per endpoint (config:read, config:write, keys:read, keys:write, peers:write, queue:read, queue:write, logs:read, setup:write, secure:read, secure:write, plus own:<address> for a principal confined to one address). GET /api/v1/openapi.json is generated from the server’s own route table and is the authoritative list for a given build.

secure:read and secure:write are separate because revoking a secure-link message destroys the only copy of it; an irreversible deletion does not belong behind a capability named “read”.

Generating or revoking key material is never done here: private material belongs to the pepsi-crypto database role alone. Generating a server-managed key (POST /api/v1/identities), registering a user’s own public key (POST /api/v1/identities/client) and revoking an identity (PATCH /api/v1/identities/{id} with {"status": "revoked"} and an optional revocation_reason, which destroys a signing-only key’s private half) are instead asked for, as generate-identity, register-client-key and revoke-identity rows in pepsi.setup_task that pepsi-setup apply re-validates and performs as pepsi-crypto (see below). A revoking PATCH is therefore asynchronous: it carries status alone and answers with the queued task; the identity reads revoked once the applier has run. Registering involves no private material, but a key registered as the user’s own becomes their public face and verifies signatures as theirs, so a compromised web tier must not be able to plant one. Producing a CSR and importing an issued certificate are not offered over HTTP; use pepsi-keys(1).

Self-service on one’s own keys. A principal confined to one address (own:<address>) may reach, for that address only, the three key writes of the pEp-style self-service: POST /api/v1/identities ({"address": ..., "vks": true|false}, vks optional and defaulting to [pepsi-keys] VKS_PUBLISH), POST /api/v1/identities/client ({"address": ..., "key": "<armoured OpenPGP public key>"}, at most 64 KiB) and PATCH /api/v1/identities/{id} (whose vks_wanted field, when true, records a key-server upload request for an active OpenPGP identity and publishes it over WKD; the retry job pepsi-keys identity publish --retry makes the upload; and whose status: revoked asks for a revocation). Each handler re-checks the address; another address’s identity answers 404. The two POST routes and a revoking PATCH answer with the queued task ({"task": ..., "kind": ..., "applier_notified": ...}), not with the key, and the principal that asked may follow that task: GET /api/v1/setup/tasks/ID answers any principal for a task it queued itself (404 for anybody else’s, as for one that does not exist; ownership is the task’s requester_key – the token’s selector, the account’s row number or the peer’s login, none of them ever reused – not the requested_by display name), and GET /api/v1/setup/tasks lists only those tasks to a principal without setup:write. The console’s key forms answer with the task’s page, /ui/setup/tasks/ID, on the same terms. DELETE /api/v1/identities/{id} stays operator-only (keys:write): a user retires a key, they do not erase its row. The console offers the same through the Generate a server-managed key and Register my own key forms on the identities page and the Upload to the key server and Revoke actions on an identity’s page.

The applier admits these three task kinds with its own authorisation gate: the requesting principal must have held keys:write, or own:<address> for exactly the address named in the task’s parameters (setup:write alone is not enough). It then requires the address to be at a [pepsi-ingress] ACCEPTED_DOMAINS domain and, for a registration, one OpenPGP certificate whose User IDs name the address; for a revocation, the identity must belong to that address. Without the applier all three answer 503 setup_applier_unavailable, like every other applier path; enqueuing also needs the [pepsi-admin] CONFIG_DB connection (503 setup_write_unavailable otherwise).

The second factor. Each of the three bodies (and the console’s forms) takes an optional "otp": "123456", the address owner’s current second-factor code (see pepsi-keys(1), otp). This server cannot check it – it holds no grant on pepsi.otp_key at all – so it only passes the code on in the task’s parameters, where the applier checks it for a task admitted on own:<address> alone, before acting; a missing, wrong, replayed or locked code fails the task with the reason. A principal holding keys:write is not asked. DELETE /api/v1/otp/ADDRESS (keys:write, never reachable under own:) queues a reset-otp task that removes the address’s second factor, which is how a locked one is unlocked. The synchronous flag changes of PATCH /api/v1/identities/{id} (is_primary, published, vks_wanted) are performed here and so are not second-factor gated.

Two operations this server deliberately refuses: writing the configuration overlay (503 unless [pepsi-admin] CONFIG_DB names a connection authenticating as the pepsi-config role, which the server verifies at start-up), and every privileged setup request when the applier package is not installed (503, see The privileged applier, and doing without it below).

The retention policy of the audit log and, when it is enabled, the mail log is applied by this binary’s prune command, not by the server: every mail-processing account may append to those tables and not erase from them, and the pepsi-httpd role is the one service role that holds DELETE on them (the only other holder is the operator’s pepsi-config role). The shipped pepsi-log-prune.timer runs it daily as that account, so retention holds whether or not the web server is running — it used to be a background task of serve, started only when a listener served the administrative surface, and a deployment without the console kept every record for ever.

Note

One binary serves the public and the administrative surface. A bug in the shared server process is therefore a bug in both; the isolation is the operator’s to configure — bind the administrative listener to a UNIX socket or to loopback.

GET /secure/TOKEN

The secure-link fallback portal’s entry page. Without a session it asks for the PIN; with a valid session cookie it renders the message. Answers 404 when the token is unknown (or the portal is not configured), 410 when the message has expired and 429 while the token is locked out. See the manual’s secure-link chapter and pepsi-stage-secure-link(1).

POST /secure/TOKEN

Verify the PIN. On success it sets the session cookie and redirects (303) to the page above; on a wrong PIN it re-renders the form with 401, and on the attempt that exhausts MAX_ATTEMPTS it answers 429 and locks the token for a period that doubles with each further lockout. A submission carrying no PIN is answered 401 as well, but is not counted against MAX_ATTEMPTS: it is not a guess, and a client that re-submits the form empty would otherwise spend a legitimate recipient’s attempts for them. The PIN is submitted in the request body and never appears in a URL.

GET /secure/TOKEN/part/N

Download attachment N of the message. Authorised by the session, never by the PIN. Always served Content-Type: application/octet-stream with Content-Disposition: attachment and a sandboxing Content-Security-Policy, whatever media type the message claimed — a text/html part rendered inline from the portal’s own origin is precisely what the design refuses.

POST /secure/TOKEN/reply

Compose a reply, as multipart/form-data (a text field and optional files). Authorised by the session. The reply’s recipient and envelope sender are taken from the stored row and are not inputs; the body is always text/plain; uploads are capped by MAX_REPLY_SIZE / MAX_REPLY_FILES while the body streams; and the injected message deliberately does not carry state.local_origin, so it cannot inherit submission privileges. Answers 404 when REPLY_STAGE is unset (replying is opt-in per deployment).

Every portal response carries, per route rather than per host, Content-Security-Policy: default-src 'none' (with a per-response nonce for the one inline stylesheet and no script source at all), Referrer-Policy: no-referrer, X-Content-Type-Options: nosniff, X-Frame-Options: DENY and Cache-Control: no-store. The session cookie is Path=/secure/-scoped, HttpOnly, Secure and SameSite=Strict, so sharing an origin with the endpoints above is safe by construction rather than by deployment discipline.

85.1.3.1.6. The privileged applier, and doing without it

pepsi-httpd never performs a privileged change. It records an intent row in pepsi.setup_task and connects to a doorbell socket; systemd’s socket activation turns that connection into a short-lived root pepsi-setup apply, which validates the row and does the work. The full trust model is in pepsi-setup(1), The applier’s trust model.

That makes the capability removable, and Debian packages it separately:

pepsi

The mail server proper: the binaries — including pepsi-setup itself — the schema, the templates and every other unit (the separately packaged pepsi-stage-detect-language and pepsi-telemetry aside). A deployment administered from a terminal needs nothing else.

pepsi-httpd-admin

pepsi-setup-apply.socket and pepsi-setup-apply.service, and nothing else. It is what turns an HTTP request into a change under /etc/pepsi. pepsi Recommends it, so a default installation has it; apt remove pepsi-httpd-admin succeeds without touching pepsi.

With that package removed there is nothing to drain pepsi.setup_task. The loop is broken at the mechanism, not at the button: an insert into that table becomes inert, whatever happens to the web tier. (A source install gets the same lever from make install INSTALL_ADMIN_UNITS=no.)

Inert, and it stays that way: putting the package back does not act on whatever accumulated while it was gone. Its postinst runs pepsi-setup apply --clear, which deletes every queued row without executing one, so arming the applier is never also the event that runs a request nobody remembers making. See pepsi-setup(1).

85.1.3.1.6.1. What the console does then

Read-only, and it says so. Every setting the setup interview asks about is still rendered, with its staged answer or its default; every control that would change one is disabled, under a banner naming the missing package. The pages are not hidden: an operator who deliberately removed the applier still has to be able to see what the deployment is configured to do.

The API refuses the corresponding mutations with a status and a code of their own rather than a generic failure:

HTTP/1.1 503 Service Unavailable
{"code": "setup_applier_unavailable",
 "hint":  "the administrative package 'pepsi-httpd-admin' is not installed …",
 "detail": {"package": "pepsi-httpd-admin", "unit": "pepsi-setup-apply.socket"}}

It applies to POST /api/v1/setup/tasks, /setup/preflight, /setup/dns-check, /setup/certificates, PUT /api/v1/setup/answers, POST /api/v1/identities, POST /api/v1/identities/client, a revoking PATCH /api/v1/identities/{id} and DELETE /api/v1/otp/{address}, and to the console’s own form posts — POST /ui/setup/{step}, /ui/setup/tasks, /ui/identities/request/{generate,register}, /ui/identities/{id}/revoke and /ui/domains/check, which is the “check now” button on the domains page. Every one of them reaches the queue through a single enqueuing function that asks first, so the refusal is one decision rather than one per surface; the disabled control on the page is presentation over that check, not a substitute for it. Answer staging is included deliberately: a draft answer’s only purpose is to be applied, and a wizard that banks six steps of them on a host where nothing can apply them is a silent no-op. DELETE /api/v1/setup/answers is not gated — discarding drafts cannot cause a privileged change, and a read-only console still has to be able to clear a half-finished interview.

Note the distinct code. setup_write_unavailable (no CONFIG_DB connection) and setup_applier_unavailable have different remedies, and the applier is reported first when both apply: configuring a database role would not have helped.

What is not affected is /api/v1/config and the console’s configuration pages. Those write pepsi.config_override, the database overlay every component reads at run time — not a file in /etc. Their only gate is [pepsi-admin] CONFIG_DB, because the applier has nothing to do with them and widening the read-only rule to cover them would remove a capability the package split was not about. The two layers are described under Configuration layering in the manual.

85.1.3.1.6.2. How it detects this

By looking at two files, which between them answer installed and armed:

1. pepsi-setup-apply.socket exists as a unit file in a directory systemd searches (/etc/systemd/system, /run/systemd/system, /usr/local/lib/systemd/system, /usr/lib/systemd/system, /lib/systemd/system); and

2. [pepsi-admin] APPLY_SOCKET exists, is a socket, and is writable by this account — connecting to a UNIX socket needs write permission.

Two stats and an access, no privilege — which matters, because the server has already dropped to the unprivileged pepsi-httpd account and cannot ask dpkg or systemd anything. It deliberately does not connect: connecting is the doorbell, and would start a root process on every page render.

Neither test is redundant. Without the first, a stale socket node reads as a live doorbell — and stale nodes happen: systemd’s RemoveOnStop= defaults to off (the shipped unit sets it; a hand-edited one may not), and a package whose files are deleted rather than stopped leaves the node behind outright. That is exactly the case this feature exists for, and getting it wrong would fail open. Without the second, a freshly installed package whose socket has never been enabled would read as ready and every task would sit pending for ever.

The check fails closed. A path that cannot be examined, a path that is not a socket, a permission error — every uncertainty reads as “not available” and the console goes read-only. The recoverable failure is being told to install something that is already installed; the unrecoverable one is believing a change was made.

Two consequences:

• A site that drains the queue some other way — pepsi-setup apply --once from cron, a hand-written unit, a host without systemd, unit files somewhere systemd does not search — has no doorbell (or no unit file) and reads as unavailable. [pepsi-admin] APPLIER = yes overrides the probe. It is a promise the operator makes; nothing here can check it.

• The converse: the socket can be present while the service unit is masked or broken, in which case a task is enqueued and stays pending. The task page shows that status rather than pretending otherwise. Both units ship in one package, so this is a hand-made state rather than a packaging outcome.

[pepsi-admin] APPLIER = no makes the console read-only by configuration alone, for a deployment that cannot change its package set.

85.1.3.1.7. Configuration

The following are documented in pepsi.conf(5): the [pepsi-httpd] section’s options (MAX_CONNECTIONS, MAX_CONNECTIONS_PER_IP, DB_POOL_SIZE, the ADDIN / ADDIN_URL pair that enables the Outlook add-in, and the optional RESUME_AUTHORIZATION_TOKEN that guards POST /resume), the [pepsi-httpd-listener-<name>] and [pepsi-httpd-cert-<name>] socket and SNI-certificate sections (including the listener flags ADMIN, LIST_API, LIST_API_TESTING and LISTS), the [pepsi-admin] section that configures the administrative API, the [pepsi-list] section that holds the Mailman API’s API_USER/API_PASS credential and API_RATE_LIMIT and the web interface’s WEB_RATE_LIMIT/WEB_SEARCH_RATE_LIMIT, the [pepsi-autoconfig] section the autoconfiguration document is built from, and the [pepsi-secure-link] section that enables the portal.

85.1.3.1.8. Privilege dropping

To bind privileged ports (80/443) pepsi-httpd is often started as root. It does not serve as root: once every configured listener has been bound (and any TLS key material read), the process drops to the unprivileged service account pepsi-httpd, dropping all supplementary groups and switching to that account’s group and user id before accepting a single connection. The account must therefore exist before the server is started as root; create it (for example useradd --system --no-create-home --shell /usr/sbin/nologin pepsi-httpd) as part of installation.

If the drop cannot be completed while running as root — most often because the pepsi-httpd account does not exist — the server logs an error and exits without serving, rather than risk running as root. When started by a non-root user no privilege change is performed (the server simply runs as the invoking user); it will never run as root.

85.1.3.1.9. TLS key material

The certificate and private key of every TLS listener — its own TLS_CERT/TLS_KEY and each [pepsi-httpd-cert-*] SNI certificate — are read once, at start-up, before the privilege drop described above. Where they are read from depends on how the server was started.

Under systemd (the packaged deployment). The unit runs as the unprivileged pepsi-httpd user from the outset — it never holds root — and so cannot open certbot’s root-only /etc/letsencrypt/{live,archive}. It does not have to: pepsi-setup(1) writes a drop-in /etc/systemd/system/pepsi-httpd.service.d/10-tls-credentials.conf with one LoadCredential= entry per file. systemd opens those files as root when it starts the unit and passes private copies to the service under $CREDENTIALS_DIRECTORY — a per-unit tmpfs directory readable by this one service and invisible to every other unit — where pepsi-httpd looks each configured path up before falling back to reading the path itself. The configuration is untouched by this: TLS_CERT/TLS_KEY still name the real files, and the credential names are derived from those paths.

Two consequences:

• Re-run pepsi-setup run after adding, moving or removing a certificate. The drop-in is regenerated from the configuration and deliberately lists only files that exist, because systemd refuses to start a unit whose credential source is missing.

• A credential is materialised when the unit starts, so a renewed certificate reaches clients only after a restart. The certbot deploy hook pepsi-setup installs performs it (systemctl try-restart).

Started directly, without systemd. The server is started as root, reads the files itself and only then drops privileges, so no credential is involved and the file permissions never matter.

A certificate that cannot be loaded is reported and skipped, not fatal: the SNI host names it would have served stop being available over HTTPS while every other certificate keeps working. Only a TLS listener left without a single usable certificate makes the server exit. One unreadable file therefore cannot take the MTA-STS policy of unrelated domains — or the metrics endpoint — down with it.

85.1.3.1.10. Commands

serve

Run the server until interrupted. Requires that the schema has been installed with pepsi-setup(1).

prune

Delete audit records older than [pepsi-admin] EVENT_RETENTION_DAYS, mail-log records older than MAIL_LOG_RETENTION_DAYS and expired administrative sessions, then exit. Run as root, it first drops to the pepsi-httpd account, whose database role holds the DELETE grant. pepsi-log-prune.timer runs it daily; on a system without systemd, run it from cron.

85.1.3.1.11. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

85.1.3.1.12. Signals

SIGINT

Initiate shutdown and exit cleanly.

SIGTERM

Not caught: the default disposition ends the process at once, cutting any request in flight. Nothing is lost by that — the queue is in PostgreSQL and this server keeps no state of its own — and it is how systemctl stop ends the unit.

85.1.3.1.13. Exit Status

0

Clean shutdown.

1

An error occurred (for example a malformed configuration file, an unreadable certificate, or a failed database connection). The reason is written to the log.

85.1.3.1.14. Examples

Serve over HTTPS:

pepsi-httpd -c /etc/pepsi/pepsi.conf serve

Fetch a policy for testing (resolving the SNI host to the server):

curl --resolve mta-sts.example.org:443:203.0.113.7 \
     https://mta-sts.example.org/.well-known/mta-sts.txt

Check that a user’s key is published, the way a correspondent’s client would:

gpg --locate-external-key alice@example.org

85.1.3.1.15. See Also

pepsi-config(1), pepsi.conf(5), pepsi-dispatch(1), pepsi-keys(1), pepsi-setup(1)

85.1.3.1.16. Bugs

Report bugs to the Pepsi issue tracker.