70.1.3. pepsi-httpd

serve MTA-STS policy, Web Key Directory, the Outlook add-in and metrics

Manual section:

1

70.1.3.1.1. Name

pepsi-httpd - HTTP/HTTPS server for MTA-STS, the Web Key Directory, the Outlook add-in and metrics.

70.1.3.1.2. Synopsis

pepsi-httpd [GLOBAL-OPTIONS] serve

70.1.3.1.3. Description

pepsi-httpd is the Pepsi HTTP/HTTPS server. It binds one or more listeners and routes each request through a generic dispatch table, matching by HTTP method and URL shape: a route’s path is a list of segments, each of which is a literal, a named capture of one segment, or a capture of everything that remains. Routes are tried in registration order and the first whose method and path match wins. It answers a handful of endpoints and is designed so further endpoints are added as single table entries.

Each [pepsi-httpd-listener-<name>] section binds one socket — SERVE = tcp (BIND_TO/PORT, default port 443), unix (UNIXPATH) or systemd (socket activation) — with MODE = plain or tls. A TLS listener selects its certificate per connection from the client’s SNI host name: every [pepsi-httpd-cert-<name>] section lists one or more SNI host names with a TLS_CERT/TLS_KEY pair, and the listener’s own TLS_CERT/TLS_KEY (if any) is the fallback for connections that present no matching SNI.

When another web server already owns ports 80/443, pepsi-setup(1) instead configures a reverse-proxy deployment. The listener stays socket-activated (SERVE = systemd) but plain-HTTP (MODE = plain); the switch happens in the pepsi-httpd.socket unit instead: pepsi-setup drops a drop-in override (/etc/systemd/system/pepsi-httpd.socket.d/10-reverse-proxy.conf) that resets the shipped ListenStream=443 and rebinds it to a UNIX socket at /run/pepsi/httpd.sock (SocketGroup=www-data), so systemd passes that socket’s fd to pepsi-httpd exactly as it would the port-443 socket. The front server terminates TLS and forwards the mta-sts.<domain> requests to it; in this mode pepsi-httpd holds no certificates of its own. Deleting the drop-in and running systemctl daemon-reload restores the direct :443 bind. See pepsi-setup(1) and --no-reverse-proxy.

70.1.3.1.4. Endpoints

GET /.well-known/mta-sts.txt

Return the MTA-STS policy (RFC 8461), built from the [pepsi] MTA_STS_* options and the [pepsi-ingress] HOSTNAME (the mx). The policy is served only when the request Host is mta-sts.<domain> for a domain in ACCEPTED_DOMAINS; any other host (or MTA_STS_MODE = none) yields 404. This replaces hosting the policy file by hand; pepsi-setup(1) still prints the _mta-sts TXT record to publish in DNS.

GET /mail/config-v1.1.xml

GET /.well-known/autoconfig/mail/config-v1.1.xml

Return the mail account autoconfiguration document (draft-ietf-mailmaint-autoconfig): the XML a mail client fetches, knowing only the user’s address, to discover which servers to use, on which ports, under which transport security and with which authentication.

These are the first two rungs of the fallback chain the draft defines, and the Host selects between them: the first is served when the request Host is autoconfig.<domain> (the required rung, and the one clients try first), the second when it is <domain> itself. In both cases DOMAIN must be in ACCEPTED_DOMAINS; any other host yields 404. The draft’s optional ?emailaddress= parameter is accepted and ignored — the document names %EMAILADDRESS%, the placeholder the client substitutes, so no request-controlled text is ever interpolated into the response.

The response is text/xml; charset=utf-8, cacheable for an hour, and — as the draft requires — public: it carries no authentication, because a client has to read it before it can know how to authenticate.

The document is built from [pepsi-autoconfig], and the endpoint answers 404 until that section names at least one incoming server (IMAP_HOST or POP3_HOST). The outgoing server is derived from the [pepsi-ingress-listener-*] flagged SUBMISSION = yes unless SMTP_HOST overrides it. Publishing requires an autoconfig.<domain> DNS record and a certificate covering it; see pepsi.conf(5).

GET /.well-known/openpgpkey/hu/HASH

GET /.well-known/openpgpkey/policy

GET /.well-known/openpgpkey/DOMAIN/hu/HASH

GET /.well-known/openpgpkey/DOMAIN/policy

The Web Key Directory (draft-koch-openpgp-webkey-service), in both its direct form (served under the domain itself) and its advanced form (served under openpgpkey.<domain>, with the domain repeated in the path). HASH is the z-base-32 SHA-1 of the lower-cased local part, which is stored and indexed alongside the domain, so a request costs one indexed lookup.

A key is returned only for a domain in ACCEPTED_DOMAINS and only from an identity that is published, active and OpenPGP. The response is the binary transferable public key — not armor — with Content-Type: application/octet-stream and Access-Control-Allow-Origin: * as the specification requires. An unknown hash yields 404 with an empty body.

The policy file is a 200 with a zero-length body. It carries no flags, but it must exist: GnuPG treats its absence as “this domain runs no Web Key Directory” and gives up before asking for a key.

The ?l=<local-part> parameter is read and ignored. Honouring it would turn the endpoint into a lookup by caller-supplied address rather than by a hash the caller already had to know.

The endpoint serves only this deployment’s own published identities (crypto_identity). It never serves a cached correspondent key (peer_key): that material was never verified by us and is not ours to publish under our own name. There is no option to change this.

There is deliberately no rate limiting and no uniform-404 shaping here. A Web Key Directory is a public oracle by construction, the hash covers only the local part (so it answers guesses rather than enumerating), and the addresses are on the outside of every message the domain sends. See pepsi-keys(1) and the manual’s key-management chapter.

For the advanced form, openpgpkey.<domain> must resolve to this host and the served certificate must cover the name; it is an SNI name on the same HTTPS listener, not a second listener. pepsi-setup(1) adds it to the certbot request and reports any that does not resolve.

GET /addin/manifest.xml

GET /addin/taskpane.html

The Outlook add-in: a manifest and a one-page task pane that let a user of a mail system behind this gateway ask for a message to be signed or encrypted. The pane sets X-Pepsi-Sign / X-Pepsi-Encrypt on the message being composed; pepsi-stage-encrypt(1) reads them and strips them before the message goes on the wire.

Both routes yield 404 unless [pepsi-httpd] ADDIN is set. Two placeholders are substituted per request: this gateway’s public https:// origin (from ADDIN_URL, else the request’s Host; the scheme is always https, because Outlook refuses to load add-in resources over plain HTTP) and a manifest identifier derived from that origin — stable for a deployment and distinct between deployments, so two gateways sideloaded into one Exchange organisation do not collide. A Host containing anything other than what a host name and port may contain is refused with 400 rather than escaped.

Distribution is by sideloading: the Exchange administrator points “Integrated apps” at the manifest URL. See the manual’s “Microsoft Exchange as a gateway” chapter.

GET /metrics

Pipeline statistics in the Prometheus text exposition format. Live gauges (pepsi_stage_active_messages, pepsi_pause_backlog) are read straight from the queue; counters (pepsi_stage_timeouts_total, pepsi_stage_crashes_total, pepsi_stage_messages_total, pepsi_stage_duration_seconds_total and the global pepsi_stages_executed_total / pepsi_messages_processed_total) come from the statistics tables that pepsi-dispatch(1) updates. The endpoint is unauthenticated; restrict access by binding it to a private/localhost listener.

POST /resume

Release a paused message back into the pipeline: the matching row is set from paused to pending and the dispatcher is notified so the responsible stage runs again. This is the target of the GNU Taler pepsi-resume payment webhook (see pepsi-setup(1) and [pepsi-payments]), called when an order is paid.

The request must carry Authorization: Bearer <token> matching the configured RESUME_AUTHORIZATION_TOKEN (compared in constant time), and a JSON body {"message_id": "<token>"} naming the message’s external token (the merchant order_id). Responses: 200 when the message exists (a paused message was resumed, or it was already non-paused — the call is idempotent), 404 when the token is unknown, 401 on a failed authorization, 400 on a malformed body. The endpoint is disabled (404) when RESUME_AUTHORIZATION_TOKEN is not configured.

/api/v1/…

The administrative API — the queue, the health summary, the configuration, the key store, the audit log. Served only on a listener flagged ADMIN = yes, and only to an authenticated principal. See The administrative surface below and the manual’s “The administrative API” chapter for the endpoint reference, the scope table and the error shape.

/ui, /ui/…

The administration console: a server-rendered browser front end for the queue, the key store, the configuration and the logs. Served on the same ADMIN = yes listeners as /api/v1, under the same authentication, the same scope checks and the same audit log — it is a client of the API and adds no capability to it. The pages are

/ui (dashboard), /ui/login, /ui/logout, /ui/queue, /ui/queue/ID, /ui/queue/ID/{requeue,bounce,cancel}, /ui/identities, /ui/identities/ID, /ui/identities/ID/{publish,unpublish,primary,revoke,delete}, /ui/peers, /ui/peers/ID/delete, /ui/ca-trust, /ui/ca-trust/ID/delete, /ui/config, /ui/config/SECTION, /ui/logs, /ui/logs/{mail,tls,dns}, /ui/domains and the single stylesheet /ui/static/console.css.

The console ships no JavaScript and loads nothing from another host; every action is a form, every destructive one is confirmed, and every mutating form carries the session’s CSRF token as a hidden field (a mutating request whose Origin names another host is refused outright). It performs no service control — there is no restart, shutdown or backup button, deliberately. See the manual’s “The administration console” chapter.

70.1.3.1.5. The administrative surface

A listener carrying ADMIN = yes additionally serves /api/v1 and the /ui console. On every other listener those routes answer a plain 404, byte-for-byte what any unknown path gets, so publishing the public listener does not publish administration and a public listener cannot be probed for whether administration is enabled on this deployment.

pepsi-httpd refuses to serve them on a flagged listener that would carry them in cleartext off this host: plaintext TCP is accepted only on a loopback address, while TLS listeners and UNIX sockets always qualify. A listener that fails the test logs a warning at start-up and serves the public endpoints only; pepsi-setup(1) reports the same at configuration time.

Three authentication mechanisms resolve to one identity model, tried in this order — an explicitly presented credential always wins, so a deliberately narrow token is never silently widened:

  1. Authorization: Bearer pepsi_<id>.<secret> — a row in pepsi.api_token. Only the digest of the secret half is stored.

  2. A session cookie from POST /api/v1/auth/login, with a CSRF token required on every mutating request. Sessions have a 30-minute idle timeout and a 12-hour hard lifetime.

  3. SO_PEERCRED on a UNIX-socket listener: root, or a member of [pepsi-admin] ADMIN_GROUP, is an administrator with no credential at all. This is what makes first-run bootstrap work and what the shipped configuration relies on.

PAM is deliberately not supported: it would put a privileged authentication path and system-account semantics inside the mail server’s web tier.

Authorisation is by scope, declared per endpoint (config:read, config:write, keys:read, keys:write, peers:write, queue:read, queue:write, logs:read, setup:write, secure:read, secure:write, plus own:<address> for a principal confined to one address). GET /api/v1/openapi.json is generated from the server’s own route table and is the authoritative list for a given build.

secure:read and secure:write are separate because revoking a secure-link message destroys the only copy of it; an irreversible deletion does not belong behind a capability named “read”.

Three operations this server deliberately refuses: generating key material (501; private material belongs to the pepsi-crypto database role alone — use pepsi-keys(1)), writing the configuration overlay (503 unless [pepsi-admin] CONFIG_DB names a connection authenticating as the pepsi-config role, which the server verifies at start-up), and every privileged setup request when the applier package is not installed (503, see The privileged applier, and doing without it below).

pepsi-httpd also applies the retention policy of the audit log and, when it is enabled, the mail log: it is the one component that holds DELETE on those tables and does not process mail.

Note

One binary serves the public and the administrative surface. A bug in the shared server process is therefore a bug in both; the isolation is the operator’s to configure — bind the administrative listener to a UNIX socket or to loopback.

GET /secure/TOKEN

The secure-link fallback portal’s entry page. Without a session it asks for the PIN; with a valid session cookie it renders the message. Answers 404 when the token is unknown (or the portal is not configured), 410 when the message has expired and 429 while the token is locked out. See the manual’s secure-link chapter and pepsi-stage-secure-link(1).

POST /secure/TOKEN

Verify the PIN. On success it sets the session cookie and redirects (303) to the page above; on a wrong PIN it re-renders the form with 401, and on the attempt that exhausts MAX_ATTEMPTS it answers 429 and locks the token for a period that doubles with each further lockout. A submission carrying no PIN is answered 401 as well, but is not counted against MAX_ATTEMPTS: it is not a guess, and a client that re-submits the form empty would otherwise spend a legitimate recipient’s attempts for them. The PIN is submitted in the request body and never appears in a URL.

GET /secure/TOKEN/part/N

Download attachment N of the message. Authorised by the session, never by the PIN. Always served Content-Type: application/octet-stream with Content-Disposition: attachment and a sandboxing Content-Security-Policy, whatever media type the message claimed — a text/html part rendered inline from the portal’s own origin is precisely what the design refuses.

POST /secure/TOKEN/reply

Compose a reply, as multipart/form-data (a text field and optional files). Authorised by the session. The reply’s recipient and envelope sender are taken from the stored row and are not inputs; the body is always text/plain; uploads are capped by MAX_REPLY_SIZE / MAX_REPLY_FILES while the body streams; and the injected message deliberately does not carry state.local_origin, so it cannot inherit submission privileges. Answers 404 when REPLY_STAGE is unset (replying is opt-in per deployment).

Every portal response carries, per route rather than per host, Content-Security-Policy: default-src 'none' (with a per-response nonce for the one inline stylesheet and no script source at all), Referrer-Policy: no-referrer, X-Content-Type-Options: nosniff, X-Frame-Options: DENY and Cache-Control: no-store. The session cookie is Path=/secure/-scoped, HttpOnly, Secure and SameSite=Strict, so sharing an origin with the endpoints above is safe by construction rather than by deployment discipline.

70.1.3.1.6. The privileged applier, and doing without it

pepsi-httpd never performs a privileged change. It records an intent row in pepsi.setup_task and connects to a doorbell socket; systemd’s socket activation turns that connection into a short-lived root pepsi-setup apply, which validates the row and does the work. The full trust model is in pepsi-setup(1), The applier’s trust model.

That makes the capability removable, and Debian packages it separately:

pepsi

Every binary — including pepsi-setup itself — the schema, the templates and every other unit. A deployment administered from a terminal needs nothing else.

pepsi-httpd-admin

pepsi-setup-apply.socket and pepsi-setup-apply.service, and nothing else. It is what turns an HTTP request into a change under /etc/pepsi. pepsi Recommends it, so a default installation has it; apt remove pepsi-httpd-admin succeeds without touching pepsi.

With that package removed there is nothing to drain pepsi.setup_task. The loop is broken at the mechanism, not at the button: an insert into that table becomes inert, whatever happens to the web tier. (A source install gets the same lever from make install INSTALL_ADMIN_UNITS=no.)

Inert, and it stays that way: putting the package back does not act on whatever accumulated while it was gone. Its postinst runs pepsi-setup apply --clear, which deletes every queued row without executing one, so arming the applier is never also the event that runs a request nobody remembers making. See pepsi-setup(1).

70.1.3.1.6.1. What the console does then

Read-only, and it says so. Every setting the setup interview asks about is still rendered, with its staged answer or its default; every control that would change one is disabled, under a banner naming the missing package. Hiding the pages would be the easy reading of “read-only” and the wrong one — an operator who deliberately removed the applier still has to be able to see what the deployment is configured to do.

The API refuses the corresponding mutations with a status and a code of their own rather than a generic failure:

HTTP/1.1 503 Service Unavailable
{"code": "setup_applier_unavailable",
 "hint":  "the administrative package 'pepsi-httpd-admin' is not installed …",
 "detail": {"package": "pepsi-httpd-admin", "unit": "pepsi-setup-apply.socket"}}

It applies to POST /api/v1/setup/tasks, /setup/preflight, /setup/dns-check, /setup/certificates and PUT /api/v1/setup/answers, and to the console’s own form posts — POST /ui/setup/{step}, /ui/setup/tasks and /ui/domains/check, which is the “check now” button on the domains page. Every one of them reaches the queue through a single enqueuing function that asks first, so the refusal is one decision rather than one per surface, and the disabled control on the page is presentation over it rather than instead of it. Answer staging is included deliberately: a draft answer’s only purpose is to be applied, and a wizard that banks six steps of them on a host where nothing can apply them is the silent no-op this whole design is avoiding. DELETE /api/v1/setup/answers is not gated — discarding drafts cannot cause a privileged change, and a read-only console still has to be able to clear a half-finished interview.

Note the distinct code. setup_write_unavailable (no CONFIG_DB connection) and setup_applier_unavailable have different remedies, and the applier is reported first when both apply: configuring a database role would not have helped.

What is not affected is /api/v1/config and the console’s configuration pages. Those write pepsi.config_override, the database overlay every component reads at run time — not a file in /etc. They keep their existing gate ([pepsi-admin] CONFIG_DB), because the applier has nothing to do with them and widening the read-only rule to cover them would remove a capability the package split was not about. The two layers are described under Configuration layering in the manual.

70.1.3.1.6.2. How it detects this

By looking at two files, which between them answer installed and armed:

  1. pepsi-setup-apply.socket exists as a unit file in a directory systemd searches (/etc/systemd/system, /run/systemd/system, /usr/local/lib/systemd/system, /usr/lib/systemd/system, /lib/systemd/system); and

  2. [pepsi-admin] APPLY_SOCKET exists, is a socket, and is writable by this account — connecting to a UNIX socket needs write permission.

Two stats and an access, no privilege — which matters, because the server has already dropped to the unprivileged pepsi-httpd account and cannot ask dpkg or systemd anything. It deliberately does not connect: connecting is the doorbell, and would start a root process on every page render.

Neither test is redundant. Without the first, a stale socket node reads as a live doorbell — and stale nodes happen: systemd’s RemoveOnStop= defaults to off (the shipped unit sets it, an older or hand-edited one may not), and a package whose files are deleted rather than stopped leaves the node behind outright. That is exactly the case this feature exists for, and getting it wrong would fail open. Without the second, a freshly installed package whose socket has never been enabled would read as ready and every task would sit pending for ever.

The check fails closed. A path that cannot be examined, a path that is not a socket, a permission error — every uncertainty reads as “not available” and the console goes read-only. The recoverable failure is being told to install something that is already installed; the unrecoverable one is believing a change was made.

Two consequences worth knowing:

  • A site that drains the queue some other way — pepsi-setup apply --once from cron, a hand-written unit, a host without systemd, unit files somewhere systemd does not search — has no doorbell (or no unit file) and reads as unavailable. [pepsi-admin] APPLIER = yes overrides the probe. It is a promise the operator makes; nothing here can check it.

  • The converse: the socket can be present while the service unit is masked or broken, in which case a task is enqueued and stays pending. The task page shows that status rather than pretending otherwise. Both units ship in one package, so this is a hand-made state rather than a packaging outcome.

[pepsi-admin] APPLIER = no makes the console read-only by configuration alone, for a deployment that cannot change its package set.

70.1.3.1.7. Configuration

The [pepsi-httpd] section’s options (MAX_CONNECTIONS, DB_POOL_SIZE and the optional RESUME_AUTHORIZATION_TOKEN that guards POST /resume), the [pepsi-httpd-listener-<name>] and [pepsi-httpd-cert-<name>] socket and SNI-certificate sections (including the listener flag ADMIN), and the [pepsi-admin] section that configures the administrative API, are documented in pepsi.conf(5). the [pepsi-secure-link] section that enables the portal, together with the [pepsi-httpd-listener-<name>] and [pepsi-httpd-cert-<name>] socket and SNI-certificate sections, are documented in pepsi.conf(5).

70.1.3.1.8. Privilege dropping

To bind privileged ports (80/443) pepsi-httpd is often started as root. It does not serve as root: once every configured listener has been bound (and any TLS key material read), the process drops to the unprivileged service account pepsi-httpd, dropping all supplementary groups and switching to that account’s group and user id before accepting a single connection. The account must therefore exist before the server is started as root; create it (for example useradd --system --no-create-home --shell /usr/sbin/nologin pepsi-httpd) as part of installation.

If the drop cannot be completed while running as root — most often because the pepsi-httpd account does not exist — the server logs an error and exits without serving, rather than risk running as root. When started by a non-root user no privilege change is performed (the server simply runs as the invoking user); it will never run as root.

70.1.3.1.9. TLS key material

The certificate and private key of every TLS listener — its own TLS_CERT/TLS_KEY and each [pepsi-httpd-cert-*] SNI certificate — are read once, at start-up, before the privilege drop described above. Where they are read from depends on how the server was started.

Under systemd (the packaged deployment). The unit runs as the unprivileged pepsi-httpd user from the outset — it never holds root — and so cannot open certbot’s root-only /etc/letsencrypt/{live,archive}. It does not have to: pepsi-setup(1) writes a drop-in /etc/systemd/system/pepsi-httpd.service.d/10-tls-credentials.conf with one LoadCredential= entry per file. systemd opens those files as root when it starts the unit and passes private copies to the service under $CREDENTIALS_DIRECTORY — a per-unit tmpfs directory readable by this one service and invisible to every other unit — where pepsi-httpd looks each configured path up before falling back to reading the path itself. The configuration is untouched by this: TLS_CERT/TLS_KEY still name the real files, and the credential names are derived from those paths.

Two consequences worth knowing:

  • Re-run pepsi-setup run after adding, moving or removing a certificate. The drop-in is regenerated from the configuration and deliberately lists only files that exist, because systemd refuses to start a unit whose credential source is missing.

  • A credential is materialised when the unit starts, so a renewed certificate reaches clients only after a restart. The certbot deploy hook pepsi-setup installs performs it (systemctl try-restart).

Started directly, without systemd. The server is started as root, reads the files itself and only then drops privileges, so no credential is involved and the file permissions never matter.

A certificate that cannot be loaded is reported and skipped, not fatal: the SNI host names it would have served stop being available over HTTPS while every other certificate keeps working. Only a TLS listener left without a single usable certificate makes the server exit. One unreadable file therefore cannot take the MTA-STS policy of unrelated domains — or the metrics endpoint — down with it.

70.1.3.1.10. Commands

serve

Run the server until interrupted. Requires that the schema has been installed with pepsi-setup(1).

70.1.3.1.11. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

70.1.3.1.12. Signals

SIGINT, SIGTERM

Initiate shutdown and exit.

70.1.3.1.13. Exit Status

0

Clean shutdown.

1

An error occurred (for example a malformed configuration file, an unreadable certificate, or a failed database connection). The reason is written to the log.

70.1.3.1.14. Examples

Serve over HTTPS:

pepsi-httpd -c /etc/pepsi/pepsi.conf serve

Fetch a policy for testing (resolving the SNI host to the server):

curl --resolve mta-sts.example.org:443:203.0.113.7 \
     https://mta-sts.example.org/.well-known/mta-sts.txt

Check that a user’s key is published, the way a correspondent’s client would:

gpg --locate-external-key alice@example.org

70.1.3.1.15. See Also

pepsi-config(1), pepsi.conf(5), pepsi-dispatch(1), pepsi-keys(1), pepsi-setup(1)

70.1.3.1.16. Bugs

Report bugs to the Pepsi issue tracker.