34. pepsi-httpd

The HTTP/HTTPS server: MTA-STS policy, the Web Key Directory, Prometheus metrics, the secure-link portal, the administrative API and console, and the mailing-list pages.

34.1. Role

pepsi-httpd answers HTTP requests for the Pepsi system. Among other things it serves the MTA-STS policy file, the Web Key Directory that publishes our users’ OpenPGP keys, and a Prometheus /metrics page; the full list follows. It is built around a generic dispatch table so that further endpoints can be added without touching the server core. Run it with pepsi-httpd serve.

HTTP/1.1 (RFC 9112) with the semantics of RFC 9110, over hyper; TLS is RFC 8446 over rustls. What it serves is defined elsewhere, and each endpoint has its own governing document:

  • /.well-known/mta-sts.txt — the policy file of RFC 8461, whose content the relay stages fetch and honour on the way out.

  • /.well-known/openpgpkey/… — the OpenPGP Web Key Directory (draft-koch-openpgp-webkey-service, an Internet-Draft rather than an RFC), in both the direct and advanced layouts.

  • /.well-known/pepsi-dkim-retired/… — retired DKIM private keys, published when [pepsi-dkim] PUBLISH_RETIRED_KEYS is on (pepsi-dkim(1)); a Pepsi convention, not a standard.

  • /mail/config-v1.1.xml and /.well-known/autoconfig/mail/config-v1.1.xml — mail account autoconfiguration (draft-ietf-mailmaint-autoconfig), described below.

  • /secure/… — the secure-link portal (The secure-link fallback portal), whose session cookie follows RFC 6265.

  • /api/v1/… and /ui — the administrative API and console (The administrative API, The administration console), served only on listeners flagged ADMIN = yes, authenticated with the RFC 6750 Bearer scheme or a session cookie.

  • POST /resume — the GNU Taler merchant webhook: an Authorization: Bearer request carrying {"message_id": "<token>"} flips the matching paused row back to pending and wakes the dispatcher, so pepsi-stage-anti-spam runs again the moment an order is paid. The endpoint is disabled (404) unless [pepsi-httpd] RESUME_AUTHORIZATION_TOKEN is set.

  • /addin/manifest.xml, /addin/taskpane.html, /addin/readpane.html, /addin/pepsi-crypto.js and /addin/icon-<size>.png — the Outlook add-in (the compose pane that asks for signing or encryption, the read pane that shows the gateway’s verdict; Mail client extensions), answering 404 unless [pepsi-httpd] ADDIN is on.

  • /3.0/… and /3.1/… — the GNU Mailman 3 REST API (The GNU Mailman 3 REST API), served only on listeners flagged LIST_API = yes and authenticated with the single [pepsi-list] API_USER/API_PASS pair.

  • /lists/…, /archives/list/… and /robots.txt — the public mailing-list pages, the member and list-owner accounts and the archive (The administration console, Archives), served only on listeners flagged LISTS = yes.

34.2. Features

  • Multiple listeners. Each [pepsi-httpd-listener-<name>] section binds one socket exactly like the ingress listeners: SERVE = tcp (BIND_TO/PORT, default port 443), unix (UNIXPATH) or systemd (socket activation), with MODE = plain or tls.

  • SNI certificate selection. A TLS listener chooses its certificate per connection from the client’s SNI host name: each [pepsi-httpd-cert-<name>] section lists one or more SNI host names and a TLS_CERT/TLS_KEY pair. A listener’s own TLS_CERT/TLS_KEY is served to a client that sends no SNI, and to an SNI name that no section lists, but only when that certificate is valid for the name; any other SNI name fails the handshake (strict SNI). (MTA-STS requires a certificate valid for mta-sts.<domain>.)

  • Key material via systemd credentials. The server runs unprivileged and cannot read certbot’s root-only /etc/letsencrypt; under systemd it does not need to. pepsi-setup writes a LoadCredential= drop-in, systemd opens each certificate and key as root when it starts the unit, and the server reads the private copy handed to it under $CREDENTIALS_DIRECTORY. A certificate that still cannot be loaded is skipped with an error rather than taking the whole server down — only a listener left with no usable certificate is fatal.

  • Generic routing. Requests are matched by method and URL shape: a route’s path is a list of segments, each a literal, a named capture of one segment, or a capture of the remainder. Routes are tried in registration order, first match wins, so GET /.well-known/mta-sts.txt, GET /.well-known/openpgpkey/$DOMAIN/hu/$HASH and a future GET /foo/$ID are one table entry each.

  • MTA-STS policy. GET /.well-known/mta-sts.txt returns the RFC 8461 policy (built from [pepsi] MTA_STS_* and the ingress HOSTNAME) when the request Host is mta-sts.<domain> for a domain in ACCEPTED_DOMAINS; any other host yields 404.

  • Web Key Directory. GET /.well-known/openpgpkey/... publishes our own users’ OpenPGP keys, in both the direct and the advanced form, answered from the key store by an indexed (domain, local-part hash) lookup. The response is the binary key with Access-Control-Allow-Origin: *; the policy file is a zero-length 200 (its absence makes GnuPG give up before asking for a key); an unknown hash is an empty-bodied 404. It serves only identities that this deployment holds and has marked published — never a cached correspondent key — and it deliberately applies no rate limiting, because the protocol is a public oracle by design. See Key management.

  • Retired DKIM keys. GET /.well-known/pepsi-dkim-retired/<domain>/<selector>.pem serves, on every listener, a private key that the signing daemon retired and wrote to its public directory under [pepsi-dkim] PUBLISH_RETIRED_KEYS, so that mail signed with it proves nothing about who sent it. Everything else there is a 404: the option off, a name that the daemon could not have written, a missing file, a symbolic link. This server holds no live DKIM key: the console’s DKIM page and GET /api/v1/keys/dkim read the daemon’s public status file.

  • Prometheus metrics. GET /metrics exposes live per-stage gauges (active and paused messages, read straight from the queue) and counters written by pepsi-dispatch (per-stage timeouts, crashes, message count and total processing time for averages, and the global stage/message totals). Those figures — and the stage names labelling them — describe how much mail the deployment carries and how its pipeline is built, so the endpoint is administrative: it is served only on an ADMIN = yes listener and answers the ordinary 404 elsewhere. It asks for no credential of its own, because a scraper has none to present; the listener flag is the access control. Binding the server privately is not an alternative: the same process must answer mta-sts.<domain> and openpgpkey.<domain> from the public internet.

  • The administrative surface. A listener flagged ADMIN = yes additionally serves the /api/v1 administrative API (The administrative API), the /ui administration console (The administration console) and /metrics. On every other listener those paths answer the ordinary 404. The console is a client of the API — same authentication, same scopes, same audit log — and performs no service control.

34.3. Mail client autoconfiguration

Setting up a mail account by hand means answering eight questions that the person who owns the account is usually the least equipped to answer: two hostnames, two ports, two notions of transport security and two authentication methods. draft-ietf-mailmaint-autoconfig fixes that: the provider publishes the answers, and the client fetches them from the address alone.

Pepsi serves the two rungs of that chain that a provider is expected to publish. The Host decides which is being asked for:

https://autoconfig.example.org/mail/config-v1.1.xml             (required)
https://example.org/.well-known/autoconfig/mail/config-v1.1.xml (optional)

Both return the same text/xml document, and both are public: the draft requires it, and the reason is not an oversight but a sequencing problem — a client has to learn which authentication mechanism to use before it can authenticate. The document therefore contains no secret, and Pepsi keeps it that way by naming %EMAILADDRESS%, the placeholder that the client substitutes, rather than any real address. The draft’s optional ?emailaddress= parameter is accepted and ignored, which also means no request-controlled text is ever interpolated into a document served from the mail domain’s own origin.

<?xml version="1.0" encoding="UTF-8"?>
<clientConfig version="1.1">
  <emailProvider id="example.org">
    <domain>example.org</domain>
    <displayName>Example Mail</displayName>
    <displayShortName>Example</displayShortName>
    <incomingServer type="imap">
      <hostname>mail.example.org</hostname>
      <port>993</port>
      <socketType>SSL</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <outgoingServer type="smtp">
      <hostname>mail.example.org</hostname>
      <port>587</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
  </emailProvider>
</clientConfig>

The outgoing half is derived, not configured. Its port, transport security and authentication are read off the [pepsi-ingress-listener-*] flagged SUBMISSION = yes, so a deployment that moves submission from STARTTLS on 587 to implicit TLS on 465 gets a corrected document without touching it. The commonest way in which a published configuration goes wrong is that it stops matching the server, and here there is nothing to forget to update. SMTP_HOST overrides the derivation for a deployment whose public endpoint is a load balancer rather than the listener.

The incoming half cannot be derived, because Pepsi does not serve mailboxes; it hands them to an MDA (pepsi-stage-relay-to-lmtp). So IMAP_HOST or POP3_HOST must name whatever server the deployment pairs Pepsi with, and until one of them does, both endpoints answer 404. That is deliberate: a client that finds a configuration document stops walking the fallback chain, so an incomplete one leaves the user worse off than none at all.

Publishing needs two things outside the configuration file: an autoconfig.<domain> DNS record pointing here, and a certificate that covers that name. Clients try the https://autoconfig.… URL first, and a certificate error there is simply a failed lookup. A deployment that does not want to add the name can rely on the /.well-known/ form, at the cost of being found only by the clients that try it.

34.4. Security

  • Unprivileged, and confined in the database. When started as root (to bind port 443), the server reads its certificates and drops to the pepsi-httpd account before accepting a connection; under systemd it never holds root at all. Its database role is cut down to what its code uses: envelope columns of the queue but never message headers or bodies, nothing on the tables it has no use for (settings, whitelist, otp_key, …), read-only access to others, and never the private_wrapped column of the key store (The privilege split, An attacker who has the database). It cannot write the configuration overlay except through the separate [pepsi-admin] CONFIG_DB connection that authenticates as pepsi-config, and every privileged change is only requested, as a pepsi.setup_task row that the root applier re-validates (pepsi-setup).

  • Administration is a listener property. /api/v1, /ui and /metrics exist only on ADMIN = yes listeners, which must be TLS, a UNIX socket or loopback plaintext; everywhere else they are an ordinary 404 (The administrative API).

  • Sign-in by e-mail gives an end user a session confined to own:<address>; its second factor is checked by the applier, never by this server (Signing in to the console by e-mail, Signing in by e-mail).

  • Resource limits. MAX_CONNECTIONS in total and MAX_CONNECTIONS_PER_IP per IPv4 address or IPv6 /64, checked before a connection takes a slot; 30-second deadlines on the TLS handshake, the request headers and the request body; at most one Argon2id password verification per CPU at a time, off the request executor (a sign-in that finds no free slot within five seconds is answered as rate-limited); and a per-target hourly budget on every public form that makes the server send mail to an address (two mails per address per hour for the list pages, ENROL_MAIL_PER_HOUR for console enrolment). The figures are tabulated in Availability.

  • Log retention is pepsi-httpd prune, run daily by pepsi-log-prune.timer: the pepsi-httpd role is the one service role that holds DELETE on the audit and mail logs; every other component may only append to them.

What a compromise of this server would still give an attacker is stated in A compromised component.

34.5. Configuration

[pepsi-httpd]: MAX_CONNECTIONS (default 256), MAX_CONNECTIONS_PER_IP (default 32), DB_POOL_SIZE, RESUME_AUTHORIZATION_TOKEN, ADDIN (default no), ADDIN_URL (which must be an https:// origin – plain http:// is refused at start-up unless the host is loopback, so a test needs no certificate) and REQUIRE_WEBAUTHN (default no; which kinds of account must use a security key, see Security keys). The administrative surface reads [pepsi-admin], and the portal [pepsi-secure-link]. Listeners and certificates live in [pepsi-httpd-listener-<name>] and [pepsi-httpd-cert-<name>] sections. The served domains, the MTA-STS mx host and the policy itself come from the shared [pepsi-ingress] and [pepsi] sections. See Configuration.

The Web Key Directory needs no configuration of its own — it follows each identity’s published flag — but the advanced method needs an openpgpkey.<domain> DNS record per served domain, and the HTTPS listener’s certificate must cover that name. pepsi-setup run asks certbot for it and reports any name that does not resolve.

[pepsi-autoconfig] configures the autoconfiguration document: IMAP_HOST / POP3_HOST (either one enables the feature) with their _PORT, _SOCKET and _AUTH; the optional SMTP_* overrides; and DISPLAY_NAME, DISPLAY_SHORT_NAME and DOCUMENTATION_URL. See pepsi.conf(5).

34.6. See also

Architecture, Key management, pepsi-dispatch, pepsi-keys, pepsi-setup, pepsi.conf(5).