30. pepsi-httpd

The HTTP/HTTPS server: MTA-STS policy, the Web Key Directory, Prometheus metrics, the secure-link portal, the administrative API and console, and the mailing-list pages.

30.1. Role

pepsi-httpd answers HTTP requests for the Pepsi system. Among other things it serves the MTA-STS policy file, the Web Key Directory that publishes our users’ OpenPGP keys, and a Prometheus /metrics page; the full list follows. It is built around a generic dispatch table so further endpoints can be added without touching the server core. Run it with pepsi-httpd serve.

HTTP/1.1 (RFC 9112) with the semantics of RFC 9110, over hyper; TLS is RFC 8446 over rustls. What it serves is defined elsewhere, and each endpoint has its own governing document:

  • /.well-known/mta-sts.txt — the policy file of RFC 8461, whose content the relay stages fetch and honour on the way out.

  • /.well-known/openpgpkey/… — the OpenPGP Web Key Directory (draft-koch-openpgp-webkey-service, an Internet-Draft rather than an RFC), in both the direct and advanced layouts.

  • /mail/config-v1.1.xml and /.well-known/autoconfig/mail/config-v1.1.xml — mail account autoconfiguration (draft-ietf-mailmaint-autoconfig), described below.

  • /secure/… — the secure-link portal (The secure-link fallback portal), whose session cookie follows RFC 6265.

  • /api/v1/… and /ui — the administrative API and console (The administrative API, The administration console), served only on listeners flagged ADMIN = yes, authenticated with the RFC 6750 Bearer scheme or a session cookie.

  • POST /resume — the GNU Taler merchant webhook: an Authorization: Bearer request carrying {"message_id": "<token>"} flips the matching paused row back to pending and wakes the dispatcher, so pepsi-stage-anti-spam runs again the moment an order is paid. The endpoint is disabled (404) unless [pepsi-httpd] RESUME_AUTHORIZATION_TOKEN is set.

  • /addin/manifest.xml and /addin/taskpane.html — the Outlook add-in, answering 404 unless [pepsi-httpd] ADDIN is on.

  • /3.0/… and /3.1/… — the GNU Mailman 3 REST API (The GNU Mailman 3 REST API), served only on listeners flagged LIST_API = yes and authenticated with the single [pepsi-list] API_USER/API_PASS pair.

  • /lists/…, /archives/list/… and /robots.txt — the public mailing-list pages, the member and list-owner accounts and the archive (The administration console, Archives), served only on listeners flagged LISTS = yes.

30.2. Features

  • Multiple listeners. Each [pepsi-httpd-listener-<name>] section binds one socket exactly like the ingress listeners: SERVE = tcp (BIND_TO/PORT, default port 443), unix (UNIXPATH) or systemd (socket activation), with MODE = plain or tls.

  • SNI certificate selection. A TLS listener chooses its certificate per connection from the client’s SNI host name: each [pepsi-httpd-cert-<name>] section lists one or more SNI host names and a TLS_CERT/TLS_KEY pair. A listener’s own TLS_CERT/TLS_KEY is the fallback for connections with no SNI match. (MTA-STS requires a certificate valid for mta-sts.<domain>.)

  • Key material via systemd credentials. The server runs unprivileged and cannot read certbot’s root-only /etc/letsencrypt; under systemd it does not need to. pepsi-setup writes a LoadCredential= drop-in, systemd opens each certificate and key as root when it starts the unit, and the server reads the private copy handed to it under $CREDENTIALS_DIRECTORY. A certificate that still cannot be loaded is skipped with an error rather than taking the whole server down — only a listener left with no usable certificate is fatal.

  • Generic routing. Requests are matched by method and URL shape: a route’s path is a list of segments, each a literal, a named capture of one segment, or a capture of the remainder. Routes are tried in registration order, first match wins, so GET /.well-known/mta-sts.txt, GET /.well-known/openpgpkey/$DOMAIN/hu/$HASH and a future GET /foo/$ID are one table entry each.

  • MTA-STS policy. GET /.well-known/mta-sts.txt returns the RFC 8461 policy (built from [pepsi] MTA_STS_* and the ingress HOSTNAME) when the request Host is mta-sts.<domain> for a domain in ACCEPTED_DOMAINS; any other host yields 404.

  • Web Key Directory. GET /.well-known/openpgpkey/... publishes our own users’ OpenPGP keys, in both the direct and the advanced form, answered from the key store by an indexed (domain, local-part hash) lookup. The response is the binary key with Access-Control-Allow-Origin: *; the policy file is a zero-length 200 (its absence makes GnuPG give up before asking for a key); an unknown hash is an empty-bodied 404. It serves only identities this deployment holds and has marked published — never a cached correspondent key — and it deliberately applies no rate limiting, because the protocol is a public oracle by design. See Key management.

  • Prometheus metrics. GET /metrics exposes live per-stage gauges (active and paused messages, read straight from the queue) and counters written by pepsi-dispatch (per-stage timeouts, crashes, message count and total processing time for averages, and the global stage/message totals). Those figures — and the stage names labelling them — describe how much mail the deployment carries and how its pipeline is built, so the endpoint is administrative: it is served only on an ADMIN = yes listener and answers the ordinary 404 elsewhere. It asks for no credential of its own, because a scraper has none to present; the listener flag is the access control. Binding the server privately is not an alternative: the same process must answer mta-sts.<domain> and openpgpkey.<domain> from the public internet.

  • The administrative surface. A listener flagged ADMIN = yes additionally serves the /api/v1 administrative API (The administrative API), the /ui administration console (The administration console) and /metrics. On every other listener those paths answer the ordinary 404. The console is a client of the API — same authentication, same scopes, same audit log — and performs no service control.

30.3. Mail client autoconfiguration

Setting up a mail account by hand means answering eight questions — two hostnames, two ports, two notions of transport security, two authentication methods — that the person who owns the account is usually the least equipped to answer. draft-ietf-mailmaint-autoconfig fixes that by having the provider publish the answers, and the client fetch them from the address alone.

Pepsi serves the two rungs of that chain a provider is expected to publish. The Host decides which is being asked for:

https://autoconfig.example.org/mail/config-v1.1.xml             (required)
https://example.org/.well-known/autoconfig/mail/config-v1.1.xml (optional)

Both return the same text/xml document, and both are public: the draft requires it, and the reason is not an oversight but a sequencing problem — a client has to learn which authentication mechanism to use before it can authenticate. The document therefore contains no secret, and Pepsi keeps it that way by naming %EMAILADDRESS%, the placeholder the client substitutes, rather than any real address. The draft’s optional ?emailaddress= parameter is accepted and ignored, which also means no request-controlled text is ever interpolated into a document served from the mail domain’s own origin.

<?xml version="1.0" encoding="UTF-8"?>
<clientConfig version="1.1">
  <emailProvider id="example.org">
    <domain>example.org</domain>
    <displayName>Example Mail</displayName>
    <displayShortName>Example</displayShortName>
    <incomingServer type="imap">
      <hostname>mail.example.org</hostname>
      <port>993</port>
      <socketType>SSL</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <outgoingServer type="smtp">
      <hostname>mail.example.org</hostname>
      <port>587</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
  </emailProvider>
</clientConfig>

The outgoing half is derived, not configured. Its port, transport security and authentication are read off the [pepsi-ingress-listener-*] flagged SUBMISSION = yes, so a deployment that moves submission from STARTTLS on 587 to implicit TLS on 465 gets a corrected document without touching it — the commonest way a published configuration goes wrong is that it stopped matching the server, and there is nothing here to forget to update. SMTP_HOST overrides the derivation for a deployment whose public endpoint is a load balancer rather than the listener.

The incoming half cannot be derived, because Pepsi does not serve mailboxes; it hands them to an MDA (pepsi-stage-relay-to-lmtp). So IMAP_HOST or POP3_HOST must name whatever the deployment pairs with, and until one of them does, both endpoints answer 404. That is deliberate: a client that finds a configuration document stops walking the fallback chain, so an incomplete one leaves the user worse off than none at all.

Publishing needs two things outside the configuration file: an autoconfig.<domain> DNS record pointing here, and a certificate covering that name — clients try the https://autoconfig.… URL first, and a certificate error there is simply a failed lookup. A deployment unwilling to add the name can rely on the /.well-known/ form, at the cost of being found only by the clients that try it.

30.4. Configuration

[pepsi-httpd]: MAX_CONNECTIONS (default 256), DB_POOL_SIZE, RESUME_AUTHORIZATION_TOKEN, ADDIN (default no) and ADDIN_URL (which must be an https:// origin – plain http:// is refused at start-up unless the host is loopback, so a test needs no certificate). The administrative surface reads [pepsi-admin], and the portal [pepsi-secure-link]. Listeners and certificates live in [pepsi-httpd-listener-<name>] and [pepsi-httpd-cert-<name>] sections. The served domains, the MTA-STS mx host and the policy itself come from the shared [pepsi-ingress] and [pepsi] sections. See Configuration.

The Web Key Directory needs no configuration of its own — it follows each identity’s published flag — but the advanced method needs an openpgpkey.<domain> DNS record per served domain, and the HTTPS listener’s certificate must cover that name. pepsi-setup run asks certbot for it and reports any that does not resolve.

[pepsi-autoconfig] configures the autoconfiguration document: IMAP_HOST / POP3_HOST (either one enables the feature) with their _PORT, _SOCKET and _AUTH; the optional SMTP_* overrides; and DISPLAY_NAME, DISPLAY_SHORT_NAME and DOCUMENTATION_URL. See pepsi.conf(5).

30.5. See also

Architecture, Key management, pepsi-dispatch, pepsi-keys, pepsi-setup, pepsi.conf(5).