23. The GNU Mailman 3 REST API

pepsi-httpd serves the GNU Mailman 3 REST API under /3.0/ and /3.1/. This is not an API of Pepsi’s own design: it is somebody else’s interface, reimplemented so that software written for GNU Mailman 3 works against Pepsi unchanged. mailmanclient, Postorius and HyperKitty are the clients it exists for.

The version implemented is 3.3.10. That precision matters: “Mailman 3” is not a contract, because the attribute set drifts between point releases while the /3.1/ path does not. The interface is documented by the GNU Mailman project at https://docs.mailman3.org/ and that documentation — not this chapter — is the specification. What follows is what an operator needs in order to run it, the three places where Pepsi has to answer something honest rather than copy something it does not have, and the one resource Pepsi adds.

Note

Attribution. The resource tree, the attribute names, the enumeration values, the wire conventions and the error messages are GNU Mailman’s, copyright the Free Software Foundation, and are reproduced here in order to be compatible with them. See vendor/PEPSI-VENDORING.md. Pepsi is not GNU Mailman and is not endorsed by the GNU Mailman project.

This chapter is about the interface. Mailing lists explains what the resources mean (a list, a member, a moderation action, a digest), and its section How the subsystem is put together shows where this API sits relative to the operator’s /api/v1 and the public member pages. The archive’s own resources are described in Archives. Installation explains how to turn the listener on and which credentials it needs; The administration console describes the browser consoles, which are clients of the same model; and Supported Features lists what Pepsi implements at all.

23.1. There are two APIs in this tree

An operator will meet both and they have nothing in common but the server:

/api/v1

Pepsi’s own administrative API (The administrative API). Its audience is the operator: the queue, the key store, the logs, the configuration. It has accounts, scopes, sessions, bearer tokens and CSRF, its collections are {items, total, limit, offset} and its errors are {code, hint, detail}.

/3.0/ and /3.1/

This one. Its audience is mailing-list software. It has one shared password and no scopes, its collections are {start, total_size, entries} and its errors are Falcon’s {title, description}.

They are deliberately not unified. The whole value of this one is that its shapes are not ours to improve.

23.2. Where it is served

Only on a listener flagged LIST_API = yes. On every other listener the /3.0/ and /3.1/ routes answer a plain 404, byte for byte what any unknown path gets, so a public listener cannot be probed for whether the API is enabled on this deployment.

The rule that the administrative API follows applies here too, and for the same reason: the credential is one static password, so it must not cross a cleartext network.

  • a TLS listener is always allowed;

  • a UNIX-socket listener is always allowed (it never leaves the host);

  • a plaintext TCP listener is allowed only on a loopback address;

  • a systemd-activated socket in cleartext is refused, because the server cannot see where systemd bound it.

A listener that asks for the flag but is bound somewhere else does not serve the API, and pepsi-httpd says so loudly at start-up rather than failing quietly.

[pepsi-httpd-listener-mailman]
# What every existing mailman.cfg-shaped client already points at.
BIND_TO = 127.0.0.1:8001
MODE = plain
LIST_API = yes

[pepsi-list]
API_USER = restadmin
API_PASS = a-long-random-string

Warning

API_USER and API_PASS are one shared credential for the whole site. Anyone who holds it can read and change every list, every subscription and every user record. That is upstream’s model, not a simplification of it; keep the listener on loopback or behind TLS, and do not reuse the password anywhere else. pepsi-httpd refuses to start when a listener asks for the API and the credential is incomplete: an API whose whole authorisation model is that password, serving without one, is not a degraded mode. Keep API_PASS in a secrets.d fragment through an @inline-secret@ directive rather than inline in the world-readable pepsi.conf; pepsi-setup run moves one it finds written inline into secrets.d/pepsi-list.secret (see Mail appliances store secrets world-readable or unprotected).

23.3. Authentication

The API uses HTTP Basic authentication against that one pair, compared in constant time. There are no sessions, no CSRF tokens, no per-user accounts and no token minting: a client sends Authorization: Basic ... on every request. A failure is 401 with

WWW-Authenticate: Basic realm="mailman3-rest",charset="utf-8"

and that realm string is upstream’s, because a client may display it.

Pepsi adds a rate limiter, which upstream does not have. It is a superset and cannot break a correct client. A compatibility suite sends hundreds of requests in seconds, however, so the default is deliberately high: 3000 requests per minute per source address, settable with [pepsi-list] API_RATE_LIMIT. A 429 from this API is Pepsi’s, not Mailman’s.

23.4. The two versions differ in four ways

It is a common mistake to assume that they differ in only one:

  1. the six *_uri template attributes are list attributes in /3.0/ and absent in /3.1/, which reaches them through /uris instead;

  2. /templates/... exists only in /3.0/; /uris/..., /lists/<id>/uris, /domains/<host>/uris and /plugins exist only in /3.1/. Each is a 404 on the other version;

  3. identifiers are serialised differently. /3.0/ emits a UUID as uuid.int, a decimal integer of up to thirty-nine digits; /3.1/ emits uuid.hex, a string. This reaches member_id, user_id and every self_link built from them, and it is why /3.1/ exists at all — the integer form is not representable in every JavaScript runtime;

  4. every self_link and Location carries its own version prefix.

23.5. Template URIs

A template override is a URI that the server fetches, whichever version wrote it: a *_uri attribute in /3.0/, or /uris in /3.1/. So whoever holds the REST credential chooses what the server reads. Pepsi accepts mailman: (the built-in text) and http(s):, fetched under the limits in [pepsi-list] TEMPLATE_FETCH_*, and answers 400 to a file: URI, which upstream would read from its own disk. A file: row that is in the table anyway (written by hand) is never read: the notice is rendered as if the row were absent.

23.6. Three honest analogues

Three resources describe things that GNU Mailman has and Pepsi does not have. None of them is faked.

/queues

Upstream’s queues are directories of .pck files. Pepsi has one table: a message in flight is a row in pepsi.workqueue at some stage. So this resource answers with the real counts of the messages of lists that exist. in counts every one still in flight, retry those paused for a later attempt, and bad and shunt the two terminal states that the dispatcher never re-queues; the pipeline’s other mail is not a list’s queue. The workqueue_id stands in for a filebase, and the table’s name appears where a directory would be.

POST /queues/in injects a message at the list’s posting stage, where it runs the whole posting chain. A message posted to bad or shunt is stored failed or timeout at that stage, and nothing runs it, just as nothing runs upstream’s queues of those names (a deployment running pepsi-failure-bouncer(1) drops it after MIN_AGE). A message posted to retry is stored paused, and the dispatcher resumes it. DELETE /queues/<name>/<id> removes one row. There is no virgin queue: a message that Pepsi composes itself is an ordinary row at its response stage. pepsi-queue(1) remains the real tool.

/plugins

Always an empty collection. Pepsi has no plugin interface and no pluggable archivers. /3.0/ has no such resource.

/reserved

Upstream’s own test hook, which upstream marks as not part of the stable API. GET /reserved/reset truncates every list and archive table, so it exists only on a listener that explicitly opted in with LIST_API_TESTING = yes; anywhere else it is a 404 indistinguishable from an unknown path, and a test asserts that default. It is there because the compatibility suites reset state between test classes in-process, which an out-of-process server cannot offer. It also restarts the counters of the tables it empties, so the first held message after a reset is request 1 again, as on upstream’s test database. DELETE /reserved/uids/orphans is a successful no-op: identifiers here are v4 UUIDs and there is no orphan table to cull.

23.7. One resource of Pepsi’s own

GET /<api>/lists/<id>/pepsi answers the per-list settings that Pepsi invented (Settings Pepsi invented). It reports each of search_trigram, archive_show_addresses, hold_dmarc_failures and archive_retention_days with the list’s own value, or with the default (null where the site setting applies). PATCH on the same path sets any of them, validated exactly as pepsi-list list set-ext validates them; an empty value removes the list’s own value, and an unknown key is a 400.

They are not attributes under /config, and must not become any: a whole-resource PUT /config has to carry every writable attribute, so one more there would break every client built against 3.3.10. No upstream client asks for this path, so none of them can notice it.

23.8. What needs configuring beyond the credential

[pepsi-list] RELEASE_STAGE

The [stage-<name>] section running pepsi-stage-list-post. A moderator accepting a held message, and POST /queues/<name>, both put a message back into the pipeline there, and only the deployment knows what that section is called. Without it those two operations answer 400 and say so — which beats answering 204 and dropping the message, because a moderator who accepts a post and watches it never arrive cannot tell that from a delivery failure.

[pepsi-list] DEFAULT_LANGUAGE

The bottom of the preference chain, reported by GET /<api>/system/preferences. Defaults to en.

23.9. A worked example

Keep the credential out of the command line: on a shared machine, it would appear in every process listing and in the shell history. Put it in a file instead and let curl read it:

$ cat > ~/.mailman-netrc <<'EOF'
machine localhost login restadmin password a-long-random-string
EOF
$ chmod 600 ~/.mailman-netrc
$ alias mmcurl='curl -sS --netrc-file ~/.mailman-netrc'

$ mmcurl http://localhost:8001/3.1/system/versions
{"mailman_version":"GNU Mailman 3.3.10 (Tom Sawyer; implemented by Pepsi …)",
 "python_version":"…","api_version":"3.1",
 "self_link":"http://localhost:8001/3.1/system/versions",
 "http_etag":"\"…\""}

$ mmcurl -X POST --data 'mail_host=lists.example.com' \
    http://localhost:8001/3.1/domains

$ mmcurl -X POST --data 'fqdn_listname=devel@lists.example.com' \
    http://localhost:8001/3.1/lists

$ mmcurl 'http://localhost:8001/3.1/lists?count=10&page=1'

mailman_version is worth a second look. It has to begin with the string that a client matches on, and it has to be true. So it names the interface version first, and it says what is actually running inside the parentheses, where upstream puts a release code name. Pepsi does not claim to be GNU Mailman. This resource is the only place where a version is shown, and it needs the API credential (see Missing HTTPS and HTTP security-header hardening on webmail).

POST /lists/<id>/digest: bump advances the volume, and send and periodic both go through the sender that the CLI uses, so an issue sent from a client and one sent from a timer are the same code path and carry the same numbering.

23.10. What is not implemented

  • A rejected held message or subscription is dropped without the rejection notice that upstream sends. Accepting, discarding and deferring are complete.

  • A subscription that the API completes sends no welcome message: send_welcome_message is accepted and has no effect.

  • [mta] verp_format, verp_regexp, verp_confirm_format and verp_probes are not configuration. Every copy is VERP’d, in upstream’s ${bounces}+${local}=${domain} form with an authentication code appended as one more = field (keyed with [pepsi-list] VERP_SECRET), and probes are on by default ([pepsi-list] BOUNCE_PROBES); no API resource exposes a VERP address, so a client sees no difference.

23.11. Testing it

tests/22-list-api-test.sh runs upstream’s own test suites against a deployed Pepsi: mailmanclient 3.3.5 (344 doctest statements plus 36 test functions) and Postorius 1.3.13 (30 files, 248 test functions), each with only its server-starting fixture replaced. The replacements live in contrib/compat/ and are versioned beside the pinned requirements, so “unmodified upstream tests” stays literally true of everything except the fixture. contrib/compat/run-local.sh runs the same two suites against a build tree instead of a deployment: a throwaway database, a configuration with only this listener, and the tree’s own pepsi-httpd and pepsi-dispatch, in a private network namespace so that port 9001, which mailmanclient’s doctests name, is free.

Both suites pass. What remains are strict expected failures, each one a declared difference listed in Declared differences from GNU Mailman 3: the fixture marks the test (or, in mailmanclient’s single using.rst doctest, the individual statement) and a marked test that starts passing fails the run. The fixture also lengthens the passwords that the suites give their users: Pepsi would refuse the short ones, and none of those tests is about passwords. See Test Suite.