23. The GNU Mailman 3 REST API¶
pepsi-httpd serves the GNU Mailman 3 REST API under /3.0/ and
/3.1/. This is not an API of Pepsi’s own design: it is somebody else’s
interface, reimplemented so that software written for GNU Mailman 3 works
against Pepsi unchanged. mailmanclient, Postorius and HyperKitty are the
clients it exists for.
The version implemented is 3.3.10. That precision matters: “Mailman 3” is
not a contract, because the attribute set drifts between point releases while
the /3.1/ path does not. The interface is documented by the GNU Mailman
project at https://docs.mailman3.org/ and that documentation — not this
chapter — is the specification. What follows is what an operator needs in order
to run it, the three places where Pepsi has to answer something honest
rather than copy something it does not have, and the one resource Pepsi adds.
Note
Attribution. The resource tree, the attribute names, the enumeration
values, the wire conventions and the error messages are GNU Mailman’s,
copyright the Free Software Foundation, and are reproduced here in order to
be compatible with them. See vendor/PEPSI-VENDORING.md. Pepsi is not
GNU Mailman and is not endorsed by the GNU Mailman project.
This chapter is about the interface. Mailing lists explains what the
resources mean (a list, a member, a moderation action, a digest), and its
section How the subsystem is put together shows where this API sits relative to the
operator’s /api/v1 and the public member pages. The archive’s own resources
are described in Archives. Installation explains how to turn the
listener on and which credentials it needs; The administration console describes the browser
consoles, which are clients of the same model; and Supported Features lists what
Pepsi implements at all.
23.1. There are two APIs in this tree¶
An operator will meet both and they have nothing in common but the server:
/api/v1Pepsi’s own administrative API (The administrative API). Its audience is the operator: the queue, the key store, the logs, the configuration. It has accounts, scopes, sessions, bearer tokens and CSRF, its collections are
{items, total, limit, offset}and its errors are{code, hint, detail}./3.0/and/3.1/This one. Its audience is mailing-list software. It has one shared password and no scopes, its collections are
{start, total_size, entries}and its errors are Falcon’s{title, description}.
They are deliberately not unified. The whole value of this one is that its shapes are not ours to improve.
23.2. Where it is served¶
Only on a listener flagged LIST_API = yes. On every other listener the
/3.0/ and /3.1/ routes answer a plain 404, byte for byte what any
unknown path gets, so a public listener cannot be probed for whether the API is
enabled on this deployment.
The rule that the administrative API follows applies here too, and for the same reason: the credential is one static password, so it must not cross a cleartext network.
a TLS listener is always allowed;
a UNIX-socket listener is always allowed (it never leaves the host);
a plaintext TCP listener is allowed only on a loopback address;
a systemd-activated socket in cleartext is refused, because the server cannot see where systemd bound it.
A listener that asks for the flag but is bound somewhere else does not serve
the API, and pepsi-httpd says so loudly at start-up rather than failing
quietly.
[pepsi-httpd-listener-mailman]
# What every existing mailman.cfg-shaped client already points at.
BIND_TO = 127.0.0.1:8001
MODE = plain
LIST_API = yes
[pepsi-list]
API_USER = restadmin
API_PASS = a-long-random-string
Warning
API_USER and API_PASS are one shared credential for the whole
site. Anyone who holds it can read and change every list, every
subscription and every user record. That is upstream’s model, not a
simplification of it; keep the listener on loopback or behind TLS, and do
not reuse the password anywhere else. pepsi-httpd refuses to start when
a listener asks for the API and the credential is incomplete: an API whose
whole authorisation model is that password, serving without one, is not a
degraded mode. Keep API_PASS in a secrets.d fragment through an
@inline-secret@ directive rather than inline in the world-readable
pepsi.conf; pepsi-setup run moves one it finds written inline into
secrets.d/pepsi-list.secret (see
Mail appliances store secrets world-readable or unprotected).
23.3. Authentication¶
The API uses HTTP Basic authentication against that one pair, compared in
constant time. There are no
sessions, no CSRF tokens, no per-user accounts and no token minting: a client
sends Authorization: Basic ... on every request. A failure is 401 with
WWW-Authenticate: Basic realm="mailman3-rest",charset="utf-8"
and that realm string is upstream’s, because a client may display it.
Pepsi adds a rate limiter, which upstream does not have. It is a superset
and cannot break a correct client. A compatibility suite sends hundreds of
requests in seconds, however, so the default is deliberately high: 3000
requests per minute per source address, settable with [pepsi-list]
API_RATE_LIMIT. A
429 from this API is Pepsi’s, not Mailman’s.
23.4. The two versions differ in four ways¶
It is a common mistake to assume that they differ in only one:
the six
*_uritemplate attributes are list attributes in/3.0/and absent in/3.1/, which reaches them through/urisinstead;/templates/...exists only in/3.0/;/uris/...,/lists/<id>/uris,/domains/<host>/urisand/pluginsexist only in/3.1/. Each is a404on the other version;identifiers are serialised differently.
/3.0/emits a UUID asuuid.int, a decimal integer of up to thirty-nine digits;/3.1/emitsuuid.hex, a string. This reachesmember_id,user_idand everyself_linkbuilt from them, and it is why/3.1/exists at all — the integer form is not representable in every JavaScript runtime;every
self_linkandLocationcarries its own version prefix.
23.5. Template URIs¶
A template override is a URI that the server fetches, whichever version wrote
it: a *_uri attribute in /3.0/, or /uris in /3.1/. So whoever
holds the REST credential chooses what the server reads. Pepsi accepts
mailman: (the built-in text) and http(s):, fetched under the limits in
[pepsi-list] TEMPLATE_FETCH_*, and answers 400 to a file: URI,
which upstream would read from its own disk. A file: row that is in the
table anyway (written by hand) is never read: the notice is rendered as if the
row were absent.
23.6. Three honest analogues¶
Three resources describe things that GNU Mailman has and Pepsi does not have. None of them is faked.
/queuesUpstream’s queues are directories of
.pckfiles. Pepsi has one table: a message in flight is a row inpepsi.workqueueat some stage. So this resource answers with the real counts of the messages of lists that exist.incounts every one still in flight,retrythose paused for a later attempt, andbadandshuntthe two terminal states that the dispatcher never re-queues; the pipeline’s other mail is not a list’s queue. Theworkqueue_idstands in for afilebase, and the table’s name appears where a directory would be.POST /queues/ininjects a message at the list’s posting stage, where it runs the whole posting chain. A message posted tobadorshuntis storedfailedortimeoutat that stage, and nothing runs it, just as nothing runs upstream’s queues of those names (a deployment running pepsi-failure-bouncer(1) drops it afterMIN_AGE). A message posted toretryis storedpaused, and the dispatcher resumes it.DELETE /queues/<name>/<id>removes one row. There is novirginqueue: a message that Pepsi composes itself is an ordinary row at its response stage. pepsi-queue(1) remains the real tool./pluginsAlways an empty collection. Pepsi has no plugin interface and no pluggable archivers.
/3.0/has no such resource./reservedUpstream’s own test hook, which upstream marks as not part of the stable API.
GET /reserved/resettruncates every list and archive table, so it exists only on a listener that explicitly opted in withLIST_API_TESTING = yes; anywhere else it is a404indistinguishable from an unknown path, and a test asserts that default. It is there because the compatibility suites reset state between test classes in-process, which an out-of-process server cannot offer. It also restarts the counters of the tables it empties, so the first held message after a reset is request1again, as on upstream’s test database.DELETE /reserved/uids/orphansis a successful no-op: identifiers here are v4 UUIDs and there is no orphan table to cull.
23.7. One resource of Pepsi’s own¶
GET /<api>/lists/<id>/pepsi answers the per-list settings that Pepsi
invented (Settings Pepsi invented). It reports each of
search_trigram, archive_show_addresses, hold_dmarc_failures and
archive_retention_days with the list’s own value, or with the default
(null where the site setting applies). PATCH on the same path sets any of
them, validated exactly as pepsi-list list set-ext validates them; an empty
value removes the list’s own value, and an unknown key is a 400.
They are not attributes under /config, and must not become any: a
whole-resource PUT /config has to carry every writable attribute, so one
more there would break every client built against 3.3.10. No upstream client
asks for this path, so none of them can notice it.
23.8. What needs configuring beyond the credential¶
[pepsi-list] RELEASE_STAGEThe
[stage-<name>]section runningpepsi-stage-list-post. A moderator accepting a held message, andPOST /queues/<name>, both put a message back into the pipeline there, and only the deployment knows what that section is called. Without it those two operations answer400and say so — which beats answering204and dropping the message, because a moderator who accepts a post and watches it never arrive cannot tell that from a delivery failure.[pepsi-list] DEFAULT_LANGUAGEThe bottom of the preference chain, reported by
GET /<api>/system/preferences. Defaults toen.
23.9. A worked example¶
Keep the credential out of the command line: on a shared machine, it would
appear in every process listing and in the shell history. Put it in a file
instead and let curl read it:
$ cat > ~/.mailman-netrc <<'EOF'
machine localhost login restadmin password a-long-random-string
EOF
$ chmod 600 ~/.mailman-netrc
$ alias mmcurl='curl -sS --netrc-file ~/.mailman-netrc'
$ mmcurl http://localhost:8001/3.1/system/versions
{"mailman_version":"GNU Mailman 3.3.10 (Tom Sawyer; implemented by Pepsi …)",
"python_version":"…","api_version":"3.1",
"self_link":"http://localhost:8001/3.1/system/versions",
"http_etag":"\"…\""}
$ mmcurl -X POST --data 'mail_host=lists.example.com' \
http://localhost:8001/3.1/domains
$ mmcurl -X POST --data 'fqdn_listname=devel@lists.example.com' \
http://localhost:8001/3.1/lists
$ mmcurl 'http://localhost:8001/3.1/lists?count=10&page=1'
mailman_version is worth a second look. It has to begin with the string
that a client matches on, and it has to be true. So it names the interface
version first, and it says what is actually running inside the parentheses,
where upstream puts a release code name. Pepsi does not claim to be GNU
Mailman. This resource is the only place where a version is shown, and it needs
the API credential (see Missing HTTPS and HTTP security-header hardening on webmail).
POST /lists/<id>/digest: bump advances the volume, and send and
periodic both go through the sender that the CLI uses, so an issue sent from
a client and one sent from a timer are the same code path and carry the same
numbering.
23.10. What is not implemented¶
A rejected held message or subscription is dropped without the rejection notice that upstream sends. Accepting, discarding and deferring are complete.
A subscription that the API completes sends no welcome message:
send_welcome_messageis accepted and has no effect.[mta] verp_format,verp_regexp,verp_confirm_formatandverp_probesare not configuration. Every copy is VERP’d, in upstream’s${bounces}+${local}=${domain}form with an authentication code appended as one more=field (keyed with[pepsi-list] VERP_SECRET), and probes are on by default ([pepsi-list] BOUNCE_PROBES); no API resource exposes a VERP address, so a client sees no difference.
23.11. Testing it¶
tests/22-list-api-test.sh runs upstream’s own test suites against a
deployed Pepsi: mailmanclient 3.3.5 (344 doctest statements plus 36 test
functions) and Postorius 1.3.13 (30 files, 248 test functions), each with only
its server-starting fixture replaced. The replacements live in
contrib/compat/ and are versioned beside the pinned requirements, so
“unmodified upstream tests” stays literally true of everything except the
fixture. contrib/compat/run-local.sh runs the same two suites against a
build tree instead of a deployment: a throwaway database, a configuration with
only this listener, and the tree’s own pepsi-httpd and pepsi-dispatch,
in a private network namespace so that port 9001, which mailmanclient’s
doctests name, is free.
Both suites pass. What remains are strict expected failures, each one a
declared difference listed in Declared differences from GNU Mailman 3: the fixture
marks the test (or, in mailmanclient’s single using.rst doctest, the
individual statement) and a marked test that starts passing fails the run. The
fixture also lengthens the passwords that the suites give their users: Pepsi
would refuse the short ones, and none of those tests is about passwords. See
Test Suite.