70. pepsi-archive

Search, export, import, purge and repair the mailing-list archive.

70.1. Role

pepsi-archive is the operator’s command-line tool for the mailing-list archive. It connects through the shared [pepsi-postgres] section and, started as root, continues as the pepsi service account before connecting, so no sudo -u pepsi is needed. It is not setuid and must not be made so. Reference: pepsi-archive(1).

The archive is a reimplementation of GNU Mailman 3’s HyperKitty: its data model, its threading rules, its Message-ID hash and its URL scheme are upstream’s, copyright the Free Software Foundation and its contributors, under the GPL. See Archives, which says so at greater length and names what was taken.

70.2. The one it exists for

purge is the command this tool exists for: remove a message somebody posted by accident from a public archive. It deletes the message, its attachments and its votes, renumbers the thread, repairs the counters, and writes an event_log row naming who did it.

It leaves a tombstone — the Message-ID, kept for [pepsi-list] TOMBSTONE_RETENTION days (default 90) — during which the archive refuses to store that message again. There are exactly two ways a purged message comes back, a re-delivery and a re-run import, and both are accidents. --forget skips the tombstone.

70.3. Commands

  • search QUERY — search as the operator, who sees every list. --list scopes the search, which is both faster (one partition) and better (the list’s own language dictionary). The output says which mechanism answered, and says so when the list does not index message bodies for substring search.

  • show LIST ID — print one archived message. ID may be a Message-ID, a Message-ID hash, or an archive URL containing one.

  • export --list ADDRESS — write the list’s archive to standard output as an mboxrd mbox, oldest first; --since / --until take RFC 3339 timestamps.

  • import --list ADDRESS FILE… — import mbox files; see below.

  • expire --list ADDRESS --before TIMESTAMP — delete everything older than the timestamp, in batches. Batched because the predicate is a date while the partitioning is by list, so it cannot prune: one statement over a year of a busy list would hold a long transaction. The retention sweep of pepsi-list tasks --once runs the same deletion daily for every list with a retention (see Archives).

  • reindex — rebuild search_text and trgm_text. This is how a changed trigram tier takes effect, in both directions and with no schema change.

  • recount — rebuild the counter columns from the rows themselves; the repair for the one documented way to break them, which is writing to the archive tables directly.

  • rebuild-threads — reattach replies whose parent arrived after them and renumber every thread. Upstream leaves those threads split for ever; this is the repair.

70.4. Import, and the round trip

import is the universal path into the archive and the other half of the export round trip — which is the cheapest possible test of both.

The mbox repairs are in the reader, not in a separate script an operator has to remember. A body line beginning From `` does not split a message: a separator must follow a blank line *and* carry a four-digit year. A missing ``Message-ID is generated deterministically from the message’s own bytes, so re-running an interrupted import produces the same id rather than a duplicate. The <abc@host> (added by postmaster@…) form is repaired; a missing Date falls back to the mbox envelope line; a folded Subject is joined; and Content-Length is dropped, because it described a framing that does not exist once the message is out of the mbox.

Threads are reattached and renumbered at the end, automatically (unless --no-rebuild). An out-of-order import otherwise leaves them split — which is what happens with upstream’s batch mode, and which nobody notices for years.

A message that cannot be imported is reported: to --rejects when a file is named and to standard error when one is not, and the command exits non-zero unless --skip-bad says the loss is expected. Re-running is safe, because every message is idempotent on its Message-ID, so resumption keys on content rather than on a file’s timestamp.

70.5. Configuration

[pepsi-list]: SEARCH_TRIGRAM — the site ceiling on the substring-search tier (off, short by default, or full), which a list chooses up to with pepsi-list list set-ext; TOMBSTONE_RETENTION in days, swept by pepsi-list tasks --once; ARCHIVE_RETENTION in days, the default the retention sweep of pepsi-list tasks --once applies to a list without its own archive_retention_days (0, the default, keeps everything); and ARCHIVE_PARTITIONS, used by pepsi-setup to create the partitions (and compared against them by pepsi-list check) and fixed at install, because PostgreSQL has no online re-partitioning.

70.6. See also

Archives, Mailing lists, pepsi-list, pepsi-stage-list-post, pepsi-httpd, pepsi-archive(1).