85.1.48. pepsi-archive

search, export, purge and repair the mailing-list archive

Manual section:

1

85.1.48.1.1. Name

pepsi-archive - the operator’s tool for Pepsi’s mailing-list archive.

85.1.48.1.2. Synopsis

pepsi-archive [GLOBAL-OPTIONS] search QUERY [–list ADDRESS] [–limit N]

pepsi-archive [GLOBAL-OPTIONS] show LIST MESSAGE-ID|HASH

pepsi-archive [GLOBAL-OPTIONS] export –list ADDRESS [–since TS] [–until TS]

pepsi-archive [GLOBAL-OPTIONS] purge –list ADDRESS ID [–forget]

pepsi-archive [GLOBAL-OPTIONS] expire –list ADDRESS –before TS

pepsi-archive [GLOBAL-OPTIONS] reindex [–list ADDRESS]

pepsi-archive [GLOBAL-OPTIONS] recount [–list ADDRESS]

pepsi-archive [GLOBAL-OPTIONS] rebuild-threads [–list ADDRESS]

pepsi-archive [GLOBAL-OPTIONS] import –list ADDRESS [–skip-bad] [–rejects FILE] [–no-rebuild] FILE…

85.1.48.1.3. Description

pepsi-archive is the operator’s command-line tool for the mailing-list archive. It connects through the shared [pepsi-postgres] section and, when started as root, continues as the pepsi service account. It is not setuid and must not be made so.

The archive is a reimplementation of GNU Mailman 3’s HyperKitty: its data model, its threading rules, its Message-ID hash and its URL scheme are upstream’s, copyright the Free Software Foundation and its contributors, so that a migrated site’s archive links keep working. The manual’s Archives chapter says what each of the two search mechanisms is good at and what a purge does and does not delete.

85.1.48.1.4. The one it exists for

purge is the command this tool exists for: remove a message somebody posted by accident from a public archive. It deletes the message, its attachments and its votes, renumbers the thread, repairs the counters and writes an event_log row naming who did it.

It leaves a tombstone: the Message-ID, kept for [pepsi-list] TOMBSTONE_RETENTION days (default 90), during which the archive refuses to store that message again. There are exactly two ways a purged message comes back — a re-delivery and a re-run import — and both are accidents. --forget skips the tombstone.

85.1.48.1.5. Commands

search QUERY

Search as the operator, who sees every list. --list scopes the search, which is both faster (one partition) and better (the list’s own language dictionary). The output says which mechanism answered, and says so when the list does not index message bodies for substring search.

show LIST ID

Print one archived message. ID may be a Message-ID, a Message-ID hash, or an archive URL containing one.

export --list ADDRESS

Write the list’s archive to standard output as an mboxrd mbox, oldest first. --since and --until take RFC 3339 timestamps.

purge --list ADDRESS ID [--forget]

See above.

expire --list ADDRESS --before TIMESTAMP

Delete everything older than the timestamp, in batches. Batched because the predicate is a date while the partitioning is by list, so it cannot prune: one statement over a year of a busy list would hold a long transaction. The same deletion runs on a schedule as the retention sweep of pepsi-list tasks --once (the pepsi-list-tasks.timer unit), for every list with a retention; see ARCHIVE_RETENTION below.

reindex [--list ADDRESS]

Rebuild search_text and trgm_text. This is how a changed trigram tier takes effect, in both directions and with no schema change.

recount [--list ADDRESS]

Rebuild the counter columns from the rows themselves. The repair for the one documented way to break them: writing to the archive tables directly.

rebuild-threads [--list ADDRESS]

Reattach replies whose parent arrived after them, and renumber every thread. Upstream leaves those threads split for ever; this is the repair.

import --list ADDRESS [--skip-bad] [--rejects FILE] [--no-rebuild] FILE…

Import one or more mbox files into a list. The universal path, and the other half of the export round trip — which is the cheapest possible test of both.

The repairs are in the reader, not in a separate script an operator has to remember: a body line beginning From `` does not split a message (a separator must follow a blank line *and* carry a four-digit year); a missing ``Message-ID is generated deterministically from the message’s own bytes, so re-running an interrupted import produces the same id rather than a duplicate; <abc@host> (added by postmaster@…) is repaired; a missing Date falls back to the mbox envelope line; a folded Subject is joined; and Content-Length is dropped, because it described a framing that does not exist once the message is out of the mbox.

Threads are reattached and renumbered at the end, automatically. An out-of-order import otherwise leaves them split — which is what happens with upstream’s batch mode, and which nobody notices for years.

A message that cannot be imported is reported, to --rejects when a file is named and to standard error when one is not, and the command exits non-zero unless --skip-bad says the loss is expected. Re-running is safe: every message is idempotent on its Message-ID, so resumption keys on content rather than on a file’s timestamp.

85.1.48.1.6. Configuration

[pepsi-list]

SEARCH_TRIGRAM

Site ceiling on the substring-search tier: off, short (default) or full. A list chooses up to it with pepsi-list list set-ext.

TOMBSTONE_RETENTION

How long a purge tombstone is kept, in days (default 90). The sweep that removes expired ones runs in pepsi-list tasks --once.

ARCHIVE_RETENTION

The site’s archive retention in days, applied by the retention sweep of pepsi-list tasks --once: a list’s own archive_retention_days (set with pepsi-list list set-ext) wins, 0 included; a list without one uses this value, whose own default is 0; and 0 keeps everything.

ARCHIVE_PARTITIONS

Used by pepsi-setup(1), which creates the partitions, and compared against them by pepsi-list check. Fixed at install: PostgreSQL has no online re-partitioning.

85.1.48.1.7. Exit status

Non-zero on a configuration or database failure, or when the message named does not exist. A verb that found nothing to do is not a failure.

85.1.48.1.8. See also

pepsi-list(1), pepsi-stage-list-post(1), pepsi-stage-list(1), pepsi-setup(1), pepsi.conf(5).