95.1.54. pepsi-archive

search, export, purge and repair the mailing-list archive

Manual section:

1

95.1.54.1.1. Name

pepsi-archive - the operator’s tool for Pepsi’s mailing-list archive.

95.1.54.1.2. Synopsis

pepsi-archive [GLOBAL-OPTIONS] search QUERY [–list ADDRESS] [–limit N]
pepsi-archive [GLOBAL-OPTIONS] show LIST MESSAGE-ID|HASH
pepsi-archive [GLOBAL-OPTIONS] export –list ADDRESS [–since TS] [–until TS]
pepsi-archive [GLOBAL-OPTIONS] purge –list ADDRESS ID [–forget]
pepsi-archive [GLOBAL-OPTIONS] expire –list ADDRESS (–before DATE | –older TIMEFRAME) [-y]
pepsi-archive [GLOBAL-OPTIONS] reindex [–list ADDRESS]
pepsi-archive [GLOBAL-OPTIONS] recount [–list ADDRESS]
pepsi-archive [GLOBAL-OPTIONS] rebuild-threads [–list ADDRESS]
pepsi-archive [GLOBAL-OPTIONS] import –list ADDRESS [–skip-bad] [–rejects FILE] [–no-rebuild] [–mboxrd] FILE…

95.1.54.1.3. Description

pepsi-archive is the operator’s command-line tool for the mailing-list archive. It connects through the shared [pepsi-postgres] section and, when started as root, continues as the pepsi service account. It is not setuid and must not be made so.

The archive reimplements HyperKitty from GNU Mailman 3. Its data model, its threading rules, its Message-ID hash and its URL scheme are upstream’s (copyright the Free Software Foundation and its contributors), so that the archive links of a migrated site keep working. The manual’s Archives chapter says what each of the two search mechanisms is good at, and what a purge does and does not delete.

95.1.54.1.4. The one it exists for

purge is the command that this tool exists for: it removes from a public archive a message that somebody posted by accident. It deletes the message, its attachments and its votes, renumbers the thread, repairs the counters and writes an event_log row naming who did it.

It leaves a tombstone: the Message-ID, kept for [pepsi-list] TOMBSTONE_RETENTION days (default 90), during which the archive refuses to store that message again. Pepsi itself archives a purged message again in only two ways, through a re-delivery or through a re-run import, and both are accidents. --forget skips the tombstone.

95.1.54.1.5. Commands

search QUERY

Search as the operator, who sees every list. --list scopes the search, which is both faster (one partition) and better (the list’s own language dictionary). The output says which mechanism answered, and says so when the list does not index message bodies for substring search.

show LIST ID

Print one archived message. ID may be a Message-ID, a Message-ID hash, or an archive URL containing one.

export --list ADDRESS

Write the list’s archive to standard output as an mboxrd mbox, oldest first. --since and --until take RFC 3339 timestamps.

purge --list ADDRESS ID [--forget]

See above.

expire --list ADDRESS (--before DATE | --older TIMEFRAME) [-y|--yes]

Delete every archived message of the list whose date is before a cutoff. Give exactly one of the two options:

--before DATE

YYYY, YYYY-MM or YYYY-MM-DD means the start of that year, month or day in UTC, so --before 2024 deletes everything up to the end of 2023 and keeps all of 2024. A full RFC 3339 timestamp is accepted too.

--older TIMEFRAME

An age such as 2 years, 42 months, 10 minutes or 5402 s (an ISO 8601 duration like P2Y works too). The cutoff is the current time minus that age. Years and months are counted on the calendar in UTC. Because the command line does not change, it can run from a timer or a cron job, for example to meet a data-protection deletion deadline: pepsi-archive expire --list team@lists.example.org --older '2 years' --yes.

The command first counts the messages that would go and asks for confirmation. -y or --yes skips the question. When standard input is not a terminal and --yes is missing, nothing is deleted and the command exits with an error. When no message is old enough, it says so and succeeds without asking. Each run that deletes something is recorded in the audit log as archive.expire. Expired messages leave no tombstone, unlike purge.

The date is the one the archive shows, which comes from the message’s Date: header. The deletion works in batches because the predicate is a date while the table is partitioned by list, so PostgreSQL cannot prune partitions: one statement over a year of a busy list would hold a long transaction. The same deletion runs on a schedule as the retention sweep of pepsi-list tasks --once (the pepsi-list-tasks.timer unit), for every list with a retention; see ARCHIVE_RETENTION below.

reindex [--list ADDRESS]

Rebuild search_text and trgm_text. This is how a changed trigram tier takes effect, in both directions and with no schema change.

recount [--list ADDRESS]

Rebuild the counter columns from the rows themselves. This repairs the one documented way of breaking them, which is writing to the archive tables directly.

rebuild-threads [--list ADDRESS]

Reattach replies whose parent arrived after them, and renumber every thread. Upstream leaves those threads split for ever; this is the repair.

import --list ADDRESS [--skip-bad] [--rejects FILE] [--no-rebuild] [--mboxrd] FILE…

Import one or more mbox files into a list. This is the universal way into the archive. It is also the other half of the export round trip, and that round trip is the cheapest possible test of both commands.

--mboxrd reads the >From quoting of body lines as mboxrd: every line that is a run of > followed by From `` loses one ``>, which is the exact inverse of export. Without it the files are read as mboxo, the format Mailman 2 and other older archivers write: only a line that is exactly >From loses its >, and >>From stays as the author wrote it. Use --mboxrd for a Pepsi export and any other mboxrd file.

The repairs are in the reader, not in a separate script that an operator has to remember: a body line beginning From `` does not split a message (a separator must follow a blank line *and* carry a four-digit year); a missing ``Message-ID is generated deterministically from the message’s own bytes, so re-running an interrupted import produces the same id rather than a duplicate; <abc@host> (added by postmaster@…) is repaired; a missing Date falls back to the mbox envelope line; a folded Subject is joined; and Content-Length is dropped, because it described a framing that does not exist once the message is out of the mbox.

Threads are reattached and renumbered at the end, automatically. Otherwise an out-of-order import leaves them split. That is what happens with upstream’s batch mode, and nobody notices it for years.

A message that cannot be imported is reported, to --rejects when a file is named and to standard error when one is not, and the command exits non-zero unless --skip-bad says the loss is expected. Re-running is safe: the import recognises each message by its Message-ID, so a resumed import goes by content rather than by a file’s timestamp.

95.1.54.1.6. Configuration

[pepsi-list]

SEARCH_TRIGRAM

The site’s ceiling for the substring-search tier: off, short (default) or full. A list chooses any tier up to it with pepsi-list list set-ext.

TOMBSTONE_RETENTION

How long a purge tombstone is kept, in days (default 90). The sweep that removes expired ones runs in pepsi-list tasks --once.

ARCHIVE_RETENTION

The site’s archive retention in days, applied by the retention sweep of pepsi-list tasks --once: a list’s own archive_retention_days (set with pepsi-list list set-ext) wins, 0 included; a list without one uses this value, whose own default is 0; and 0 keeps everything.

ARCHIVE_PARTITIONS

pepsi-setup(1) reads it to create the partitions, and pepsi-list check compares it against them. Fixed at install: PostgreSQL has no online re-partitioning.

95.1.54.1.7. Exit status

Non-zero on a configuration or database failure, or when the message named does not exist. A verb that found nothing to do is not a failure.

95.1.54.1.8. See also

pepsi-list(1), pepsi-stage-list-post(1), pepsi-stage-list(1), pepsi-setup(1), pepsi.conf(5).