95.1.54. pepsi-archive¶
search, export, purge and repair the mailing-list archive
- Manual section:
1
95.1.54.1.1. Name¶
pepsi-archive - the operator’s tool for Pepsi’s mailing-list archive.
95.1.54.1.2. Synopsis¶
95.1.54.1.3. Description¶
pepsi-archive is the operator’s command-line tool for the mailing-list
archive. It connects through the shared [pepsi-postgres] section and, when
started as root, continues as the pepsi service account. It is not
setuid and must not be made so.
The archive reimplements HyperKitty from GNU Mailman 3. Its data model, its threading rules, its Message-ID hash and its URL scheme are upstream’s (copyright the Free Software Foundation and its contributors), so that the archive links of a migrated site keep working. The manual’s Archives chapter says what each of the two search mechanisms is good at, and what a purge does and does not delete.
95.1.54.1.4. The one it exists for¶
purge is the command that this tool exists for: it removes from a public
archive a message that somebody posted by accident. It deletes the message, its
attachments and its votes, renumbers the thread, repairs the counters and
writes an event_log row naming who did it.
It leaves a tombstone: the Message-ID, kept for [pepsi-list]
TOMBSTONE_RETENTION days (default 90), during which the archive refuses to
store that message again. Pepsi itself archives a purged message again in only
two ways, through a re-delivery or through a re-run import, and both are
accidents.
--forget skips the tombstone.
95.1.54.1.5. Commands¶
- search QUERY
Search as the operator, who sees every list.
--listscopes the search, which is both faster (one partition) and better (the list’s own language dictionary). The output says which mechanism answered, and says so when the list does not index message bodies for substring search.- show LIST ID
Print one archived message. ID may be a
Message-ID, a Message-ID hash, or an archive URL containing one.- export
--listADDRESS Write the list’s archive to standard output as an
mboxrdmbox, oldest first.--sinceand--untiltake RFC 3339 timestamps.- purge
--listADDRESS ID [--forget] See above.
- expire
--listADDRESS (--beforeDATE |--olderTIMEFRAME) [-y|--yes] Delete every archived message of the list whose date is before a cutoff. Give exactly one of the two options:
--beforeDATEYYYY,YYYY-MMorYYYY-MM-DDmeans the start of that year, month or day in UTC, so--before 2024deletes everything up to the end of 2023 and keeps all of 2024. A full RFC 3339 timestamp is accepted too.--olderTIMEFRAMEAn age such as
2 years,42 months,10 minutesor5402 s(an ISO 8601 duration likeP2Yworks too). The cutoff is the current time minus that age. Years and months are counted on the calendar in UTC. Because the command line does not change, it can run from a timer or a cron job, for example to meet a data-protection deletion deadline:pepsi-archive expire --list team@lists.example.org --older '2 years' --yes.
The command first counts the messages that would go and asks for confirmation.
-yor--yesskips the question. When standard input is not a terminal and--yesis missing, nothing is deleted and the command exits with an error. When no message is old enough, it says so and succeeds without asking. Each run that deletes something is recorded in the audit log asarchive.expire. Expired messages leave no tombstone, unlike purge.The date is the one the archive shows, which comes from the message’s
Date:header. The deletion works in batches because the predicate is a date while the table is partitioned by list, so PostgreSQL cannot prune partitions: one statement over a year of a busy list would hold a long transaction. The same deletion runs on a schedule as the retention sweep ofpepsi-list tasks --once(the pepsi-list-tasks.timer unit), for every list with a retention; seeARCHIVE_RETENTIONbelow.- reindex [
--listADDRESS] Rebuild
search_textandtrgm_text. This is how a changed trigram tier takes effect, in both directions and with no schema change.- recount [
--listADDRESS] Rebuild the counter columns from the rows themselves. This repairs the one documented way of breaking them, which is writing to the archive tables directly.
- rebuild-threads [
--listADDRESS] Reattach replies whose parent arrived after them, and renumber every thread. Upstream leaves those threads split for ever; this is the repair.
- import
--listADDRESS [--skip-bad] [--rejectsFILE] [--no-rebuild] [--mboxrd] FILE… Import one or more mbox files into a list. This is the universal way into the archive. It is also the other half of the
exportround trip, and that round trip is the cheapest possible test of both commands.--mboxrdreads the>Fromquoting of body lines asmboxrd: every line that is a run of>followed byFrom `` loses one ``>, which is the exact inverse ofexport. Without it the files are read asmboxo, the format Mailman 2 and other older archivers write: only a line that is exactly>Fromloses its>, and>>Fromstays as the author wrote it. Use--mboxrdfor a Pepsi export and any othermboxrdfile.The repairs are in the reader, not in a separate script that an operator has to remember: a body line beginning
From `` does not split a message (a separator must follow a blank line *and* carry a four-digit year); a missing ``Message-IDis generated deterministically from the message’s own bytes, so re-running an interrupted import produces the same id rather than a duplicate;<abc@host> (added by postmaster@…)is repaired; a missingDatefalls back to the mbox envelope line; a foldedSubjectis joined; andContent-Lengthis dropped, because it described a framing that does not exist once the message is out of the mbox.Threads are reattached and renumbered at the end, automatically. Otherwise an out-of-order import leaves them split. That is what happens with upstream’s batch mode, and nobody notices it for years.
A message that cannot be imported is reported, to
--rejectswhen a file is named and to standard error when one is not, and the command exits non-zero unless--skip-badsays the loss is expected. Re-running is safe: the import recognises each message by itsMessage-ID, so a resumed import goes by content rather than by a file’s timestamp.
95.1.54.1.6. Configuration¶
[pepsi-list]
SEARCH_TRIGRAMThe site’s ceiling for the substring-search tier:
off,short(default) orfull. A list chooses any tier up to it withpepsi-list list set-ext.TOMBSTONE_RETENTIONHow long a purge tombstone is kept, in days (default 90). The sweep that removes expired ones runs in
pepsi-list tasks --once.ARCHIVE_RETENTIONThe site’s archive retention in days, applied by the retention sweep of
pepsi-list tasks --once: a list’s ownarchive_retention_days(set withpepsi-list list set-ext) wins,0included; a list without one uses this value, whose own default is0; and0keeps everything.ARCHIVE_PARTITIONSpepsi-setup(1) reads it to create the partitions, and
pepsi-list checkcompares it against them. Fixed at install: PostgreSQL has no online re-partitioning.
95.1.54.1.7. Exit status¶
Non-zero on a configuration or database failure, or when the message named does not exist. A verb that found nothing to do is not a failure.
95.1.54.1.8. See also¶
pepsi-list(1), pepsi-stage-list-post(1), pepsi-stage-list(1), pepsi-setup(1), pepsi.conf(5).