44. pepsi-detect-language

Classify one message the way the pipeline would — offline, for diagnosis and tuning.

44.1. Role

pepsi-detect-language is a diagnostic and tuning tool, not a stage. It reads a single message from a file or standard input, runs pepsi-stage-detect-language’s own classification over it, and prints the resulting state.language value — and nothing else — to standard output.

It never opens the database and needs no pepsi.ingress row, so a message that was classified wrongly in production can be investigated on a saved copy, on any host, as often as you like. Reference: pepsi-detect-language(1).

It is the same executable as pepsi-stage-detect-language, installed as a second name for it: the two share the entire classification path — so what the tool reports is what the pipeline did, not a re-implementation that can drift — and the statically linked language models, which is why they also ship in the same (separate) package.

44.2. Diagnosing a misclassification

--explain writes every intermediate step to standard error, leaving standard output untouched:

pepsi-detect-language --explain message.eml >/dev/null

It shows how the input was framed, which candidate languages were in play and where that set came from, the body text that was extracted, the prose left after filtering it — the single most useful block, since that, and not the extracted text, is what the detector was given — and the confidence of every candidate with the TOP_N and 5 % rules applied.

44.3. Why there are two text blocks

The stage does not classify the body as extracted. It first removes everything that cannot be a word: URLs, e-mail addresses, base64 keys, DNS records, identifiers. This is not tidiness — measured on real mail, those tokens decide the answer. A Mastodon follow notification and a DNS-record dump both classified as Yoruba, and a spam body of two bare URLs as Tsonga, purely on the strength of the machine text in them; with that text removed all three are correctly English.

It used to be worse still. The stage ran lingua’s low-accuracy mode, which looks for an n-gram unique to one candidate language before it scores anything and returns that language at probability 1 if exactly one matches — so a single non-word decided the entire message, with total confidence and without the rest of the body being examined. <graham@grahamcox.co.uk> in a quoted attribution line turned 3.3 kB of English into Estonian; a base64 DKIM key turned a DNS notification into Yoruba, an IPv6 address in the same message into Polish. That mode is no longer used (it also measured slower, and needed more memory, than full accuracy once the filtering was in place).

So when a classification looks wrong, read the second block. If the prose there is the message a human would read and the answer is still wrong, the candidate set is the next thing to try (below). If the second block says there is no prose, the body genuinely held none — a bulk mailer whose plain part is two bare URLs — and no language is the right answer.

44.4. Tuning LANGUAGES

The usual cause of a confident wrong answer is the candidate set: a language the traffic never contains still competes for probability, and short texts are where it wins. --languages overrides the configured set (and skips reading the configuration entirely), so a set can be tried before it is configured:

pepsi-detect-language --languages "en de" message.eml

* selects every supported language, exactly as in the option — which is what the pepsi-setup wizard configures unless the operator narrows it, and so the set to compare against when a production classification looks wrong:

pepsi-detect-language --languages '*' message.eml

Over a corpus:

for m in ~/Maildir/cur/*; do
    printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m"
done | sort | uniq -c

44.5. Input

  • mbox, RFC 5322 or a bare body — all three are accepted, and which one it is being given is detected. A leading mbox From_ separator line is stripped (with mboxrd >From quoting undone); a bare body is wrapped in a minimal Content-Type. A header block is recognised only when every line before the first blank line is a field or a folded continuation, so a body opening with Re: the meeting is not mistaken for one; --body-only forces that reading.

  • Headers are kept, and never classified. Header text does not reach the detector — the stage extracts inline body parts only — but Content-Type, charset and Content-Transfer-Encoding decide what the body is. Discarding them would classify a quoted-printable, base64 or HTML message from its encoding rather than its prose, which is one of the failures the tool exists to find.

  • The plain alternative wins, unless it is a stub. For a multipart/alternative the text/plain part is what a sender wrote, so it is preferred. When it holds no prose at all — the bulk-mailer shape, two bare URLs where the message should be — the HTML sibling is used instead, and --explain says so.

  • Line endings are normalised to CRLF as pepsi-ingress does, so a message read from a file classifies exactly as the queued copy did.

44.6. Output

One line: the Accept-Language-style string, e.g. en;q=1, de;q=0.45. When the stage would record no language — no inline text, too little of it, or nothing above the confidence threshold — standard output is empty and the reason goes to standard error. That is the stage’s own behaviour, not an error.

44.7. See also

pepsi-stage-detect-language, pepsi-stage-block-language, pepsi-detect-language(1), pepsi.state(7).