51. pepsi-detect-language

Classify one message the way the pipeline would — offline, for diagnosis and tuning.

51.1. Role

pepsi-detect-language is a diagnostic and tuning tool, not a stage. It reads a single message from a file or standard input, runs pepsi-stage-detect-language’s own classification over it, and prints the resulting state.language value — and nothing else — to standard output.

It never opens the database and needs no pepsi.workqueue row, so a message that was classified wrongly in production can be investigated on a saved copy, on any host, as often as you like. Reference: pepsi-detect-language(1).

It is the same executable as pepsi-stage-detect-language, installed as a second name for it: the two share the entire classification path — so what the tool reports is what the pipeline did, not a re-implementation that can drift — and the statically linked language models, which is why they also ship in the same (separate) package.

51.2. Diagnosing a misclassification

--explain writes every intermediate step to standard error, leaving standard output untouched:

pepsi-detect-language --explain message.eml >/dev/null

It shows how the input was framed, which candidate languages were in play and where that set came from, the body text that was extracted, the prose left after filtering it — the single most useful block, since that, and not the extracted text, is what the detector was given — and the confidence of every candidate with the top-three and 5 % rules applied.

51.3. Why there are two text blocks

The stage does not classify the body as extracted. It first removes everything that cannot be a word: URLs, e-mail addresses, base64 keys, DNS records, identifiers. This is not tidiness: left in, those tokens decide the answer, and a notification e-mail, a DNS-record dump or a spam body of two bare URLs is classified from its machine text as a language nobody wrote in it.

The stage runs lingua in full-accuracy mode. Its low-accuracy mode looks for an n-gram unique to one candidate language before it scores anything and returns that language at probability 1 if exactly one matches, so a single non-word — an address in a quoted attribution line, a base64 key, an IPv6 address — would decide the entire message with total confidence, without the rest of the body being examined.

So when a classification looks wrong, read the second block. If the prose there is the message a human would read and the answer is still wrong, the candidate set is the next thing to try (below). If the second block says there is no prose, the body genuinely held none — a bulk mailer whose plain part is two bare URLs — and no language is the right answer.

51.4. Tuning LANGUAGES

The usual cause of a confident wrong answer is the candidate set: a language the traffic never contains still competes for probability, and short texts are where it wins. --languages overrides the configured set (and skips reading the configuration entirely), so a set can be tried before it is configured:

pepsi-detect-language --languages "en de" message.eml

* selects every supported language, exactly as in the option — which is what the pepsi-setup wizard configures unless the operator narrows it, and so the set to compare against when a production classification looks wrong:

pepsi-detect-language --languages '*' message.eml

Without --languages the candidate set comes from the configuration: the single [stage-*] section running pepsi-stage-detect-language. When a configuration has more than one, name the one to use with --stage NAME; when it has none, the stage’s built-in default set is used and a warning says so.

Over a corpus:

for m in ~/Maildir/cur/*; do
    printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m"
done | sort | uniq -c

51.5. Input

• mbox, RFC 5322 or a bare body — all three are accepted, and which one it is being given is detected. A leading mbox From_ separator line is stripped (with mboxrd >From quoting undone); a bare body is wrapped in a minimal Content-Type. A header block is recognised only when every line before the first blank line is a field or a folded continuation, so a body opening with Re: the meeting is not mistaken for one; --body-only forces that reading.

• Headers are kept, and never classified. Header text does not reach the detector — the stage extracts inline body parts only — but Content-Type, charset and Content-Transfer-Encoding decide what the body is. Discarding them would classify a quoted-printable, base64 or HTML message from its encoding rather than its prose, which is one of the failures the tool exists to find.

• The plain alternative wins, unless it is a stub. For a multipart/alternative the text/plain part is what a sender wrote, so it is preferred. When it holds no prose at all — the bulk-mailer shape, two bare URLs where the message should be — the HTML sibling is used instead, and --explain says so.

• Line endings are normalised to CRLF as pepsi-ingress does, so a message read from a file classifies exactly as the queued copy did.

51.6. Output

One line: the Accept-Language-style string, e.g. en;q=1, de;q=0.45. When the stage would record no language — no inline text, too little of it, or nothing above the confidence threshold — standard output is empty and the reason goes to standard error. That is the stage’s own behaviour, not an error.

51.7. See also

pepsi-stage-detect-language, pepsi-stage-block-language, pepsi-detect-language(1), pepsi.state(7).