55. pepsi-detect-language¶
Classify one message the way the pipeline would — offline, for diagnosis and tuning.
55.1. Role¶
pepsi-detect-language is a diagnostic and tuning tool, not a stage. It
reads a single message from a file or standard input, runs
pepsi-stage-detect-language’s own classification over it, and prints the
resulting state.language value, and nothing else, to standard output.
It never opens the database and needs no pepsi.workqueue row, so a message that
was classified wrongly in production can be investigated on a saved copy, on any
host, as often as you like. Reference: pepsi-detect-language(1).
It is the same executable as pepsi-stage-detect-language, installed as a
second name for it. The two share the entire classification path, so what the
tool reports is what the pipeline did, not the result of a re-implementation that
can drift. They also share the statically linked language models, which is why
both names ship in the same (separate) package.
55.2. Diagnosing a misclassification¶
--explain writes every intermediate step to standard error, leaving standard
output untouched:
pepsi-detect-language --explain message.eml >/dev/null
It shows how the input was framed, which candidate languages were in play and where that set came from, the body text that was extracted, the prose left after filtering it, and the confidence of every candidate with the top-three and 5 % rules applied. The filtered prose is the single most useful block, because that, and not the extracted text, is what the detector was given.
55.3. Why there are two text blocks¶
The stage does not classify the body as extracted. It first removes everything that cannot be a word: URLs, e-mail addresses, base64 keys, DNS records, identifiers. This is not mere tidiness: if those tokens are left in, they decide the answer, and a notification e-mail, a DNS-record dump or a spam body of two bare URLs is classified from its machine text as a language that nobody wrote in it.
The stage runs lingua in full-accuracy mode. Its low-accuracy mode looks for
an n-gram unique to one candidate language before it scores anything and
returns that language at probability 1 if exactly one matches. A single
non-word, such as an address in a quoted attribution line, a base64 key or an
IPv6 address, would then decide the entire message with total confidence,
without the rest of the body being examined.
So when a classification looks wrong, read the second block. If the prose there is the message a human would read and the answer is still wrong, the candidate set is the next thing to try (below). If the second block says there is no prose, the body genuinely held none (as with a bulk mailer whose message is two bare URLs), and no language is the right answer.
55.4. Tuning LANGUAGES¶
The usual cause of a confident wrong answer is the candidate set: a language the
traffic never contains still competes for probability, and short texts are where
it wins. --languages overrides the configured set (and skips reading the
configuration entirely), so a set can be tried before it is configured:
pepsi-detect-language --languages "en de" message.eml
* selects every supported language, exactly as in the option. The
pepsi-setup wizard configures this set unless the operator narrows it, so
it is the set to compare against when a production classification looks wrong:
pepsi-detect-language --languages '*' message.eml
Without --languages the candidate set comes from the configuration: the
single [stage-*] section running pepsi-stage-detect-language. When a
configuration has more than one, name the one to use with --stage NAME; when
it has none, the stage’s built-in default set is used and a warning says so.
Over a corpus:
for m in ~/Maildir/cur/*; do
printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m"
done | sort | uniq -c
55.5. Input¶
mbox, RFC 5322 or a bare body: all three are accepted, and the tool detects which one it is given. A leading mbox
From_separator line is stripped (with mboxrd>Fromquoting undone); a bare body is wrapped in a minimalContent-Type. A header block is recognised only when every line before the first blank line is a field or a folded continuation, so a body opening withRe: the meetingis not mistaken for one;--body-onlyforces that reading.Headers are kept, and never classified. Header text does not reach the detector, since the stage extracts only inline body parts.
Content-Type,charsetandContent-Transfer-Encodingstill decide what the body is. Discarding them would classify a quoted-printable, base64 or HTML message from its encoding rather than its prose, which is one of the failures the tool exists to find.What is displayed is what is read. For a
multipart/alternativethe HTML part is what most clients show, so it is classified, without the elements that an inline style or thehiddenattribute hides, without invisible characters, and with look-alike letters inside a Latin word folded. When it holds no prose at all, only images and links, thetext/plainsibling is used instead, and--explainsays so.Line endings are normalised to CRLF as pepsi-ingress does, so a message read from a file classifies exactly as the queued copy did.
55.6. Output¶
One line: the Accept-Language-style string, e.g. en;q=0.55, de;q=0.45. When
the stage would record no language (no inline text, no prose in it, too little
prose, or nothing above the confidence threshold), standard output is empty
and the reason goes to standard error. That is the stage’s own behaviour, not an
error.
55.7. See also¶
pepsi-stage-detect-language, pepsi-stage-block-language, pepsi-detect-language(1), pepsi.state(7).