70.1.22. pepsi-detect-language

classify one message the way the pipeline would, offline

Manual section:

1

70.1.22.1.1. Name

pepsi-detect-language - diagnostic and tuning tool for Pepsi’s language detection.

70.1.22.1.2. Synopsis

pepsi-detect-language [OPTIONS] [FILE]

70.1.22.1.3. Description

pepsi-detect-language answers one question: what would pepsi-stage-detect-language(1) record for this message? It reads a single message, runs the stage’s own classification over it, and writes the resulting state.language value — and nothing else — to standard output.

It is a diagnostic and tuning tool, not part of the pipeline. It never opens the database, never touches the queue and needs no pepsi.ingress row, so it is safe to run on a saved copy of a message that was classified wrongly in production, on a host that is not the mail server, and repeatedly with different –languages sets while narrowing down the option to configure.

Use it to:

• reproduce a misclassification from a saved message, and see with –explain why the detector chose what it chose — the extracted text, the prose that survived filtering (which is what the detector actually saw), and the confidence of every candidate language;

• tune the stage’s LANGUAGES option: a candidate language that the traffic never contains still competes for probability, and dropping it is often the whole fix. Re-run the same message against different sets with –languages until the answer is right, then configure that set;

• check a corpus in a loop (for m in Maildir/cur/*; do …; done) before and after a configuration change.

It is the same executable as pepsi-stage-detect-language(1), installed as a second name for it — the two share the whole classification path, and the statically linked language models, which is why they are also packaged together.

70.1.22.1.4. Input

The message is read from FILE, or from standard input when no file is given (or when FILE is -). One message per invocation. Three input shapes are accepted, and which one it is being given is detected:

mbox

A leading From_ separator line — the mbox convention: From followed by a space and the envelope sender — is stripped, and mboxrd >From quoting in the body is undone. (Unquoting is done only when a separator line was actually found: elsewhere, a line beginning >From is somebody’s quoted text.) Note that a file holding several mbox messages is treated as one message; split it first.

a complete RFC 5322 message

Used as-is, exactly as the stage would see it after pepsi-ingress(1) stored it.

a bare body, with no header block

Wrapped in a minimal Content-Type: text/plain header — declaring utf-8 or iso-8859-1 according to what the bytes are — so the MIME parser reads it as text.

A header block is recognised only when every line before the first blank line is a header field or a folded continuation of one. That is what keeps a body opening with something like Re: the meeting from being mistaken for a one-field header block; –body-only forces the bare-body reading when the input really is such a body.

Line endings are normalised to CRLF, as pepsi-ingress(1) does, so a message saved out of a file classifies identically to the copy the queue held.

The header block is kept, and that is deliberate. No header text ever reaches the classifier — the stage extracts only inline body parts, so Subject:, From: and the Received: trace are discarded from what is classified. But Content-Type, charset and Content-Transfer-Encoding decide what the body is: strip them and a quoted-printable, base64 or HTML message would be classified from its encoding rather than its prose — which is one of the misclassifications this tool exists to find.

70.1.22.1.5. Output

On success, one line on standard output: the Accept-Language-style string the stage would merge into state.language, for example:

en;q=1, de;q=0.45

Nothing else is ever written to standard output, so the value can be captured directly:

lang=$(pepsi-detect-language message.eml)

When the stage would record no language — the message has no inline text, none of that text is prose (a body of nothing but URLs), there is too little prose to classify, or no candidate language reaches the 5 % confidence threshold — standard output stays empty and the reason is reported on standard error. That is not an error: it is what the stage does, advancing the message with no language key added.

The format is described in pepsi-stage-detect-language(1) and pepsi.state(7).

70.1.22.1.6. Options

–languages CODES

Detect among these candidate languages instead of the configured ones, written as the stage’s LANGUAGES option is (whitespace- or comma-separated ISO 639-1 codes, at least two, e.g. "en de fr"; * selects every supported language, as it does in the option). The configuration file is then not read at all, so this is also how to run the tool on a host with no Pepsi configuration. This is the tuning knob: the candidate set is the option that most often explains a wrong answer — compare --languages '*' (what the pepsi-setup wizard configures by default) against a set narrowed to the languages your traffic actually contains.

–stage NAME

Take the candidate languages from [stage-NAME]. Needed only when a configuration runs the detection stage more than once (for example on the inbound and submission paths with different language sets); with a single such stage it is found automatically.

–body-only

Treat the input as a bare message body even if it opens with something that looks like a header block.

–explain

Write the intermediate steps to standard error: how the input was framed, the candidate set and where it came from, the extracted body text, the prose left after filtering it (which is what the detector was actually given, and whether it came from the HTML alternative), the confidence of every candidate language and which of them the top-three and 5 % rules kept, and the final outcome. Standard output is unaffected, so –explain can be added to any invocation without disturbing a script reading the result.

The two text blocks are the ones to read together. The stage does not classify the body as extracted: it first removes everything that cannot be a word — URLs, e-mail addresses, base64 blobs, DNS records, identifiers — because a single such token makes lingua return one language at probability 1 without looking at the rest of the message. If a classification looks wrong, the question is usually what is in the second block, not the first.

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations. Only the candidate-language set is taken from it.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info). Logs go to standard error.

-h, –help; -V, –version

Print a usage summary / the version and exit.

70.1.22.1.7. Configuration

Without –languages, the candidate set is the LANGUAGES option of the configured detection stage — the single [stage-<name>] section whose PROGRAM is pepsi-stage-detect-language, or the one named by –stage. If the configuration runs no such stage, the stage’s built-in default set is used and a warning says so. No other option is read, and nothing is written.

70.1.22.1.8. Exit Status

0

The message was classified. Standard output holds the state.language string, or is empty if the stage would record none.

1

The message could not be read, the configuration could not be loaded (pass –languages to run without one), –stage named a section that does not exist, several stages run the program and none was named, or –languages was invalid. The reason is written to standard error.

70.1.22.1.9. Examples

What did the pipeline make of this message?

pepsi-detect-language -c /etc/pepsi/pepsi.conf message.eml

Why did it say that?

pepsi-detect-language --explain message.eml >/dev/null

Would a narrower candidate set fix it?

pepsi-detect-language --languages "en de" message.eml

Classify a body typed on the terminal (end with Ctrl-D):

pepsi-detect-language --languages "en de fr" -

Survey a maildir before changing LANGUAGES:

for m in ~/Maildir/cur/*; do
    printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m"
done | sort | uniq -c

70.1.22.1.10. See Also

pepsi-stage-detect-language(1), pepsi-stage-block-language(1), pepsi-stage-vacation(1), pepsi.conf(5), pepsi.state(7)

70.1.22.1.11. Bugs

Report bugs to the Pepsi issue tracker.