95.1.30. pepsi-detect-language¶
classify one message the way the pipeline would, offline
- Manual section:
1
95.1.30.1.1. Name¶
pepsi-detect-language - diagnostic and tuning tool for Pepsi’s language detection.
95.1.30.1.2. Synopsis¶
pepsi-detect-language [OPTIONS] [FILE]
95.1.30.1.3. Description¶
pepsi-detect-language answers one question: what would
pepsi-stage-detect-language(1) record for this message? It reads a single
message, runs the stage’s own classification over it, and writes the resulting
state.language value to standard output. Nothing else goes there.
It is a diagnostic and tuning tool, not part of the pipeline. It never opens
the database, never touches the queue and needs no pepsi.workqueue row, so it
is safe to run on a saved copy of a message that was classified wrongly in
production, on a host that is not the mail server, and repeatedly with different
–languages sets while narrowing down the option to configure.
Use it to:
reproduce a misclassification from a saved message, and see with –explain why the detector chose what it chose: the extracted text, the prose that survived filtering (which is what the detector actually saw), and the confidence of every candidate language;
tune the stage’s LANGUAGES option: a candidate language that the traffic never contains still competes for probability, and dropping it is often the whole fix. Re-run the same message against different sets with –languages until the answer is right, then configure that set;
check a corpus in a loop (
for m in Maildir/cur/*; do …; done) before and after a configuration change.
It is the same executable as pepsi-stage-detect-language(1), installed
under a second name. The two share the whole classification path and the
statically linked language models, which is why they are also packaged together.
The name under which the executable is invoked decides which program runs. On a
build without that symlink, you reach the tool as
pepsi-stage-detect-language pepsi-detect-language [OPTIONS] [FILE], which
re-executes itself under the second name.
95.1.30.1.4. Input¶
The message is read from FILE, or from standard input when no file is given (or
when FILE is -), one message per invocation. The tool accepts three input
shapes and detects which one it has been given:
- mbox
A leading
From_separator line is stripped; by the mbox convention, this line isFromfollowed by a space and the envelope sender. The mboxrd>Fromquoting in the body is undone, but only when a separator line was actually found: elsewhere, a line beginning>Fromis somebody’s quoted text. Note that a file holding several mbox messages is treated as one message; split it first.- a complete RFC 5322 message
Used as-is, exactly as the stage would see it after pepsi-ingress(1) stored it.
- a bare body, with no header block
Wrapped in a minimal
Content-Type: text/plainheader, so that the MIME parser reads it as text. The header declaresutf-8oriso-8859-1, according to what the bytes are.
A header block is recognised only when every line before the first blank line
is a header field or a folded continuation of one. This rule keeps a body that
opens with something like Re: the meeting from being mistaken for a
one-field header block; –body-only forces the bare-body reading when the
input really is such a body.
Line endings are normalised to CRLF, as pepsi-ingress(1) does, so a message
saved out of a file classifies identically to the copy the queue held.
The header block is kept, and that is deliberate. No header text ever
reaches the classifier: the stage extracts only inline body parts, so
Subject:, From: and the Received: trace are discarded from what is
classified. But Content-Type, charset and Content-Transfer-Encoding
decide what the body is. Without them, a quoted-printable, base64 or HTML
message would be classified from its encoding rather than from its prose, and
that is one of the misclassifications that this tool exists to find.
95.1.30.1.5. Output¶
On success, one line on standard output: the Accept-Language-style string
that the stage would merge into state.language, for example:
en;q=0.55, de;q=0.45
Nothing else is ever written to standard output, so the value can be captured directly:
lang=$(pepsi-detect-language message.eml)
Sometimes the stage records no language: the message has no inline text,
none of that text is prose (a body of nothing but URLs), there is too little
prose to classify, or no candidate language reaches the 5 % confidence
threshold. Standard output then stays empty, and the reason is reported on
standard error. That is not an error: the stage does the same, and advances the
message without adding a language key.
The format is described in pepsi-stage-detect-language(1) and pepsi.state(7).
95.1.30.1.6. Options¶
- –languages CODES
Detect among these candidate languages instead of the configured ones, written in the same way as the stage’s LANGUAGES option (whitespace- or comma-separated ISO 639-1 codes, at least two, e.g.
"en de fr";*selects every supported language, as it does in the option). The configuration file is then not read at all, so this is also how to run the tool on a host with no Pepsi configuration. This is the setting to tune: the candidate set is the option that most often explains a wrong answer. Compare--languages '*'(what the pepsi-setup wizard configures by default) with a set narrowed to the languages that your traffic actually contains.- –stage NAME
Take the candidate languages from
[stage-NAME]. Needed only when a configuration runs the detection stage more than once (for example on the inbound and submission paths with different language sets); with a single such stage it is found automatically.- –body-only
Treat the input as a bare message body even if it opens with something that looks like a header block.
- –explain
Write the intermediate steps to standard error: how the input was framed, the candidate set and where it came from, the extracted body text, the prose left after filtering it (which is what the detector was actually given, and whether it came from the plain alternative because the displayed HTML one held no prose), the confidence of every candidate language and which of them the top-three and 5 % rules kept, and the final outcome. Standard output is unaffected, so –explain can be added to any invocation without disturbing a script that reads the result.
Read the two text blocks together. The stage does not classify the body as extracted: it first removes everything that cannot be a word (URLs, e-mail addresses, base64 blobs, DNS records, identifiers), because such tokens are not words in any language, and a body dominated by them is classified from them rather than from its prose. If a classification looks wrong, the question is usually what is in the second block, not the first.
- -c FILE, –config FILE
Read the configuration from FILE instead of searching the default locations. Only the candidate-language set is taken from it.
- -L LOGLEVEL, –log LOGLEVEL
Set the logging verbosity (default
[pepsi] LOG, elseinfo). Logs go to standard error.- -v, –verbose
Show log messages from all sources.
- -h, –help; -V, –version
Print a usage summary / the version and exit.
95.1.30.1.7. Configuration¶
Without –languages, the candidate set is the LANGUAGES option of the
configured detection stage: the single [stage-<name>] section whose
PROGRAM is pepsi-stage-detect-language, or the one named by –stage.
If the configuration runs no such stage, the stage’s built-in default set is used
and a warning says so. No other option is read, and nothing is written.
95.1.30.1.8. Exit Status¶
- 0
The message was classified. Standard output holds the
state.languagestring, or is empty if the stage would record none.- 1
The message could not be read, the configuration could not be loaded (pass –languages to run without one), –stage named a section that does not exist or that runs another program, several stages run the program and none was named, or –languages was invalid. The reason is written to standard error.
95.1.30.1.9. Examples¶
What did the pipeline make of this message?
pepsi-detect-language -c /etc/pepsi/pepsi.conf message.eml
Why did it say that?
pepsi-detect-language --explain message.eml >/dev/null
Would a narrower candidate set fix it?
pepsi-detect-language --languages "en de" message.eml
Classify a body typed on the terminal (end with Ctrl-D):
pepsi-detect-language --languages "en de fr" -
Survey a maildir before changing LANGUAGES:
for m in ~/Maildir/cur/*; do
printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m"
done | sort | uniq -c
95.1.30.1.10. See Also¶
pepsi-stage-detect-language(1), pepsi-stage-block-language(1), pepsi-stage-vacation(1), pepsi.conf(5), pepsi.state(7)
95.1.30.1.11. Bugs¶
Report bugs to the Pepsi issue tracker.