54. pepsi-stage-detect-language¶
Detect the human language(s) of a message body and record them in the state.
54.1. Role¶
pepsi-stage-detect-language identifies the natural language(s) used in a
message body and records them under state.language for later stages or
templates to use, then advances the message to NEXT_STAGE. It only annotates
the state — it never modifies the message or drops, bounces or pauses it.
Reference: pepsi-stage-detect-language(1).
54.2. Features¶
Shared MIME text extraction: a full MIME parser in
pepsi-commonextracts from the message the inline body text that the reader is shown:text/plainparts verbatim andtext/htmlparts reduced to plain text — each decoded from its content-transfer-encoding and character set to UTF-8. For amultipart/alternativethe HTML alternative is read, since most clients display it; the plain one only when the HTML holds no prose. HTML is rendered without<head>,<style>,<script>,<template>, comments and the elements that an inlinestyleor thehiddenattribute hides (style sheet classes are not evaluated); invisible and format characters are removed; and look-alike letters of another script inside a Latin word are folded with the UTS #39 confusables data, so a sender cannot show the reader one language and the classifier another (Content concealed from spam filters (HTML/CSS, invisible chars, bad QR)).Prose filtering: the extracted text is reduced to what a human wrote in words before anything is classified. Tokens that cannot be words (URLs, addresses, base64 keys, identifiers, anything with a digit or an internal full stop in it) are dropped, and then lines with too few letters to be a sentence. This is not mere tidiness: otherwise a body that is mostly machine text is classified by its machine text, and a DNS-record dump or a body of two bare URLs gets a confident verdict for a language that nobody wrote in it. The rules skip the length, digit, full-stop and capitalisation tests for scripts that do not separate words with spaces (Han, Kana, Hangul, Thai, …), which would otherwise delete a Chinese or Thai message in its entirety.
Skips what it cannot read: attachments (including text attachments) and encrypted or opaque bodies (
multipart/encrypted,application/pkcs7-mime, …) carry no readable text. Such a message advances unchanged, with nolanguagekey added, and so does a message whose body holds no prose or too little prose to classify. A body that is only URLs has no language, and recording none is the honest answer.Statistical detection: the surviving prose is classified with the lingua detector, built from the candidate languages of the
LANGUAGESoption. The detector receives only the first 20 000 characters of prose, since the first few kilobytes decide the language. The stage keeps the most likely languages whose probability is at least 5 %, at most three of them.Accept-Language output: the result is stored as an HTTP
Accept-Language-style string, e.g.en;q=0.55, de;q=0.45, where eachqis that language’s detection probability (the probabilities over all candidates sum to 1).Built once per worker, at full accuracy: the detector is expensive to construct, so each worker process builds it a single time (per candidate-language set) and reuses it for every message.
lingua’s low-accuracy mode is deliberately not used: it short-circuits on a single n-gram that only one candidate owns, so one address or key can decide the verdict for a whole body, and it preloads extra n-gram count models, so it brings no advantage.
54.3. Configuration¶
[stage-<name>]: PROGRAM = pepsi-stage-detect-language, NEXT_STAGE
(required) and LANGUAGES (optional). LANGUAGES lists at least two
distinct ISO 639-1 codes, separated by whitespace or commas, and defaults to a
broad common set; narrow it to the languages you expect, since each enabled
language costs worker memory.
The single token * stands for every language the detector supports. See
pepsi-stage-detect-language(1).
54.4. Placement and limits¶
The stage reads plaintext, so on a branch that decrypts it belongs after
pepsi-stage-decrypt and before pepsi-stage-reencrypt;
pepsi-setup warns when re-encryption comes before it, since it would then see
only ciphertext (Client custody (custody = client, the MUA key)). The 20 000-character bound on what is
classified is one of the parsing limits in Availability, and a message
whose content makes the classifier panic advances without a language rather
than being retried.
54.5. State¶
Inputs: none from
state; the stage reads the stored message body.Outputs: when a language is detected, the
Accept-Language-style string is merged intostateunderlanguage; otherwisestateis untouched.
54.6. See also¶
pepsi-detect-language (the offline diagnostic and tuning tool: the same executable under a second name, which reports what this stage would record for a message read from a file), pepsi-stage-check-whitelist, pepsi-stage-anti-spam, Supported Features, pepsi-stage-detect-language(1).