54. pepsi-stage-detect-language

Detect the human language(s) of a message body and record them in the state.

54.1. Role

pepsi-stage-detect-language identifies the natural language(s) used in a message body and records them under state.language for later stages or templates to use, then advances the message to NEXT_STAGE. It only annotates the state — it never modifies the message or drops, bounces or pauses it. Reference: pepsi-stage-detect-language(1).

54.2. Features

  • Shared MIME text extraction: a full MIME parser in pepsi-common extracts from the message the inline body text that the reader is shown: text/plain parts verbatim and text/html parts reduced to plain text — each decoded from its content-transfer-encoding and character set to UTF-8. For a multipart/alternative the HTML alternative is read, since most clients display it; the plain one only when the HTML holds no prose. HTML is rendered without <head>, <style>, <script>, <template>, comments and the elements that an inline style or the hidden attribute hides (style sheet classes are not evaluated); invisible and format characters are removed; and look-alike letters of another script inside a Latin word are folded with the UTS #39 confusables data, so a sender cannot show the reader one language and the classifier another (Content concealed from spam filters (HTML/CSS, invisible chars, bad QR)).

  • Prose filtering: the extracted text is reduced to what a human wrote in words before anything is classified. Tokens that cannot be words (URLs, addresses, base64 keys, identifiers, anything with a digit or an internal full stop in it) are dropped, and then lines with too few letters to be a sentence. This is not mere tidiness: otherwise a body that is mostly machine text is classified by its machine text, and a DNS-record dump or a body of two bare URLs gets a confident verdict for a language that nobody wrote in it. The rules skip the length, digit, full-stop and capitalisation tests for scripts that do not separate words with spaces (Han, Kana, Hangul, Thai, …), which would otherwise delete a Chinese or Thai message in its entirety.

  • Skips what it cannot read: attachments (including text attachments) and encrypted or opaque bodies (multipart/encrypted, application/pkcs7-mime, …) carry no readable text. Such a message advances unchanged, with no language key added, and so does a message whose body holds no prose or too little prose to classify. A body that is only URLs has no language, and recording none is the honest answer.

  • Statistical detection: the surviving prose is classified with the lingua detector, built from the candidate languages of the LANGUAGES option. The detector receives only the first 20 000 characters of prose, since the first few kilobytes decide the language. The stage keeps the most likely languages whose probability is at least 5 %, at most three of them.

  • Accept-Language output: the result is stored as an HTTP Accept-Language-style string, e.g. en;q=0.55, de;q=0.45, where each q is that language’s detection probability (the probabilities over all candidates sum to 1).

  • Built once per worker, at full accuracy: the detector is expensive to construct, so each worker process builds it a single time (per candidate-language set) and reuses it for every message. lingua’s low-accuracy mode is deliberately not used: it short-circuits on a single n-gram that only one candidate owns, so one address or key can decide the verdict for a whole body, and it preloads extra n-gram count models, so it brings no advantage.

54.3. Configuration

[stage-<name>]: PROGRAM = pepsi-stage-detect-language, NEXT_STAGE (required) and LANGUAGES (optional). LANGUAGES lists at least two distinct ISO 639-1 codes, separated by whitespace or commas, and defaults to a broad common set; narrow it to the languages you expect, since each enabled language costs worker memory. The single token * stands for every language the detector supports. See pepsi-stage-detect-language(1).

54.4. Placement and limits

The stage reads plaintext, so on a branch that decrypts it belongs after pepsi-stage-decrypt and before pepsi-stage-reencrypt; pepsi-setup warns when re-encryption comes before it, since it would then see only ciphertext (Client custody (custody = client, the MUA key)). The 20 000-character bound on what is classified is one of the parsing limits in Availability, and a message whose content makes the classifier panic advances without a language rather than being retried.

54.5. State

  • Inputs: none from state; the stage reads the stored message body.

  • Outputs: when a language is detected, the Accept-Language-style string is merged into state under language; otherwise state is untouched.

54.6. See also

pepsi-detect-language (the offline diagnostic and tuning tool: the same executable under a second name, which reports what this stage would record for a message read from a file), pepsi-stage-check-whitelist, pepsi-stage-anti-spam, Supported Features, pepsi-stage-detect-language(1).