43. pepsi-stage-detect-language

Detect the human language(s) of a message body and record them in the state.

43.1. Role

pepsi-stage-detect-language identifies the natural language(s) used in a message body and records them under state.language for later stages or templates to use, then advances the message to NEXT_STAGE. It only annotates the state — it never modifies the message or drops, bounces or pauses it. Reference: pepsi-stage-detect-language(1).

43.2. Features

  • Shared MIME text extraction: a full MIME parser in pepsi-common pulls the inline body text out of the message — text/plain parts verbatim and text/html parts reduced to plain text — each decoded from its content-transfer-encoding and character set to UTF-8. For a multipart/alternative the plain alternative is preferred over the HTML one — unless it holds no prose, the shape a bulk mailer produces when it puts two bare URLs where the message should be, in which case the HTML sibling is used.

  • Prose filtering: the extracted text is reduced to what a human wrote in words before anything is classified. Tokens that cannot be words (URLs, addresses, base64 keys, identifiers, anything with a digit or an internal full stop in it) are dropped, and then lines with too few letters to be a sentence. This is not tidiness: measured over real mail, a body that is mostly machine text classifies on the machine text — a DNS-record dump and a Mastodon notification both came out Yoruba, a body of two bare URLs came out Tsonga. See pepsi-detect-language for the measurements. The rules skip the length, digit and capitalisation tests for scripts that do not separate words with spaces (Han, Kana, Hangul, Thai, …), which would otherwise delete a Chinese or Thai message in its entirety.

  • Skips what it cannot read: attachments (including text attachments) and encrypted or opaque bodies (multipart/encrypted, application/pkcs7-mime, …) carry no readable text, so such a message — or one whose body holds no prose, or too little of it to classify — advances unchanged, with no language key added. A body that is only URLs has no language, and recording none is the honest answer.

  • Statistical detection: the surviving prose is classified with the lingua detector, built from the candidate languages of the LANGUAGES option. The most likely at most three languages whose probability is at least 5 % are kept.

  • Accept-Language output: the result is stored as an HTTP Accept-Language-style string, e.g. en;q=1, de;q=0.45, where each q is that language’s detection probability.

  • Built once per worker, at full accuracy: the detector is expensive to construct, so each worker process builds it a single time (per candidate-language set) and reuses it for every message. lingua’s low-accuracy mode is deliberately not used: with the prose filter in place it measured less accurate, slower on long bodies, and larger in resident memory (it preloads extra n-gram count models), so there is nothing it buys.

43.3. Configuration

[stage-<name>]: PROGRAM = pepsi-stage-detect-language, NEXT_STAGE and LANGUAGES (optional; whitespace/comma-separated ISO 639-1 codes, at least two, defaulting to a broad common set — narrow it to the languages you expect, since each enabled language costs worker memory). See pepsi-stage-detect-language(1).

43.4. State

  • Inputs: none from state; the stage reads the stored message body.

  • Outputs: when a language is detected, the Accept-Language-style string is merged into state under language; otherwise state is untouched.

43.5. See also

pepsi-detect-language (the offline diagnostic and tuning tool: the same executable under a second name, which reports what this stage would record for a message read from a file), pepsi-stage-check-whitelist, pepsi-stage-anti-spam, Supported Features, pepsi-stage-detect-language(1).