.. This file is part of PEPSI. Copyright (C) 2026 Pepsi contributors PEPSI is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. =========================== pepsi-stage-detect-language =========================== *Detect the human language(s) of a message body and record them in the state.* Role ==== ``pepsi-stage-detect-language`` identifies the natural language(s) used in a message body and records them under ``state.language`` for later stages or templates to use, then advances the message to ``NEXT_STAGE``. It only annotates the state — it never modifies the message or drops, bounces or pauses it. Reference: :manpage:`pepsi-stage-detect-language(1)`. Features ======== * **Shared MIME text extraction:** a full MIME parser in ``pepsi-common`` pulls the inline body text out of the message — ``text/plain`` parts verbatim and ``text/html`` parts reduced to plain text — each decoded from its content-transfer-encoding and character set to UTF-8. For a ``multipart/alternative`` the plain alternative is preferred over the HTML one — unless it holds no prose, the shape a bulk mailer produces when it puts two bare URLs where the message should be, in which case the HTML sibling is used. * **Prose filtering:** the extracted text is reduced to what a human wrote in words before anything is classified. Tokens that cannot be words (URLs, addresses, base64 keys, identifiers, anything with a digit or an internal full stop in it) are dropped, and then lines with too few letters to be a sentence. This is not tidiness: measured over real mail, a body that is mostly machine text classifies on the machine text — a DNS-record dump and a Mastodon notification both came out Yoruba, a body of two bare URLs came out Tsonga. See :doc:`pepsi-detect-language` for the measurements. The rules skip the length, digit and capitalisation tests for scripts that do not separate words with spaces (Han, Kana, Hangul, Thai, …), which would otherwise delete a Chinese or Thai message in its entirety. * **Skips what it cannot read:** attachments (including text attachments) and encrypted or opaque bodies (``multipart/encrypted``, ``application/pkcs7-mime``, …) carry no readable text, so such a message — or one whose body holds no prose, or too little of it to classify — advances unchanged, with no ``language`` key added. A body that is only URLs has no language, and recording none is the honest answer. * **Statistical detection:** the surviving prose is classified with the `lingua `_ detector, built from the candidate languages of the ``LANGUAGES`` option. The most likely **at most three** languages whose probability is at least 5 % are kept. * **Accept-Language output:** the result is stored as an HTTP ``Accept-Language``-style string, e.g. ``en;q=1, de;q=0.45``, where each ``q`` is that language's detection probability. * **Built once per worker, at full accuracy:** the detector is expensive to construct, so each worker process builds it a single time (per candidate-language set) and reuses it for every message. ``lingua``'s low-accuracy mode is deliberately not used: with the prose filter in place it measured less accurate, slower on long bodies, *and* larger in resident memory (it preloads extra n-gram count models), so there is nothing it buys. Configuration ============= ``[stage-]``: ``PROGRAM = pepsi-stage-detect-language``, ``NEXT_STAGE`` and ``LANGUAGES`` (optional; whitespace/comma-separated ISO 639-1 codes, at least two, defaulting to a broad common set — narrow it to the languages you expect, since each enabled language costs worker memory). See :manpage:`pepsi-stage-detect-language(1)`. State ===== * **Inputs:** none from ``state``; the stage reads the stored message body. * **Outputs:** when a language is detected, the ``Accept-Language``-style string is merged into ``state`` under ``language``; otherwise ``state`` is untouched. See also ======== :doc:`pepsi-detect-language` (the offline diagnostic and tuning tool: the same executable under a second name, which reports what this stage would record for a message read from a file), :doc:`pepsi-stage-check-whitelist`, :doc:`pepsi-stage-anti-spam`, :doc:`../features`, :manpage:`pepsi-stage-detect-language(1)`.