..
  This file is part of PEPSI.
  Copyright (C) 2026 GNUnet e.V.

  PEPSI is free software; you can redistribute it and/or modify it under the
  terms of the GNU Affero General Public License as published by the Free
  Software Foundation; either version 3, or (at your option) any later version.

===========================
pepsi-stage-detect-language
===========================

*Detect the human language(s) of a message body and record them in the state.*

Role
====

``pepsi-stage-detect-language`` identifies the natural language(s) used in a
message body and records them under ``state.language`` for later stages or
templates to use, then advances the message to ``NEXT_STAGE``. It only annotates
the state — it never modifies the message or drops, bounces or pauses it.
Reference: :manpage:`pepsi-stage-detect-language(1)`.

Features
========

* **Shared MIME text extraction:** a full MIME parser in ``pepsi-common`` pulls
  the inline body text out of the message — ``text/plain`` parts verbatim and
  ``text/html`` parts reduced to plain text — each decoded from its
  content-transfer-encoding and character set to UTF-8. For a
  ``multipart/alternative`` the plain alternative is preferred over the HTML one —
  unless it holds no prose, the shape a bulk mailer produces when it puts two bare
  URLs where the message should be, in which case the HTML sibling is used.
* **Prose filtering:** the extracted text is reduced to what a human wrote in
  words before anything is classified. Tokens that cannot be words (URLs,
  addresses, base64 keys, identifiers, anything with a digit or an internal full
  stop in it) are dropped, and then lines with too few letters to be a sentence.
  This is not tidiness: a body that is mostly machine text otherwise classifies
  on the machine text, and a DNS-record dump or a body of two bare URLs gets a
  confident verdict for a language nobody wrote in it. The rules skip the length, digit and capitalisation tests for scripts that do not
  separate words with spaces (Han, Kana, Hangul, Thai, …), which would otherwise
  delete a Chinese or Thai message in its entirety.
* **Skips what it cannot read:** attachments (including text attachments) and
  encrypted or opaque bodies (``multipart/encrypted``, ``application/pkcs7-mime``,
  …) carry no readable text, so such a message — or one whose body holds no prose,
  or too little of it to classify — advances unchanged, with no ``language`` key
  added. A body that is only URLs has no language, and recording none is the
  honest answer.
* **Statistical detection:** the surviving prose is classified with the `lingua
  <https://github.com/pemistahl/lingua-rs>`_ detector, built from the candidate
  languages of the ``LANGUAGES`` option. Only the first 20 000 characters of
  prose are handed to it — a language is decided by the first few kilobytes —
  and the most likely **at most three** languages whose probability is at least
  5 % are kept.
* **Accept-Language output:** the result is stored as an HTTP
  ``Accept-Language``-style string, e.g. ``en;q=1, de;q=0.45``, where each ``q``
  is that language's detection probability.
* **Built once per worker, at full accuracy:** the detector is expensive to
  construct, so each worker process builds it a single time (per
  candidate-language set) and reuses it for every message. ``lingua``'s
  low-accuracy mode is deliberately not used: it short-circuits on a single
  n-gram only one candidate owns, so one address or key can decide the verdict
  for a whole body, and it preloads extra n-gram count models, so there is
  nothing it buys.

Configuration
=============

``[stage-<name>]``: ``PROGRAM = pepsi-stage-detect-language``, ``NEXT_STAGE``
*(required)* and ``LANGUAGES`` (optional; whitespace/comma-separated ISO 639-1
codes, at least two distinct ones, defaulting to a broad common set — narrow it
to the languages you expect, since each enabled language costs worker memory).
The single token ``*`` stands for every language the detector supports. See
:manpage:`pepsi-stage-detect-language(1)`.

State
=====

* **Inputs:** none from ``state``; the stage reads the stored message body.
* **Outputs:** when a language is detected, the ``Accept-Language``-style string
  is merged into ``state`` under ``language``; otherwise ``state`` is untouched.

See also
========

:doc:`pepsi-detect-language` (the offline diagnostic and tuning tool: the same
executable under a second name, which reports what this stage would record for a
message read from a file), :doc:`pepsi-stage-check-whitelist`,
:doc:`pepsi-stage-anti-spam`, :doc:`../features`,
:manpage:`pepsi-stage-detect-language(1)`.