.. This file is part of PEPSI. Copyright (C) 2026 GNUnet e.V. PEPSI is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. ===================== pepsi-detect-language ===================== *Classify one message the way the pipeline would — offline, for diagnosis and tuning.* Role ==== ``pepsi-detect-language`` is a **diagnostic and tuning tool**, not a stage. It reads a single message from a file or standard input, runs :doc:`pepsi-stage-detect-language`'s own classification over it, and prints the resulting ``state.language`` value — and nothing else — to standard output. It never opens the database and needs no ``pepsi.workqueue`` row, so a message that was classified wrongly in production can be investigated on a saved copy, on any host, as often as you like. Reference: :manpage:`pepsi-detect-language(1)`. It is the **same executable** as ``pepsi-stage-detect-language``, installed as a second name for it: the two share the entire classification path — so what the tool reports is what the pipeline did, not a re-implementation that can drift — and the statically linked language models, which is why they also ship in the same (separate) package. Diagnosing a misclassification ============================== ``--explain`` writes every intermediate step to standard error, leaving standard output untouched:: pepsi-detect-language --explain message.eml >/dev/null It shows how the input was framed, which candidate languages were in play and where that set came from, the body text that was extracted, **the prose left after filtering it** — the single most useful block, since that, and not the extracted text, is what the detector was given — and the confidence of every candidate with the top-three and 5 % rules applied. Why there are two text blocks ============================= The stage does not classify the body as extracted. It first removes everything that cannot be a word: URLs, e-mail addresses, base64 keys, DNS records, identifiers. This is not tidiness: left in, those tokens decide the answer, and a notification e-mail, a DNS-record dump or a spam body of two bare URLs is classified from its machine text as a language nobody wrote in it. The stage runs ``lingua`` in full-accuracy mode. Its low-accuracy mode looks for an n-gram *unique* to one candidate language before it scores anything and returns that language at probability ``1`` if exactly one matches, so a single non-word — an address in a quoted attribution line, a base64 key, an IPv6 address — would decide the entire message with total confidence, without the rest of the body being examined. So when a classification looks wrong, read the *second* block. If the prose there is the message a human would read and the answer is still wrong, the candidate set is the next thing to try (below). If the second block says there is no prose, the body genuinely held none — a bulk mailer whose plain part is two bare URLs — and no language is the right answer. Tuning ``LANGUAGES`` ==================== The usual cause of a confident wrong answer is the candidate set: a language the traffic never contains still competes for probability, and short texts are where it wins. ``--languages`` overrides the configured set (and skips reading the configuration entirely), so a set can be tried before it is configured:: pepsi-detect-language --languages "en de" message.eml ``*`` selects every supported language, exactly as in the option — which is what the :doc:`pepsi-setup` wizard configures unless the operator narrows it, and so the set to compare against when a production classification looks wrong:: pepsi-detect-language --languages '*' message.eml Without ``--languages`` the candidate set comes from the configuration: the single ``[stage-*]`` section running ``pepsi-stage-detect-language``. When a configuration has more than one, name the one to use with ``--stage NAME``; when it has none, the stage's built-in default set is used and a warning says so. Over a corpus:: for m in ~/Maildir/cur/*; do printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m" done | sort | uniq -c Input ===== * **mbox, RFC 5322 or a bare body** — all three are accepted, and which one it is being given is detected. A leading mbox ``From_`` separator line is stripped (with mboxrd ``>From`` quoting undone); a bare body is wrapped in a minimal ``Content-Type``. A header block is recognised only when every line before the first blank line is a field or a folded continuation, so a body opening with ``Re: the meeting`` is not mistaken for one; ``--body-only`` forces that reading. * **Headers are kept, and never classified.** Header text does not reach the detector — the stage extracts inline body parts only — but ``Content-Type``, ``charset`` and ``Content-Transfer-Encoding`` decide what the body *is*. Discarding them would classify a quoted-printable, base64 or HTML message from its encoding rather than its prose, which is one of the failures the tool exists to find. * **The plain alternative wins, unless it is a stub.** For a ``multipart/alternative`` the ``text/plain`` part is what a sender wrote, so it is preferred. When it holds no prose at all — the bulk-mailer shape, two bare URLs where the message should be — the HTML sibling is used instead, and ``--explain`` says so. * Line endings are normalised to CRLF as :doc:`pepsi-ingress` does, so a message read from a file classifies exactly as the queued copy did. Output ====== One line: the ``Accept-Language``-style string, e.g. ``en;q=1, de;q=0.45``. When the stage would record no language — no inline text, too little of it, or nothing above the confidence threshold — standard output is **empty** and the reason goes to standard error. That is the stage's own behaviour, not an error. See also ======== :doc:`pepsi-stage-detect-language`, :doc:`pepsi-stage-block-language`, :manpage:`pepsi-detect-language(1)`, :manpage:`pepsi.state(7)`.