.. This file is part of PEPSI. Copyright (C) 2026 Pepsi contributors PEPSI is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. ===================== pepsi-detect-language ===================== *Classify one message the way the pipeline would — offline, for diagnosis and tuning.* Role ==== ``pepsi-detect-language`` is a **diagnostic and tuning tool**, not a stage. It reads a single message from a file or standard input, runs :doc:`pepsi-stage-detect-language`'s own classification over it, and prints the resulting ``state.language`` value — and nothing else — to standard output. It never opens the database and needs no ``pepsi.ingress`` row, so a message that was classified wrongly in production can be investigated on a saved copy, on any host, as often as you like. Reference: :manpage:`pepsi-detect-language(1)`. It is the **same executable** as ``pepsi-stage-detect-language``, installed as a second name for it: the two share the entire classification path — so what the tool reports is what the pipeline did, not a re-implementation that can drift — and the statically linked language models, which is why they also ship in the same (separate) package. Diagnosing a misclassification ============================== ``--explain`` writes every intermediate step to standard error, leaving standard output untouched:: pepsi-detect-language --explain message.eml >/dev/null It shows how the input was framed, which candidate languages were in play and where that set came from, the body text that was extracted, **the prose left after filtering it** — the single most useful block, since that, and not the extracted text, is what the detector was given — and the confidence of every candidate with the ``TOP_N`` and 5 % rules applied. Why there are two text blocks ============================= The stage does not classify the body as extracted. It first removes everything that cannot be a word: URLs, e-mail addresses, base64 keys, DNS records, identifiers. This is not tidiness — measured on real mail, those tokens decide the answer. A Mastodon follow notification and a DNS-record dump both classified as Yoruba, and a spam body of two bare URLs as Tsonga, purely on the strength of the machine text in them; with that text removed all three are correctly English. It used to be worse still. The stage ran ``lingua``'s low-accuracy mode, which looks for an n-gram *unique* to one candidate language before it scores anything and returns that language at probability ``1`` if exactly one matches — so a single non-word decided the entire message, with total confidence and without the rest of the body being examined. ```` in a quoted attribution line turned 3.3 kB of English into Estonian; a base64 DKIM key turned a DNS notification into Yoruba, an IPv6 address in the same message into Polish. That mode is no longer used (it also measured slower, and needed more memory, than full accuracy once the filtering was in place). So when a classification looks wrong, read the *second* block. If the prose there is the message a human would read and the answer is still wrong, the candidate set is the next thing to try (below). If the second block says there is no prose, the body genuinely held none — a bulk mailer whose plain part is two bare URLs — and no language is the right answer. Tuning ``LANGUAGES`` ==================== The usual cause of a confident wrong answer is the candidate set: a language the traffic never contains still competes for probability, and short texts are where it wins. ``--languages`` overrides the configured set (and skips reading the configuration entirely), so a set can be tried before it is configured:: pepsi-detect-language --languages "en de" message.eml ``*`` selects every supported language, exactly as in the option — which is what the :doc:`pepsi-setup` wizard configures unless the operator narrows it, and so the set to compare against when a production classification looks wrong:: pepsi-detect-language --languages '*' message.eml Over a corpus:: for m in ~/Maildir/cur/*; do printf '%s\t%s\n' "$(pepsi-detect-language --languages "en de" "$m")" "$m" done | sort | uniq -c Input ===== * **mbox, RFC 5322 or a bare body** — all three are accepted, and which one it is being given is detected. A leading mbox ``From_`` separator line is stripped (with mboxrd ``>From`` quoting undone); a bare body is wrapped in a minimal ``Content-Type``. A header block is recognised only when every line before the first blank line is a field or a folded continuation, so a body opening with ``Re: the meeting`` is not mistaken for one; ``--body-only`` forces that reading. * **Headers are kept, and never classified.** Header text does not reach the detector — the stage extracts inline body parts only — but ``Content-Type``, ``charset`` and ``Content-Transfer-Encoding`` decide what the body *is*. Discarding them would classify a quoted-printable, base64 or HTML message from its encoding rather than its prose, which is one of the failures the tool exists to find. * **The plain alternative wins, unless it is a stub.** For a ``multipart/alternative`` the ``text/plain`` part is what a sender wrote, so it is preferred. When it holds no prose at all — the bulk-mailer shape, two bare URLs where the message should be — the HTML sibling is used instead, and ``--explain`` says so. * Line endings are normalised to CRLF as :doc:`pepsi-ingress` does, so a message read from a file classifies exactly as the queued copy did. Output ====== One line: the ``Accept-Language``-style string, e.g. ``en;q=1, de;q=0.45``. When the stage would record no language — no inline text, too little of it, or nothing above the confidence threshold — standard output is **empty** and the reason goes to standard error. That is the stage's own behaviour, not an error. See also ======== :doc:`pepsi-stage-detect-language`, :doc:`pepsi-stage-block-language`, :manpage:`pepsi-detect-language(1)`, :manpage:`pepsi.state(7)`.