85.1.30. pepsi-stage-block-language

route a message by its detected body language(s)

Manual section:

1

85.1.30.1.1. Name

pepsi-stage-block-language - the language-policy stage of the Pepsi pipeline.

85.1.30.1.2. Synopsis

pepsi-stage-block-language [GLOBAL-OPTIONS] worker

85.1.30.1.3. Description

This stage defaults to FUSION = yes: when stage fusion is enabled ([pepsi] ALLOW_FUSION, the default) and this stage is folded into the unified pepsi binary, a predecessor may run it in its own worker process instead of dispatching it separately. See pepsi-dispatch(1) and pepsi.conf(5).

pepsi-stage-block-language is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It loads that pepsi.workqueue row (refusing to act unless its status is running), reads its [stage-<stage>] section, scores the human language(s) recorded under state.language (by pepsi-stage-detect-language(1)) against a configured whitelist and blacklist, and routes the message accordingly: above the score threshold it advances to NEXT_STAGE, otherwise it is blocked. What blocking does is the ENFORCEMENT option — a bounce, or a flag on a message that is delivered anyway (see Enforcement below). The rest of state (state.dsn, the ingress origin, …) is preserved, and this stage never adds anything to it.

85.1.30.1.4. Scoring

state.language is an HTTP Accept-Language-style string of code;q=quality items, where each quality is the detection probability of that language. The stage turns it into a single number:

score = Σ q over languages listed in WHITELIST
      − Σ q over languages listed in BLACKLIST

Languages on neither list contribute nothing. For example, with English whitelisted and French blacklisted, a message recorded as en;q=0.7, fr;q=0.3 scores 0.7 − 0.3 = 0.4.

A list entry is a language range, matched against a detected tag by RFC 4647 §3.3.1 basic filtering: the range matches when it equals the tag, or equals a prefix of it followed by -. So a plain de matches a detected de-CH, and a narrower de-CH — which is also accepted in the lists — matches only a detected de-CH, not a bare de.

If no language was detected (state.language is absent), the score is computed against the pseudo-distribution none;q=1. The literal pseudo-code none may itself appear in WHITELIST or BLACKLIST, so an operator decides the fate of undetected mail: blacklisting none scores it −1, whitelisting it scores +1, and listing it nowhere leaves it at 0.

A message that carries an explicit state.spam = false verdict — set by an upstream whitelist match (pepsi-stage-check-whitelist(1)) or by ingress for mail addressed solely to the reserved <Postmaster> mailbox (RFC 5321 §4.5.1) — bypasses the language policy entirely and advances to NEXT_STAGE regardless of its detected language.

Otherwise, a message whose score is strictly greater than THRESHOLD advances to NEXT_STAGE; one at or below it is blocked. A message that is already a bounce (the null envelope sender) is exempt in both enforcement modes — it advances to NEXT_STAGE regardless of its score, because a bounce must never be bounced and a flag would then claim a bounce that was never going to happen.

Note

Because the comparison is strict and unlisted languages contribute 0, an “allow all except a blacklist” policy needs a negative THRESHOLD. With the default THRESHOLD = 0.0 and only a BLACKLIST set, every neutral-language message scores 0 — which is not greater than 0 — and is blocked. pepsi-setup(1) warns about this particular misconfiguration (a BLACKLIST with an empty WHITELIST and a non-negative THRESHOLD).

This stage never pauses a message, and the message body is never loaded.

85.1.30.1.5. Enforcement

ENFORCEMENT decides what happens to a blocked message. Both modes act on exactly the same set of messages: everything hard bounces, soft flags.

hard (the default)

The message is routed to BOUNCE_STAGE with a permanent failure-DSN state, so pepsi-stage-bounce(1) emits a bounce only if the sender’s NOTIFY requests FAILURE (the default when absent). Either way the message is not delivered. Nothing about the message itself is rewritten. pepsi-setup(1) refuses a hard stage with no BOUNCE_STAGE, and its worker refuses to start on such a section, since it would otherwise fail every message the policy rejects.

A message with several envelope recipients is first split so that each recipient is blocked on its own row, and each therefore gets its own per-recipient group in the DSN (RFC 3464 §2.3.1) and is judged by its own NOTIFY. Without the split the report would name one address and leave the sender believing the others had been delivered.

soft

The message is delivered anyway, marked, and advances to NEXT_STAGE:

  • SUBJECT_FLAG_LABEL (default [!LANG]) is appended to its Subject:, in the header block and in the row’s subject column, so the mark is visible both to the recipient and to pepsi-queue(1). It is appended only when the subject does not already contain it, case-insensitively, so a reply quoting it back through this stage does not collect a second one. A message that arrived with no Subject: field gains none — the stage does not invent a header — but its column is still marked.

  • An X-Pepsi-Detected-Languages header is added, carrying the detection in the same code;q=value form as an HTTP Accept-Language header. It is re-rendered from the distribution that was actually scored — so the field and the decision cannot disagree, with each quality rounded to two decimals — and for undetected mail it carries the pseudo-distribution none;q=1 that was scored. Any copy of that field already on the message is removed first: it is not in the X-Pepsi-* namespace that pepsi-stage-decrypt(1) strips wholesale, so an inbound message may carry a forged one, and the field a delivered message ends up with must be this server’s answer rather than the sender’s.

  • BOUNCE_STAGE is not consulted, and no DSN is generated.

This is the mode to run first. The flags accumulating in real mailboxes are a faithful preview of what enforcement would refuse, so WHITELIST, BLACKLIST and THRESHOLD can be tuned against genuine traffic before a correspondent’s mail is thrown away. Being an ordinary stage option, ENFORCEMENT is per-address overridable (pepsi-settings(1)) — one account can stay in training while the rest of the deployment enforces.

Warning

Rewriting the Subject: breaks the sender’s DKIM signature and this server’s own ARC AMS: both over-sign Subject. That is the same trade pepsi-stage-vacation(1) makes for its VACATION_TAG, and it is free on a branch that ends in local delivery and not free on one that relays the message onward. Put ENFORCEMENT = soft on the former. hard rewrites no header at all.

85.1.30.1.6. Configuration

Options live in the stage’s own [stage-<name>] section (PROGRAM = pepsi-stage-block-language): NEXT_STAGE, BOUNCE_STAGE, the WHITELIST/BLACKLIST language lists, the THRESHOLD, and the ENFORCEMENT/SUBJECT_FLAG_LABEL pair above. They are documented in pepsi.conf(5).

85.1.30.1.7. State

Inputs: state.language (the Accept-Language-style string written by pepsi-stage-detect-language(1); treated as none;q=1 when absent) and, for a bounced message, state.dsn and the envelope recipient.

Outputs: on the pass path the stage advances and leaves state untouched. Under ENFORCEMENT = hard a blocked message reroutes to BOUNCE_STAGE, merging the failure-DSN keys pepsi-stage-bounce(1) consumes (state.bounce). Under soft the stage writes no state at all: the mark is in the message, not in the row’s metadata. The state layout is described in pepsi.state(7).

Transitions (the score is computed from WHITELIST/BLACKLIST, with the pseudo-language none scoring undetected mail):

  • state.spam = false → advance to NEXT_STAGE, unscored;

  • score above THRESHOLD → advance to NEXT_STAGE;

  • score at or below THRESHOLD, message is a bounce (null sender) → advance to NEXT_STAGE, unmarked, in either mode;

  • score at or below THRESHOLD, ENFORCEMENT = hard, more than one envelope recipient → the row is split into one row per recipient, each left pending at this same stage, and each judged again on its own next pass;

  • score at or below THRESHOLD, ENFORCEMENT = hard → reroute to BOUNCE_STAGE carrying state.bounce (a failure DSN);

  • score at or below THRESHOLD, ENFORCEMENT = soft → rewrite the Subject:/X-Pepsi-Detected-Languages headers and advance to NEXT_STAGE.

The stage never pauses. It fails a message only when a per-address setting turns on ENFORCEMENT = hard for a stage with no BOUNCE_STAGE and the policy blocks it.

85.1.30.1.8. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

85.1.30.1.9. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

85.1.30.1.10. Exit Status

0

The message was processed (advanced or routed to the bounce stage).

1

An error occurred (message not found or not running, or a per-address setting that breaks the stage’s section — an invalid language code, both WHITELIST and BLACKLIST empty, an overlap between them, or a block with no BOUNCE_STAGE). The reason is written to the log.

A fault of the host (the database, a template or helper that cannot be used) is not reported as a failure: the message is paused and retried, as Stage errors in pepsi-dispatch(1) describes. A section that does not parse makes the worker refuse to start (status 78) instead of failing each message in turn.

85.1.30.1.11. Examples

Score and route message 42 (it must be running):

echo 42 | pepsi-stage-block-language -c /etc/pepsi/pepsi.conf worker

A pipeline section that accepts English mail, rejects French-looking mail, and lets undetected mail through (0 exceeds the -0.1 threshold):

[stage-block-language]
PROGRAM = pepsi-stage-block-language
NEXT_STAGE = srs
BOUNCE_STAGE = bounce
WHITELIST = en
BLACKLIST = fr
THRESHOLD = -0.1

The same policy in training mode: nothing is refused, and the mail that would have been refused arrives with a marked subject and a header saying why:

[stage-block-language]
PROGRAM = pepsi-stage-block-language
NEXT_STAGE = local
BOUNCE_STAGE = bounce
WHITELIST = en
BLACKLIST = fr
THRESHOLD = -0.1
ENFORCEMENT = soft

A French-looking message then reaches the recipient as:

X-Pepsi-Detected-Languages: fr;q=0.87, en;q=0.09
Subject: Bonjour [!LANG]

Keep one user in training while everybody else is enforced (pepsi-settings(1)):

pepsi-settings set alice@example.org block-language ENFORCEMENT soft

85.1.30.1.12. See Also

pepsi-config(1), pepsi-stage-detect-language(1), pepsi-stage-bounce(1), pepsi-dispatch(1), pepsi-settings(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

85.1.30.1.13. Bugs

Report bugs to the Pepsi issue tracker.