70.1.23. pepsi-stage-block-language

route a message by its detected body language(s)

Manual section:

1

70.1.23.1.1. Name

pepsi-stage-block-language - the language-policy stage of the Pepsi pipeline.

70.1.23.1.2. Synopsis

pepsi-stage-block-language [GLOBAL-OPTIONS] worker

70.1.23.1.3. Description

This stage defaults to FUSION = yes: when stage fusion is enabled ([pepsi] ALLOW_FUSION, the default) and this stage is folded into the unified pepsi binary, a predecessor may run it in its own worker process instead of dispatching it separately. See pepsi-dispatch(1) and pepsi.conf(5).

pepsi-stage-block-language is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It loads that pepsi.ingress row (refusing to act unless its status is running), reads its [stage-<stage>] section, scores the human language(s) recorded under state.language (by pepsi-stage-detect-language(1)) against a configured whitelist and blacklist, and routes the message accordingly: above the score threshold it advances to NEXT_STAGE, otherwise it is blocked. What blocking does is the ENFORCEMENT option — a bounce, or a flag on a message that is delivered anyway (see Enforcement below). The rest of state (state.dsn, the ingress origin, …) is preserved, and this stage never adds anything to it.

70.1.23.1.4. Scoring

state.language is an HTTP Accept-Language-style string of code;q=quality items, where each quality is the detection probability of that language. The stage turns it into a single number:

score = Σ q over languages listed in WHITELIST
      − Σ q over languages listed in BLACKLIST

Languages on neither list contribute nothing. For example, with English whitelisted and French blacklisted, a message recorded as en;q=0.7, fr;q=0.3 scores 0.7 − 0.3 = 0.4.

If no language was detected (state.language is absent), the score is computed against the pseudo-distribution none;q=1. The literal pseudo-code none may itself appear in WHITELIST or BLACKLIST, so an operator decides the fate of undetected mail: blacklisting none scores it −1, whitelisting it scores +1, and listing it nowhere leaves it at 0.

A message that carries an explicit state.spam = false verdict — set by an upstream whitelist match (pepsi-stage-check-whitelist(1)) or by ingress for mail to the reserved <Postmaster> mailbox (RFC 5321 §4.5.1) — bypasses the language policy entirely and advances to NEXT_STAGE regardless of its detected language.

Otherwise, a message whose score is strictly greater than THRESHOLD advances to NEXT_STAGE; one at or below it is blocked. A message that is already a bounce (the null envelope sender) is exempt in both enforcement modes — it advances to NEXT_STAGE regardless of its score, because a bounce must never be bounced and a flag would then claim a bounce that was never going to happen.

Note

Because the comparison is strict and unlisted languages contribute 0, an “allow all except a blacklist” policy needs a negative THRESHOLD. With the default THRESHOLD = 0.0 and only a BLACKLIST set, every neutral-language message scores 0 — which is not greater than 0 — and is blocked. pepsi-setup(1) warns about this particular misconfiguration (a BLACKLIST with an empty WHITELIST and a non-negative THRESHOLD).

This stage never pauses a message, and the message body is never loaded.

70.1.23.1.5. Enforcement

ENFORCEMENT decides what happens to a blocked message. Both modes act on exactly the same set of messages — everything hard bounces, soft flags — and that correspondence is the point.

hard (the default)

The message is routed to BOUNCE_STAGE with a permanent failure-DSN state, so pepsi-stage-bounce(1) emits a bounce only if the sender’s NOTIFY requests FAILURE (the default when absent). Either way the message is not delivered. Nothing about the message itself is rewritten. This is the behaviour the stage had before the option existed, so a configuration written without it is unchanged.

soft

The message is delivered anyway, marked, and advances to NEXT_STAGE:

  • SUBJECT_FLAG_LABEL (default [!LANG]) is appended to its Subject:, in the header block and in the row’s subject column, so the mark is visible both to the recipient and to pepsi-queue(1). It is appended only when the subject does not already contain it, case-insensitively, so a reply quoting it back through this stage does not collect a second one. A message that arrived with no Subject: field gains none — the stage does not invent a header — but its column is still marked.

  • A Pepsi-Detected-Languages header is added, carrying the detection in the same code;q=value form as an HTTP Accept-Language header — the same text state.language holds, and for undetected mail the pseudo-distribution none;q=1 that was actually scored. Any copy of that field already on the message is removed first: it is not in the X-Pepsi-* namespace that pepsi-stage-decrypt(1) strips wholesale, so an inbound message may carry a forged one, and the field a delivered message ends up with must be this server’s answer rather than the sender’s.

  • BOUNCE_STAGE is not consulted, and no DSN is generated.

This is the mode to run first. The flags accumulating in real mailboxes are a faithful preview of what enforcement would refuse, so WHITELIST, BLACKLIST and THRESHOLD can be tuned against genuine traffic before a correspondent’s mail is thrown away. Being an ordinary stage option, ENFORCEMENT is per-address overridable (pepsi-settings(1)) — one account can stay in training while the rest of the deployment enforces.

Warning

Rewriting the Subject: breaks the sender’s DKIM signature and this server’s own ARC AMS: both over-sign Subject. That is the same trade pepsi-stage-vacation(1) makes for its VACATION_TAG, and it is free on a branch that ends in local delivery and not free on one that relays the message onward. Put ENFORCEMENT = soft on the former. hard rewrites no header at all.

70.1.23.1.6. Configuration

Options live in the stage’s own [stage-<name>] section (PROGRAM = pepsi-stage-block-language): NEXT_STAGE, BOUNCE_STAGE, the WHITELIST/BLACKLIST language lists, the THRESHOLD, and the ENFORCEMENT/SUBJECT_FLAG_LABEL pair above. They are documented in pepsi.conf(5).

70.1.23.1.7. State

Inputs: state.language (the Accept-Language-style string written by pepsi-stage-detect-language(1); treated as none;q=1 when absent) and, for a bounced message, state.dsn and the envelope recipient.

Outputs: on the pass path the stage advances and leaves state untouched. Under ENFORCEMENT = hard a blocked message reroutes to BOUNCE_STAGE, merging the failure-DSN keys pepsi-stage-bounce(1) consumes (state.bounce). Under soft the stage writes no state at all: the mark is in the message, not in the row’s metadata. The state layout is described in pepsi.state(7).

Transitions (the score is computed from WHITELIST/BLACKLIST, with the pseudo-language none scoring undetected mail):

  • state.spam = false → advance to NEXT_STAGE, unscored;

  • score above THRESHOLD → advance to NEXT_STAGE;

  • score at or below THRESHOLD, message is a bounce (null sender) → advance to NEXT_STAGE, unmarked, in either mode;

  • score at or below THRESHOLD, ENFORCEMENT = hard → reroute to BOUNCE_STAGE carrying state.bounce (a failure DSN);

  • score at or below THRESHOLD, ENFORCEMENT = soft → rewrite the Subject:/Pepsi-Detected-Languages headers and advance to NEXT_STAGE.

The stage never pauses or fails.

70.1.23.1.8. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

70.1.23.1.9. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

70.1.23.1.10. Exit Status

0

The message was processed (advanced or routed to the bounce stage).

1

An error occurred (message not found or not running, misconfigured stage — e.g. an invalid language code, an overlap between WHITELIST and BLACKLIST, or a block with no BOUNCE_STAGE — or a database error). The reason is written to the log.

70.1.23.1.11. Examples

Score and route message 42 (it must be running):

pepsi-stage-block-language -c /etc/pepsi/pepsi.conf 42

A pipeline section that accepts English mail, rejects French-looking mail, and lets undetected mail through (0 exceeds the -0.1 threshold):

[stage-block-language]
PROGRAM = pepsi-stage-block-language
NEXT_STAGE = srs
BOUNCE_STAGE = bounce
WHITELIST = en
BLACKLIST = fr
THRESHOLD = -0.1

The same policy in training mode: nothing is refused, and the mail that would have been refused arrives with a marked subject and a header saying why:

[stage-block-language]
PROGRAM = pepsi-stage-block-language
NEXT_STAGE = local
BOUNCE_STAGE = bounce
WHITELIST = en
BLACKLIST = fr
THRESHOLD = -0.1
ENFORCEMENT = soft

A French-looking message then reaches the recipient as:

Pepsi-Detected-Languages: fr;q=0.87, en;q=0.09
Subject: Bonjour [!LANG]

Keep one user in training while everybody else is enforced (pepsi-settings(1)):

pepsi-settings set alice@example.org block-language ENFORCEMENT soft

70.1.23.1.12. See Also

pepsi-config(1), pepsi-stage-detect-language(1), pepsi-stage-bounce(1), pepsi-dispatch(1), pepsi-settings(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

70.1.23.1.13. Bugs

Report bugs to the Pepsi issue tracker.