95.1.31. pepsi-stage-block-language

route a message by its detected body language(s)

Manual section:

1

95.1.31.1.1. Name

pepsi-stage-block-language - the language-policy stage of the Pepsi pipeline.

95.1.31.1.2. Synopsis

pepsi-stage-block-language [GLOBAL-OPTIONS] worker

95.1.31.1.3. Description

This stage defaults to FUSION = yes: when stage fusion is enabled ([pepsi] ALLOW_FUSION, the default) and this stage is folded into the unified pepsi binary, a predecessor may run it in its own worker process instead of dispatching it separately. See pepsi-dispatch(1) and pepsi.conf(5).

pepsi-stage-block-language is a stage program run by pepsi-dispatch(1) as a persistent worker that reads message ids on standard input. For each id it loads the pepsi.workqueue row (refusing to act unless its status is running) and reads its [stage-<stage>] section. It then scores the human language(s) that pepsi-stage-detect-language(1) recorded under state.language against a configured whitelist and blacklist, and routes the message accordingly: above the score threshold it advances to NEXT_STAGE, otherwise it is blocked. The ENFORCEMENT option decides what blocking does: a bounce, or a flag on a message that is delivered anyway (see Enforcement below). The rest of state (state.dsn, the ingress origin, …) is preserved; the stage adds to it only the state.bounce of a hard block (see State below).

95.1.31.1.4. Scoring

state.language is an HTTP Accept-Language-style string of code;q=quality items, where each quality is the detection probability of that language. The stage turns it into a single number:

score = Σ q over languages listed in WHITELIST
      − Σ q over languages listed in BLACKLIST

Languages on neither list contribute nothing. For example, with English whitelisted and French blacklisted, a message recorded as en;q=0.7, fr;q=0.3 scores 0.7 − 0.3 = 0.4.

A list entry is a language range, matched against a detected tag by RFC 4647 §3.3.1 basic filtering: the range matches when it equals the tag, or equals a prefix of it followed by -. So a plain de matches a detected de-CH, and a narrower de-CH, which the lists also accept, matches only a detected de-CH, not a bare de.

If no language was detected (state.language is absent), the score is computed against the pseudo-distribution none;q=1. The literal pseudo-code none may itself appear in WHITELIST or BLACKLIST, so an operator decides the fate of undetected mail: blacklisting none scores it −1, whitelisting it scores +1, and listing it nowhere leaves it at 0.

Some messages carry an explicit state.spam = false verdict, set by an upstream whitelist match (pepsi-stage-check-whitelist(1)) or by ingress for mail addressed solely to the reserved <Postmaster> mailbox (RFC 5321 §4.5.1). Such a message bypasses the language policy entirely and advances to NEXT_STAGE regardless of its detected language.

Otherwise, a message whose score is strictly greater than THRESHOLD advances to NEXT_STAGE; one at or below it is blocked. A message that is already a bounce (the null envelope sender) is exempt in both enforcement modes: it advances to NEXT_STAGE regardless of its score, because a bounce must never be bounced and a flag would then claim a bounce that was never going to happen.

Note

Because the comparison is strict and unlisted languages contribute 0, an “allow all except a blacklist” policy needs a negative THRESHOLD. With the default THRESHOLD = 0.0 and only a BLACKLIST set, every neutral-language message scores 0, which is not greater than 0, so it is blocked. pepsi-setup(1) warns about this particular misconfiguration (a BLACKLIST with an empty WHITELIST and a non-negative THRESHOLD).

This stage never pauses a message, and the message body is never loaded.

95.1.31.1.5. Enforcement

ENFORCEMENT decides what happens to a blocked message. Both modes act on exactly the same set of messages: everything that hard bounces, soft flags.

hard (the default)

The message is routed to BOUNCE_STAGE with a permanent failure-DSN state, so pepsi-stage-bounce(1) emits a bounce only if the sender’s NOTIFY requests FAILURE (the default when absent). Either way the message is not delivered. Nothing about the message itself is rewritten. pepsi-setup(1) refuses a hard stage with no BOUNCE_STAGE, and its worker refuses to start on such a section, since it would otherwise fail every message that the policy rejects.

A message with several envelope recipients is first split so that each recipient is blocked on its own row, and each therefore gets its own per-recipient group in the DSN (RFC 3464 §2.3.1) and is judged by its own NOTIFY. Without the split the report would name one address and leave the sender believing the others had been delivered.

A message whose content came out of, or failed to open from, a ciphertext without integrity protection (an S/MIME EnvelopedData, AES-CBC; recorded by pepsi-stage-decrypt(1) in state.crypto.in) is not bounced: it is flagged and advanced as under soft. Whether a modified CBC ciphertext opens decides what language is detected, so a bounce for one outcome and none for the other would tell its sender whether the padding was valid.

soft

The message is delivered anyway, marked, and advances to NEXT_STAGE:

  • SUBJECT_FLAG_LABEL (default [!LANG]) is appended to its Subject:, in the header block and in the row’s subject column, so the mark is visible both to the recipient and to pepsi-queue(1). It is appended only when the subject does not already contain it, case-insensitively, so a reply quoting it back through this stage does not collect a second one. A message that arrived with no Subject: field gains none, because the stage does not invent a header, but its column is still marked.

  • An X-Pepsi-Detected-Languages header is added, carrying the detection in the same code;q=value form as an HTTP Accept-Language header. It is rendered again from the distribution that was actually scored, with each quality rounded to two decimals, so the field and the decision cannot disagree. For undetected mail, it carries the pseudo-distribution none;q=1 that was scored. Any copy of that field already on the message is removed first. The field is in the X-Pepsi-* namespace that pepsi-stage-decrypt(1) strips from inbound mail wholesale, but a path without that stage would otherwise let a forged copy through; the field that a delivered message ends up with must be this server’s answer rather than the sender’s.

  • BOUNCE_STAGE is not consulted, and no DSN is generated.

This is the mode to run first. The flags that accumulate in real mailboxes are a faithful preview of what enforcement would refuse, so WHITELIST, BLACKLIST and THRESHOLD can be tuned against genuine traffic before a correspondent’s mail is thrown away. Since ENFORCEMENT is an ordinary stage option, it can be overridden per address (pepsi-settings(1)), so one account can stay in training while the rest of the deployment enforces.

Warning

Rewriting the Subject: breaks the sender’s DKIM signature and this server’s own ARC AMS: both over-sign Subject. This is the same trade-off that pepsi-stage-vacation(1) makes for its VACATION_TAG. It costs nothing on a branch that ends in local delivery, but it does cost something on a branch that relays the message onward. Put ENFORCEMENT = soft on the former. hard rewrites no header at all.

95.1.31.1.6. Configuration

Options live in the stage’s own [stage-<name>] section (PROGRAM = pepsi-stage-block-language): NEXT_STAGE, BOUNCE_STAGE, the WHITELIST/BLACKLIST language lists, the THRESHOLD, and the ENFORCEMENT/SUBJECT_FLAG_LABEL pair above. They are documented in pepsi.conf(5).

95.1.31.1.7. State

Inputs: state.language (the Accept-Language-style string written by pepsi-stage-detect-language(1); treated as none;q=1 when absent) and, for a bounced message, state.dsn and the envelope recipient.

Outputs: on the pass path the stage advances and leaves state untouched. Under ENFORCEMENT = hard a blocked message reroutes to BOUNCE_STAGE, merging the failure-DSN keys that pepsi-stage-bounce(1) consumes (state.bounce). Under soft the stage writes no state at all: the mark is in the message, not in the row’s metadata. The state layout is described in pepsi.state(7).

Transitions (the score is computed from WHITELIST/BLACKLIST, with the pseudo-language none scoring undetected mail):

  • state.spam = false → advance to NEXT_STAGE, unscored;

  • score above THRESHOLD → advance to NEXT_STAGE;

  • score at or below THRESHOLD, message is a bounce (null sender) → advance to NEXT_STAGE, unmarked, in either mode;

  • score at or below THRESHOLD, ENFORCEMENT = hard, more than one envelope recipient → the row is split into one row per recipient, each left pending at this same stage, and each judged again on its own next pass;

  • score at or below THRESHOLD, ENFORCEMENT = hard, content from a ciphertext without integrity protection → as under soft;

  • score at or below THRESHOLD, ENFORCEMENT = hard → reroute to BOUNCE_STAGE carrying state.bounce (a failure DSN);

  • score at or below THRESHOLD, ENFORCEMENT = soft → rewrite the Subject:/X-Pepsi-Detected-Languages headers and advance to NEXT_STAGE.

The stage never pauses. It fails a message only when a per-address setting turns on ENFORCEMENT = hard for a stage with no BOUNCE_STAGE and the policy blocks it.

95.1.31.1.8. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

95.1.31.1.9. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default [pepsi] LOG, else info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

95.1.31.1.10. Exit Status

The outcome of each message is reported to pepsi-dispatch(1) on the worker’s status line, not as the exit status. A message that is not running fails, and so does a message for which a per-address setting breaks the stage’s section (an invalid language code, both WHITELIST and BLACKLIST empty, an overlap between them, or a block with no BOUNCE_STAGE). A fault of the host (the database, a template or helper that cannot be used) is not a failure: the message is paused and retried, as Stage errors in pepsi-dispatch(1) describes.

0

The worker ran until its standard input was closed.

1

A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.

78

The worker refused to start: a [stage-*] section it serves does not parse (e.g. both WHITELIST and BLACKLIST empty). The dispatcher requeues what it handed over and retries the stage later.

95.1.31.1.11. Examples

Score and route message 42 (it must be running):

echo 42 | pepsi-stage-block-language -c /etc/pepsi/pepsi.conf worker

A pipeline section that accepts English mail, rejects French-looking mail, and lets undetected mail through (0 exceeds the -0.1 threshold):

[stage-block-language]
PROGRAM = pepsi-stage-block-language
NEXT_STAGE = srs
BOUNCE_STAGE = bounce
WHITELIST = en
BLACKLIST = fr
THRESHOLD = -0.1

The same policy in training mode: nothing is refused, and the mail that would have been refused arrives with a marked subject and a header saying why:

[stage-block-language]
PROGRAM = pepsi-stage-block-language
NEXT_STAGE = local
BOUNCE_STAGE = bounce
WHITELIST = en
BLACKLIST = fr
THRESHOLD = -0.1
ENFORCEMENT = soft

A French-looking message then reaches the recipient as:

X-Pepsi-Detected-Languages: fr;q=0.87, en;q=0.09
Subject: Bonjour [!LANG]

Keep one user in training while everybody else is enforced (pepsi-settings(1)):

pepsi-settings set alice@example.org block-language ENFORCEMENT soft

95.1.31.1.12. See Also

pepsi-config(1), pepsi-stage-detect-language(1), pepsi-stage-bounce(1), pepsi-dispatch(1), pepsi-settings(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

95.1.31.1.13. Bugs

Report bugs to the Pepsi issue tracker.