95.1.33. pepsi-stage-secretary

hold mail from unknown senders until they confirm by replying

Manual section:

1

95.1.33.1.1. Name

pepsi-stage-secretary - the confirm-to-send stage of the Pepsi pipeline.

95.1.33.1.2. Synopsis

pepsi-stage-secretary [GLOBAL-OPTIONS] worker

95.1.33.1.3. Description

pepsi-stage-secretary is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It implements confirm-to-send, after qmail’s qsecretary: mail from a sender that no whitelist knows is held, and the sender is asked, in their language, to confirm by replying. The reply adds the sender to the whitelist and releases every message held for them; later mail from them is recognised by pepsi-stage-check-whitelist(1) ahead of this stage and never held again.

Without a reply by HOLD_TIME the held mail takes the timeout path: it is rerouted to BOUNCE_STAGE when one is wired (a DSN, a quarantine or a discard, as the operator chooses), and otherwise deleted. Each recipient of the held row reaches BOUNCE_STAGE on a row of its own, so that a DSN reports on every one of them and honours each recipient’s NOTIFY.

The header block is loaded, the body never is: a challenge quotes nothing of the held message but a short hint of its subject.

95.1.33.1.4. What it does and does not stop

Confirm-to-send is weaker than pay-to-send (pepsi-stage-anti-spam(1)). A spammer with a working mailbox can automate the reply, and the declaration that the sender agrees to (PENALTY) is a legal deterrent, not an economic one. It is still useful: most bulk mail comes from addresses that cannot receive, and it costs a legitimate correspondent one reply, once. The two can be combined; see UNCHALLENGEABLE_STAGE.

95.1.33.1.5. Per message

The stage decides in this order:

  1. A reply to a challenge. A recipient is the reply address <CONTROL_LOCAL_PART>-<cookie>@<domain>, where <cookie> is 32 hex digits and <domain> one of LOCAL_DOMAINS (default [pepsi-ingress] ACCEPTED_DOMAINS). A row that mixes reply addresses with ordinary recipients is split first, both halves coming back to this stage. Then:

    • a null-sender reply means the challenge bounced: its address cannot receive, so the held messages are woken and take the timeout path now rather than after HOLD_TIME;

    • a reply marked as automatic is ignored: Auto-Submitted: other than no, Precedence: bulk|list|junk, an X-Auto-Response-Suppress: covering auto-replies, or a multipart/report body. An out-of-office responder that answers From: instead of the null envelope sender must not agree on its owner’s behalf;

    • any other reply confirms, provided its envelope sender or its From: addr-spec is the challenged address. The sender is added to the whitelist, every message held for the challenge is released, and the challenge is deleted. A reply to an unknown or expired cookie, or from anybody else, is logged and changes nothing.

    The reply itself is consumed (deleted) in every case.

  2. Locally submitted mail (state.local_origin) passes to NEXT_STAGE. A null-sender message (a bounce, an automatic reply) cannot be challenged, so it has to prove that it answers mail that this host sent. It passes when it carries a Pepsi-Origin field that verifies against the [pepsi-origin] secret and whose nonce is still tracked (two weeks). The field may be anywhere in the message, since a delivery report returns our header block in its body. Any other null-sender message is unchallengeable (step 6), so a forged bounce does not get a stranger’s mail past the gate. With proof-of-origin switched off ([pepsi-origin] ENABLED = no) no bounce can be proven, and every one passes; pepsi-setup(1) warns.

  3. state.spam = false (a whitelisted sender) passes; a message released by a confirmation passes too, gaining state.spam = false so a pay-to-send stage downstream agrees.

  4. state.spam = true takes the timeout path at once.

  5. A held message waking up: abandoned (its challenge bounced) or past its deadline → the timeout path, deleting the challenge with the last message it held, unless its challenge was answered; woken early → paused again until the deadline.

  6. First encounter. The recipients are grouped by the whitelist that WHITELIST_NAME expands to for each (a {login}/{localpart} template can name a different list per recipient), and a message whose recipients fall into several groups is fanned out into one row per group. Then the message is unchallengeable, and goes to UNCHALLENGEABLE_STAGE, when:

    • an RFC 3834 rule says it must not be answered (the same rules that pepsi-stage-vacation(1) applies: mailing-list mail, Precedence: bulk, Auto-Submitted:, service senders such as noreply@, delivery reports, and a sender whose state.auth.spf or state.auth.dmarc is fail);

    • the sender is also one of the recipients;

    • From: does not name exactly one mailbox, or its addr-spec is not the envelope sender: the challenge goes to MAIL FROM and the whitelist matches From:, so the two must be one address;

    • REQUIRE_AUTHENTICATED is on and neither state.auth.spf nor state.auth.dmarc is pass;

    • the sender has been sent MAX_CHALLENGES_PER_SENDER challenges in the last 24 hours, across every whitelist;

    • WHITELIST_NAME does not expand for the recipients (a {login} template and a recipient with no account), or there is no served domain to put the reply address at.

    Otherwise the message joins the open challenge for its (whitelist, sender) pair, or opens one and sends the challenge, and is paused until the challenge expires. A message joining an existing challenge inherits its expiry, not a fresh HOLD_TIME; only the message that opened a challenge sends one.

A held row records state.secretary: challenge (the cookie), deadline (epoch seconds) and whitelist, later confirmed or abandoned; mail sent to UNCHALLENGEABLE_STAGE carries state.secretary.unchallengeable (the reason). The timeout path removes state.secretary entirely. See pepsi.state(7).

95.1.33.1.6. The challenge

The challenge is a text/plain message injected at RESPONSE_STAGE, where it is signed and relayed like any outbound mail. It has the null envelope sender and Auto-Submitted: auto-replied (RFC 3834), so a compliant peer — another Pepsi secretary included — never answers it. It is To: the envelope sender; From: is the reply address, shown under the protected recipient’s address as display name, and Reply-To: repeats it, so any mail client’s Reply reaches it. It threads under the held message through In-Reply-To:/References: and carries Content-Language:.

It never quotes the held message’s body or attachments: a challenge goes to an address that, for spam, is usually forged, so anything it quoted would be delivered by us to a stranger.

95.1.33.1.6.1. The text

Per language, highest first:

  1. MESSAGE_<LANG> in the stage’s own section, which is per-address overridable, so that a user can set their own text through pepsi-settings(1) or pepsi-stage-edit-settings(1);

  2. <TEMPLATE>.<lang>.body under [pepsi] TEMPLATE_DIR (default secretary-challenge.<lang>.body).

The first source wins per language, not wholesale: a user’s MESSAGE_EN does not displace the shipped German template for a German sender. The language is chosen as pepsi-stage-vacation(1) chooses it: state.language in descending confidence, then DEFAULT_LANGUAGE, each by exact tag, primary subtag or a refining region tag. The same rules apply as there: two spaces in a text become a newline, and the fields below render verbatim whichever brace style is used.

A message whose content came out of, or failed to open from, an S/MIME EnvelopedData (AES-CBC, no integrity protection) counts as having no detected language: the language depends on whether a modified CBC ciphertext opened, and a challenge written in it would tell the sender whether it did.

The subject is SUBJECT_<LANG> for the language the text was chosen in, else the shipped subject for that language (the ten languages the templates ship in), else SUBJECT; it is a Mustache text too. An explicit SUBJECT switches the shipped translations off, so it is used for every language without a SUBJECT_<LANG>. Should nothing render (no text for the language or the default, or a template that does not compile), the built-in English text is sent and a warning logged: a challenge that is not sent costs the sender their message.

95.1.33.1.6.2. Fields

{{SENDER_NAME}}

The sender’s From: display name, or their address; always the address for a sender that did not pass SPF or DMARC (challenged only under REQUIRE_AUTHENTICATED = no), whose envelope sender may be forged.

{{RECIPIENT}}

The held message’s envelope recipient(s), comma-separated.

{{SUBJECT_HINT}}

The held message’s subject, RFC 2047-decoded and cut to SUBJECT_HINT_LENGTH characters (with … when cut). That is enough for a correspondent to recognise their mail, but too little to carry a payload. Absent for an empty subject, a length of 0, a message that arrived encrypted (whose subject carries the decryption outcome; see pepsi-stage-bounce(1)), or a sender that did not pass SPF or DMARC: quoting its words would let a forger have this host deliver text of their choosing to a third party.

{{ORIGINAL_DATE}}

The held message’s Date: — absent when it has none.

{{DEADLINE}}

When the challenge expires, as YYYY-MM-DD HH:MM UTC.

{{REPLY_ADDRESS}}

The reply address.

{{PENALTY}}

The configured PENALTY — absent when there is none, so a template leaves the sentence out with {{#PENALTY}}…{{/PENALTY}}.

95.1.33.1.6.3. The confirmation notice

With CONFIRM_NOTICE on, a confirmed sender is also told that their mail was delivered: a null-sender notice from <CONTROL_LOCAL_PART>@<domain>, whose text comes from CONFIRM_MESSAGE_<LANG> or secretary-confirmed.<lang>.body (with a built-in English fallback) and whose subject is the shipped one for the language the text was chosen in (English otherwise); it has no subject option.

95.1.33.1.8. Options

All of these are per-address overridable except NEXT_STAGE and BOUNCE_STAGE, which are always taken from the operator’s configuration.

NEXT_STAGE

Where passing, whitelisted and confirmed mail goes. Required.

RESPONSE_STAGE

Stage the challenge (and the confirmation notice) is injected at, so it is signed and relayed. Required. Both are marked as Pepsi’s own (state.notice, kind secretary), and a NOTICE_SIGNER stage signs them (see pepsi-stage-encrypt(1)), except while a per-address override applies to the challenged message, since an override can change the wording.

BOUNCE_STAGE

The timeout path. Unset, timed-out mail is deleted, and so is mail that cannot be challenged when UNCHALLENGEABLE_STAGE is unset as well. Most newsletters and transactional mail cannot be challenged. pepsi-setup(1)’s wizard sets it to pepsi-stage-bounce(1), so that such mail is refused visibly. An unanswered challenge can mean a forged sender, and that stage sends the DSN only when the message authenticated the sender’s domain (BOUNCE_UNAUTHENTICATED).

UNCHALLENGEABLE_STAGE

Where mail that must not be challenged goes (see above), including a null-sender message without a proof of origin. Unset, it takes the timeout path, which refuses it through BOUNCE_STAGE, and deletes a null-sender message (a bounce is never bounced). Point it at a pepsi-stage-anti-spam(1) stage to ask such senders to pay instead, at NEXT_STAGE to deliver it, or at a spam filter.

WHITELIST_NAME

The one whitelist a confirmed sender is added to; may be a {login} / {localpart} template expanded per recipient, exactly as in pepsi-stage-check-whitelist(1). Required. A check-whitelist stage ahead of this one must consult it, or a confirmed sender is challenged again on every message (pepsi-setup(1) warns).

Every stranger who answers a challenge is written into this list, so an account owner may only point it into their own namespace: a value they set by mail through pepsi-stage-edit-settings(1) must expand, for their address, to a <login>/... name of the account that address resolves to ({login}/friends, say), or be the whitelist the operator’s configuration already names for them. A global name, another user’s list, or any value for an address that is no local account is refused there, and nothing changes.

The operator may name any whitelist, a shared one included, in the INI file, in pepsi.config_override at any scope (global, domain:, address:) and in a pepsi.settings row written with pepsi-settings(1). The stage uses the effective value as it finds it.

CONTROL_LOCAL_PART

The local part with which reply addresses start. Default secretary. At most 30 ASCII letters, digits, ., _ or -, neither starting nor ending with . or - and without ..; it is lower-cased.

HOLD_TIME

How long a held message waits for its sender. Default 120 h. Use h/m/s units.

REQUIRE_AUTHENTICATED

Challenge only a sender whose domain passed SPF or DMARC. Default yes: it is the backscatter gate.

MAX_CHALLENGES_PER_SENDER

Challenges one address may be sent per 24 hours, across all whitelists. Default 5; 0 means no limit. A challenge that bounces still counts.

DKIM_REQUIRED

auto (the default), yes or no: the dkim_required of the whitelist row written for a confirmed sender. auto mirrors the held mail: DKIM is required if any message held for the challenge passed DKIM (or ARC together with DMARC). When none did but one passed DMARC (an aligned SPF pass, or a trusted ARC sealer’s override), the row is written with dmarc_required instead, so a sender whose domain does not sign is still bound to an authenticated identity rather than to an address anyone can forge. no writes neither.

CONFIRM_NOTICE

Tell a confirmed sender their mail was delivered. Default no.

PENALTY

Free text naming what the sender agrees to pay if their message was unsolicited (e.g. EUR:50). Unset by default.

SUBJECT_HINT_LENGTH

Characters of the held subject quoted in the challenge. Default 8; 0 omits the hint; at most 64.

TEMPLATE

Base name of the challenge template files. Default secretary-challenge.

SUBJECT

Subject of the challenge when no SUBJECT_<LANG> fits. Default Please confirm your message to {{RECIPIENT}}, which stands for its shipped translations: left unset, a text chosen in one of the ten shipped languages gets that language’s subject. Set explicitly, it is used for every language without a SUBJECT_<LANG>.

DEFAULT_LANGUAGE

Language used when none of the sender’s detected languages has a text. Default en; pepsi-setup(1) requires a text for it.

MESSAGE_<LANG>, SUBJECT_<LANG>, CONFIRM_MESSAGE_<LANG>

Per-language overrides of the challenge text, its subject and the confirmation text.

LOCAL_DOMAINS, TARGETS, RECIPIENT_DELIMITER

The shared locality options: which domains a reply address may be at, and how {login}/{localpart} expand.

95.1.33.1.9. Placement

On the inbound path:

  • after pepsi-stage-detect-language(1), so the challenge is in the sender’s language;

  • after pepsi-stage-check-whitelist(1), whose state.spam = false is what lets a known sender pass (this stage never queries the whitelist itself);

  • before pepsi-stage-aliases(1), the list router and local delivery, none of which must see a reply address first (an @domain catch-all or a - RECIPIENT_DELIMITER would rewrite it).

The outbound path should run pepsi-stage-auto-whitelist(1) into the same WHITELIST_NAME, so the people your users write to are never challenged when they answer. Spam milters stay ahead of check-whitelist by default; the manual’s “Spam filters and the secretary” describes moving them behind this stage.

95.1.33.1.10. Database

Three tables. pepsi.secretary_challenge holds one open challenge per (whitelist, sender): the cookie, the expiry and whether any held message passed DKIM. pepsi.secretary_sent records that a challenge was sent, without its cookie, for the 24-hour cap. It is kept apart so that a challenge that is answered, bounces or expires does not give its sender the budget back. pepsi.secretary_answered records that a challenge was answered, until a day after it would have expired. A message that was still joining the challenge at that moment finds the answer there as soon as it is paused. No reply is ever matched against this table, so a late reply still matches nothing. Five functions do the work in one statement each: secretary_challenge_claim() (race-safe: two workers claiming one pair open one challenge), secretary_confirm(), secretary_release_answered(), secretary_abandon() and secretary_expire(). Expired rows are swept as new challenges are claimed; there is no cron job to run.

95.1.33.1.11. Subcommands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

95.1.33.1.12. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default [pepsi] LOG, else info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

95.1.33.1.13. Exit Status

The outcome of each message is reported to pepsi-dispatch(1) on the worker’s status line, not as the exit status. A message that is not running, or a per-address setting that breaks the stage’s section, fails that message; a fault of the host (the database, a template or helper that cannot be used) is not a failure: the message is paused and retried, as Stage errors in pepsi-dispatch(1) describes.

A challenge that cannot be queued is a fault of the host. It is withdrawn, and the message is retried rather than held for a reply that nobody was asked for. Withdrawing deletes the challenge, unless another held message has joined it meanwhile, and gives the sender’s MAX_CHALLENGES_PER_SENDER slot back.

0

The worker ran until its standard input was closed.

1

A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.

78

The worker refused to start: a [stage-*] section it serves does not parse. The dispatcher requeues what it handed over and retries the stage later.

95.1.33.1.14. Examples

A pipeline section on the inbound path:

[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = secretary
WHITELIST_NAME = correspondents

[stage-secretary]
PROGRAM = pepsi-stage-secretary
NEXT_STAGE = local
RESPONSE_STAGE = dkim-sign
WHITELIST_NAME = correspondents
PENALTY = EUR:50

One user’s own challenge text, entered by the operator:

pepsi-settings -c /etc/pepsi/pepsi.conf set alice@example.org secretary \
    MESSAGE_EN "Hi {{SENDER_NAME}},  please reply once to reach me."

95.1.33.1.15. See Also

pepsi-stage-check-whitelist(1), pepsi-stage-auto-whitelist(1), pepsi-stage-anti-spam(1), pepsi-stage-vacation(1), pepsi-whitelist(1), pepsi-settings(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

RFC 3834 (automatic responses).

The manual’s Threat model chapter (the remote-sender and availability sections) states the backscatter and rate claims the challenge is held to; its Security model chapter (Unknown recipients and backscatter, Resource limits) describes the shared automatic-reply rules.

95.1.33.1.16. Bugs

Report bugs to the Pepsi issue tracker.