85.1.32. pepsi-stage-secretary

hold mail from unknown senders until they confirm by replying

Manual section:

1

85.1.32.1.1. Name

pepsi-stage-secretary - the confirm-to-send stage of the Pepsi pipeline.

85.1.32.1.2. Synopsis

pepsi-stage-secretary [GLOBAL-OPTIONS] worker

85.1.32.1.3. Description

pepsi-stage-secretary is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It implements confirm-to-send, after qmail’s qsecretary: mail from a sender that no whitelist knows is held, and the sender is asked — in their language — to confirm by replying. The reply adds the sender to the whitelist and releases every message held for them; later mail from them is recognised by pepsi-stage-check-whitelist(1) ahead of this stage and never held again.

Without a reply by HOLD_TIME the held mail takes the timeout path: it is rerouted to BOUNCE_STAGE when one is wired (a DSN, a quarantine, a discard — the operator’s choice), and otherwise deleted.

The header block is loaded, the body never is: a challenge quotes nothing of the held message but a short hint of its subject.

85.1.32.1.4. What it does and does not stop

Confirm-to-send is weaker than pay-to-send (pepsi-stage-anti-spam(1)). A spammer with a working mailbox can automate the reply, and the declaration the sender agrees to (PENALTY) is a legal deterrent, not an economic one. It is still useful: most bulk mail comes from addresses that cannot receive, and it costs a legitimate correspondent one reply, once. The two combine — see UNCHALLENGEABLE_STAGE.

85.1.32.1.5. Per message

The stage decides in this order:

1. A reply to a challenge. A recipient is the reply address <CONTROL_LOCAL_PART>-<cookie>@<domain>, where <cookie> is 32 hex digits and <domain> one of LOCAL_DOMAINS (default [pepsi-ingress] ACCEPTED_DOMAINS). A row that mixes reply addresses with ordinary recipients is split first, both halves coming back to this stage. Then:

   • a null-sender reply means the challenge bounced: its address cannot receive, so the held messages are woken and take the timeout path now rather than after HOLD_TIME;

   • a reply marked as automatic — Auto-Submitted: other than no, Precedence: bulk|list|junk, an X-Auto-Response-Suppress: covering auto-replies, a multipart/report body — is ignored: an out-of-office responder that answers From: instead of the null envelope sender must not agree on its owner’s behalf;

   • any other reply confirms, provided its envelope sender or its From: addr-spec is the challenged address. The sender is added to the whitelist, every message held for the challenge is released, and the challenge is deleted. A reply to an unknown or expired cookie, or from anybody else, is logged and changes nothing.

   The reply itself is consumed (deleted) in every case.

2. A null-sender message (a bounce) and locally submitted mail (state.local_origin) pass to NEXT_STAGE.

3. state.spam = false (a whitelisted sender) passes; a message released by a confirmation passes too, gaining state.spam = false so a pay-to-send stage downstream agrees.

4. state.spam = true takes the timeout path at once.

5. A held message waking up: abandoned (its challenge bounced) or past its deadline → the timeout path, deleting the challenge with the last message it held; woken early → paused again until the deadline.

6. First encounter. The recipients are grouped by the whitelist WHITELIST_NAME expands to for each (a {login}/{localpart} template can name a different list per recipient), and a message whose recipients fall into several groups is fanned out into one row per group. Then the message is unchallengeable, and goes to UNCHALLENGEABLE_STAGE, when:

   • an RFC 3834 rule says it must not be answered (the same rules pepsi-stage-vacation(1) applies: mailing-list mail, Precedence: bulk, Auto-Submitted:, service senders such as noreply@, delivery reports);

   • the sender is also one of the recipients;

   • From: does not name exactly one mailbox, or its addr-spec is not the envelope sender — the challenge goes to MAIL FROM and the whitelist matches From:, so the two must be one address;

   • REQUIRE_AUTHENTICATED is on and neither state.auth.spf nor state.auth.dmarc is pass;

   • the sender has been sent MAX_CHALLENGES_PER_SENDER challenges in the last 24 hours, across every whitelist;

   • WHITELIST_NAME does not expand for the recipients (a {login} template and a recipient with no account), or there is no served domain to put the reply address at.

   Otherwise the message joins the open challenge for its (whitelist, sender) pair, or opens one and sends the challenge, and is paused until the challenge expires. A message joining an existing challenge inherits its expiry, not a fresh HOLD_TIME; only the message that opened a challenge sends one.

A held row records state.secretary: challenge (the cookie), deadline (epoch seconds) and whitelist, later confirmed or abandoned; mail sent to UNCHALLENGEABLE_STAGE carries state.secretary.unchallengeable (the reason). The timeout path removes state.secretary entirely. See pepsi.state(7).

85.1.32.1.6. The challenge

The challenge is a text/plain message injected at RESPONSE_STAGE, where it is signed and relayed like any outbound mail. It has the null envelope sender and Auto-Submitted: auto-replied (RFC 3834), so a compliant peer — another Pepsi secretary included — never answers it. It is To: the envelope sender; From: is the reply address, shown under the protected recipient’s address as display name, and Reply-To: repeats it, so any mail client’s Reply reaches it. It threads under the held message through In-Reply-To:/References: and carries Content-Language:.

It never quotes the held message’s body or attachments: a challenge goes to an address that, for spam, is usually forged, so anything it quoted would be delivered by us to a stranger.

85.1.32.1.6.1. The text

Per language, highest first:

1. MESSAGE_<LANG> in the stage’s own section — per-address overridable, so a user’s own text through pepsi-settings(1) or pepsi-stage-edit-settings(1);

2. <TEMPLATE>.<lang>.body under [pepsi] TEMPLATE_DIR (default secretary-challenge.<lang>.body).

The first source wins per language, not wholesale: a user’s MESSAGE_EN does not displace the shipped German template for a German sender. The language is chosen as pepsi-stage-vacation(1) chooses it: state.language in descending confidence, then DEFAULT_LANGUAGE, each by exact tag, primary subtag or a refining region tag. The same rules apply as there: two spaces in a text become a newline, and the fields below render verbatim whichever brace style is used.

The subject is SUBJECT_<LANG> for the language the text was chosen in, else SUBJECT; it is a Mustache text too. Should nothing render (no text for the language or the default, or a template that does not compile), the built-in English text is sent and a warning logged — a challenge that is not sent costs the sender their message.

85.1.32.1.6.2. Fields

{{SENDER_NAME}}

The sender’s From: display name, or their address.

{{RECIPIENT}}

The held message’s envelope recipient(s), comma-separated.

{{SUBJECT_HINT}}

The held message’s subject, RFC 2047-decoded and cut to SUBJECT_HINT_LENGTH characters (with … when cut) — enough for a correspondent to recognise their mail, too little to carry a payload. Absent for an empty subject or a length of 0.

{{ORIGINAL_DATE}}

The held message’s Date: — absent when it has none.

{{DEADLINE}}

When the challenge expires, as YYYY-MM-DD HH:MM UTC.

{{REPLY_ADDRESS}}

The reply address.

{{PENALTY}}

The configured PENALTY — absent when there is none, so a template leaves the sentence out with {{#PENALTY}}…{{/PENALTY}}.

85.1.32.1.6.3. The confirmation notice

With CONFIRM_NOTICE on, a confirmed sender is also told that their mail was delivered: a null-sender notice from <CONTROL_LOCAL_PART>@<domain>, whose text comes from CONFIRM_MESSAGE_<LANG> or secretary-confirmed.<lang>.body (with a built-in English fallback) and whose subject is fixed.

85.1.32.1.7. The cookie

The cookie is a random 128-bit identifier from the operating system’s generator, looked up in the database; there is no secret to provision and nothing to rotate. A stolen cookie is worthless: a confirmation must come from the challenged address, and forging that address gains nothing unless the forger can also read its mail, which is exactly what the cookie proves.

Late replies have no effect. A confirmation deletes the challenge, the timeout path deletes it, and every lookup ignores a challenge past its expiry, so a reply arriving after the deadline matches nothing, whether or not the row has been swept yet.

85.1.32.1.8. Options

All of these are per-address overridable.

NEXT_STAGE

Where passing, whitelisted and confirmed mail goes. Required.

RESPONSE_STAGE

Stage the challenge (and the confirmation notice) is injected at, so it is signed and relayed. Required.

BOUNCE_STAGE

The timeout path. Unset, timed-out mail is deleted. An unanswered challenge usually means a forged sender, so a DSN here is a second piece of backscatter; pepsi-setup(1)’s wizard leaves it unset.

UNCHALLENGEABLE_STAGE

Where mail goes that must not be challenged (see above). Unset, it takes the timeout path. Point it at a pepsi-stage-anti-spam(1) stage to ask such senders to pay instead, at NEXT_STAGE to deliver it, or at a spam filter.

WHITELIST_NAME

The one whitelist a confirmed sender is added to; may be a {login} / {localpart} template expanded per recipient, exactly as in pepsi-stage-check-whitelist(1). Required. A check-whitelist stage ahead of this one must consult it, or a confirmed sender is challenged again on every message (pepsi-setup(1) warns).

Every stranger who answers a challenge is written into this list, so an account owner may only point it into their own namespace: a value they set by mail through pepsi-stage-edit-settings(1) must expand, for their address, to a <login>/... name of the account that address resolves to ({login}/friends, say), or be the whitelist the operator’s configuration already names for them. A global name, another user’s list, or any value for an address that is no local account is refused there, and nothing changes.

The operator may name any whitelist, a shared one included, in the INI file, in pepsi.config_override at any scope (global, domain:, address:) and in a pepsi.settings row written with pepsi-settings(1). The stage uses the effective value as it finds it.

CONTROL_LOCAL_PART

The local part reply addresses start with. Default secretary. At most 30 ASCII letters, digits, ., _ or -.

HOLD_TIME

How long a held message waits for its sender. Default 120 h. Use h/m/s units.

REQUIRE_AUTHENTICATED

Challenge only a sender whose domain passed SPF or DMARC. Default yes: it is the backscatter gate.

MAX_CHALLENGES_PER_SENDER

Challenges one address may be sent per 24 hours, across all whitelists. Default 5; 0 means no limit. A challenge that bounces still counts.

DKIM_REQUIRED

auto (the default), yes or no: the dkim_required of the whitelist row written for a confirmed sender. auto mirrors the held mail — required if any message held for the challenge passed DKIM (or ARC together with DMARC).

CONFIRM_NOTICE

Tell a confirmed sender their mail was delivered. Default no.

PENALTY

Free text naming what the sender agrees to pay if their message was unsolicited (e.g. EUR:50). Unset by default.

SUBJECT_HINT_LENGTH

Characters of the held subject quoted in the challenge. Default 8; 0 omits the hint; at most 64.

TEMPLATE

Base name of the challenge template files. Default secretary-challenge.

SUBJECT

Subject of the challenge when no SUBJECT_<LANG> fits. Default Please confirm your message to {{RECIPIENT}}.

DEFAULT_LANGUAGE

Language used when none of the sender’s detected languages has a text. Default en; pepsi-setup(1) requires a text for it.

MESSAGE_<LANG>, SUBJECT_<LANG>, CONFIRM_MESSAGE_<LANG>

Per-language overrides of the challenge text, its subject and the confirmation text.

LOCAL_DOMAINS, TARGETS, RECIPIENT_DELIMITER

The shared locality options: which domains a reply address may be at, and how {login}/{localpart} expand.

85.1.32.1.9. Placement

On the inbound path:

• after pepsi-stage-detect-language(1), so the challenge is in the sender’s language;

• after pepsi-stage-check-whitelist(1), whose state.spam = false is what lets a known sender pass — this stage never queries the whitelist itself;

• before pepsi-stage-aliases(1), the list router and local delivery, none of which must see a reply address first (an @domain catch-all or a - RECIPIENT_DELIMITER would rewrite it).

The outbound path should run pepsi-stage-auto-whitelist(1) into the same WHITELIST_NAME, so the people your users write to are never challenged when they answer. Spam milters stay ahead of check-whitelist by default; the manual’s “Spam filters and the secretary” describes moving them behind this stage.

85.1.32.1.10. Database

Two tables. pepsi.secretary_challenge holds one open challenge per (whitelist, sender): the cookie, the expiry and whether any held message passed DKIM. pepsi.secretary_sent records that a challenge was sent, without its cookie, for the 24-hour cap — kept apart so a challenge that is answered, bounces or expires does not give its sender the budget back. Four functions do the work in one statement each: secretary_challenge_claim() (race-safe: two workers claiming one pair open one challenge), secretary_confirm(), secretary_abandon() and secretary_expire(). Expired rows are swept as new challenges are claimed; there is no cron job to run.

85.1.32.1.11. Subcommands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

85.1.32.1.12. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

85.1.32.1.13. Exit Status

0

The message was processed (forwarded, held, rerouted or consumed).

1

An error occurred (message not found or not running, or a per-address setting that breaks the stage’s section). The reason is written to the log.

A fault of the host (the database, a template or helper that cannot be used) is not reported as a failure: the message is paused and retried, as Stage errors in pepsi-dispatch(1) describes. A section that does not parse makes the worker refuse to start (status 78) instead of failing each message in turn.

A challenge that cannot be queued is such a fault. It is withdrawn — the challenge deleted, unless another held message has joined it meanwhile, and the sender’s MAX_CHALLENGES_PER_SENDER slot given back — and the message is retried, rather than held for a reply nobody was asked for.

85.1.32.1.14. Examples

A pipeline section on the inbound path:

[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = secretary
WHITELIST_NAME = correspondents

[stage-secretary]
PROGRAM = pepsi-stage-secretary
NEXT_STAGE = local
RESPONSE_STAGE = dkim-sign
WHITELIST_NAME = correspondents
PENALTY = EUR:50

One user’s own challenge text, entered by the operator:

pepsi-settings -c /etc/pepsi/pepsi.conf set alice@example.org secretary \
    MESSAGE_EN "Hi {{SENDER_NAME}},  please reply once to reach me."

85.1.32.1.15. See Also

pepsi-stage-check-whitelist(1), pepsi-stage-auto-whitelist(1), pepsi-stage-anti-spam(1), pepsi-stage-vacation(1), pepsi-whitelist(1), pepsi-settings(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

RFC 3834 (automatic responses).

85.1.32.1.16. Bugs

Report bugs to the Pepsi issue tracker.