95.1.33. pepsi-stage-secretary¶
hold mail from unknown senders until they confirm by replying
- Manual section:
1
95.1.33.1.1. Name¶
pepsi-stage-secretary - the confirm-to-send stage of the Pepsi pipeline.
95.1.33.1.2. Synopsis¶
pepsi-stage-secretary [GLOBAL-OPTIONS] worker
95.1.33.1.3. Description¶
pepsi-stage-secretary is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It implements confirm-to-send, after qmail’s qsecretary: mail from a sender that no whitelist knows is held, and the sender is asked, in their language, to confirm by replying. The reply adds the sender to the whitelist and releases every message held for them; later mail from them is recognised by pepsi-stage-check-whitelist(1) ahead of this stage and never held again.
Without a reply by HOLD_TIME the held mail takes the timeout path: it is
rerouted to BOUNCE_STAGE when one is wired (a DSN, a quarantine or a
discard, as the operator chooses), and otherwise deleted. Each recipient of the
held row reaches BOUNCE_STAGE on a row of its own, so that a DSN reports on
every one of them and honours each recipient’s NOTIFY.
The header block is loaded, the body never is: a challenge quotes nothing of the held message but a short hint of its subject.
95.1.33.1.4. What it does and does not stop¶
Confirm-to-send is weaker than pay-to-send (pepsi-stage-anti-spam(1)). A spammer with a working mailbox can automate the reply, and the declaration that the sender agrees to (PENALTY) is a legal deterrent, not an economic one. It is still useful: most bulk mail comes from addresses that cannot receive, and it costs a legitimate correspondent one reply, once. The two can be combined; see UNCHALLENGEABLE_STAGE.
95.1.33.1.5. Per message¶
The stage decides in this order:
A reply to a challenge. A recipient is the reply address
<CONTROL_LOCAL_PART>-<cookie>@<domain>, where<cookie>is 32 hex digits and<domain>one of LOCAL_DOMAINS (default[pepsi-ingress] ACCEPTED_DOMAINS). A row that mixes reply addresses with ordinary recipients is split first, both halves coming back to this stage. Then:a null-sender reply means the challenge bounced: its address cannot receive, so the held messages are woken and take the timeout path now rather than after HOLD_TIME;
a reply marked as automatic is ignored:
Auto-Submitted:other thanno,Precedence: bulk|list|junk, anX-Auto-Response-Suppress:covering auto-replies, or amultipart/reportbody. An out-of-office responder that answersFrom:instead of the null envelope sender must not agree on its owner’s behalf;any other reply confirms, provided its envelope sender or its
From:addr-spec is the challenged address. The sender is added to the whitelist, every message held for the challenge is released, and the challenge is deleted. A reply to an unknown or expired cookie, or from anybody else, is logged and changes nothing.
The reply itself is consumed (deleted) in every case.
Locally submitted mail (
state.local_origin) passes to NEXT_STAGE. A null-sender message (a bounce, an automatic reply) cannot be challenged, so it has to prove that it answers mail that this host sent. It passes when it carries aPepsi-Originfield that verifies against the[pepsi-origin]secret and whose nonce is still tracked (two weeks). The field may be anywhere in the message, since a delivery report returns our header block in its body. Any other null-sender message is unchallengeable (step 6), so a forged bounce does not get a stranger’s mail past the gate. With proof-of-origin switched off ([pepsi-origin] ENABLED = no) no bounce can be proven, and every one passes; pepsi-setup(1) warns.state.spam = false(a whitelisted sender) passes; a message released by a confirmation passes too, gainingstate.spam = falseso a pay-to-send stage downstream agrees.state.spam = truetakes the timeout path at once.A held message waking up: abandoned (its challenge bounced) or past its deadline → the timeout path, deleting the challenge with the last message it held, unless its challenge was answered; woken early → paused again until the deadline.
First encounter. The recipients are grouped by the whitelist that WHITELIST_NAME expands to for each (a
{login}/{localpart}template can name a different list per recipient), and a message whose recipients fall into several groups is fanned out into one row per group. Then the message is unchallengeable, and goes to UNCHALLENGEABLE_STAGE, when:an RFC 3834 rule says it must not be answered (the same rules that pepsi-stage-vacation(1) applies: mailing-list mail,
Precedence: bulk,Auto-Submitted:, service senders such asnoreply@, delivery reports, and a sender whosestate.auth.spforstate.auth.dmarcisfail);the sender is also one of the recipients;
From:does not name exactly one mailbox, or its addr-spec is not the envelope sender: the challenge goes to MAIL FROM and the whitelist matchesFrom:, so the two must be one address;REQUIRE_AUTHENTICATED is on and neither
state.auth.spfnorstate.auth.dmarcispass;the sender has been sent MAX_CHALLENGES_PER_SENDER challenges in the last 24 hours, across every whitelist;
WHITELIST_NAME does not expand for the recipients (a
{login}template and a recipient with no account), or there is no served domain to put the reply address at.
Otherwise the message joins the open challenge for its (whitelist, sender) pair, or opens one and sends the challenge, and is paused until the challenge expires. A message joining an existing challenge inherits its expiry, not a fresh HOLD_TIME; only the message that opened a challenge sends one.
A held row records state.secretary: challenge (the cookie), deadline
(epoch seconds) and whitelist, later confirmed or abandoned; mail sent to
UNCHALLENGEABLE_STAGE carries state.secretary.unchallengeable (the reason).
The timeout path removes state.secretary entirely. See pepsi.state(7).
95.1.33.1.6. The challenge¶
The challenge is a text/plain message injected at RESPONSE_STAGE, where it
is signed and relayed like any outbound mail. It has the null envelope sender
and Auto-Submitted: auto-replied (RFC 3834), so a compliant peer — another
Pepsi secretary included — never answers it. It is To: the envelope sender;
From: is the reply address, shown under the protected recipient’s address as
display name, and Reply-To: repeats it, so any mail client’s Reply reaches it.
It threads under the held message through In-Reply-To:/References: and
carries Content-Language:.
It never quotes the held message’s body or attachments: a challenge goes to an address that, for spam, is usually forged, so anything it quoted would be delivered by us to a stranger.
95.1.33.1.6.1. The text¶
Per language, highest first:
MESSAGE_<LANG>in the stage’s own section, which is per-address overridable, so that a user can set their own text through pepsi-settings(1) or pepsi-stage-edit-settings(1);<TEMPLATE>.<lang>.bodyunder[pepsi] TEMPLATE_DIR(defaultsecretary-challenge.<lang>.body).
The first source wins per language, not wholesale: a user’s MESSAGE_EN
does not displace the shipped German template for a German sender. The language is
chosen as pepsi-stage-vacation(1) chooses it: state.language in descending
confidence, then DEFAULT_LANGUAGE, each by exact tag, primary subtag or a
refining region tag. The same rules apply as there: two spaces in a text become a
newline, and the fields below render verbatim whichever brace style is used.
A message whose content came out of, or failed to open from, an S/MIME
EnvelopedData (AES-CBC, no integrity protection) counts as having no
detected language: the language depends on whether a modified CBC ciphertext
opened, and a challenge written in it would tell the sender whether it did.
The subject is SUBJECT_<LANG> for the language the text was chosen in, else
the shipped subject for that language (the ten languages the templates ship in),
else SUBJECT; it is a Mustache text too. An explicit SUBJECT switches the
shipped translations off, so it is used for every language without a
SUBJECT_<LANG>. Should nothing render (no text for the
language or the default, or a template that does not compile), the built-in English
text is sent and a warning logged: a challenge that is not sent costs the sender
their message.
95.1.33.1.6.2. Fields¶
{{SENDER_NAME}}The sender’s
From:display name, or their address; always the address for a sender that did not pass SPF or DMARC (challenged only underREQUIRE_AUTHENTICATED = no), whose envelope sender may be forged.{{RECIPIENT}}The held message’s envelope recipient(s), comma-separated.
{{SUBJECT_HINT}}The held message’s subject, RFC 2047-decoded and cut to SUBJECT_HINT_LENGTH characters (with
…when cut). That is enough for a correspondent to recognise their mail, but too little to carry a payload. Absent for an empty subject, a length of 0, a message that arrived encrypted (whose subject carries the decryption outcome; see pepsi-stage-bounce(1)), or a sender that did not pass SPF or DMARC: quoting its words would let a forger have this host deliver text of their choosing to a third party.{{ORIGINAL_DATE}}The held message’s
Date:— absent when it has none.{{DEADLINE}}When the challenge expires, as
YYYY-MM-DD HH:MM UTC.{{REPLY_ADDRESS}}The reply address.
{{PENALTY}}The configured PENALTY — absent when there is none, so a template leaves the sentence out with
{{#PENALTY}}…{{/PENALTY}}.
95.1.33.1.6.3. The confirmation notice¶
With CONFIRM_NOTICE on, a confirmed sender is also told that their mail was
delivered: a null-sender notice from <CONTROL_LOCAL_PART>@<domain>, whose text
comes from CONFIRM_MESSAGE_<LANG> or secretary-confirmed.<lang>.body (with
a built-in English fallback) and whose subject is the shipped one for the
language the text was chosen in (English otherwise); it has no subject option.
95.1.33.1.8. Options¶
All of these are per-address overridable except NEXT_STAGE and BOUNCE_STAGE, which are always taken from the operator’s configuration.
- NEXT_STAGE
Where passing, whitelisted and confirmed mail goes. Required.
- RESPONSE_STAGE
Stage the challenge (and the confirmation notice) is injected at, so it is signed and relayed. Required. Both are marked as Pepsi’s own (
state.notice, kindsecretary), and aNOTICE_SIGNERstage signs them (see pepsi-stage-encrypt(1)), except while a per-address override applies to the challenged message, since an override can change the wording.- BOUNCE_STAGE
The timeout path. Unset, timed-out mail is deleted, and so is mail that cannot be challenged when UNCHALLENGEABLE_STAGE is unset as well. Most newsletters and transactional mail cannot be challenged. pepsi-setup(1)’s wizard sets it to pepsi-stage-bounce(1), so that such mail is refused visibly. An unanswered challenge can mean a forged sender, and that stage sends the DSN only when the message authenticated the sender’s domain (BOUNCE_UNAUTHENTICATED).
- UNCHALLENGEABLE_STAGE
Where mail that must not be challenged goes (see above), including a null-sender message without a proof of origin. Unset, it takes the timeout path, which refuses it through BOUNCE_STAGE, and deletes a null-sender message (a bounce is never bounced). Point it at a pepsi-stage-anti-spam(1) stage to ask such senders to pay instead, at NEXT_STAGE to deliver it, or at a spam filter.
- WHITELIST_NAME
The one whitelist a confirmed sender is added to; may be a
{login}/{localpart}template expanded per recipient, exactly as in pepsi-stage-check-whitelist(1). Required. A check-whitelist stage ahead of this one must consult it, or a confirmed sender is challenged again on every message (pepsi-setup(1) warns).Every stranger who answers a challenge is written into this list, so an account owner may only point it into their own namespace: a value they set by mail through pepsi-stage-edit-settings(1) must expand, for their address, to a
<login>/...name of the account that address resolves to ({login}/friends, say), or be the whitelist the operator’s configuration already names for them. A global name, another user’s list, or any value for an address that is no local account is refused there, and nothing changes.The operator may name any whitelist, a shared one included, in the INI file, in
pepsi.config_overrideat any scope (global,domain:,address:) and in apepsi.settingsrow written with pepsi-settings(1). The stage uses the effective value as it finds it.- CONTROL_LOCAL_PART
The local part with which reply addresses start. Default
secretary. At most 30 ASCII letters, digits,.,_or-, neither starting nor ending with.or-and without..; it is lower-cased.- HOLD_TIME
How long a held message waits for its sender. Default
120 h. Useh/m/sunits.- REQUIRE_AUTHENTICATED
Challenge only a sender whose domain passed SPF or DMARC. Default
yes: it is the backscatter gate.- MAX_CHALLENGES_PER_SENDER
Challenges one address may be sent per 24 hours, across all whitelists. Default
5;0means no limit. A challenge that bounces still counts.- DKIM_REQUIRED
auto(the default),yesorno: thedkim_requiredof the whitelist row written for a confirmed sender.automirrors the held mail: DKIM is required if any message held for the challenge passed DKIM (or ARC together with DMARC). When none did but one passed DMARC (an aligned SPF pass, or a trusted ARC sealer’s override), the row is written withdmarc_requiredinstead, so a sender whose domain does not sign is still bound to an authenticated identity rather than to an address anyone can forge.nowrites neither.- CONFIRM_NOTICE
Tell a confirmed sender their mail was delivered. Default
no.- PENALTY
Free text naming what the sender agrees to pay if their message was unsolicited (e.g.
EUR:50). Unset by default.- SUBJECT_HINT_LENGTH
Characters of the held subject quoted in the challenge. Default
8;0omits the hint; at most64.- TEMPLATE
Base name of the challenge template files. Default
secretary-challenge.- SUBJECT
Subject of the challenge when no
SUBJECT_<LANG>fits. DefaultPlease confirm your message to {{RECIPIENT}}, which stands for its shipped translations: left unset, a text chosen in one of the ten shipped languages gets that language’s subject. Set explicitly, it is used for every language without aSUBJECT_<LANG>.- DEFAULT_LANGUAGE
Language used when none of the sender’s detected languages has a text. Default
en; pepsi-setup(1) requires a text for it.- MESSAGE_<LANG>, SUBJECT_<LANG>, CONFIRM_MESSAGE_<LANG>
Per-language overrides of the challenge text, its subject and the confirmation text.
- LOCAL_DOMAINS, TARGETS, RECIPIENT_DELIMITER
The shared locality options: which domains a reply address may be at, and how
{login}/{localpart}expand.
95.1.33.1.9. Placement¶
On the inbound path:
after pepsi-stage-detect-language(1), so the challenge is in the sender’s language;
after pepsi-stage-check-whitelist(1), whose
state.spam = falseis what lets a known sender pass (this stage never queries the whitelist itself);before pepsi-stage-aliases(1), the list router and local delivery, none of which must see a reply address first (an
@domaincatch-all or a-RECIPIENT_DELIMITER would rewrite it).
The outbound path should run pepsi-stage-auto-whitelist(1) into the same WHITELIST_NAME, so the people your users write to are never challenged when they answer. Spam milters stay ahead of check-whitelist by default; the manual’s “Spam filters and the secretary” describes moving them behind this stage.
95.1.33.1.10. Database¶
Three tables. pepsi.secretary_challenge holds one open challenge per (whitelist,
sender): the cookie, the expiry and whether any held message passed DKIM.
pepsi.secretary_sent records that a challenge was sent, without its cookie,
for the 24-hour cap. It is kept apart so that a challenge that is answered,
bounces or expires does not give its sender the budget back.
pepsi.secretary_answered records that a challenge was answered, until a day
after it would have expired. A message that was still joining the challenge at
that moment finds the answer there as soon as it is paused. No reply is ever
matched against this table, so a late reply still matches nothing.
Five functions do the work in one statement each:
secretary_challenge_claim() (race-safe: two workers claiming one pair open one
challenge), secretary_confirm(), secretary_release_answered(),
secretary_abandon() and secretary_expire(). Expired rows are swept as new
challenges are claimed; there is no cron job to run.
95.1.33.1.11. Subcommands¶
- worker
Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.
95.1.33.1.12. Global Options¶
- -c FILE, –config FILE
Read the configuration from FILE instead of searching the default locations.
- -L LOGLEVEL, –log LOGLEVEL
Set the logging verbosity (default
[pepsi] LOG, elseinfo).- -v, –verbose
Show log messages from all sources.
- -h, –help; -V, –version
Print a usage summary / the version and exit.
95.1.33.1.13. Exit Status¶
The outcome of each message is reported to pepsi-dispatch(1) on the
worker’s status line, not as the exit status. A message that is not
running, or a per-address setting that breaks the stage’s section, fails
that message; a fault of the host (the database, a template or helper that
cannot be used) is not a failure: the message is paused and retried, as
Stage errors in pepsi-dispatch(1) describes.
A challenge that cannot be queued is a fault of the host. It is withdrawn, and the message is retried rather than held for a reply that nobody was asked for. Withdrawing deletes the challenge, unless another held message has joined it meanwhile, and gives the sender’s MAX_CHALLENGES_PER_SENDER slot back.
- 0
The worker ran until its standard input was closed.
- 1
A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.
- 78
The worker refused to start: a
[stage-*]section it serves does not parse. The dispatcher requeues what it handed over and retries the stage later.
95.1.33.1.14. Examples¶
A pipeline section on the inbound path:
[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = secretary
WHITELIST_NAME = correspondents
[stage-secretary]
PROGRAM = pepsi-stage-secretary
NEXT_STAGE = local
RESPONSE_STAGE = dkim-sign
WHITELIST_NAME = correspondents
PENALTY = EUR:50
One user’s own challenge text, entered by the operator:
pepsi-settings -c /etc/pepsi/pepsi.conf set alice@example.org secretary \
MESSAGE_EN "Hi {{SENDER_NAME}}, please reply once to reach me."
95.1.33.1.15. See Also¶
pepsi-stage-check-whitelist(1), pepsi-stage-auto-whitelist(1), pepsi-stage-anti-spam(1), pepsi-stage-vacation(1), pepsi-whitelist(1), pepsi-settings(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)
RFC 3834 (automatic responses).
The manual’s Threat model chapter (the remote-sender and availability sections) states the backscatter and rate claims the challenge is held to; its Security model chapter (Unknown recipients and backscatter, Resource limits) describes the shared automatic-reply rules.
95.1.33.1.16. Bugs¶
Report bugs to the Pepsi issue tracker.