95.1.27. pepsi-stage-autocrypt-learn

learn correspondents’ keys from the mail they send

Manual section:

1

95.1.27.1.1. Name

pepsi-stage-autocrypt-learn - the key-learning stage of the Pepsi pipeline.

95.1.27.1.2. Synopsis

pepsi-stage-autocrypt-learn [GLOBAL-OPTIONS] worker

95.1.27.1.3. Description

pepsi-stage-autocrypt-learn is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It loads that pepsi.workqueue row (refusing to act unless its status is running) and reads its [stage-<stage>] section.

It is where an inbound message teaches Pepsi a public key. It learns keys from four sources, all of them things that a correspondent put in a message they sent us:

  • the sender’s Autocrypt: header (Autocrypt Level 1 §2.1);

  • an attached application/pgp-keys part;

  • the certificate that made an S/MIME signature over the message: a certificate is learnt only when the signature verifies under its key, never because the SignedData merely carries it (a chain, a stranger’s certificate attached beside the signer’s, a certs-only message);

  • the Autocrypt-Gossip: fields of a message that arrived encrypted, which introduce the other recipients’ keys (Level 1 §5.3).

Everything it learns is written to pepsi.peer_key through the same keydisc_resolve path that pepsi-keydisc(1) uses, so a key learnt here also releases any message parked waiting for that address’s key.

The stage never drops, pauses, bounces or rewrites a message. It learns and advances, so NEXT_STAGE is mandatory.

It also records the opposite direction: that a correspondent provably holds one of our keys (see Proof that a correspondent holds our key), and what a correspondent’s S/MIME client can read (see S/MIME capabilities).

95.1.27.1.4. S/MIME capabilities

An S/MIME signature may carry the RFC 8551 §2.5.2 SMIMECapabilities attribute: the list of ciphers that the signer’s client reads. For each signing certificate, the stage records whether that list names id-aes256-GCM, that is, whether the client reads the AES-256-GCM AuthEnvelopedData that Pepsi emits by default. The record goes into pepsi.smime_capability, the per-peer record by which pepsi-stage-encrypt(1) chooses the S/MIME container. Thunderbird lists no AES-GCM, so its users are recorded as needing AES-256-CBC EnvelopedData from their first signed message on.

A claim counts only when the signature over it verifies under a certificate that the message itself carries (and that is not a CA), and it is keyed on that certificate rather than on any address: whoever signs a claim can describe only their own key, so no sender filter is needed and none of the trust rules for keys apply. Chain validation is not required. A signature with no attribute records nothing (silence is not “CBC only”); two signatures in one message that disagree about one certificate cancel out; more than eight claims in one message teach none; and an existing record is replaced only by a claim whose signingTime is not older, so a replayed old message cannot revert a correspondent who has changed client. The same LEARN_KEYS and spam gates as for keys apply.

A correspondent who has never sent a signed message has no learnt record and gets AuthEnvelopedData; configure one with pepsi-keys peer capability set (pepsi-keys(1)), which also takes precedence over a learnt record.

95.1.27.1.5. Where to put it

This is a stage of its own rather than part of pepsi-stage-decrypt(1) because it has to run at a different place in the pipeline, and placing it wrongly quietly undoes that:

  • After pepsi-stage-decrypt. Gossip fields live inside the ciphertext, so there is nothing to read until the message has been opened.

  • After every stage that can decide that a message is junk: pepsi-stage-check-whitelist(1) and any other stage that sets state.spam. Autocrypt Level 1 §5.3 says peer state SHOULD be ignored for a message the client believes to be spam, and a stage placed before that belief exists cannot honour it. pepsi-stage-decrypt(1) runs first on the inbound path by design, so that belief does not yet exist there.

  • Before anything that answers or delivers — pepsi-stage-vacation(1), the relay stages, the local-delivery stages. A key that is learnt after the reply has been sent comes too late to encrypt the reply.

pepsi-setup --wizard places it correctly when end-to-end cryptography is enabled.

95.1.27.1.6. What it will not learn

The gates are Autocrypt Level 1’s, plus Pepsi’s own where Pepsi is deliberately stricter. A field or key that fails any of them is skipped; the message is never failed for it.

  • No key at all when LEARN_KEYS is off: the stage learns nothing, gossip included: gossip widens what a message may teach, so it cannot be on while the narrower rung is off. (The proof described under Proof that a correspondent holds our key is still recorded.)

  • Nothing from a message scored as spam, unless LEARN_FROM_SPAM says otherwise.

  • No key whose private half is known. A key on the known-compromised list is skipped, from every source, gossip included. That list is pepsi.compromised_key: the badkeys blocklist, Debian weak keys included, and the moduli that the daily pepsi-keys peer screen factored.

  • Nothing from a message offering more than eight keys for the sender: past that cap the material is a keyring stapled to a message, and taking an arbitrary subset of an attacker-chosen list would be a silent choice, so the message teaches none of them.

  • Only the sender’s own key from the first three sources: a message may legitimately carry a keyring of third parties, and storing those as though the sender vouched for them would let one correspondent seed the store for everybody. Introducing a third party is what gossip is for, and it lands on a rung of its own.

  • Gossip only from a message that arrived encrypted, and only when the arriving header block carried no Autocrypt-Gossip: field of its own. Anything on the delivery path can write such a field, so a message that carries one teaches no gossip at all. Neither fact is visible from the committed plaintext, so pepsi-stage-decrypt(1) records both under state.crypto.in and this stage reads them there. A message that never met a decrypt stage has no such record, and absent means no: such a deployment learns headers and attached keys, and no gossip.

  • No gossip for an address the message does not visibly name: a field’s addr must appear in the message’s own To:, Cc: or Reply-To: (Level 1 §5.3’s receiver rule). Without it a sender could introduce a key for any address in the world by mentioning it in a header that nobody reads.

  • Never a gossiped key for an address this host serves. A peer key for a local address is what our own outbound mail to that user would be encrypted with, so a gossip field naming one is a key substitution, not an introduction. The test is LOCAL_DOMAINS, and it applies to gossip alone: the sender’s own material is filtered by address rather than by locality (see below for what stops a forged From: from mattering).

  • Never the sender’s own address by gossip (their own key belongs on the rung above), never two fields naming one address (nothing here can arbitrate between two claims about a stranger), and nothing from a message with more than 50 gossip fields (all of them are ignored), each field carrying at most 16 KiB of keydata= (a minimal transferable key is under 4 kB base64 even for RSA-4096, so that is generous by a factor of four; an oversized field is skipped, not fatal). The separate 10 KiB limit that some readers will have met is the Autocrypt Level 1 §2.1 cap on the Autocrypt: header itself, and is never applied to a gossip field.

95.1.27.1.7. What a learnt key is worth

Little, on purpose. Harvested keys sit at rank inbound, or at inbound-signed when the message carried a DKIM signature that passed at ingress, was aligned with From: and covered the key. The signature covers the key when its h= lists the Autocrypt: header, or, for an attached key or signer certificate, when it covers the body. The stage reads this from state.auth.dkim and the Authentication-Results fragment that ingress recorded; nothing is re-verified here. An S/MIME signer certificate is also lifted to inbound-signed when DMARC passed at ingress (state.auth.dmarc = pass, by aligned DKIM or aligned SPF): the domain vouched for the sender, and the certificate signed the message. An OpenPGP key is not, because nothing but a DKIM signature binds a header field or an attachment to the message. Gossiped ones sit at gossip. Those are the bottom three rungs of the trust ladder (see pepsi-keydisc(1)):

  • such a key can never displace one from discovery or from an operator;

  • a signature checked against one is reported valid-untrusted at best, never valid, so it never sets state.signature_verified, the state key on which pepsi-stage-check-whitelist(1) gates a signature_required row;

  • a peer_key row is ignored entirely for an address for which this host holds a crypto_identity: the lookup that the crypto stages use replaces such an address’s cached keys with the public half of the identity. A message forging From: a local user can therefore leave a row behind, but can never have it used for that user.

Within a rung the conflict rule is newest wins on the message’s Autocrypt effective date, so a correspondent who reinstalls their client stops receiving mail encrypted to a key they no longer hold. Across the two inbound rungs the rule is one-way: a key that a DKIM-signed message brought is never replaced by one from an unsigned message, however new, while a signed key replaces an unsigned one whatever its date. So once a correspondent’s domain has signed for their key, forging their From: no longer redirects their mail; what remains is protection against a passive adversary for correspondents whose domain signs nothing, which is all Autocrypt claims. An operator who wants opportunistic encryption but no third-party introductions sets [pepsi-keydiscovery] MIN_TRUST = harvested; one who wants it only for signed claims sets inbound-signed.

95.1.27.1.8. When a correspondent’s key changes

Newest-wins is keyed on the From: header, which the sender writes, and inbound authentication is fail-open, so a message that merely claims to come from a correspondent can replace the key that we encrypt to them with. Autocrypt accepts that; Pepsi accepts it too, but never silently:

  • An audit row. The database writes a key.peer.rotate row to pepsi.event_log (severity notice, actor autocrypt:inbound, autocrypt:inbound-signed or autocrypt:gossip, subject the correspondent’s address) in the same statement as the replacement, with the protocol, the displaced and the new fingerprints and both effective dates. There is no rotation without its row. It is listed by GET /api/v1/events and the console’s event log, and summarised by pepsi-status(1) under Correspondent key changes.

  • A notice to the people it affects. With RESPONSE_STAGE set, Pepsi mails a notice to each local envelope recipient of the message that caused the change. These are the people to whom this correspondent just wrote, and their reply will be encrypted to the new key. The notice names the correspondent and both fingerprints, and says whether the new key was the correspondent’s own or introduced by gossip (and by whom). Each fingerprint is printed as hex and, beneath it, in the PGP word list (one word per byte, two- and three-syllable words alternating), the form to compare aloud; the new key’s trust rung is named (inbound-signed when the correspondent’s domain signed the message with an aligned DKIM signature that covered the key, inbound when nothing did, gossip). When a pepsi-stage-encrypt section answers key commands (it has a RESPONSE_STAGE and a KEYS_CONTROL_LOCAL_PART that is not none), the notice also gives the exact subject with which a reader who has compared the key pins it (pepsi-stage-encrypt(1)): pin FPR CODE, sent to that address at the recipient’s domain. It is null-sender, Auto-Submitted: auto-generated, From: postmaster@<recipient's domain>, rendered from the key-rotation.<lang>.body template under [pepsi] TEMPLATE_DIR for the message’s detected state.language with an English fallback (which pepsi-setup(1) requires to exist), and injected at RESPONSE_STAGE. “Local” is a recipient at a LOCAL_DOMAINS domain; nobody elsewhere is ever mailed, and nobody is told about their own address. Local users who correspond with the address but did not receive this message are not told: Pepsi keeps no record of who corresponds with whom, and building one for this would be the worse trade. The audit row covers them. A notice that cannot be queued (a template that cannot be read, a database error) is logged at warn and not retried: a second pass would find the new key already stored and have no change to report, so the audit row is then the only record.

  • A log line and a telemetry count (autocrypt-rotation).

A replacement by a strictly better source is the ladder working as designed and is not a rotation: for example, a WKD or DANE answer displacing a harvested key, or the owner’s own Autocrypt: header displacing a gossiped one. The one better-source replacement that is treated as a rotation, with the row, the notice and the ACCEPT_ROTATION hold, is a key from a DKIM-signed message displacing a different key from the same correspondent’s unsigned mail: both are the correspondent’s own claims, and the recipients should hear that the domain now vouches for another key.

A gossiped key that its owner confirms is promoted. When the sender’s own Autocrypt: header (or an attached key naming them) carries the same key that gossip already stored for them, the row moves from the gossip rung to inbound (or to inbound-signed when that message was DKIM-signed). This is Autocrypt Level 1 §5.3’s promotion. Because the key did not change it is neither a rotation nor held, and nobody is sent a notice; the database writes one key.peer.promote row (severity info, actor autocrypt:inbound, detail the protocol, the fingerprint, from_source gossip, source inbound and both effective dates) in the same statement, and the stage logs it. The row’s effective date becomes the sender’s own, so a later rotation by them is judged against their own claim and not the date that a third party’s message carried. The verdict that a signature can reach is unchanged (valid-untrusted at best, as for any harvested key), but the row now defends at the inbound rank, so a later gossip field offering a different key for the address is refused, and MIN_TRUST = harvested admits it. ACCEPT_ROTATION plays no part: nothing is displaced.

ACCEPT_ROTATION = expired narrows the rule for an operator who prefers refusing a genuine rotation to accepting a forged one: the newer key is taken only when every stored key for that address and protocol is dead — its validity revoked or expired, or its expires_at in the past. Otherwise the stored key is kept, and the offer is held: a key.peer.rotate.held row (severity warning, naming the kept and the offered fingerprints), the same notice saying the new key was not accepted, and the autocrypt-rotation-held count. Deadness is read from what is stored, and every learnt key is stored with the expiry that its own material states: for OpenPGP the date in the primary key’s current self-signature, for S/MIME the certificate’s notAfter. A key that has lapsed is therefore dead from that moment. A key that states no expiry is dead only once it is marked revoked or expired, or an operator removes it (pepsi-keys peer remove) or imports the new one by hand (pepsi-keys peer import, which outranks the harvested rung). The policy is read through the stage section, so a pepsi.settings override for a recipient applies to a stage listed in EDITABLE_STAGES.

95.1.27.1.9. Proof that a correspondent holds our key

With [stage-encrypt] ATTACH_KEYS_AS_FILES on, pepsi-stage-encrypt(1) attaches the sender’s public key as a file until the correspondent provably holds it. This stage writes that proof, one row per (fingerprint, peer_address) in pepsi.peer_has_own_key, when an inbound message shows it:

  • it was decrypted with one of our keys (state.crypto.in.decrypted and decrypted_with), and was not merely passed through for a user’s own mail client;

  • it carries a signature whose verdict is valid or valid-untrusted and that covers the plaintext (signature.covers_plaintext), so a stranger who wraps somebody else’s ciphertext in their own signature proves nothing;

  • the signer is the From: address itself (never the envelope fallback used for key learning), so a forged From: cannot suppress the attachment.

The store then re-checks that the fingerprint names an active MTA key (custody = local) of the address the message was decrypted for, and prunes, in the same statement, rows whose fingerprint no longer names an active identity. The record is keyed on the fingerprint, so a new key starts being attached again. A peer who encrypts without signing keeps receiving the file, which does no harm.

This runs before every key-learning gate: it records something about our own key, not about theirs, so neither LEARN_KEYS nor the spam belief has anything to say about it. Like everything else here it never fails the message.

95.1.27.1.10. Configuration

Options live in the stage’s own [stage-<name>] section (PROGRAM = pepsi-stage-autocrypt-learn): LEARN_KEYS, LEARN_GOSSIP, LEARN_FROM_SPAM, ACCEPT_ROTATION (newest, the default, or expired), the optional RESPONSE_STAGE for key-change notices, the shared locality options LOCAL_DOMAINS, TARGETS and RECIPIENT_DELIMITER (parsed, but only LOCAL_DOMAINS is consulted), and a mandatory NEXT_STAGE. The stage also parses [pepsi-keydiscovery], whose NEGATIVE_TTL it hands to the shared store-and-release path. They are documented in pepsi.conf(5).

The stage is unprivileged: it writes only public key material (and the fact that a correspondent holds our public key), so it runs as the ordinary pepsi service account and is folded into the multi-call pepsi binary. It carries no setuid or setgid bit.

95.1.27.1.11. State

Inputs: state.spam (the belief upstream stages record); state.crypto.in.encrypted, state.crypto.in.decrypted and state.crypto.in.outer_gossip, written by pepsi-stage-decrypt(1), which together decide whether this message’s gossip may be believed. The sender is taken from the from_header column, falling back to the envelope sender.

Outputs: none on the message — its state is left untouched. The side effects are pepsi.peer_key rows, pepsi.peer_has_own_key rows, key.peer.rotate/key.peer.rotate.held/key.peer.promote rows in pepsi.event_log, and the key-change notices injected at RESPONSE_STAGE (token <token>-key-rotation-<n>). The proof additionally reads state.crypto.in.decrypted_with, state.crypto.in.for_client and state.crypto.in.signature. The state layout is described in pepsi.state(7).

Transitions: always advances to NEXT_STAGE. There is no branch; the stage never pauses, fails, reroutes or finishes. Every error in learning is logged and then ignored: a message is being delivered, and nothing about a stranger’s key material may stand in the way of it. Key material that does not parse is the sender’s business and logged at debug; material that parsed and could not be stored (a database error) is this host’s, and logged at warn, so an operator notices a store that has stopped learning. The configuration is the exception: the stage’s section, [pepsi-keydiscovery] and the [pepsi] crypto policy are checked when the worker starts (a broken one makes it refuse to start, exit 78, and the dispatcher holds the queue), and a section that stops parsing under a running worker is retried rather than silently skipped.

95.1.27.1.12. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

95.1.27.1.13. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default [pepsi] LOG, else info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

95.1.27.1.14. Exit Status

0

The worker exited cleanly (standard input closed).

1

An error occurred (a misconfigured stage or a database error). The reason is written to the log.

78

The worker refused to start: its section, [pepsi-keydiscovery] or the [pepsi] crypto policy does not parse. The dispatcher requeues what it handed over and retries the stage later.

95.1.27.1.15. Examples

Learn from message 42 (it must be running):

echo 42 | pepsi-stage-autocrypt-learn -c /etc/pepsi/pepsi.conf worker

An inbound pipeline placing it after the spam checks and before delivery:

[stage-check-whitelist]
PROGRAM = pepsi-stage-check-whitelist
NEXT_STAGE = autocrypt-learn
WHITELIST_NAME = correspondents

[stage-autocrypt-learn]
PROGRAM = pepsi-stage-autocrypt-learn
NEXT_STAGE = local

Tell recipients when a correspondent’s key changes, and accept a change only once the old key is dead:

[stage-autocrypt-learn]
PROGRAM = pepsi-stage-autocrypt-learn
NEXT_STAGE = local
RESPONSE_STAGE = dkim-sign
ACCEPT_ROTATION = expired

Keep opportunistic encryption but refuse third-party introductions:

[stage-autocrypt-learn]
PROGRAM = pepsi-stage-autocrypt-learn
NEXT_STAGE = local
LEARN_GOSSIP = no

95.1.27.1.16. See Also

pepsi-stage-decrypt(1), pepsi-stage-encrypt(1), pepsi-keydisc(1), pepsi-keys(1), pepsi-stage-check-whitelist(1), pepsi-dispatch(1), pepsi-status(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

The manual’s Security chapter (A correspondent’s key changing, Which S/MIME container a correspondent gets) and its Threat model chapter (Keys of correspondents) state what learnt keys are trusted for.

95.1.27.1.17. Bugs

Report bugs to the Pepsi issue tracker.