9. Key management

End-to-end cryptography — OpenPGP and S/MIME — needs somewhere to keep keys. Pepsi keeps them in the database, in three tables that together are the key store, and manages them with one program, pepsi-keys. This chapter explains what is stored, how the private half is protected, who is allowed to reach it, and the two design decisions that most often surprise people: why signing and encryption material are separate, and why an address that only ever receives mail ends up with no key at all.

The key store is deliberately not the DKIM/ARC key store. Those are per-domain keys, held as files under [pepsi] KEY_DIR and read by the signing stages; see Installation. What follows is about per-address end-to-end material, which has an entirely different lifecycle.

Note

Today the store is populated and managed by pepsi-keys, filled with correspondents’ keys by the discovery layer described in Finding a correspondent’s key, and published by all three channels described in Publishing our users’ keys. What is still to come is the stages that consume it — signing, encrypting and decrypting message bodies.

9.1. Three kinds of material

pepsi.crypto_identity — our own addresses

The key pairs of the addresses this deployment serves, private half included, wrapped at rest. This is what signs outbound mail and decrypts inbound mail. Each row records the address, the protocol (openpgp or smime), what the material is capable of, its algorithm and fingerprint, the public material itself, the wrapped private material and the id of the key that wrapped it, plus the lifecycle columns (status, is_primary, published, expires_at, revoked_at, private_purged_at).

pepsi.peer_key — other people’s addresses

Remote correspondents’ public keys and certificates, cached however they were obtained. This is what encrypts outbound mail and verifies inbound signatures. Each row carries the source it came from, whether the lookup that produced it was DNSSEC-validated, the last validity verdict reached about it, and a cache deadline.

pepsi.ca_trust — trust anchors

The CA certificates an inbound S/MIME chain must reach for its signature to count as trusted. Pepsi ships no default anchor set: a signature that chains to a CA nobody chose is reported untrusted rather than bad, so an empty store degrades to “we cannot vouch for this” instead of to “this is forged”, and an operator who adds an anchor is making a decision rather than inheriting one.

9.2. Custody: how a private key is stored

The private half of an identity lives in the database, but never in the clear. It is sealed with AES-256-GCM under a key-encryption key (KEK) derived from a secret that lives only in the filesystem, in a secrets.d fragment named by [pepsi-crypto] KEY_WRAP_SECRET.

The property this buys is worth stating plainly, because it is the whole point:

The database alone never yields a private key. A stolen dump, a replica, a backup tape or an SQL-injection foothold in an unrelated component produces ciphertext and nothing else. The key that opens it is a second, differently guarded artefact and is never a database value.

The stored blob is a version octet, a 96-bit random nonce and the AES-GCM ciphertext. The version octet exists so a future construction can coexist with this one; an unrecognised version is an error, never a silent fallback.

The additional authenticated data binds the ciphertext to the row it belongs to — the address, the protocol and the purpose, length-prefixed. Without that binding, an attacker with write access to the table but not the KEK could move Alice’s wrapped signing key into Bob’s row and have the pipeline sign Bob’s mail with it. The AAD is authenticated but not stored, so such a blob simply fails to open in the wrong place, exactly as a tampered one does.

The KEK itself is derived from the configured secret with HMAC-SHA-256 under a fixed label, mixing in the key id. Two consequences follow: the secret may be any string (a passphrase, base64, hex — the derivation does not care), and every distinct KEY_WRAP_KEY_ID yields an independent 256-bit key. There is deliberately no start-up passphrase: unattended restart and socket activation have to keep working.

Warning

pepsi-setup refuses a KEY_WRAP_SECRET shorter than 16 characters. It is the single key that opens every stored private key in the deployment; generate it from a real entropy source rather than typing one.

9.3. The privilege boundary

Wrapping is one half of the custody model. The other half is a database privilege boundary, provisioned by pepsi-setup and re-asserted on every pepsi-setup run:

• a login role pepsi-crypto is created with the ordinary pipeline access (SELECT/INSERT/UPDATE/DELETE across the schema, sequences, functions) plus the crypto_identity.private_wrapped column;

• every ordinary service role — pepsi (the dispatcher and every stage worker), pepsi-ingress, pepsi-httpd, pepsi-telemetry — has its table-level grant on crypto_identity revoked and replaced with a column-level grant listing every column except private_wrapped.

So a compromise of an unrelated stage — relay, bounce, aliases — yields the public half of every identity and nothing more, even with full database access under that role. The column list is read back from information_schema rather than hard-coded, so a column added by a later change is covered automatically and the grant cannot go quietly stale against the schema.

DELETE has no column granularity in PostgreSQL and stays whole: a stage that can delete an identity row can deny service, but it still cannot read a key.

Important

PostgreSQL peer authentication keys off the effective uid, not the gid. A setgid bit grants group membership — which is what the mode of the KEK fragment tests — but not a database identity. A program that must be the pepsi-crypto role has to be that account: run as it, or setuid to it. pepsi-keys does exactly that, assuming the identity for the duration of each database call and giving it back afterwards, in the same shape pepsi-whitelist uses for its own account.

Reaching a private key therefore takes both things: the database role and the key-encryption key. Neither alone is enough, and they are guarded by different mechanisms — a PostgreSQL grant and a file mode.

9.3.1. Who may manage what

An identity is keyed on the lower-cased address, and the passwd login that owns that address is recorded alongside it when the locality rules ([pepsi-crypto] LOCAL_DOMAINS, RECIPIENT_DELIMITER, TARGETS — the same options local delivery uses) resolve one. That column is the access rule: an unprivileged caller of pepsi-keys may see and change only identities whose login is their own, taken from the real uid’s passwd entry. An identity with no owning account — a role address, a hosted domain, an Exchange-fronted deployment — belongs to nobody in particular and is operator-only; treating “unowned” as “everybody’s” would make every role address writable by every local user.

Peer keys and trust anchors have no per-user namespace at all. A peer key is somebody else’s key, cached for the whole deployment, and the trust store is policy; both are operator-only, listing included.

pepsi-keys ships without a setuid bit, unlike pepsi-whitelist. One binary able to open every private key in a deployment is a far larger prize than a single whitelist table, so by default only the operator runs it. A site that wants users to manage their own identities installs it setuid pepsi-crypto; the ownership rule above is enforced regardless, so that is a site decision and not a code change.

9.4. Signing and encryption are separate

The store splits signing material from encryption material wherever the protocol allows it. The reason is an asymmetry that is easy to miss:

Pepsi delivers plaintext to the mailbox. A private decryption key is therefore needed only for mail still in flight and for whatever ciphertext an operator has archived — never to read one’s own mailbox. A private signing key, once retired, can do nothing legitimate whatever: no honest process ever re-signs old mail. Keeping it is pure forgery liability.

So retirement — at expiry, and identically at revocation — does two different things:

• a signing row’s private half is destroyed, leaving the public half so signatures made while the key was valid can still be verified;

• an encryption row’s private half is retained, so in-flight and archived ciphertext stays readable.

private_purged_at records when material was dropped, which keeps “deliberately destroyed” distinguishable from “never held” — an auditable state rather than an absence.

For OpenPGP the split is intrinsic and free: a transferable key is a sign-capable primary with an encryption subkey, so one row of purpose both represents it. For S/MIME it means two certificates, one with digitalSignature and one with keyEncipherment/keyAgreement. Both the self-signed and the certificate-request paths handle the pair naturally — identity generate creates both, and --csr emits a request over each.

9.4.1. Single-certificate S/MIME, and what it costs

Two certificates per identity is a real expense when certificates are bought from a CA per address. [pepsi] CRYPTO_SMIME_SHARED_KEY = yes therefore issues one certificate carrying both key usages instead. It is an administrator-only INI option — a security/cost trade, not a per-user preference, and structurally out of reach of the per-address override layer.

The trade is not hidden:

• the forgery-liability benefit is given up. A shared certificate’s private half follows the decryption rule at retirement, because destroying it would take the ability to decrypt with it; it leaves the store only through an explicit pepsi-keys identity forget-private;

• revocation becomes all-or-nothing. Revoking a compromised signing key also ends the ability to receive new encrypted mail at that certificate.

Shared mode is RSA only. pepsi-setup refuses it together with an elliptic-curve CRYPTO_SMIME_ALGORITHM rather than silently producing a certificate half the world will not accept: one EC key doing both ECDSA and ECDH is cross-algorithm key reuse that several S/MIME clients reject outright.

9.4.2. Capability, never shape

CRYPTO_SMIME_SHARED_KEY decides only what newly created identities look like. Toggling it never rewrites, re-keys or invalidates anything already issued, so at any moment the table may legitimately hold split identities, shared identities, and both for the same address — an old shared certificate still valid while new split ones are in use.

Every lookup therefore resolves by capability:

signing      ->  purpose IN ('sign',    'both')
encryption   ->  purpose IN ('encrypt', 'both')

and there is deliberately no query that returns “the identity” for an address. Code that assumed there were exactly two rows, or exactly one, would become a bug the moment an operator flipped the switch. When reading pepsi-keys identity list output, read the PURPOSE column the same way.

9.5. Creating identities

There are three ways material enters crypto_identity.

Generation. pepsi-keys identity generate creates the key pair here. OpenPGP defaults to Ed25519 ([pepsi] CRYPTO_OPENPGP_ALGORITHM), which is instantaneous to generate; RSA is available at 2048, 3072 or 4096 bits. S/MIME defaults to RSA at CRYPTO_GENERATE_RSA_BITS and can use NIST P-256 or P-384. Each generated certificate is self-signed, which is enough to start immediately, and --csr additionally emits a PKCS#10 request over the same key so a real CA can issue a replacement without re-keying.

Pepsi does not run a certificate authority of its own. That was considered and rejected: it would make Pepsi hold a key far more valuable than any single user’s, with its own custody and revocation problem.

Import. pepsi-keys identity import stores material generated elsewhere — the certificate a CA issued over an existing key, or a key pair a user already had. The declared --purpose is what every later lookup matches on, so it must describe what the material can actually do.

Automatic creation, lazily. [pepsi-crypto] AUTO_CREATE_IDENTITY (on by default) permits an identity to be generated for a served address without an operator asking. The trigger is deliberately lazy: creation happens the first time a local sender explicitly asks for protection — a marker in the subject, or the equivalent header from a mail client add-in — and never merely because an address appeared in an envelope. A user who never uses the feature therefore never has key material and is never exposed by it. Rows created this way are marked auto_created.

Warning

The one surprising consequence. The trigger is on the outbound path. An address that only ever receives mail therefore stays keyless, gets no Web Key Directory entry and publishes no DNS record — so outsiders have nothing to encrypt to it with, and never will, however long they wait.

An operator who wants inbound coverage must pre-provision those addresses:

pepsi-keys identity generate role@example.org --protocol openpgp

This is a design decision, not an oversight: eager creation for every address ever seen would mint key material — and publish it — for people who never asked for any.

A generated or imported identity is published by default (--no-publish opts out, and identity publish --off changes it later). A key nobody can fetch cannot be encrypted to, so publication is the useful default; the flag is the per-address escape hatch.

9.6. Publishing our users’ keys

The mirror image of “Finding a correspondent’s key” below: how other people find our users’ keys. There are three channels, and they behave so differently that treating them as one setting would be a mistake.

Channel	Who serves it	Reversible?	Turned on by
Web Key Directory	us (pepsi-httpd)	yes, immediately	the identity’s published flag — on by default
Key server	a third party	no	[pepsi-keys] VKS_PUBLISH — off by default
DNS (OPENPGPKEY / SMIMEA)	us (our zone)	yes, on the next zone push	publishing the record by hand

Every identity is published when it is created, however it was created. The opt-out is per address, not per identity, because that is the unit a user reasons about (“am I discoverable?”) — and because a per-identity opt-out permits a useless half-state in which the signing certificate is published and the encryption certificate is not, so correspondents can verify but cannot encrypt.

9.6.1. The Web Key Directory

pepsi-httpd answers the four Web Key Directory paths for every domain in [pepsi-ingress] ACCEPTED_DOMAINS:

GET /.well-known/openpgpkey/hu/<hash>              direct   (on <domain>)
GET /.well-known/openpgpkey/policy                 direct
GET /.well-known/openpgpkey/<domain>/hu/<hash>     advanced (on openpgpkey.<domain>)
GET /.well-known/openpgpkey/<domain>/policy        advanced

Each identity carries the WKD local-part hash of its address (z-base-32 of the SHA-1 of the lower-cased local part), stored and indexed together with the domain, so a request is answered with one lookup rather than by hashing every candidate row. The SHA-1 there is a naming function fixed by the specification, not a security one, so it is not gated by CRYPTO_ALLOW_WEAK_DIGESTS.

The response is the binary transferable public key — not armor — with Content-Type: application/octet-stream and Access-Control-Allow-Origin: *, which the specification requires so a browser-based client can fetch a key from a domain other than its own origin. The policy file is a zero-length 200: it carries no flags, but it has to exist, because GnuPG reads its absence as “this domain runs no Web Key Directory” and gives up before asking for a key.

The ?l=<local-part> parameter a client may send is read and thrown away. Honouring it would turn the endpoint into a lookup by caller-supplied address, which is a much broader thing than answering a hash the caller already had to know.

Deployment takes one DNS record and one certificate name per served domain. The advanced method — the one every current client tries first — is fetched from openpgpkey.<domain>, so that name must resolve to the pepsi-httpd host and the certificate it serves must cover it. Both are the same listener: openpgpkey.<domain> and the apex are two SNI names on one HTTPS listener, not two listeners. pepsi-setup run adds openpgpkey.<domain> to the names it asks certbot for and reminds you about any that does not resolve; a domain that already publishes keys is flagged, because for it the advanced lookup is failing today rather than merely unconfigured. The direct method on the apex works without any of this, so a missing openpgpkey host is a degradation, not a breakage.

Important

The endpoint serves our own published identities only. It reads crypto_identity, the table of addresses we hold key material for, and never peer_key, the cache of correspondents’ keys that discovery filled in. Serving those would make Pepsi a key server for material it never verified and does not vouch for, published under its own domain name. There is no configuration option that changes this.

Note

A Web Key Directory answers an unlimited number of guesses, and Pepsi does not pretend otherwise. There is no rate limiter on the endpoint and no uniform-404 shaping: a request for an address that publishes a key gets the key, and a request for one that does not gets a 404, promptly and as often as anybody cares to ask.

This is a decision, not an oversight. The protocol is a public oracle by construction — its whole purpose is that anybody may ask whether an address publishes a key — the hash covers only the local part, so it answers guesses rather than enumerating a list, and the addresses are on the outside of every message the domain sends anyway. Shaping the 404s would cost a client the ability to tell “no key” from “server broken” in order to buy an obstacle that a wordlist walks straight past. If you need the addresses to be secret, a key directory is the wrong thing to be running.

9.6.2. Key servers, and why they are an operator decision

keys.openpgp.org and the servers that speak its protocol accept a key permanently. They can later be told the key is revoked; they can never be told to forget it, and neither can the address it was uploaded for.

That is why uploading is gated on [pepsi-keys] VKS_PUBLISH, an operator-level switch that is off by default:

• not per identity, because the person who understands that an upload cannot be withdrawn is the operator, not whoever happened to generate a key — and because a per-user opt-in is a step essentially nobody takes, so making it per-user would mean key-server discoverability never materialises at all;

• not simply following published, because uploading a user’s address to a third party should be something an operator decided, never a side effect of a default.

With the switch off, a one-off upload is still available and prints a confirmation naming exactly what cannot be taken back:

pepsi-keys identity publish 42 --vks

Publication is a two-step protocol, and that is why there is a retry job. The upload returns a token; only the token authorises asking the server to mail a confirmation link to the address; and only after that link is followed does the server serve the key by address. A key that was uploaded but never confirmed is stored yet unfindable — the failure that looks like success. So:

pepsi-keys identity publish --retry      # from cron, e.g. hourly

works through every published OpenPGP identity that is not verified yet, spaced by VKS_RETRY_INTERVAL and giving up after VKS_MAX_ATTEMPTS with the reason left in the row for you to read (pepsi-keys identity show). Each round asks the key server whether the address is already served before uploading again, so a confirmation followed by anybody — the stage below, a human reading the mailbox, an earlier run whose database write did not land — ends the loop.

The confirmation mail itself is followed by pepsi-stage-vks-confirm, a small stage on the inbound path. It is a separate program because what it does — fetch a URL found in inbound mail — is a genuinely dangerous primitive, and isolating it means the restrictions are in one small place whose position in the pipeline is visible in the configuration. That program’s page lists them; the short version is that the mail must come from the configured key-server host, be authenticated by SPF or DMARC, link only to that host, and be addressed to somebody we are actually waiting for a confirmation for. Anything else is delivered to the user, who can click the link themselves.

9.6.3. Unpublishing, and what it cannot do

pepsi-keys identity unpublish 42

stops the Web Key Directory serving the identity at once, and clears its key-server bookkeeping so the retry job forgets it.

It does not remove anything from a key server, because nothing can. If the key was uploaded, the command says so and points at the only real remedy:

pepsi-keys identity revoke 42 --upload

which builds a revocation certificate — signed by the key it retires — and publishes it on top of the uploaded key. Note the ordering inside that command: the upload happens first, because revoking a signing identity destroys the private key the revocation certificate needs. If the upload fails, nothing is revoked, so you can fix the key server and try again.

9.6.4. DNS: OPENPGPKEY and SMIMEA

pepsi-keys dns <address> prints the DNS records that publish an address’s published, active identities: OPENPGPKEY (RFC 7929) for OpenPGP material and SMIMEA (RFC 8162) for S/MIME certificates, both under a hashed owner name so the zone does not enumerate the addresses it serves. pepsi-setup run prints the same records for every published identity, alongside the DKIM, SPF and MTA-STS ones (capped at a readable number — for a large deployment, ask per address).

S/MIME has no Web Key Directory equivalent and no key server, so SMIMEA is its publication channel.

Warning

Publish key records only in a DNSSEC-signed zone. An unsigned OPENPGPKEY record is a key handed out by whoever can answer for the zone, which is not an improvement on having no key at all — and unlike a WKD answer, a DNS answer carries no other evidence. Both commands repeat this as a comment in their own output.

The DNS owner name and the WKD hash are not interchangeable: the DNS one is SHA-256 truncated to 28 octets, hex-encoded, over the case-sensitive local part (RFC 7929 §3), while the WKD one is z-base-32 of SHA-1 over the lower-cased local part. Confusing them produces records nothing ever queries, which is why they are separate functions with a test asserting they differ.

9.7. Backing up and rotating the key-encryption key

Danger

If the key-encryption key is lost, every stored private key is gone. Back up secrets.d/pepsi-crypto.secret separately from the database, and treat it with the care its contents deserve.

The blow is softened by the same asymmetry that governs retirement: mailboxes hold plaintext, so losing the KEK does not make anybody’s mail unreadable. What is lost is any ciphertext still in flight or archived — and, far more disruptively, every published identity is invalidated at once. Correspondents still hold public keys the deployment can no longer use, so the whole organisation has to re-key and re-publish. Plan for that as a disaster-recovery scenario, not as an inconvenience.

Rotation, by contrast, is routine and incremental by construction, because each row records the id of the key that sealed it:

1. put the new secret in secrets.d/pepsi-crypto.secret as KEY_WRAP_SECRET, and keep the old one beside it as KEY_WRAP_SECRET_<OLD-ID> (the suffix is the old wrap_key_id, upper-cased);

2. point [pepsi-crypto] KEY_WRAP_KEY_ID at the new id;

3. run pepsi-keys wrap rotate.

Between steps 2 and 3 the deployment keeps working: new material is sealed under the new key while old material still opens under the retired one. Only after step 3 reports everything rotated may the retired secret be removed. Each re-wrap is a guarded update keyed on the old id, so a concurrent rotation — or a key re-generated while the sweep ran — is never written over; rows whose key has no configured secret are reported and skipped rather than failing the run. --dry-run lists the work first.

9.8. Peer keys

A peer key is cached with the source it came from, and the source decides what happens when a newly discovered key contradicts a stored one. The order of those sources is the trust ladder tabulated in Finding a correspondent’s key below; it lives in a database function so it can be re-ordered by re-running the procedures file rather than by a schema patch, and the conflict rule is applied server-side, in the same statement that stores the key, so no client can skip it.

A key that outranks everything stored for an address replaces it; at equal or lower rank the stored key is kept and the new one is not inserted, so a weak source can never quietly displace a better-attested one. A pinned row is never displaced, whatever the rank — pinning is how an operator makes a trust-on-first-use decision permanent.

Two rules bound how broad an entry can be:

• an entry may name a whole domain (*@domain), but only when it was entered by hand. That is enforced by a database constraint, not by convention, so no discovery mechanism can generalise one address’s answer to a domain;

• an exact address match always wins. A *@domain row is consulted only when no exact row exists, and pepsi-keys peer show marks it as the fallback it is.

9.9. Finding a correspondent’s key

Everything above is about keys once they are in the store. Getting them there for someone else’s address is key discovery: given an e-mail address, find the public key or certificate, record where it came from and how much that source is worth, and cache it. The program is pepsi-keydisc, and the whole feature is configured by [pepsi-keydiscovery].

Finding a key is easy. Knowing whether to believe it is the entire problem, which is why the source and the DNSSEC flag are stored with every key and why this section spends most of its length on ranking rather than on protocols.

9.9.1. Where a key can come from

dane

DNS OPENPGPKEY (RFC 7929) and SMIMEA (RFC 8162), under a hashed local part. Refused without the resolver’s DNSSEC AD bit — see When there is no validating resolver.

wkd-advanced

The Web Key Directory at openpgpkey.<domain> — a host the domain had to delegate deliberately.

wkd-direct

The Web Key Directory at the domain itself, <domain>/.well-known/openpgpkey.

ldap

A configured directory. Optional, and compiled in only with the ldap cargo feature, because a deployment with no directory should not link one.

vks

A verifying key server (keys.openpgp.org by default, or any server speaking the same paths). Note that this is VKS, not the older unverified HKP: an HKP server will serve a key anybody uploaded for anybody’s address.

inbound

Harvested from a message that already arrived — an application/pgp-keys part, an S/MIME signer certificate, or an Autocrypt: header. Only material claiming the sender’s own address is kept. Not a discovery service: it runs where the message already is and costs no network I/O, which is why it may not appear in SOURCES.

gossip

Read out of the Autocrypt-Gossip: fields inside a message pepsi-stage-decrypt decrypted: the other To:/Cc: recipients’ keys, which the sender attached so that a reply-all can be encrypted to somebody one has never had direct mail from (Autocrypt Level 1 §5.3). The only route by which one correspondent may speak about another, and hence its own rung at the bottom of the ladder. Like inbound it is not a service and may not appear in SOURCES; see pepsi-stage-decrypt for the five rules that decide whether a field counts.

manual / api

An operator running pepsi-keys peer import, or an authenticated submission. Not discovered at all; listed here because it is the top of the same ladder.

9.9.2. The trust ladder

This is the order everything is judged by — best first. It decides which key wins a conflict, what MIN_TRUST compares against, and how much extra time a still-running method is granted by the stop rule below.

Rank	Source	Why there
0	own	Not discovered at all, and not really a rung: the public half of an identity this deployment holds for the address. It pre-empts the ladder rather than topping it — see Our own users’ keys are not discovered — so it clears every floor and no peer_key row for that address is consulted.
1	manual / api	An operator who typed a fingerprint in meant it. Also the only source permitted to enter a *@domain row.
2	dane with the AD bit	The only source with cryptographic provenance: the domain published it and DNSSEC proves the answer was not rewritten in transit.
3	wkd (advanced)	Fetched over verified HTTPS from openpgpkey.<domain>, a host the domain set up on purpose.
4	wkd-direct	The same, from the domain itself. A weaker signal of intent than a dedicated host, hence its own rank.
5	ldap	A directory the operator configured, and therefore semi-trusted — but still a directory somebody else runs.
6	vks	Proves only that somebody controlling that mailbox uploaded it: the server sent a confirmation mail and somebody clicked it. A real check, and a much weaker one than “the domain publishes this”.
7	inbound (harvested, Autocrypt)	Trust on first use, of what the address’s owner said about their own address. Also where an unvalidated DANE answer lands: a DNS record nobody signed is worth no more than a scrape.
8	gossip (Autocrypt-Gossip)	One correspondent’s introduction of another (Autocrypt Level 1 §5.3), read out of the ciphertext of a message this host decrypted. The same material as rank 7 and a different claim: not “this is my key” but “this is somebody else’s”. §5.3 keeps the two in separate peer states precisely so that a third party’s introduction can never displace what an owner vouched for, and that is what a rung of its own buys.

[pepsi-keydiscovery] MIN_TRUST is the floor, and it defaults to any — the bottom rung, whatever the bottom currently is, so gossiped, harvested and Autocrypt keys are all accepted. That is a decision, not laziness: the alternative to encrypting to a trust-on-first-use key is not encrypting to a better key, it is sending cleartext. Opportunistic encryption to an unauthenticated key still defeats passive interception, which is the threat most mail actually faces. The ranking is not thereby pointless — it arbitrates conflicts, it is recorded per key, and an operator who would rather send nothing than send to an unauthenticated key has one option to raise.

The default tracks the bottom rather than naming a number, and the two Autocrypt-shaped spellings are where that pays off. MIN_TRUST = harvested names rank 7 exactly, so it accepts what a correspondent advertised about themselves and refuses a stranger’s introduction of them — a genuinely useful middle setting, and one that did not exist while rank 7 was the bottom. The introductions are still stored either way; the floor decides only whether they may be encrypted to.

9.9.3. Our own users’ keys are not discovered

There is one address for which the whole ladder is the wrong question: one this deployment made the keys for. If pepsi-keys issued an identity for alice@example.org, then Alice’s key is not something to be found, weighed and compared — it is on the shelf, and it is the only key her mail can be encrypted to that she will be able to open.

That matters because peer_key is not a trusted table. Part of what fills it is harvesting from inbound mail, and harvesting keys on the From: header, which the sender writes. Inbound authentication is deliberately fail-open — only a definite DMARC failure under DMARC_ENFORCE rejects — so on a deployment that does not enforce DMARC on its own domain, a message claiming From: alice@example.org and carrying an Autocrypt: header will seed a peer key for Alice with a stranger’s key in it. Two things then go wrong, and they are the reason this is a defence and not a tidiness rule: mail from one of our users to another would be encrypted to the planted key, and a signature made with it would verify, setting state.signature_verified for an outsider.

So both crypto stages ask “is this one of ours” before they ask “what did discovery find”. When the answer is yes, our own key is the answer: the correspondent’s peer_key rows are not consulted for that address at all, whatever their rank and however they arrived. state.crypto records it as own.

Important

The rule is conditional on holding a key, not on serving the address, and the difference is a deployment rather than a technicality. A user who runs GnuPG in their own client, has never asked Pepsi to encrypt anything and has never uploaded a key to us has no identity here — and their real key is one this host can only ever learn from discovery, harvesting or gossip. A blanket “never believe a discovered key for an address we serve” rule would send that user cleartext for ever. So: if we made the keys, they are the ones that get used; if we did not, the address is treated like any other correspondent’s, trust-on-first-use exposure included.

The corollary is the cost, and it is worth stating plainly: for an address we do hold an identity for, a signature made with the user’s own separate key no longer verifies, however well-attested that key is. An operator who wants that key believed should import it (pepsi-keys identity import), which is also how the user starts receiving mail encrypted to it.

9.9.3.1. Finding the users who bring their own key

The rule above leaves a residue by design: a served address we hold no identity for is still trust-on-first-use, because that is the only channel by which a bring-your-own-GnuPG user’s real key can reach this host. What an operator can do about it is not a setting — it is an introduction. Import that user’s public key as an identity and the exposure is over for that address: from then on our own row pre-empts whatever the cache holds.

So the deployment needs a way to notice them, and that is GET /api/v1/peers/unclaimed (see The administrative API): the addresses that are

• served by this deployment (their domain is one of [pepsi-ingress] ACCEPTED_DOMAINS),

• held by no active encryption identity of ours, and

• nevertheless present in peer_key — usually harvested from their own outbound mail’s Autocrypt: header, which is exactly what a user running GnuPG in their own client produces.

Each row names the cached fingerprints, their source and their rung on the ladder, so an operator can tell a key the user themselves advertised from a third party’s gossiped introduction before acting on it. The report is read-only and changes nothing; it is a worklist, not an alarm, and an empty one is the ordinary state of a deployment whose users all hold identities here. The console shows the same list under Correspondent keys (/ui/peers).

Note

Ask the user before importing. The key in the cache is one somebody advertised; confirming out of band that it is really theirs is what turns a trust-on-first-use row into a trusted one, and is the entire value of the exercise.

Which lifecycle states count differs between the two uses, for the reason expiry exists:

• Encrypting to the address counts active identities only. A retired key must not receive new mail, and a deployment that retired a user’s key without issuing another is in the same position as one that never held a key for them — so discovery takes over.

• Verifying a signature from the address counts everything but revoked. A signature made last year was made with last year’s key; refusing to look at it would report our own users’ mail as unverifiable every time a key rolled over. revoked is excluded because that is the operator saying the material must not be used at all.

9.9.4. The pipeline never blocks on a key server

Discovery is asynchronous and outside the stages. A stage that needs a key it does not have does no network I/O at all: it commits whatever it has already produced, pauses the message, and enqueues a request keyed on the address. A pepsi-keydisc instance answers the request and, in the same round-trip that stores the key, releases every message parked on that address.

Three consequences follow, and each is the point of the design rather than a side effect:

• A slow key server cannot hold a stage worker. That problem is removed rather than bounded — there is no worker waiting on it to bound.

• A hundred messages to one correspondent cost one lookup. The request is keyed on the address, so they deduplicate onto it and are released together.

• A request that settles without finding a key still releases its waiters. The services find keys; the stages decide about mail. Nothing in the discovery layer ever decides a message’s fate.

The methods run in parallel: every method in SOURCES starts at once, each as its own process (pepsi-keydisc@dane, @wkd-advanced, @wkd-direct, @vks, @ldap). The two Web Key Directory methods are separate ranks and therefore genuinely concurrent, not a fallback chain — running direct only “if advanced fails” would silently accept the weaker answer whenever the stronger one was merely slow. One process per method also means a hung LDAP bind cannot stall WKD, and that turning a method off is systemctl disable pepsi-keydisc@ldap.

Keep SOURCES and the enabled units in step. SOURCES is the roster the stop rule waits for, so an instance whose method it does not list refuses to start, and a method it lists with nothing running costs every request the full TIMEOUT before it gives up.

9.9.5. RANK_GRACE: the speed-versus-thoroughness dial

Running everything at once raises the only interesting question in the whole fan-out: an answer has arrived from a mediocre source, and a better one is still running. Wait, or go?

RANK_GRACE is the single knob that answers it. When a lookup succeeds, every still-running method that ranks better is granted

RANK_GRACE × (how many ranks better it is)

of additional time, measured from the moment the answer in hand arrived. When that window expires, the best result in hand wins. The knob spans the whole range of reasonable policies:

Setting	Behaviour
0	The first usable answer wins outright. Fastest; the ranking then only arbitrates conflicts in the store, never the race.
5 s (default)	One rank up gets 5 s, three ranks up gets 15 s. A slow WKD-advanced server cannot hold a message behind a key-server hit that already arrived, but a nearly-as-fast better source still gets to win.
≥ TIMEOUT	Every method is waited for and the ranking applies in full. Slowest and most thorough; pepsi-setup notes this configuration rather than objecting to it, because it is a legitimate choice.

Two other deadlines bound the same fan-out: METHOD_TIMEOUT (15 s) is one method’s own network budget, and TIMEOUT (60 s) ends the whole request whatever is still running. A request that ends with nothing is not an error; it is a correspondent with no key.

9.9.6. The negative cache, and why it matters more than it sounds

When a request settles without a key, that outcome is remembered: for NEGATIVE_TTL (24 h) after an authoritative “there is no key here”, and for the much shorter ERROR_TTL (1 h) after a network failure — “the server was down” is worth re-trying sooner than “there is no key”.

The rule that makes this worth its own paragraph:

A message for an address with a fresh negative entry never pauses. It takes the stage’s no-key path immediately.

Without that, every message to a correspondent who simply has no key would pause for the full TIMEOUT to re-learn an answer already known — turning one absent key into a permanent per-message delay for that whole correspondence. The accepted cost is the other direction: a key published five minutes ago stays invisible until the entry ages out. Lower NEGATIVE_TTL if that trade is wrong for your deployment; that is exactly what the option is for.

The same short-circuit fires when the deadline of an outstanding request has passed with nobody answering it — the signature of a deployment running no discovery service at all. The message takes the no-key path rather than parking again, which is what stops it parking for ever.

9.9.7. Lookup privacy

Looking an address up tells somebody that you are about to write to it. Which somebody depends on the method: DNS tells the domain’s authoritative servers and your resolver’s path, WKD tells the correspondent’s own domain, and a key server tells its operator.

Two controls exist, and they are not interchangeable:

[pepsi-keydiscovery] ALLOW_DOMAINS / DENY_DOMAINS

The operator’s, and the recipient-side one. A non-empty ALLOW_DOMAINS is exclusive — only those domains are ever looked up — and DENY_DOMAINS always wins over it. This is the control to reach for when the question is “never query that domain”.

[stage-encrypt] DISCOVERY = no

The per-user one, overridable per address through the pepsi.settings layer. Cached keys are still used; a cache miss goes straight to the no-key path instead of parking.

Important

State the limit plainly, because the option’s name invites the wrong reading. The per-address settings layer resolves an outbound message on its envelope sender. So setting DISCOVERY = no for an address means

• “this user’s mail never triggers a lookup” —

and not

• “never look this correspondent up”.

There is deliberately no per-address recipient denylist: it would need a new column and a second list to stay consistent with DENY_DOMAINS, and two overlapping denylists that can disagree are worse than one that cannot. The recipient-side control is the operator’s, above.

9.9.8. When there is no validating resolver

DANE discovery is AD-gated: an OPENPGPKEY or SMIMEA answer the resolver did not DNSSEC-validate is refused outright. That is not pedantry and it is not the same policy as TLS Reporting, where a forged rua costs an attacker a copy of a report. An unvalidated key record is a key from whoever can answer for the zone, which is not an improvement on having no key at all — so an unvalidated answer is not a weaker answer, it is no answer.

Pepsi trusts a validating resolver rather than validating DNSSEC itself, the same operational model its DANE/TLSA support uses for transport security. The consequence is worth stating bluntly:

Warning

A deployment without a validating resolver gets no DANE results at all — silently, because the method dutifully answers “no key” for every address, and the best-ranked source simply never wins. pepsi-setup probes for this and reports it, but the fix is yours: point Pepsi at a validating resolver (unbound, or systemd-resolved with DNSSEC=yes over a trusted path), or drop dane from SOURCES so nothing waits for it.

9.9.9. Inbound mail waits too

Verification pauses on the same mechanism as encryption: when an inbound signature is from a signer whose key is not cached, the message parks while discovery runs, so the first message from a new correspondent gets a real verdict instead of unverifiable.

That buys a real property and costs a real delay, on exactly the mail somebody is waiting for. Three things blunt it, and the first is the one to reach for:

• INBOUND_TIMEOUT is separately tunable. An operator who feels this delay can drop it to a few seconds without touching the outbound TIMEOUT at all. pepsi-setup warns above five minutes.

• The negative cache means the second and later messages from an unknown signer never pause — only the first pays.

• Requests deduplicate per address, so a flood from one signer costs one lookup rather than one per message.

Whether the parked row holds plaintext depends on the protocol, and the difference is not cosmetic. Under the layering rules an inbound signature is inside the ciphertext, so the decryption has to happen before the signer — and therefore the missing key — is even known. What may then be committed differs:

• An S/MIME signature nested inside an envelope survives decryption as an ordinary MIME layer, so the message is parked with the plaintext committed and is decrypted exactly once.

• An OpenPGP signature lives only inside the packet stream and is gone the moment the plaintext is out. Committing the plaintext would destroy the very evidence the key is being fetched to check, so such a message is parked unchanged and decrypted again on resume. The second decryption is the price of a correct verdict.

Note

One exposure to know about. A message parked with its plaintext committed — the S/MIME case above — sits in pepsi.ingress decrypted for up to INBOUND_TIMEOUT.

This is the same exposure delivery would create a moment later anyway: the mailbox gets plaintext. It is a longer window, not a new one. If that window matters to you, it is INBOUND_TIMEOUT that sets its length. See pepsi-stage-decrypt for the rule as the stage applies it.

9.9.10. Who uses the store

The discovery layer is complete: the sources, the ranking, the request queue and the negative cache all work, and pepsi-keys peer refresh --address exercises the whole fan-out end to end today.

Both consumers exist. pepsi-stage-encrypt signs locally submitted mail as its From: author and encrypts it per recipient, parking a message whose recipient key is not cached until a discovery service settles the request. pepsi-stage-decrypt is its inbound counterpart: it opens mail addressed to recipients this host serves — trying every non-revoked identity of theirs, including expired ones, because old mail must stay readable — verifies the signature against the peer_key cache and the ca_trust anchors, and harvests the sender’s key out of the message itself. It parks on the signer’s key the same way, so even the first message from a new correspondent gets a real verdict rather than unverifiable.

What is not here yet is re-encryption for storage: a decrypted message is filed in the mailbox as plaintext, which is what makes server-side decryption useful at all — the gateway model already assumes the mail store is trusted. Re-encrypting to the user’s own client key would need a second key per user, a way to learn it and a policy for when it is missing; that is a feature in its own right and is recorded as deferred in docs/ROADMAP.md.

9.10. A CLI walk-through

Give an address an OpenPGP identity, look at it, and check that GnuPG agrees the public half is a key:

pepsi-keys identity generate alice@example.org \
    --protocol openpgp --name 'Alice Example'
pepsi-keys identity list --address alice@example.org
pepsi-keys identity show 1
pepsi-keys identity export 1 | gpg --import

Give the same address S/MIME material — a signing and an encryption certificate, each self-signed for immediate use, with a certificate request over each key to send to a CA:

pepsi-keys identity generate alice@example.org --protocol smime --csr

When the CA answers, store its certificate over the key that is already in the store and make it the primary signing material. Nothing is re-keyed and nothing already sent stops being readable:

pepsi-keys identity import alice@example.org \
    --protocol smime --purpose sign --public alice-sign.pem --primary

Publish the address in a signed zone:

pepsi-keys dns alice@example.org >> example.org.zone

Withdraw a signing key after a laptop is stolen. The private half is destroyed at once; the public half stays so last month’s signatures still verify:

pepsi-keys identity revoke 1 --reason 'laptop stolen'
pepsi-keys identity show 1        # private material: purged at …

Retire everything that has passed its validity — the same rule, applied in bulk, and the natural body of a periodic timer:

pepsi-keys identity retire

Trust a correspondent’s key by hand and pin it, so a later discovery answer cannot replace it:

pepsi-keys peer import bob@example.com --protocol openpgp --file bob.pgp --pin
pepsi-keys peer show bob@example.com

Decide which CAs an inbound S/MIME signature may chain to:

pepsi-keys ca add /etc/ssl/certs/our-ca.pem --label 'Our CA'
pepsi-keys ca list

And, once a year or after an incident, roll the key-encryption key:

pepsi-keys wrap rotate --dry-run
pepsi-keys wrap rotate

9.11. Configuration summary

The algorithm and security policy live in [pepsi] and are administrator-only: they are security decisions with secure defaults, not preferences, and because the per-address override layer only overrides [stage-<name>] sections they are structurally out of reach of pepsi-stage-edit-settings and any web-admin surface.

Those options are CRYPTO_ALLOW_DOWNGRADE, CRYPTO_ALLOW_WEAK_DIGESTS, CRYPTO_MIN_RSA_BITS, CRYPTO_GENERATE_RSA_BITS, CRYPTO_INLINE_PGP, CRYPTO_OPENPGP_ALGORITHM, CRYPTO_SMIME_ALGORITHM and CRYPTO_SMIME_SHARED_KEY.

The key store’s own custody and lifecycle settings live in [pepsi-crypto]: KEY_WRAP_SECRET and KEY_WRAP_KEY_ID (plus KEY_WRAP_SECRET_<ID> for a retired key), AUTO_CREATE_IDENTITY, IDENTITY_VALIDITY_DAYS (730 by default), and the locality options LOCAL_DOMAINS / RECIPIENT_DELIMITER / TARGETS that decide which account an address belongs to. A deployment that does no end-to-end cryptography can omit the section entirely; as soon as it says anything, pepsi-setup insists on a usable KEY_WRAP_SECRET, since a key store that cannot store keys is a configuration mistake rather than a preference. The exhaustive reference is pepsi.conf(5).

Key-server publication has its own section, [pepsi-keys]: VKS_PUBLISH (off by default — see above for why it is an operator switch), VKS_SERVER (defaulting to the first [pepsi-keydiscovery] VKS_SERVERS entry, so a deployment that already chose a key server to read from does not name it twice), and the retry knobs VKS_MAX_ATTEMPTS, VKS_RETRY_INTERVAL and VKS_BATCH. Web Key Directory publication needs no configuration at all: it follows each identity’s published flag.

9.12. See also

The two stages that use this store are pepsi-stage-encrypt (outbound) and pepsi-stage-decrypt (inbound); The secure-link fallback portal is what happens when a correspondent has no key at all, and Client interoperability records which third-party clients have actually been measured against what those stages emit. pepsi-keydisc is the discovery service, pepsi-keys the operator CLI and pepsi-stage-vks-confirm the key-server confirmation follow-up. RFC Index maps each standard named here to the module that realises it, and Microsoft Exchange as a gateway is the deployment shape this store was built for.

pepsi-keys, pepsi-setup, Architecture, Configuration, pepsi-keys(1), pepsi.conf(5), pepsi-setup(1).