85.1.10. pepsi-stage-relay-to-internet

deliver a message directly to the recipient’s MX hosts

Manual section:

1

85.1.10.1.1. Name

pepsi-stage-relay-to-internet - the direct-to-MX delivery stage of the Pepsi pipeline.

85.1.10.1.2. Synopsis

pepsi-stage-relay-to-internet [GLOBAL-OPTIONS] worker

pepsi-stage-relay-to-internet [GLOBAL-OPTIONS] mta-sts DOMAIN

85.1.10.1.3. Description

pepsi-stage-relay-to-internet is a stage program run by pepsi-dispatch(1) as a persistent worker reading message ids on standard input. It loads that pepsi.workqueue row (refusing to act unless its status is running), reads its [stage-<stage>] section, and delivers the message directly to the recipient domain’s mail exchangers, using the message’s own envelope sender (the null sender <> for bounces). Delivery is one recipient per attempt: a row that still holds several recipients is first split into one row per recipient (each with its slice of state.dsn), all of which re-enter this stage. A message carrying more than MAX_HOP_COUNT Received: header fields is treated as a mail loop and fails permanently.

It performs its own DNS MX lookup: candidate hosts are tried in ascending MX preference (equal-preference hosts in random order), a domain with no MX records falls back to its address records (the implicit MX of RFC 5321 §5.1), and a non-existent domain or RFC 7505 null MX is a permanent failure — the null MX reported as 556 with the enhanced status 5.1.10 (RFC 7505 §4.1), so the bounce says that the address can never be deliverable rather than looking like any other 550. Each MX host’s A/AAAA addresses are selected with Happy-Eyeballs and cached, per address, in the dns_address table; the ADDRESS_FAMILY setting (intersected with the families the host can route) chooses which to use. Connections are made on port 25. Every query is made for the fully-qualified name, so a search list in /etc/resolv.conf is never applied: without that, a domain with no MX records would be looked up a second time under each search domain, and the resulting NXDOMAIN — not the domain’s own NODATA — would be reported, turning an ordinary implicit-MX delivery into a permanent failure.

Unless disabled with MTA_STS = no, the recipient domain’s MTA-STS policy (RFC 8461) is honoured: an enforce policy restricts delivery to the listed MX hosts over STARTTLS with a certificate that validates for the host, failing the message (transient) when no MX matches. A testing policy is validated the same way — the attempt and its outcome are recorded for TLS Reporting — but never blocks delivery: a transport-security failure (or a policy that matches no MX) falls back to opportunistic STARTTLS over all MX hosts, so the domain owner learns from the reports whether enforce would break mail before switching to it. Domains with no policy use opportunistic STARTTLS. Policy lookups fail open only when nothing is cached.

Policies are cached for their max_age (capped at one week). The _mta-sts TXT id is re-checked on each use of a cached policy (RFC 8461 §3.1): a changed id triggers an immediate refetch, while a transiently unreachable record (or a failed refetch of a changed policy) keeps the still-valid cached policy. A policy host that returns a transient 5xx/429 is treated as “fetch failed” (fail open, not cached as “no policy”), whereas a 404 or a non-text/plain body is a definitive absence.

DANE (RFC 7672): when DANE is warn (the default) or strict and the destination MX host publishes DNSSEC-validated TLSA records, the MX certificate is authenticated against them instead of by PKIX — this takes precedence over MTA-STS for that connection. DANE rests on DNSSEC: the configured resolver (DNS_SERVERS, or resolv.conf) must validate and be reached over a trusted path, because Pepsi trusts the response AD (Authentic Data) bit rather than validating itself. TLSA (and MX) answers without the AD bit are treated as “no DANE”. In strict mode a usable-but-mismatching record (or a failed TLSA or MX-security lookup, e.g. for a DNSSEC-bogus record) defers the message as a transient failure; in warn mode the mismatch is logged and delivery proceeds. That precedence holds only where DANE is at least as strong as what it replaces: for a host covered by an enforce MTA-STS policy, warn mode is not used (a warn-mode mismatch accepts any certificate, which would discard the policy’s RFC 8461 §5 validation requirement) — the certificate is validated by PKIX and the substitution is logged. strict mode still takes precedence there. A domain that simply publishes no TLSA records is delivered normally even in strict mode (DANE is opportunistic — it only bites when records exist). A domain that publishes TLSA records of which none is usable (a PKIX usage, or a matching type this build does not implement) is not the same thing: per RFC 7672 §2.2 it has still committed the host to TLS, so STARTTLS becomes mandatory for it — unauthenticated, but never cleartext — and a hop that does not offer it defers the message.

SMTP TLS Reporting (RFC 8460): when [pepsi-tlsrpt] SEND_REPORTS is set, the outcome of each TLS session — a success, or a classified failure (starttls-not-supported, certificate/handshake or DANE/MTA-STS validation) — is recorded as an aggregate counter in pepsi.tls_session, keyed by the applied policy (tlsa/sts/no-policy-found) and MX host. pepsi-tlsrpt(1) later compiles these into daily reports. Recording is best-effort (a database hiccup never fails a delivery) and is skipped for a report message itself (state.tlsrpt).

On success the row is removed (or, if the stage sets NEXT_STAGE, advanced). A transient failure pauses the message with an exponential backoff (re-queued by pepsi-dispatch(1) when its timeout elapses); past MAX_LIFETIME the failure is treated as permanent. A permanent failure of an ordinary message is routed to the stage’s BOUNCE_STAGE (or, with none configured, the row is marked failed). A permanent failure of a bounce (null sender) is never bounced again: a copy is delivered to POSTMASTER if configured, otherwise it is discarded, and the row is then removed.

DSN (RFC 3461): when the recipient’s MX advertises the DSN extension, the parameters the origin requested (RET/ENVID and the recipient’s NOTIFY/ORCPT, read from the row’s state.dsn) are propagated to it; otherwise they are omitted. On a permanent failure the recipient’s NOTIFY/ORCPT and the ENVID are handed to the bounce stage so a failure DSN is generated only when requested. This stage preserves state.dsn.

When the global [pepsi] ORIGINATE_SUCCESS_DSN is enabled and a successful delivery’s recipient requested NOTIFY=SUCCESS, the message is routed (after delivery) to BOUNCE_STAGE, which emits a positive (Action: delivered) report to the sender. With the flag off, or without a SUCCESS request, a successful delivery is simply finished.

When DELAY_DSN_AFTER is set and a still-undelivered message has been queued at least that long, a one-shot “delayed” (Action: delayed) DSN is sent to the sender — but only if the sender requested NOTIFY=DELAY (delay has no implicit default). The original message keeps being retried; the warning is a separate message routed through BOUNCE_STAGE, and is sent at most once.

Proof of origin: when the shared [pepsi-origin] section is configured (on by default — pepsi-setup(1) generates a secret if none is set), each outbound message is stamped with a Pepsi-Origin header before delivery — an HMAC over the originating sender account, a fresh 128-bit nonce, a timestamp and our HOSTNAME — and the nonce is recorded in pepsi.origin_nonce (valid for two weeks). The same stamping is shared with pepsi-stage-relay-to-smarthost(1) so a message takes a proof whichever outbound path it uses. A returning bounce embeds this header, letting pepsi-stage-anti-spam(1) confirm the bounce is a reply to mail we actually sent. The header is prepended like the Received: trace header (above any existing signatures, so it disturbs none) and is left unsigned. A nonce that cannot be recorded (a database error) holds the message back — it is retried like any host fault, see pepsi-dispatch(1) — because a header whose nonce is not on record would make every bounce to the message look forged. Without the [pepsi-origin] section no header is added; a section whose secret cannot be read stops the stage’s workers from starting (the queue is held, and the journal says why), exactly as pepsi-setup(1) refuses it.

Content downgrading (RFC 6152 / RFC 6531): the message is matched to the extensions the MX actually advertises rather than sent verbatim. When the body contains 8-bit octets and the MX supports 8BITMIME the envelope carries BODY=8BITMIME; when it does not, the MIME tree is walked and each 8-bit leaf part is re-encoded with a 7-bit content-transfer-encoding (quoted-printable for text/*, base64 otherwise). What a part declares is not taken as proof of what it holds: a part labelled base64 that carries raw 8-bit octets is re-encoded too, and a multipart/* whose declared boundary= occurs nowhere in its body has no parts to preserve and is encoded whole. The result is then re-tested, because the walk is best-effort over a message nobody validated (a tree nested past the depth cap is copied through unchanged): a body still carrying 8-bit octets is a permanent failure (554) rather than something to put on the wire, which RFC 6152 §3 forbids “under any circumstances”.

When the message needs SMTPUTF8 (UTF-8 in the headers or envelope) and the MX supports it the envelope carries SMTPUTF8; when it does not, UTF-8 header fields are rewritten as RFC 2047 encoded-words — in the three places RFC 2047 §5 allows one, so the field still parses: an unstructured field body, a display-name phrase (a quoted-string becomes one encoded-word, without the quotes) and the inside of a comment (the parentheses stay). A non-ASCII Content-Type or Content-Disposition parameter, for which §5 has no encoded-word, takes the RFC 2231 extended form instead (filename*=UTF-8''caf%C3%A9.txt, split into *0*/*1* sections when one line would not hold it); ASCII parameters are copied as written. This covers every MIME part’s header block, not only the message’s, since that is where an attachment’s filename sits. A non-ASCII boundary= has no encoded form at all — it must match the body’s delimiter lines octet for octet, and RFC 2046 allows it only 7-bit characters — so the multipart gets a fresh random ASCII boundary and its delimiter lines are rewritten to match, rather than the message being bounced. A part header that cannot be converted (say, raw Latin-1 octets) is copied as it stands and left to the 8BITMIME re-test, the transport seeing it as body content. A non-ASCII envelope address, and a non-ASCII octet inside a header address, cannot be downgraded — such a message is a permanent failure (bounced) towards a non-SMTPUTF8 hop. Re-encoding a body necessarily breaks any body-covering signature already on the message (the originator’s DKIM body hash and the ARC-Message-Signature from pepsi-stage-arc(1)); this is unavoidable, since the MX’s capabilities are only known after MX selection, and rare in practice.

85.1.10.1.4. Configuration

Pipeline wiring and every operational option live in the stage’s own [stage-<name>] section (PROGRAM = pepsi-stage-relay-to-internet): the SERVER_NAME/POSTMASTER identities, the connection timeouts, the retry-backoff schedule and MAX_LIFETIME, DELAY_DSN_AFTER, MAX_HOP_COUNT, the DNS settings, MTA_STS/MTA_STS_TIMEOUT, ADDRESS_FAMILY and DANE. They are documented in pepsi.conf(5).

85.1.10.1.5. State

Inputs (read from the row’s state; all optional):

  • state.dsn — the RFC 3461 parameters (message-level ret/envid and the first recipient’s notify/orcpt); propagated to the MX when it advertises DSN and consulted to gate failure/success/delay reports.

  • state.origin — the 8BITMIME/SMTPUTF8 hints (body_8bit, smtputf8) used by the content-downgrading decision.

Outputs (merged into the row’s state):

  • On a transient failure (pause): attempts, last_error and, once a delay DSN has been emitted, delay_sent.

  • On a permanent failure with a BOUNCE_STAGE (reroute), or on a delay warning (an enqueued clone): a state.bounce object (kind = permanent or delay) for pepsi-stage-bounce(1). For an SMTP-level failure it also records the structured next-hop detail — remote_mta (the recipient’s domain), smtp_code, enhanced_status, phase and reply_text — so the bounce can state precisely why delivery was refused.

  • On a successful delivery with ORIGINATE_SUCCESS_DSN and NOTIFY=SUCCESS: a state.bounce object with kind = success.

  • On a permanent failure with no BOUNCE_STAGE (fail): last_error.

state.dsn and state.origin are always preserved. The state layout is described in full in pepsi.state(7).

Transitions (the retry schedule is RETRY_INITIAL / RETRY_FACTOR / RETRY_MAX_INTERVAL, bounded by MAX_LIFETIME):

  • successful delivery → finish (the row is deleted), or advance to NEXT_STAGE if one is set; with [pepsi] ORIGINATE_SUCCESS_DSN and a recipient that asked for NOTIFY=SUCCESS it instead reroutes to BOUNCE_STAGE to emit a positive (Action: delivered) DSN;

  • transient failure → pause for the next retry interval;

  • still queued past DELAY_DSN_AFTER with a NOTIFY=DELAY recipient → enqueue a one-shot delay-DSN clone at BOUNCE_STAGE (the original stays queued and keeps retrying);

  • permanent failure, or MAX_LIFETIME exhausted → reroute to BOUNCE_STAGE, or fail (terminal failed) if no BOUNCE_STAGE is configured;

  • an undeliverable bounce (null sender) is never re-bounced: a copy is delivered to POSTMASTER (if set) and the row is finished.

85.1.10.1.6. Commands

worker

Run as a persistent pepsi-dispatch(1) worker, reading message ids on standard input.

mta-sts DOMAIN

Diagnostic: look up and print the MTA-STS policy that would apply when delivering to DOMAIN. Uses the system resolver and needs no message or stage configuration.

85.1.10.1.7. Global Options

-c FILE, –config FILE

Read the configuration from FILE instead of searching the default locations.

-L LOGLEVEL, –log LOGLEVEL

Set the logging verbosity (default info).

-v, –verbose

Show log messages from all sources.

-h, –help; -V, –version

Print a usage summary / the version and exit.

85.1.10.1.8. Exit Status

The outcome of each message is reported to pepsi-dispatch(1) on the worker’s status line, not as the exit status.

0

The worker ran until its standard input was closed, or the mta-sts lookup completed.

1

A fatal error occurred (unreadable configuration, the database could not be opened, or standard input/output failed). The reason is written to the log.

85.1.10.1.9. Examples

Run the stage on message 42 (it must be running):

echo 42 | pepsi-stage-relay-to-internet -c /etc/pepsi/pepsi.conf worker

Check which MTA-STS policy applies to a recipient domain:

pepsi-stage-relay-to-internet -c /etc/pepsi/pepsi.conf mta-sts example.com

85.1.10.1.10. See Also

pepsi-config(1), pepsi-stage-bounce(1), pepsi-stage-dkim-sign(1), pepsi-stage-relay-to-smarthost(1), pepsi-dispatch(1), pepsi.conf(5), pepsi.state(7), pepsi-setup(1)

85.1.10.1.11. Bugs

Report bugs to the Pepsi issue tracker.