95.1.15. pepsi-stage-relay-to-lmtp

deliver a message to a local MDA (Dovecot) over LMTP

Manual section:

1

95.1.15.1.1. Name

pepsi-stage-relay-to-lmtp - the LMTP local-delivery stage of the Pepsi pipeline.

95.1.15.1.2. Synopsis

pepsi-stage-relay-to-lmtp [GLOBAL-OPTIONS] worker

95.1.15.1.3. Description

pepsi-stage-relay-to-lmtp is a stage program run by pepsi-dispatch(1). It loads the message’s pepsi.workqueue row (refusing to act unless its status is running), reads its [stage-<stage>] section, and hands the message over LMTP (RFC 2033) to a Mail Delivery Agent, typically Dovecot, for final local delivery. The MDA applies each recipient’s Sieve (RFC 5228) script as it files the mail, so server-side filtering (fileinto, vacation, redirect, flag setting, …) is provided by the MDA rather than by Pepsi.

The stage delivers one recipient per row: it first splits a message with several recipients into one row for each, dropping a recipient that is named twice, so that each LMTP transaction has one recipient. LMTP replies name no recipient, so after a fault nothing could tell whose reply a later one is, and one slow Sieve program would hold up every recipient of the transaction. The cost is one connection and one Sieve run per recipient; the rows share one stored body. The stage itself never builds a delivery-status notification. It delivers, and otherwise routes the recipient onward to a configured stage, recording the reason under state.bounce so that a downstream pepsi-stage-bounce(1) (or a relay) can act on it. Each recipient’s fate follows its reply (its end-of-DATA reply, or the refusal of its RCPT TO):

  • 2xx: delivered; the recipient leaves the row. When the recipient asked for NOTIFY=SUCCESS, [pepsi] ORIGINATE_SUCCESS_DSN is set and the message is not itself a bounce, a positive notification is routed to the optional NOTIFY_STAGE.

  • a permanent reject is routed by reason, which the stage classifies from the reply’s RFC 3463 enhanced status. A bad destination mailbox (x.1.x, or x.2.x other than x.2.2, or a bare 550 user-unknown with no enhanced status) goes to UNKNOWN_MAILBOX_STAGE when the recipient is on one of LOCAL_DOMAINS, else to NEXT_STAGE. An over-quota reject (x.2.2) goes to QUOTA_LIMIT_STAGE (else NEXT_STAGE), a Sieve / local policy reject (x.7.x) goes to SIEVE_REJECT_STAGE (else NEXT_STAGE). Anything else goes to BOUNCE_STAGE when the recipient is on one of LOCAL_DOMAINS and that option is set, and otherwise to NEXT_STAGE.

  • 4xx (e.g. server busy): a transient failure. The recipient is retried with exponential backoff until MAX_LIFETIME, then routed by the same classification of reasons.

  • no reply after the whole message was sent: a lost acknowledgement. A timeout cancels nothing at the MDA, which usually files the mail all the same, so a retry may deliver a second copy. The first retry waits at least 15 minutes and the second at least 30, and a worker killed or crashed at this stage counts like a reply that did not come. After three such attempts, or when the stage gives up for any reason after one of them, it stops and routes a report to BOUNCE_STAGE (else NEXT_STAGE) with state.bounce.kind unknown. The sender is told that the mail may have arrived, and to ask the recipient. RFC 3464 has no action for this, so the report says Action: failed with Status: 4.4.2: the attempts have been abandoned.

A recipient that arrived with an RFC 7293 RRVS= parameter has it passed on when the MDA advertises RRVS. Dovecot, for one, does not. When the MDA does not advertise it, the default no-support action R refuses that recipient here with 5.7.19 (routed like any x.7.x reject). With C the stage delivers it, and writes the request as a Require-Recipient-Valid-Since: header field.

A delivered recipient’s row ends. A routed recipient becomes a pending row at its target stage, and a deferred one stays on its row for the next retry. A DELAY warning (when due and NOTIFY=DELAY was requested) is routed to NOTIFY_STAGE. All of this is one statement with the row’s final update (its deletion, or its pause until the retry). If that statement fails, for example because the database is unreachable, the error is retried and a recipient that the MDA accepted gets a second copy: delivery is at least once.

A failure of the whole session before any per-recipient reply, whether in connecting, the greeting, STARTTLS, authentication or the data transfer, does not concern one recipient. After a transient one the whole message is retried with backoff; after a permanent one (or any once MAX_LIFETIME has passed) every recipient is routed onward without classifying the reason: to BOUNCE_STAGE when it is on one of LOCAL_DOMAINS and that option is set, otherwise to NEXT_STAGE. Some permanent refusals concern the message rather than the session: the refusal of MAIL FROM, of a message larger than the server’s SIZE limit, or of content that cannot be downgraded for the server. Such a refusal is instead taken as every recipient’s reply and routed per recipient as above.

Any other error of the stage, such as the database refusing the fan-out, is retried as in every stage (after one minute, doubling up to an hour) until MAX_LIFETIME; after that the message goes to BOUNCE_STAGE when the section sets one (the option every [stage-<name>] section accepts), or is failed.

Because the MDA decides which addresses are local, this stage performs no locality test of its own (unlike pepsi-stage-relay-to-maildir(1)): every recipient is offered to the MDA. LOCAL_DOMAINS only decides where an unknown mailbox goes afterwards. An unknown mailbox on one of our own domains must not reach a relay stage, because the relay would look up our own MX and post the message straight back to this host; so it goes to UNKNOWN_MAILBOX_STAGE. On any other domain (an alias target that expanded off-site) it goes on to NEXT_STAGE. Place the stage where only recipients meant to be local arrive, usually after alias expansion (pepsi-stage-aliases(1)). It is not terminal: NEXT_STAGE is mandatory. Set it to a pepsi-stage-bounce(1) to bounce undeliverable recipients, or to a relay stage to forward them off-site. A message that reaches the stage with no NEXT_STAGE configured is marked failed before the MDA is contacted, so nothing is delivered.

Unlike local Maildir writing, LMTP needs no privileged helper: the MDA performs the per-user delivery itself, so this stage runs as the unprivileged pepsi user and carries no setuid/setgid bit. It is folded into the unified pepsi binary. A PASSWORD does not belong in the world-readable pepsi.conf: when pepsi-setup(1) writes the configuration it moves every PASSWORD into the secrets.d/pepsi.secret fragment (mode 0640, readable by the pepsi account) and leaves an @inline-secret@ directive in its place. The MDA is trusted with what the stage hands it, like every external helper that the operator wires in (see the Threat model chapter of the Pepsi manual).

95.1.15.1.4. Recycled addresses (RFC 7293)

For every recipient the MDA accepted, the stage records the passwd login behind the address, with its uid, in pepsi.login_assignment (see pepsi-setup(1), The login assignment record), exactly as pepsi-stage-relay-to-maildir(1) does. The address must be on LOCAL_DOMAINS, its sub-address (the site-wide [pepsi] RECIPIENT_DELIMITER) is stripped, and the lower-cased local part must name a passwd entry whose uid lies in the range for regular accounts in /etc/login.defs. A login that comes back under another uid, or after it was found gone, has a new owner: the database drops the previous owner’s per-address state at that moment, and logs it. This needs no configuration; it is what lets a site that delivers through an MDA notice a recycled login at all.

The MDA’s virtual users are not covered: a recipient that passwd does not know is delivered but not recorded, so a change of its owner inside the MDA is not noticed, and the previous owner’s settings and whitelists stay. The lookups are made before the MDA is contacted. During a directory outage (an LDAP or sssd backend that is down) the message is therefore retried like any other error while nothing has been delivered yet, and the outage is never read as “no such user”. The recording follows the MDA’s per-recipient replies, so a recipient that the MDA refused is not recorded. A database error at that point is logged rather than retried, because the MDA has already filed the message and the next delivery records the login.

95.1.15.1.5. Options

Run mode:

worker

Run as a persistent pepsi-dispatch(1) worker, reading workqueue row ids on standard input.

Configuration (in the stage’s own [stage-<name>] section):

PROGRAM

Must be pepsi-stage-relay-to-lmtp.

NEXT_STAGE

Required. Where a recipient the MDA could not deliver is routed (a bad destination mailbox, and the fallback for the reasons below). Set it to a pepsi-stage-bounce(1) to bounce undeliverable recipients, or to a relay stage (e.g. pepsi-stage-srs(1) → a smarthost) to forward them off-site. pepsi-setup rejects a configuration without it.

QUOTA_LIMIT_STAGE

Where to route a recipient the MDA refused for being over quota (RFC 3463 x.2.2). Defaults to NEXT_STAGE. Set it to a bounce stage when NEXT_STAGE is a relay, so quota rejects bounce rather than relay off-site.

SIEVE_REJECT_STAGE

Where to route a recipient refused by Sieve / local policy (RFC 3463 x.7.x). Defaults to NEXT_STAGE.

UNKNOWN_MAILBOX_STAGE

Where to route a recipient on one of LOCAL_DOMAINS that the MDA refused as an unknown mailbox (x.1.x, a bare 550). Defaults to NEXT_STAGE. Set it to a bounce stage whenever NEXT_STAGE can reach a relay: relaying an unknown address on our own domain delivers it back to this host. pepsi-setup --wizard sets it to bounce.

LOCAL_DOMAINS

The domains whose mailboxes the MDA holds (space- or comma-separated). Defaults to [pepsi-ingress] ACCEPTED_DOMAINS. Read for UNKNOWN_MAILBOX_STAGE, for the login assignment record (only a recipient on these domains is looked up in passwd; see Recycled addresses above), and by pepsi-ingress(1)’s recipient check, which asks the MDA whether an address on these domains exists before accepting it (see VERIFY_RECIPIENTS).

NOTIFY_STAGE

Where positive SUCCESS delivery notifications and DELAY warnings are routed (as a cloned message carrying state.bounce.kind); set it to a pepsi-stage-bounce(1), which renders the report. When unset, no positive or delay notifications are originated. The stage never builds a notification itself.

SOCKET

Path of the MDA’s LMTP UNIX-domain socket (e.g. /run/dovecot/lmtp). This is the common layout for local delivery, with no TLS or authentication. Exactly one of SOCKET and HOST must be set.

HOST / PORT

Reach the MDA over TCP instead (PORT defaults to 24). Use for an MDA on another host.

TLS

Transport security for a HOST target: off (default), starttls, or tls (implicit). TLS_VERIFY (default yes) and TLS_CA control PKIX verification; TLS_CLIENT_CERT/TLS_CLIENT_KEY present a client certificate for mutual TLS / SASL EXTERNAL.

AUTH

SASL mechanism for a TCP target: none (default), plain, login or external. Authentication is refused over cleartext, so any non-none AUTH requires TLS = starttls or tls (and external requires a client certificate). USERNAME/PASSWORD supply the credential (both required for plain and login; for external an optional USERNAME is the authorisation identity). A SOCKET target accepts no AUTH at all.

SERVER_NAME

Name announced in LHLO and stamped in the Received: header. Defaults to [pepsi-ingress] HOSTNAME; one of the two must be set.

RETRY_INITIAL, RETRY_MAX_INTERVAL, RETRY_FACTOR, MAX_LIFETIME

When to retry a recipient that was deferred transiently, and when to give up. Durations use h/m/s units. Defaults: 60 s, 15 m, 2 and 120 h.

DELAY_DSN_AFTER

Emit a one-shot “delayed” notification to NOTIFY_STAGE once a still-undelivered message has been queued this long and the (first deferred) recipient asked for NOTIFY=DELAY. Unset by default (no delay warnings); has no effect without NOTIFY_STAGE.

CONNECT_TIMEOUT, COMMAND_TIMEOUT, DATA_TIMEOUT

Per-operation LMTP timeouts. Defaults: 60 s, 120 s and 600 s. A timeout ends the session: replies already received stand, and every recipient still unresolved is deferred. A retry can deliver a second copy to a recipient whose acknowledgement was late.

95.1.15.1.6. See Also

pepsi-dispatch(1), pepsi-stage-relay-to-maildir(1), pepsi-stage-aliases(1), pepsi-stage-bounce(1), pepsi.conf(5), pepsi.state(7).

RFC 2033 (LMTP), RFC 5228 (Sieve).