33. pepsi-dispatch

The single long-lived coordinator that drives messages through the stages.

33.1. Role

pepsi-dispatch is the only process that starts stage programs. Run exactly one per system, as pepsi-dispatch serve (the one subcommand; the global -c/-L/-v flags come before it). It claims newly queued rows and feeds each to an idle worker process of the row’s stage, requeues paused messages when their retry time is due, and recovers from crashes and timeouts. It processes no message content itself. When started as root, it drops to the pepsi service account, which the workers inherit. Reference: pepsi-dispatch(1).

33.2. Features

  • Event-driven, batched claiming. LISTENs on the workqueue, config_changed, telemetry_changed and pepsi_list_changed channels with a periodic POLL_INTERVAL heartbeat. On any wake it drains its pending internal events (so per-stage capacities are current) and then claims work for every stage with spare capacity in one UPDATE … RETURNING — each stage up to its own remaining capacity (a per-stage LATERAL … ORDER BY workqueue_id LIMIT cap that stops at the cap via the partial pending index). With exactly one dispatcher claiming serially as the sole pending``→``running writer, the claim needs no FOR UPDATE SKIP LOCKED.

  • Fair scheduling across senders. By default (FAIR_SCHEDULING) the claim shares each stage’s worker time among senders, in the manner of the Linux CFS/EEVDF scheduler, so that one sender’s burst does not delay everybody queued behind it. A sender is the authenticated account, else the connecting address (an IPv6 one by its /64), else the envelope domain of a locally created message. FAIR_FIFO_PERCENT of the slots stay oldest-first, which bounds how long any message can wait. The accounting is kept in memory only.

  • Persistent, pipelined workers. Each stage runs PROGRAM worker (found on PATH unless absolute) as long-lived processes that connect to the database once and read message ids from standard input, answering with a one-line status (0 on success) per message, in order. A worker is handed up to QUEUE_LIMIT ids at once (written to its stdin in a single buffer); it processes them strictly serially, but the next id is already queued, so the coordinator saves one round-trip per message. A stage’s in-flight capacity is therefore QUEUE_LIMIT × PARALLELISM, and because the worker count is unchanged it costs no extra database connections.

  • Elastic per-stage pools. Workers are started on demand up to a stage’s PARALLELISM; work is packed onto the fewest workers, and surplus workers are stopped after WORKER_IDLE_TIMEOUT — so a quiet stage collapses toward zero workers and a busy one grows to its cap. After MAX_MESSAGES a worker recycles its child (once its pipeline drains); a stage whose PROGRAM cannot start is held off with a short cooldown instead of spinning.

  • Advancing is the stage’s write. A stage that advances a message sets the row pending at the next stage itself, in the same update that moves stage; the dispatcher never rewrites a row on success. When the worker reports its status, the coordinator moves on to its next claim, which picks the row up for the next stage’s pool (so an advance needs no notification). A worker may also fuse a fast successor into the same pass (see pepsi-dispatch(1), Stage fusion).

  • Retry scheduling. Sleeps until the earliest paused row’s timeout (or a due worker reap / spawn cooldown / the heartbeat), then flips due rows back to pending and re-scans.

  • Crash & watchdog recovery. A worker reports a non-zero status only for an error that its stage marked permanent (or after giving up on retries); that one message is failed and the worker survives. A crashed child, or a running message unanswered within MAX_RUNTIME (timed per message, from when it reached the head of the worker’s queue or the worker’s next line), gives that message a strike: it is paused and retried after a minute, then two, and only the third strike sets it failed (crash) or timeout (hang), with state.failure_class crashed or timed-out. The worker’s other pipelined messages are requeued to pending rather than lost. A child that dies before printing ready, or exits 78 (configuration or schema refused), blames nothing: its messages go back to pending and the stage is held off for a short cooldown. Reasons are written under a status = 'running' guard, so that they never overwrite a row that a stage has already moved on.

  • Stage errors are retried. An error that a stage does not mark permanent is treated as a host fault: the worker pauses the message (state.last_error, state.temporary_failures; back-off one minute doubling to an hour) and reports success; past the stage’s MAX_LIFETIME it reroutes to BOUNCE_STAGE or fails it (failure_class = retries-exhausted).

  • Database trouble is not the message’s fault. A worker that loses its database connection reports status 74; the dispatcher pauses the message for 30 s and retries it for as long as the database stays away. A worker that finds the database connection-exhausted reports 75; the message is requeued and the stage running the most workers has its parallelism halved for five minutes.

  • Exactly one dispatcher. A session-scoped PostgreSQL advisory lock makes a second dispatcher refuse to start (after waiting about ten seconds for a predecessor that is still exiting): two would un-claim each other’s rows and deliver, bounce or settle payments twice.

  • Start-up / shutdown safety. Resets orphaned running rows (from a previous dispatcher) to pending on start; on SIGINT/SIGTERM stops workers and resets their (and any claimed-but-unassigned) rows.

  • Isolation. At most one writer ever touches a given queue row, so the shared pool runs READ COMMITTED; queue operations are still wrapped in a retry helper that re-runs a transient 40001/40P01 failure as cheap insurance.

  • Config propagation. Passes CONFIG_FILE to workers via -c (before the worker subcommand) so they load the same configuration.

  • Hot reload of the pipeline. On a config_changed notification it rebuilds its own stage table from the database overlay and retires every worker — idle ones at once, busy ones when their current message reports back — so a [stage-*] change takes effect without a restart. A reloaded graph that does not parse ends the process (exit 78; the unit restarts with back-off) rather than routing by a pipeline that the workers no longer share; an overlay that cannot be read keeps the current graph and retries. Its own [pepsi-dispatch] options are not reloaded. A telemetry_changed notification likewise retires the workers so that their telemetry sinks follow the change, and a pepsi_list_changed one (a mailing list or list domain added, renamed or removed) retires them so that the router reloads its route map.

  • Reload of the configuration file. Every two seconds it compares the modification time and size of the file it was started with (and of CONFIG_FILE when that names another) with what it last read; a changed file is re-read and handled like a config_changed notification, so an edit to pepsi.conf reaches every stage without a restart. A file that does not parse is not applied: the error is logged and the running configuration kept.

  • Statistics. Accumulates per-stage throughput, kills/timeouts, crashes and processing time plus the global stage/message totals in memory, and flushes the deltas to the stage_stats/dispatch_stats tables in a single transaction every STATS_INTERVAL, whenever the pipeline goes idle (at most once a second, so a finished burst’s numbers do not wait for the next tick), and on shutdown; nothing is written when there are no deltas. It is the only writer of those tables. The figures of stages fused into another stage’s worker pass come back on that pass’s status line instead of being written by the worker, so that the workers of a stage do not contend for that stage’s single row. These are exported by pepsi-httpd’s /metrics.

33.3. Configuration

[pepsi-dispatch]: MAX_RUNTIME (default 300 s), WORKER_IDLE_TIMEOUT (default 5 s), POLL_INTERVAL (default 30 s), STATS_INTERVAL (default 60 s), DB_POOL_SIZE (default 8), CONFIG_FILE (no default; without it the workers fall back to the default configuration search), FAIR_SCHEDULING (default yes), FAIR_HALF_LIFE (default 60 s) and FAIR_FIFO_PERCENT (default 10). Every duration must lie in (0, 168 h]. Per-stage worker pools are sized by PARALLELISM (default 4), QUEUE_LIMIT (default 4, the messages pipelined to one worker at once) and MAX_MESSAGES (default 1000) in each [stage-<name>] section. See pepsi.conf(5) and Configuration.

33.4. See also

Architecture, pepsi-queue, pepsi-ingress, pepsi-dispatch(1). Security: the dispatcher and its workers run as the pepsi account (The privilege split), what a compromised worker gets (An attacker who compromises a stage worker), and the scheduling and runtime limits in the availability table (Availability).