33. pepsi-dispatch¶
The single long-lived coordinator that drives messages through the stages.
33.1. Role¶
pepsi-dispatch is the only process that starts stage programs. Run exactly
one per system, as pepsi-dispatch serve (the one subcommand; the global
-c/-L/-v flags come before it). It claims newly queued rows and feeds
each to an idle worker process of the row’s stage, requeues paused messages
when their retry time is due, and recovers from crashes and timeouts. It processes no
message content itself. When started as root, it drops to the pepsi
service account, which the workers inherit. Reference: pepsi-dispatch(1).
33.2. Features¶
Event-driven, batched claiming.
LISTENs on theworkqueue,config_changed,telemetry_changedandpepsi_list_changedchannels with a periodicPOLL_INTERVALheartbeat. On any wake it drains its pending internal events (so per-stage capacities are current) and then claims work for every stage with spare capacity in oneUPDATE … RETURNING— each stage up to its own remaining capacity (a per-stageLATERAL … ORDER BY workqueue_id LIMIT capthat stops at the cap via the partialpendingindex). With exactly one dispatcher claiming serially as the solepending``→``runningwriter, the claim needs noFOR UPDATE SKIP LOCKED.Fair scheduling across senders. By default (
FAIR_SCHEDULING) the claim shares each stage’s worker time among senders, in the manner of the Linux CFS/EEVDF scheduler, so that one sender’s burst does not delay everybody queued behind it. A sender is the authenticated account, else the connecting address (an IPv6 one by its /64), else the envelope domain of a locally created message.FAIR_FIFO_PERCENTof the slots stay oldest-first, which bounds how long any message can wait. The accounting is kept in memory only.Persistent, pipelined workers. Each stage runs
PROGRAM worker(found onPATHunless absolute) as long-lived processes that connect to the database once and read message ids from standard input, answering with a one-line status (0on success) per message, in order. A worker is handed up toQUEUE_LIMITids at once (written to its stdin in a single buffer); it processes them strictly serially, but the next id is already queued, so the coordinator saves one round-trip per message. A stage’s in-flight capacity is thereforeQUEUE_LIMIT × PARALLELISM, and because the worker count is unchanged it costs no extra database connections.Elastic per-stage pools. Workers are started on demand up to a stage’s
PARALLELISM; work is packed onto the fewest workers, and surplus workers are stopped afterWORKER_IDLE_TIMEOUT— so a quiet stage collapses toward zero workers and a busy one grows to its cap. AfterMAX_MESSAGESa worker recycles its child (once its pipeline drains); a stage whosePROGRAMcannot start is held off with a short cooldown instead of spinning.Advancing is the stage’s write. A stage that advances a message sets the row
pendingat the next stage itself, in the same update that movesstage; the dispatcher never rewrites a row on success. When the worker reports its status, the coordinator moves on to its next claim, which picks the row up for the next stage’s pool (so an advance needs no notification). A worker may also fuse a fast successor into the same pass (see pepsi-dispatch(1), Stage fusion).Retry scheduling. Sleeps until the earliest
pausedrow’stimeout(or a due worker reap / spawn cooldown / the heartbeat), then flips due rows back topendingand re-scans.Crash & watchdog recovery. A worker reports a non-zero status only for an error that its stage marked permanent (or after giving up on retries); that one message is
failedand the worker survives. A crashed child, or a running message unanswered withinMAX_RUNTIME(timed per message, from when it reached the head of the worker’s queue or the worker’snextline), gives that message a strike: it is paused and retried after a minute, then two, and only the third strike sets itfailed(crash) ortimeout(hang), withstate.failure_classcrashedortimed-out. The worker’s other pipelined messages are requeued topendingrather than lost. A child that dies before printingready, or exits78(configuration or schema refused), blames nothing: its messages go back topendingand the stage is held off for a short cooldown. Reasons are written under astatus = 'running'guard, so that they never overwrite a row that a stage has already moved on.Stage errors are retried. An error that a stage does not mark permanent is treated as a host fault: the worker pauses the message (
state.last_error,state.temporary_failures; back-off one minute doubling to an hour) and reports success; past the stage’sMAX_LIFETIMEit reroutes toBOUNCE_STAGEor fails it (failure_class = retries-exhausted).Database trouble is not the message’s fault. A worker that loses its database connection reports status
74; the dispatcher pauses the message for 30 s and retries it for as long as the database stays away. A worker that finds the database connection-exhausted reports75; the message is requeued and the stage running the most workers has its parallelism halved for five minutes.Exactly one dispatcher. A session-scoped PostgreSQL advisory lock makes a second dispatcher refuse to start (after waiting about ten seconds for a predecessor that is still exiting): two would un-claim each other’s rows and deliver, bounce or settle payments twice.
Start-up / shutdown safety. Resets orphaned
runningrows (from a previous dispatcher) topendingon start; onSIGINT/SIGTERMstops workers and resets their (and any claimed-but-unassigned) rows.Isolation. At most one writer ever touches a given queue row, so the shared pool runs
READ COMMITTED; queue operations are still wrapped in a retry helper that re-runs a transient40001/40P01failure as cheap insurance.Config propagation. Passes
CONFIG_FILEto workers via-c(before theworkersubcommand) so they load the same configuration.Hot reload of the pipeline. On a
config_changednotification it rebuilds its own stage table from the database overlay and retires every worker — idle ones at once, busy ones when their current message reports back — so a[stage-*]change takes effect without a restart. A reloaded graph that does not parse ends the process (exit78; the unit restarts with back-off) rather than routing by a pipeline that the workers no longer share; an overlay that cannot be read keeps the current graph and retries. Its own[pepsi-dispatch]options are not reloaded. Atelemetry_changednotification likewise retires the workers so that their telemetry sinks follow the change, and apepsi_list_changedone (a mailing list or list domain added, renamed or removed) retires them so that the router reloads its route map.Reload of the configuration file. Every two seconds it compares the modification time and size of the file it was started with (and of
CONFIG_FILEwhen that names another) with what it last read; a changed file is re-read and handled like aconfig_changednotification, so an edit topepsi.confreaches every stage without a restart. A file that does not parse is not applied: the error is logged and the running configuration kept.Statistics. Accumulates per-stage throughput, kills/timeouts, crashes and processing time plus the global stage/message totals in memory, and flushes the deltas to the
stage_stats/dispatch_statstables in a single transaction everySTATS_INTERVAL, whenever the pipeline goes idle (at most once a second, so a finished burst’s numbers do not wait for the next tick), and on shutdown; nothing is written when there are no deltas. It is the only writer of those tables. The figures of stages fused into another stage’s worker pass come back on that pass’s status line instead of being written by the worker, so that the workers of a stage do not contend for that stage’s single row. These are exported by pepsi-httpd’s/metrics.
33.3. Configuration¶
[pepsi-dispatch]: MAX_RUNTIME (default 300 s), WORKER_IDLE_TIMEOUT
(default 5 s), POLL_INTERVAL (default 30 s), STATS_INTERVAL (default 60 s),
DB_POOL_SIZE (default 8), CONFIG_FILE (no default; without it the
workers fall back to the default configuration search), FAIR_SCHEDULING
(default yes), FAIR_HALF_LIFE (default 60 s) and FAIR_FIFO_PERCENT
(default 10). Every duration must lie in (0, 168 h]. Per-stage worker pools are sized by PARALLELISM (default 4),
QUEUE_LIMIT (default 4, the messages pipelined to one worker at once) and
MAX_MESSAGES (default 1000) in each [stage-<name>] section. See
pepsi.conf(5) and Configuration.
33.4. See also¶
Architecture, pepsi-queue, pepsi-ingress,
pepsi-dispatch(1). Security: the dispatcher and its workers run as
the pepsi account (The privilege split), what a compromised worker gets
(An attacker who compromises a stage worker), and the scheduling and runtime limits in the availability
table (Availability).