ADR 0018: Distributed Task Queue and Scheduler¶
Status: Proposed Date: 2026-05-24
Context¶
crates/spikard-http/src/background.rs provides an in-process, fire-and-forget task
runtime: a bounded queue, a concurrency cap, and graceful drain. It does not survive a
restart and does not coordinate across instances. The toolbox needs durable background
work — persisted jobs, retries, and cron scheduling — for services that must not lose
work, while keeping the lightweight in-process path for tasks that do not need
durability.
Decision¶
- Use apalis in a new
spikard-taskscrate for durable background processing: persisted jobs, retries, prioritization, result tracking, and cron-scheduled jobs, with Redis, PostgreSQL, SQLite, and MySQL backends. - A
TaskHandlertrait mirrorsMessageHandler(ADR 0015): the job payload is JSON arguments, and the apalis worker invokes the host handler through the same FFI callback machinery. The worker runs inside the unifiedApplicationruntime. TaskQueueConfigandCronScheduleDTOs live inspikard-core::services. The scheduler is an apalis cron source that enqueues jobs on aCronSchedule.- Backends behind Cargo features (
postgres,redis,sqlite,mysql,cron), using rustls. The in-processBackgroundRuntimestays as the non-durable fast path;spikard-tasksis the durable upgrade, selected by configuration.
Consequences¶
- The existing
fixtures/background_tasks.jsonenqueue-returns-202 shape maps cleanly onto a "request enqueues a durable job" handler, so HTTP-triggered enqueue keeps working. - Two task paths coexist (in-process versus durable); documentation must make the trade-off explicit (latency and simplicity versus durability and cross-instance coordination).
- apalis is pre-1.0; pin the version and track its release. The
TaskHandlerboundary insulates host code from apalis API churn. - An in-memory queue backend (or SQLite
:memory:via the SQL backend) is the cross-language parity mock per ADR 0022; containerized Redis and PostgreSQL cover the durable backends in Rust integration tests.