tinyjobs design notes¶
Why tinyjobs is built the way it is. The documentation covers usage; this covers the reasoning, including for the parts that are not written yet. Sections describing unbuilt features say so.
Delivery semantics¶
tinyjobs guarantees at-least-once execution.
The failure window is narrow but real: a worker can finish a job and die before it writes SUCCESS
back to the store. The lease on that job eventually expires, a reaper requeues it, and the job runs a
second time. There is no way to close this window without a transaction that spans both the queue and
whatever your task touches.
The practical consequence is a contract with you, the author of the task:
Task functions must be idempotent. Running one twice with the same arguments must be indistinguishable from running it once.
In practice this means: upsert instead of insert, check-then-act inside a transaction, use the job id as a natural deduplication key for external calls, and make external API calls with an idempotency key of their own.
Where we can help:
- Enqueue-side deduplication.
with_options(key="invoice-sync:123")refuses to create a second job while one with the same key is pending or running. This makes enqueueing effectively-once, which is the half of the problem that actually shows up in practice: a retried HTTP request submitting the same job three times. - Attempt visibility. Every attempt is recorded as an
Executionrow, so a duplicate run is visible after the fact rather than silent. - Transactional enqueue. On Postgres and SQLite,
enqueuejoins your application's own transaction: if the transaction rolls back, the job is never created. This removes the classic dual-write bug where a job starts running against a row that was never committed. It is a genuine advantage of a database-backed queue over a broker-backed one; see Transactional enqueue for the API and what each backend needs for it.
Guarantees¶
The numbers on the "Measured" page came from harnesses written for each run and thrown away, so a reader had to take them on trust, and a store configured against the guarantees, a Redis on snapshots or a SQLite file on NFS, broke them silently. The point of a small queue is that it can be trusted, so the guarantees are declared in one place, checked against the store that is actually configured, and provable on the reader's own machines with a harness that ships in the package. What tinyjobs promises, on every backend:
- A job acknowledged by
enqueueruns, at least once, until it succeeds or uses its attempts: a worker killed outright leaves it to its lease, and the reaper returns it. - A job finishing while its worker dies may run again, never be lost; the second run is visible as a second execution, so idempotent tasks make the whole thing effectively once.
- A cron schedule fires once per due time, from any number of workers, without a leader.
- A deduplication key refuses a second job while one is pending or running.
- On Postgres, a job enqueued in a transaction that rolls back never exists.
What the store must provide for those to hold is the store's own durability and the clock discipline described under each backend. Both are now checked rather than assumed.
What the store must provide¶
tinyjobs check asks the backend to audit its own configuration, through an audit() the
backend may define, read like claim_job rather than required by the protocol, returning the
same Finding rows as the health check, so a plugin without one is unaffected and the
dashboard's overview shows the result beside the queues. The audit reads settings, never writes,
and costs one call: a few pragmas, the mount table and the free space on SQLite, one INFO on
Redis, one query of three settings and the recovery state on Postgres.
- SQLite. The journal mode must be
walandsynchronousat leastNORMAL, which the passing line says holds across process crashes, not power loss, since in WAL modeNORMALcan roll back the last commits on a power cut, the one case this promise leaves toFULLand its fsync per commit; the file must be on a local filesystem that survives a reboot, read from/proc/self/mountinfoon Linux, where the mount is the one whose point is the file itself if there is one, else the deepest one containing the file among those on the file's device, byst_devof the file, or of its directory before the file exists, against the table'smajor:minor, so a mount hidden under a later or a moved one is never taken, else, when no entry carries that device, as on a btrfs subvolume, the deepest containing mount of any device; octal escapes are decoded and undecodable bytes kept as the paths carry them. NFS, CIFS and SMB, 9p, Ceph, Gluster, Lustre, GPFS, WebDAV and the FUSE mounts of object stores fail, since SQLite's locking is unreliable there and WAL needs shared memory;tmpfsandramfsfail, since a reboot empties them, and so dooverlay,fuse-overlayfsandaufs, a container's writable layer, gone with the container; a file that is itself a mount point, as a Docker bind mount of the one file makes it, fails too, since its-waland-shmfiles land outside the mount and vanish with the container; and the disk holding the file must have room, 64 MB at least, since SQLite refuses every write with "database or disk is full" when it runs out and a worker then fails every outcome. The file audited is the one SQLite reports inPRAGMA database_list, not a reading of the name given, so a URI'smode=memoryorvfs=memdb, the last value when a parameter repeats, counts as in memory and a name withoutfile:is the literal filename it is to SQLite. An in-memory database audits clean; a file opened through afile:URI, whose pragmas tinyjobs leaves to the caller, is audited as the file it is and told to set WAL itself when it has not. - Redis.
appendonlymust be on, since on snapshots alone a crash loses the queue since the last one, which in testing was all of it, and the last snapshot, and the last append-only write whenappendonlyis on, must have succeeded, since Redis refuses every write withMISCONFafter a failed snapshot while save points are configured, which is the default, whether or not the append-only file is on;maxmemory-policymust not be anallkeys-*policy whenevermaxmemoryis set, since those evict jobs under pressure without a word, whilenoevictionand thevolatile-*policies leave the jobs alone, as tinyjobs sets no TTL on any key, and fail enqueues withOOMinstead;used_memorymust be under 90% ofmaxmemory, since past it enqueues fail withOOM, or jobs are already being evicted; and the server must be the primary, since a replica refuses writes. All of it comes fromINFO, which ACL-restricted users usually keep whereCONFIGis taken away, and a field missing or in an unexpected form,role,aof_enabled, the write and snapshot statuses,maxmemory,maxmemory_policyorused_memory, is reported as not checked. - Postgres.
synchronous_commitmust not beoff, since a crash then loses the last commits and a job acknowledged to the caller vanishes, which is the one thing promised not to happen. It is a session setting, so tinyjobs does not leave it to the server: every connection it opens sets it toonwhen the session starts withoff, and keepslocal,remote_writeorremote_applyas they are, since forcingonwould weaken a server that waits for its standbys. The audit reads the session value, which fails only when something reset it after tinyjobs, and the server's default frompg_settings.reset_val, named beside a passing value when it isoff, since a job enqueued through the caller's own connection and transaction commits under that default and the caller's own rows with it.fsyncmust be on, since off it can lose or corrupt what was committed, except on Neon, told by itsneon.timeline_idsetting, which runs every compute withfsyncoff and keeps durability in its storage layer; and the connection must not be in recovery, since a standby refuses every write. The settings are read withcurrent_setting(name, true), so a server that lacks one answers NULL, reported as not checked, rather than an error.
Each problem is one failing finding naming the setting and its remedy, so check exits 1 and a
probe or the dashboard shows it, and what passed, or could not be checked, is one line after the
failures, the only line when nothing fails, saying what the store runs with. A setting the
server will not reveal, a mount table or a disk that does not answer, is reported as not
checked, not as a failure, since the guarantee may still hold.
A development setup breaks the guarantees on purpose, a Redis on snapshots in a compose file or
a SQLite file on the container's layer, and a readiness probe on check would keep such a
container from starting. tinyjobs check --store-warnings, store_warnings=True on
app.check(), turns the audit's failures into findings that pass with warning set, printed as
WARN by the command, and leaves the exit status to the rest of the check. The switch is on the
command and the call, not on the app, since the problem is in the store's configuration and not
in the code, and an app option that accepts losing jobs would end up in production settings.
Proving it on your own setup¶
tinyjobs verify URL is the crash-recovery harness of the "Measured" page, shipped: it runs on a store that holds
no other jobs, which it checks first and refuses otherwise, and leaves it as it found it but for
the schema rows, so it can run against a fresh file, a spare Redis database or a scratch Postgres
database, before a deploy or in a pipeline. The URL is required rather than taken from --app,
since a quiet production store would pass the check and then receive the harness's jobs; an
in-memory store is refused, since the workers could not share it.
tinyjobs verify redis://localhost:6379/9 --duration 30s --rate 50 --workers 3 --kill-every 5s
It starts the workers as subprocesses of tinyjobs worker on a tiny app of its own,
tinyjobs.verify:app, whose one task sleeps 20 to 100 ms and records its own run in a ledger, a
SQLite file in a temporary directory that the workers share, with INSERT OR IGNORE on the job's
number: the ledger is the side effect, and inserting it that way is the idempotency the contract
asks of every task. The URL and the ledger reach the workers through the environment, and each
run sends on a queue named after a random token and consumes only that queue, so two runs on one
store, or a worker of the application's own, never take each other's jobs. The workers run with a
short lease and reap interval, 5 s and 1 s, so a kill costs seconds, not the minute of the
defaults, with --no-scheduler, in a session of their own, so a Ctrl-C reaches the verifier
alone, and with their stdin on a pipe from the verifier that they watch: when it closes, because
the verifier was killed or died, each worker sends itself SIGTERM, so no run leaves workers
behind, though a verifier killed outright leaves its temporary directory and its jobs in the
store, which a new file, a flushed Redis database or dropped Postgres tables clear. The verifier
sends
jobs at the rate for the duration, numbering them by attempt so a send that failed after the
store had committed cannot share a number with the next; every --kill-every it stops the
oldest worker not yet signalled, alternating SIGTERM and SIGKILL, and starts another in its
place. Only a worker that has run a job is stopped, which the ledger knows from a token each
worker gets in its environment and records with its first run, a token rather than a pid since a
pid can be reused by the replacement, since a SIGTERM to a worker still importing, before its
handlers are installed, ends it with the signal's default and would look like a refusal to stop;
a SIGKILL goes only to a worker holding a running job, read from the running jobs'
worker_id, which carries the pid, so it lands on a job in flight unless the job ended in the
milliseconds between the query and the signal, and the lease is exercised rather than left to
chance, which is why --settle must exceed --lease plus the reap interval, since the
cut-short job comes back only when its lease ends, and why the command sizes --settle to twice
the lease when the lease is raised without it; a kill due while no worker is ready, or none
holds a job for a SIGKILL, is tried again a quarter of a second later, and one due
during a stall, a suspended verifier or a send the store held for seconds, is taken once, not
made up for, so the stops stay a kill interval apart. A worker that exits on its own is an
unexpected death; one that outlives the shutdown timeout after SIGTERM did not stop cleanly,
whatever it was doing, and one that exits non-zero on SIGTERM did not stop cleanly if it had
run a job. When the sending stops it waits up to --settle (60 s)
for its own queue to show nothing queued or running, so an application's job on another queue
cannot keep it waiting, stops the workers cleanly, then reads every job of its queue back and
reports:
sent 1,500 jobs: 1,500 succeeded, 0 failed, 0 cancelled, 0 unfinished, 0 lost
workers stopped 5 times, 3 by SIGTERM and 2 by SIGKILL, 0 unexpected deaths
4 jobs ran more than once, after the kills; every job applied its side effect once
end-to-end p50 69 ms, p95 108 ms, kills included
0 enqueue failures; the store is left as it was found
ok
Unfinished means still queued or running at the end, which a run cut short leaves behind. The
jobs are matched by number, read from each job's arguments and from the ledger's rows, never by
count, so a loss and a phantom cannot cancel out: lost means a number acknowledged by the store
and not found in it; never acknowledged means a number found whose send had been reported as
failed; enqueued more than once means two jobs carrying one number, which the ledger's INSERT
OR IGNORE would otherwise hide; missing means a number sent whose side effect never happened.
Ran more than once is the job's claims field, so no per-job query is needed, and the only
place a second run shows, since the ledger's INSERT OR IGNORE records each number once
however many times its job ran: applied once is the ledger's numbers against the numbers sent,
every number applied at least once and none that was not sent. Latency is
finished_at - created_at from the job rows. The verdict is ok and exit 0 only when nothing
was lost, failed or left unfinished, every job marked the ledger once, every enqueue went
through, no worker died unexpectedly or refused to stop, the store settled in time and answered
throughout, the run went to its end, some worker ran a job and at least one was SIGKILLed,
since a run too short for that proves nothing about leases, no job was cancelled, which only
someone else could have done, and nothing was left in the store;
otherwise the failing reasons are the verdict and the exit status is 1, with the tail of the
workers' log when one died or refused to stop, while a store refused, unreachable or settings
that make no sense are a configuration error and exit 2 before anything runs. The header and
the errors show the URL with its password masked. Afterwards the queued jobs of a cut-short run are
cancelled, in every case, since nothing would ever consume a queue named after the run, and
the run's finished jobs purged, unless --keep leaves them in the store, for tinyjobs jobs
and the dashboard, or other jobs appeared during the run, of another task or of another run on
the same store, in which case nothing is purged, the run's own finished jobs stay too and the
report says so. A job still running at the end, which a settle that timed out leaves behind a
SIGKILL, is cancelled with the queued ones, which is safe once every worker has been reaped,
and counted as unfinished, so the verdict fails and the store is still clean for the next run;
a run that failed before clearing says the store was not cleared. The workers' environment is
the verifier's own without any TINYJOBS_VERIFY_* variable inherited from the shell, so a
leftover prefix cannot send them to another namespace.
What it does not do: stop the store, skew clocks or saturate the machine. Those runs are on the "Measured" page, measured once by hand, and their value is the number, not the harness. The verifier answers the one question a reader has before trusting a queue: on my store, with my settings, does a killed worker lose a job. It needs a POSIX platform for the signals.
Core concepts¶
Four entities. Keeping them distinct is what makes retries, cron and monitoring simple instead of tangled.
Task: the registered function¶
@app.task
async def generate_pdf(document_id: int) -> str:
...
A Task is a declaration: a callable, a stable name, and its default options (retries, timeout, queue, priority). It lives for the lifetime of the process and is never persisted. Only its name is.
That name is the contract between the code and the store. It defaults to module.function, but it is
overridable (@app.task(name="pdf.generate")) and should be for anything long-lived: renaming a
module must not orphan the jobs already sitting in the queue.
Job: a request to execute¶
job = generate_pdf.enqueue(123)
A Job is a persistent record: this task, these arguments, this status. It is created by enqueue,
survives restarts, and is the unit users cancel, inspect and query.
Execution: a single attempt¶
One Job produces one or more Executions:
Job #123
├── attempt 1 → FAILED (ConnectionError, 0.4s, worker-a)
├── attempt 2 → FAILED (ConnectionError, 0.4s, worker-a)
└── attempt 3 → SUCCESS (2.1s, worker-b)
This is the piece most queue libraries omit, and its absence is felt immediately in production: with attempts collapsed into the job row, you lose every traceback but the last one, you cannot tell a job that succeeded first try from one that barely survived, and you cannot chart failure rates. An Execution row carries its own status, timings, worker id, and traceback on failure.
Schedule: a rule that creates jobs¶
@app.task(cron="*/5 * * * *")
async def refresh_cache():
...
A Schedule is not a Job. It is a rule with a next firing time; when that time arrives it creates a Job and computes the next one. Keeping it separate means cron jobs get ordinary retry behaviour, ordinary history, and ordinary cancellation, with no special-casing anywhere in the worker.
How they flow¶
@app.task Schedule (a coroutine in every worker)
│ │
│ registers │ cron fires
▼ │
Task ─────── enqueue() ──────► Job ◄────────────┘
│
│ claimed by a worker
▼
Execution ── retry ──► Execution
Data model¶
The SQLite schema as schema.py creates it, under the default prefix. Other backends are free to
store this differently, as long as the protocol contract holds. payload holds the serialized
(args, kwargs), codec names its encoding, hand_backs counts hand-backs in a row, claims
every claim ever and is never reset, run_at is the earliest execution time as unix seconds,
lease_expires_at is set only while running, runs_after is the job a job was sent after, waiting_on the same id for as long as it waits
and null otherwise, and
next_run_at on a schedule is its
compare-and-set column. tinyjobs_install_lock stays empty: a first enqueue inside a caller's
transaction writes to it to take the file's write lock before it reads, as
Transactional enqueue explains.
CREATE TABLE IF NOT EXISTS tinyjobs_install_lock (
taken INTEGER
);
CREATE TABLE IF NOT EXISTS tinyjobs_meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS tinyjobs_jobs (
id TEXT PRIMARY KEY,
task TEXT NOT NULL,
payload BLOB NOT NULL,
codec TEXT NOT NULL,
queue TEXT NOT NULL DEFAULT 'default',
status TEXT NOT NULL,
priority INTEGER NOT NULL DEFAULT 0,
attempts INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL DEFAULT 1,
hand_backs INTEGER NOT NULL DEFAULT 0,
claims INTEGER NOT NULL DEFAULT 0,
run_at REAL NOT NULL,
created_at REAL NOT NULL,
finished_at REAL,
lease_expires_at REAL,
worker_id TEXT,
dedup_key TEXT,
result BLOB,
last_error TEXT,
runs_after TEXT,
waiting_on TEXT
);
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_claim
ON tinyjobs_jobs (queue, priority DESC, run_at ASC)
WHERE status = 'queued';
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_lease
ON tinyjobs_jobs (lease_expires_at)
WHERE status = 'running';
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_by_status
ON tinyjobs_jobs (status, id DESC);
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_by_queue
ON tinyjobs_jobs (queue, id DESC);
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_purge
ON tinyjobs_jobs (status, finished_at)
WHERE finished_at IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS tinyjobs_jobs_dedup
ON tinyjobs_jobs (dedup_key)
WHERE dedup_key IS NOT NULL AND status IN ('queued', 'held', 'waiting', 'running');
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_waiting
ON tinyjobs_jobs (waiting_on)
WHERE waiting_on IS NOT NULL;
CREATE TABLE IF NOT EXISTS tinyjobs_executions (
id INTEGER PRIMARY KEY,
job_id TEXT NOT NULL REFERENCES tinyjobs_jobs(id) ON DELETE CASCADE,
attempt INTEGER NOT NULL,
claim INTEGER NOT NULL,
status TEXT NOT NULL,
worker_id TEXT NOT NULL,
started_at REAL NOT NULL,
finished_at REAL,
error_type TEXT,
traceback TEXT
);
CREATE UNIQUE INDEX IF NOT EXISTS tinyjobs_executions_claim
ON tinyjobs_executions (job_id, claim);
CREATE TABLE IF NOT EXISTS tinyjobs_schedules (
name TEXT PRIMARY KEY,
cron TEXT NOT NULL,
timezone TEXT NOT NULL,
next_run_at REAL NOT NULL,
last_run_at REAL,
last_job_id TEXT
);
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_held
ON tinyjobs_jobs (task, queue, priority DESC, run_at ASC)
WHERE status = 'held';
CREATE INDEX IF NOT EXISTS tinyjobs_jobs_running_by_task
ON tinyjobs_jobs (task)
WHERE status = 'running';
CREATE TABLE IF NOT EXISTS tinyjobs_limits (
task TEXT PRIMARY KEY,
last_at REAL NOT NULL DEFAULT 0
);
The last three serve Limits: the held index is the per-task line a claim admits from,
the running index answers how many of a task run right now, and tinyjobs_limits remembers when a
rate-limited task last started.
Notes on a few choices:
- UUIDv7 job ids, not autoincrement integers. They are sortable by time (so they behave like integers for pagination and index locality) but they can be generated client-side and remain unique across backends and across a future sharded store.
run_aton every job. Delayed jobs, retry backoff and cron output are all just "a queued job with a futurerun_at". This is the single most load-bearing column in the schema: it lets one state (QUEUED) cover three features that would otherwise need states of their own.codecper row, not global. You can flip the default codec without breaking jobs already in flight, and a mixed-codec queue during a migration is legal by construction.- The claim index is partial, and column order matters.
WHERE status = 'queued'keeps the index holding only claimable rows, so completed history, however much of it accumulates, is invisible to the hot path. Measured on 200k rows of which 500 were queued: the index carried 500 entries and the claimSELECTran in ~5 µs. The column order must matchORDER BY priority DESC, run_at ASCexactly; get it wrong and SQLite silently addsUSE TEMP B-TREE FOR ORDER BY, sorting the result set on every single claim. This is what makes retention a pure disk-space policy rather than a performance one. - Partial unique index for deduplication. Enforced by the database, not by a read-then-write in Python, which would race under concurrency.
- Timestamps are unix floats. Comparisons stay cheap, no timezone ambiguity in storage, and the presentation layer formats them.
Lifecycle¶
Job status¶
┌─────────┐
│ CREATED │ transient: the row is written as QUEUED (or WAITING)
└────┬────┘
↓
┌─────────┐
┌──────────────────►│ QUEUED │◄─────────────────┐
│ └────┬────┘ │
│ │ claim (run_at <= now) │
│ ↓ │
│ ┌─────────┐ │
│ │ RUNNING │ │ lease expired
│ └────┬────┘ │ (worker died)
│ │ │
│ ┌─────────────────┼─────────────────┐ │
│ ↓ ↓ └─────┘
│ ┌─────────┐ ┌────────┐
│ │ SUCCESS │ │ FAILED │ (this attempt)
│ └─────────┘ └───┬────┘
│ │
│ attempts < max
│ and retryable?
│ ╱ ╲
│ yes no
│ │ │
└───────────────────┘ ↓
run_at = now + backoff ┌────────┐
│ FAILED │ terminal
└────────┘
From QUEUED, HELD, WAITING or RUNNING, an explicit request moves the job to CANCELLED (terminal).
One more state sits beside QUEUED: HELD. A claim that finds a due job of a task at its limit,
as Limits explains, moves the job to HELD instead of taking it, and a later claim with
room for that task moves it to RUNNING. A held job is still in the queue, counts as queued, and can be
cancelled; it only stays out of the claim's way, so the claim never scans past jobs it cannot take.
WAITING sits before QUEUED: a job sent with after= is written as WAITING and moves to
QUEUED when the job it names succeeds, or to CANCELLED when that one fails for good or is
cancelled, as Jobs after another explains. A waiting job is not due, so no
claim considers it and no backlog figure counts it.
Terminal states: SUCCESS, FAILED, CANCELLED. Everything else is in flight.
Job status and Execution status are different enums, and they answer different questions. A job
whose attempt 1 failed is not FAILED. It goes back to QUEUED with a future run_at, while the
execution row for attempt 1 stays FAILED forever. Job.FAILED means one thing only: retries are
exhausted, this job is dead. That distinction is why a separate RETRYING job status is unnecessary.
Cancellation is included from day one. It costs an enum value and one predicate in the claim
query today; retrofitting it means migrating a live table later. Semantics: cancelling a QUEUED job
prevents it from ever being claimed. Cancelling a RUNNING job sets the flag and stops future
retries. The in-flight attempt is not killed, because Python cannot safely interrupt arbitrary
code. An attempt that finishes after the cancellation cannot report: the store refuses its outcome.
Execution status¶
RUNNING → SUCCESS | FAILED | TIMEOUT | LOST | CANCELLED
LOST is what the reaper writes when a lease expires: we know the attempt started, we will never
know how it ended. Keeping it distinct from FAILED matters: a spike in LOST means workers are
being OOM-killed, which is a completely different investigation from a spike in FAILED.
Layering: who knows what¶
user code
│
Task name, callable, default options
│
App registry, configuration, codec selection
│
Backend jobs, state, leases, persistence ← the plugin seam
│
Worker claim → execute → record → repeat
│
Executor async / thread / process
│
callable
The rule that keeps this honest: the Backend never knows how a function is executed, and the Executor never knows how state is stored. The Worker is the only component that touches both, and it is kept as dumb as possible: a loop.
The Scheduler sits beside the Worker, not above it. It knows about cron expressions and next firing times. It never executes anything; it only creates jobs.
Serialization¶
This is the decision with the longest shadow, so the reasoning is written out in full.
Decision¶
JSON with typed extensions is the default codec. Pickle is available, per-task or per-job, and is disabled unless explicitly enabled in configuration. The codec name is stored on every job row.
Why not pickle by default¶
Pickle is the obvious choice. It serializes almost anything, it is in the standard library, and it is what Celery originally did. It is still the wrong default, for four reasons that compound:
-
It becomes a remote code execution sink the moment the store is shared.
pickle.loadswill execute arbitrary code by design (__reduce__), so deserializing is equivalent to executing whatever produced the payload. In a local SQLite file that is roughly as trusted as your own source tree. But the Redis and Postgres backends are network services, frequently unauthenticated in development, occasionally exposed by accident. Anyone able to write a job row would gain code execution inside every worker. Choosing pickle would have designed in a vulnerability that opened by itself when those backends shipped. -
It throws away the main benefit of a database-backed queue. With JSON, diagnosing a bad job is
SELECT task, payload FROM tinyjobs_jobs WHERE status = 'failed'. With pickle it is an opaque blob that only a Python process with the right modules importable can even read. Nosqlite3shell, nogrep, no dashboard, no support engineer looking at a customer's database. -
It couples the payload to your code, exactly when you can least afford it. A job pickled by code that defines
Invoicewith three fields, unpickled by a worker still running the two-field version, raises on load. During a rolling deploy you have mixed worker versions by definition, that is what rolling means. JSON degrades (an unknown field is ignored); pickle fails closed. The same applies across Python upgrades and any refactor that moves a class between modules. -
The use case does not need it. Task arguments should be identifiers, not objects.
enqueue(123)notenqueue(document). This is not merely a serialization preference: a job may sit in the queue for minutes, then be retried ten minutes after that. A task carrying a snapshot of an object is operating on stale data by the time it runs, which is a correctness bug independent of how the snapshot was encoded. Reloading from the source of truth inside the task is the correct pattern, and it needs nothing richer than a primary key. A codec that cannot serialize your arguments is usually telling you the arguments are wrong.
What the default codec actually handles¶
Plain JSON is too thin: no datetime, no Decimal, no UUID. So the default codec is JSON plus a
tagged-value convention, roughly 60 lines of standard library:
{"$t": "datetime", "$v": "2026-08-31T17:34:00+02:00"}
{"$t": "decimal", "$v": "19.99"}
{"$t": "uuid", "$v": "..."}
{"$t": "set", "$v": [1, 2, 3]}
{"$t": "bytes", "$v": "<base64>"}
Covered out of the box: str, int, float, bool, None, list, dict, tuple, set,
frozenset, bytes, datetime, date, time, timedelta, Decimal, UUID and Path.
Anything whose reconstruction needs a class (a @dataclass, an Enum, a domain object)
must be registered, and this is a deliberate refusal rather than a gap:
app.codec.register_dataclass(Money)
app.codec.register_enum(Priority)
app.codec.register_type(MyAdapter())
The alternative would be to store a class path in the payload and import it on decode, which hands anyone able to write a job row an arbitrary-import primitive, and therefore arbitrary module-level code execution inside every worker. That is the same hole pickle opens, reached by a different route, so the same answer applies: nothing gets imported by name on the strength of the payload.
Encoding an unregistered type raises SerializationError at enqueue time, in the caller's own
traceback, naming the type and the three registration helpers. One line per type, once, and it makes
refusing pickle reasonable: an explicit, reviewable, safe conversion instead of an implicit one that
also executes code.
The pickle escape hatch¶
Some payloads genuinely resist a clean encoding, and a library that says "no" to them is a library people work around badly. So pickle stays, gated:
app = TinyJobs("sqlite:///jobs.db", allow_pickle=True) # off by default
@app.task(codec="pickle")
def train_model(model): ...
Enabling it logs a warning naming the security tradeoff. Backends that transport jobs over a network
may refuse it outright via configuration. Because codec is stored per job, none of this is a
one-way door: you can migrate task by task, in either direction, with a mixed queue in between.
Rejected alternatives¶
marshal: fast, stdlib, but the format is explicitly documented as unstable across Python versions. Unusable for anything persisted.- JSON only, no extensions: pushes
datetimehandling into every task in the codebase. - A bundled MessagePack/CBOR implementation: better wire size, but no better type coverage than extended JSON, and vendoring an encoder contradicts the zero-dependency goal for a gain that is irrelevant at this scale. Worth revisiting only as an optional codec if payload size ever becomes a measured problem.
The backend protocol¶
The seam that makes storage swappable. It is narrow on purpose, and async on purpose; see Public API for why.
class Backend(Protocol):
capabilities: Capabilities
# --- lifecycle -------------------------------------------------------
async def setup(self) -> None: ... # create schema, run migrations
async def close(self) -> None: ...
# --- producer side ---------------------------------------------------
async def enqueue(self, job: Job) -> Job: ...
async def enqueue_many(self, jobs: Sequence[Job]) -> list[Job]: ... # see note
# --- consumer side ---------------------------------------------------
async def claim(self, *, worker_id: str, queues: Sequence[str],
limit: int, lease_seconds: float,
limits: Mapping[str, Limit] | None = None) -> list[Job]: ...
async def heartbeat(
self, job_ids: Sequence[str], lease_seconds: float, worker_id: str | None = None
) -> Collection[str] | None: ... # the ids whose lease it renewed
async def complete(self, job: Job, result: bytes | None) -> bool: ...
async def fail(self, job: Job, error: ExecutionError, retry_at: float | None,
execution_status: ExecutionStatus = ExecutionStatus.FAILED) -> bool: ...
async def release(self, job: Job, delay: float = 0.0) -> bool: ...
async def release_many(self, jobs: Sequence[Job]) -> int: ...
async def hand_back(self, job: Job, delay: float = 0.0) -> bool: ...
async def release_held(self, tasks: Collection[str],
queues: Sequence[str] | None = None) -> int: ...
# --- control ---------------------------------------------------------
async def cancel(self, job_id: str) -> bool: ...
async def requeue(self, job_id: str) -> bool: ...
async def get(self, job_id: str) -> Job | None: ...
async def list(self, *, status: JobStatus | None = None, queue: str | None = None,
limit: int = 100, cursor: str | None = None) -> Page: ...
async def executions(self, job_id: str) -> list[Execution]: ...
async def counts_by_status(self) -> dict[str, int]: ...
async def stats(self, now: float) -> Stats: ...
# --- schedules --------------------------------------------------------
async def schedules(self) -> list[ScheduleState]: ...
async def register_schedules(self, states: Sequence[ScheduleState]) -> None: ...
async def fire_schedule(self, name: str, *, expected: float, next_run_at: float,
jobs: Sequence[Job]) -> list[Job] | None: ...
# --- housekeeping -----------------------------------------------------
async def reap_expired(self) -> int: ...
async def purge(self, policy: RetentionPolicy, now: float) -> int | None: ...
async def expire_results(self, task: str, ttl: float, now: float,
batch: int = 500, since: float = 0.0) -> int: ...
BaseBackend gives defaults for setup, close, enqueue_many, hand_back, release_many,
release_held, purge, expire_results and the three schedule methods, so a store writes the
other fourteen. It also defines follow_clock, which a store that reads its own server's time
overrides. Beside the protocol the core looks for a few names a store may leave out, and does
without them:
claim_job(job_id, *, worker_id, lease_seconds), which lets an eager call run the one job it sent;wait_for_jobs, which lets a worker sleep until a job arrives instead of polling;transient_errors, a tuple of the store's own exceptions that mean an outage, through which a worker keeps retrying a job's outcome instead of losing it;reserve, which tells a pooled store how many jobs a worker runs and returns, as a whole number of at least 1, how many rounds of hand-backs the worker should allow for when it stops, andspare, which has the store size its pool to what was reserved;audit(), which answerstinyjobs check;refuse_held_transaction, with which SQLite refuses a write made while the caller holds a transaction on the same file;- for a store that declares
transactional_enqueue,enqueue_in(tx, jobs)andaenqueue_in(tx, jobs).
These are found by name, not declared, and they are the part of this seam most likely to change before 1.0 (see Compatibility).
enqueue_many earns its place as a distinct method rather than a loop in the core, because the
transaction boundary is the optimization. A thousand separate enqueue calls on SQLite means a
thousand fsyncs; the same thousand inside one transaction means one, which is two orders of
magnitude. Redis backends will want pipelining, Postgres a multi-row INSERT. None of that is
reachable if the core does the looping. After v1.0 a new method could only arrive with a default, so
it goes in now, with a default implementation in BaseBackend (a loop over enqueue()), so that
simple backends get the correct behaviour for free and the bundled stores override it from the
start.
claim_job claims one job by its id, only while it is queued and due, which is how an eager call
runs the job it sent and no other (see Testing). It is optional: without it an eager
call drains every due job, as earlier releases did. It is a lookup by
primary key, under FOR UPDATE SKIP LOCKED on Postgres: 200 rounds of three apps each racing it
against a plain claim for the same job gave the job to exactly one of them on SQLite, Postgres and
Redis, and one that finds nothing to claim takes 0.2 to 0.3 ms.
A backend may also declare transient_errors, the exception types that mean the store could not be
reached. When complete or fail raises one, the worker writes again after 0.25 s, doubling up to
4 s, until a second before its lease would run out, where the last try lands. Every outcome write
checks that the worker still holds that claim, so a write repeated after its reply was lost changes
nothing, and the worker then reads the execution of its own claim to learn whether the first one
landed; that read is retried the same way. A job whose task returned but whose outcome is still
being written when the worker shuts down is not handed back to the queue: its lease runs out and the
reaper decides, so max_attempts still holds. Redis and Postgres declare theirs, and SQLite declares
a write lock that another process held past busy_timeout: four 7 second locks taken by another
process while three workers drained a stream of jobs lost 3 outcomes of 1308, whose jobs then ran
twice, and now lose none. A claim that meets that lock is a one-line warning like any other
transient claim error. The error keeps the SQLite error code and name of the one it replaces.
The contract a backend must honour¶
The signatures are the easy part. These invariants are what an implementation is actually promising, and the conformance suite that checks them ships with the package, so a plugin runs the same tests as the bundled backends:
from tinyjobs.testing import BackendContract
class MyBackendTests(BackendContract, unittest.IsolatedAsyncioTestCase):
async def make_backend(self, clock):
return MyBackend("my://test", clock=clock) # a fresh, empty store per test
It drives the backend through a FrozenClock, so leases, delays and retention are tested without
sleeping, and it races several clients against one store for the claim, dedup and cron invariants.
Tests for a capability the backend does not declare are skipped. A backend whose clients cannot share
a store, like memory://, sets independent_clients = False and races one client against itself.
claimis atomic. A job is handed to exactly one worker. Two concurrent claims never return the same job. This is the one hard requirement; everything else is comfort.claimrespectsrun_at. Never return a job scheduled for the future.claimrespectsstatus. Never return a cancelled or terminal job.- A claimed job carries a lease. If the holder stops calling
heartbeat,reap_expiredeventually makes the job claimable again. complete,fail,releaseandhand_backare idempotent, and all are no-ops on a job the caller no longer holds. The holder is the worker and the claim, a per-job counter that only ever grows, so a run that stalls past its lease, or was cancelled and retried, cannot report for the run that replaced it, even when the same worker claimed the job again. Executions are keyed by claim too, so a retried job keeps the history of its earlier runs, and cancelling a running job marks its executioncancelled.enqueuewith a duplicate activededup_keyreturns the existing job rather than creating a second one.fire_scheduleis a compare-and-set. It advancesnext_run_atonly if it still holdsexpected, and inserts the jobs in the same transaction, so a firing is either fully recorded or not at all. A caller that loses the race getsNoneand creates nothing.register_scheduleskeeps a storednext_run_atunless the cron or the timezone changed. That is what makes a restart pick up where it left off, and what lets the misfire policy see the firings it missed.
Capabilities¶
Not every store can do everything. A Redis list has no cheap priority ordering, for instance. Backends declare what they support so the core can degrade predictably instead of failing at runtime:
@dataclass(frozen=True, slots=True)
class Capabilities:
priority: bool = True
delayed: bool = True
dedup: bool = False
transactional_enqueue: bool = False
retention: bool = False
schedules: bool = False
max_payload_bytes: int | None = None
loop_bound: bool = False
result_ttl: bool = False
limits: bool = False
after: bool = False
A store names only what differs from the defaults, so a capability added later leaves it as it was.
max_payload_bytes is declared and not yet enforced by the core.
Requesting an unsupported feature raises at enqueue time with a clear message, not silently later.
limits says the store enforces a task's concurrency= and rate= at the claim, which then takes
limits=, a mapping of task name to Limit; a worker or a drain whose app registers a limited task
refuses a store without it at start, since enqueue is not where the limit would be lost.
after says the store keeps a job sent with after= waiting and settles it when its parent ends;
without it the enqueue raises.
loop_bound is not a feature but a constraint: a backend whose connections belong to the event loop
that opened them, as an asyncio Redis or Postgres client's do, sets it, and the app opens it on the
facade's loop and routes every call there (see Async core, sync facade).
Plugin discovery¶
Third-party backends register through entry points. importlib.metadata is stdlib, so this stays
dependency-free:
# in tinyjobs-redis's pyproject.toml
[project.entry-points."tinyjobs.backends"]
redis = "tinyjobs_redis:RedisBackend.from_url"
The entry point names a factory, which the app calls as factory(url, clock=clock, **options):
the URL it was given, the app's clock, which the store reads instead of time.time(), and the
keyword options passed to TinyJobs.
The user's side of the swap, in full:
app = TinyJobs("sqlite:///jobs.db") # before
app = TinyJobs("redis://localhost/0") # after
The URL scheme selects the backend. A scheme the package does not bundle is looked up in the entry
point group, with a helpful error if nothing is registered for it. A few answers still come from the
bundled schemes only, such as which stores Django can join a transaction on, which are shared
between processes, and which get a connection pool of their own in worker.run() and may run more
than one loop; a plugin cannot give them yet. Bundled: sqlite://, memory://, redis:// behind
the redis extra and postgres:// behind the postgres extra, each client imported only when its
URL is used.
The SQLite backend¶
The default, and the one that has to be right. SQLite is a good queue substrate. It is transactional, it fsyncs, and in WAL mode readers never block the writer, provided you respect its one constraint: a single writer at a time.
Connection setup¶
PRAGMA journal_mode = WAL; -- readers don't block the writer
PRAGMA synchronous = NORMAL; -- a process crash loses nothing, a power cut the last commits
PRAGMA busy_timeout = 5000; -- wait for the write lock instead of failing
PRAGMA foreign_keys = ON;
Every write transaction is opened as BEGIN IMMEDIATE. Taking the write lock upfront turns lock
contention into a bounded wait instead of a mid-transaction SQLITE_BUSY and a forced retry, the
classic source of "database is locked" in multi-process SQLite.
One connection per thread, kept in a thread-local, never shared across threads.
A cancelled call still queued behind another is taken off the backend's thread at once; one
already running gets half a second, then the cancel goes on while it finishes in that thread. A
release at shutdown is one write transaction for every job. With another process holding the write
lock, four jobs released at shutdown let the app close 6.5 seconds after the signal, where each
release waited out busy_timeout behind the others and the close took 22.6.
A claim inserts its execution rows in one executemany and builds its jobs after the commit, which
cut its median hold of the write lock from 101 to 89 µs. Every bound on a call shares one timer per
event loop instead of arming its own, and an outcome write enters its bound without an await. Four
processes of eight slots drain 30,000 no-op jobs from one file at 5,233 a second, against 5,007 before
the last of these changes and 5,232 before the shutdown budget (medians of twelve interleaved rounds).
Inside a worker or a closing app, BEGIN IMMEDIATE waits for the lock a tenth of a second at a
time, up to busy_timeout, and stops once the shutdown budget is spent, since interrupt() does
not end SQLite's own wait and the exiting interpreter joins the backend's thread: with the write
lock held elsewhere and a 2 second shutdown timeout, the CLI worker exited 1.81 to 1.97 seconds
after the signal, where it took 6.2 to 6.3 (7.7 at 4 seconds, now 3.85 to 3.89). A write cut by its
own deadline still waits, so an outcome cut on a lock lands once the lock is let go. Setting the
timeout around every write made the SQLite worker and drain about 2 percent slower, so a WAL
connection keeps the short timeout between writes: another statement that finds the lock taken there
is retried until busy_timeout, and a write made without a shutdown budget sets the full timeout
back first. Without WAL the timeout is set back after each BEGIN.
Switching a fresh file to WAL needs an exclusive lock, and SQLite answers "database is locked" at
once instead of honouring busy_timeout. Two workers starting together on a new file lost that race
in 11 of 60 tries, so the switch is retried until busy_timeout runs out.
Tables and the prefix¶
The five tables are tinyjobs_jobs, tinyjobs_executions, tinyjobs_schedules,
tinyjobs_meta and tinyjobs_limits, with the empty tinyjobs_install_lock beside them, the indexes
tinyjobs_jobs_claim and so on, and prefix= renames them all, as on
Postgres and Redis, so several apps share one file and a file shared with an application's own
tables, which a transactional enqueue needs, collides with nothing: jobs and meta are names an
application is likely to have taken, and before 0.14 tinyjobs took them. The prefix follows the
Postgres rule, lowercase letters, digits and underscores, forty characters at most.
A file from before 0.14 holds the bare names. The first 0.14 process to open it with the default
prefix renames the four tables in place, which SQLite does without copying and while rewriting the
foreign key in executions, drops the old idx_* indexes for the DDL to create them under their
new names, migrates whatever older schema version it finds, and leaves one thing behind: a bare
meta table holding schema_version 4 and nothing else. A worker from 0.7 to 0.13 reads meta
first and refuses a version newer than its own, so it stops with "upgrade the package" instead of
creating its four bare tables beside the renamed ones and running a second queue in the same file;
one from before 0.7 creates its bare tables, empty, and refuses right after, which leaves nothing
in them, and a bare meta at version 4 or more is read as that stub, never as a queue to migrate.
A legacy queue is recognised by its shape, not by its names alone: bare meta, jobs and
executions with the columns tinyjobs gave them and a schema_version from 1 to 3. An
application's own jobs and meta tables, names that are common, are left alone and the
prefixed tables are created beside them.
A file created by 0.14 has no such stub, since a meta table in an application's database is the
collision the prefix exists to avoid; an older worker pointed at such a file would start a queue of
its own, which is a downgrade, not an upgrade, and is not supported. A legacy queue is migrated
into the default prefix only: opening such a file with another prefix is refused with the reason,
since two apps with two prefixes could otherwise each claim the old tables for their own.
The claim¶
The heart of the system. On SQLite >= 3.35 it is one statement, atomic by definition:
UPDATE tinyjobs_jobs
SET status = 'running',
worker_id = :worker,
attempts = attempts + 1,
lease_expires_at = :now + :lease
WHERE id IN (
SELECT id FROM tinyjobs_jobs
WHERE queue IN (:queues)
AND status = 'queued'
AND run_at <= :now
ORDER BY priority DESC, run_at ASC
LIMIT :batch
)
RETURNING *;
Verified against SQLite 3.51. On older libraries (Debian 11 ships 3.34) the backend falls back to
SELECT ids, then UPDATE ... WHERE id = ? AND status = 'queued' inside BEGIN IMMEDIATE, checking
rowcount to detect a lost race. Same guarantees, one extra round trip. The version check happens
once at setup().
Polling¶
SQLite has no LISTEN/NOTIFY, so workers poll. Naive polling either burns CPU or adds latency, so the
interval adapts: after a claim that returned work, poll again immediately; after an empty claim, back
off geometrically from ~5 ms to a configurable ceiling (default 500 ms), with jitter so that N workers
starting together do not synchronize into a thundering herd.
For the local single-machine case polling could be helped along: a wakeup file descriptor would
let an in-process enqueue nudge a sleeping worker directly, dropping enqueue-to-start latency
to sub-millisecond without changing the polling fallback that covers cross-process enqueues. It
is not built; Postgres gets the same effect from LISTEN/NOTIFY.
Crash recovery¶
A claimed job holds a lease. The worker refreshes it (heartbeat) at one third of the lease duration
while the job runs, naming itself, so a worker that lost the job cannot extend the lease of the one
that holds it now. A plugin gets worker_id only when its heartbeat names that parameter; an
older two-argument one, or one hidden behind a *args, **kwargs wrapper, is called without it.
heartbeat returns the ids whose lease it renewed; a job missing from that answer, and not already
recording its outcome, is logged once as no longer held, by id, and left out of later heartbeats. A
heartbeat or a claim that runs out of time or hits a transient error is a one-line warning, not a
traceback.
An answer that is not a collection of ids, such as a plugin's count, is logged once and the worker
keeps beating without tracking lost leases; a count used to end the heartbeats silently. A heartbeat
unanswered after a quarter of the lease is joined by a second one, without cutting the first, and
the two get until the lease they renew would lapse: with the first call of every heartbeat never
answering, the second renewed the lease with no warning. A cut call costs Redis its connection, and a
fresh one three round trips more, so on a slow link one late heartbeat used to make every later one
late too.
On Redis over a link adding 300 ms each way, with a 6 second lease and eight jobs at a time, 218
jobs ran once each with no heartbeat error, where 27 heartbeats used to fail and a job whose outcome
got a second and a half ran twice. Over 70 seconds on that link, eight jobs ran once with no
heartbeat warning, where from the first clock reading due all 20 heartbeats failed. If
the worker is killed by SIGKILL, by the OOM killer or by a yanked power cable, the lease
simply expires.
Any worker can run the reaper; it is another single atomic statement:
UPDATE tinyjobs_jobs
SET status = CASE WHEN attempts < max_attempts THEN 'queued' ELSE 'failed' END,
lease_expires_at = NULL, worker_id = NULL,
run_at = :now
WHERE status = 'running' AND lease_expires_at < :now
RETURNING id;
The abandoned Execution rows are marked LOST. This is precisely the at-least-once window: the job
may have completed its side effects before dying. Hence the idempotency contract.
Housekeeping¶
Retention is on by default, and it is a disk-space policy, nothing more. The partial claim index means history never touches the hot path, so "how much do I keep" is decoupled from "how fast is my queue". That is measurable: claiming a job takes the same 11 microseconds whether the table holds nothing or half a million finished jobs. Defaults:
| Status | Kept for | Why |
|---|---|---|
success |
7 days | Enough to answer "did it run last week?" |
cancelled |
7 days | Same |
failed |
30 days | These are the ones you come back to |
Set any of them to 0 to keep forever, or use KEEP_EVERYTHING to disable pruning entirely.
The sweep is claimed the same way a schedule is fired, with one conditional UPDATE on a marker row
in tinyjobs_meta:
UPDATE tinyjobs_meta SET value = :now
WHERE key = 'last_purge_at'
AND (CAST(value AS REAL) = 0 OR CAST(value AS REAL) <= :now - :interval)
RETURNING key;
Exactly one worker's UPDATE matches, so forty workers still produce one sweep per interval with no
leader to elect. The = 0 arm exists because a fresh database starts the marker at zero, and without
it the first sweep would never be due on a clock whose value is smaller than the interval.
Deletion happens in bounded batches, oldest first, each its own short write, so a large backlog never
becomes one transaction that stalls every claim. A sweep stops at 50,000 jobs, and the worker that
reached that cap starts another at once, without waiting for the interval, until a sweep comes back
under it; the others still wait for the marker. Before this, one sweep an hour capped pruning at 1.2
million jobs a day, so a queue finishing more than about 14 jobs a second kept its history forever,
and failed jobs, swept after success, were never reached: in a soak finishing 1,600 jobs a second
on Redis, 3,500 failed jobs were still kept past their retention. executions rows
follow their job through ON DELETE CASCADE, which the connection enables with
PRAGMA foreign_keys = ON.
Two indexes make this cheap, and both follow the same rule as the claim index: match the query, or
SQLite silently sorts. tinyjobs_jobs_purge is (status, finished_at) WHERE finished_at IS NOT NULL,
which matches the sweep exactly, and being partial it only holds terminal rows. Since a terminal job
is never updated again, only deleted, that index costs one write per job and nothing on the claim
path. Getting it wrong turns the sweep quadratic: measured on 100,000 expired jobs, with the index
0.22 seconds, without it 4.45 seconds, and the gap widens with size.
Pruning does not shrink the file. SQLite puts the freed pages on a freelist and reuses them, so the
database plateaus rather than shrinking, which is exactly what a soak run shows: flat at 6.9 MB for
as long as you care to watch. Returning space to the filesystem needs VACUUM, which wants exclusive
access, so it stays a manual command and never runs on its own.
Where the limits are¶
Expect low thousands of jobs per second for enqueue and claim on a local SSD, dominated by fsync
rather than by Python. That covers most applications. It will not cover a high-volume ingestion
pipeline, and it degrades on network filesystems. SQLite's locking over NFS is unreliable, so the
file belongs on a local disk, which the audit in tinyjobs check verifies on Linux. Both cases
are the reason the backend seam exists.
The Redis backend¶
Bundled rather than a separate tinyjobs-redis package: until the protocol stops changing
incompatibly at v1.0, a change to it would otherwise need two releases in lockstep. It depends on
nothing but the public protocol, so it can move out later without touching a line. The client
library is an extra, tinyjobs[redis], and is imported only when a redis:// URL is used, so the
core stays dependency-free. A url redis-py cannot read raises ConfigurationError, without its
reason once the url holds a password, since that reason may quote part of it. So does a connection
option the client does not take, named unless the url holds a password, before the host or as a
password= option: a raw ? in the password made the rest of the url such an option, which
redis-py raised at the first command as a TypeError quoting the password, and a raw & in a
password= value made the part after it one. A unix:// url whose socket path holds an @ behind
a password is refused too, since redis-py took the rest of the password for the path and quoted it
when the connection failed. So are decode_responses, which broke every call, and a timeout= that
is not a number, which redis-py 5.0 kept as text and failed every call with; one that is a number is
read as one. So are the client's own retries (retry with any retries, retry_on_timeout,
retry_on_error), on the url or as keywords. A client handed in, redis-py 8's default included,
gives tinyjobs its settings, not its connections: tinyjobs opens its own pool from them with no
retries and leaves the client as it was, since redis-py keeps the retry on each connection and
pinning a shared pool would take the retries from other code using it. A call sent again after its
answer was lost runs its script twice, so one enqueue counted two queued jobs and the complete
sent again found the job no longer running, returned false and left the counts at one queued and
one success. Refusing them is simpler than making every script safe to run twice, and tinyjobs
already recovers a lost answer itself: outcomes are written again only after checking the claim, and
a lost claim is handed back by its token.
On Python 3.12 a command written to a connection the server had reset raised asyncio's
TypeError: 'NoneType' object is not callable, a bug fixed in 3.13, so a job whose outcome met it
was failed as if the error were its own: 11 of 2,000 jobs through a 20 second outage ended failed.
That exact error, raised by asyncio's write, now fails the call as a lost connection, which is
retried, and all 2,000 ended success. psycopg does not write through asyncio transports, so
Postgres never meets it.
Atomicity¶
Every state change is one Lua script: enqueue, claim, complete, fail, release, hand back, cancel, requeue,
heartbeat, reap, a purge batch, a cron firing. Redis runs a script without interleaving any other
command, which gives each transition the guarantee SQLite gets from BEGIN IMMEDIATE. The scripts
share one library, and in it a single pair of functions, leave and enter, knows which indexes
each status lives in. A transition is leave(old), write the hash, enter(new), so no script can
forget an index.
Data model¶
All keys sit under {prefix}, which by default is {tinyjobs}:
| key | type | holds |
|---|---|---|
job:<id> |
hash | the job's fields, the same columns as the SQLite table |
executions:<id> |
hash | attempt number to a JSON record |
ready:<queue> |
sorted set | due jobs, score -priority, member run_at\|id, so order is priority then age |
waiting:<queue> |
sorted set | queued jobs not due yet, by run_at; a claim promotes the due ones first (the name predates the waiting status and has nothing to do with it) |
queued:<queue> |
sorted set | every queued job by run_at, for the oldest-due figure |
leases |
sorted set | running jobs by lease expiry, for the reaper |
finished:<status> |
sorted set | terminal jobs by finished_at, for retention |
index:all, index:status:<s>, index:queue:<q> |
sorted sets | ids for listing; UUIDv7 ids sort by creation |
dedup:<key> |
string | the active job holding a dedup key |
children:<id> |
set | the jobs waiting on a job, settled when it ends |
counts, queued_tasks |
hashes | counters kept by enter and leave, so stats is O(queues) |
schedule:<name>, schedules |
hash, set | cron state; firing compares next_run_at and swaps it |
Every machine keeps Redis's time. The app's default clock is a FollowingClock: once the backend
has read Redis's TIME, now() is that reading plus the time elapsed since, counted on a clock that
never steps and keeps counting through a suspend (CLOCK_BOOTTIME on Linux, CLOCK_MONOTONIC on
macOS). A step of the wall clock therefore changes nothing, whether it lands between two calls or
during a reading. The backend reads TIME again before a call goes on after a connection error or
when the wall clock and the elapsed clock disagree by half a second. A reading thirty seconds old,
or one whose round trip was slow, is read again beside the call that finds it due, which goes on
with the reading it has, and one a minute old before the call. Only the
TIME command itself is timed, on a connection already taken from the pool and opened. A reading
whose round trip took more than half a second only renews the reading held when the two agree
within half that round trip plus a quarter of a second; when they do not, it is taken, and either
way the next try waits a second. An older reading never replaces a newer one. A reading is only
as exact as half its round trip, so a run_at built on a slow one can be up to a quarter of a
second early: under a soak that held a four-core host at a load of 40, 2,400 delayed jobs started
0.06 to 0.24 seconds before their delay was up, and none earlier. Without this a host an hour ahead reaped every live lease the others held.
A host can still be on a stale reading for a moment: after a failover to a node whose clock differs,
until its next reading, or during a suspend its elapsed clock did not count. Leases do not depend on
it. A following host marks its now with a ~, and for it the scripts read TIME themselves: the
reaper takes the earlier of the two, so it never reaps by a time ahead of Redis's, and a claim or a
heartbeat stamps the later of the two lease ends, so no lease is written to end before Redis says it
should. Everything else written in that moment, a delayed run_at or a cron slot, is off by the
stale reading until the next one, at most a minute later; the app also reads again before it
builds a job whenever the reading is not to be trusted. The one thing a following host cannot survive is Redis's own
time jumping ahead, as after a failover to a node whose clock is ahead: every lease stamped before
the jump by more than its length looks expired and is reaped, so its job runs again, or fails when it
has no attempts left. Redis's nodes therefore stay on NTP.
Instants the caller supplies, a run_at and Retry(at=...), are read on the local wall clock and
moved by the difference between the two. A clock the caller passes, FrozenClock or any object with
now() and monotonic(), is used as it is and never follows, which is how the contract suite drives
Redis by hand. With the default clock job ids take their time from Redis's too, so listings put the
newest job first whichever host sent it; with any other clock they take the wall clock. Readings due at the same moment share one round trip, and a clock shared by
apps on two servers whose times differ by more than a second raises instead of mixing them.
Where the limits are¶
- Persistence is Redis's job. On its default snapshots a crash loses everything since the last
one, which in testing was the whole queue.
appendonly yeswith the defaulteveryseclost nothing when Redis was killed mid-run. The Redis page says so where people will read it. - One hash slot. The hash tag puts every key in the same slot, which is what lets a script touch them all. It runs on a single Redis or a primary with replicas; a cluster would hold the queue on one node.
- Script length bounds latency. Redis serves nobody else while a script runs. Enqueues go in batches of 100 and a claim promotes at most 100 due delayed jobs, which kept the slowest call at 18 ms when 100,000 delayed jobs fell due at once. Claiming stays at 0.7 ms with 200,000 queued. The one script without a bound is the one that settles the jobs waiting on a job that ends: cancelling a parent with 20,000 jobs waiting on it took 0.27 seconds, 13 microseconds a child.
- Polling, not blocking. Workers poll with the same adaptive interval as on SQLite. A blocking pop would save the idle round trips but cannot express priorities across queues.
- No transactional enqueue, ever. Redis shares no transaction with your database, so the Postgres feature has no Redis counterpart.
- A bounded pool that waits. redis-py 8 caps its pool at 100 and raises
MaxConnectionsErrorthe moment a caller finds it full, which turned half of 200 concurrentaenqueue()calls into errors. The backend uses a blocking pool of the same size instead, so 400 concurrent requests to a FastAPI endpoint all succeeded, and a caller only fails afterpool_timeout. redis-py 5.0.1 connects while holding that pool's lock, so a call cancelled mid-connect held it untilpool_timeout(2.00 s in the test, 20 s by default), stalled every other call, and kept its connection counted as in use. On 5.0.1 the backend swaps in the 5.0.2 acquire, which connects outside the lock: the cancel returns at once and the connection goes back.
The Postgres backend¶
Bundled like Redis, behind the postgres extra (psycopg and psycopg-pool), and imported only
when a postgres:// or postgresql:// URL is used. It runs on psycopg 3's async API with an
AsyncConnectionPool, so it is loop_bound and the app routes every call onto its own loop.
Tables¶
Five tables, named after the prefix and optionally inside a schema=: <prefix>_jobs,
<prefix>_executions, <prefix>_schedules, <prefix>_meta and <prefix>_limits, with the same
columns as SQLite except that <prefix>_limits also carries the running count of each task.
Before setup reads any of them it finds them in the catalog, along the schemas the connection's
search_path named when the pool opened, passing over a table of a tinyjobs name that a role it
does not trust owns, or sits in a schema such a role owns, when a trusted one of that name comes later
on the path, with a warning naming the schema, so a role allowed to create schemas cannot stop new
processes by planting one earlier. It checks that all five sit in one schema and that the
tables and the schema belong to the connecting role or to a role that can become it (SET on
Postgres 16 and later, membership before), and that no role inherits the privileges of without
being able to become it too; a schema owned by pg_database_owner counts as the database owner's. Postgres runs a trigger, a policy or a view's function with the privileges of the
role reading or writing the row, so a table owned by anyone else would let that role act as
tinyjobs. For the same reason a table is refused too when it grants TRIGGER, or REFERENCES on
the table or any of its columns, to such a role or to PUBLIC, since a foreign key from that role's own table would make a purge fire
its triggers, or when code that runs on it belongs to one: a function behind a trigger, constraint,
default, generated column, index or policy, an operator, a domain or type, or an index's operator
class. Setup and every tx= call find that code by following, in pg_depend, the objects attached
to the tables down through every object a user created, and the same goes for each table that
references them by a foreign key, whose triggers a cascade fires, and each table that inherits from
them, whose rows tinyjobs' statements reach. A table they reference, such as a lookup, runs nothing
of its owner's on tinyjobs' statements and is not counted, and neither is a view, a function or a stored
regclass that merely names the tables runs nothing during tinyjobs' statements and is not counted,
so no role can block the tables by creating one. The guard stays near a tenth of a millisecond. Such tables are refused before a row is read, even
when tinyjobs connects as a superuser, was made a member of their owner, or the database belongs to
someone else. With no tables yet, the schema they would be created in is checked the same way, and
created only when it does not exist, so a role that owns its schema needs no CREATE on the
database. Setup
then reads the stored schema version and creates nothing when it matches, so a fleet starting at
once costs a few catalog reads and one SELECT each. Otherwise it takes a transaction-scoped
advisory lock and checks again under it. Any role that may connect can take an advisory lock, so
one held by a session whose role cannot become setup's own is not waited for, a lock let go between
the attempt and the reading of its holders is tried once more before that is concluded, and one
held for a minute, or what is left of setup's deadline, whatever the statement_timeout, is waited
out rather than obeyed: setup
creates the tables without it, and a concurrent creation it then trips over sends it round again,
up to three times. The wait blocks on the lock rather than polling a clock: four workers starting
together on a fresh prefix over a link adding 300 ms each way started quietly, one after another,
where three of four created the tables without the lock after five seconds and two of them blamed
a role that could not become theirs. A lock held 45 seconds was waited for under a one second
statement_timeout, where the timeout failed setup after 1 second, and after 30 by default.
Between the CREATE statements and the rows the meta table is seeded with, setup checks the placement and the
owners once more in the same transaction, so a table another role creates in the gap is caught
before tinyjobs writes a row into it, and the whole creation rolls back. A store written by a
newer release is refused and left alone. Every setup then checks that each tinyjobs index exists as
a valid index of the right table, unique where it must be, and that the statistics below exist on
the jobs table, so an object of that name another role made first stops setup instead. One that is
missing, invalid or of the wrong kind is rebuilt with CREATE INDEX CONCURRENTLY, on a connection
of its own while a second one holds the meta table's instance row FOR UPDATE, so workers keep
claiming and enqueueing while it builds. Only a role that may write the meta table can take that row
lock, though one allowed to LOCK it, as MAINTAIN allows, can hold the table; each attempt waits
half a second for the table, and neither connection is ended by transaction_timeout or
idle_in_transaction_session_timeout. Other setups starting meanwhile wait up to 30 seconds for
that rebuild instead of racing it, and do not wait on the table again for what it left: a plain
index is kept for the next setup with a warning and a unique one stops setup with the reason, naming
the pid, role and application of each session still holding the meta table. The executions
index claims insert against is the exception: claims cannot run without it, so it is built in one
plain step, during which they wait instead of failing. Unique indexes go first. A build waits up to
30 seconds for every transaction open in the database when it started, since that is what
CONCURRENTLY waits for, whether or not they touch the table; past that a unique index stops setup
with the reason and the oldest sessions it waited for, and the plain ones left are kept for the next setup with a warning, without waiting
again, since jobs run without them, only more slowly. If active jobs with the same dedup key got in
while that index was missing, setup names those jobs and asks to cancel all but one of each group.
CREATE INDEX IF NOT EXISTS skips a name another relation already holds, which would leave claims
failing and dedup keys ignored, so then setup names the index and stops, and the next setup after
that relation is dropped creates it. From then on every statement names the tables with the
schema setup found them in, so a table of the same name created later earlier on a search_path
cannot take the jobs, and tinyjobs' own connections run with search_path set to pg_catalog,
pg_temp, so no function or operator someone created elsewhere is ever called. The statements that
run on a caller's connection for tx= name every function, type and operator with pg_catalog, so
the caller's search_path cannot change them either. The caller may connect as another role, or
switch with SET ROLE between two calls, so every tx= call runs the ownership check through the
caller's connection, against the role current at that moment, before anything touches the tables;
a role the tables' owner does not answer to never writes through them, and none of the owner's
triggers, defaults or index expressions runs with its privileges. A tx= call holds the
connection's lock from the check to its last statement, as psycopg's own executemany does, so no
other statement on that connection, such as a RESET ROLE from another task or thread, lands
anywhere inside it. The check and the first insert share one pipeline sync, so the check costs no
round trip (a libpq older than 14 has no pipeline mode and spends one); a failing check makes
Postgres skip the insert unparsed. Inside a caller's own pipeline they join it without a sync, so an
autocommit pipeline still commits as one; there the check is read before the insert is sent, so a
refusal sends nothing more and leaves the pipeline as any failed statement does, for the caller to
sync. The insert sent behind the check is never prepared: psycopg counts a statement Postgres skipped
as prepared, and on an autocommit connection no rollback would clear that, so every later call
would fail. A tx= call that needs more than one batch on an idle autocommit connection runs in a
transaction of its own, so it stores all its jobs or none, as an enqueue through the pool does. Values are bound by the server, arrays in binary, so neither the caller's search_path
nor its standard_conforming_strings or client encoding changes what runs or what is stored. This
uses psycopg internals present in 3.2 and 3.3; on a psycopg without them tx= raises
UnsupportedCapabilityError rather than run unguarded. A connection whose client encoding is
SQL_ASCII is refused, since psycopg reads its text as bytes. A guard inside the insert
statement would not do: a statement-level BEFORE trigger fires before any row reaches the guard,
and the planner may evaluate an immutable column default before the statement starts. A refused
call leaves the caller's transaction aborted. Setup also reads the store's identity: a
random instance id written into the meta table when the tables are created, the database's oid and,
when the role may read it, the cluster's system identifier. A copy made with CREATE DATABASE ...
TEMPLATE or a restored dump keeps the instance id but not the other two. A tx= connection whose
identity differs makes tinyjobs read its own again on a fresh connection of its own, with the same
settings and timeouts as the pool's and a connect timeout, before refusing, so an app that rides
through a move of the same store, such as a switchover to a replica, follows it.
id, queue, status and dedup_key use the C collation, so ids sort as bytes, the way UUIDv7
ids are ordered on SQLite and Redis, whatever the database's locale. A btree entry holds about
2.7 kB, so the unique index covers a dedup key's SHA-256 digest, stored beside it, and the queue
indexes cover a queue's first 500 characters, with the full name compared on top; neither has a
length limit. Queues longer than that which share their first 500 characters share index entries,
so each is claimed by reading its siblings' rows too. Extended statistics tell the planner that the
prefix and the name go together; without them it multiplies the two conditions' selectivities, and
after a few prepared listings by queue and status Postgres picked a generic plan that read the whole
primary key, 14 ms instead of 0.12 at 300,000 jobs. Many jobs go in batches of at most 1,000 jobs and 64 MiB of payloads, keys and errors,
in one transaction, so no array parameter nears Postgres' 1 GB limit on a value. The jobs table has a fillfactor
of 85 and an autovacuum tuned to run often: a job row is updated several times in its life, and the
free space lets most of those updates stay on the same page. There is no index on the lease, so a
heartbeat touches no indexed column and is a HOT update; the reaper scans the running jobs instead,
which are few.
The claim¶
A claim is one statement. A recursive query walks the priorities present in the wanted queues from
the highest down, using the partial index on (left(queue, 500), priority DESC, run_at) over queued
jobs, and
for each level that holds a due job takes the due jobs with FOR UPDATE SKIP LOCKED, oldest first,
until it has enough. Each step reads a span of index entries per queue below the last level, in index
order, and either finds the highest level with a due job or skips every level the spans covered,
since they hold each such level's oldest job; over several queues it skips only down to the highest
of the spans' last entries. The span starts at one entry, doubles up to 128 while steps cross several
levels, and drops back to one when it finds a due level, or when a span of at least 16 entries stays
inside one level. So the common
claim reads one entry per queue, delayed jobs spread one per priority are skipped up to 128 per step,
and a large delayed level costs one index descent instead of a scan.
Over several queues, each level visits the queues in the order of their oldest due job, so a batch
can take a queue's newer jobs before another queue's next one, and the next claim starts from that
other queue. Rows are locked only as the claim takes them, so a concurrent claim never skips a due
job that nobody is taking. The same statement marks them running, stamps the lease and opens their
executions, and it stamps a random token of its own and takes a transaction advisory lock derived
from it, which only that claim's transaction can hold. A claim that ends without its answer, from a
lost connection, a server that stopped answering or a cancelled call, even one cancelled again
while it gives up, may still commit later, so tinyjobs keeps its token, its lease, its worker and
the backend it was sent on, named by process id and start time, in memory. Passes over them run
one at a time. Each, as the login role, first ends the backend if it holds the claim's lock, so a
claim still running takes nothing, unless the role may not run pg_terminate_backend, which is
logged once and leaves the claim to commit, then reads pg_stat_activity and runs one UPDATE that hands back the running jobs
carrying those tokens, without counting the attempt and with their executions removed. A token is
forgotten once that UPDATE hands back its jobs, or when its backend, as read before the UPDATE,
is gone, belongs to another role, or has gone idle since the first pass saw it, since a backend
only goes idle once the claim, if it arrived, has ended. A backend that sat idle since the first
pass may still receive a claim delayed on the way, or be idle in PgBouncer's pool; its token is kept
until the backend changes state or goes. Passes run before a claim and on a heartbeat, at most once
a second while some backend is busy or unseen, when a wait for jobs is also cut to a second, and
once every third of the lease while all sit idle. Closing runs passes for up to five seconds while
some backend is busy, so a claim still running there is ended too. A pass that fails is logged and
tried again later, and never fails the claim it runs before. No timing is guessed, and no
backend that might serve someone else is ended: only the holder of the claim's own lock. A process
without such tokens runs no extra statement; the lock costs a claim about 3 µs. A process that dies
loses its tokens, like any jobs it held: their leases expire. Workers never wait on each other's rows. Walking levels rather than sorting matters when
a backlog is not due: with 50,000 delayed jobs at one priority ahead of a due one, the plain ORDER
BY priority DESC, run_at read 205 buffers per claim and the level walk 50. Spans matter when the
delayed work spreads over many priorities: with 20,000 delayed jobs each at its own priority above a
due one, a walk of one level per step read 100,000 buffers and took 57 ms per claim, the spans 1,200
buffers and 4 ms.
Time¶
As on Redis, every machine follows the server's clock: the backend reads clock_timestamp() on
setup and after a connection error, and again every thirty seconds beside a call while the pool has
a second idle connection and no worker on it is stopping, so the reading never takes the connection
a release needs, or before the call once the reading is a minute old; the statements that decide by time
compare against the earlier of the caller's now and statement_timestamp(). A claim or a heartbeat
stamps the later of the two lease ends. So a host on a stale reading never reaps a live lease and
never writes one that ends too soon.
Waking workers¶
Idle workers do not wait for their next poll. After an enqueue that left a job due, the backend sends
pg_notify with the queue as payload, cut to its first 1,000 characters so it stays under
Postgres's limit, at most once per queue every 50 ms; a burst sends one notification and a trailing
one. A queue with a trailing notification pending waits for it rather than sending its own: 30
enqueues 4.4 ms apart sent 5 notifications, two of them 3 ms apart, and now send 4, 50 ms apart.
A caller with a deadline does not wait for the notification, which is sent on its own: at a
600 ms round trip a release of eight jobs under a one second deadline returned after 0.61 seconds,
where waiting for the notification took it past the deadline and the worker logged them unreleased
though they were queued. The channel is tinyjobs_ and a hash of the instance id, which always fits in a channel name.
So every spelling of one store, by schema= or by search_path, shares one channel, and two stores
never do. Each worker process holds one extra
connection that LISTENs, and a notification for one of its queues ends its idle wait at once,
unless it claimed since. A lost listening connection is reopened within a second, and every reopen
wakes the workers once, since notifications sent meanwhile are gone. Closing while its LISTEN is still on
the way closes that connection instead of sending a cancel, which psycopg waits five seconds for:
over a link adding 300 ms each way, workers stopped right after going idle used to log a failed
cancel five seconds after they stopped. A server
idle_session_timeout closes an idle listener too, and it reopens the same way; idle pooled
connections it closes are replaced, and psycopg_pool logs a warning for each. Polling stays as the fallback, so
notify=False or a dropped notification costs latency, never a job.
Joining the caller's transaction¶
tx= takes the caller's psycopg connection or cursor, sync or async, and writes the job on it:
with psycopg.connect(DATABASE_URL) as connection, connection.transaction():
connection.execute("INSERT INTO orders (id) VALUES (%s)", (42,))
send_receipt.with_options(tx=connection).enqueue(42)
The insert is written once as a generator of (statement, parameters) steps, driven by the pool's
async connections, by the caller's async connection on the caller's loop, and by the caller's sync
connection in the caller's thread. A sync enqueue therefore never crosses to the backend's loop with
a connection that belongs to another thread. The notification goes inside the same transaction, and
Postgres delivers it only at commit, so a worker never wakes for a job it cannot see yet; a rollback
takes both. The tables are named with their schema here too, so the caller's search_path or a
temporary table of the same name cannot divert the job. The first use of a connection reads the
store's identity through it and raises ConfigurationError when it differs, which catches a
connection to another database, even one holding tinyjobs tables of its own or a copy of this one.
Dedup keys work inside the transaction as outside: a second enqueue with a held key returns the
holder, including one written earlier in the same transaction. Jobs sharing a key inside one call
become one job before anything is written, and the first one given is the one stored. Every call
inserts its jobs in key order, so two bulk enqueues racing on the same keys wait for each other
instead of deadlocking. A holder written by another transaction that has not committed makes the
insert wait for it, as any unique index does. Eager mode refuses tx=, because it would run the job
before the transaction could commit, and so does every backend that cannot join one.
In Django, OPTIONS = {"url": ..., "database": "default"} enqueues through that database's own
connection, so a task enqueued inside transaction.atomic() is written in its transaction, and a
rolled-back savepoint takes its tasks with it. aenqueue runs in the thread that holds the
transaction, through sync_to_async(thread_sensitive=True), except in eager mode, which joins no
transaction and would hold that thread while the job needs it.
Connections¶
Each process keeps a pool of 1 to 10 connections (min_connections=, max_connections=), plus the
listening one, whatever its workers run. Growing it with their concurrency bought nothing: with three
workers of 16 jobs on a 40 connection server, over a link adding 300 ms each way, 1200 jobs took 67.3
seconds with at most 32 connections, no warning and none of 270 other connections refused; grown,
they took 66.2 seconds with 36, 15 warnings and 107 of 267 refused. Claims did not starve behind the
outcome writes either: no worker logged a failed claim. Resizing the pool opens only the connections
it lacks, never past max_connections=: psycopg's own resize opens one per step of the minimum,
which with both connections of a two connection pool busy opened a third. Every pooled connection
is set to read committed, a 30 second statement_timeout and a one minute
idle_in_transaction_session_timeout, with jit off and synchronous_commit raised to on when
the session starts with off: the recursive queries behind the
trust guard carry cost estimates high enough that a server built with LLVM compiled them at every
call, which took a setup 3.5 seconds instead of 50 milliseconds and a tx= call 1.3 seconds
instead of 10. On a server that can compile, a tx= call turns jit off for its own statements
without a round trip of its own: on an idle autocommit connection the setting travels in the same
pipeline as the statement and ends with it, and inside the caller's transaction the caller's value
is kept on the server and put back with the last statement, or after a failure; every connection,
the listening one too, is named tinyjobs in pg_stat_activity unless the URL names it and keeps
TCP keepalives and a one minute tcp_user_timeout on, and the listening one gives up connecting
after pool_timeout, two seconds at least. Pooled connections give up connecting after
pool_timeout too, between two and five seconds, unless the url sets connect_timeout, and a new
connection whose setup statements get no answer within five seconds is closed and another opened:
with the server accepting connections but not answering their setup for a while, jobs started 4.9
seconds after it answered again, where none had started 120 seconds later. While other connections
keep getting answers, the setup keeps waiting, five seconds at a time and a minute at most (cut
at 60.0 seconds, where it ran to 65), since
it is then queued behind a pooler that opens one server at a time rather than unanswered: two
workers of 32 behind a PgBouncer pool of 25 over a link adding 300 ms each way started with no
warning and kept the app's wait for a server under 2.4 seconds, where 37 to 44 setups were cut,
each warned, and the connections opened again queued behind the rest, the app waiting up to 9.2.
A pooled statement is never cut short while bytes move
in either direction or while the server is still working on it, waiting on a lock, a synchronous
standby or its disk; a deadline would fail a large claim over a slow link, or report a commit
waiting on a standby as failed while it then applies. After ten silent seconds tinyjobs asks the
server what the statement's backend is doing, on an idle pooled connection when the pool has one,
which gets a second to take the question and answer it, and otherwise, or when it does not, on a
fresh connection, one per backend shared by every statement asking at the time, which closes once
none is: twenty statements silent together behind a held row lock over that slow link kept a
process at 11 sessions, where each opened a connection of its own and it reached 18. The question
names the
backend by its process id and start time, so a reused id is never taken for it, runs as the login
role, which sees its own sessions even when the url or ALTER ROLE ... SET role switches to
another, and uses only pg_catalog operators whatever the url's search_path. If the backend is
gone, the call raises OperationalError, which the worker treats like a lost connection. If the
backend has been blocked sending to us for those ten seconds, or has waited five minutes for bytes
from us with nothing moving on our side, an answer lost on its way or a request stuck in a proxy,
tinyjobs ends that backend first, so nothing it was doing takes effect afterwards, then raises;
a login role that may not run pg_terminate_backend raises without ending it. A
server that cannot be asked within ten seconds is not taken for gone, since a PgBouncer whose
server slots this pool already holds looks the same: the call waits for those five minutes of
silence, then raises. A claim waits half its lease instead, asking after silences of a third of
that, between one and ten seconds, and no question nor wait between them runs past that half: with
the server cut off, a claim leased for 6, 30 and 60 seconds gave up after 3.0, 15.0 and 30.0. That
half covers the whole claim: the sweep of earlier unanswered claims, which gets a quarter of the
lease so one that hears nothing leaves the claim the rest, the clock reading or the wait for another
call's, every wait for a pooled connection, and the cancel request and the wait after it when the
claim is cancelled. With a 6 second lease and the server cut off, the claim after a lost answer gave
up after 3.0 seconds rather than 40, and one due to read the clock after 3.0 rather than 321. With
no connection to ask on, a slow claim and a lost answer look the same, so a claim still running
when the server has taken no new connection for half its lease is given up too, and settled like
any claim without its answer: it is ended if still running, and its jobs come back without the
attempt counted. A heartbeat gets a quarter of its lease the same way, its sweep half of that, and
the next one starts a third of the lease after the last one started, so a heartbeat that hears nothing
leaves the next one time to renew the lease: with the connections already open frozen and new ones
answering, a heartbeat on a 12 second lease gave up within 4.0 seconds with at least 8.0 left on
the lease, where it used to wait past 40 with the lease lapsed. The worker gives an outcome or a
hand-back one full lease from when it starts, and during shutdown no more than the shutdown
deadline. An outcome whose call ran out of time is checked against the job's execution history on a
fresh call within a second, bounded by the worker's own end rather than by the job's deadline that
just ran out; if it landed, it counts as recorded and its hooks run. Once a statement goes unanswered, the
pooled connections idle since before it are closed and replaced rather than used, since a frozen
network path or a hung proxy silences them all alike. A call cut by its
deadline or cancelled sends the server a cancel request and waits for the statement to stop, both
within half a second once the deadline has passed. A backend whose stop is not confirmed in that
time has its connection closed and is ended with pg_terminate_backend, matched by process id and
start time, before the next statement this backend sends, so a statement left waiting on a lock
never applies afterwards. Every statement tinyjobs sends on a pooled connection starts with a
comment carrying a secret random token of that connection, /* tinyjobs <token> */, first so
track_activity_query_size never cuts it, and the same for every statement so prepared statements
still work, including the BEGIN, COMMIT and ROLLBACK psycopg sends itself on Postgres 16
and later; each pooled connection also holds a session advisory lock on a secret random key. A
backend is ended, as the login role and in the same statement that looks, only while its current or
last statement carries the token, whatever its state and whenever it arrived: running, idle with the
statement still on its way, or a statement that reached the server after the cut. Where the role
sees no statement text, with track_activities off, the lock stands in for the token, and the same
for a bare transaction command: before 16, pg_stat_statements keys those by their text, so a token
on them added two entries per connection, 40 after 20 short runs where there are now 2. Nothing else
runs our token or holds our lock, and PgBouncer's reset (DISCARD ALL) drops the lock before it
hands the backend to another client, whose statements carry no token, so another client is never
ended; neither clock takes part. Through a proxy that delivered the statement three seconds late,
past the cut, onto a row lock, the statement was ended and the job stayed running, where it used to
land as success once the lock was let go; through a session pooler that handed our backend to
another process, that process's three second statement ran to its end. A COMMIT that reached the
server late, after the cut, is ended too, by its token or, before 16, by the lock: it used to be
left to apply all 100 jobs of a cut batch enqueue; none apply. A claim delayed on its way
is then handed back by the sweep of unanswered claims.
The next statement waits for a pass already running rather than going ahead of it, and a backend
not gone within a second of being asked stays queued for the next pass. Cancelling a call bounds
the cancel request at five seconds on psycopg 3.2 and 3.3 alike, and the call ends in its
cancellation even when the server answers the cancel with another error, so it is never retried.
These paths rely on private psycopg and psycopg-pool names, the pool's connection bookkeeping, the
query pipeline and the cursor's execution internals, so both are required below 3.4 and a test
checks every one of those names is still there. Pooled connections ask for UTF8 unless the URL
names a client encoding, so a SQL_ASCII
database works too. Every connection unused for ten seconds beyond the minimum closes together,
not one per interval, and a connection that failed to open is not retried once no caller waits
for it: four workers of 16 on a 40-slot server peaked at 29 sessions of theirs instead of 36, and
refused none of the app's own instead of 8 in 265. Callers kept waiting a quarter of a second get
connections opened for all of them at once, up to the maximum, rather than one after another: four
callers behind connects of 0.6 seconds were all served within 0.94 seconds instead of 1.83, and over
a link adding 300 ms each way bursts of 150 jobs after 13 idle seconds took 9.6 seconds instead of
10.1, 9.55 before the pool gave idle connections back, with the same peak of 20 to 21 sessions.
Every connection is replaced after half an hour. Deadlocks, serialization
failures and a session the server closed under idle_session_timeout, which it only does while
waiting for a command, so before the statement ran, are retried three times with a warning, except inside the caller's transaction, which only
the caller can retry. Before using a pooled connection, tinyjobs reads what the server already sent on
it without a round trip, under a microsecond, and hands a closed one back to be replaced and takes
the next at once. So the connections a restart or pg_terminate_backend closed while idle cost
neither an error nor a wait: with eight of them dead, the next three statements took under a
second together. A read that still loses its connection mid-flight, such as get, a listing or the
clock, runs once more on a fresh one; writes do not, since the server may have applied them, and
nothing is retried after waiting pool_timeout for a connection. Options the backend does not know
and libpq does not either, such as a misspelled max_conections=, whether passed as arguments or
in the url's query string, are refused by TinyJobs(...)
itself rather than surfacing as a pool that never opens; so is a url whose host or port came out
malformed, which is what a raw @, /, ? or # in the password does, and an error about the url
never quotes it once it holds a password; cursor_factory is not taken either, since
tinyjobs picks its own cursors. On a caller's connection whose cursors bind raw, tx= uses a plain
cursor of its own. A fork leaves the parent's connections alone: the
child opens its own.
Where the limits are¶
- Behind PgBouncer, session pooling only, with its default
server_reset_query. Each connection is configured once when opened, and the listener needs a session of its own. Transaction pooling would lose both, and without the reset a backend could reach another client still holding our lock and our last statement, which is what marks it as ours. - Every reaper pass looks at every waiting job. Each worker's pass reads the jobs waiting on
another and looks up the parent of each, to settle the ones whose parent ended unseen. A few
thousand waiting jobs cost nothing to speak of, fifty thousand about a tenth of a second a pass,
and a million of them under fifty workers more than a million lookups a second for as long as
they wait; the answer there is a longer
--reap-intervalon most of the workers. - A held snapshot stops the cleanup. A transaction left open anywhere on the server, or a replication slot that stopped advancing, keeps autovacuum from removing finished job rows, and the jobs table bloats until it ends.
statement_timeoutis the server's limit, and so is what it counts. A purge or a claim that needs longer than 30 seconds fails, and Postgres counts sending a result, and receiving the parameters of a statement not yet prepared, as part of the statement, so a payload slower than that to move fails at once with nothing applied. Raise it withstatement_timeout=or turn it off withNone; the watch on silent connections stays on either way.- Supported versions are the ones the community supports, 14 to 18 today. CI runs the suite on 14 and 18.
The scheduler¶
Cron looks like the easy part and contains most of the subtle bugs. What we commit to:
@app.task(cron="*/15 * * * *", timezone="Europe/Rome", misfire="skip", overlap=False)
async def refresh_cache() -> None:
...
A cron task is an ordinary task with a rule attached. It can still be enqueued by hand, and the jobs the rule creates are ordinary jobs: same retries, same history, same cancellation. It is called with no arguments, so a signature that requires some is rejected when the decorator runs, as is a cron expression, a timezone or a misfire policy that cannot work. A mistake surfaces at import, not at midnight.
Where it runs: inside every worker, by default¶
tinyjobs worker runs the scheduler as one more coroutine on its event loop. Not as a separate
process, and not on a designated leader.
This is the dividend of leader-free firing below. Because the compare-and-set guarantees exactly one
job per tick no matter how many schedulers race for it, running fifty schedulers is not a bug to be
prevented, it is free high availability. There is no leader to elect, no failover to test, and no
deployment where cron silently stops because one special process died. --no-scheduler opts out, and
tinyjobs scheduler runs one standalone for anyone who wants the isolation.
The cost is a polling loop per worker, which the tick makes negligible: the scheduler sleeps until the
earliest next_run_at rather than waking on a fixed interval, capped at 30 seconds so a worker that
slept on stale state rereads it soon. A tick that fires nothing is one SELECT over the schedules
table.
Leader-free firing¶
Running three workers must not create three jobs per tick. The usual answer is leader election; ours is cheaper, because the store already gives us atomicity:
BEGIN IMMEDIATE;
UPDATE tinyjobs_schedules SET next_run_at = :computed_next
WHERE name = :name AND next_run_at = :expected;
-- only if that matched one row:
INSERT INTO tinyjobs_jobs ...;
COMMIT;
Exactly one worker's UPDATE matches. That one creates the job, the others get zero rows and move
on. The comparison is against the exact value the scheduler read, not against its own clock, so two
workers whose clocks disagree by a few seconds still cannot both win. And the job goes in inside the
same transaction as the advance, so a worker killed halfway leaves either both or neither: there is no
window where the schedule moved on and the job was never written.
Registration is the other half. Every scheduler upserts its declarations when it starts, and the
upsert keeps the stored next_run_at unless the cron or the timezone changed. A restart therefore
resumes where the previous process stopped, which is what lets the misfire policy see what it missed,
while a changed expression starts afresh from now.
Timezones and DST¶
Schedules store an IANA timezone name and are evaluated with zoneinfo (stdlib). This makes the
awkward cases explicit rather than accidental:
- A fixed time of day,
30 2 * * *, on the day a DST gap swallows 02:30. Policy: fire at the first valid instant after the gap, 03:00. - The same schedule on the day 02:30 happens twice. Policy: fire once, on the first occurrence.
- A schedule whose hour field is a wildcard (
*/10 * * * *,0 * * * *) is about frequency, not about a time of day, so it follows the clock as it actually ticks: the repeated hour runs twice, and the hour the gap removes does not run at all. The interval between firings stays what you wrote. This is the rule Vixie cron uses. - The default timezone is UTC, which has none of these problems. Local time is opt-in.
"UTC"never touches the tz database; any other name needs one, which Windows lacks unlesstzdatais installed.
Misfires¶
If the scheduler was down from 09:00 to 11:00 and a schedule fires every 15 minutes, that is eight missed firings. Guessing here is how you get a thundering herd on restart, so it is a per-schedule policy:
skip(default): forget them, resume from now.run_once: create a single catch-up job.run_all: create all eight, each with therun_atit should have had, oldest first. For accounting-style workloads where every period matters. At most 100 are created per tick, so a long outage on a frequent schedule is worked off over consecutive ticks instead of in one transaction.
"Missed" needs a threshold, because a firing a second late is not missed. That threshold is the
misfire grace, 60 seconds by default (--misfire-grace). A firing at most that late counts as on
time, under every policy. So does a firing that falls inside the grace at the end of an outage, which
is why a restart that lands a few seconds after the minute still runs that minute; under run_once that
job doubles as the catch-up.
The clock can also go the other way. If it moves back, by hand or by a VM restored from a snapshot,
the stored next_run_at is suddenly far in the future and the schedule would sit silent until the
clock caught up. A stored firing later than the next one the cron allows from now + grace is
treated as exactly that, and recomputed from now. The grace is the tolerance, so clocks that merely
disagree by less than it between machines are never mistaken for a jump.
Overlap¶
By default a schedule will not create a new job while its previous one is still queued, held or running,
implemented with the dedup key, so it needs no extra machinery. overlap=True opts out. A run that is
waiting out its retry backoff is still queued, so it holds the key too: a failing hourly job does not
pile up a second one behind it. run_all requires overlap=True, since its catch-up jobs would
otherwise collapse into one.
Pausing and firing by hand¶
Two things an operator does to a schedule without touching code: stop it for a while, and run it now. Both are the firing above with no jobs in it.
app.pause_schedule("report.hourly") # tinyjobs schedules pause report.hourly
app.resume_schedule("report.hourly") # tinyjobs schedules resume report.hourly
app.run_schedule("report.hourly") # tinyjobs schedules run report.hourly
A paused schedule is one whose next firing never comes: next_run_at is set to NEVER, 2^62
seconds past the epoch, with the same compare-and-set that a firing uses, so a firing that races
the pause cannot write over it, and a pause that loses the race reads the row again and retries
until it holds, so it never reports a schedule paused that is not. Every backend already has the
operation. No column, no migration, no new script. The scheduler leaves a row at NEVER alone
instead of reading it as a clock that jumped back, tinyjobs schedules prints paused in place
of the time, and check reports the schedule as paused rather than overdue. Registration keeps a
stored time unless the cron or the timezone changed, so a pause survives restarts and deploys, and
a changed cron starts the schedule afresh, as it always did, which un-pauses it. Resuming computes
the next time from the task's cron and timezone, or from the stored ones when this process does
not register the task, so it works from any process; firing by hand needs the task, since the
job it creates is the task's.
Running by hand is an ordinary enqueue carrying the schedule's dedup key, so with overlap=False
a run that is already queued, held or running comes back instead of a second one, and the schedule's own
next time is untouched: a job fired by hand at 10:40 does not move the 11:00 firing.
Cron syntax¶
Standard five-field expressions (minute hour day month weekday) with *, ranges, steps, lists, and
three-letter names; 0 and 7 are both Sunday. When both the day of the month and the weekday are
restricted, either one matching is enough, as in every cron since Vixie. Plus the familiar macros:
@hourly, @daily, @midnight, @weekly, @monthly, @yearly, @annually. An expression that can
never fire, such as 0 0 30 2 *, is rejected rather than accepted and left waiting.
Seconds-level precision is out of scope. At that frequency you want a long-running task with a sleep, not a scheduler.
Where the limits are¶
Measured on an Apple silicon laptop, every schedule set to * * * * *, all of them firing on the
same minute boundary, which is the worst case:
| one scheduler, in process | about 110 µs per firing, one transaction each: 10,000 in 1.05 s |
| four workers racing, 2,000 schedules | 8,000 firings, each exactly once, p95 0.57 s late, max 0.62 s |
| four workers racing, 10,000 schedules | 40,000 firings, each exactly once, p95 6.4 s late, max 7.5 s |
| 56 schedulers racing, one schedule, 12 minutes | 12 jobs, no duplicates, p95 15 ms late |
50 SIGKILLs in 10 minutes, 4 workers and a scheduler |
10 jobs, no gaps, no duplicates, max 4 ms late |
Racing schedulers share one write lock, and every loser still takes it for an UPDATE that matches
nothing, so lateness grows with schedules times workers. Applications have tens of schedules, where
none of this is visible; at ten thousand firing on the same minute the tail is seconds late, and still
exactly once.
The other limits are about time rather than volume:
- A task that blocks the event loop blocks that worker's scheduler. An
async defcallingtime.sleep()for 150 seconds on an every-minute schedule made its own worker skip the two firings it slept through. The same task with--no-schedulerand a standalonetinyjobs schedulerfired every minute within 36 ms; the jobs queued behind the blocked worker, but none were lost. Other workers cover for a blocked one, unless they block the same way. - Restarts. A graceful stop and a
SIGKILLbehave the same, because the state is the row. Down for 35 seconds across a firing: it ran 25 seconds late on restart. Down for 85 seconds: the firing more than a minute late was skipped and the next one ran 15 seconds late. Down for four and a half minutes:skipdropped four firings,run_allcreated all five in order. - Clock jumps. Forward by an hour: treated as an outage, missed firings skipped. Back by an hour: realigned on the next tick, instead of falling silent for the hour.
- Rolling deploys that change a cron. While both versions run, each firing is computed by
whichever version wins it, and it is still never doubled, since the compare-and-set does not care
whose expression produced the value. Once only the new version runs, every next time comes from the
new expression; the
croncolumn thattinyjobs schedulesprints catches up the next time a new worker starts.
Execution model¶
A worker is a loop: claim a batch, dispatch each job to the right executor, record the outcome. All the interesting decisions are in the dispatch.
Executors¶
| Task kind | Executor | Timeout | Notes |
|---|---|---|---|
async def |
worker event loop | hard | asyncio.timeout, real cancel |
def |
daemon thread pool | soft | see below |
def, isolated=True |
a job process | hard | killed at the timeout, replaced on a crash |
A single worker process runs an event loop plus a thread pool. Async and sync tasks share the process
without either starving the other. Tasks declared isolated=True run in job processes the worker
starts on demand, up to processes, and keeps for the next job.
Isolated tasks¶
The job process is spawned, not forked: a worker has threads, a running event loop and open
sockets, none of which survive a fork intact, and since 3.12 a fork from a process with threads is
a deprecation warning. Spawning costs an interpreter start and the task module's imports, paid once per process,
which is why processes are kept and reused; jobs_per_process retires one after that many jobs, for
code that leaks, and the default is never.
The job's own data is never pickled across the boundary. The parent sends the task's name, the
module that registered it, the function's own module and qualified name, the job's payload bytes
and codec; the process imports the registering module, or the function's when the task is not
there, as after a helper in another module called app.task, and picks among its module-level
objects the isolated Task with that name over that function, refusing to guess when two apps
registered one; it decodes the arguments with the task's own app, calls the function and encodes
the result the same way. The task is found by its name, like a task is on enqueue, and not by the
function's module and qualified name alone: a function imported from elsewhere, registered with
app.task(...)(function), or shared by two apps would otherwise be found in the wrong place or
not at all. What this asks of the user is only that the Task sit at module level of the module
that registered it, which @app.task does by itself. process_setup is found by its own module
and qualified name, resolved once in the worker the way the process will resolve it, so a bound
method or an alias is refused at start instead of failing every job. A task registered in
__main__ resolves through __mp_main__, where the spawn machinery re-imports a guarded script;
it does not re-import a python -c command, a REPL or a package's __main__.py, so a task or
setup registered there is refused at registration. The process sends
started right before it calls the function, and only then does timeout= start counting: the
first job on a fresh process used to pay for the interpreter start and the imports, and a module
slower to import than the timeout failed every job forever, each failure killing the process and
the next job starting another. Coming up has its own bound, which covers the interpreter and the worker's main module; the
setup, django.setup() and the module import have a separate one of ten minutes, long enough
for a model to load and short enough that a connect that hangs does not hold the slot forever. An exception comes back
pickled when it round-trips, since it is the process's own object and the pipe already carries
pickles; failing that, as its class name, module and message, from which the parent rebuilds it
when the class is in its own modules, so retry_on matches it and TimeoutError still means
TIMEOUT, and keeps the name and the remote traceback otherwise; either way the child's
traceback becomes the direct cause of the error the parent holds, so a plain formatter and
Sentry print the child's frames before the parent's. Retry and Abandon cross by their
fields. A process_setup function, found by name the same way, runs once in each process
before its first job; it is set on the app, so that workers, app.drain() and eager mode all
honour it, or on one worker. django.setup() runs on its own when DJANGO_SETTINGS_MODULE is
set, because a job process starts from the task module's imports and nothing the worker did at
startup. app.drain() and eager mode start a process for the call and stop it after, so there
the setup runs each time.
The parent waits on the pipe from the event loop, so an isolated job costs no thread; the request
is written from a thread, since a payload larger than the pipe buffer would otherwise stall the
loop until the process had finished importing. A timer polls the process while a reply is pending,
because a grandchild holding the pipe would hide the process's death from the reader, and the
parent reads the pipe in whatever pieces arrive and frames the messages itself, so a process that
dies halfway through a message cannot leave the loop waiting for the rest. The parent's end is
closed only once nothing is being written to it, since closing a descriptor under a writer in
another thread could hand the request to whichever file took the number next. A process retired
after jobs_per_process is reaped from the loop shortly after, so a long run does not pile up
descriptors or zombies. At
timeout= the process is killed, the job fails with TIMEOUT and the pool starts a fresh process
for the next job; a process that exits or dies of a signal fails its job with ProcessDied, naming
the exit status or the signal, and retries like any failure. A process that dies before it got to
the function is not the job's fault, so the job is tried once more on a fresh process; one that
died while idle is reaped when the pool reaches for it, not blamed on the next job. On shutdown a
job still running at the end of the budget is killed and handed back, where a thread can only be
left running: the one case where the process executor is stricter than the thread pool, because it
can be. The processes are not daemonic, so a task may start processes of its own; at interpreter
exit the pool kills whatever is left, and a process
whose worker died under it notices the parent change and exits, so a hung task never outlives the
worker that could have killed it.
A job process ignores SIGINT and SIGTERM so that a Ctrl-C, or a TERM sent to the worker's
group, stops the worker, which decides, and not the job under it; it does so with a handler that does
nothing rather than SIG_IGN, which survives exec and would leave every subprocess the task
starts deaf to terminate(). It also moves into a process group of its own, so that the kill at
the timeout, at shutdown or at the worker's death takes whatever the task started with it, since
a process the task forked would otherwise outlive the job it belonged to. A process that exits
or crashes on its own leaves its subprocesses to themselves, as any process does, and a group of
its own means a job process stopped by the terminal's tostop would wait for its timeout.
The pool is the only one that reaps its processes: each is taken out of multiprocessing's
bookkeeping as soon as it starts, so no other thread's Process.start() can collect its exit
status under the pool, misreport its death or free its pid before the pool is done with it; and
when something else in the worker did collect it, a wait on any child or SIGCHLD ignored, the
pool notices, lets the process go and never signals the pid again. The parent blocks both signals
around the spawn so the mask is inherited through exec and a signal in the first milliseconds, before the
process has installed its handlers, stays pending and is discarded once it has; without that, a
Ctrl-C during a spawn killed the process with a bootstrap traceback and the job failed with
ProcessDied.
processes defaults to concurrency or the CPU count, whichever is lower, and never fewer than
loops, since isolated tasks are there for CPU-bound and crash-prone work. Isolated jobs beyond that wait in their slot for a free
process; the claim loop does not know a job is isolated until it has it, and a second kind of slot
would add a scheduling problem for a case that processes=concurrency already solves for those who
want every slot backed by a process. With loops, the processes are shared out among the loops like
the slots.
There is no --concurrency-model flag. The alternative is tempting, so here is the argument
against it. The thing such a flag would select, threads versus processes, is not a property of the
deployment, it is a property of the individual task: a function that holds the GIL for four seconds
needs a process whether you run one worker or forty. isolated=True expresses that at the
declaration site, where the person who knows the answer is standing. A global flag would ask the same
question again at the layer with the least information, and answer it the same way for every task.
Scaling out is therefore "run more worker processes", supervised by systemd, supervisord or your orchestrator, which you need regardless the moment you have two machines. Apart from the job processes of isolated tasks, the worker does not fork or supervise children itself; that stays out of v1, where adding it later is purely additive and removing it would not be.
Knobs: --concurrency N sets the thread pool size and doubles as the default ceiling for concurrent
async tasks, overridable with --async-concurrency when the default does not fit.
Event loops¶
Redis and Postgres connections belong to one event loop, which is the app's loop thread for ordinary
calls. worker.run() owns a loop of its own, so it opens a second connection pool on it, through a
private copy of the app that shares the tasks, codecs, hooks and clock, and claims and records
outcomes there: each is a plain await rather than a hop to the loop thread and back. The app
itself is left alone, so def tasks enqueuing, other threads and other workers on it carry on
when the run ends and its pool closes. On one process with one Redis, this took no-op jobs from
2,180 to 2,930 a second. await worker.start() runs on a loop it does not own and keeps using the
app's connection.
One loop is one thread, so one process stops there however many cores it has. loops=N runs N-1
more workers in threads of the same process, each with its own loop and pool. concurrency is
shared out among them; only the first fires cron and prunes. Stopping, forcing and signals reach all
of them, and a loop that fails stops the others and its error comes out of run(). With the GIL on
they take turns and gain little; on a free-threaded build they run in parallel. SQLite is refused:
one writer means extra loops only wait on each other.
Timeouts are not uniform, and pretending otherwise would be a lie¶
A running Python thread cannot be killed. There is no safe interrupt for arbitrary code; the
CPython APIs that come close corrupt interpreter state. So for a sync task in a thread pool, a timeout
can only mark the job TIMEOUT and move on. The thread keeps running until the function returns, and
it keeps occupying a pool slot. The timeout counts from when the thread starts, so a job waiting
for a free thread behind one that timed out is never recorded as timed out before it ran: it used to
be failed with TIMEOUT while queued and then run anyway, and with retries left it ran again.
A thread still running after its job ended counts against concurrency until it returns, and a
warning names its task, so the worker claims only jobs it has a thread for; a drain or an eager
call runs the next job on a thread added in its place and drops it once the old one returns: a
drain behind such a thread finished in 0.5 seconds and an eager call past its timeout returned in
0.51, where they waited 3.0 and 4.0 for the thread. Nothing caps those added threads, so a
process that keeps sending jobs that hang holds one thread per hung job, as it would calling the
functions directly: 40 eager calls of a task with a 0.05 second timeout that sleeps for three
seconds held 41 threads and about 40 MB at most, all 40 ended failed, and they returned in 2.2
seconds. An earlier cap of eight such threads per app kept that bounded, but it
made drains hand jobs back and eager calls wait for or borrow threads, and every rule it needed
broke another case. A thread that fails to start fails its job, where the job was handed
back as if the threads were shut down and the pool counted the dead thread, so no later job ran. The
threads are daemon threads, so a process whose worker stopped with one still running exits, with a
warning naming them, where it used to hang at interpreter exit until killed. A job with a 0.3 second
timeout claimed behind such a thread used to stay running for as long as the thread ran, its lease
renewed so no other worker could take it; it now stays queued for any other worker, and ran here
once the thread returned. With two workers of concurrency 2, one holding two runaway threads, all
six quick jobs ran on the other, where two were held running after 24 seconds.
This catches people out, so, plainly:
- A sync task stuck in an infinite loop leaks a thread, and the job slot with it, for the life of the process.
- A blocking socket read with no timeout of its own blocks past your
timeout=.
Mitigations, in order of preference: pass timeouts to the library you are calling (every HTTP client
has one); or write the task as async def, where cancellation actually works; or declare it
isolated=True, which runs it in a process the worker kills at the timeout. The documentation says
all of this at the point where timeout= is introduced, not in a footnote.
Graceful shutdown¶
On SIGTERM (or the first SIGINT), the worker stops claiming, keeps heartbeating in-flight jobs,
and returns within --shutdown-timeout, or 1.25 seconds when it is shorter. The last quarter of it,
one second at least, is kept for releases; jobs get the rest to finish. A shorter budget would leave
nothing for releases: with a timeout of 0, three async jobs stayed running until their lease expired
and three def jobs whose threads returned could not get a connection to record their outcome, where
now the first are back in the queue and the others recorded. A close that starts once that budget is
spent keeps its own limits rather than the spent one, and heartbeats for jobs still running in threads
wait out a lock as before the stop: an enqueue behind a locked table that the close cut off at once
now lands, and a heartbeat behind a 1.2 second lock taken after the worker's end renews its lease,
where it failed after 0.1 seconds; that wait lasts only as long as the heartbeat itself, so it never
holds the exit: on SQLite, a heartbeat stuck on a lock when the heartbeat stopped kept the process
alive 5 seconds longer, its thread still in SQLite's busy wait, where now it gives up within a slice.
A check of an outcome whose answer was lost ends with the timeout, without the half second a cut call
otherwise gets to unwind: with SQLite locked and a 2 second timeout the worker returned after 1.75
seconds, the close's moment before the timeout, where it took 2.50. Every other call made under an end
the shutdown sets is cut the same way, so an outcome written as a kept thread returns ends with that
thread's window rather than half a second past it. A claim still waiting on the backend shares that
wait: jobs it brings back in time run like the others, and one that has not answered by then is given
up and its statement stopped on the server. Anything still running is then released back to QUEUED
in one call for all of them, a single round trip on Redis, so another worker picks it up immediately
rather than waiting for the lease to expire; a release that has not landed when the timeout ends is
left to its lease. With the server cut off and --shutdown-timeout 3, the worker returned 3.0 seconds
after the signal and the process exited after 3.1; it took 9.6 before, and under uvicorn with
shutdown_timeout=5 6.2 rather than 12.2. Closing ends a connection still being opened at once rather
than waiting for it: over a link adding 300 ms each way, closing while one was being opened took under
0.01 seconds with nothing logged, where it took 1.0 and psycopg_pool logged that it could not stop
its task. A backend that went silent is not asked to end what it left behind at close; one that still
answers gets half a second a pass for it. Once the stop is asked the backend keeps a connection open
for each running job plus two for the releases and the heartbeat, within the pool size, and opens
those it lacks, so outcomes written as threads return need not connect first; on Redis the idle
connections stay free for other calls meanwhile, where a call waited a second behind them, and a stop
asked outside a run reserves nothing; the heartbeat stops renewing the jobs about to be released, and
a clock reading that falls due is taken beside a call rather than before it. Over a link that adds 300
ms each way, eight jobs on a 2 second timeout were all released, on Postgres and on Redis, with the
worker back after 1.61 seconds, where every run used to log all eight unreleased and Redis left them
running until their lease ran out. A second SIGINT ends the wait at once: a release already sent
gets a second, and the process exited 2.6 seconds after the second signal with the server cut off,
where it took 6.6. The process itself exits within the timeout: a wake-up still on its way at exit
drops its connection, without the pool's warning about it, instead of asking the server to stop it,
which over that link cost another second, so with a 3 second timeout it exited 2.65 seconds after the
signal rather than 3.6, and with 5 seconds and 16 jobs after 4.4 rather than 5.3.
The timeout is one budget for the process, from the signal to its exit. Its last moment, a quarter
of the release share and half a second at most, is the close's: waiting for outcome batches, ending
statements left on the server, closing the pool and finishing the loop each get only what is left
of the budget, and a statement cut once it is spent drops its connection after a twentieth of a
second. The worker's own shares, above, are carved out of the rest, so it returns that moment
before the timeout. A def job's thread is the one exception, since it cannot be stopped: once it
returns, its outcome gets a release share of its own, the moment included. At close no batch of
outcomes whose writers ran out of time is waited for, and a cut pipeline drops its connection once
the cancel is sent. With Postgres locked and a 4 second timeout,
eight jobs ending during the stall or just before it let the process exit 3.78 to 3.84 seconds after
the signal, where it took 4.53 to 9.58; eight def jobs whose threads returned 3.3 seconds after
the signal let it exit after 4.33 to 4.37, where their outcomes waited on the lock with no deadline
and it took 33; with a claim stuck on the lock it exited at 4.07, the interpreter's own exit
included. On Redis paused, the same shapes exited after 3.80 to 3.82 and 4.35 to 4.37, where the
threads took 5.14.
A job whose outcome is being written when the jobs' share ends is not cancelled: the write keeps
its deadline, which ends a second before the timeout, or half the release share when that is
shorter, and the check of a write whose answer is lost gets the rest. With a 2 second timeout and the
outcome held on a row lock for 1.3 seconds, it was recorded and its hooks ran, where it used to be
cut and left running for its lease. Outcomes that would wait for a connection go out together, in
one pipelined round trip of their own statements on both backends: over a link adding 300 ms each
way, four jobs ending just before a 3 second stop were all recorded with their hooks run, where
Postgres lost 6 of 16 and Redis left every one running with no hook. Each outcome keeps its own
answer, so of two claims of one job in a batch only the live one lands, where both read as written
and the stale one's failure was stored; an outcome its text cannot encode, or that the server
refuses, fails alone and the rest of its batch lands: 29 of 29 beside such a poison outcome, where
26 or, on Redis, 5 landed and the others stayed running. Texts holding a lone surrogate are
stored escaped. The statements carry their values as bound parameters, as alone, so a result or a
traceback never shows in pg_stat_activity or in the statement the server logs on an error, where
the batch wrote them into its text. An outcome is cut at its own deadline: outcomes with five
seconds or more left share a batch, and those with less share one with others whose time left is
within a factor of two, each batch cut at the earliest deadline among its outcomes, so a cut job
stays running as when written alone, where a batch without a deadline wrote it success. With
leases of four seconds, eight outcomes waiting for one connection over a 0.7 second round trip
went out in one batch, where each went alone, two timed out waiting and their jobs ran again. A retry
due in a batch wakes its listeners as when written alone, which took 0.10 seconds where it timed out
after 1.5. Closing the backend settles outcomes still waiting for a batch with the close error,
where two calls never returned; on Redis that error is not one a write is retried after, and a
script is refused once the backend is closed, where the retry reopened a connection on the closed
client and wrote the outcome after close() returned. A Redis batch waits for a connection only
when the pool is full, and takes the first one any call gives back: beside an outcome written
alone that held its connection 2.5 seconds, another under a one second deadline was written at
once, where it waited for that one, timed out after 1.55 seconds and left its job running.
On Redis a script missing from the server is loaded once per batch
and the batch sent again, one load where twenty of them took 6.1 seconds over that link.
A cancelled job still awaiting in its cleanup when the timeout
ends is cancelled once more and left behind, logged, for the reaper: with --shutdown-timeout 4 and a
cleanup awaiting 15 seconds the worker stopped after 4.0 seconds, where it took 18.0. A job that
swallows every cancellation used to keep the process alive anyway, since asyncio.run waits without
end for the tasks it cancels at exit; the worker runs its own loop instead, waits for them only until
the close end, and keeps the ones still pending referenced so their coroutines are never closed
outside the loop: with --shutdown-timeout 2 such a job let the process exit after 2.0 seconds, where
it never did. A worker leaving signals to its caller still owns Ctrl-C while run() drives its loop,
as asyncio.run did: the first one asks for the stop and run() raises KeyboardInterrupt once it
ends; a worker handling its own signals returns instead, so its exit keeps status 0 however many
presses it took. Every press counts until the loop answers it, through either signal path and on
loops without signal handlers alike, and a second press always forces the stop: the loop answers it
by cancelling and releasing the running jobs, and when it has not answered within 0.2 seconds, or
an earlier press is still unanswered, the press raises in whatever blocks the loop. The press marks the worker as forcing before the
loop runs again, so jobs claimed in the same batch that have not started yet go back to the queue
without running: with three blocking jobs claimed together the exit took 12 seconds, one job after
another, where it now takes 0.02. The loop is told to force the stop at the same time, so a job that catches the interrupt and goes on awaiting is cancelled
anyway, and the interrupt is raised again every 0.2 seconds until the loop answers. A worker
handling its own signals whose loop still has not answered a second later, as when a job catches
every interrupt in a blocking loop, ends the process with status 130 and leaves its jobs to the
reaper; a worker leaving signals to its caller keeps raising, as Python itself would. That interrupt is never a job's outcome: a job it ends goes back to the queue like a
cancelled one, where it used to be
recorded as failed and spend its attempt, and the run winds down with the forced budget. A job that
blocked the loop after the first press used to ignore every later one; two presses close together
could lose the force and wait out a 30 second budget. Both now end within the forced budget. Two
presses the kernel receives before it delivers the first are one signal, as for any process, and
are taken as one stop. A stop
asked while the backend is still connecting ends the connect, and so does one asked before the
run, where a silent server held Ctrl-C until its pool gave up 18 seconds later, and presses that land after a run ends no longer reach the next
run of the same worker. A press that lands once the run is already stopping counts as the second, so a worker handling its
own signals still forces a run that settles after its handlers are gone, where the left-behind job
that then blocked the loop held it 13.8 seconds. A worker that handles its own signals takes SIGTERM for the whole run, its start and teardown
included, and hands each handler back without passing through the default, so a second SIGTERM
while it tears down counts as a press where it killed the process with status 143. A handler set
outside Python, by faulthandler.register, a C extension or the program embedding the interpreter,
is put back as it was, and an unanswered press only re-sends a signal whose handler is the
worker's own, so a SIGINT handler the program kept never sees a Ctrl-C nobody pressed.
A signal sent to a process is delivered to one thread of it, any that does not block it, and
Python runs its handler only on the main thread, at the next instruction that thread runs. A main
thread that a job holds in a blocking call runs none, and a signal that another thread received
does not interrupt the call. So a press that landed on the control thread, the app's loop thread or a pool thread, while a job
blocked the loop, waited for the job to return, and two presses of one signal that waited
together were handled as one: seen once in 1,200 forced stops on Linux, and once on the release
runners, where the worker had not left after 15 seconds. Blocking the stop signals in the threads
tinyjobs starts would send them to the main thread, and was tried twice: a blocked mask is
inherited by every thread and every process started from the thread that has it, and each attempt
found another place where code that is not tinyjobs' runs on such a thread, an async job under
app.drain(), a coroutine given to run_sync, a dashboard request, whose child processes then
ignored terminate(). What ships instead leaves every mask alone and forwards the press. Python
writes the number of each signal the process receives, on whichever thread, to the loop's wakeup
channel, which the loop reads at its next turn. While a run's handlers are in place a thread
watches that channel without reading it: a stop signal it finds unread twice, a twentieth of a
second apart, means the loop is blocked, and the thread then sends the main thread SIGURG,
once for each such signal. SIGURG does nothing by itself and its handler is empty; the Go runtime
uses it to the same end. But it interrupts the system call the main thread is in, and Python then
runs every handler that is due, the stop signal's among them, which is what happens when the
kernel picks the main thread in the first place. A loop that reads its signals in time is never
nudged, and nothing is sent that a stop signal did not cause. A run started without
handle_signals still takes a Ctrl-C the program left at its default, and gives Python its
loop's wakeup channel for as long as it lasts, if none is set, so that the same thread sees it;
so does a worker on a loop that takes no signal handlers of its own, which a signal on another
thread used to leave asleep until its next timer.
None of this happens when SIGURG has a handler already, the program's own or one a library set
outside Python, or on a loop that is not asyncio's own; the disposition and the wakeup channel
are put back as they were found, in a process forked meanwhile too, unless something else took
them in between. Two presses of one signal less than a tenth of a second apart, the first on
another thread, can still count as one, and a main thread that blocks the stop signals around a
call is interrupted by the SIGURG all the same. When a run of
tinyjobs worker, or of manage.py tinyjobs worker run with the process's own arguments, ends cleanly, its
handlers become ones that exit with status 0: they give stdout and stderr half a second to flush,
and a second signal during that flush exits at once. So a press that lands after the run, while the
code after the command, its threads or the exit hooks run, keeps that status (on Linux 6 of 60 such
presses killed it with status 130), and a process held at exit by a thread, a child process, an
exit hook or a stalled pipe still stops on the next signal. Once the interpreter finalizes, it puts
the default handlers back, as it does for any Python program, and a later run of the command in the
same process starts again from the defaults. A run that fails
keeps its failure status. A program that calls tinyjobs.cli.main() itself keeps
the signal handlers it had. A run forced while it settles frees the slots of the jobs it leaves
behind, so the next run of the same worker claims again, where a worker of concurrency 1 never
claimed another job. Raised inside
a job, the interrupt was recorded as that job's failure and the worker went on serving; interrupted
with such a job and a 1 second timeout, run() now returns within its 1.25 second budget, where it
hung. The loop's default executor runs on daemon threads that are not joined at exit, so a job
awaiting asyncio.to_thread no longer holds the process: with an 8 second call and a 0.3 second
timeout it exited after 0.02 seconds, where it took 9.4. A run forgets the jobs it left behind, so the
same worker run again claims as before rather than counting them against its concurrency and renewing
their leases, and run() refuses a running loop before it touches the thread's loop.
A job running in a thread cannot be stopped, and releasing it would let another worker run it while
the thread still does. So at shutdown the worker leaves such a job running instead: it keeps its
lease renewed, records its outcome when the thread returns, within the release share of the timeout
from that moment, once more for each further pool's worth of running jobs, and returns after that.
Over a link adding 300 ms each way, six such jobs with --shutdown-timeout 3 or 5 were all
recorded on Redis and on Postgres, and twelve on Postgres's ten connections, where two to four of
six used to lose their outcome and run again, or be marked failed with max_attempts=1. It keeps the job as soon as the jobs' share ends, so a
thread that returns while another job's cleanup still holds the wait is recorded too: beside such a
job it ended success, where it used to stay running. A drain does the same: stopped while its
job runs in a thread, it waits for the thread and records the outcome, where it used to release the
job, which another worker then claimed while the thread still ran it. A second signal does not change this: it ends the wait at
once and keeps such a job the same way, where its thread used to be waited for under the cut
deadline, so its outcome was given up and the job ran again; with SIGTERM then SIGINT the job now
ran once and was recorded. A def job is not claimed while no thread is free for it; one still
queued in the thread pool at shutdown is taken out of the pool before it is released, so it never
runs here afterwards, and one whose call has started is kept like a running one. With a 20 second
def job and the server cut off, the worker exited 17.4 seconds after the signal, once the thread was
done and its outcome had been given up; before, the job was released while its thread kept running.
A claim whose caller is cancelled while it is in flight still lands in the backend, so tinyjobs lets
it finish and hands its jobs back, as if they were never claimed; otherwise a cancelled drain would
leave its job running until its lease ends. The claim runs on tinyjobs' own loop, not the
caller's, so this holds when the caller's loop ends with the claim in flight (asyncio.run
returning, Ctrl-C under it) and when the process exits: that loop finishes such claims before it
stops, including a hand-back sent while that loop is being closed, until the caller's end: the end
of its shutdown for a worker, its deadline for any other caller that has one. A claim the worker
gave up at shutdown is still handed back if it lands before the worker's share of the timeout ends,
and the app's close and the loop wait for that; past it neither waits, and its jobs come back
through the sweep of unanswered claims or their lease. A claim landing 0.1 seconds after the worker
gave it up, with a 0.2 second hand-back, was handed back before the close, where the close went
ahead at once and left the job running. Closing it cancels every other task at once and waits
five seconds at most, or what is left of a worker's shutdown budget, and half a second more for a
task that swallowed the cancel, which is cancelled once again: a task that keeps ignoring
cancellation is logged and left behind, so the close never hangs. At exit SQLite runs only the calls
that return jobs (complete, fail, release, hand back) on the loop thread once the interpreter
refuses new work to its executor; a claim is refused, so a worker still running stops instead of
claiming jobs it cannot start. A job whose job threads were already shut down is handed back without
using an attempt, never recorded as failed. Once the interpreter is exiting, tinyjobs starts no new
loop thread, and the exit closes only the connections no live thread is using, so a worker left
running in a daemon thread cannot crash the process. A job task cancelled before it starts is
released the same way as one cancelled while it runs.
These guards cost little per job. A drain keeps one renewal timer for all the jobs it holds, a SQLite call waits on a future its thread fills rather than on a shielded task, a job without a timeout starts without a message back to the loop, and a claim runs inside the worker's loop task with a timer that gives it up at shutdown. On Redis a script call and a batch of outcomes go straight to a pooled connection, past the client's command and pipeline wrappers, which cost about 6 microseconds of CPU a call; a claim token is a per-process random prefix and a counter, and a claim's answer leaves the token out. These paths and the pool bookkeeping rely on private redis-py names, so redis is required below 9 and a test checks those names are still there. Draining 10,000 jobs took 3.03 seconds in memory, 3.84 on SQLite and 5.46 on Redis, against 3.01, 3.82 and 5.57 before these guards existed; three CLI workers of concurrency 8 ran 4,408 jobs a second on SQLite and 6,148 on Redis, against 4,547 and 6,269.
A drain leases each job for a minute and renews the lease every 20 seconds while the job runs, so
the job of a drain killed mid-run comes back within a minute, where its one-hour lease stranded it
for an hour; and it writes a job's outcome again after an error that means the store could not be
reached, as a worker does, where it raised and left the job running: 3,000 jobs drained through
six half-second Redis cuts all ended success, each run once, where up to 6 stayed running. A
Redis claim carries a token; when its answer is lost, the next claim, at once and then once a second
at most until that lease would end, hands back the jobs still running under it as if never
claimed, as the sweep of unanswered claims does on Postgres, where they stayed running until their
lease ended and then counted as a failed attempt. A claim its caller gives up, cancelled or timed
out, is handed back the same way, where it left the job running until its lease ended. A worker
meets this whenever a reply is lost: with every twentieth claim answer lost, a worker
of concurrency 8 ran 2,000 jobs to success in 1.1 seconds, where without tokens 79 ended failed,
each once its 10 second lease ran out. The tokens are kept in one
sorted set, one entry per running job, removed with its outcome: 10,000 running jobs cost about 2 MB
more and nothing once done, and claims are within 5% of their speed without tokens, where a set per
claim, kept for its whole lease, doubled the keys and slowed claims by up to 25%. A drain given up
by its caller while a sync job's thread runs renews that
lease until the thread returns, where it stopped at once, so with the lease scaled down twentyfold
the lease lapsed and the next drain ran the job again beside the first thread; it ran once.
A worker or scheduler installs SIGINT and SIGTERM handlers only on the main thread; elsewhere it
runs without them, since Python delivers signals to the main thread only. Workers on one loop share
one handler, so a signal reaches each of them. When the last one stops it puts back what it found: a
handler the loop already had, or the process's previous handler, including one set outside Python
(read and restored with sigaction). A handler replaced meanwhile is kept. On asyncio's own loop the
shared handler is a loop handler. Elsewhere (uvloop, or a loop without add_signal_handler) it is a
Python handler that schedules the stop on the loop, so the loop's own handlers stay in place and keep
receiving the signal.
Inside a web server¶
app.serve(worker=True) runs the same worker as a task on the server's own loop, and
app.lifespan() wraps it in the shape FastAPI and Starlette expect. The server owns the process
signals, so an embedded worker installs no handlers: leaving the lifespan is its SIGTERM, with the
same drain and release. handle_signals=False is accepted there and True is refused.
A worker, a drain or a scheduler can be started again after it stops, on the same loop or a new one,
and stop() is safe to call from any thread; called before the start, it makes that run return at
once.
Where the limits are:
- One loop for both. An
asynctask that blocks stalls every request in that process, not only other jobs. - One GIL for both. CPU-bound
deftasks raise request latency; "Measured" has the numbers. Moving them totinyjobs workerchanges nothing in the task code. - No forced stop. The standalone worker's second signal has no equivalent, because the server
keeps the signals. Under
docker stopwith the default 30 s timeout, a 15 s job was killed at 10 s and finished 76 s after the restart, once its lease expired. Withshutdown_timeout=5the container exited cleanly and the released job finished 16 s after the restart. - One worker per server process.
uvicorn --workers 4means four workers on one queue, which the claim and the leader-free scheduler already handle.
Limits¶
@app.task(concurrency=2)
async def rebuild_index(tenant_id: int): ...
@app.task(rate="10/m")
async def call_partner_api(order_id: int): ...
concurrency=2 means at most two jobs of this task run at once, counted across every worker on
the store. rate="10/m" means two starts of this task are at least six seconds apart, so no window
of a minute holds more than ten. Both can be set on one task, and both are kept in the store, not
in a worker: a worker's own concurrency says how many jobs one process runs, a task's says how
many of that task the whole deployment runs. The limit is about the task, so app.send(name) from
a process without the task's code is limited like any other job, and two tasks never share a limit.
Why a rate is spacing and not a bucket. A token bucket lets an idle task start N jobs at once and then one per interval, which puts 2N - 1 starts in the first window; the quota it protects usually does not. Even spacing is the one reading of "ten a minute" that holds in every window, it needs one number per task, the time of the last start, and a backlog drains at the rate asked for. The cost is that a task idle for an hour does not catch up with a burst, which for a quota is the point. A burst allowance is a later option if usage asks for it, not a different design.
Enforced at the claim, through the store. The only place that knows what runs everywhere is the
store, so the worker passes the limits of the tasks it registers with every claim, and the claim
decides. An app without limited tasks pays little: SQLite's claim is the single statement it was,
Redis' script walks the ready sets as before, and Postgres' claim gains one step that raises the
running count of any claimed task that has a tinyjobs_limits row, which with no limits ever
declared is a join against an empty table; every exit from running on Postgres lowers that count
the same way. A task that once had a limit keeps its row there, so its claims and exits keep
locking that one row after the limit is gone, which on a no-op claim-and-complete loop measured
about half the throughput of a task without a row; an operator who wants it back deletes the row
once no process declares the limit, and a process that still does creates it again from the
running jobs. With limits, a candidate job of a limited task is admitted when the task's running count,
as the store keeps it, is under its concurrency, and, for a rate, when its last start is at least
an interval ago and no other job of the task is admitted in the same claim. A rate admits one job of
a task per claim, which is what spacing means.
A job the limit stops is held, not skipped. Skipping would make every claim read past every due
job of a saturated task, and a flood of one rate-limited task would put thousands of index entries
in front of everything else. So the claim moves such a job to HELD, a status of its own outside
the partial index the claim reads, and the index stays clean: each held job costs one write when it
is held and one when it is admitted, however many claims pass in between. Held jobs form a line per
task, ordered like the queue, by priority then run_at, and a claim with room for a task merges
the line with the due jobs it reads in that same order, so the limit never changes the order the
queue promised: a held job goes before a fresh one that is due later, and after a fresh one of a
higher priority, which is then the one admitted while the held one keeps its place. The line
respects the worker's queues: a held job in a queue the worker does not serve is left for one that
does. Holding is only ever applied to jobs the claim reached, so a held job can be seen in
tinyjobs jobs, in the dashboard and in counts_by_status, and cancelling it works as for a queued
one. A hand-back or release returns a job to QUEUED, and the next claim decides again.
Who wakes a held job. Nobody has to: the next claim with room does, and the worker knows when
that is. A job of a concurrency-limited task finishing in a worker raises that worker's room
flag, which ends its idle wait, so it claims again at once rather than after a poll; a worker whose
own claim started a rate-limited job notes when that task's interval ends and bounds its idle wait
to that moment, so the next start of a backlog comes an interval after the last and not an interval
plus a poll. Both are hints a worker gives itself; room another worker opened is found by polling,
so a worker that registers a rate-limited task also bounds its idle poll at the smallest interval
among its tasks, never below 50 ms, where it would otherwise back off to half a second, and a worker
on a store that wakes it, as Postgres does through LISTEN, bounds its wait the same way. A held job of a task
whose limit was raised in a new deployment is admitted by the first claim that sees the room, since
the claim admits as many from the line as the task has room for. A limit removed is the one case
no claim covers, since a claim only reads the lines of the tasks it has limits for; so a worker or a
drain, as it starts, returns to the queue the held jobs of the tasks it registers without a limit,
through release_held(tasks=..., queues=...), within the queues it serves, and they are claimed as
the unlimited jobs they now are; the worker repeats the release on every reaper pass, since a worker
on an older build that still has the limit holds them again on its next claim, and a release only at
start would leave whatever it holds after that start held for good. A task not registered here is
not touched, since its limit is someone else's to know, and a queue not served here is not touched
either, so a service that registers a task only to send it, on queues of its own, leaves the
service that runs it alone. While old and new builds overlap the two sides take turns, which is
bounded churn and ends with the deployment. The release is best effort: any store error there is
logged and the worker goes on, since the next pass tries again.
Held jobs count as queued. Every signal that counts due jobs counts held ones with them: the
per-task queued counts and the oldest due job of each queue behind tinyjobs stats, the
dashboard's overview, the tinyjobs_store_queued_jobs and tinyjobs_store_oldest_due_seconds
metrics and tinyjobs check --max-wait. A held job is due by definition, since the claim reached
it, and a backlog stuck behind a saturated task is exactly what --max-wait exists to report.
The running count. SQLite is a single writer, so the claim counts the task's running rows under
the write lock, through a partial index on (task) WHERE status = 'running', and nothing can drift.
Redis keeps running_tasks beside queued_tasks, adjusted by the one pair of functions every
transition goes through, so it is exact by construction. Postgres runs claims concurrently and reads
under READ COMMITTED, where a count in a statement cannot see a claim that committed after the
statement's snapshot, so there the count is a column on the task's row in tinyjobs_limits, locked
FOR UPDATE by any claim that has a candidate of the task and read again after the lock, raised
by the claim and lowered by every transition out of RUNNING, including the reaper and the sweep
of lost claims. The reaper also writes the counts back from the running rows once per interval for
the whole store, so a deployment that mixed a build without a task's limit with one that has it
corrects itself within half a minute.
The row is locked only when the task has a candidate, so claims of unlimited work never wait on
it. A pg_try_advisory_xact_lock guards the claim as before; when it is not taken nothing is
admitted, the counts are not raised, and holding still happens, which is correct either way.
The claim with limits, per store.
- SQLite runs the claim as a loop in one write transaction: it reads the task's running counts
and last starts once, then in each slice a batch of due jobs and, for each limited task with room,
the head of its held line, merges them in queue order, admits or holds each, and reads the next
slice after a slice held something, up to eight slices, so a saturated task's jobs leave the
index without starving the work behind them. The rows are taken with the
UPDATE ... WHERE id = ? AND status = ?of the pre-3.35 path, since a decision is made per row. - Postgres keeps one statement. The level walk picks due jobs as before; the tasks among them that
have a
tinyjobs_limitsrow, and the limited tasks with a held job in the wanted queues, have their rows locked; the held lines are readFOR UPDATE SKIP LOCKEDup to each task's room; a window numbers the candidates per task in queue order; the ones within the room are claimed, the others of limited tasks are held, and the task rows are raised, all in data-modifying CTEs of the same statement. The read of due jobs is bounded by the batch, so when it comes back full, a held job that sorts after its last row waits for a later claim, and the line never jumps ahead of fresh work the queue put before it. When a run holds jobs and still falls short of the batch, the backend runs the claim again, up to eight times, each run a complete claim of its own with its own token, all within the half lease one claim is given, so the jobs of the first run come back with lease time left; the jobs it held have left the claim index, so each run reads the next due jobs. A claim without limits still raises the running count of any task that has a row, so a worker that does not register a task's limit, or hands such a job back as unknown, cannot drive the count below the jobs actually running. The rate is checked and stamped against the same server-bounded time as the rest of the claim, so two machines whose clocks disagree cannot start one task twice within an interval. The claim locks, in task order, the limits row of every task among the jobs it read, whether or not this worker limits it, and raises the count of each task it starts that has a row, so a worker that limits other tasks still counts the running jobs of this one. Over several queues, the claim reads each queue's jobs at a priority level before the next queue's, so a full read can leave earlier due jobs of another queue unread; the claim finds the first unread due job of each wanted queue at the last level it read, and a held job that sorts after the earliest of them waits for a later claim. Held jobs form a line per task and queue, read through a partial index on(task, left(queue, 500), priority DESC, run_at ASC)over held jobs, so a claim on one queue never reads past the held jobs of another, and long queue names share the 500-character key the claim index uses. A claim reads a further slice only when the previous one held jobs and its read of due jobs came back full, since a short read means nothing else was due. A task's row in the limits table is created the first time a process claims with the task's limit, starting from the jobs of the task running at that moment; once the creation has committed, the process counts again, locking the task's running jobs for share before the row, in the same order as every transition out of running, which locks the job and then the row, so a completion under way settles first instead of deadlocking with the count. Two narrow cases remain until the next reaper pass writes the count back: a claim of the task that took its snapshot before the row existed and commits after the recount began leaves the row one low, and a transition out of running that took its snapshot before the row existed and reaches its job only after the recount locked the running jobs leaves the row one high, which underconcurrency=1stops the task until that pass. Once created, a row is never deleted: a worker that registers the task without a limit returns the task's held jobs in its own queues to the queue, in batches of 5,000 each its own statement so a large backlog never runs into the statement timeout, and leaves the row with its running count and last start, so a stub of the task in another service cannot reset the spacing of a rate; a process that finds a row missing, as after an operator empties the table, creates it again from the running jobs and reads the next slice. A rate stamp is not rolled back when its claim is swept as lost, so a lost claim costs one interval. The reaper writes the running counts back from the jobs once per interval for the whole store, not once per worker: each pass first tries to move alast_recount_attime in the meta table by at least thirty seconds, forward or, after a clock step left the stored time ahead, back, and only the pass that moves it locks the limits rows and sets each from one grouped count of the running jobs, read through a partial index on(task)over running jobs; the passes of the other workers skip it. The upgrade from the first schema version runs without the statement timeout, so building its indexes on a large table is not cancelled halfway. Schema version 2 is reached in the setup transaction of the first 0.15 process: the DDL creates the limits table and the held index, then the dedup index is recreated withheldin its predicate and the statusCHECKconstraint is swapped for one that allowsheld, named<prefix>_jobs_status_checkwhatever the old one was called, validated on the spot, since every row satisfied the stricter one. The pass holds the setup lock and blocks writes to the jobs table while it scans it, which is the stop-the-world step the upgrade notes name. - Redis keeps one script. It reads
running_tasksandlimit:<task>, and a task's held jobs sit in one sorted set per queue,held:<length>:<task>:<queue>, the task's byte length first so that no two task and queue names share a key, with the ready set's order. The claim reads the head of the line of each queue it serves, drops any entry whose job is gone or no longer held, and reads on.held_queues:<task>names the queues with a line, whichrelease_heldreads when it is given no queues. Each slice merges the fresh rows with those heads, and when a queue's read comes back full, nothing that sorts after its last row is decided in that slice. The claim holds what it cannot admit and reads a further slice only when a slice held something and a read came back full, up to four slices. Held jobs count inqueued_tasks, andheld_due:<queue>gives their oldestrun_atto the stats. One reap pass a minute across the workers, gated by a store-widerunning_recounted_attime checked in the same script, and the move to schema 3 rebuildrunning_tasksfrom the running index, so a count left wrong by a 0.14 process is corrected within a minute while every other pass costs one read.
What the limits do not do. Eager mode runs the task inline at the call and ignores limits, since
there is no claim. app.drain() claims and so honours them, which means a drain returns with jobs
held for a rate until the clock moves. A limit on a task the worker does not register is unknown to
that worker, which claims the job as unlimited and hands it back as unknown, as before. A backend
plugin that does not declare limits in its capabilities refuses to start a worker or a drain whose
app registers a limited task, rather than run it unlimited.
Jobs after another¶
report = build_report.enqueue(42)
email_report.with_options(after=report).enqueue(42)
The second job runs only once the first has succeeded. This is the one piece of workflow most applications ask a queue for, and it is kept as one piece on purpose: a job names at most one parent, a parent can have any number of children, and nothing else exists. No "after all of these", no group, no chord, no result passed down the line. Those are where a queue starts needing a scheduler of its own, with state to join and partial failures to define, and a tree of single parents needs none of it: every job's fate depends on one row.
A status, not a flag. A child is written as WAITING with its parent's id on the row
(runs_after; after on a Job). A status of its own keeps it out of the claim's partial index,
so claims never scan past waiting jobs however many there are, and out of the backlog figures,
since a job that waits on another is not late. HELD could not be reused: workers hand held jobs
of tasks they know no limit for back to the queue, which would start a child early.
And a column that is only set while it waits. runs_after stays on the row for good, for
jobs show and for a retry to ask the parent again. The lookups that settle children go through
a second column, waiting_on, which holds the same id while the job waits and is null from the
moment it is released or cancelled, with a partial index on it where it is not null. The first
version looked children up by runs_after and status = 'waiting', and that predicate gave both
SQL planners a choice: the index of jobs by status serves it too, and with statistics taken
before the waiting jobs arrived, after an upgrade or a burst, or with the sampled ones
PRAGMA optimize leaves, they took it, reading every waiting job once per descendant. A lookup by
waiting_on alone has one index that can serve it, whatever the statistics say.
Settled where the parent ends. Every path that ends a job also settles what waits on it, in
the same atomic step, with the exceptions Postgres makes below: complete queues the children; a final fail, cancel and the reaper
failing a job out of attempts cancel every descendant, since a waiting child's own children can
only be waiting too. A released child keeps its run_at unless that has passed, in which case it
becomes due now, so the oldest-due figure measures how long it waited for a worker and not for its
parent. A cancelled child gets last_error "the job this one was to run after did not succeed",
no execution row, and no hooks, as with any job cancelled before it ran. An attempt that failed
and will be retried ends nothing, so nothing moves.
On SQLite the settling is two statements in the write transaction that records the outcome, a
plain UPDATE by parent for a success and a recursive one for a failure, both on the partial
index of waiting jobs by parent. On Redis it is a loop inside the same script, over the set
children:<parent> that enter and leave keep. Neither has a window: the store's single writer
orders the child's insert and the parent's end.
Postgres has a window, and closes it twice. There the outcome statement gains a CTE that
releases or cancels the children it can see. The walk down the line is a lateral lookup and
waiting_on is declared all-distinct to the planner, because left to the statistics of a few
parents with thousands of children each it chose joins that reread every waiting job at every
step. What it can
see is every child committed before the statement began. A child whose insert commits after that, because the enqueue raced the parent's end or sat
in the application's transaction while the parent finished, was placed by a snapshot in which the
parent still ran, and is seen by nobody. Taking a lock on the parent at insert would close the
window and let an application transaction block the workers from claiming the parent or recording
its outcome, which is the wrong side to make wait. So the insert locks nothing, and the reaper
pass settles stragglers: every waiting job whose parent is terminal is released or cancelled by
the same rules, at most reap_interval late. The pass reads the waiting jobs through their
partial index, so it costs nothing while none wait and one lookup of a parent per waiting job
while some do; the upgrade analyzes the new column so that this holds from the first pass on a
store that came from 0.16, and a new store reads its whole table in the pass, 12 ms at 100,000
jobs, only until autovacuum has analyzed it once. Retention plays along: a purge skips a parent that still has a waiting child, so a straggler
finds its parent's outcome. What is left is a child whose insert was still uncommitted when
retention removed its parent, which takes a transaction held open for longer than the retention
horizon: the pass cannot know how that parent ended, so it cancels the child with a last_error
that says the parent is gone, and a retry runs it. Released children wake their queue's
listeners like an insert does.
That window is not rare when children are sent right behind parents that finish fast: four senders against four workers doing nothing else left 51 and 60 of 2,400 children waiting for the pass. So an enqueue outside a caller's transaction looks once more at the children it has just stored as waiting, by id, after its insert has committed, and settles the ones whose parent ended in between. It is one more statement for a child sent that way, two when it finds something to settle, and none for any other job: a single child takes 0.65 ms to send where it took 0.32, against 0.16 for a plain job, and it left 4 and 9 of the same 2,400: what remains for the pass is a parent whose outcome statement was itself still running across both, or a child sent inside a transaction that commits later.
The CTE is not free: Postgres plans every outcome statement, and a second look at the jobs table,
with its partial indexes to prove, takes a completion from about 150 to about 195 microseconds
whether or not anything waits. So a store pays for it only once it has held a waiting job. The
first child ever sent leaves a following: row in the meta table, for good. It is an insert of a
row of its own when it sees none, never an update of a shared one, so it waits for nobody, a
REPEATABLE READ transaction that began before the first row committed adds its own instead of
failing, and two that race leave two rows. Only two SERIALIZABLE transactions that both send the
first child a store ever held are a write skew, and Postgres fails one of them as it would any
other. The role that sends a child therefore needs INSERT on the meta table as well as on the
jobs; the reaper
pass leaves one if the store holds waiting jobs and none is there, as after a rollback. A
completion looks for such a row, which costs a fifth as much, and a process that reads it set queues the children of the job it
just completed with one more statement and uses the statement with the CTE from then on. If that
one extra statement fails, the reaper pass does its work.
Placed at insert. The parent has to be in the store when the child is sent; a missing one
raises JobNotFoundError and the job is not written, nor is anything else of the call unless
it joined a caller's transaction, where what the call had written is the caller's to roll back,
or retention removed a parent between two of the hundred-job scripts of one large call on Redis. A parent that already succeeded
gives a child that is QUEUED at once, one that already failed or was cancelled a child stored
as CANCELLED, and the Job that enqueue returns says which. A waiting job holds its
deduplication key like any job in flight.
Requeue asks the parent again. requeue on a failed or cancelled job with a parent places it
as the insert would: QUEUED if the parent succeeded or is no longer in the store; WAITING if the parent
is in flight again; refused while the parent is failed or cancelled. A parent that is gone cannot
say how it ended, and refusing there would leave a job that failed on its own impossible to retry
once retention had removed its parent, so the request of whoever retries it is taken as the answer. Requeueing a parent does not
bring back the children its failure cancelled: nothing records which cancelled jobs were cancelled
by it rather than by hand, and a guess there would run jobs somebody stopped.
Eager mode needs nothing of its own: a parent sent eagerly has usually ended before the child is sent, so the child is placed queued and runs at once, or is placed cancelled. A parent that is still queued, because it is delayed or has a retry ahead, leaves the child waiting like anywhere else, until a drain or a worker finds the parent due and runs both in order.
Retries¶
@app.task(
max_attempts=5,
retry_on=(ConnectionError, TimeoutError), # default: all exceptions
retry_backoff="exponential", # or "fixed", or a callable
retry_base=1.0,
retry_max=600.0,
retry_jitter=True,
)
async def sync_invoice(invoice_id: int): ...
Delay for attempt n is min(retry_base * 2 ** (n - 1), retry_max), multiplied by a random factor in
[0.5, 1.0] when jitter is on. Jitter is the default and it is not cosmetic: when a downstream service
goes down and comes back, un-jittered exponential backoff makes every failed job retry in the same
instant and knock it over again.
A retry is not a new job. It is the same job with attempts incremented and run_at pushed into the
future, plus a new Execution row. The job id is stable for its whole life, which is what makes it
usable as an idempotency key downstream.
Two escapes from the default policy, raised from inside a task:
raise Retry(in_=30): retry at a time you choose, ignoring the backoff curve. For honouring aRetry-Afterheader.raise Abandon("reason"): fail immediately and permanently, skipping remaining attempts. For errors that will never succeed on retry, like a 422.
Jobs that exhaust their attempts stay in the store as FAILED with the full traceback of every
attempt. tinyjobs jobs retry <id> resets the counter and requeues, giving you a dead letter queue without a
second queue to operate. A retry is refused while another active job holds the same dedup key. A retried
job is due at once; one cancelled before any worker took it keeps its place from when it was sent.
Results¶
Off by default. Most background jobs are run for their side effects, and storing return values that nobody reads is pure write amplification on the hot path.
@app.task(store_result=True)
async def render_report(report_id: int) -> str:
return "/tmp/report.pdf"
job = render_report.enqueue(42)
result = job.wait(timeout=30).result() # polls, then decodes the stored value
Results are serialized with the same codec as the arguments and stored on the job row, so they live as long as the job does and go when retention purges it, or earlier with a TTL:
app = TinyJobs("sqlite:///jobs.db", max_result_bytes=64 * 1024)
@app.task(store_result=True, result_ttl=3600)
async def render_report(report_id: int) -> str: ...
A large result belongs in object storage, with the job returning a key, and max_result_bytes,
off by default, makes that a rule rather than advice: an encoded result over the cap fails the job
with ResultTooLarge, and a result the codec refuses fails it with UnencodableResult; both are
a ResultError, a kind of SerializationError. The function already ran, so neither failure is
retried: retrying would run the side effects again to hit the same deterministic error. The cap is
checked in the worker after encoding, so it holds for isolated tasks too, whose result is decoded
from the job process and encoded again; a job process whose result the codec refuses answers with
that, so the parent fails the job the same way.
result_ttl drops the stored result that many seconds after the job finished, leaving the row
and its history; job.result() then returns None. The sweep runs at every retention check of
every worker's housekeeping loop, a quarter of the retention interval and at most five minutes,
for the tasks that worker registers, without the election that pruning uses: it is an idempotent
batched UPDATE per task with a TTL, repeated until a batch comes back short, over the same
(status, finished_at) index that pruning walks. A pass over the tasks stops at
maximum_per_sweep results, like pruning, and the next pass starts from the task after the one
that used them up, so a busy task cannot starve the others; passes go on until one comes back
short everywhere, as pruning does, and every check starts again from the first task. A task
whose sweep fails is logged once and sits out the rest of that check, as a failing prune does.
A task that came back short has nothing older than its cutoff, so the worker keeps that cutoff,
less one check interval of slack for a job finished just before it on a clock slightly behind, as
the floor of its next sweep: the first sweep after a start walks every old success of the task
once, and the ones after it only the window since the last. On Redis a per-task sorted set of
the jobs holding a result, scored by finished_at, is kept by complete and trimmed by the sweep
and by pruning, so the sweep never scans the finished jobs. A result stored before the set
existed is not in it and lives until its job is purged.
job.wait(timeout=...) and await job.awaited(timeout=...) poll with the same adaptive backoff as
the worker loop, and return the job rather than raising on failure: a job that failed still has a
status, an attempt count and a last error worth reading. Only the timeout raises.
They are a convenience for scripts, tests and glue code, not a synchronisation primitive. Blocking a request thread on a background job means you did not need a background job.
The dashboard¶
tinyjobs dashboard --app tasks:app serves a page of queues, jobs and schedules, with the same
service calls behind it as the CLI: stats, check, jobs, get, executions, retry,
cancel, schedules, pause_schedule, resume_schedule and run_schedule. It adds no
dependency. The pages are HTML rendered in Python with html.escape, one stylesheet and one
script of a hundred lines, served by the same stdlib http.server plumbing as the Prometheus
exporter, in tinyjobs.serving, and loaded only when the command runs or the app is mounted.
Flower needs Tornado and rq-dashboard needs Flask; the point of tinyjobs is that pip install
tinyjobs is the whole install, and a dashboard that changed that would have cost more than it
gave. An optional extra was considered and rejected for the same reason: there is nothing to put
in it. Should the pages ever want a frontend build or a framework, tinyjobs[dashboard] is the
place, and adding it later breaks nobody.
The core is one object, Dashboard(app), whose handle(request) is a coroutine taking a
Request of method, path, query and root and returning a Response of status, body, content
type and headers. The action forms carry no fields, so no entry point reads a request body: the
stdlib server discards one up to 64 KiB and refuses a larger one with 413 before acting, and a
WSGI or ASGI server keeps it. A schedule name is one URL
segment, percent-encoded in the form action and decoded by the server, so a name with spaces or
accents works, and one with a slash does not, so such a schedule gets no buttons. The core calls
the app's async methods, so it runs on any loop: the stdlib server and the WSGI callable hand
each request to app.run_sync, under the name of the writing call it carries so
that the Postgres guard against a held Django transaction applies to a retry from the page as it
does to an enqueue from a view; the ASGI callable awaits it in place. Mounting is wsgi(app) and
asgi(app) from tinyjobs.dashboard, at any prefix, since every link is built from the
request's root (SCRIPT_NAME, root_path). Each page is built as a dict first and rendered
after, so the same handlers answer /api/... with JSON: that is what the tests assert on, and
what a script that wants the numbers reads, with curl.
The pages are what the runbook needs. The overview is stats and check on one screen: counts by
queue and status, the oldest due job, leases past their time, queued jobs by task and every
finding of the health check. The job list is the store's own list: newest first, by status and
queue, fifty a page, with the cursor the backend returns, so it costs the same on a million rows
as on ten, except status and queue together, which walk the status index and filter by queue; a filter by task would need an index no backend has, so a task is searched from the
overview's counts and by its queue, and a job by id in the address bar. The job page shows the
job's fields, its arguments and result decoded with the app's codecs (a payload the app cannot
decode shows the error in their place), the last error, and every attempt with its traceback
folded under a <details>, with retry and cancel where the lifecycle allows them. The schedules
page is tinyjobs schedules with its three actions as buttons. A worker list was left out: the
store knows workers only through the jobs they hold, and a list of worker ids taken from running
jobs would be empty exactly when the question matters. A heartbeat table is a backend change, and a
later one.
Actions are POST forms answered with a redirect back to the page and a one-line notice, so a
refresh never repeats them. The page reloads its main block every five seconds through fetch,
keeps the notice and the open tracebacks across the reload, and skips it while a form control or a
keyboard-focused link has focus, text is selected, a filter has unsent edits or the tab is hidden. Timestamps are shown in UTC with the age beside them, both
from the app's clock, so a test with a frozen clock reads the page it expects; the lists show the
age alone, with the clock time on hover, so nine columns fit on one screen.
The server binds 127.0.0.1:8070 by default, --host 0.0.0.0 exposes it, and the bind errors are
the exporter's, with their remedies. The command serves until Ctrl-C or, when it owns the
process, SIGTERM, through stop_on_signals() in tinyjobs.exiting, the one module beside the
worker's handlers that touches the signal API; it installs its handlers only on the main thread and
only over the defaults, and restores them on exit. HEAD is answered like GET without the body, so curl -I
and a load balancer's probe work; a wrong method is a 405 with Allow, decided before the store
is asked, and an unknown path a 404. There is no authentication and no CSRF guard, by design: the
dashboard retries and cancels jobs, so it belongs behind the reverse proxy that already guards the
rest of the deployment, where Flower and rq-dashboard sit too; a login of its own would be a second
one to configure and keep secure. A page that cannot reach the store answers 503 with the error,
so a load balancer sees the store's trouble rather than a hung page: every page's reads run under
the exporter's budget, shared through within() in tinyjobs.deadlines, and on the overview the
stats query and the health check share one deadline, the check reusing the stats it is handed
instead of asking the store twice. The actions are not cut short, since a write cancelled
mid-flight could still commit and then be reported as a failure.
Public API¶
Async core, sync facade¶
The backend protocol is async. Every method a backend implements is a coroutine. On top of that
sits a synchronous facade, so enqueuing still works from a script, a Django view or a __main__
block with no event loop in sight.
An earlier version of this document argued the opposite, and the argument was half right. The
diagnosis was correct: sqlite3 is a blocking C library, the standard library ships no async driver
for it, and an async def enqueue sitting directly on a blocking sqlite3 call is not asynchronous.
It is a synchronous call that has taken your event loop hostage.
The conclusion drawn from it was wrong. The problem is not the async keyword, it is where the
thread hop lives. Put it inside the SQLite backend and async def claim becomes honest, because it
really does yield control:
async def claim(self, ...):
return await self._call(self._claim_sync, ...)
Put it in the architecture instead, and every future backend pays for a limitation that belongs to SQLite alone. A natively async backend would have to expose synchronous methods, which means running its own event loop on another thread and bridging back to it. Since the whole point of the backend seam is Redis, Postgres and whatever comes after, that is the wrong trade.
The old shape also leaked in a way that is easy to measure. Because asyncio.to_thread spreads calls
across the default executor's thread pool, and SQLite connections are thread-local, a worker doing
nothing unusual ended up holding ten connections instead of one. Moving the hop into the backend,
onto a dedicated single-worker executor, brings it back to one and serializes writes inside the
process for free, which is exactly what SQLite's single-writer model wants anyway.
job = generate_pdf.enqueue(123) # sync facade, from anywhere
job = await generate_pdf.aenqueue(123) # native path
The facade drives a single background event loop, started lazily on the first synchronous call, and
marshals coroutines onto it. The obvious alternative, one asyncio.run() per call, was rejected
because connection pools in async network clients are bound to the loop that created them; a fresh
loop per enqueue would break precisely the backends this design exists to support.
That loop is also where such a backend lives. Redis declares Capabilities(loop_bound=True), and the
app's connection is always opened on the facade's loop, whoever connects first: a sync call,
ready(), a lifespan or an embedded worker; worker.run() opens a pool of its own on its loop, as
described under Event loops. A call from any other loop hops there and awaits the result, and cancelling it
cancels its work there. One app therefore serves any number of loops at once, a lifespan and a
worker thread with its own asyncio.run() included, and a loop that ends or is stopped takes nothing
with it. Before this rule a FastAPI app on Redis without a lifespan answered 421 of 1,000 mixed def
and async def requests with a 500, and a worker on a second loop failed every claim; now neither
does. SQLite and memory:// hold nothing bound to a loop and are used in place from any of them.
The hop costs about a tenth of a millisecond, 3,600 sequential enqueues a second from one loop
instead of 5,200 when that loop held the connection; four workers still drain 20,000 jobs in 2.1
seconds.
Closing is ordered. aclose() refuses new calls with AppClosedError, waits up to five seconds for
the calls already running, closes the backend and stops the facade's loop. A call caught by the
close gets AppClosedError too, never a RuntimeError from a shut-down executor or a closed pool:
four threads calling enqueue, start and close and three event loops calling aenqueue, ready
and aclose at random for 9 s saw nothing else, on SQLite and on Redis, and asyncio logged no
unretrieved exception. Once a close begins, a drain or eager call that would start is refused with
AppClosedError, and an eager call whose async def job the close cuts raises it too: the job ran
once and is left queued, where it was started again in a new drain and ran twice.
A loop being shut down refuses new work instead of running it beside the
closing connection. A
sync call on a closed app raises without starting the loop thread, and an app dropped without
close() stops that thread and closes its connections when it is collected, so python -X dev stays
quiet at exit. That cleanup runs on a threading.Thread of its own: run from a bare OS thread, it
would leave a dummy thread in threading for good on 3.12, and every later fork would
count it as a foreign thread and skip the pause. At exit tinyjobs waits for these cleanups,
including one whose thread is registered but has not started yet: joining it straight away raised
RuntimeError and skipped the rest of the exit cleanup. Closing stops the app's workers first and waits for the jobs they are running, up to
their shutdown_timeout, so a finished job is never left running. A job that closes its own app
does not wait at all: it tells every worker to stop and returns, and the close takes effect once its
runner has recorded the outcome and the other workers have finished, so two jobs in two workers that
each close the app do not wait on each other. A worker or
scheduler whose app is closed stops instead of retrying. After
os.fork() the child leaves the parent's loop thread and connections alone, never closing them, and
connects on its first call, so an app started before a preforking server forks works in every child.
A fork waits for SQLite calls already running in other threads, and for the write transactions open
there, up to ten seconds, so the child never inherits a lock held by a thread it does not have, and
new calls wait from the moment the fork starts until it is done. SQLite keeps its file locks in
per-process memory that a fork copies, so a child forked while another thread still holds a write
open, past those ten seconds, cannot write to that database at all. A thread that forks inside a transaction does not wait for itself, and the child
cannot commit or roll back the parent's transaction. When tinyjobs' own threads are the only other
threads, the fork first pauses them, before the wait, and they resume right after, so Python's
multi-threaded fork() warning stays quiet and an idle app adds no delay. A SQLite executor thread
that the pause retires leaves its connection to the thread that replaces it, so no thread is left
closing it when the fork happens. A SQLite call queued, or waiting at the fork gate, when the fork
starts steps aside: it runs on the next executor once the fork is done, so that executor's thread
retires too. Cleanups inherited by a child, such as the close of a connection whose thread the child
does not have, do nothing there. The pause gives up rather
than wait for work: a loop that does not stop within 0.1 s (a job blocking it) or a job running in a
worker thread through asyncio.to_thread forks with the threads running, and the warning is then
accurate. So does a process with threads Python did not start (a BLAS pool, a gRPC channel), where
Python warns whatever tinyjobs does. A call already waiting on the loop when it pauses keeps waiting
and gets its result once the loop resumes, a call made during the pause is served then, and a close
that lands during a pause still stops the loop and closes it. A closed SQLite backend stays closed
across a fork. A signal handler that raises while a fork waits, Ctrl-C or a SIGTERM handler calling
sys.exit(), does not cut the wait short: the wait starts again, up to three times, and the exception is raised from the
fork hook afterwards, where CPython reports it and drops it, as it does for any fork hook. Without
tinyjobs the handler would run after fork() returns; with it, that exception is lost.
A signal handler may call the synchronous API. The calling thread waits on a lock of its own, which
is released when the work is done, so a handler that interrupts a sync call in progress is served.
Two cases cannot be: the handler interrupted its thread while that thread held one of tinyjobs'
locks, which only async code takes, or while it was inside the fork hooks, where the work it would
wait for needs that same thread to go on. There the call raises InterruptedCallError at once
instead of hanging, close() included, which then leaves the app open; setting a flag in the handler
and making the call after it returns always works. tinyjobs' locks are plain C RLocks it keeps weak
references to, and whether a thread holds one is read from the lock itself, so a thread that is only
waiting for a lock does not count as holding it. A forked child frees every such lock another thread
held at the fork and keeps the ones its own thread holds.
A handler that raises, such as Ctrl-C, cannot leave tinyjobs half-done either, and tinyjobs never
touches signal handlers, signal masks or the wakeup fd to get there. Python runs handlers only on the
main thread, so the main thread keeps no books. A synchronous call, close() included, builds its
request, hands it to tinyjobs' control thread through a SimpleQueue and waits on the request's
lock. That thread starts and stops the loops and schedules the call, and the coroutine is made on the
loop, so an interrupted call is either cancelled before it starts or finishes without its waiter; a
close keeps going on its own thread and the app still ends closed. The fork hooks follow the same
rule. The hooks that run after a fork are C callables, so no handler can run inside them. The hook
before it only adds entries keyed by its own thread, which those C hooks remove, and asks the control
thread to pause the loops and exit; in the parent a C hook starts the thread that resumes them, and a
child settles itself from the first such thread it starts. Each pause carries its fork's own resume
callable and is dropped once that fork is over, so a pause left behind by an interrupted hook never
stops the loops during a later fork, and a pause that outlasts its fork resumes the loops itself. A handler that raises inside a fork hook
is reported and dropped, as CPython does for every fork hook, and the fork goes on. SQLite's fork gate
and connection pool keep their state in dicts and sets changed one C call at a time. A write nested
inside another joins it through a savepoint, as in 0.6, so a failure inside it undoes only its own
work, whether it sits in a plain block, a coroutine, a @contextmanager or @asynccontextmanager
helper or an ExitStack. A nested write held by a generator that other code drives with next() is
different, because the outer code keeps writing while the generator is suspended: closing that
generator keeps what was done, and any other failure rolls back the whole transaction if anything
changed since the nested write opened, schema and user_version included. A nested write
interrupted before it finished poisons the outer one, which rolls back instead of committing. A nested
write still suspended when the nested write it joined exits passes to the outer one: its work is
part of that transaction, and a later failure of it, or the exit of the write it left having
failed, rolls the whole transaction back. A root write that ends while such a write is still
suspended commits or rolls back without it and retires its connection: the pool forgets it, closes
it once no call runs on it, and every later statement through it, from any frame, cursor or
thread, fails with ProgrammingError instead of writing outside a transaction, as does a new
write or pool.connection opened from inside the block of the write cut loose. One that ends while
a write that joined it is still running on the same thread, as when a generator dependency is
closed from inside the endpoint's own write, waits for that write and then commits or rolls back,
without any write that was only suspended; a rolled-back outer makes that write raise instead of
committing, and so does an outer poisoned meanwhile, even when that write failed itself. One that ends while a joined write runs on another thread retires the connection the
same way, stops the statement in flight, then rolls back and raises. A transaction whose
connection another thread used meanwhile, as when asyncio.to_thread outlives a timeout, retires
it at the end the same way, so that thread's later statements fail instead of committing on their
own: a committing root waits out a statement still running there, a failing root stops it first,
and a failing nested write rolls the whole transaction back. A write entered by hand inside an open one must
exit before it, or the outer one rolls back, raises and retires the connection. Every connection
the pool closes is retired first, so a close never lands under a statement another thread is
running. A write that would
join a transaction SQLite has already ended, as INSERT OR ROLLBACK does, is refused, and the
outer write rolls back. A write entered by hand and dropped without exiting is rolled back when it is collected,
together with any write that joined it meanwhile, and logs a warning on tinyjobs.sqlite. A write counts as open while the with block that holds it runs: the block where it was
entered, or, for a write opened inside some __enter__, the block that entered that; the frame's
exception table and the jumps inside the block tell this exactly. A write entered through an
ExitStack counts as open while that stack still holds its exit, wherever the stack moves, and one
opened by a generator outside any block, such as a pytest fixture, counts as open until it exits.
So a later write never joins a dead one, and the fork gate counts a write as held until its exit is
done. A later write joins an open one when it runs inside its block, in the same contextvars
context, or in the same asyncio task; otherwise it gets a connection of its own and waits for the
lock, so concurrent FastAPI requests served by one thread never share a transaction; two asyncio
tasks on one thread are the exception described below. Context is
what Python's contextvars says it is: asyncio.to_thread and FastAPI give each call its own copy,
while loop.run_in_executor runs everything in the worker thread's one context, so work sent there
joins a transaction an earlier call left open on that thread. That also
means a write opened by a sync FastAPI endpoint does not join the one its sync dependency holds,
since FastAPI runs each in a separate context; use the connection the dependency yields. A
transaction whose entry or exit was interrupted is rolled back by the next tinyjobs call on that
thread, sync or async, even while something still holds the interrupted write, and at once when
nothing does. A write may exit on another thread than the one that entered it and commits there,
untouched by the first thread meanwhile, and its connection stays open if that first thread ends.
Two asyncio tasks on one thread share its connection, so a task that writes while another task's
transaction is open gets an error instead of joining it; asyncio.run called inside a write joins
it, one task at a time. A tinyjobs call made while the thread is
opening or closing a transaction, which is where a signal handler lands, is refused with
InterruptedCallError, so the half-open transaction is neither healed nor joined. An interrupted
write that a later one rolled back refuses to commit. All of this relies on each of those C calls
being atomic, which the GIL guarantees and a free-threaded build keeps with its per-object locks on
built-in types; a handler still runs only on the main thread, between bytecodes, on both. Only the main thread is checked for holding a lock, since a handler cannot run
anywhere else; a fork in progress counts on every thread. Async code on a loop in the main thread is
outside this: asyncio.run and uvicorn turn Ctrl-C into a cancellation, which tinyjobs handles at
every await, and a handler that raises in the middle of that code is not covered. The tests raise
an exception at every point of a call, a close, a drain, a transaction and a fork where CPython can
run a handler, and check that nothing is left behind.
The main thread keeps no books because CPython itself is not safe against a handler raising at an
arbitrary instruction. On 3.13 and 3.14, the backward jump of a loop whose body ends inside a try is
missing from the exception table. An exception raised there skips the enclosing except and
finally: 200 of 200 trials on 3.13 and 3.14, none on 3.12. On 3.14, with lock: checks
for signals after __enter__ returns but before the block is protected, so the lock stays held. On
3.12, an interrupted Thread.is_alive() reports a live thread as dead, so a fork reads
whether the control thread has exited from the thread's handle instead, which 3.14 renamed from
_handle to _os_thread_handle.
The first connection is elected across loops and threads: one caller connects, and the others wait
on a thread-safe future without blocking their own loop and share its error if it fails, so a server
that refuses the connection is asked once, not once per waiter. If the caller's loop stops first, a
waiter takes over; a Redis connection is made on the facade's loop and finishes either way. A second
backend made in that race is closed, as is one whose setup() fails. An asyncio.Lock did this
before, and it belongs to one loop: two loops connecting at once deadlocked in every trial, and four
loops racing four threads hung all of them. With the election, 0 of 100 such trials fail on SQLite
and 0 of 50 on Redis. A connection that never reaches the app's loop, because a close retired that
loop or cancelled the caller first, ends its election too, so the next caller connects. A 3 s race
of seven callers hit that 16 to 59 times; the election used to stay open, and when the loop kept
running every later caller waited on it: on Python 3.14 against a disk-synced Postgres, 1 race in
69 left all seven stuck until the app was closed, and 0 in 80 do now.
Calling a synchronous method from inside a running loop raises SynchronousCallInsideEventLoopError
and names the awaitable to use instead, rather than deadlocking or silently blocking the caller's
loop.
Every call connects lazily, except after the app has been closed. close(), aclose() and leaving
async with app or app.serve() mark it closed, and a later enqueue() raises AppClosedError
rather than quietly opening a connection nobody will close; so do drain() and a scheduler tick.
start(), ready(), serve() and starting a worker open it again. Nested async with blocks
share one connection, which closes when the outermost one ends.
Async does not make the queue faster on its own. On SQLite the limit is still fsync. It pays when
there are many concurrent I/O waits in flight, which is the network-backend case, and it is why the
protocol has to be async before those backends exist rather than after.
Task functions are unaffected by any of this. async def runs on the loop, def runs in a pool,
and you write whichever fits the work.
Defining an app¶
from tinyjobs import TinyJobs
app = TinyJobs("sqlite:///jobs.db")
@app.task
async def generate_pdf(document_id: int) -> str:
...
@app.task(queue="slow", max_attempts=5, timeout=300)
def transcode(video_id: int) -> None:
...
@app.task(cron="*/5 * * * *", timezone="Europe/Rome")
async def refresh_cache() -> None:
...
@app.task(concurrency=2, rate="10/m")
async def call_partner_api(order_id: int) -> None:
...
Enqueuing¶
job = generate_pdf.enqueue(123)
job = generate_pdf.with_options(
delay=60, # or run_at=<datetime>
priority=10,
queue="urgent",
key=f"pdf:{123}", # deduplication
).enqueue(123)
Why with_options rather than enqueue(123, delay=60). Options and task arguments cannot share
one keyword namespace without a silent collision: the moment somebody writes a task that takes a
parameter called queue, priority or key, none of which are exotic names, their argument would
be swallowed as an option, or the option would land in their function. The bug would surface as a job
running in the wrong queue, or a TypeError far from its cause. Separating them costs seven
characters and removes the failure mode entirely. with_options returns a lightweight view and never
mutates the task, so it is safe to build per call site. A priority must fit in 32 bits, from -2³¹ to
2³¹ - 1, which every backend stores exactly; anything else raises ValueError at enqueue.
enqueue performs no work: it serializes the arguments, writes one row, returns. job.id and
job.status are available immediately.
enqueue()
↓ validate against the task signature ← fails here, not in the worker
↓ serialize (args, kwargs)
↓ backend.enqueue()
↓ row committed
↓ return Job
Arguments are checked against the task's signature with inspect.signature().bind() at enqueue
time. A typo becomes a TypeError in the caller's traceback, right where it was written, instead of
a failed job discovered in a log an hour later.
Enqueuing without the task's code¶
That validation is only possible because enqueue is a method on a Task object, so reaching it means
the function is registered in this process, by construction. But there is a legitimate pattern where
it is not: a producer that dispatches work to workers whose code it deliberately does not import, such
as a small API service feeding a heavy ML worker it shares nothing with.
Rather than weaken enqueue with a warning for both cases, that pattern gets its own entry point:
app.send("ml.train_model", args=(dataset_id,), queue="gpu", priority=5)
No signature to check, no import attempted, and the caller has said so explicitly by choosing this
method. The two APIs answer the two situations exactly, and neither has to compromise: task.enqueue()
is strict because it can afford to be, app.send() is permissive because that is the whole point of
reaching for it.
Transactional enqueue¶
The dual-write bug is old and common: you commit a row, enqueue a job about it, and the enqueue succeeds while the commit rolls back, or the worker claims the job a millisecond before the commit lands and finds nothing there. A broker-backed queue cannot fix this, because the broker and your database have no transaction in common. A database-backed queue can, and on Postgres tinyjobs does:
with psycopg.connect(DATABASE_URL) as connection, connection.transaction():
connection.execute("INSERT INTO documents (id, name) VALUES (%s, %s)", (123, "contract.pdf"))
generate_pdf.with_options(tx=connection).enqueue(123)
# both committed, or neither
tx= goes on with_options, app.send and app.asend, and takes the connection or a cursor the
caller already has, sync or async, which is the one an ORM or a request handler holds. Borrowing it
rather than handing one out is what makes the job part of a transaction tinyjobs did not start; the
settings the claim relies on live on tinyjobs's own pool, and an insert needs none of them. How the
Postgres backend joins it is in
Joining the caller's transaction. tx= on a backend whose
Capabilities.transactional_enqueue is False raises UnsupportedCapabilityError at enqueue time,
and so does eager mode.
On SQLite it does the same thing, with one condition: the queue has to live in the same file as
your data. SQLite documents that a transaction spanning ATTACHed databases is not atomic in WAL
mode, and WAL is not negotiable here, since it is what lets readers work while a worker writes. So
the tinyjobs tables carry a prefix, tinyjobs_jobs, tinyjobs_executions, tinyjobs_schedules
and tinyjobs_meta, plus the empty tinyjobs_install_lock, as they do on Postgres, and point the
app at the file your application already uses:
app = TinyJobs("sqlite:///app.db")
connection = sqlite3.connect("app.db")
with connection:
connection.execute("INSERT INTO documents (id, name) VALUES (?, ?)", (123, "contract.pdf"))
generate_pdf.with_options(tx=connection).enqueue(123)
connection.close()
tx= takes a sqlite3.Connection or a sqlite3.Cursor. The insert runs on it, on the thread that
holds it, inside whatever transaction it has open, and tinyjobs neither begins nor commits one:
with the module's default transaction control the first insert opens the transaction, as psycopg
does, and the caller's commit() ends it; a connection in autocommit mode commits at once, which is
an ordinary enqueue. aenqueue runs the same insert inline on the event loop's thread, since a
sqlite3 connection is bound to the thread that opened it; the insert is a few hundred
microseconds, and the one wait it can have, for the write lock, is bounded by the connection's own
timeout; the first call of a cold backend costs more, since it also opens a short connection on
that thread to read the schema version, and may run the install inline there. The connection
must be open on the file the app uses, compared by real path from PRAGMA database_list, or the
call raises ConfigurationError naming both files, since an insert
into another file would commit with nothing; an in-memory store refuses tx, since workers could
not share it anyway. The dedup key is enforced on the caller's connection by the same partial unique
index, and the holder it finds is read in the caller's snapshot, so a job sent twice inside one
transaction comes back once.
The first tinyjobs call of a process can well be an enqueue inside a transaction, a Django request
that writes an order and sends its receipt from the same atomic() block. By then the caller's
connection holds the file's write lock, so a setup through tinyjobs's own connection would wait
for it, in the same thread, until busy_timeout ran out. So a join needs no setup: when the app is
not yet connected, enqueue_in takes a backend built cold, with no connection of its own, which
reads the schema version once through a short connection of its own, outside the caller's
transaction and with no wait at all, since the caller may hold the file's exclusive lock, in
rollback-journal mode, once its transaction outgrew the page cache; a busy answer counts as not
current. When the file lacks the schema, or the check could not read, the backend installs it
through the caller's connection, the same DDL and migrations, where SQLite keeps DDL
transactional. Python's sqlite3 opens a transaction on its own only before an insert, update or
delete, never before DDL, so the install runs in a savepoint when the caller's transaction is
already open, and rolls back with it, and inside a BEGIN IMMEDIATE of its own otherwise, which
waits for the write lock up to the connection's own timeout, so that it is one atomic,
serialized unit whatever the connection's transaction mode. Inside the savepoint the first
statement is a write, since an install that read first from a deferred transaction failed in 3
of 4 first requests started together on a new file: DELETE FROM tinyjobs_install_lock WHERE 0,
an empty table the schema carries for that one purpose, takes the write lock and changes nothing,
and when the table is not there yet, creating it is the write. Creating and dropping a scratch
table instead failed with "database table is locked" whenever the caller still had a cursor
open, as a Django iterator() inside atomic() does, and a legacy migration inside such a
transaction still fails that way on its DROP INDEX; rolling the savepoint back then ends the
read in progress too, so that error is reworded to say to read the rows first, or to let an
ordinary call migrate the file. The reworded errors keep SQLite's error code and name, for a
caller that retries on them; the renames run with legacy_alter_table off, so the foreign key in executions
follows the table whatever the connection has set. Because an install inside the caller's
transaction can roll back with it, the backend takes the schema as known only once its own setup,
or a check outside any transaction, found it current; until then each call checks again, and a
"no such table" on the insert after that makes the backend check and install once more before
giving up. A transaction that read the file before a worker installed the tables cannot see
them, and the "no such table" SQLite then answers is reworded to say so; a transaction that read
before its enqueue while another connection wrote gets SQLite's SQLITE_BUSY_SNAPSHOT, reworded
to name BEGIN IMMEDIATE. The app connects as usual on its next ordinary call after the commit, and finds the schema
there; before the commit its own setup would wait for the caller's write lock. Which backends join
cold is a fact about the backend, held by the loader beside loop_bound; Postgres connects
first, since its join needs the store's identity from setup.
What the caller's transaction costs is the write lock: from its first write to its commit, every
worker on the file waits for it, up to busy_timeout, so the block that enqueues should be as
short as the one that writes the order. And a transaction that read before it writes meets WAL's
one rule: SQLite refuses the upgrade at once, with "database is locked", when another connection
wrote since the read began, and a worker writes every few milliseconds. tinyjobs's own pool opens
every write with BEGIN IMMEDIATE for that reason, and the enqueue through a caller's connection
writes before it reads: the insert is its first statement, and a dedup key is looked up only after
the insert hit the unique index, so a plain BEGIN whose first statement is the enqueue never
meets the rule. Before the enqueue wrote first, it read the schema version on the caller's
connection, and 16 of 300 calls inside a deferred transaction failed that way while another
connection committed every millisecond. A caller's transaction that reads before it enqueues
should begin immediately, which under Django is "transaction_mode": "IMMEDIATE" in the database
OPTIONS.
Under Django, "database": "default" in OPTIONS now works on a SQLite alias as on Postgres:
the backend enqueues through that alias's raw connection, inside transaction.atomic(), the
alias's NAME must be the file in url, and an alias on another engine than the one in url is
refused with both named. tinyjobs sets the file to WAL mode the first time its own pool opens it,
or when a first enqueue outside any transaction installs the schema, which is persistent and which
Django's connections read and write through without change; tinyjobs check audits it as any
store, and switches it when it can.
A tinyjobs call that writes (an enqueue, a send, a cancel, a retry, a purge or a drain) made while
the same context holds an open backend.transaction(), tinyjobs's own write scope on the same
file, is still refused at once: it would run on another connection and wait for the write lock that
scope holds. An asyncio task started inside the block is not refused, since it can wait while the
block goes on; a thread handed the context, as asyncio.to_thread does, is. The refusal sees only
the context that opened the transaction. Inside the block, commit(), rollback(),
executescript(), with connection: and setting isolation_level or autocommit are refused
because they would end the transaction early, and a transaction SQLite ends by itself, as
INSERT OR ROLLBACK does, makes the block raise when it exits, since every statement after that
point committed on its own.
Task discovery¶
Workers must import task modules to populate the registry. A job row stores a task name, and a worker that never imported the module cannot resolve it.
tinyjobs worker --app myproject.tasks:app
tinyjobs worker --app myproject:app --import "myproject.jobs.*"
A job whose task is unknown to the worker is left QUEUED and logged, never failed: the usual cause
is a misconfigured --import or a partially rolled-out deploy, and failing the job would destroy work
that a correctly configured worker could still run. A hand-back gives the attempt back: attempts
and the execution history read as if the claim never happened, and only the job's hand_backs
counts it, so a task allowed three attempts still gets three once a worker knows it. Any other
outcome resets hand_backs, so it counts the hand-backs in a row, including those of a claim
whose caller was cancelled. A backend without hand_back
falls back to release, which uses the attempt up. Each hand-back
waits twice as long as the one before, from unknown_task_delay (60 s) up to
unknown_task_max_delay (an hour). A fixed delay wrote an ERROR line every minute, forever, for a
job retention never prunes because it never finishes; doubling brings that to 24 a day once the
delay reaches the hour.
Django tasks¶
tinyjobs.contrib.django.TinyJobsBackend is a django.tasks backend. A Django task has no name of
its own, only an import path, so the job stores that path and the registry takes a resolver: a name
it has not seen is looked up once, wrapped as a tinyjobs task and kept.
The resolver is the trust boundary. A job row is data, and whoever can write one should not be able
to make the worker import arbitrary modules, so it only imports modules that belong to an app in
INSTALLED_APPS, and only accepts a django.tasks.Task whose path is the job's; enqueue refuses
any other task with InvalidTask, so it cannot be sent. Anything else stays unknown and is handed
back like any unregistered task; an import that raises is logged with its traceback. A task bound to
another alias is rebound to this one, since Task.using(backend=...) sends it here without changing
the module attribute. So the alias a task is declared with is a default, not a boundary: whoever can
write a job row can already name any task of the installed apps. Django validated the task when it
was enqueued, so the copies made for a stored job, rebound or carrying the job's queue and priority,
are not validated again: a queue since dropped from QUEUES, or a priority outside Django's range
sent through app.send, still runs and reads back. manage.py tinyjobs also imports each installed
app's tasks module first, so check counts every task found there, whatever alias it is declared
with, before any job arrives.
Django builds one backend instance per thread. They share one TinyJobs per alias and options, so a
WSGI server does not open a connection pool per thread. The async methods go through the sync ones
in a thread: Django under ASGI has no lifespan to connect at startup, and the first connection,
raced from the server's loop and from a sync view's thread at once, turned 33 of 400 requests into
500s. Through one loop, 1,000 out of 1,000 succeed.
The mapping to TaskResult follows what tinyjobs already stores. Executions give worker_ids,
started_at and the errors; the exception path comes from the last line of the stored traceback,
because the execution keeps only the class name. Cancelled reads as FAILED, since Django has no
such status, and so does a task whose worker died. Neither raised anything, so each gets a
JobCancelled or WorkerLost error from tinyjobs.contrib.django.exceptions, and errors[-1]
always exists on a failed result. A Django task gets one attempt unless it asks for more, below.
takes_context needs the running job inside the task. The execution service puts it in a context
variable, and the thread executor copies the context into the thread that runs a def task.
Signals fire as Django's own backend fires them; in eager mode task_enqueued arrives last, because
the task has already run by the time enqueue() returns.
Task options. django.tasks knows a priority and a queue, and nothing of a timeout, retries,
a job process or a limit. Django leaves the door open for exactly this: @task hands every keyword
it does not know to the backend's task_class, from 6.1 on, so the backend's own class,
tinyjobs.contrib.django.Task, is Django's task with the tinyjobs options as fields: timeout,
max_attempts and the retry_* family, isolated, concurrency, rate and result_ttl, with
the names, defaults and meanings of @app.task. They are declared where the task is,
@task(concurrency=2, rate="10/m"), validated by validate_task with the rest, so a malformed
rate is an InvalidTask at import, and carried through using() like any field. Django 6.0's
@task takes no other keyword, so tinyjobs.contrib.django.task is the same decorator with the
keywords passed on, for 6.0 and for whoever prefers the import to say where the options come
from. Two things the
decorator cannot reach get a second way in, "tasks" in the backend's OPTIONS, a mapping from
import path to options that wins over the decorator field by field: a task of a third-party app,
and a project that keeps another backend, such as Django's immediate one, for some environment,
where an unknown keyword on @task would be a TypeError. A path there that names no task of the
installed apps is a ConfigurationError when manage.py tinyjobs starts, since a typo would
otherwise leave a task quietly unlimited. retry_on takes exception classes or their dotted
paths, which is what a settings file wants, where importing project code is awkward; a path is
checked for its shape where it is declared and imported when the task is
resolved, not when the backend is built, which happens inside the import of the first module that
declares a task and would find the project half imported. There is no store_result, which
get_result() needs and stays on, and no cron, since a Django task has no schedule.
The adapter builds the tinyjobs task from those options, so everything downstream is the code
@app.task already runs: the limit reaches the claim through the registry, which is why
manage.py tinyjobs resolves every discovered task before the worker starts, and the retry policy
decides in the lifecycle service. A task outside its app's tasks module is resolved only when its
first job is claimed, by a claim that could not carry its limit; a worker or a drain that finds a
limit on a task its claim did not name hands the job back and lets the next claim, which reads the
registry again, decide it. A task the resolver finds but cannot build, a dotted retry_on that
does not import in this process, is left queued with the unknown task's backoff and an error in
the log, and a drain raises. @app.task now refuses at declaration what it used to accept in
silence, a max_attempts below one, a timeout that is not a positive number, an unknown backoff,
since the same check has to answer for values that arrive from a settings file.
What changes is what Django sees of an attempt's end. task_finished used to fire from the
wrapper around the task, which knows only what the function did. It does not know a timeout, which
the executor decides outside it while an async def task is cancelled and a def task keeps its
thread and may return long after its job was failed; nor a result too large to store; nor a job
process that died; nor, now, whether a failure will be retried. Asking the wrapper to guess, and a
ledger to reconcile its guess with the lifecycle's decision, left a window in every ordering. So
task_finished now fires from the app's own success and failure hooks, in the worker, after the
outcome is recorded and only when no attempt is left, with the result read back from the store:
the status and errors get_result() would give at that moment. A failed attempt that will run
again has not finished, so the task reads READY with the attempt's error in errors.
task_started still fires from the wrapper, for every attempt, with context.attempt counting
them, on the thread that runs a def task.
What the hook sends from the worker's loop, it sends from a thread started for that one
delivery, which closes the database connections its receivers opened before it ends, whatever
their CONN_MAX_AGE, since nothing would use them again, and with the task's error raised around
the send for a failure, because Django's own receiver logs the traceback it finds in flight. Neither a pool nor asend would do. The
loop's default pool is bounded and shared with what tinyjobs itself needs, the pipe to a job
process among it, so slow receivers would starve the worker, and in eager mode a chain of
receivers each enqueueing the next task would block once it was as deep as the pool. And under
asend asgiref runs sync receivers of a loop without
an outer sync context on one thread for the whole process, which made four receivers of 0.4 s
take 1.6 s on a worker of four and deadlocked an eager enqueue called from such a thread, and
Django itself connects a sync receiver, its logger, to every signal. The price of reading the
result back is two point reads for each job that ends, while it still holds its slot. If they
fail, the signal still fires, built from the job as it was claimed and the error in hand. Two ends
have no signal, because no worker is there to see them: a job cancelled while it runs, and a job
the reaper fails after its worker died; get_result() reports both as FAILED. A receiver that
raises is logged and no longer fails the task, since the task had already ended.
One task after another. task.after(result) returns the task with follows set, the way
using returns it with another queue, and the enqueue passes it on as after=. follows is an
ordinary field of the task class because Django copies and pickles a task by its fields, a
result included, and a field kept out of the constructor would break both; it is left out of the
"tasks" settings and tinyjobs' decorator refuses it, since the job a task runs after exists
only once something has been sent. A result read back carries it, a cancelled one says in its
JobCancelled error that the task before it did not succeed, and a missing parent raises
JobNotFoundError from enqueue. A task with follows reports a run_after only while it
waits: once released its run_at is the moment it was released whenever the time asked for had
passed, and the two cannot be told apart.
Isolated Django tasks. A job process finds an isolated task by importing its module and
looking for the tinyjobs task among the module's names, and a Django task module holds none. So a
task may carry a locator, a module-level function with its arguments, which the job process calls
for the task instead of searching; the Django backend's locator takes the alias and the path and
asks that alias's registry, the same resolver and the same trust boundary as the worker's. The job
process also receives the job and sets the context variable the thread executor would have copied,
which takes_context and the signals need; that holds for every isolated task, not only Django's.
The Django wrapper runs whole inside the job process, so task_started fires there, where the
task's own code and its database connections live, and it reads the job's earlier attempts
through a store connection of its own, one per job process. task_finished fires in the worker
like any other task's, which is the only place that knows a process killed at its timeout or dead
of a crash ended the task. async def tasks cannot be isolated,
as anywhere else.
Operations¶
Observability¶
Logging goes through the stdlib logging module, on the tinyjobs logger and its children, with the
job id in the message and, on every record about a job, the same facts as extra fields: job_id,
task, queue and attempt, plus worker_id on the worker's own records, duration_ms on an
outcome, and error, error_message and will_retry on a failure, whose last attempt also
carries the traceback twice: as a traceback field, the child's own traceback alone for an
isolated task, and as exc_info, the rebuilt error with that traceback chained as its direct
cause, which is what Sentry and a plain formatter read; a record about a schedule carries
schedule. The two formatters in tinyjobs.logs prefer the field: JSONFormatter writes one
JSON object per line with time, level, logger, message, the fields, and traceback when
there is one, and TextFormatter prints the traceback under the line. --log-format installs
them on every CLI command, on the tinyjobs logger when the host left it alone and on the root
logger in a standalone run, and, when the flag was given, warns when either already has
handlers. No logging is configured by the library. A library that calls
basicConfig is a library that fights your application.
app.metrics hooks the job events and keeps counters with no dependency: tinyjobs_jobs_total
by task and outcome (success, failure, retry), a tinyjobs_job_duration_seconds histogram of
successful attempts by task, and tinyjobs_jobs_running by task, which an internal end event
decrements even when an attempt is cancelled or its outcome is not recorded, since the public hooks
fire only for recorded outcomes; it keys the running attempts by job and claim, so an end without a
start, or two ends, cannot drive the gauge negative. The collector is created with the app's hooks,
so nothing is missed before the first scrape, and one per app is shared by its lanes, so --loops N
counts once; it takes a lock, since the lanes' loops run on different threads. render() writes the
Prometheus text format with a tinyjobs_info gauge carrying the version and, given a Stats,
adds the store's gauges: tinyjobs_store_jobs by queue and status, tinyjobs_store_queued_jobs
by task, tinyjobs_store_oldest_due_seconds by queue and tinyjobs_store_expired_leases.
tinyjobs worker --metrics-port 9464 serves it at /metrics from a thread on the stdlib
http.server, loaded only when a worker serves metrics, asking the store for its stats on each
scrape, so a 15 second scrape is one stats query per 15 seconds, with five seconds to answer
on the scrape's clock, not the query's, since a driver can take as long again to deliver a
cancel; under Prometheus' default scrape timeout of ten, a store that cannot answer, or not in
time, costs the scrape its gauges, not the counters. A host name binds IPv4 first, a taken port
is refused rather than retried on the other family, and the socket has SO_REUSEADDR so a
restart takes its port back at once, which on macOS also lets it sit beside a server holding the
same port on every address, where Linux refuses. An embedded worker returns
app.metrics.render(await app.astats()) from a route of its own. The counters are per process,
as Prometheus expects; the store's numbers are the same from every worker, so scrape them from one,
or aggregate them with max() across instances rather than sum().
Lifecycle hooks for metrics, tracing or alerting, with no framework assumptions:
@app.on_job_start
def _(job): ...
@app.on_job_success
def _(job, result, duration): ...
@app.on_job_failure
def _(job, exc, will_retry): ...
Hooks run in the worker. Success and failure fire after the outcome is recorded, so a hook that reads
the job back sees it. They may be sync or async. One that raises is logged on tinyjobs.hooks and
ignored: observability must never change what happens to a job.
Health¶
tinyjobs stats gives counts by queue and status, how long the oldest due job has waited, and the
number of jobs currently past their lease, which is the metric to alert on, because it is the one
that means workers are dying.
tinyjobs check is the same data turned into a verdict, exiting 1 on any problem so it can back a
probe or a cron alert. It fails when the app registers no tasks, when queued jobs name a task the app
does not register (the usual sign of a missing --import), when a job is past its lease, when a due
job has waited longer than --max-wait, and when a cron schedule is more than a minute overdue, which
means nothing is firing it. It also fails when the store is configured against the
guarantees, through the backend's audit(). Reaching the store and
its schema version is checked by connecting at all. Clock skew is not: on one SQLite file there is
no second clock to compare against.
Schema migrations¶
A schema_version row in tinyjobs_meta, forward-only migrations applied at setup(). Workers refuse to
start against a schema newer than they understand, so a partial rollback cannot corrupt a queue.
Version 3 adds jobs.hand_backs and jobs.claims, and keys executions by claim. The migration
numbers each job's existing executions in the order they started, and sets claims to the larger of
attempts and that count, so a job retried under 0.6 keeps every row. On Redis the store moves to
version 2: a job without hand_backs reads as zero, and a job without claims gets the same
renumbering the first time a script touches it or it is read, so get, list and executions
already show the adopted order and claims. Version 4 moves the SQLite tables under the prefix,
as Tables and the prefix describes: a file from before 0.14 is renamed in
place, its indexes recreated under their new names, and a bare meta stub left behind for older
processes to refuse.
Version 6 on SQLite, 3 on Postgres and 4 on Redis add runs_after and waiting_on to the jobs, the partial index
of waiting jobs by parent, and waiting to the statuses a dedup key is held in and, on Postgres,
to the status check. Upgrading to 0.17 is a stop-the-world step: an older worker would end a
parent and leave its children waiting.
It is meant to be the last one. Beside schema_version every store now keeps compatible_since,
the oldest schema version whose code works on the store as it is, written by whichever release
installs or upgrades it. A process that finds a store newer than itself starts anyway when
compatible_since is at or below its own version, runs no DDL and leaves both rows alone; it
refuses, as before, when the row is higher or missing. The promise that goes with it: from 0.17
a release that changes the store keeps the previous minor working beside it, so a rolling deploy
from one minor to the next needs no stop, and a release that cannot keep the promise says so in
its upgrade note and raises compatible_since, which makes older processes refuse to start;
the ones already running are what the note's stop is for. Two minors apart is not covered: the store allows it only when every release between
kept the row where it was. Keeping the promise includes keeping the tables, columns and indexes
the previous minor checks at setup, since a process working beside a newer store creates nothing, and
on Postgres verifies what it knows.
Upgrading to 0.14 is a stop-the-world step as well, for the application processes that enqueue too, since a 0.13 process left running works on tables that are no longer there, as SQLite spells out; a 0.13 process started afterwards refuses the stub before running any DDL, so it leaves nothing behind.
Upgrading from 0.6 is a stop-the-world step. A 0.6 worker writes executions by attempt, which no longer matches the layout, so stop every 0.6 process before the first 0.7 one starts. Once the store is upgraded, a 0.6 process refuses to start against it. 0.6 runs its schema statements before it checks the version, so 0.7 keeps an empty index under the 0.6 unique index's name: that statement becomes a no-op, and a refused 0.6 start cannot stop a running 0.7 worker from retrying jobs.
Security notes¶
- Pickle is off unless explicitly enabled; see Serialization.
- Task names come from the registry only. A job row naming an unregistered task is never imported dynamically. That would be an arbitrary-import primitive for anyone able to write a row.
- The SQLite file holds serialized arguments. If tasks carry sensitive data, file permissions are the boundary; the library does not encrypt.
Testing¶
Testing background jobs is normally miserable, so it is a design constraint rather than an afterthought.
# 1. Eager mode: every enqueue runs its own job before it returns
app = TinyJobs("memory://", eager=True)
# 2. Manual draining: real queue semantics, deterministic ordering
app = TinyJobs("memory://")
generate_pdf.enqueue(1)
app.drain() # run everything due, highest priority first, then stop
# 3. Fake clock: test delays and backoff without waiting
clock = FrozenClock(time.time())
app = TinyJobs("memory://", clock=clock)
flaky.enqueue()
app.drain() # first attempt fails, the retry is due in 10s
clock.advance(10)
app.drain()
memory:// is not a second implementation. It is the SQLite backend on a private in-memory database,
so a test exercises the same claim, the same dedup index and the same retry bookkeeping as
production, with nothing to keep in sync. Eager mode claims the job an enqueue sent by its id and
runs it, so a delayed job or a retry with backoff stays queued until the clock reaches it and a drain
runs it, exactly as it would under a worker. It runs no other job: Celery, RQ, Huey and Django's own
test backends run only the job sent, and running every due job made the sent one wait behind older
ones and, with nested sends and threads, led to waits on jobs other calls held. A call whose job
another drain or worker claimed first waits until it is no longer running; that wait has no
cycle, since the job it waits on was inserted by the call itself and is run by a drain or worker
that runs one job at a time without waiting on the caller. A dedup hit comes back as it is, since
the job it found may be the caller's own. An eager send takes 0.6 to 0.8 ms on SQLite and in memory,
0.9 on
Postgres and 1.3 on Redis at the median, where running every due job took 1.5 to 2.8.
drain() runs jobs one at a time, fires hooks, and raises UnknownTaskError on a job no task in the
app can run, leaving it queued. It does not fire cron schedules; enqueue the task to test it.
On writing C extensions¶
The original plan said "C extension: only after benchmarks". That instinct is right; the conclusion it leads to is stronger, and writing it down now saves months of work later.
There is probably nothing here worth writing in C. A job queue's cost is not CPU:
- The dominant cost of
enqueueisfsync, the kernel and the disk, not the interpreter. - The dominant cost of
claimis the SQLite B-tree lookup, already C. - JSON encoding uses CPython's C accelerator, already C.
- Cron parsing computes a next firing time in microseconds, once per schedule per tick.
- Actual job execution is user code, which is the entire point.
The remaining Python is object construction and a loop. Making it twice as fast would move an end-to-end number by a fraction of a percent, because the time is spent waiting for a disk.
There is also a direct conflict with the primary goal. A C extension breaks "pip install, no dependencies." It means building wheels for every platform × Python version × architecture, a CI matrix to maintain, and a source build, with a working compiler, for anyone on a combination you did not publish. "Pure Python, zero dependencies, works everywhere" is the strongest single advantage over Celery. Spending it on a speedup that profiling has not yet asked for is a bad trade.
Policy, if the question returns: C is permitted only for a hot path a profile shows is worth more
than 20% end-to-end, must ship with a pure-Python fallback selected at import time, and must remain
optional, so that pip install tinyjobs never requires a compiler. The first version is pure Python, and it
should be benchmarked honestly before anyone opens a .c file.
Roadmap¶
v0.1, the core loop
Task registry, task.enqueue() with signature validation, app.send() by name, SQLite backend with
the atomic partial-index claim, leases and the reaper, a worker with sync and async executors and a
graceful shutdown, retries with backoff, queues, priorities, delayed jobs and deduplication keys,
the JSON codec, tinyjobs worker, tinyjobs jobs, and the backend plugin seam.
v0.1.1 and v0.2, results
Job.refresh(), Job.result() and Job.wait().
v0.3, retention
Finished jobs pruned automatically, and tinyjobs purge.
v0.4, time and comfort
Cron schedules with the embedded leader-free scheduler, misfire and overlap policies, timezone and
DST handling; lifecycle hooks; the memory:// backend, eager and drain test modes;
tinyjobs stats / check.
v0.5, Redis A Redis backend on Lua scripts, and the contract suite every backend runs.
v0.6, FastAPI FastAPI and Starlette lifespans with an embedded worker.
v0.7, Django
The django.tasks backend and manage.py tinyjobs. On Redis, the server's clock followed instead
of each machine's.
v0.8, Postgres
A Postgres backend with LISTEN/NOTIFY wake-ups, enqueue inside the caller's transaction, also
through Django's transaction.atomic().
v0.9, free threading
A worker that keeps the GIL off on free-threaded builds, and loops= to run several event loops
in one process.
v0.10, isolation
isolated=True, a job process per task, spawned on demand and reused, killed at the timeout and
replaced when it crashes, with processes and jobs_per_process.
v0.11, operations Schedules paused, resumed and fired by hand, a result size cap and a per-task result TTL, structured log fields and a JSON log format, a Prometheus exporter on the hooks, and a guide to running it.
v0.12, the dashboard A dependency-free dashboard on the stdlib server and the same service calls as the CLI: queues and health, jobs and their attempts, retry and cancel, schedules paused, resumed and run from the page, mountable in a WSGI or ASGI app.
v0.13, guarantees
The guarantees written down in one place, tinyjobs check auditing the store's configuration
against them through the backend's audit(), and tinyjobs verify to prove them on any store:
workers killed under steady load, nothing lost, every side effect applied once.
v0.14, the queue in your database
Transactional enqueue on SQLite: prefixed tables, so the queue lives in the application's own file,
tx= taking a sqlite3 connection, a cold join that installs the schema through the caller's
connection, and Django's "database" option on SQLite aliases.
v0.15, limits
concurrency= and rate= on a task, kept in the store and enforced at the claim across every
worker: a job the limit stops is held out of the claim's way and admitted, in the order the queue
promised, when the task has room again.
v0.16, Django task options
The tinyjobs options on a django.tasks task: timeout, retries, isolated, concurrency and
rate, through the backend's task class or the "tasks" option, with isolated tasks located by
the backend in the job process.
v0.17, one job after another
after=job: a job waits, in a status of its own, until another has succeeded, and is cancelled
with everything behind it when that one fails. The store settles it in the step that ends the
parent. With it, compatible_since, so that later upgrades between neighbouring minors need no
stop.
v0.18, settling
No feature. A changelog and a compatibility page that says what is public and
what a release may change; the exceptions and return types a caller meets exported from the
package; NewerStoreError for a store too new; the in-memory template backend made to pass the
contract; and a release check that upgrades a store of the release before and shares it between
the two versions.
v1.0, commitment A backend protocol that changes only compatibly, a documented plugin API, full documentation. After this the protocol is a compatibility promise. Compatibility lists what has to be true of the protocol and of the public surface first.
Beyond
Joining a SQLAlchemy session's transaction, designed against a real application rather than against a
guess. A --workers N that forks and supervises N workers, if running one per core under a
supervisor turns out to be the friction people report. Groups and chords only if usage demands
them. They are where task queues usually go to become frameworks.