Retries, persistence, and visibility for FastAPI's BackgroundTasks
Production-grade background tasks. No workers, no brokers, drop-in.
The problem
BackgroundTasks is fine, until production finds the gaps
You shipped background_tasks.add_task(send_email, address=email) and it worked in dev. Then you deployed it, a task failed, and you had no idea. No retry happened. No log survived. The user never got their email. You found out three days later when they complained.
Without fastapi-taskflow
- Tasks fail silently, no retry or backoff
- Tasks compete with request handlers for the same event loop
- Pending tasks lost on every restart
- No task IDs, no status, no history
- No priority, every task waits in the same queue
- Retried requests run the task again with no deduplication
- No dashboard, no logs, no stack traces on failure
- No persistence, everything lives in memory
With fastapi-taskflow
- Automatic retries with configurable delay and backoff
- Concurrency caps keep tasks from starving request handlers
- Pending tasks requeued automatically on the next startup
- UUID per task, full lifecycle tracking and status history
- Priority queues so urgent tasks never wait behind batch jobs
- Idempotency keys prevent duplicate execution on retried requests
- Live dashboard over SSE with per-task logs and stack traces
- SQLite in dev, Redis, PostgreSQL, or MySQL in prod
Quick look
Looks like FastAPI. Works like a task system.
from fastapi import BackgroundTasks, FastAPI
from fastapi_taskflow import TaskAdmin, TaskManager, task_log
task_manager = TaskManager(snapshot_db="tasks.db")
app = FastAPI()
TaskAdmin(app, task_manager, auto_install=True)
@task_manager.task(retries=3, delay=1.0, backoff=2.0)
def send_email(address: str) -> None:
task_log(f"Sending to {address}")
... # your logic here
@app.post("/signup")
def signup(email: str, background_tasks: BackgroundTasks):
task_id = background_tasks.add_task(send_email, address=email)
return {"task_id": task_id}
The route signature does not change. auto_install=True handles the rest. Read the Quick Start →
Dashboard
Live visibility out of the box
Fit
Who this is for
Good fit
- Teams already using FastAPI's native
BackgroundTasks - Apps that need retries, status tracking, and a dashboard without adding Celery
- Services where background work runs inside the same process as the web server
- Multi-instance deployments on SQLite (same host) or Redis, PostgreSQL, MySQL (any host)
- Teams who want structured task logs, encryption, and observability hooks without new infrastructure
Not a good fit
- Tasks that must run on dedicated worker machines completely separate from the web server
- Workflows that need message broker routing, fan-out, or cross-language workers
Features
What you get
Reliability
Tasks that survive failures and restarts
Automatic retries with configurable delay and exponential backoff. Unfinished tasks are saved on shutdown and re-dispatched on the next startup. Idempotency keys prevent duplicate execution across retried requests and webhooks.
Visibility
See what happened without digging
Live admin panel at /tasks/dashboard. Every task has a UUID, a status, per-task logs, and a full stack trace on failure. All in one place, nothing to configure.
Observability
Structured logs from inside your tasks
Call task_log(message, level=, **extra) from anywhere in a running task. Extras flow to observers as structured fields. FileLogger, StdoutLogger, and InMemoryLogger included.
Persistence
SQLite in dev, PostgreSQL in prod
SQLite works with zero setup. Swap to Redis, PostgreSQL, or MySQL with one line. All backends support task history, requeue, idempotency, and scheduled task locking.
Multi-instance
Scale out without coordination overhead
Requeue claiming is atomic, so only one instance picks up each pending task. Task history is shared across all nodes. Idempotency keys work cross-instance.
Control
Run tasks exactly the way you need
Priority queues, eager dispatch, async concurrency semaphore, dedicated sync thread pool, and a process executor for CPU-bound work. Set defaults on the decorator and override per call.
Scheduling
Interval and cron, distributed-safe
Register periodic tasks with every= or a cron expression. A distributed lock ensures only one instance fires each entry per interval.
Security
Sensitive args, never in plain text
Fernet encryption for task args and kwargs at enqueue time. Never stored in the task store, the database, or any log file. One key, one config option.
Adoption
Keep your existing route signatures
One line at startup hooks into FastAPI's injection. Your background_tasks: BackgroundTasks annotations stay exactly as they are. You get task IDs on every existing route with no other changes.
Positioning
Not a Celery replacement
fastapi-taskflow does not compete with Celery, ARQ, Taskiq, or Dramatiq. Those tools are built for distributed workers, message brokers, and high-throughput task routing across separate machines.
This library is for teams using FastAPI's native BackgroundTasks who want retries, visibility, and resilience without adding worker infrastructure. It supports multi-instance deployments with a shared SQLite file (same host) or Redis, PostgreSQL, MySQL (any host), including atomic requeue claiming, idempotency keys, and shared task history across instances.
CPU-bound tasks can run in a ProcessPoolExecutor via executor='process', which bypasses the GIL and runs workers in separate OS processes. This covers most in-process CPU workloads. If your tasks need to run on separate machines entirely, use a proper task queue.
Contributing
Get involved
Bug reports, feature requests, and pull requests are welcome. For questions or direct feedback, see the contributing page.