Background jobs and queues

Celery, Redis and the scheduler.

Long-running work runs in Celery workers (a standard worker and a browsing worker) with Redis as the broker. A scheduler service triggers periodic tasks and Flower provides a monitoring dashboard.

Why it is split out

Scraping, analysis and other slow work must not block API responses. Putting them on a queue keeps the API responsive and lets workers retry independently.

Local development

All of these services are started by the compose file. See Local development setup.