Thursday 2 p.m.–6 p.m.

Distributed task processing using Celery

Narahari Allamraju

Audience level:
Lecture Room E2


Celery is a distributed task queue that can be used to off load intensive tasks to a set of workers that complete the task in the background. This session will explain how we can use Celery in scenarios that are more complex than a simple add example and how you can use the various features of Celery to make your tasks execute efficiently.


This talk will go over -

  • Basics of Celery and Celery tasks
  • What are the advanced features that are available in Celery
  • A set of example scenarios where you might want to use Celery
  • A step by step analysis on how you should design your task solution so that it can be trivially distributed across Celery workers
  • An analysis on how you can make sure that all workers get access to the data they need to execute the task without running into contentions
  • Applying features like subtasks, routing,grouping and chaining
  • Other aspects to take note of - monitoring Celery workers

The examples chosen will be scenarios that closely mirror some of the business scenarios out there and will give a good real life application example.