Thursday 2 p.m.–6 p.m.
Distributed task processing using Celery
Narahari Allamraju
- Audience level:
- All
- Room:
- Lecture Room E2
Description
Celery is a distributed task queue that can be used to off load intensive tasks to a set of workers that complete the task in the background. This session will explain how we can use Celery in scenarios that are more complex than a simple add example and how you can use the various features of Celery to make your tasks execute efficiently.
Abstract
This talk will go over -
- Basics of Celery and Celery tasks
- What are the advanced features that are available in Celery
- A set of example scenarios where you might want to use Celery
- A step by step analysis on how you should design your task solution so that it can be trivially distributed across Celery workers
- An analysis on how you can make sure that all workers get access to the data they need to execute the task without running into contentions
- Applying features like subtasks, routing,grouping and chaining
- Other aspects to take note of - monitoring Celery workers
The examples chosen will be scenarios that closely mirror some of the business scenarios out there and will give a good real life application example.
