Implement a queue to do asynchronous processing of batches

[hltcoe-bot]

Right now when a requester uploads a CSV file, the creation of tasks occurs happens while the user waits for the response of the form submission. Right now we have a limit on POST size of 100 MB and that can be a lot of CSV rows. As we scale up, we'll seem more batches with thousands of tasks.

The best way to handle this is to add the CSV parsing and task creation to a queue. This could be a poor man's queue using the database and a cron job or using something like ZeroMQ. For previous projects (non-Django) I've used Celery to abstract away the differences between the different queue systems out there.

Poster: Cash Costello id: 206

[charman]

FWIW, Celery appears to have strong Django support:

Previous versions of Celery required a separate library to work with Django, but since 3.1 this is no longer the case. Django is supported out of the box

https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

The current Django documentation mentions Celery:

https://docs.djangoproject.com/en/3.2/topics/db/transactions/#performing-actions-after-commit