Usage¶
Celery distributed task queue module for Invenio.
Invenio-Celery is a small discovery layer that takes care of discovering and loading tasks from other Invenio modules, as well as providing configuration defaults for Celery usage in Invenio. Invenio-Celery relies on Flask-CeleryExt for integrating Flask and Celery with application factories.
Defining tasks¶
Invenio modules that wish to define Celery tasks should use the
@shared_task
decorator (usually in tasks.py
):
# mymodule/tasks.py
from celery import shared_task
@shared_task
def sum(x, y):
return x + y
Additionally the Invenio module should add the task module into the
invenio_celery.tasks
entry point:
# setup.py
setup(
# ...
entry_points=[
'invenio_celery.tasks' : [
'mymodule = mymodule.tasks'
]
]
)
Using tasks¶
Invenio modules that need to call tasks do not need to do anything special as long as the Invenio-Celery extension has been initialized. Hence calling tasks is as simple as:
from mymoudle.tasks import sum
result = sum.delay(2, 2)
Periodic tasks¶
Periodic tasks can be configured via CELERYBEAT_SCHEDULE
configuration
variable:
# config.py
CELERYBEAT_SCHEDULE = {
'indexer': {
'task': 'invenio_indexer.tasks.process_bulk_queue',
'schedule': timedelta(minutes=5),
},
}
For further information about see Periodic Tasks chapter of the Celery documentation.
Celery workers¶
Invenio-Celery hooks into the Celery application loading process so that when
a worker starts, all the tasks modules defined in invenio_celery.tasks
will
be imported and cause the tasks to be registered in the worker. Note that this
only happens on the Celery worker side which needs to know upfront all the
possible tasks.
For further details on how to setup Celery and define an Celery application factory please see Flask-CeleryExt