Integration layer between Celery and Invenio.
Home Page: https://invenio-celery.readthedocs.io
License: MIT License
Celery distributed task queue module for Invenio.
Invenio-Celery is a small discovery layer that takes care of discovering and loading tasks from other Invenio modules, as well as providing configuration defaults for Celery usage in Invenio. Invenio-Celery relies on Flask-CeleryExt for integrating Flask and Celery with application factories.
Further documentation is available on https://invenio-celery.readthedocs.io/
Please update (or unpin) msgpack-python to msgpack as old msgpack-python is not working properly with celery newer than 4.2.1 (where celery is serializing also time when task finished, and msgpack fails on this)
Very important notes for existing users
PyPI package name
TL;DR: When upgrading from msgpack-0.4 or earlier, don't do pip install -U msgpack-python. Do pip uninstall msgpack-python; pip install msgpack instead.
Package name on PyPI was changed to msgpack from 0.5. I upload transitional package (msgpack-python 0.5 which depending on msgpack) for smooth transition from msgpack-python to msgpack.
Sadly, this doesn't work for upgrade install. After pip install -U msgpack-python, msgpack is removed and import msgpack fail.
In invenio-celery:
install_requires=[
# Avoid version 4.3 it breaks the emails with datetime
# https://github.com/celery/celery/pull/5606
'celery>=4.2.1,<4.3.0',
''Flask-CeleryExt>=0.3.0',
]```
In Flask-CeleryExt (an invenio-managed project):
```python
install_requires=[
'celery>=3.1;python_version<"3.7"',
'celery>=4.3;python_version=="3.7"',
]This means that dependency resolution fails when installing invenio-celery in Python 3.7. I know that celery 4.4 is still in RC, but can we relax one of these so that it can be installed?
Because of some required renaming in celery (i.e. async is now a keyword in python3.7) running Invenio with Celery is not possible in Python 3.7. This is planned to be fixed in Celery 5.x...
Relevant celery issues: celery/celery#4500, celery/celery#4849
running compile_catalog
error: [Errno 2] No such file or directory: 'invenio_celery/translations/'
ValueError: Cannot find file (or expand pattern): 'dist/*'
running upload_docs
It would be nice to have an example of how to use InvenioCelery (and how to initialize without entry-points).
/examples/app.py)__init__.py)Entrypoint loading fails when package is defined in entrypoints.
Add support for loading tasks via entry_points and modules.
Invenio modules should be using the @shared_task decorator to declare tasks.
The celery application needs a way to load these tasks. This should be possible via providing entry points and/or list of modules.
# setup.py (declaring celery tasks in a Invenio module).
setup(
entry_points={
'invenio_celery.tasks': [
'invenio_fungenerator = invenio_fungenerator',
]
}
)The celery application should then be able to load the tasks similar to this:
from flask_celeryext import create_celery_app
from invenio_celery import load_tasks
from .factory import create_app
celery = create_celery_app(create_app)
celery.autodiscover_tasks(load_tasks(entry_point='invenio_celery.tasks'))The following cookiecutter change:
inveniosoftware/cookiecutter-invenio-module#98
should be propagated to this Invenio module.
Namely, in run-tests.sh, the sphinx for doctests is invoked after pytest run:
$ tail -3 ./\{\{\ cookiecutter.project_shortname\ \}\}/run-tests.sh
sphinx-build -qnNW docs docs/_build/html && python setup.py test && sphinx-build -qnNW -b doctest docs docs/_build/doctestThis sometimes led to problems on Travis CI with the second sphinx-build run due
to "disappearing" dependencies after the example application was tested.
A solution that worked for invenio-marc21 (see
inveniosoftware/invenio-marc21#49 (comment))
and that was integrated in cookiecutter (see
inveniosoftware/cookiecutter-invenio-module#98) was to
run doctest execution in pytest, removing the second sphinx-build invocation.
This both solved Travis CI build failures and simplified test suite execution.
Note that this change may necessitate to amend the code tests etc so that things
would be executed with the Flask application context (see
inveniosoftware/invenio-marc21@09e98fc).
It appears that there is currently no way to add a key to the CELERYBEAT_SCHEDULE variable, instead of completely overwriting it (with no knowledge of what we're overwriting).
Such functionality would enable modules to declare what they want to add to the beat, instead of the administrator of the overlay being responsible of creating this schedule on their own.
Example use-case:
The future checker requires being able to run tasks periodically, and adding these periodic tasks to celery beat is the closest-to-native-invenio solution we could come up with.
Related to #53
Package version (if known):
For notifications, I am trying to serialize the draft of a record with the .dumps() method of the record class, such that it can be passed around for a celery task.
When starting the task, I'll get an kombu.exceptions.EncodeError: can not serialize 'UUID' object error. Checking the serialized draft, all fields are fine, except for the versions. The UUID fields of the versions are not dumped as strings but as UUID objects 'next_draft_id': UUID('c0d626f3-84ae-4c19-8205-4dfe546f1323').
in records-resources, UUID fields are explicitly converted to str: https://github.com/inveniosoftware/invenio-records-resources/blob/master/invenio_records_resources/records/api.py#L157-L169
in drafts-resources, this conversion is missing: https://github.com/inveniosoftware/invenio-drafts-resources/blob/master/invenio_drafts_resources/records/systemfields/versions.py#L137-L146
In the json serializer, UUID fields are handled: https://github.com/pallets/flask/blob/41d4f62909bb426c84e9d057151f7d734695320a/src/flask/json/provider.py#L109-L122
To be added
To serialize a UUID field, when it is passed to a task, no matter the serializer selected.
Because the extension is calling celery.autodiscover_tasks with an empty string ('') as the related_name argument, this line eventually fails in celery (responsible commit: https://github.com/WoaDmulL/celery/commit/c089f86e338c852043bc2ee25be2b82fd53a45a6).
Since it's not very convenient to change all of the invenio package entrypoints to support a invenio_package + tasks naming convention, there's two solutions:
celery to <4.2From a quick read of https://steve.dignam.xyz/2023/05/20/many-problems-with-celery/, there seem to be some configuration options/tactics that might prevent some of the common issues we're facing in both Invenio tests and production setups of Celery.
We can evaluate each of them and see if they're applicable or come with their own caveats.
(Feel free to add more links/resources to this issue so that we can make more educated decisions).
param, returns, raises, versionadded)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.