Coder Social home page Coder Social logo

django-celery-transactions's Introduction

django-celery-transactions

Travis Version Downloads Coverage Status

django-celery-transactions holds on to Celery tasks until the current database transaction is committed, avoiding potential race conditions as described in Celery's user guide. Send tasks from signal handlers without fear!

Django and Celery coverage

  • Django 1.6, 1.7 and 1.8 are supported and tested. 1.9 will likely require refactoring.
  • Celery 3.0.x and 3.1.x are supported, but the tests are running with 3.1.18

Features

  • If the transaction is rolled back, the tasks are discarded. Django's transaction middleware does this if an exception is raised.
  • If transactions aren't being managed, tasks are sent as normal. This means that sending tasks from within Django's shell will work as expected, as will the various transaction decorators commit_manually, commit_on_success, etc.

Installation & Use

  1. Install django-celery-transactions from PyPI:
$ pip install django-celery-transactions
  1. Use the patched decorator to create your tasks:
from djcelery_transactions import task
from models import Model


@task
def print_model(model_pk):
    print Model.objects.get(pk=model_pk)
  1. Then use them as normal:
from django.db import transaction
from models import Model
from tasks import print_model


# This task will be sent after the transaction is committed. This works
# from anywhere in the managed transaction block (e.g. signal handlers).
def view(request):
    model = Model.objects.create(...)
    print_model.delay(model.pk)


# This task will not be sent (it is discarded) as the transaction
# middleware rolls the transaction back when it catches the exception.
def failing_view(request, model_pk):
    print_model.delay(model_pk)
    raise Exception()


# This task will be sent immediately as transactions aren't being
# managed and it is assumed that the author knows what they're doing.
@transaction.commit_manually
def manual_view(request, model_pk):
    print_model.delay(model_pk)
    transaction.commit()

Caveats

Due to the task being sent after the current transaction has been committed, the PostTransactionTask provided in this package does not return an celery.result.AsyncResult as the original celery Task does.

Thus, print_model.delay(model_pk) simply returns None. In order to track the task later on, the task_id can be predefined in the apply_async method:

    from celery.utils import uuid

    u = uuid()
    print_model.apply_async((model_pk), {}, task_id=u)

Compatibility with CELERY_ALWAYS_EAGER

There are 2 main reasons for CELERY_ALWAYS_EAGER:

  1. Running task synchronously and returning EagerResult, Celery's user guide

  2. Being able to run code (often tests) without a celery broker.

For this second reason, the intended behavior will often conflict with transactions handling, which is why you should then also use CELERY_EAGER_TRANSACTION

    CELERY_ALWAYS_EAGER = True
    CELERY_EAGER_TRANSACTION = True

Run test suite

$ python setup.py test

django-celery-transactions's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

django-celery-transactions's Issues

0.2 fails with Django 1.6 and MySQL backend

The 0.2 version on PyPi takes issue with Django's 1.6 DatabaseWrapper, see below.

Django==1.6
django-celery-transactions==0.2.0

Exception Type:    AttributeError
Exception Value:    
'DatabaseWrapper' object has no attribute 'closed_in_transaction'
Exception Location: /tmp/myapp/env/local/lib/python2.7/site-packages/djcelery_transactions/transaction_signals.py in __patched__exit__, line 214
Python Executable:  /tmp/myapp/env/bin/python
Python Version: 2.7.6
Python Path:    
['/tmp/myapp/market',
 '/tmp/myapp/env/lib/python2.7',
 '/tmp/myapp/env/lib/python2.7/plat-x86_64-linux-gnu',
 '/tmp/myapp/env/lib/python2.7/lib-tk',
 '/tmp/myapp/env/lib/python2.7/lib-old',
 '/tmp/myapp/env/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/tmp/myapp/env/local/lib/python2.7/site-packages',
 '/tmp/myapp/env/local/lib/python2.7/site-packages/newrelic-2.34.0.29',
 '/tmp/myapp/env/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info',
 '/tmp/myapp/env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info',
 '/tmp/myapp/market']Exception Type:    AttributeError
Exception Value:    
'DatabaseWrapper' object has no attribute 'closed_in_transaction'
Exception Location: /tmp/myapp/env/local/lib/python2.7/site-packages/djcelery_transactions/transaction_signals.py in __patched__exit__, line 214
Python Executable:  /tmp/myapp/env/bin/python
Python Version: 2.7.6
Python Path:    
['/tmp/myapp/market',
 '/tmp/myapp/env/lib/python2.7',
 '/tmp/myapp/env/lib/python2.7/plat-x86_64-linux-gnu',
 '/tmp/myapp/env/lib/python2.7/lib-tk',
 '/tmp/myapp/env/lib/python2.7/lib-old',
 '/tmp/myapp/env/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/tmp/myapp/env/local/lib/python2.7/site-packages',
 '/tmp/myapp/env/local/lib/python2.7/site-packages/newrelic-2.34.0.29',
 '/tmp/myapp/env/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info',
 '/tmp/myapp/env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info',
 '/tmp/myapp/market']

Django 1.9 compatibility

I work on a project that uses django-celery-transactions and Django 1.8, but would like to upgrade to Django 1.9. The README warns the 1.9 "will likely require refactoring", so I investigated it a bit and the result is better than expected.

It looks like django-celery-transactions is fine with Django 1.9, despite 1.9's addition of an on_commit() hook, and the monkey-patching of Atomic.__exit__ will not disable any existing features. Django's on_commit() is actually a method on database connections and doesn't require any code in transaction.Atomic's __exit__ method. That __exit__ method seems to be identical between 1.8 and 1.9, except for a 2-line change; Django commit 00a1d4d042a7afd139316982c9b57e87d26a894f removes the 'elif connection.features.autocommits_when_autocommit_is_off' block.

Unfortunately it looks like we can't just drop the monkey-patching and use 1.9's on_commit feature because it's not sufficient to implement what djcelery-transactions needs. 1.9 only has an on-commit hook, and the indexing needs a rollback hook because it needs to either send the tasks (in the case of a successful commit) or discard them (in the case of a rollback). Maybe we could add a wrapper for the __enter__ method that cleared the task queue on initial entry to an atomic block?

I can file a pull request to update the README if you like.

Django support

EDIT: By looking at the code it seems Django 1.6 to 1.8 is supported. If that's correct feel free to close the issue - unless you want to update the docs ;)

Hi,

it seems that this fork seems to be maintained in contrary to @bradleyayers original repository. I wondered which django versions are supported by this fork, since it is not clearly mentioned in the docs (started a discussion here bradleyayers#17).

Regards

Jakob

Catching exceptions when Celery broker is down

Hello!

First: thank's for this package, it is really useful in our project!
I have one problem: In one of my views a signal is triggered which then calls a celery task like this:

def post_save_donation(sender, instance, created, raw, using, **kwargs):
    create_something.delay(pk)

post_save.connect(post_save_donation, sender=Donation)

The task is defined like this:

from djcelery_transactions import task

@task
def create_something(donation_pk):
    # do something

Everything works great, but:

If RabbitMQ is down when I am calling the task no error is thrown during the create_something.delay(pk) call. But the view throws this error:

[Errno 111] Connection refused

The question now is: Why is this error thrown in the view but not in the signal handler where the create_something.delay(pk) is called?

Any help is greatly appreciated

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.