stored / django-dde

Django Distributed Exporter

Django Distributed Exporter (DDE)

Asynchronous and Distributed Data Exporter from a Django QuerySet

DDE main goal is to get data from a QuerySet and export to an output format (currently CSV only) using Celery for task distribution.

Installation

pip install django-dde

Requirements

DDE uses Celery to manage its tasks. Currently our broker defaults to Redis so be sure to install the dependencies need, for more information click here.

File management is dealt using Django's Default Storage. For more information on how to use it please click here.

We developed DDE using Django 1.11 and Python 3.5. Other versions are pending tests. Let us know if works in your virtual environment.

Usage

First of all:

pip install django-dde

Excelent! Now you must ensure that in your settings.py you have these configurations:

EXPORTER_FROM_EMAIL = '[email protected]'

Now you have to configure a periodic task using Celery, please refere to this link.

Now in the code:

from exporter.models import Exporter

exporter = Exporter.objects.create_exporter(
    queryset=Model.objects.queryset, # Change Model to your Model
    email=[e-mail], # E-mail that will receive the link to the exporter,
    limit_per_task=100, # How many entries each task will proccess
)

And that's it! Simple, right?

Development setup

Installing our dependencies in your virtual environment is pretty straight foward:

pip install -r requirements/development.txt

But you can use our Docker to test DDE. This method already install all dependencies on a container, creates a Redis container and you ready to go. We already configured some shortcuts to help:

make build # Builds docker image, be sure to run this first
make build-no-cache # Build/Rebuilds (cleaning cache) docker image

make tests # Run all tests
make bash # Enters Docker's Bash

make help # Shows the available commands

Release History

• 0.1
  • Initial version

Meta

Stored E-commerce

Distributed under the MIT license. See LICENSE for more information.

https://github.com/stored

Contributing

1. Fork it (https://github.com/yourname/yourproject/fork)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request

Remember: Every feature created or fixed must have a test written and/or fixed.

django-dde's Issues

Definir politicas de retentativa

Create Celery workers on docker's up for testing

Remove task call from create_exporter on manager and move to model

Remove ExporterStatusException and use KeyError

Currently que got a Custom Exception (ExporterStatusException) which verifies if the status that will be set is valid. The right is to to raise a KeyError Exception and remove ExporterStatusException for better standardization.

---

Atualmente nos temos uma Custom Exception (ExporterStatusException) que verifica se o status que será setado é válido. Precisamos remover esta custom exception e utilizar o KeyError no lugar para padronizar.

Add S3 support and custom handler

Não é possível utilizar a mesma coluna duas vezes

Add Travis

Bugs won't find themselves alone.

---

Bugs não se encontram sozinhos.

create teardown to remove tempfiles *.pkl and reports/

Add treatment to False and None fields Attribute

Add general filename preffix / Adicionar prefixo geral para nome do arquivo

Files outputed by DDE uses the Queryset's Model to generate the filename, we need to have a filename to preffix and/or replace this.

--

Arquivos gerados pelo DDE usam o Model do Queryset para buscar um nome e colocar no nome do arquivo, seria excelente ter um prefixo (ou até mesmo trocar o nome do arquivo).

File must not be .read() when joining files

Write tests for utils

Remove join_files from Manager and create a Handler

Improve default join_files

Valor default estão com erro.

Remove celery_app

Exception raised when limit_per_task < Queryset.count()

Fix get headers when default value is set