Charts regeneration on data change about django-chartflo

[synw]

Its to research on different possible implementations for data regeneration. I provided one that works for me as an example but it may be interesting for the users to have different approaches.

This implementation reflects what I need and the tooling I use, but people might be more comfortable using well known tools like Celery, and the needs vary so feedback and ideas are welcome here.

I'll start experimenting using workers for regeneration with a super experimental not even documented asynchronous jobs runner of my own that I have to test and develop. It should be similar to any other worker for the code to process the events queue. Maybe we can provide events queue processing code and leave the rest to the user, I don't know I have to make a test implementation to see what it would be like

[synw]

They have interesting paradigms. It seems useful if you use some async libs in python, but without it achieving concurrency with threads looks limited. Their schedulers look nice: they might be interesting to look at. Did you use this framework already?

I use Go for the concurrency, it is way much better than python in this category. I think the concurrency story in python is a mess, and it is more complicated and much slower than Go, so I stick to use the right tool for the job

Btw a first external implementation of generators is available in django-chartmodels: I switched it to the new Chartflo api.

[synw]

I'm actually dealing with the complexity that is now in the core because of the new features, trying to minimize it and to optimize the api design, writing dashboards and refine it little by little. It is not the moment for me to invest in this: I can't see any clear benefit and mostly don't understand it. An example of how we could benefit from it would help

without the need of a worker

I don't see how we can avoid a worker except for immediate processing of changes

[synw]

This question is now solved with the gencharts management command

[brylie]

What is the goal of this task?

[brylie]

By the way, what are your thoughts on RxPY?

[brylie]

I just learned that JavaScript has a Stage 1 Observable proposal that uses semantics from the Rx project. I was just curious about reactive programming in Python, since I am coming from a background in Meteor.js.

[synw]

I found a way: declare the models used in the chart object, and to what generator is attached to. A management commands checks for model changes, pick the right generators and run them

[synw]

I removed the django-extensions script system for a more portable one that allows to embed generators and dashboards in external apps. At startup all the chartflo.py files in apps will be auto-discovered and referenced as generators.

We now have all the needed functionalities to make the module usable: the heavy lifting has been done. I am pretty much satisfied of the api now even if it still needs adjustments, and no major change should occur anymore but it is not considered stable yet

I plan to test all this by embedding dashboards in some apps and verify all runs as expected. Once we have a bit of stability we can think about releasing

[synw]

We now have a management command that checks the events queue for changes and runs the associated generators. Each chart declares the models it is associated with and the generator he depends on.

[brylie]

Each chart declares the models it is associated with and the generator he depends on.

By the way, can this be done at the 'query' level? E.g. if a chart is powered by a specific query, such as as filtering results by date, could we listen for changes that are related to the query?

I am coming from Meteor.js, where we define publications (which are basically database queries that return a cursor) and subscribe to changes that fall within the publication scope.

[synw]

A lot of things have changed recently and an explanation on how it works could help:

Charts creation: generator

1. Gather data with a query, filter anything you want on it: example in the users dashboard
2. Construct a chart passing a query or a dictionary dataset (or a pandas dataframe: to be implemented) to the generate or generate_timeseries method of a ChartController instance that will encode the data and create an Altair chart object. The generator and related models are declared and passed to the function as arguments. A Django Chart instance is created or updated.
3. Under the hood it calls the generate method on the Chart model that will finish the job, recording data and generating html files

Charts update: worker

To update the charts we use a time based worker with the gencharts -s=10 management command, where s is the number of minutes between the runs. It is also possible not to use this option, for example if you prefer to manage this using cronjobs or anything else.
Note: before the first run intialize every generator running this management command: gen chartmodels -all where chartmodels is the name of an app containing the generator

The update process is based on an event queue: the involved models must be registered in settings to be watched: an event will be fired on every create, update, delete operation. The worker processes the events that occured since its last run

E.g. if a chart is powered by a specific query, such as as filtering results by date, could we listen for changes that are related to the query?

The changes detection occurs at model level: a model is attached to a chart: if this model changes the generator associated with the chart will run.

[Edit]: thinking about it I have an old object_level_monitoring branch in Mqueue. In theory it should be possible to implement object level change detection, but I don't feel the need for it now. I want to stabilize and not introduce anymore complexity for the moment. There are still quite a lot of adjustments to make but the core design is stabilized

I am coming from Meteor.js, where we define publications (which are basically database queries that return a cursor) and subscribe to changes that fall within the publication scope.

This is the pub/sub approach. I would love to have something like Rethinkdb changefeeds available but it is not available with the databases we use in Django. I heard that recent Postgresql have a changefeeds mechanism where you can listen to changes on models, but it would break the compatibility with the other Django databases.
Now we could use Redis to implement both of the mechanisms: events queue processing at regular time interval or listen to immediate changes. Both are implemented today using the django-mqueue events. Note that django-mqueue has a Redis hook, so it would be easy to implement, just changing the place where the worker retrieves the events: I use this when a lot of events are fired.
Alternatively we could record the events into other databases and process them from there

[brylie]

What about using the RxPY Observable / Observer classes?

[synw]

These are not very clear to me but if you have inspiration, motivation to try it out and if you feel that we could benefit from it please go ahead

[brylie]

I honestly think you would be in a better position to develop Chartflo with the Observer pattern.

Basically, an Observable can be made from any iterable. An Observer will subscribe to the Observable. Any time the Observable changes, the Observer will be signaled, and can iterate over the changes.

Reactive extensions are the basis for the proposed ECMAScript Observable type, and have been implemented in other languages (including Python, Scala, Java, .NET, Clojure, and Swift). So, it is probably a good time investment to delve a bit into how they work.

[synw]

The question is what do we get involving this layer? Is the complexity worth it?

[brylie]

One benefit is that Observables are somewhat declarative, handling event propagation (complexity) automatically (without the need of a worker, long polling, etc).

It extends the observer pattern to support sequences of data and/or events and adds operators that allow you to compose sequences together declaratively while abstracting away concerns about things like low-level threading, synchronization, thread-safety, concurrent data structures, and non-blocking I/O.

http://reactivex.io/intro.html

The RxPY package might not be too large of a dependency (~174KB).

The observer pattern using Observable/Observer classes may also become a part of the Python language, as more async/reactive applications are built with Python (particularly as Python is drawing some inspiration from the JavaScript developments in this regard).