Coder Social home page Coder Social logo

mypebble / django-pgviews Goto Github PK

View Code? Open in Web Editor NEW
193.0 13.0 41.0 163 KB

Fork of django-postgres that focuses on maintaining and improving support for Postgres SQL Views.

License: The Unlicense

Python 100.00%
sql-views django postgres python pg-views sql

django-pgviews's Introduction

SQL Views for Postgres

Gitter Circle CI

Adds first-class support for PostgreSQL Views in the Django ORM

Installation

Install via pip:

pip install django-pgviews

Add to installed applications in settings.py:

INSTALLED_APPS = (
  # ...
  'django_pgviews',
)

Examples

from django.db import models

from django_pgviews import view as pg


class Customer(models.Model):
    name = models.CharField(max_length=100)
    post_code = models.CharField(max_length=20)
    is_preferred = models.BooleanField(default=False)

    class Meta:
        app_label = 'myapp'

class PreferredCustomer(pg.View):
    projection = ['myapp.Customer.*',]
    dependencies = ['myapp.OtherView',]
    sql = """SELECT * FROM myapp_customer WHERE is_preferred = TRUE;"""

    class Meta:
      app_label = 'myapp'
      db_table = 'myapp_preferredcustomer'
      managed = False

NOTE It is important that we include the managed = False in the Meta so Django 1.7 migrations don't attempt to create DB tables for this view.

The SQL produced by this might look like:

CREATE VIEW myapp_preferredcustomer AS
SELECT * FROM myapp_customer WHERE is_preferred = TRUE;

To create all your views, run python manage.py sync_pgviews

You can also specify field names, which will map onto fields in your View:

from django_pgviews import view as pg


VIEW_SQL = """
    SELECT name, post_code FROM myapp_customer WHERE is_preferred = TRUE
"""


class PreferredCustomer(pg.View):
    name = models.CharField(max_length=100)
    post_code = models.CharField(max_length=20)

    sql = VIEW_SQL

Usage

To map onto a View, simply extend pg_views.view.View, assign SQL to the sql argument and define a db_table. You must always set managed = False on the Meta class.

Views can be created in a number of ways:

  1. Define fields to map onto the VIEW output
  2. Define a projection that describes the VIEW fields

Define Fields

Define the fields as you would with any Django Model:

from django_pgviews import view as pg


VIEW_SQL = """
    SELECT name, post_code FROM myapp_customer WHERE is_preferred = TRUE
"""


class PreferredCustomer(pg.View):
    name = models.CharField(max_length=100)
    post_code = models.CharField(max_length=20)

    sql = VIEW_SQL

    class Meta:
      managed = False
      db_table = 'my_sql_view'

Define Projection

django-pgviews can take a projection to figure out what fields it needs to map onto for a view. To use this, set the projection attribute:

from django_pgviews import view as pg


class PreferredCustomer(pg.View):
    projection = ['myapp.Customer.*',]
    sql = """SELECT * FROM myapp_customer WHERE is_preferred = TRUE;"""

    class Meta:
      db_table = 'my_sql_view'
      managed = False

This will take all fields on myapp.Customer and apply them to PreferredCustomer

Features

Updating Views

Sometimes your models change and you need your Database Views to reflect the new data. Updating the View logic is as simple as modifying the underlying SQL and running:

python manage.py sync_pgviews --force

This will forcibly update any views that conflict with your new SQL.

Dependencies

You can specify other views you depend on. This ensures the other views are installed beforehand. Using dependencies also ensures that your views get refreshed correctly when using sync_pgviews --force.

Note: Views are synced after the Django application has migrated and adding models to the dependency list will cause syncing to fail.

Example:

from django_pgviews import view as pg

class PreferredCustomer(pg.View):
    dependencies = ['myapp.OtherView',]
    sql = """SELECT * FROM myapp_customer WHERE is_preferred = TRUE;"""

    class Meta:
      app_label = 'myapp'
      db_table = 'myapp_preferredcustomer'
      managed = False

Materialized Views

Postgres 9.3 and up supports materialized views which allow you to cache the results of views, potentially allowing them to load faster.

However, you do need to manually refresh the view. To do this automatically, you can attach signals and call the refresh function.

Example:

from django_pgviews import view as pg


VIEW_SQL = """
    SELECT name, post_code FROM myapp_customer WHERE is_preferred = TRUE
"""

class Customer(models.Model):
    name = models.CharField(max_length=100)
    post_code = models.CharField(max_length=20)
    is_preferred = models.BooleanField(default=True)


class PreferredCustomer(pg.MaterializedView):
    name = models.CharField(max_length=100)
    post_code = models.CharField(max_length=20)

    sql = VIEW_SQL


@receiver(post_save, sender=Customer)
def customer_saved(sender, action=None, instance=None, **kwargs):
    PreferredCustomer.refresh()

Postgres 9.4 and up allow materialized views to be refreshed concurrently, without blocking reads, as long as a unique index exists on the materialized view. To enable concurrent refresh, specify the name of a column that can be used as a unique index on the materialized view. Unique index can be defined on more than one column of a materialized view. Once enabled, passing concurrently=True to the model's refresh method will result in postgres performing the refresh concurrently. (Note that the refresh method itself blocks until the refresh is complete; concurrent refresh is most useful when materialized views are updated in another process or thread.)

Example:

from django_pgviews import view as pg


VIEW_SQL = """
    SELECT id, name, post_code FROM myapp_customer WHERE is_preferred = TRUE
"""

class PreferredCustomer(pg.MaterializedView):
    concurrent_index = 'id, post_code'
    sql = VIEW_SQL

    name = models.CharField(max_length=100)
    post_code = models.CharField(max_length=20)


@receiver(post_save, sender=Customer)
def customer_saved(sender, action=None, instance=None, **kwargs):
    PreferredCustomer.refresh(concurrently=True)

Custom Schema

You can define any table name you wish for your views. They can even live inside your own custom PostgreSQL schema.

from django_pgviews import view as pg


class PreferredCustomer(pg.View):
    sql = """SELECT * FROM myapp_customer WHERE is_preferred = TRUE;"""

    class Meta:
      db_table = 'my_custom_schema.preferredcustomer'
      managed = False

Sync Listeners

django-pgviews 0.5.0 adds the ability to listen to when a post_sync event has occurred.

view_synced

Fired every time a VIEW is synchronised with the database.

Provides args:

  • sender - View Class
  • update - Whether the view to be updated
  • force - Whether force was passed
  • status - The result of creating the view e.g. EXISTS, FORCE_REQUIRED
  • has_changed - Whether the view had to change

all_views_synced

Sent after all Postgres VIEWs are synchronised.

Provides args:

  • sender - Always None

Django Compatibility

Django Version Django-PGView Version
1.4 and down Unsupported
1.5 0.0.1
1.6 0.0.3
1.7 0.0.4
1.9 0.1.0
1.10 0.2.0

Django 1.7 Note

Django 1.7 changed how models are loaded so that it's no longer possible to do sql = str(User.objects.all().query) because the dependent models aren't yet loaded by Django.

Django 1.9 Note

You now have to use the .view module directly.

Django 1.10 Note

When updating to Django 1.10, if you see AttributeError: can't set attribute when you try to migrate or run tests, you need to check your migrations for where _base_manager or _default_manager get set on the model and replace it with objects inside the migration.

This also applies to Django MPTT who have covered this in a bit more detail.

Python 3 Support

Django PGViews supports Python 3 in versions 0.0.7 and above.

Django 3.0 Note

Changed from django.utils import six to import six in the views.py file, and added six as a dependency module due to the module being removed from Django.

Django 4.0 Note

Removed default_app_config.

django-pgviews's People

Contributors

alex-j-jackson avatar annefly avatar caioariede avatar codingjoe avatar gitter-badger avatar jheld avatar kennydude avatar killerdom1123 avatar ktosiek avatar ryanolf avatar scott-w avatar toanant avatar tpict avatar zacharyvoase avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-pgviews's Issues

AppRegistryNotReady raises

The scenario is out of the box. When model is get created it cannot release projections and raises AppRegistryNotReady.

(see django-pgviews/view.py line 214)

Maybe I missed something on how to defer loading until registry is get populated...

For this particular project I use Django ver. 2.0
Based on documentation it is not supported, but I'd be glad to contribute myself. Just need some input on direction to go.

Thank you

Failing when using django-tenant-schemas

django-pgviews always assumes that there will be only one schema public and that all the tables/views will belong to it. When using multiple schemas, see: https://github.com/bernardopires/django-tenant-schemas -- views that belong to a tenant fail on App.ready().

Traceback (most recent call last):
  File "/var/www/student-platform/student-platform/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/tenant_schemas/management/commands/migrate_schemas.py", line 42, in handle
    executor.run_migrations(tenants=[self.schema_name])
  File "/usr/local/lib/python3.6/site-packages/tenant_schemas/migration_executors/base.py", line 58, in run_migrations
    run_migrations(self.args, self.options, self.codename, public_schema_name)
  File "/usr/local/lib/python3.6/site-packages/tenant_schemas/migration_executors/base.py", line 31, in run_migrations
    MigrateCommand(stdout=stdout, stderr=stderr).execute(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 257, in handle
    self.verbosity, self.interactive, connection.alias, apps=post_migrate_apps, plan=plan,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/sql.py", line 51, in emit_post_migrate_signal
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 175, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 175, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/var/www/student-platform/student-platform/django_pgviews/apps.py", line 29, in sync_pgviews
    vs.run(force=True, update=True)
  File "/var/www/student-platform/student-platform/django_pgviews/models.py", line 25, in run
    backlog = self.run_backlog(backlog, force, update)
  File "/var/www/student-platform/student-platform/django_pgviews/models.py", line 56, in run_backlog
    index=view_cls._concurrent_index)
  File "/usr/local/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/var/www/student-platform/student-platform/django_pgviews/view.py", line 121, in create_view
    cursor.execute('CREATE OR REPLACE VIEW {0} AS {1};'.format(view_name, view_query))
psycopg2.ProgrammingError: relation "xxxxxxxxx" does not exist
LINE 11:           FROM xxxxxxxxx

Additional indices on materialized views

Is there any way to have custom indices on materialized views, except for the one used for concurrent update?

If not, would it be possible to at least get a signal after the view is created (but still in the same transaction)? Then we'd be able to build additional indices in the same transaction.

importerror with django 1.9

I have the library installed, and I'm running django 1.9 and python3.5

Here is the error I get on startup:
ImportError: No module named 'django_pgviews'
It is listed in my installed_apps.
I'm running version 0.1.0.
Ideas?

Views syncing on every migrate

I have created some materialized views using view.MaterializedView as a base class. I find that every time I run migration, the views are recreated -- dropped and created again.

Is there some way to disable this behaviour?

Tests materialized views

Hey guys,

I have a issue when test materialized views with pytest.

Basically, when I refresh the materialized views by endpoint works, but when I refresh with tests, don't works.

The refresh method works, but don't update datas on materialized view.

Thank you!

Module Import Correction

it should be :

from django_pgviews import view as pg

and not " import django_postgres.view as pg"

querying the MaterializedView

I have a materialized view that is a cached set of results from a few database tables.

I see that you have an option for a model = ForeignKey() field in the read only MV, but I can't quite use that...unless you're just calling it "model" to make it simple to look at, instead of being a hook the app knows about.

Basically, when I call MyMV.objects.count(), I get an error that the relation doesn't exist. I have managed=True, abstract=False, and I have the db_name set correctly. Calling refresh on the class in ipython does seem to work (takes awhile to finish anyway).

Any ideas?

cannot drop view whily syncing

Hi,
we saw that there is an issue with syncing views, in case you change views that are dependencies for other views:

How to reproduce the error

  1. create 2 Views like this
class Test1View(django_pgviews.View):
    sql = "SELECT last_name, email from auth_user;"

    class Meta:
        db_table = 'myapp_test1view'
        managed = False


class Test2View(django_pgviews.View):
    dependencies = [
        'myapp.Test1View',
    ]
    sql = "SELECT email from common_test1view;"

    class Meta:
        db_table = 'common_test2view'
        managed = False
  1. run manage.py sync_pgviews --force

  2. change the sql of Test1View to be SELECT last_name, first_name, email from auth_user;

  3. run manage.py sync_pgviews --force
    --> You will encounter the following error during sync:
    psycopg2.InternalError: cannot drop view myapp_test1view because other objects depend on it

I will try to come up with a fix for this, unless you know another way how to get around this error.

Does django-pgviews allow Updating of views?

I'm trying to update an object in Django Admin but I'm getting an error

DETAIL:  View columns that are not columns of their base relation are not updatable.

wanted to enquire if the package allows update.

How to refresh materialised view in case of multidb in django using manage.py?

I am using Postgresql as database in my django project. I am using materialised views also using django-pgviews which is django module for using materialised views in django.

To refreshing materialised views I use command below

python manage.py sync_pgviews

But now I am using multidb in my django project. Which is easily manageable like

python manage.py migrate # This will apply sql to default database
python manage.py migrate --database=tableau # This will apply sql to tableau database

I was expecting --database=tableau option to work with python manage.py sync_pgviews also. But this does not work.

python manage.py sync_pgviews # refresh default db views
python manage.py sync_pgviews --database=tableau # This does not work

How can I refresh materialised view of other database using django using manage.py?

Django 3.0 and Python 3 Support

We would really like to support Django 3 in out app but use django-pgviews quite alot and it uses on six which is now removed. From what I can tell the only usages are in views.py and I tried working on a PR but I do not quite understand how the six.with_metaclass function works and how to replace it.

If anyone is able to advise me I would be happy to submit a PR.

Python 3 support

People somehow maintain packages working both with python 2 and python 3. In this case I think it will be extremely easy โ€“ looks like the only inconsistent thing is except Exception, e notation. Can I help you with this?

Materialized views

Another extremely simple to create feature: am I right that it will take only to add MATERIALIZED into View creation syntax?

id Column does not exist

I am trying to use a MaterializedView to hold some aggregated queries from one of my model tables. Everything seems fine, however when I try to use the result like a queryset calling MyView.objects.all() I get the error column per_machine.id does not exist. It seems that django expects an id column to always be present but that unlike a normal model, and AutoField isn't assigned by default.
Is this just the expected behaviour and if so is the intention that one should specify an explicit id column or am I simply misusing the resulting queryset by calling objects.all()?

Error with multiple field indexing

 File "../lib/python3.6/site-packages/django_pgviews/view.py", line 116, in create_view
    cursor.execute('CREATE UNIQUE INDEX {0}_{1}_index ON {0} ({1})'.format(view_name, index))
psycopg2.ProgrammingError: syntax error at or near ","
LINE 1: CREATE UNIQUE INDEX event_view_id, title, external...

Document projection

You are using projection = ['myapp.Customer.*',] in the example, but I can't find any reference to what it does or how it is to be used.

0.5.1 PYPI build fails looking for README.md

Receive the following message when I run pip install django-pgviews

Downloading django-pgviews-0.5.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-FEpTji/django-pgviews/setup.py", line 9, in
LONG_DESCRIPTION = open('README.md').read()
IOError: [Errno 2] No such file or directory: u'README.md'

Running pip install "django-pgviews<0.5.1" installs the library correctly.

Views cannot depend on other views

You cannot define a view which uses another view and reliably get the sync_pgviews command to work correctly.

Proposed solution:

class MyView(View):
    class Meta:
         dependencies = ('my_app.View', 'my_app.Otherview',)

Clarify Foreign Keys

In the README, we should clarify how ForeignKey relationships work in the context of SQL Views.

__init__() got an unexpected keyword argument 'providing_args'

When use django-pgviews==0.5.7 with Django 4 raise thise error:


  File ".../lib/python3.8/site-packages/django_pgviews/signals.py", line 4, in <module>
    view_synced = Signal(
TypeError: __init__() got an unexpected keyword argument 'providing_args'

Use create/rename for materialized views

Currently sync_pgviews will lock all transactions using a given materialized view while it's being recreated.

I think it would be better to first create the materialized view under a different name, and only then drop the old one and rename the new one. Or even better, have an option to use migration scripts for managing the views.

Switch CI system for better build matrix

Hi, I'd like to propose a switch to travis-ci.
I think thee is an advantage when it comes to the support of multiple python and Django versions.
Travis-CI has a more advanced build matrix.

I'd be willing to set it up. As a plus I could even do a full 'django-cc' integration.

Let me know if you're interested and I'll open up a PR.

ForeignKey in view?

Hi,

Maybe this is a dumb question, but is it possible to define a pg.View with a ForeignKey in its projection? The reason I ask, is its natural given an model with an IP address in a model (and postgres) to say something like:

select v.*,o.id AS organisation_id from data_visitor v LEFT OUTER JOIN v.source_ip <<= data_network n LEFT OUTER JOIN data_organisation o ON n.organisation_id = o.id

and then say:
class VisitorOrganisation(pg.View):
projection = ['Visitor.*', models.ForeignKey(...)]
sql = ...

but this fails for me under django v1.8 - models.IntegerField in place of the foreign key is an ugly hack, but it works. Perhaps there is a better way to achieve the ability to identify an organisation for a given visitor's IP address using the postgres specific SQL?

Projection not working with MaterializedView

When I use a string Projection and try to filter on a field my view should have, I get this error FieldError: cannot resolve keyword 'x' into field. Choices are: id.

After a quick investigation in the ViewMeta, it seems the fields from my real model are not being copied immediately and are put in a deferred situation because they are strings. The thing is, as they are deferred, the super new is called without them being attributed and that apparently causes django to miss them. I then only get one field on my view in the end, which is id.

As the issue was with the projection being a string, I tried putting projection = MyModel._meta.fields but got an error with reverse name clashes on the foreign keys.

Do you have any solution for this problem ?
I don't want to specify every field on my view especially since they are the same.

Sometimes Django 2.1.2 return this random error

ProgrammingError: relation "misuper_stockitemmaterializedview" does not exist
LINE 1: ...ct"."unit_stock", "misuper_product"."colors" FROM "misuper_s...

The error is random, because from 10.000 request, only 3-4 fails with that error in the endpoint. Any reason for that?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.