niwinz / djorm-pgarray

PostgreSQL array field for Django.

djorm-pgarray

NOTE: pypi package name changed from djorm-ext-pgarray to more simple djorm-pgarray

PostgreSQL array field for Django.

Documentation: http://niwinz.github.io/djorm-pgarray/

djorm-pgarray's Issues

Bug with the default form field

I am using SmallIntegerArrayField() (version 1.0.2, Django==1.6.5) in one of my model. When a model form is used for the model (admin or other) and a user types "1,2,3" in the field and submits the form and a validation error occurs, the value in the field becomes "1,,,2,,,3" (similarly "123" becomes "1,2,3").

I investigated a bit and found that in this function https://github.com/niwibe/djorm-pgarray/blob/0ff2027fd571f85c5339220bf871843eae072286/djorm_pgarray/fields.py#L232, the type of parameter value is string (not list). If I comment out join operation in the function, it works fine.

How to properly delete an object that uses pg_array fields

I have a simple model that includes three pg_array fields:

class GroundTrack(models.Model):
    class Meta:
        app_label = 'simulation'
    objects = GroundTrackManager()
    tle = models.ForeignKey(tle.TwoLineElement, unique=True)
    latitude = pgarray_fields.FloatArrayField()
    longitude = pgarray_fields.FloatArrayField()
    timestamp = pgarray_fields.BigIntegerArrayField()

When I try to delete one of the rows of the table, a "RuntimeError: maximum recursion depth exceeded in cmp":

python manage.py shell$: GroundTrack.objects.all()[0].delete()

... and the exception (shortened beforehand):

File "/home/rtubio/repositories/satnet/server/WebServices/services/configuration/models/segments.py", line 170, in spacecraft_deleted
        return instance.groundtrack.delete()
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 694, in delete
    collector.collect([self])
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 197, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 17, in CASCADE
    source_attr=field.name, nullable=field.null)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 193, in collect
    sub_objs = self.related_objects(related, new_objs)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 213, in related_objects
    **{"%s__in" % related.field.name: objs}
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 593, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 611, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1204, in add_q
    clause = self._add_q(where_part, used_aliases)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1240, in _add_q
    current_negated=current_negated)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1128, in build_filter
    lookup_type, value)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 1112, in get_lookup_constraint
    (Constraint(alias, targets[0].column, sources[0]), lookup_type, value), AND)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/utils/tree.py", line 104, in add
    data = self._prepare_data(data)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 61, in _prepare_data
    if is_iterator(value):
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/local/lib/python2.7/site-packages/django/utils/itercompat.py", line 30, in is_iterator
    return isinstance(x, collections.Iterator)
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/lib/python2.7/abc.py", line 141, in __instancecheck__
    subtype in cls._abc_negative_cache):
File "/home/rtubio/repositories/satnet/server/WebServices/.venv/lib/python2.7/_weakrefset.py", line 75, in __contains__
    return wr in self.data
RuntimeError: maximum recursion depth exceeded in cmp

Is there any specific way for deleting an object when using any of the fields form pg_array?

Table creation in Django 1.7

Using migrations in Django 1.7 to create a table containing an array field creates the table without the field.

ArrayField(dbtype='date')

There does not seem to be any functionality for creating an ArrayField of date or datetime objects. Am I mistaken?

Migrating to 1.8's django.contrib.postgres.fields.ArrayField

Do you (or anyone following along) have any experience about the migration path between djorm-pgarray fields and the new Postgres fields in 1.8? Just curious, thanks for this package!

Tag version 1.2 in git

Could you please tag v1.2 in git?

ArrayFormField with dimension > 1

This is a very useful library but I've hit one problem I can't figure out. I have a field:

ArrayField(dbtype="int", dimension=2)

When it renders in a form I get a text box but I can't find a format that works to enter the data. I tried '[[1],[2]]' and '[1],[2]' to no avail. Is there a way to do this or is it a bug/limitation?

Thanks!

How to use PostgreSQL array operator in Django 1.6?

I browse the source code find that the array operator can only use in Django 1.7, is there anyway use it in Django 1.6?

django.db.utils.ProgrammingError: operator class "text_pattern_ops" does not accept data type text[]

In our development we are using django==1.8.2 and we have this strange problem on initial migrations. To be more specific the fields.py:

class LanguagesField(TextArrayField):
    def __init__(self, *args, **kwargs):
        kwargs.update({
            'verbose_name': _('языковые версии'),
            'choices': settings.LANGUAGES,
            'default': ('en',),
            'db_index': True
        })

        super(LanguagesField, self).__init__(*args, **kwargs)

    def formfield(self, **kwargs):
        kwargs['widget'] = forms.CheckboxSelectMultiple(attrs={'class': 'uioff'})
        kwargs['required'] = True

        return super(LanguagesField, self).formfield(**kwargs)

In models.py:

langs = LanguagesField()

The initial migration:

fields = [
...

('langs', multilanguages.fields.LanguagesField(default=('en',), choices=[(b'en', '\u0410\u043d\u0433\u043b\u0438\u0439\u0441\u043a\u0438\u0439'), (b'ru', '\u0420\u0443\u0441\u0441\u043a\u0438\u0439'), (b'tr', '\u0422\u0443\u0440\u0435\u0446\u043a\u0438\u0439')], dbtype='text', verbose_name='\u044f\u0437\u044b\u043a\u043e\u0432\u044b\u0435 \u0432\u0435\u0440\u0441\u0438\u0438', db_index=True)),
...
]

And error on ./manage.py migrate

django.db.utils.ProgrammingError: operator class "text_pattern_ops" does not accept data type text[]

dbtype / type_cast are not Django-esque

I think it would make a lot more sense if we had multiple classes for each database type, instead of shoehorning all of the various types into a single class. This sort of behavior is not present in any of the default Django fields.

In my opinion, it would be more Pythonic/Django-esque if there were a class for each database type, just like there are for the default Django classes. This could potentially make validation and preparation clearer.

class ArrayField(fields.Field):
    """ Abstract base. Implement common stuff that all ArrayFields have.
    """

class TextArrayField(ArrayField):
    """ Implement text[] specific stuff.
    """

class EmailArrayField(TextArrayField):
    """ Add Email validation and other snazzy stuff.
    """

class IntegerArrayField(ArrayField):
    """ Implement int[] specific stuff.
    """

This is just one proposal. It might be possible to do something clever by having TextArrayField inherit both from ArrayField and TextField, or by having TextArrayField inherit from TextField and use some kind of metaclass to transform TextField into TextArrayField.

Django Admin

Does this library support editing pgarrays from the Django admin console?

Incorect work with migrations in Django 1.7

Problem with default values of fields.
For example: roles = IntegerArrayField(blank=True, default=[0, 1])
It create migration like:

...
field=djorm_pgarray.fields.IntegerArrayField(default=[0, 1]),
...

It causes error when migrate: django.db.utils.DataError: missing "]" in array dimensions

Temporarily I manualy changed migration to field=djorm_pgarray.fields.IntegerArrayField(default='{0, 1}'). It works fine.

Do you have any ideas to fix it?

Unicode values and `ArrayFormField`

https://github.com/niwibe/djorm-ext-pgarray/blob/d48b6e7f6d8e2e3c63bf977c81dd5a7b7463c534/djorm_pgarray/fields.py#L143

This line causes problems when displaying values outside ASCII range:

...
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/django/template/debug.py", line 91, in render
    output = force_text(output)
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/django/utils/encoding.py", line 100, in force_text
    s = s.__unicode__()
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/django/forms/forms.py", line 425, in __str__
    return self.as_widget()
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/django/forms/forms.py", line 475, in as_widget
    return widget.render(name, self.value(), attrs=attrs)
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/django/forms/forms.py", line 513, in value
    return self.field.prepare_value(data)
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/djorm_pgarray/fields.py", line 143, in prepare_value
    return self.delim.join(str(v) for v in value)
  File "/home/paczkowski/.virtualenvs/itemizer/local/lib/python2.7/site-packages/djorm_pgarray/fields.py", line 143, in <genexpr>
    return self.delim.join(str(v) for v in value)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

My array field type is text: ArrayField(dbtype='text') and I'm using unmodified ModelAdmin. Values in the column: {Żółty,Клиент,こんにちは}

TLDR: where's PyPI release?

TL;DR: WHERE'S THE PYPI RELEASE?

Please create a new release on Python Package Index.

Longer version:

Hi!

came here from version 0.10 to implement deconstruct method for Django 1.7 and IT IS ALREADY THERE! THANKS!

This package is great.

Thank you for creating it.

Model class is read multiple times, but pop removes the dbtype argument

I am working on upgrading to Django 1.7 and a new djorm_pgarray module. I'm having an issue where all models that use this:

myattribute = ArrayField(dbtype="varchar(255)", ...)

are showing up in the migration as:

('myattribute', djorm_pgarray.fields.ArrayField(default=None, null=True,  blank=True))

It seems to be because the model is loaded twice, and the fields.py line:

self._array_type = kwargs.pop("dbtype", "int")

is actually removing the varchar(255) so when the class is read a second time, the dbtype is missing.

This causes all future ORM operations to break, of course...

How to remove an element from the array?

Hi,

I have been using django_pgarray for a while now. However, I see no way to say, add/remove a single element from the array or another array in the documentation.

Is this doable? If so, how?

Text field only works for 1D arrays

Hi,
Not sure if I'm just not getting it.

It looks like that for a text array field it'll only ever handle a 1D array, everything else gets converted to a Unicode string (with the text of what the arrays would be).

Is that correct/deliberate?

Thanks for the lib!

(Example straight from the DB)
{"[1, 2]","[3, 4]"}
instead of:
{["1", "2"],["3", "4"]}

Default null and blank kwargs are inconsistent with default null and blank kwargs for Django fields

According to the Django docs, the default values for Field.null and Field.blank are False.

Why do blank and null default to True for ArrayField?

https://docs.djangoproject.com/en/1.6/ref/models/fields/#null

Multiple DB representations for blank values

Under some conditions, NULL will get inserted for blank values, but sometimes {} (an empty array) will get inserted. This is a similar to if you created a CharField(blank=True, null=True). The default behavior for Django is to set NOT NULL on CharField and insert zero-length strings (literally: '') for blank values. I think that to be consistent with existing Django fields, we should insert a zero-length array (literally: {}) for blank values.

This is related to issue #27 which I posted a moment ago.

Unable to update

When updating an existing model, I get a DatabaseError

array value must start with "{" or dimension information

It seems that query is just using %s and is passing a list as the parameter.

Set empty list/tuple as default

I tried to make ArrayField to have empty tuple as default, but it is converted to string and thus default value is this ["(", ")"]. What am I missing here?

Arrays stopped working for our type.

We use an array of macaddr, with a custom class to represent MAC addresses in our Python code. But with recent versions of pgarray, this has started failing with “TypeError: invalid postgreSQL type: macaddr.” Of course it's not an invalid type — it's just a type that pgarray does not support.

Supporting more types might help, but in our case we already have a working custom type which would be a pain to replace. The smoothest and most durable way to solve the problem from our perspective would be to make fields.TYPES extensible. We find we can work around the problem by having our code add an entry to map 'macaddr' to our custom type.

admin form cannot handle non-string arrays

I have an array field defined by

times = ArrayField(dbtype="float(53)",dimension=1)

but I get a DatabaseError when trying to save "1.0,2.0,3.0" into the admin form

column "times" is of type double precision[] but expression is of type text[] LINE 1: ...te", "left_sweep", "sort") VALUES ('', '', '', 2, ARRAY['1.0...

in "postgresql_psycopg2/base.py in execute, line 54"

I can add the model from the shell without a problem, passing a list of floats. Similarly, I can retrieve the list from the shell.

But when I go to edit the instance on the admin form I get a TypeError:

sequence item 0: expected string, float found

in "djorm_pgarray/fields.py in prepare_value, line 92"

I wish I could offer a bugfix, but I don't quite understand how prepare_value() is doing its thing

Use GIN index by default

Btree indexes are not ideal for arrays.
By default GIN should be used, or at least it should be provided as an option.
The JSONField guys might have some useful pointers.
https://github.com/yjmade/django-pgjsonb/tree/master/django_pgjsonb

How work with pgarray?

Quering don't works. No one. How use it?

System: python2.7, PostgreSQL 9.3.5, Django==1.6.7, djorm-pgarray==1.0.4, etc

Model:

class SomeModel(models.Model):
    title = models.TextField()
    tags = IntegerArrayField()

In shell:

SomeModel.objects.create(title='name1', tags=[1,2])
SomeModel.objects.filter(tags__overlap=[1]) # -> FieldError: Join on field 'tags' not permitted. Did you misspell 'overlap' for the lookup type?
SomeModel.objects.filter(tags__contains=[1]) # -> Operator doesn't exist. LINE 1: ...s_somemodel" WHERE "users_somemodel"."tags"::text LIKE ARRAY...
SomeModel.objects.filter(tags__contained_by=[1, 2, 3]) # -> FieldError: Join on field 'tags' not permitted. Did you misspell 'contained_by' for the lookup type? 

What I doing wrong?

It's works:

SomeModel.objects.all().extra(where=['''tags <@ ARRAY[1,2,3]'''])
[<SomeModel: SomeModel object>]