• in django > 1.6, if elasticsearch is not started, all the tests fail, in django 1.4 only a warning is shown
• the message when elasticsearch is not running is not clear/visible enough
• in django 1.7 there are some warnings

Hi,

I have a use case where I want to use the same model, but which use two different serializers. Each serializer should be on its own elastic-search URL.

Is this possible?

EG:
localhost:9200/django/model-Item/
localhost:9200/django/model-TaskNew/

class Item(EsIndexable, models.Model):
    title = models.CharField("Title", max_length=10000, blank=True)
    created = models.DateTimeField("Created", auto_now_add=True)

    class Elasticsearch(EsIndexable.Elasticsearch):
        serializer_class = ItemEsSerializer, ItemNewEsSerializer
        fields = ['title', 'created']

complete is an awesome api. However lack of payload (basically document id) renders it useless for many usecases.

Basically: Enable the storing of payloads, it defaults to false.

For example:
a. Tag.es.complete('name', 'p')
[u'Play/Outdoor', u'Performing Arts', u'Personality', u'Phonetics', u'Phonics']

b. City.es.complete('city', 'n')
[u'New Delhi']

However, without payload (document id's) its not usable for multiple usecases where the id is passed along to the client/in the workflow.

Checked the mappings for a connection_field city:
"city_complete": {
"max_input_length": 50,
"payloads": false,
"analyzer": "simple",
"preserve_position_increments": true,
"type": "completion",
"preserve_separators": true
},

hi， I wrote test code

model

from django_elasticsearch.models import EsIndexable

class UserInfoSE(EsIndexable, models.Model):
    tel = models.CharField(max_length=50, null=True, unique=False, default='')

view

UserInfoSE.es.queryset.all()

and settings:

ELASTICSEARCH_URL = 'http://192.168.99.101:9200'

but I got some error: Unable to get repr for <class 'django_elasticsearch.query.EsQueryset'>

Because for now we can't do something like MyModel.es.all().deserialize()[5:15].
The idea would be that the .deserialize() method only sets a flag on the query, then on evaluation, the deserialize method would be call.

Thank you for this work! I tried django-elasticsearch with python 3 and "patatras" I realized that it's restricted to python 2.x

I think your approach is relevant and, if interested, I may help you porting this app to python3.

Tell me.

i has this error in a clean code:

14:25 $ git diff
diff --git a/requeriments.txt b/requeriments.txt
index 47d501b..5f36098 100644
--- a/requeriments.txt
+++ b/requeriments.txt
@@ -29,3 +29,5 @@ django-ajax-selects==1.4.1
 Babel==2.3.4
 django-compressor==2.1
 django-libsass==0.7
+elasticsearch==2.3.0
+git+https://github.com/liberation/django_elasticsearch.git
diff --git a/uniresto/products/models.py b/uniresto/products/models.py
index c78781d..4928754 100644
--- a/uniresto/products/models.py
+++ b/uniresto/products/models.py
@@ -9,6 +9,8 @@ from imagekit.processors import ResizeToFill
 from image_cropping import ImageRatioField
 from enum import Enum
 from django.utils.text import slugify
+from django_elasticsearch.models import EsIndexable
+


 COMBO_PRODUCT_TYPES = (
@@ -166,7 +168,7 @@ class Category(models.Model):
         verbose_name_plural = 'Categorias de productos (Pasta, carnes, etc).'


-class ProductTemplates(models.Model):
+class ProductTemplates(EsIndexable, models.Model):
     """
     Almacena plantillas de productos por categorias. Se utiliza para agilizar el registro de todos los productos
     de marca comercial o no ya definidos.
14:36 $ 

in settings:

ELASTICSEARCH_URL='http://localhost:9200'
ELASTICSEARCH_AUTO_INDEX=True
ELASTICSEARCH_DEFAULT_INDEX='uniresto'

Tracktrace:

NameError                                 Traceback (most recent call last)
<ipython-input-3-be67e0bf0776> in <module>()
----> 1 ProductTemplates(name="caca", category=o, celiac=False, delivery=True, consumed_in_store=True, max_choices=1).save()

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/models.py in save(self, *args, **kwargs)
     52     def save(self, *args, **kwargs):
     53         self._raise_no_db_operation()
---> 54         super(EsIndexable, self).save(*args, **kwargs)
     55 
     56     def delete(self, *args, **kwargs):

/home/ignacio/uniresto/lib/python3.5/site-packages/django/db/models/base.py in save(self, force_insert, force_update, using, update_fields)
    708 
    709         self.save_base(using=using, force_insert=force_insert,
--> 710                        force_update=force_update, update_fields=update_fields)
    711     save.alters_data = True
    712 

/home/ignacio/uniresto/lib/python3.5/site-packages/django/db/models/base.py in save_base(self, raw, force_insert, force_update, using, update_fields)
    745         if not meta.auto_created:
    746             signals.post_save.send(sender=origin, instance=self, created=(not updated),
--> 747                                    update_fields=update_fields, raw=raw, using=using)
    748 
    749     save_base.alters_data = True

/home/ignacio/uniresto/lib/python3.5/site-packages/django/dispatch/dispatcher.py in send(self, sender, **named)
    199 
    200         for receiver in self._live_receivers(sender):
--> 201             response = receiver(signal=self, sender=sender, **named)
    202             responses.append((receiver, response))
    203         return responses

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/models.py in es_save_callback(sender, instance, **kwargs)
     70     if not issubclass(sender, EsIndexable):
     71         return
---> 72     instance.es.do_index()
     73 
     74 

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/managers.py in wrapper(*args, **kwargs)
     44         if args[0].instance is None:
     45             raise AttributeError("This method requires an instance of the model.")
---> 46         return f(*args, **kwargs)
     47     return wrapper
     48 

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/managers.py in do_index(self)
    122     @needs_instance
    123     def do_index(self):
--> 124         body = self.serialize()
    125         es_client.index(index=self.index,
    126                         doc_type=self.doc_type,

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/managers.py in wrapper(*args, **kwargs)
     44         if args[0].instance is None:
     45             raise AttributeError("This method requires an instance of the model.")
---> 46         return f(*args, **kwargs)
     47     return wrapper
     48 

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/managers.py in serialize(self)
    101         Note: by default, will use all the model's fields.
    102         """
--> 103         serializer = self.get_serializer()
    104         return serializer.serialize(self.instance)
    105 

/home/ignacio/uniresto/lib/python3.5/site-packages/django_elasticsearch/managers.py in get_serializer(self, **kwargs)
     88     def get_serializer(self, **kwargs):
     89         serializer = self.model.Elasticsearch.serializer_class
---> 90         if isinstance(serializer, basestring):
     91             module, kls = self.model.Elasticsearch.serializer_class.rsplit(".", 1)
     92             mod = importlib.import_module(module)

NameError: name 'basestring' is not defined

• Better nested objects (FK, M2M) handling - check for the related model Elasticsearch serializer presence
• batcher
• Automatic check for corresponding model property on an abstract field

A very common use case is to store different languages for translatable models in different indices (cf https://www.elastic.co/guide/en/elasticsearch/guide/current/one-lang-docs.html).

The main problem is to choose between a new class EsMultiIndexable(?) with the corresponding manager, or a non backward compatible change of the api of EsIndexable (index -> indices, etc).

To be concrete, I've tried to change production database with test one in UnitTests by changing the value of ELASTICSEARCH_DEFAULT_INDEX variable. The problem is, that EsIndexable.Elasticsearch.index is substituted from settings only once on class creation, and any changes in django.settings doesn't affect it. The solution is writing dynamic property.
I'm not sure, but this problem may also occure with other SETTING's variables.

It`s look like possible:
Finding Multiple Exact Values

Hi, I followed the same steps mentioned in given readme.md file

Here is my model:

from django.db import models
from django_elasticsearch.models import EsIndexable

class Product(EsIndexable, models.Model):
    product_name = models.CharField(max_length=100)
    slug = models.SlugField(max_length=100)

After this as per mentioned in steps I did this in python shell

>>> q = Product.es.search('sample')
>>> q

and I got this error in shell

ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection 
object at 0x10828e250>: Failed to establish a new connection: [Errno 
61] Connection refused) caused by: 
NewConnectionError(<urllib3.connection.HTTPConnection object at 
0x10828e250>: Failed to establish a new connection: [Errno 61] 
Connection refused)

What is that I'm missing and how do I proceed now, so that I should get resultant array

We recently deployed a Django app to a live server and ran into an odd error we didn't experience in our dev and staging environment. The error, triggered by a template inclusion on a .mlt() query, is:

AttributeError at
'EsQueryset' object has no attribute '_prefetch_related_lookups'

Error originates from django/db/models/query.py in _fetch_all, line 966 in Django 1.8.3

We solved the issue by adding these three lines:
self._prefetch_related_lookups = []
self._prefetch_done = False
self._known_related_objects = {}
to
... /django_elasticsearch/query.py in def init(self, model, fuzziness=None): of class EsQueryset(QuerySet):

Seems like this was added in Django 1.4 and wasn't reflected in the EsQuerySet class. Not sure why it wasn't triggered on our staging and only on live, so seems to be an edge case.

There are instances where you will use this library on the later parts of development..

In my case,, I just used this library and every data that is already saved before I install this package is not indexed..

By setting ELASTICSEARCH_AUTO_INDEX to true,, everything that will be saved will contain an index.. This will enable me to set index to every data that is already saved by updating or resaving it which is done manually or a lot easier approach is by re-saving them via for loop

But is there an easier way? or something easier can be done to do that? like a manage.py command or something?

Can you provide an example for autocomplete?

Here's my model:

class Article(EsIndexable, BaseContent):
    title = models.CharField(max_length=200, unique=False, blank=True, null=True)
    content = models.TextField(db_column="text", blank=True, null=True)

    class Elasticsearch(EsIndexable.Elasticsearch):
        completion_fields = ['title', 'content']

But then when i run:

Article.es.complete('title', 'test')

I get a

TransportError: TransportError(500, u'{"_shards":{"total":5,"successful":0,"failed":5,"failures":[{"index":"django","shard":0,"status":500,"reason":"BroadcastShardOperationFailedException[[[django][0] ]; nested: ElasticsearchException[failed to execute suggest]

Filter contains didn't work,,,

>>> ModelBuah.es.get(id=3)
{u'nama': u'Langsat', u'jml': 50}
>>> ModelBuah.es.filter(nama__contains='sat')
[]
>>> ModelBuah.es.queryset.filter(nama__contains='sat')
[]

This is very strange action.
Why are you do this?
I`m saving my data with some upper symbols like "hrStorageUsed".
This action is necessary?

Hi there! Excuse me if the question is dumb, I'm very new to elasticsearch and django. In docs for Elasticsearch I can see the "highlight" option to mark the words in document that matched to my query. Is it possible through django-elasticsearch? Thanks in advance!

I am trying to use native django deserialization with EsIndexable models. But django returns DeserializedObject.
Do you have any idea how to fix it?
I`ll continue trying to fix it, but any idea would be helpful.

According to this, when it encounters a method_name it can't find on the instance it just goes pass. Based on the example in test_app I assumed that there was a way to serialize @property abstract methods on the class.

Can you add a exclude_fields to class Elasticsearch to only exclude some fields?

I'm using abstract model. I have a lot of models have a uuid field (and other fields) which is useless for search. Its painfull to use fields to specify each model to exclude the uuid field.

My code like this:

class CommonAttr(models.Model):
    create_date = models.DateTimeField(u"Create data", auto_now_add=True) # exclude
    uuid = models.CharField(u'UUID', max_length=36, unique=True) # exclude

    class Meta:
        abstract = True

    class Elasticsearch(EsIndexable.Elasticsearch):
        exclude_fields = ['uuid',  'create_date']  # this is very usefull for abstract model

class A(CommonAttr):
    pass

class B(CommonAttr):
    pass

Can you add a exclude_fields to only exclude some fields?
Thank you very much.

This is outdated right?

Sorry, for my question.
How to filter by fields "must" and "range"?
I used

q = Traps.es.all().filter(time__gte="now-10d") q.filter(severity="ERROR")
but it tell me raise

RequestError: TransportError(400, u'search_phase_execution_exception', u'No query registered for [must]')

How to get range and must fields filter?

P.S.: query search create
{'query': {'filtered': {'filter': {'range': {'time': {'gte': 'now-10d'}}, 'bool': {'must': [{'term': {'severity': 'error'}}]}}}}}

By adding a doc_type attribute in the Elasticsearch class and use it in get_doc_type method, just like the index attribute.

The problem was mentioned in #4.

Hi,

Nice work on this repo.

I am trying to implement a function to modify my data called get_grouped_tags.

However, I keep getting an error 'QuerySet' object has no attribute 'tag'.

How do I handle M2M's on your serializer? Thanks

myapp/models.py

class Tag(models.Model):
    name = models.CharField("Name", max_length=5000, blank=True)
    taglevel = models.IntegerField("Tag level", null=True, blank=True)


class Item(EsIndexable, models.Model):
    title = models.CharField("Title", max_length=10000, blank=True)
    tag = models.ManyToManyField('Tag', blank=True)
    deleted = models.BooleanField(default=False)

    # def get_grouped_tags(self):
    #     tag = self.tag.order_by('taglevel')
    #     grouped_tags = {
    #         tag_level: [
    #             { 'name': tag_of_level.name, 'taglevel': tag_of_level.taglevel, }
    #             for tag_of_level in tags_of_level
    #         ] for tag_level, tags_of_level
    #         in groupby(tag, lambda tag: tag.taglevel)
    #     }
    #     return grouped_tags

    class Elasticsearch(EsIndexable.Elasticsearch):
        serializer_class = MyModelEsSerializer
        fields = ['title', 'tag']

myapp/elastic/serializers.py

class MyModelEsSerializer(EsJsonSerializer):
    def serialize_tag(self, instance, field_name):
        tag = getattr(instance.tag.order_by('taglevel'), 'tag')
        grouped_tags = {
            tag_level: [
                { 'name': tag_of_level.name, 'taglevel': tag_of_level.taglevel, }
                for tag_of_level in tags_of_level
            ] for tag_level, tags_of_level
            in groupby(tag, lambda tag: tag.taglevel)
        }
        return  grouped_tags

Currently I get this:

>>> from django_elasticsearch.models import EsIndexable
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/.../venv/lib/python2.7/site-packages/django_elasticsearch/models.py", line 4, in <module>
    from django.db.models.signals import post_save, post_delete, post_syncdb
ImportError: cannot import name post_syncdb

MIT, APL2, BSD?

Interested in playing with this at work, but I don't want to get stepped on if you decide for GPL3 down the line.

Thanks!

# datetime
typ = field.get_internal_type()
if val and typ in ('DateField', 'DateTimeField'):
    return datetime.datetime.strptime(val, '%Y-%m-%dT%H:%M:%S.%f') 

This code block raises the following error:

ValueError: unconverted data remains: +00:00

Hi liberation team,

I'm very interesting by your lib but, in your readme.md, I did'nt find information about the ES version compatibility ?

Could you let me know ?

Best Regards,

Nicolas.

The bug is in models.py => es_syncdb_callback(), line 81:
if int(get_version()[2]) > 6: returns False for django 1.10 and 1.11 as it gets 1 > 6 not 11 > 6...
The correct code (for support of django 1.10+ and django 2.0+) is:

v = get_version().split('.')
if int(v[0]) > 1 or int(v[1]) > 6 

Is there a way to fetch models with strict routing set in ElasticSearch settings?
I get TransportError(400, 'RoutingMissingException[...])
When I try to call MyModel.es.get(id=100500) method

Is there any sample or documentation how to make sysnonym token filter ?

https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html#analysis-synonym-tokenfilter

also is it possible to have django model for "synonyms_path"

I do not know if this is a bug but I tried doing:

Artist.es.flush()

However it returned:

AttributeError: 'IndicesClient' object has no attribute 'delete_mapping'

My models.py is:

# Create your models here.
class Artist(EsIndexable, models.Model):
    name = models.CharField(max_length=50)
    birth_date = models.DateField()

    # https://docs.djangoproject.com/es/1.9/ref/models/instances/#get-absolute-url
    # Used to pass a location header URL upon post
    def get_absolute_url(self):
        return reverse('artists-detail', kwargs={'artist_id':self.id})

    def __unicode__(self):
        return self.name

    def get_albums(self):
        x = Album.objects.filter(artist_id=self.id)
        # print x.values('name')
        return x.values('name')

I have this on my settings.py:

ELASTICSEARCH_AUTO_INDEX = 1

I want search via elasticsearch, but does not want to deserealize objects from elastic, but use it from database.

Is subj is possible?

Hi,

Great work on this project.

I have am indexing an Item field which points to Many Tag items as a ManyToMany relationship.

When I add, move, or delete a Tag- it does not update the ES index.

I attempted to create a m2m_changed signal, however it had no affect.

When I add an M2M to Item, or remove one, it only saves when I press save twice (ie re-enter the model).

How can I signal to update the index on an M2M? And also, how would I tell it to keep track of changes in the Tag model (eg update if I change a name within Tag)?

Thanks

#models.py

class Tag(models.Model):
    name = models.CharField("Name", max_length=5000, blank=True)
    taglevel = models.IntegerField("Tag level", null=True, blank=True)


class Item(EsIndexable, models.Model):
    title = models.CharField("Title", max_length=10000, blank=True)
    tag = models.ManyToManyField('Tag', blank=True)

def update_search(instance, **kwargs):
    Item.es.create_index()
 
@receiver(m2m_changed, sender=Task.tag.through)
def index_update(instance):
    instance.es.do_index()

We can run aggregations using the .extra method, but it would be great to have a higher-level support.
Do you guys have any plans/thoughts to support aggregations?

Hey.
My application is built on the bases of basic elasticsearch library. I've decided to ease migrations and installed this extention. Everything is fine except for my system is designed to treat elasticsearch _id as postgresql model id. In this case I can't write custom serializer deserialize(source), which doesn't contain _id field in source parameter. At the same time, it'not clear, how to set up _id in serialize(instance) method.

The current version of django-restframework (2.4) is not compatible with django 1.8.
But its API changed quite a lot, so we might need to create 2 separate modules and load the right one depending on the version of django-restframework.
For now the tests are failing with django 1.8.
We could also raise something if the user try to use django 1.8 with an old version of restframework.

I'm very keen to use this, but I'm not making it very far. Elastic search is running on 9200, I suppose I should begin by stating that. curl -GET "http://localhost:9200/"
{
"status" : 200,
"name" : "Erg",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.1",
"build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
"build_timestamp" : "2015-07-29T09:54:16Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}

Attached this to a very simple model

class Document(EsIndexable, models.Model):
     user = models.ForeignKey(User)
    docfile = models.FileField(upload_to='documents/%Y/%m/%d')
    tags = models.ManyToManyField(Tag)
    posted_date = models.DateTimeField()

then in shell:
>>> from home.models import Document
>>> Document.es.create_index()
>>> Document.es.search('value')
[]
>>> Document.es.count()
0

I've got several rows of data in the model but nothing is coming up when I search. Any thoughts?

Django version 1.8.4

How to configure subset of fields to be index at model level?

I have User model with user_name, email and password. I configured fields like below snapshot but I am still able to search the term mentioned in the email address and password.

class User(EsIndexable, models.Model):
    user_name = models.CharField(max_length=25)
    email = models.CharField(max_length=50)
    password = models.CharField(max_length=25)
class Elasticsearch(EsIndexable.Elasticsearch):
    mappings = {'user_name': {'boost': 2.0, 'analyzer': 'autocomplete'} }
    fields = ['user_name']

I don't want to index email and password. How can I configure model and what other steps required to achieve this?

Hi, I'm trying to use your wrapper (branch with support Django 1.9) and got the error.
ImportError: cannot import name 'post_syncdb'`

I'm using Elastisearch 2.2.0, Django 1.9.2 and Python 3.5.1.

Hi.
rel attribute is deprecated in django 2.0. So serializer is broken.
Most probably rel should be replaced with remote_field attribute for Django 2.0 support.

  File "/home/vagrant/venv/lib/python3.7/site-packages/django_elasticsearch/managers.py", line 46, in wrapper
    return f(*args, **kwargs)
  File "/home/vagrant/venv/lib/python3.7/site-packages/django_elasticsearch/managers.py", line 104, in serialize
    return serializer.serialize(self.instance)
  File "/home/vagrant/venv/lib/python3.7/site-packages/django_elasticsearch/serializers.py", line 167, in serialize
    return json.dumps(self.format(instance),
  File "/home/vagrant/venv/lib/python3.7/site-packages/django_elasticsearch/serializers.py", line 154, in format
    for field in fields])
  File "/home/vagrant/venv/lib/python3.7/site-packages/django_elasticsearch/serializers.py", line 154, in <listcomp>
    for field in fields])
  File "/home/vagrant/venv/lib/python3.7/site-packages/django_elasticsearch/serializers.py", line 118, in serialize_field
    if field.rel:
AttributeError: 'IntegerField' object has no attribute 'rel'

Traceback:

File "/home/gaurav/dev/proj/venv/lib/python2.7/site-packages/rest_framework/generics.py", line 151, in filter_queryset
    queryset = backend().filter_queryset(self.request, queryset, self)
  File "/home/gaurav/dev/proj/venv/lib/python2.7/site-packages/django_elasticsearch/contrib/restframework/restframework3.py", line 25, in filter_queryset
    if view.action == 'list':
AttributeError: 'AccountView' object has no attribute 'action'

AccountView is a ListCreateAPIView with filter_backends = (ElasticsearchFilterBackend, )

ManyToManyField with 'through' Model bring us to some problems

Error on line 118 in django_elasticsearch/serializers.py:
AttributeError: 'ManyToOneRel' object has no attribute 'rel'

Hi,

ES and django seem to work fine except for deleting items. When delete an item, I can no longer access it via it's pk (eg localhost:9200/django/model-X/123). It returns "false".

However, it still appears at localhost:9200/django/model-X/_search. The hits total do not change when I delete an item.

Is there a particular way to remove items form the _search that is different from accessing them at their individual URL?

I've a model whose Elasticsearch class looks like this:

mappings = {
    'name': {
        'boost': 3.0,
        'type': 'completion'
    },
    'custom_fields': {
        'type': 'object'
    }
}

fields = (
    'id',
    'name',
    'description',
    'tags',
    'taxonomy',
    'ordering_name',
    'custom_fields'
)

suggest_fields = ('name',)
facets_fields = ('parent', 'taxonomy',)
completion_fields = ('name',)

When running model.es.complete('name', q) (where q is a string) I get a very long and verbose error from Elasticsearch that boils down to the following (repeated 5 times):

ClassCastException[org.elasticsearch.index.mapper.core.StringFieldMapper cannot be cast to org.elasticsearch.index.mapper.core.CompletionFieldMapper] U

As far as I can tell, everything's defined correctly. I've tried reindexing, but always end up with the same error. Using the latest version from Git - at the time of writing - of this package, 1.9.0 of the Python package and 1.7.2 of the server. Any pointers as to where I might be going wrong would be helpful, if this isn't an issue with the library.

Hello!

I've been trying to work this into a project and running into an error when I put ElasticsearchFilterBackend into the Django REST framework DEFAULT_FILTER_BACKENDS setting.

My config looks like this:

REST_FRAMEWORK = {
    # ...
    'DEFAULT_FILTER_BACKENDS': (
        'rest_framework.filters.OrderingFilter',
        'django_elasticsearch.contrib.restframework.ElasticsearchFilterBackend'
    ),
    # ...
}

And when I try to search an endpoint, I get a TypeError: get_ordering() takes exactly 4 arguments (2 given) on line 43 of django_elasticsearch/contrib/restframework.py (ordering = self.get_ordering(request)).

Looking at the source, it looks like the method call should have queryset and view arguments passed to it as well, but simply adding those in doesn't fix the problem.