Printing out all fields of an entry. without knowing what they are. about elasticsearch-dsl-py

[davidawad]

I see, is there any way to know if the response were empty??
this will re-create that issue

import elasticsearch
from elasticsearch_dsl import Search, Q
from datetime import datetime

es = elasticsearch.Elasticsearch([{u'host': u'himanshu.addteq.com', u'port': b'9200'}])

s = Search(using=es, index= "_all")

response = s.execute()
for hit in response:
    for field in hit:
        print 'Field %s contains:%r' % (field, hit[field])

[honzakral]

Hi!
the output you are seeing is the result's repr and it is truncated since it can be potentially large. If you want to access all the fields you can very easily just iterate over them:

for hit in response:
    for field in hit:
        print 'Field %s contains:%r' % (field, hit[field])

Alternatively you can just access the rad dict of the hit through the _d_ attribute:

for hit in reponse:
    print hit._d_

note, however, that in that case you lose the ability to refer to the fields via the hit.field_name notation and would have to resolve to hit['field_name'].

Does this help? If so I will make sure it's documented. Thanks for the feedback!

[davidawad]

okay, I knew it was going to be something like that. Thanks for all your help today Honza, you've been great.

[davidawad]

I have another question, here's a revised version of that script where i'm probably going to be getting a lot of entries. In the terminal output I'm not seeing nearly as many as kibana is showing me. Is there any pagination happening?

import elasticsearch
from elasticsearch_dsl import Search, Q

es = elasticsearch.Elasticsearch([{u'host': u'192.168.4.151', u'port': b'9200'}])

s = Search(using=es, index= "_all") \
        .query("match", response="404")

response = s.execute()

if response:
        for hit in response: # each 'hit' is an object
                print hit.host+'|'+ hit.response+'|'+hit.request
                #print hit

else: ##all clear, no 404 errors. 
        print "all clear"

0:0:0:0:0:0:0:1:55081|404|/blog/geekery/xvfb-firefox.html/?
0:0:0:0:0:0:0:1:55081|404|/projects/xmlpresenter/demo/presentation-background.png
0:0:0:0:0:0:0:1:55081|404|/apple-touch-icon-precomposed.png
0:0:0:0:0:0:0:1:55143|404|/apple-touch-icon-precomposed.png
0:0:0:0:0:0:0:1:55143|404|/apple-touch-icon.png
0:0:0:0:0:0:0:1:55143|404|/blog/geekery/xvfb-firefox.html/?
0:0:0:0:0:0:0:1:55143|404|/apple-touch-icon-precomposed.png
0:0:0:0:0:0:0:1:55143|404|/apple-touch-icon.png
0:0:0:0:0:0:0:1:55143|404|/apple-touch-icon-precomposed.png
0:0:0:0:0:0:0:1:55143|404|/blog..

the output doesn't have nearly as many entries as there are on the kibana and i'm wondering if there's something happening on the inside that could be causing this.

[davidawad]

in addition the solution of printing them out by looping over them gave a KeyError(0) unfortunately :(

[honzakral]

By default we only return the top 10 results, if you want pagination, just use the slicing operator:

s = Search(...)
s = s[0:10] # or [10:20] for 2nd page etc
response = s.execute()

The KeyError thing is weird, can you reproduce it? Looks like a bug.

Also note that if response will always return True so your logic won't work. But it's a good idea and I will implement it so that it will work in the future.

[honzakral]

Sorry, forgot to add that, you can look at if response.hits.

The code you posted works fine for me, it can be data dependent though.

[davidawad]

reponse.hits is a field of the response object?
It was weird for me but either way I'll keep this issue bookmarked as I'm going to be using the specific fields for another script.

[honzakral]

hits is a field on the Response class that actually contains the hits. There is a shortcut that enables you to just iterate over the response object itself (which you are using) but the data itself is actually stored in .hits. Therefore doing:

response = Search(...).execute()
if not response.hits:
    print("No hits")

works as expected. You can also look at response.hits.total and other attributes that are on that object (everything that is returned by elasticsearch in the 'hits' key).

[davidawad]

cool, one other question. So reading the api I want to use a range of dates. Which I could normally do with

"range" : {
    "timestamp" : {
        "gt" : "2014-01-01 00:00:00",
        "lt" : "2014-01-07 00:00:00"
    }
}

now how could I use the dsl api for this?? would it be

s = Search(using=es, index= "_all") \
        .query("match", response="404")\
        .filter("timestamp", "gt"="2014-01-01 00:00:00" , "lt" : "2014-01-07 00:00:00" ) 

[ChristopherRabotin]

The proper syntax for a range filter is the following, supposing that timestamp is the name of the field:

s = Search(using=es, index= "_all") \
        .query("match", response="404")\
        .filter("range", timestamp={"gt": "2014-01-01 00:00:00" , "lt" : "2014-01-07 00:00:00"})

[davidawad]

ahh i see. In that case i am completely good. Thanks @ChristopherRabotin and of course @honzakral for all the help on multiple repos! Cheers.

[honzakral]

@ChristopherRabotin just adding that you don't need to work with the datetimes as text - elasticsearch-py supports datetime objects and will correctly serialize them when sending data to elasticsearch.