Coder Social home page Coder Social logo

tornado-es's Introduction

Build Status

Tornado-es

A tornado-powered python library that provides asynchronous access to elasticsearch.

Install

Via pip:

pip install tornadoes

Usage

Indexing a dummy document:

$ curl -XPUT 'http://localhost:9200/index_test/typo_test/1' -d '{
    "typo_test" : {
        "name" : "scooby doo"
    }
}'

Tornado program used to search the document previously indexed:

# -*- coding: utf-8 -*-

import json

import tornado.ioloop
import tornado.web

from tornadoes import ESConnection


class SearchHandler(tornado.web.RequestHandler):

    es_connection = ESConnection("localhost", 9200)

    @tornado.web.asynchronous
    def get(self, indice="index_test", tipo="typo_test"):
        query = {"query": {"match_all": {}}}
        self.es_connection.search(callback=self.callback,
                                  index=indice,
                                  type=tipo,
                                  source=query)

    def callback(self, response):
        self.content_type = 'application/json'
        self.write(json.loads(response.body))
        self.finish()

application = tornado.web.Application([
    (r"/", SearchHandler),
])

if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

Development

Setup your development environment:

make setup

Note: Don't forget to create a virtualenv first

Run tests:

make test

Note: Make sure ElasticSearch is running on port 9200

Contributing

Fork, patch, test, and send a pull request.

License

MIT © Globo.com

tornado-es's People

Contributors

blampe avatar cristianoliveira avatar gustavoluz avatar heynemann avatar kmike avatar luizgpsantos avatar matityahul avatar newgene avatar nsigustavo avatar ricobl avatar scorphus avatar sergiojorge avatar sjhewitt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tornado-es's Issues

Allow to specify keyword arguments for HTTPClient when creating instance of ESConnection

Current API does not allow to specify parameters for Tornado.AsyncHTTPClient used under the hood by Tornado-es when connecting to ES.

I stumbled on this because I'm debugging Tornado app and noticed warnings in logs

max_clients limit reached, request queued. 10 active, 11 queued requests.

these warnings are emitted by AsyncHTTPClient. I would like to adjust number of concurrent connections. Base Tornado client has variety of parameters that can be passed see here. If you look at usage of client in tornado-es you can see that you can't pass anything to http client.

This would be just matter of adding some extra arguments to __init__ and passing them to client. Is it ok if I create PR with that?

cc @luizgpsantos

Document Update

I do not see a way to update a document from viewing the source of the project. I feel there should be a way to do this.

Improve Docs

It's a little unclear what I'm able to accomplish with this project. Is it only for search? Can I index things?

It might be I'm a newbie with ElasticSearch, but I got a little lost.

Return more than just _source in ESConnection.get method

I'm also not sure returning only _source is good because there is other useful data in the result (like "found" and "version"). This change would be backwards incompatible though.

If you want only _source it may be more efficient to query it directly by appending /_source to the URL.

Extend support to Tornado 3.2

Please extend support to Tornado version 3.2. As far as I can tell from the tests, tornado-es supports it already:

$ pip list | grep tornado
tornado (3.2)
tornadoes (1.4.2)

$ make test
Cleaning up build and *.pyc files...
Done!
test_can_put_and_delete_document (unit.test_tornadoes.TestESConnection) ... ok
test_count_specific_index (unit.test_tornadoes.TestESConnection) ... ok
test_count_specific_query (unit.test_tornadoes.TestESConnection) ... ok
test_count_specific_query_with_many_parameters (unit.test_tornadoes.TestESConnection) ... ok
test_count_specific_query_with_parameters (unit.test_tornadoes.TestESConnection) ... ok
test_count_specific_type (unit.test_tornadoes.TestESConnection) ... ok
test_search_apecific_type (unit.test_tornadoes.TestESConnection) ... ok
test_search_for_specific_type_with_query (unit.test_tornadoes.TestESConnection) ... ok
test_search_specific_index (unit.test_tornadoes.TestESConnection) ... ok
test_should_access_specific_documento (unit.test_tornadoes.TestESConnection) ... ok
test_should_accumulate_searches_before_search (unit.test_tornadoes.TestESConnection) ... ok
test_should_clean_search_list_after_search (unit.test_tornadoes.TestESConnection) ... ok
test_should_generate_empty_header_with_no_index_specified (unit.test_tornadoes.TestESConnection) ... ok
test_should_make_two_searches (unit.test_tornadoes.TestESConnection) ... ok
test_simple_search (unit.test_tornadoes.TestESConnection) ... ok
test_can_put_and_delete_document (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_count_specific_index (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_count_specific_query (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_count_specific_query_with_many_parameters (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_count_specific_query_with_parameters (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_count_specific_type (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_search_apecific_type (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_search_for_specific_type_with_query (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_search_specific_index (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_should_access_specific_documento (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_should_clean_search_list_after_search (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_should_make_two_searches (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok
test_simple_search (unit.test_tornadoes.TestESConnectionWithTornadoGen) ... ok

----------------------------------------------------------------------
Ran 28 tests in 0.155s

OK

NOTE: Tests that check the number of all documents -- test_search_all_entries, test_count_all_entries, test_search_all_entries, test_count_all_entries -- were not considered.

Support newer version of tornado

The current setup.py requirements forces me to use tornado 2.4 which is old news by now.

We should test this against newer versions of tornado and make sure we are up to date with the newer releases.

update api doesn't work right

I try to use update api to update document's version ,but result seems wrong ,update add a new attribute to the document, for example,

{
  "supplier_matchcode": "SMPE",
  "article_number": "CS1342",
  "description": "Fuel Parts",
  "url": "",
  "len": 6,
  "label": "Sensor",
  "version": 0,
  "supplier_cname": "",
  "supplier": "SMPE",
  "group": "",
  "replace_numbers": "",
  "utility_number": ""
}

I use below script to update it's version

{
  "script": "ctx._source.version+=1"
}

this work right in Official elasticsearch, after update I will get this

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "brand",
        "_type": "part",
        "_id": "CS1342",
        "_score": 1,
        "_source": {
          "supplier_matchcode": "SMPE",
          "article_number": "CS1342",
          "description": "Fuel Parts",
          "url": "",
          "len": 6,
          "label": "Sensor",
          "version": 1,
          "supplier_cname": "",
          "supplier": "SMPE",
          "group": "",
          "replace_numbers": "",
          "utility_number": ""
        }
      }
    ]
  }

but tornado-es update will add a new attribute to the document ,just like below

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "brand",
        "_type": "part",
        "_id": "CS1342",
        "_score": 1,
        "_source": {
          "supplier_matchcode": "SMPE",
          "article_number": "CS1342",
          "description": "Fuel Parts",
          "url": "",
          "len": 6,
          "label": "Sensor",
          "version": 0,
          "supplier_cname": "",
          "supplier": "SMPE",
          "group": "",
          "replace_numbers": "",
          "utility_number": "",
          "script": "ctx._source.version+=1"
        }
      }
    ]
  }

and this is my update code

from tornadoes import ESConnection
self.es = ESConnection()
self.es.update(index=mona.index,type='part',uid=mona.param['id'],contents=mona.body,callback=self.callback)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.