elastic / elasticsearch-rails

Elasticsearch integrations for ActiveModel/Record and Ruby on Rails

Elasticsearch Rails

This repository contains various Ruby and Rails integrations for Elasticsearch:

• ActiveModel integration with adapters for ActiveRecord and Mongoid
• Repository pattern based persistence layer for Ruby objects
• Enumerable-based wrapper for search results
• ActiveRecord::Relation-based wrapper for returning search results as records
• Convenience model methods such as search, mapping, import, etc
• Rake tasks for importing the data
• Support for Kaminari and WillPaginate pagination
• Integration with Rails' instrumentation framework
• Templates for generating example Rails application

Elasticsearch client and Ruby API is provided by the elasticsearch-ruby project.

Installation

Install each library from Rubygems:

gem install elasticsearch-model
gem install elasticsearch-rails

Compatibility

The libraries are compatible with Ruby 3.0 and higher.

We follow Ruby’s own maintenance policy and officially support all currently maintained versions per Ruby Maintenance Branches.

The version numbers follow the Elasticsearch major versions. Currently the main branch is compatible with version 8.x of the Elasticsearch stack.

Rubygem		Elasticsearch
0.1	→	1.x
2.x	→	2.x
5.x	→	5.x
6.x	→	6.x
7.x	→	7.x
8.x	→	8.x
main	→	8.x

Check out Elastic product end of life dates to learn which releases are still actively supported and tested.

Usage

This project is split into three separate gems:

• elasticsearch-model, which contains search integration for Ruby/Rails models such as ActiveRecord::Base and Mongoid,

• elasticsearch-persistence, which provides a standalone persistence layer for Ruby/Rails objects and models

• elasticsearch-rails, which contains various features for Ruby on Rails applications

Example of a basic integration into an ActiveRecord-based model:

require 'elasticsearch/model'

class Article < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end

# Index creation right at import time is not encouraged.
# Typically, you would call create_index! asynchronously (e.g. in a cron job)
# However, we are adding it here so that this usage example can run correctly.
Article.__elasticsearch__.create_index!
Article.import

@articles = Article.search('foobar').records

You can generate a simple Ruby on Rails application with a single command (see the other available templates). You'll need to have an Elasticsearch cluster running on your system before generating the app. The easiest way of getting this set up is by running it with Docker with this command:

  docker run \
    --name elasticsearch-rails-searchapp \
    --publish 9200:9200 \
    --env "discovery.type=single-node" \
    --env "cluster.name=elasticsearch-rails" \
    --env "cluster.routing.allocation.disk.threshold_enabled=false" \
    --rm \
    docker.elastic.co/elasticsearch/elasticsearch:7.6.0

Once Elasticsearch is running, you can generate the simple app with this command:

rails new searchapp --skip --skip-bundle --template https://raw.github.com/elasticsearch/elasticsearch-rails/main/elasticsearch-rails/lib/rails/templates/01-basic.rb

Example of using Elasticsearch as a repository for a Ruby domain object:

class Article
  attr_accessor :title
end

require 'elasticsearch/persistence'
repository = Elasticsearch::Persistence::Repository.new

repository.save Article.new(title: 'Test')
# POST http://localhost:9200/repository/article
# => {"_index"=>"repository", "_id"=>"Ak75E0U9Q96T5Y999_39NA", ...}

Please refer to each library documentation for detailed information and examples.

Model

• [README]
• [Documentation]
• [Test Suite]

Persistence

• [README]
• [Documentation]
• [Test Suite]

Rails

• [README]
• [Documentation]
• [Test Suite]

Development

To work on the code, clone the repository and install all dependencies first:

git clone https://github.com/elastic/elasticsearch-rails.git
cd elasticsearch-rails/
bundle install
rake bundle:install

Running the Test Suite

You can run unit and integration tests for each sub-project by running the respective Rake tasks in their folders.

You can also unit, integration, or both tests for all sub-projects from the top-level directory:

rake test:all

The test suite expects an Elasticsearch cluster running on port 9250, and will delete all the data.

License

This software is licensed under the Apache 2 license, quoted below.

Licensed to Elasticsearch B.V. under one or more contributor
license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright
ownership. Elasticsearch B.V. licenses this file to you under
the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.

elasticsearch-rails's Issues

Incomplete index in Callback

Hi @karmi ,

Thank you for your awesome gems!

I am replacing Tire to elasticsearch-rails, the first issue is that it won't perform as_indexed_json when update_document, It' s update changed_attributes only.

Is there any solution about this? perform as_indexed_json when update_document, not only changed_attributes .

Source

https://github.com/elasticsearch/elasticsearch-rails/blob/master/elasticsearch-model/lib/elasticsearch/model/indexing.rb#L334

Model

class User << ActiveRecord::Base

  has_many :keywords

  def as_indexed_json(options={})
    { :id => id, 
      email: email,
      first_name: first_name, 
      last_name: last_name,

      full_name: "#{first_name} #{last_name}",
      research_interests: keywords.map(&:name)
    }
  end

end

Kaminari `per` method support not returning correct records

I have a query that returns 659 records:

Product.search('colt').results.total
#=> 659

If I'm using the default pagination window size of 25, this gets me up to 27 pages:

Product.search('colt').page(27).records.size
#=> 9
Product.search('colt').page(28).records.size
#=> 0

That works fine.

However, if I try to change the pagination window size to 1, I can't step through the records correctly. I'd expect 659 pages of Products with this window setting (which is actually the total that is displayed in the view using Kaminari's helpers), but the records cut out on page 27 as if I were still using the default window size of 25:

Product.search('colt').page(27).per(1).records.size
#=> 1
Product.search('colt').page(28).per(1).records.size
#=> 0

In addition to the Kaminari view helpers displaying 659 total pages, it also selects page 26 when passed {page: 2} through the controller, page 51 when passed {page: 3}, etc.

Associations not bringing back results

HI @karmi

I have the following relationship

class Discussion < ActiveRecord::Base
  has_many :replies, as: :replyable, dependent: :destroy
  ...
  ...

  def as_indexed_json(options={})
    self.as_json( include: { replies: { only: :body }} )
  end
end

class Reply < ActiveRecord::Base
  belongs_to :discussion
end

We also have multiple indexes which you gave use a solution from #30

require 'elasticsearch/model'

class Find
  def self.search(q)
    response = Elasticsearch::Model.search(q)
    if response.response.empty?
      []
    else
      response.results.map { |r| r['_type'].classify.constantize.where(id: r['_id']).to_a }.flatten
    end
  end
end


class MultipleModels < Array
  def client
    Elasticsearch::Model.client
  end

  def ancestors
    []
  end

  def default_per_page
    100
  end
end

module Elasticsearch::Model
  # Search multiple models
  #
  def search(query_or_payload, models=[], options={})
    models = [Article, Series, Discussion]

    models = MultipleModels.new(models)

    index_names    = models.map { |c| c.index_name }
    document_types = models.map { |c| c.document_type }

    search   = Searching::SearchRequest.new(
      models,
      query_or_payload,
      {index: index_names, type: document_types}.merge(options)
    )

    Response::Response.new(models, search)
  end

  module_function :search
end

It appears we have it set up right

Discussion.first.__elasticsearch__.as_indexed_json
=> {"title"=>"Some title",
 "updated_at"=>Thu, 24 Oct 2013 14:20:09 UTC +00:00,
 "class_name"=>"discussion",
 "replies"=>[{"body"=>"<p>Apples</p>"}, {"body"=>"<p>Orange</p>"}]}

but searching for Apples or Oranges gives us no results.

Setting Sort Condition on Search?

Hi again!

I'm testing this gem, and one thing I would like to do is: control sort order of search results.

One complication though: sometimes I want to be able to sort on fields that are indexed in elasticsearch (and are searchable), but aren't actually columns in my database (this data is saved under rails' store feature).

Here's an example of roughly what I would like to be able to do, based on some Q+A found for Tire:

 a = ProjectMaterial.search("aardvarks*") do |s|
    s.sort {by :store_serialized_field, "asc"}
 end

Where :store_by_serialized_value is not a column in my db, but a field saved in a rails serialized store.

In testing, I notice that your gem indexes the store_by_serialized_field, so that these values are searchable. However, I'm not sure if elasticsearch-rails yet supports a sort option when performing a search query, and secondarily I'm not sure if it supports a sort option on field that is not a db column.

as_json only: [...attributs...], methods: [...methods list...] => callback update_document does not update "methods values" in ES

Hi,

I have a mapping with methods and attributes:
def as_indexed_json options={}
as_json only: ['id', 'banned','birthday'],
methods: ['method1', 'method2', methodn']
end

when I want to update a document, only changed attributes are saved in ES, not methods...
Looking at indexing.rb code, this seems normal:
def update_document(options={})
if changed_attributes = self.instance_variable_get(:@__changed_attributes)
client.update(
{ index: index_name,
type: document_type,
id: self.id,
body: { doc: changed_attributes } }.merge(options)
)
else
index_document(options)
end
end

so, how can I autosave "methods" values to ES when they are changedat the same time as attributes ?

elasticsearch models and STI

right now, overriding index_name and document_type won't work properly with rails STI
consider this example:

class Animal < ActiveRecord::Base
  document_type 'mammal'
  index_name 'mammals'
end

class Dog < Animal
end

Animal.document_type   # 'mammal'
Animal.index_name      # 'mammals'
Dog.document_type      # 'dog'
Dog.index_name         # 'dogs'

Since document_type and index_name are redefined in base class, i would expect the same to be reflected in derived classes also, but thats not happening. So, i have to manually redefine them in all the derived classes. Is this a bug?

Multi-type search

Elasticsearch has support for searching against multiple types in a single query. But the Response::Response, Response::Results, and Response::Records classes only support searching from a single model class. It would be nice if these classes could support multiple model classes.

It looks like this could be implemented by modifying Response::Records#records. Instead of including this method directly from the adapter like this, this could be implemented like:

# PSEUDOCODE
def records
  records = results.group_by(&:type).map do |type, results_for_type|
    # determine class and adapter for type ...
    # get records from adapter ...
  end.flatten

  # sort records so they are in the same order as results ...

  records
end

Unfortunately, it looks like this would involve significant, backward-incompatible changes to all of the classes listed above.

If this is a feature that you think is worthwhile, then please let me know and I may be able to work on this. Thanks.

Searching multiple indices/types (equivalent to "Tire.search")

Unfortunately, I've spent possibly too much time looking through this new gem looking for a top level search method. Is there a way to search without using a specific model? (Like Tire.search used to do)

For the most part, I'm using the base client methods to do my searching, however, I think I'd like to handle most of the response parts through this gem. (Such as Kaminari)

Possibly, is there a way to search via Elasticsearch::Model.client.search(search_hash) and use this gem's result formatting?

I'm partially new to this lower level of ES (coming from Tire) so hopefully you know what I'm looking for. Any help is much appreciated!!

Other than having somewhat of a learning curve, LOVING these new gems for ES

no error raised when elasticsearch is unavailable

Rails controller code:

def create
  ...
  if @something.save
    redirect_to some_url, notice: 'Something was successfully created.'
  else
    render action: "new"
  end
end

If mysql is unavailable the following error is raised:
Mysql2::Error ... Can't connect to local MySQL server
or if some other error ".save" returns false (like validations).

However, if elasticsearch is unavailable at the moment when a CRUD
action occurs no error is raised and ".save" returns true ... as if all went well.
Is there a common approach to handling this? To be consistent with rails,
it seems the gem should set an error (self.errors), or better, raise an error that can
be rescued or retried ... something that makes the action a single transaction (atomic).

question: How do I add more indexes for one active_record model?

I have to override as_indexed_json to define associations and can add my custom field which I need in the index.

But when I have a user class and need two indexes for this model? For example user and expert with different fields.

Do I have to create an expert class which inherited from user and define the index there?

Or is there an other solution, to search over two different indexes but with with one model?

Something like:
User.search(index: :expert, 'foo') or User.search(index: :user, 'foo')

Callbacks when updating associated record?

So in your example https://github.com/elasticsearch/elasticsearch-rails/blob/templates/elasticsearch-rails/lib/rails/templates/searchable.rb#L60 if Authors is updated, how would the original model's index get updated? Running into this issue locally.

How do I add synonyms to a model?

Hi all,

I am not sure how to do this:
I have an ActiveRecord that has some attributes indexed. I would like to add the synonyms for some of these fields but I am not sure how to do this.. should it be put inside the mappings block? is it even possible to do that with this gem?

Thank you in advance!

unable to get records if the table primary key is not id

I'm unable to fetch records from my legacy database when the primary_key has been changed.

It looks like the default method for fetching the records when using ActiveRecord is hardcoded to with "id" as the primary key field, and will fail to fetch records even if the models primary_key attribute has been set.

Example:

class Site < ActiveRecord::Base
    self.primary_key = 'siteid'
end

response = Site.search('Test')
response.records.first
# Fails with the following error -> ERROR:  column sites.id does not exist
# should be trying to use sites.siteid

Equivalent for Tire's index_prefix

I miss Tire's ability to set a global index_prefix, without adding index_name overrides in every model:

Tire::Model::Search.index_prefix "myapp_#{Rails.env}"

Import Data Faster

Is there a way to have this command run faster bundle exec rake environment elasticsearch:import:model CLASS='model' FORCE=yby including any table joins? My model has mapping object on it that includes another table currently.

Building a queries

Hello.

In our application we have a set of complicated queries with a lots of conditions, aggregations, etc. We have to construct huge dsl query hashes.

We tried to use JBuilder for jsoning (as recommended in README), but it turned up to bad practice because we need to do a lot of checking of query params presence, it's complicate to build arrays etc.

So, we wrote our own query builder which fits our needs better:

def build_query(options)
  QueryBuilder.build do
    filtered do
      _(:sort) do
        rank { order(:desc) }   
      end

      filter do
        term { name(options[:name]) }
        term { surname(options[:surname]) }
      end
    end
  end
end

build_query(name: 'Kimchi') # { filtered: { sort: [ { rank: { order: 'desc' } } ], filter: { term: { name: 'Kimchi' } } } }

The idea is simple: if deepest value of a node is blank (like surname term value in my example) - it does not render this node in the query.

build_query({}) # { filtered: { sort: [ ... ] } } 

Example above does not render filter section because all nested nodes has blank values in their deepest points.

What do you think about such way of building queries?

Plans for Gem Progression

First off: I think the work being done in this gem and in the elasticsearch-ruby gem is awesome!

I was wondering what the planned timeline is for getting this gem into a stable, release version?

Apologies if this is not the place for such a question.

question: Feature Extraction Pattern on multiple AR classes

I like the feature extraction pattern shown here: https://github.com/elasticsearch/elasticsearch-rails/tree/master/elasticsearch-model#feature-extraction-pattern

it looks like it is meant for one class? Does anyone have any suggestions or pointers on how that would work with multiple classes? Say each class has it's own mappings definition, search, as_indexed_json etc.. Would you do one concern per indexed class?

Overwriting of search method

Hi!

After update elasticsearch-rails form 46734cd to 88b6597 broken overwriting of search via

module Searchable
  extend ActiveSupport::Concern

  included do
    include Elasticsearch::Model

    mapping do
      # ...
    end

  end

  module ClassMethods
    def search(query)
      # ...
    end
  end

But it work with:

# In: app/models/concerns/searchable.rb
#
module Searchable
  extend ActiveSupport::Concern

  included do
    include Elasticsearch::Model

    mapping do
      # ...
    end

    def self.search(query)
      # ...
    end
  end
end

I think, problem with delegation and easy to update readme.

Search on attributes does not work

Hi again,

I updated the Rails / MongoDB proof of concept app to use elasticsearch-rails instad of ReTire. Unfortunately I still have trouble searching. Am I doing it wrong?

models/ad.rb

class Ad
  include Mongoid::Document
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  field :title, type: String
  field :body, type: String
end

✗ rails console
Loading development environment (Rails 4.0.0)
[2] pry(main)> Ad.import
NoMethodError: undefined method 'as_indexed_json' for #<Ad _id: 52d8e35d4d61724105010000, title: "martin", body: nil>
from /Users/martins/.rvm/gems/ruby-2.0.0-p353@ReTire/bundler/gems/elasticsearch-rails-04a637725dbf/elasticsearch-model/lib/elasticsearch/model/adapters/mongoid.rb:80:in `block in __find_in_batches'
[3] pry(main)> Ad.search('*').size
=> 10
[4] pry(main)> Ad.search('title:"martin"').size
=> 0
[5] pry(main)> Ad.where(title: 'martin').size
=> 1

Update callbacks do not use as_indexed_json

The update callbacks mine changed_attributes directly

def update_document(options={})
   if changed_attributes = self.instance_variable_get(:@__changed_attributes)
     client.update(
        { index: index_name,
          type:  document_type,
          id:    self.id,
          body:  { doc: changed_attributes } }.merge(options)
     )
  else
    index_document(options)
  end
end

This has 2 consequences for people with a custom as_indexed_json

• any indexed attributes that are not active model attributes are not indexed
• if as_indexed_json omits some attributes then they will be included when a record is updated in this way

Perhaps the documentation should highlight this?

question: Is there possible to get the result records ids as integer?

When i have a response and get response.redords.ids i get the ids array, but the ids are strings and i have to cast it.

Is there an other solution?

Testing with elasticsearch-rails

Hi partypeople,

I'm currently implementing elasticsearch in our application and wanted to write tests. A clean and convenient way of doing this, is setting a seperate index prefix while testing[1]. This worked with Tire but is it also working with this gem? If not, what is your best practise to do this?

Another question: How do you all test with this gem in general?

💖 Thanks, 💖

b

Is percolate working with elasticsearch-rails?

Is percolate working with elasticsearch-rails?

Similar to how the tire gem worked?

Ability to set default options for index/update/delete document method's

Hi!
@karmi What do you think about ability to set default options via DSL:
https://github.com/elasticsearch/elasticsearch-rails/blob/master/elasticsearch-model/lib/elasticsearch/model/indexing.rb#L287. I think it will be useful, especially for different env's (for example for test env I'll use refresh: true option in integration tests).

Conflict with ActiveSupport delegate ?

Hi @karmi

Here is my Article model with delegate defined.

Seems conflict with Elasticsearch::Model::Support::Forwardable

Model example

class Article << ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  delegate(:content, to: :cover_letter, prefix: true)

end

Trace

11:40:37 sidekiq.1 | wrong number of arguments (2 for 1)
11:40:37 sidekiq.1 | /Users/sam/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/forwardable.rb:225:in `single_delegate'
11:40:37 sidekiq.1 | /Users/sam/web/tulip/app/models/article.rb:224:in `<class:Article>'
11:40:37 sidekiq.1 | /Users/sam/web/tulip/app/models/article.rb:31:in `<top (required)>'
11:40:37 sidekiq.1 | /Users/sam/.rvm/gems/ruby-2.0.0-p353/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:469:in `load'
...
11:40:37 sidekiq.1 | /Users/sam/.rvm/gems/ruby-2.0.0-p353/bin/sidekiq:23:in `<main>'

Thanks!

Feedback on elasticsearch-model

Hi @karmi,

I wanted to give you some feedback about elasticsearch-model but didn't find a better place to post it.

This gem is really interesting, especially the "model" part of it.

I really like the decoupling between the class that holds the data, the one that does the request, and the one that represent the response and the results.

Still, there is something that bothers me ; there is a big assumption that "model" is something that is stored in another primary datastore than Elasticsearch, accessible through ActiveRecord, Mongoid or else.

I am working on an app where a lot of data is indexed in ES from an external data source. In the "main" app, I just want to query ES, fetch results and populate collections of results with data found in ES.

The objects I'm dealing with are very simple Ruby objects (_PORO_s) on which I can add some behavior, …

I guess I could use the various parts of Elasticsearch::Model with only what I want (excluding all the Record stuff) but it seems a little tedious and brittle.

Maybe, it would be just enough to have an example, include just what's needed to have those plain Ruby objects coming from ES.

Another question : in Result, how do you deal with script_fields (with a calculated distance, for example) when the rest of the _source is also present? I guess a basic merge would do the job.

Models not being indexed automatically

I'm not seeing the import callbacks being executed. When I save a model the data never hits Elasticsearch. I have to #import every time I create or update a model. I'm including all my search stuff as a module into my model. I'm using ActiveRecord.

module Search
  module BusinessIndexer
    extend ActiveSupport::Concern

    included do
      include Elasticsearch::Model
      include Elasticsearch::Model::Callbacks

      mappings do
        indexes :name
        indexes :name_suggest, type: 'completion', payloads: true
      end

      def as_indexed_json(options = {})
        {
          name_suggest: {
            input: [
              name,
              sanitize_input(name)
            ],
            output: name,
            payload: {
              id: id,
              name: name
            }
          }
        }
      end

      private
      def sanitize_input(input)
        input.gsub(/\Athe\s/i, '') unless input.nil?
      end
    end
  end
end

Index organization

Should an app use a single index by default? My impression is that since Elasticsearch supports multiple data types in an index, this maps well to Rails' structure. It would also simplify configuration, since the index name could be specified in a config file. Is there a reason not to do this?

Right now I'm accomplishing this with:

class UserSearch
  extend ActiveSupport::Concern
  included do
    include Elasticsearch::Model
    include Elasticsearch::Model::Callbacks
    index_name "my_app_name_#{Rails.env}"
    document_type "user"
    # ...
  end
end

response.records not working with rails 3.0.x

I was investigating using this gem in rails 3.0.20. It bundle installs fine, and I was able to perform a search of my data. However, when attempting to call the "records" method on the search response, I see the following error:

NoMethodError: undefined method exec_queries' for #<ActiveRecord::Relation:0x00000005715db8> from /home/tworker/work/ove-ruby/vendor/bundle/ruby/1.9.1/gems/activerecord-3.0.20/lib/active_record/relation.rb:374:inmethod_missing'

It looks like the "exec_queries" method is only available in rails 3.2.x. There is a "to_a" method available in 3.0.x which is very similar, but simply changing the gem to call that gets into a recursion issue. Is there a simple fix you can think of to allow this gem to be used in 3.0? I'll continue to investigate as well...thank you.

Setting Elasticsearch for production

Hi there I'm trying to deploy a toy app with this gem to heroku, and I didn't see any documentation how how to set the production environment url.

I'm sorry if I overlooked something, but any help would be much appreciated.

`require': cannot load such file -- elasticsearch/model (LoadError)

I'm receiving this error: 'require': cannot load such file -- elasticsearch/model (LoadError)

I have gem "elasticsearch-rails" bundled in my Gemfile and have this at the top of my Orders model.

require 'elasticsearch/model'

class Order < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

as_indexed_json not being called on save

No matter what settings I seem to use, the as_indexed_json method is only called when I do Model.import, but not on save

AM I missing something obvious in the documentation?

NULL

changes for ES 1.0 release

This works great with es-ruby 0.4 but it is locked to that version.
What could I change to work with es-ruby 1.0.0 ?

Infinite recursion caused by as_indexed_json

class Thing < ActiveRecord::Base
  include Elasticsearch::Model
  def as_indexed_json(options={})
    default = self.__elasticsearch__.as_indexed_json(options) # <--
    other = {
      # whatever
    }
    default.merge(other)
  end
end

Wanting to extend the default behavior (intuitive, DRY, OO) causes infinite recursion.

Soon I'll submit a pull request.

Facets included in this gem?

Are facets included in this gemfile, I can't find a reference to them.

ES as primary data store

Is it possible right now to use this gem to create rails model using only ES?

e.g:

class Article
  include Elasticsearch::Model
  mappings do
    indexes :title
    indexes :body
  end

end

a = Article.new title: "hello", body: "hello body"
a.save

If so, is there any documentation about this?

Thanks.

[RAILS] elasticsearch:import:all should be depend on :environment rake task to load models

The elasticsearch:import:all task should depend on the :environment task to load models.

the existing fallback to using require will still work, but avoids issues with classes within the models directory.

Faraday::TimeoutError

Frequently I see the following in the rails log:

[!!!] Error when refreshing the index: Faraday::TimeoutError
Operation timed out after 5001 milliseconds with 0 bytes received

The code is doing this:

Share.__elasticsearch__.refresh_index!

... so that any new/updated records are shown correctly on the list page.
If it matters, I do have the patron gem in the Gemfile.
Should I increase the timeout value, if so, how?
Suggestions ?

mapping not the same when import or created by .__elasticsearch__.create_index!

with code from today:

mappings dynamic: 'false' do
indexes :id, index: :not_analyzed
indexes :birthday, type: 'date'
indexes :certified, type: 'boolean'
indexes :first_name, type: 'string', index: :not_analyzed
indexes :es_connected_until, type: 'date'
indexes :es_friend_ids, type: 'integer'
indexes :es_location, type: 'geo_point', lat_lon: true
indexes :es_profil_photo_url, type: 'string', index: :not_analyzed
indexes :es_visible_until, type: 'date'
indexes :gender, type: 'byte'
indexes :ghost_mode, type: 'boolean'
indexes :hide_age, type: 'boolean'
indexes :is_banned_by_admin, type: 'boolean'
indexes :last_connection_at, type: 'date'
indexes :last_name, type: 'string', index: :not_analyzed
end

User.elasticsearch.create_index! result:
curl -X GET http://localhost:9200/users/user/_mapping?pretty
{
"users" : {
"mappings" : {
"user" : {
"dynamic" : "false",
"properties" : {
"birthday" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"certified" : {
"type" : "boolean"
},
"es_connected_until" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"es_friend_ids" : {
"type" : "integer"
},
"es_location" : {
"type" : "geo_point"
},
"es_profil_photo_url" : {
"type" : "string",
"index" : "not_analyzed"
},
"es_visible_until" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"first_name" : {
"type" : "string",
"index" : "not_analyzed"
},
"gender" : {
"type" : "byte"
},
"ghost_mode" : {
"type" : "boolean"
},
"hide_age" : {
"type" : "boolean"
},
"id" : {
"type" : "string",
"index" : "not_analyzed"
},
"is_banned_by_admin" : {
"type" : "boolean"
},
"last_connection_at" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"last_name" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}
}

when no mapping and I import:
{
"users" : {
"mappings" : {
"user" : {
"properties" : {
"birthday" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"es_connected_until" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"es_friend_ids" : {
"type" : "long"
},
"es_location" : {
"type" : "string"
},
"es_profil_photo_url" : {
"type" : "string"
},
"es_visible_until" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"first_name" : {
"type" : "string"
},
"gender" : {
"type" : "long"
},
"ghost_mode" : {
"type" : "boolean"
},
"hide_age" : {
"type" : "boolean"
},
"id" : {
"type" : "long"
},
"is_banned_by_admin" : {
"type" : "boolean"
},
"last_connection_at" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"last_name" : {
"type" : "string"
}
}
}
}
}
}

Charlie

Problems with kaminari

This works:

Article.search(query: 'Brazil').page(1).records

But this...

Article.search(query: 'Brazil').page(1).records.current_page

Throw this error:

Hash can't be coerced into Fixnum

Because of this error I can't make the kaminari pagination helper to work on my views, am I missing something?

ruby 2.0
rails 3.2
kaminari 0.14.1
kaminari-bootstrap 0.1.3
elasticsearch-rails 0.1.0
elasticsearch-model 0.1.0

Compatibility with the paginate helper of kaminari

Hi,

I should be handy to be compatible with the paginate helper from kaminari. For that, I think there is just 3 simple methods needed:

• current_page (the argument of the page method)
• total_pages (the number of results divided by the per page)
• limit_value (the per page count)

Searching multiple indexes

Hi @karmi

We are converting our application that is currently using Tire and we want to move it over to elasticsearch-rails. We have all the indexes all set but the problem is converting the search to multiple indexes.

Here was our Tire implementation

@search_items = Tire.search(['first_index', 'second_index', 'third_index', 'fourth_index'], {size: 20, load: true}) do |search|
  search.query{|q| q.string params[:q], default_operator: "AND" } if params[:q].present?
  search.filter :or, { :not => { :exists => { :field => :published} } },
   { :term => { :published => true } }
end

Looking at your Elasticsearch-model documentation it appears we can probably use this approach Elasticsearch::Model.client.search(q: 'Something') but I am not 100% sure that searches all the indexes. Even if it does it appears that the return value is not ruby objects so making use of the return is unclear to us.

how to disable the round robin behavior

I have a situation where most of my models index/update/delete their data in a
local db and node/cluster, but I have one model that needs to index/update/delete its data in a node/cluster located on another remote server.
Even though I specify the host+port for that one model, using

Share.__elasticsearch__.client = Elasticsearch::Client.new(host: "1.2.3.4:9500")

there are times when the data gets stored in the local node/cluster.
I suspect that "round robin" is involved.

Is there a way to disable the "round robin" behavior ?

Or, is there a way to force that one model to only "talk" to the remote node/cluster ?

I would like to keep the default model usage that saves/updates/deletes to both
the local db table and the remote elasticsearch index.

Equivalent to Tire :as

Is there an equivalent to :as when mapping models?

I'm doing the following:

mappings dyanmic: false do
  indexes :suggest, type: 'completion', index_analyzer: 'simple', search_analyzer: 'simple', as: 'name'
end

I'm recieving this error so I'm assuming it's not available in elasticsearch-model.

[!!!] Error when creating the index: Elasticsearch::Transport::Transport::Errors::BadRequest
[400] {"error":"MapperParsingException[mapping [business]]; nested: MapperParsingException[Unknown field [as]]; ","status":400}

Defined class methods like index_name are not used by other proxy methods

I'm not sure whether this is intentional or truly a bug, but I think at least the readme should be updated to make things a little clearer regarding the interaction between the __elasticsearch__ proxy and the classes that include it.

Similar to what's described in the readme, I don't include Elasticsearch::Model directly in my class, and use a separate module instead. Here's an example...

module Searchable
  def self.included(base)
    base.class_eval do
      include Elasticsearch::Model
      def self.index_name
        "foo"
      end
    end
  end
end

If I include that module in an ActiveRecord class like so:

class User < ActiveRecord::Base
  include Searchable
end

and call User.index_name, it correctly outputs "foo".

However, none of the other methods that are delegated to the __elasticsearch__ proxy end up using User.index_name. That means when I call User.search, it still searches against a users index.

To show this more clearly... I went in and monkey-patched the search method to drop in a debugger:

module Elasticsearch
  module Model
    module Searching
      module ClassMethods
        def search(query_or_payload, options={})
          binding.pry
          super
        end
      end
    end
  end
end

When User.search is called, the method is delegated to the __elasticsearch__ proxy. From within that debugger, you can see that self refers to the proxy object instead of the actual class, so calling self.index_name in the SearchRequest ends up never using the method defined on my User model.

Of course, I could just go and define the other methods that are passed to the proxy, but that kind of defeats the purpose of including the module to begin with.

Am I using the module correctly, and would it be worth modifying this behavior? If so, I could send a pull request. I think it'd be pretty straightforward... in the case of #search, you should just have to pass self.target to the SearchRequest instead of self. By doing so, if the method is not defined on the proxy's target, the method will get delegated back to the proxy... but if it is, it'll actually end up being used.

Connection Refused error

Sorry, I prefer to not post this here, but we're stuck on deployment and pretty desperate. After deployment, I'm trying to build the elasticsearch indexes via this comment:

bundle exec rake environment elasticsearch:import:model CLASS='User' FORCE=true RAILS_ENV=production

And it's producing this error:

[!!!] Error when deleting the index: Faraday::ConnectionFailed
Connection refused - connect(2)
[!!!] Error when creating the index: Faraday::ConnectionFailed
Connection refused - connect(2)
rake aborted!
Connection refused - connect(2)
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.1.31/Ruby/lib/patches/net_patches.rb:7:in `block in request_with_mini_profiler'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.1.31/Ruby/lib/mini_profiler/profiling_methods.rb:40:in `step'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.1.31/Ruby/lib/patches/net_patches.rb:6:in `request_with_mini_profiler'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:80:in `perform_request'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:39:in `call'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139:in `build_response'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/faraday-0.9.0/lib/faraday/connection.rb:377:in `run_request'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/base.rb:187:in `call'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/base.rb:187:in `perform_request'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/client.rb:102:in `perform_request'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/actions/bulk.rb:81:in `bulk'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-model/lib/elasticsearch/model/importing.rb:78:in `block in import'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-model/lib/elasticsearch/model/adapters/active_record.rb:88:in `block in __find_in_batches'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/activerecord-4.0.2/lib/active_record/relation/batches.rb:75:in `find_in_batches'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/activerecord-deprecated_finders-1.0.3/lib/active_record/deprecated_finders/relation.rb:70:in `find_in_batches'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/gems/activerecord-4.0.2/lib/active_record/querying.rb:8:in `find_in_batches'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-model/lib/elasticsearch/model/proxy.rb:80:in `method_missing'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-model/lib/elasticsearch/model/adapters/active_record.rb:86:in `__find_in_batches'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-model/lib/elasticsearch/model/importing.rb:77:in `import'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-model/lib/elasticsearch/model.rb:113:in `import'
/home/deployer/apps/controltec/shared/bundle/ruby/2.0.0/bundler/gems/elasticsearch-rails-527e2075ae81/elasticsearch-rails/lib/elasticsearch/rails/tasks/import.rb:59:in `block (3 levels) in <top (required)>'
Tasks: TOP => elasticsearch:import:model
(See full trace by running task with --trace)

After the failure, elasticsearch is shutdown, and does not respond to attempts to revive it. The database is hosted on AWS, and not on the local server. I'm not sure if this has anything to do with it.

sudo service elasticsearch status # elasticsearch is not running
sudo service elasticsearch start # Starting Elasticsearch
sudo service elasticsearch status # elasticsearch is not running

This is how I'm installing elasticsearch via a capistrano task.

    run "#{sudo} apt-get update"
    run "#{sudo} apt-get install openjdk-7-jre-headless -y"
    run "wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.deb"
    run "#{sudo} dpkg -i elasticsearch-1.0.1.deb"

I know this sounds like an elasticsearch issue, but I was hoping someone could offer some clues.

Use with Draper

I have been using retire, but saw the model integration this morning in this project.

After trying to get the new search method working with pagination I could not then wrap the returned "results" in a Draper::CollectionDecorator. I also couldn't even access the id of the results.first.id properly(nil). FYI I don't want to load the actual model out of the database.

Example object (results.first)

#<Elasticsearch::Model::Response::Result:0x007fa9ee879688
 @result=
  {"_index"=>"teams",
   "_type"=>"team",
   "_id"=>"51",
   "_score"=>0.99194,
   "_source"=>
    {"id"=>51,
     "created_at"=>"2014-03-11T04:31:35Z",
     "updated_at"=>"2014-03-11T04:31:36Z",
     "full_name"=>"test_Club, test_Team",
     "temporary_id"=>"1394512295.090692",
     "lock_version"=>2,
     "avatar"=>"/original/team/default_team.png",
     "verification_level"=>0,
     "owner_id"=>771,
     "name"=>"test_Team",
     "club"=>{"id"=>711, "name"=>"test_Club"}}}>

I have the collection decorator delegating to Kaminari pagination. But upon rendering (rabl) the only thing displayed is an empty set of items, but the pagination data is available.

I can map the source by doing results.map(&:_source) but I have then lost the pagination info.

My question is... Am I doing something very wrong here or should I wait a bit longer for further enhancements to this project before moving gems(retire=>this project).

Controller

    teams = Team.search(query).page(params[:page]).per(params[:per_page])
    @teams = TeamsDecorator.decorate(teams, with: TeamDecorator)

class TeamsDecorator < Draper::CollectionDecorator
  delegate :current_page, :per_page, :offset, :total_entries, :total_pages
end

module Searching::Team::Mapping

  def self.included(base)

    base.class_eval do
      mapping do
        indexes :id, type: 'integer'
        indexes :created_at, type: "date"
        indexes :updated_at, type: "date"
        indexes :name, type: "string"
        indexes :full_name, type: "string"
        indexes :avatar, type: "string"
        indexes :lock_version, type: "integer"
        indexes :owner_id, type: "integer"
        indexes :temporary_id, type: "string"
        indexes :verification_level, type: "integer"
        indexes :club, type: "object",
          properties: {
            id: { type: "integer" },
            name: { type: "string" }
          }
        indexes :avatar, type: "string"
      end

    end

  end

end

class TeamDecorator < Draper::Decorator
  delegate_all

  def to_indexed_json
    {
      _id: id,
      created_at: created_at,
      updated_at: updated_at,
      full_name: full_name,
      temporary_id: temporary_id,
      lock_version: lock_version,
      avatar: avatar.url(:thumb),
      verification_level: verification_level,
      owner_id: owner_id,
      name: name,
      club: format_club,
      avatar: avatar.url,
    }.to_json
  end

  def format_club
    {
      id: group.try(:id),
      name: group.try(:name),
    }
  end

  def sync_title
    self.full_name
  end

  def sync_subtitle
    self.club.name
  end

end

Thanks for the help and I do hope this question is not to verbose and attempts to highlight possible implementation issues for other first time users :)

Thanks Dave

Error generating template

After running this command

rails new searchapp --skip --skip-bundle --template https://raw.github.com/elasticsearch/elasticsearch-rails/master/elasticsearch-rails/lib/rails/templates/01-basic.rb

I receive this error. Is this to be ignored?

The template [https://raw.github.com/elasticsearch/elasticsearch-rails/master/elasticsearch-rails/lib/rails/templates/01-basic.rb] could not be loaded. Error: cannot load such file -- elasticsearch