Import Data Faster about elasticsearch-rails

[karmi]

To perform the join or limit the scope of the import method, just write your own importing routine, using find_in_batches, something like:

errors       = 0

Article.where("published_on > 2013-12-01").find_in_batches(batch_size: 100) do |articles|
  response = Article.__elasticsearch__.client.bulk \
    index:   Article.index_name,
    type:    Article.document_type,
    body:    articles.map { |a| { index: { _id: a.id, data: a.__elasticsearch__.as_indexed_json } } }

  errors += response['items'].map { |k, v| k.values.first['error'] }.compact.length
end

It would make sense to add an import method to the ActiveRecord::Relation / Mongoid::Collection classes, so you'd simply call Article.where("published_on > 2013-12-01"). I'll keep that in mind.

[AaronRustad]

@bcackerman the ability to specify ActiveRecord scope on the Rake import task has recently been added in #104.

[bcackerman]

Is there a way to use force: true with the version @karmi posted?

[karmi]

@bcackerman What do you mean by using the force option in this context? As @aackerman says (thanks!), you can specify the scope in the environment variable for the Rake task, and then just use FORCE=y.

If you wanna manage it yourself, then the create_index! method accepts a force option -- please see the RDoc.

[netwire88]

Hi @karmi, has any user been able to replicate your bulk import example with Mongoid instead of ActiveRecord? I search all over the net but since Mongoid doesn't have find_in_batches, I'm not sure how to implement your example above.

[karmi]

@netwire88 It should work exactly the same for Mongoid -- there's an adapter for Mongoid shipped.

[karmi]

@netwire88 I mean, if you wanna write a custom importing code like that, just use the appropriate Mongoid methods, see eg. this comment, https://github.com/mongoid/mongoid/issues/1334#issuecomment-2419621

[netwire88]

Thanks @karmi, does the default import method that comes with elasticsearch rails use bulk import?