mongoid / mongoid_fulltext

An n-gram-based full-text search implementation for the Mongoid ODM.

Mongoid

This is the legacy fork which is no longer maintained. The official repository is now under the MongoDB organization here.

mongoid_fulltext's Issues

undefined method `db' for #<Moped::Collection

Getting below error while doing...
rake db:mongoid:create_indexes

rake aborted!
undefined method db' for #<Moped::Collection:0x007fc04fc9ad08> gems/mongoid_fulltext-0.5.8/lib/mongoid_fulltext.rb:71:infulltext_search_ensure_indexes'

Mongoid Edge/6 and Mongo v2 driver compatibility

It seems mongoid_fulltext is no longer compatible with the most recent mongoid version. Problem come from Mongo v2 Driver which no longer provides a remove_all interface.
Might want to follow the track of https://github.com/DatabaseCleaner/database_cleaner/issues/378

NoMethodError: undefined method `remove_all' for #<Mongo::Collection::View:0x0000000acd5a28>
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongoid_fulltext-0.7.1/lib/mongoid/full_text_search.rb:334:in `block in update_ngram_index'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongoid_fulltext-0.7.1/lib/mongoid/full_text_search.rb:319:in `each_pair'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongoid_fulltext-0.7.1/lib/mongoid/full_text_search.rb:319:in `update_ngram_index'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/bundler/gems/mongoid-7c2ff52a0c52/lib/mongoid/contextual/mongo.rb:687:in `yield_document'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/bundler/gems/mongoid-7c2ff52a0c52/lib/mongoid/contextual/mongo.rb:124:in `block in each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:45:in `block in each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongo-2.2.5/lib/mongo/cursor.rb:85:in `block in each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongo-2.2.5/lib/mongo/cursor.rb:85:in `each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongo-2.2.5/lib/mongo/cursor.rb:85:in `each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:44:in `each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/bundler/gems/mongoid-7c2ff52a0c52/lib/mongoid/query_cache.rb:208:in `each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/bundler/gems/mongoid-7c2ff52a0c52/lib/mongoid/contextual/mongo.rb:123:in `each'
        from C:in `each'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/mongoid_fulltext-0.7.1/lib/mongoid/full_text_search.rb:294:in `update_ngram_index'
        from (irb):1
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc1/lib/rails/commands/console.rb:65:in `start'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc1/lib/rails/commands/console_helper.rb:9:in `start'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:78:in `console'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
        from C:/Dev/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc1/lib/rails/commands.rb:18:in `<top (required)>'

Add support for Ruby 2.4.0

Seems non-trivial. Even with this:

  gem 'mongo', github: 'mongodb/mongo-ruby-driver' if RUBY_VERSION >= '2.4.0'
  gem 'bson', github: 'mongodb/bson-ruby' if RUBY_VERSION >= '2.4.0'

      TypeError:
        wrong argument type Integer (expected Proc)
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection.rb:193:in `block in write'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection.rb:192:in `each'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection.rb:192:in `write'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection.rb:162:in `deliver'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection.rb:108:in `block in dispatch'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/monitoring/publishable.rb:47:in `publish_command'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection.rb:107:in `dispatch'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/operation/executable.rb:37:in `block in execute'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server/connection_pool.rb:107:in `with_connection'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/server.rb:242:in `with_connection'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/operation/executable.rb:35:in `execute'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/operation/commands/collections_info.rb:51:in `execute'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/database/view.rb:114:in `send_initial_query'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/database/view.rb:95:in `collections_info'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/database/view.rb:55:in `collection_names'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/database.rb:116:in `collection_names'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/bundler/gems/mongo-ruby-driver-dd19e48bbb72/lib/mongo/database.rb:140:in `collections'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/gems/mongoid-6.0.3/lib/mongoid/config.rb:175:in `purge!'
      # /Users/dblock/.rvm/gems/ruby-2.4.0/gems/mongoid-6.0.3/lib/mongoid.rb:99:in `purge!'
      # ./spec/spec_helper.rb:20:in `block (2 levels) in <top (required)>'

Can't search array fields

I have an array field that stores an array of tags. I would like to search the tags along with my other fulltext fields.

When I do

   fulltext_search_in :tags

I get
NoMethodError: undefined methodencoding' for ["ghost", "accounting"]:Array`
...when trying to update the fulltext indexes.

Thanks for a great gem! I will contribute if I can figure out how to fix this.

Full search term match scoring

Here's a situation that we are experiencing after integration...

We have indexed roughly 3 million companies by name and domain.

Searching for "Tesla" produces results for lots of companies with "Tes" as the first word, but does not include the company "Tesla" even though that full word ngram can be found in our search index. We assumed that because the full search term could be matched to an ngram, that result would be scored the highest.

Does that sound right/wrong? Is there a configuration option that we missed?

Cannot call `remove_from_ngram_index` on collection

The readme suggests it's possible to call Artwork.remove_from_ngram_index

But on ruby 2.3.3 and mongoid v6 it raises

NoMethodError: undefined method `mongo_v2?' for Artwork::Class

I believe there is a context issue in the module that defines that method. The mongo_v2 is indeed defined in some of this gem's module but is not called appropriately. I believe some similar method should be provided by the Mongoid Compatibility gem and should be called directly.

NameError: uninitialized constant Mongoid::FullTextSearch::ClassMethods::Mongo

Hi,
Where did the Mongo::DESCENDING come from? I think it should be defined in Mongo, but i did not see you required this gem in your gemspec. This error occurred in 0.5.8. Would you please have a look ? Thanks!

Combining fulltext_search with other criteria?

I don't index large articles where relevance is the only concern in search results. Therefore I eventually reach a point where I need to order and/or filter the results using other mongoid criteria.

For example finding Articles matching the fulltext term "foo" but also was published in the last week. Or displaying the results in a table and wanting to sort (asc and desc) on different table columns like title, author or published timestamp.

I haven't figured out if or how to work these things into the filters options in mongoid_fulltext. Maybe it is possible. But they seem to require filters that can be "rendered" and index-time.

Limiting with other criteria when calling .fulltext_search() is possible and do work... sort of. They just throw exceptions whenever something is actually filtered and an id is not found. (Thanks to Mongoid 3's new behavior).

So, finally a question or two

Would it be possible for mongoid_fulltext to not use find(foo) but instead use .where('_id' => foo) without breaking other behaviors?

Incorrect step size when string is longer than the number of n-grams desired

In https://github.com/artsy/mongoid_fulltext/blob/master/lib/mongoid_fulltext.rb:

# Figure out how many ngrams to extract from the string. If we can't afford to extract all ngrams,
# step over the string in evenly spaced strides to extract ngrams. For example, to extract 3 3-letter
# ngrams from 'abcdefghijk', we'd want to extract 'abc', 'efg', and 'ijk'.
if bound_number_returned
   step_size = [((filtered_str.length - config[:ngram_width]).to_f / config[:max_ngrams_to_search]).ceil, 1].max
else
   step_size = 1
end

If we want to get 3 n-grams: abc, efg and ijk from abcdefghijk (11) we need a step of 4, not 3.

(11.to_f - 3) / 3 = 2.6, ceil to 3

I think this needs to not do - config[:ngram_width].

However, I wonder whether the comment is incorrect and we want the first 3 n-grams instead of skipping characters.

cc: @aaw

Not support for non-English Languages?

It failed to search for Chinese by using this great gem. Should it be improved to support searching multiple languages?

Move into the mongoid organization?

Should we donate this to https://github.com/mongoid?

[Feature] Weighted/Prioritised search fields

This is only conceptual, I don't know how this would be executed, but I want to share this idea anyway. In my app I would like to prioritise patterns matching in certain fields and weight the score dependent on that.

For example patterns matching in a title are usually more important than patterns matching in a description. This is how I would like to use the options:

class Artwork

  include Mongoid::Document
  include Mongoid::FullTextSearch

  field :title
  field :description

  fulltext_search_in :title, :description,
                     :priorities => { :title => 10, :description => 5 }

end

Is this even possible to emulate without a massive performance loss?

Indexing with timeout disabled gives an ArgumentError Exception

Following the documentation I am able to build an index using Artwork.update_ngram_index; however, some of the datasets I wish to query require the disablement of timeout.

Artwork.update_ngram_index(timeout: false)

Returns the following error:

(byebug) Artwork.update_ngram_index(timeout: false)
*** ArgumentError Exception: wrong number of arguments (given 1, expected 0)

I notice two declarations of update_ngram_index which could be where my issues stem from.
The first doesn't appear to have been updated to take in arguments and is what my code seems to be invoking.

The second was updated to allow arguments, but doesn't seem to get invoked by my code.

I believe to have worked around this issue by directly calling Artwork.each(&:update_ngram_index) though this could be an issue with my own setup if others are not experiencing this.

Bulk indexing - Saving a model with a long indexable name triggers as many queries as the number of ngrams

The update code triggers one insert per ngram

This leads to as many insert requests as there are ngrams to be indexed. This becomes terribly slow when doing batch updates, (especially if one forgets to implement the update_if option)

I suggest the following

• Use an implementation of batch update to add all the ngrams in a single request
• (might be a an idea for a new feature) use a default implementation of default_if based on dirty tracking (if the indexed fields are Mongoid fields, then it's possible to use dirty tracking with _changed? methods to know if an update is needed

Support for Mongoid localized fields

Is it possible to use a fulltext search on a translated field ? Knowing the fields behaves like a hash, it it possible to either

• 1: enable lookup for all languages at the same time,
• 2: given a language (ie I18n.locale), autocomplete only for that specific language ?

Release v. Next

@tomasc any interest in helping out? If yes, wha's your rubygems email?

Release 0.6.1

I don't see any barrier to release 0.6.1. gemspec is prepared and tests are green with both 3.0 and 3.1 versions.

https://travis-ci.org/artsy/mongoid_fulltext/builds/6020196

/cc @dblock

Creating indexes from the console with an empty DB crashes with "ns does not exist"

Context : starting a mongoid container/pod on kubernetes and trying to seed the database with

::Mongoid::Tasks::Database.create_indexes
# or rails db:create_indexes

I am not sure whether this is due to a Mongoid update or something else I am missing.
Backtrace

rails aborted!
Mongo::Error::OperationFailure: ns does not exist: my_db_production.mongoid_fulltext.index_collection_name_0 (26) (on mongodb.staging.svc.cluster.local, modern retry, attempt 1)
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/result.rb:343:in `raise_operation_failure'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/indexes/result.rb:86:in `validate!'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:29:in `block (3 levels) in validate_result'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:96:in `add_server_diagnostics'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:28:in `block (2 levels) in validate_result'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:43:in `add_error_labels'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:27:in `block in validate_result'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:82:in `unpin_maybe'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:26:in `validate_result'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/executable.rb:46:in `block in execute'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/executable.rb:45:in `tap'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/executable.rb:45:in `execute'
/usr/local/bundle/gems/mongo-2.14.0/lib/mongo/operation/shared/op_msg_or_list_indexes_command.rb:29:in `block 

The faulty lines of code (School collection is empty)

School.all.each(&:save)
::School.create_indexes

Return the `highlight` on search

If you are indexing text, by example, with ElasticSearch, you can get a portion of the text where the search was found, using highlight, so that you can highlight content matching the query.
Would be awesome if mongoid_fulltext could do the same.
Is it possible?

Cursor timing out

I'm trying to generate an ngram index for a large number of short phrases, around 3,000,000.

I'm getting the following error a few minutes after kicking off Model.update_ngram_index

 Mongo::Error::OperationFailure (cursor id 331936703090 not found (43) ...`

Is there a config option I can pass to disable cursor timeouts? Any other recommendations?

Rails4, Mongoid4 support?

Hi,
Im using Rails4 and getting this error.
Uninitialized constant Mongoid::Indexes (NameError)

are you planning for Rails4, Mongoid4 support?

Incompatible with Mongoid 5

In Indexes/Indexable, alias_method is used before create_indexes is defined, so there goes the first error.

After I move alias_method to the bottom, undefined method error is gone, but index_specification method on each model is lost for some reason, so indexes can't be created.

create_indexes without mongoid_fulltext works.