collectiveidea / delayed_job_mongoid

Mongoid backend for delayed_job

DelayedJob Mongoid backend

Requirements

Delayed Job Mongoid supports the following dependency versions:

• Ruby 2.3+
• Mongoid 5+

For Mongoid 3 and 4, use 2.x version of this gem.

Installation

Add the gem to your Gemfile:

gem 'delayed_job_mongoid'

Then create the indexes (don't forget to do this in production):

script/rails runner 'Delayed::Backend::Mongoid::Job.create_indexes'

To generate script/delayed_job:

rails generate delayed_job

That's it. Use delayed_job as normal.

Advanced Usage

Silencing Noisy Logging

By default, the Mongo Ruby Driver will print log lines every 5 seconds as Delayed Job polls your database. Use the following plugin to silence logs which happen while polling (except error level) and only log operations which happen during job execution.

# in initializers/delayed_job.rb

require 'delayed/plugins/mongo_silencer'
Delayed::Worker.plugins << Delayed::Plugins::MongoSilencer

To ensure thread safety, this plugin requires your Mongo::Logger.logger to be an instance of ActiveSupport::Logger class. It will ignore instances of the Ruby standard library Logger class.

Customizing Job Class

Delayed Job allows specifying a custom backend class for jobs. For example, you may wish to specify a different database collection name. To do this:

# in an initializer

class MyJob
  include ::Mongoid::Document

  store_in(collection: :my_jobs)

  include ::Delayed::Backend::Mongoid::Mixin
end

Delayed::Worker.backend = MyJob

# in your application

MyJob.enqueue(job_object)

delayed_job_mongoid's Issues

A branch for mongoid ~> 2.4

It would be nice, if there was a branch for delayed_job_mongoid 1.0.x which would support mongoid ~> 2.4.

I have some fix that I would be glad to submit as pull request.

Worker crashing when there is no job in the table

I'm using mongoid 2.1.4 with mongodb 2.0.0.

Running :
$ rake --trace jobs:work

The worker crashes with following issue :

You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.[]
~/.gems/ruby/1.9.1/gems/delayed_job_mongoid-1.0.4/lib/delayed/backend/mongoid.rb:60:in `reserve'

I have looked at the issue in the console and it appears that variable result is just set to nil (after the find_and_modify call).

Release a mongoid5-compatible version please

Please release a new gem version

According to https://rubygems.org/gems/delayed_job_mongoid, last release was 1.5 years ago:

2.0.0 August 2, 2012

Support for Mongoid 4.0.0

Bundler could not find compatible versions for gem "activesupport":

In Gemfile:
  delayed_job_mongoid (>= 0) ruby depends on
    activesupport (~> 3.0) ruby

  rails (~> 4.0.0.beta) ruby depends on
    activesupport (4.0.0.beta1)

Bundler could not find compatible versions for gem "mongoid":

In Gemfile:
  delayed_job_mongoid (>= 0) ruby depends on
    mongoid (~> 2.0) ruby

  mongoid (4.0.0)

Incremental memory consumption

System:

• delayed_job (4.1.5)
• delayed_job_mongoid (2.3.0)
• mongoid (6.3.0)

Problem

After some time delayed_job process is taking all memory in development environment.

It seems that problem ocurrs only when we set preload_models: true into mongoid configuration file.

It could be related with this issues of delayed_job:

• collectiveidea/delayed_job#776
• collectiveidea/delayed_job#823

Please release new gem version

delayed_job_mongoid is not writing data in backend

I have mongoid model
class RequestResponse
include Mongoid::Document
field :body, type: String
field :job_id, type: Fixnum
field :completed, type: Boolean
end
I have a class in my lib folder

class MyJob < Struct.new(:session, :url, :r_id)
def perform
rr = RequestResponse.find(r_id)
session = YAML.load session
rr.body = session.get(url).body
rr.completed = true
rr.save
end
end
I called some where in my controller

rr = RequestResponse.new
rr.save
Delayed::Job.enqueue(MyJob.new(session.to_yaml, url, rr.id),3)

I can see

rake jobs:work
1 jobs processed at 19.3392 j/s, 0 failed ...

And the result is not stored in the table for rr if i check
rr.body
it is still nill any one can help me out Thanks in advance

has_many in a model –> to_yaml error

After some gems updates I got such an issue:

[72] pry(main)> b=Block.create
=> #<Block _id: 50f6f5150b002dd0cb000011, created_at: 2013-01-16 18:44:37 UTC, updated_at: 2013-01-16 18:44:37 UTC, x: 0, y: 0, title: "", scope: :for_one, parent_id: nil, game_id: nil, task_id: nil, _type: "Block">
[73] pry(main)> b.children.to_yaml
TypeError: can't dump anonymous class Class
from /Users/ulitiy/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/syck/rubytypes.rb:6:in `to_yaml'
[74] pry(main)> b.children.class
=> Mongoid::Relations::Targets::Enumerable
[75] pry(main)> b.delay.title
TypeError: can't dump anonymous class Class

So, if I have has_many in my model, I get can't dump anonymous class Class exception when try to make delayed method call. I don't know why that happens, but deleting the row with has_many make dj work. 3 days trying to understand that issue...

Update Multiple Records with help of Single Delayed Job.

I am new to delayed_job.
Here is my code which is executed on every update:

Delayed::Job.enqueue(Incrementor.new(ids_array), 0 , 1.minute.from_now.getutc)

And the code within incrementor.rb is:

class Incrementor < Struct.new(:array)
def perform
array.each{ |p| perform_update(p) }
end
def perform_update(id)

find and update object

end

Can we do like this ?

Not working with Rails 5

When I run the rails jobs:work command I get the error:
Error while reserving job: undefined method 'find_and_modify'

Is this gem not ready for Rails 5 or is there some other issue?

Failed query during deserialization should not raise Delayed::DeserializationError

In this file:

https://github.com/collectiveidea/delayed_job_mongoid/blob/master/lib/delayed/serialization/mongoid.rb

You can see self.yaml_new raises Delayed::DeserializationError

This is not good when working with replica sets. In rare cases, when triggering a delayed job (e.g. a mailer) on new object creation, the job will be invoked and query a replica set member BEFORE the data has replicated, raising a Mongoid::Errors::DocumentNotFound which is re-raised as a Delayed::DeserializationError

Delayed::DeserializationError is NOT the appropriate error here, since it implies that the handler itself is malformed, and hence no matter how many times we try to deserialize, we will never succeed. DJ handles it by failing the job immediately without retrying. In the case of a failed query however, if we wait ~1 second we WILL succeed.

My proposal is to simply raise the Mongoid::Errors::DocumentNotFound and let the job retry X times.

Don't treat every mongoid object as root object

delayed_job fails with the following exception:

Access to the collection for Image is not allowed since it is an embedded document, please access a collection from the root document.

class User
  include Mongoid::Document
  embeds_one :image, :inverse_of => :user
end

class Image
  include Mongoid::Document
  embedded_in :user, :inverse_of => :image

  mount_uploader :asset, ImageUploader, :validate_integrity => true, :validate_processing => true
end

ImageUploader is a carrierwave uploader. I want to handle removing an asset using delayed_job.

If I do image.remove_asset! it works fine, but if I do image.delay.remove_asset! it doesn't work. I have also tried to do handle_asynchronously(:remove_asset!), but that doesn't help as well.

Problem seems in line:6 of lib/delayed/serialization/mongoid.rb, where the code is klass.find(val['attributes']['_id']). It tries to load the object from db, but the object is an embedded object so it can't be done and fails with the exception.

Exception: DeserializationError

Currently running into Delayed::DeserializationErrors exceptions. The details has been posted to Google Groups. http://groups.google.com/group/delayed_job/browse_thread/thread/9d28383df677b45e

It seems to be an issue with YAML.

I've been running through different cases to try and debug what is wrong.

• Instance method (left empty)
• Tried not passing any variables to the delayed instance method
• Tried object from Mongoid that do not contain any sub collections
• Tried un-installing and re-installing both delayed_job_mongoid + delayed_job gems + mongoid
• Running latest versions of gems Using delayed_job (2.1.1), Using delayed_job_mongoid (1.0.0), Using mongoid (2.0.0.beta.20)

Find job fails when :first and :condition is used

Hi - I ran into a problem using cedar stack on heroku. The find method failed with Mongoid::Errors::DocumentNotFound: Document not found for class Delayed::Backend::Mongoid::Job with id(s) [:first, {:conditions=>{:_id=>"4f1d8ab08fc8ea000100000b"}}].

I'm using these gems:
rails (3.1.3)
delayed_job_mongoid (1.0.8)
delayed_job (~> 3.0.0)
mongoid (>= 2.0)

This change fixed the issue though I'm probably not seeing some important conflict this change may have
christianhager@42c984f

Should the index spec be updated to match the current #reserve method?

It was updated a few years ago, but it seems that it's drifted some. For example, the index doesn't contain any knowledge of failed_at.

Add an index for `queue`

When using queues, things can get slow: findAndModify locks the table while a BasicCursor goes through each row. Ensuring a {queue: 1} index exists clears things up.

It time to update .gemspec for mongoid 2.0.0

Mongoid 2.0.0 has release.

https://rubygems.org/gems/mongoid/versions/2.0.0

Postgres Deadlock

Hello,

I have been using delayed_job for many years in my app and I couldn't be happier with the robustness I have seen.
Since a few days though I'm seeing Postgres deadlock error from time to time.
I have multiple processes working on the delayed jobs, but I don't think that should be a problem.

Error while reserving job: PG::TRDeadlockDetected: ERROR: deadlock detected
and

STATEMENT: UPDATE "delayed_jobs" SET locked_at = '2023-01-24 23:17:34.600181', locked_by = 'host:bfa497____c5 pid:27' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE (((run_at <= '2023-01-24 23:17:34.599794' AND (locked_at IS NULL OR locked_at < '2023-01-24 23:17:04.599811')) OR locked_by = 'host:bfa497______9c5 pid:27') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *

I'm a bit lost where to start debugging this. I have already done a Postgres vacuum since I read in a forum that it might help, but it didn't.

Any help is highly appreciated!

Redesign the findAndModify query and/or indexes to be more performant

Right now I have roughly 200 workers and a job table of ~7000 jobs (not many jobs, but they're long-running).

The findAndModify as it is today, with the indexes prescribed plus another for queue, is relatively slow and when you have many workers the write lock used by findAndModify brings Mongodb to its knees (which in turn makes everything glacially slower). Currently, it just uses a BasicCursor and scans the whole job table (at least if you use named queues).

The answer to this StackOverflow question is insightful though it doesn't contain any explicit suggestions.

Can this be rectified by adding or adjusting indexes?

Mongoid change

I believe...

::Mongoid.master.connection.connect_to_master

...should be changed to:

::Mongoid.master.connection.connect

.delay doesn't work

Having a problem using .delay on my mailers... I'm using dmitrydims fork for Rails 4 and Mongoid 4. Anyone else having the same problem?

Frozen Hash in Heroku

I'm getting a Frozen Hash error after the job is completed (the job is completed correctly).

Here the exception:

rake aborted!
can't modify frozen Hash
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/attributes.rb:190:in `[]='
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/attributes.rb:190:in `block in write_attribute'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/threaded/lifecycle.rb:26:in `_assigning'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/attributes.rb:180:in `write_attribute'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-3.1.4/lib/mongoid/fields.rb:437:in `block (2 levels) in create_field_setter'
/app/vendor/bundle/ruby/1.9.1/gems/delayed_job-3.0.5/lib/delayed/worker.rb:250:in `handle_failed_job'
/app/vendor/bundle/ruby/1.9.1/gems/delayed_job-3.0.5/lib/delayed/worker.rb:212:in `block in run'

• in worker.rb:212 (maybe in Job.destroy?) An Exception is rescued
• in worker.rb:250 when it executes handle_failed_job, the rake task crashes when it sets the field last_error

In local I don't have this problem.

For heroku I am using mongohq with dbversion 2.2.6 (I get the same problem also with version 2.4.6).

My delayed_job versions are:

delayed_job (3.0.5)
delayed_job_mongoid (2.0.0)

Any clue?

Thanks

Moped not getting correct config on Heroku with Mongolab

I have delayed_job_mongoid setup on a Rails app on Heroku running unicorn. I'm using Mongolab and have my mongoid.yml file set up as recommended. The app itself connects correctly to the Mongolab hosted db however the jobs:work task is never able to connect to the db. Upon reviewing the logs it appears it is trying to connect to a local db (using some sort of default connection settings) instead of using the connection settings specified in my mongoid.yml file. I've set similar apps up before with the same Heroku/Mongolab setup and not run into this problem. Any idea what I'm missing?

The exact error that I'm seeing in the logs is:

app[worker.1]: Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set #<Moped::Cluster:.. @seeds=[<Moped::Node resolved_address="127.0.0.1:27017">]>

Worker doesn't pickup jobs created by a mongoid embedded document

As the title says, you can't use the handle_asynchronously option inside an object that will be persisted as a subdocument.

In the example below, every time you create a comment, the job will be created, but when the worker picks it up for execution, nothing will happen because calling comment.find(:id) will return nil

class Post
  include Mongoid::Document
  embeds_many :comments
end

class Comment
  include Mongoid::Document
  embedded_in :post

  after_create :notify_post_author

  def notify_post_author
    Notification.create(user: post.user, text: 'you got a comment')
  end

  handle_asynchronously :notify_post_author, queue: 'notifications'
end

This is referenced in Issue #11 however it leads to another gem rather than a fix, so I'm curious as to why this wasn't addressed by this gem

MySQL server has gone away

We're using Delayed Job on Mongo for queue processing, but our workers have to interact with a MySQL server to complete our jobs.

DJ works perfectly when run through the rake tasks, but when daemonized using:

script/delayed_job start

the worker starts in the background but as soon as its up and pulls its first job, we get a 'MySQL server has gone away' error. It happens every time and only when using the daemonizer.

Any suggestions?

Problem deploying to heroku using delayed_job_mongoid

From heroku support staff:

"I've taken a look and it seems to be a problem with delayed_job 3.0.0.pre.3's gemspec. This is the gemspec we're getting from rubygems.org: https://gist.github.com/1ff12f33003b44870e59

Can you reach out to the authors and ask them to repush using ruby 1.8.7. Using ruby 1.9.x seems to cause issues. We've seen this happen before."

Rails 4.1 compatability

Just tried upgrading to Rails 4.1 and having dependency issues.

Bundler could not find compatible versions for gem "activesupport":
  In Gemfile:
    delayed_job_mongoid (>= 0) ruby depends on
      delayed_job (< 5, >= 3.0) ruby depends on
        activesupport (< 4.1, >= 3.0) ruby

    rails (~> 4.1.1) ruby depends on
      railties (= 4.1.1) ruby depends on
        activesupport (4.1.1)

Unable to use

I'm trying to use this, but get an error when I have jobs in the queue and do "rake jobs:work" : Job failed to load: undefined method `keys' for nil:NilClass. Also, this doesn't look right to me:

ree-1.8.7-2010.02 > Delayed::Job.all
=> #<Mongoid::Criteria:0x1075a1650 @klass=Delayed::Backend::Mongoid::Job, @documents=[], @selector={}, @options={}>

That happens even after I add a job.

Exception: DeserializationError

Currently running into Delayed::DeserializationErrors exceptions. The details has been posted to Google Groups. http://groups.google.com/group/delayed_job/browse_thread/thread/9d28383df677b45e

I've been running through different cases to try and debug what is wrong.

• Instance method (left empty)
• Tried not passing any variables to the delayed instance method
• Tried object from Mongoid that do not contain any sub collections
• Tried un-installing and re-installing both delayed_job_mongoid + delayed_job gems + mongoid
• Running latest versions of gems Using delayed_job (2.1.1), Using delayed_job_mongoid (1.0.0), Using mongoid (2.0.0.beta.20)

Looking at where the exception is being raised in
lib/delayed/serialization/mongoid.rb

def self.yaml_new(klass, tag, val)
  begin
    klass.find(val['attributes']['_id'])
  rescue Mongoid::Errors::DocumentNotFound
    raise Delayed::DeserializationError
  end
end

Not sure why the document isn't being found, could it be the find function expects a string than the BSON object id? (I have no idea, I'm just trying to be useful).

<Note: Re-created issue as accidentally closed the other one>

Disable logging of job polling queries

Is there a way to disable the MOPED logs from being added to the development.log or production.log file every time delayed job polls? It adds a lot of noise to the log files and makes using tail -f on them really annoying, as every 5 seconds you get a MOPED query checking for new jobs...

Issue with Mongoid Inheritance

There seems to be an issue with Mongoid Embedded 1-N (One to Many) Inheritance Classes (http://mongoid.org/docs/documents/inheritance.html) and Delayed_Job.

E.G

class User
  embeds_many :subscriptions

    def delay_test
      self.subscriptions.create!({},SubscriptionChild)
    end
end


class Subscription
  include Mongoid::Document

  embedded_in :user
end

class SubscriptionChild < Subscription
  # field goes here
end

Calling User.first.delay.delay_test results in a User#delay_test failed with NameError: uninitialized constant User::SubscriptionChild

Unless an embedded type/object of SubscriptionChild already exists in Subscription (user.subscriptions).

Seems like Delayed_Jobs worker isn't loading SubscriptionChild inherited class. Anyone know how to fix/hack around this?

Mongoid 7 support

Are there any plans to support Mongoid 7?

Assigning different worker threads to different hosts

Hello:

Does anyone know whether or not it is possible to run some delayed_job tasks from the queue in the delayed_jobs table on one set of production hosts (e.g., app0, app1, and app2), while others on another set of production hosts (e.g., app3 and app4).

For example, I would like to have app0, app1, and app2 share the load of sending out emails, while app3 and app4 handle other background tasks (e.g., importing data, etc.)

Thanks for any help or suggestions.
--Alex

Delayed Job issue with Mongo transactions and job reservation

Hello, I'm running into some Mongo transaction issues. I have a cron task that queues up hundreds of delayed job records to be consumed by 2 workers. Each of those jobs may also trigger some other delayed jobs to be consumed by a different worker. (We are running 6 workers, total). The issue comes in below (see log) and crashes the worker process:

(Note: the workers that crash are processing the spin-off Delayed Jobs, not the main one processing the original 100s of records)

Log entry:

I, [2021-05-26T14:21:57.379990 #835]  INFO -- : 2021-05-26T14:21:57-0700: [Worker(delayed_job.0 host:jobsserver.staging.webserver.com pid:835)] 

Error while reserving job: Retryable write with txnNumber 11 is prohibited on session eae817b0-643f-4a5a-aacc-baae7ff9dbd9 - S8oMyTnRmLmiiib96wHuUHEe1v5gZMNm0MYQT5hrBs4= because a newer retryable write with txnNumber 406 has already started on this session. (225) (on web-staging-shard-00-00.stlfg.mongodb.net:27017, modern retry, attempt 1)

Could these be a bug?

Other system details:

ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

    rails (5.2.5)
    mongoid (7.0.5)
      activemodel (>= 5.1, < 6.1)
      mongo (>= 2.5.1, < 3.0.0)
    delayed_job (4.1.9)
      activesupport (>= 3.0, < 6.2)
    delayed_job_mongoid (2.3.1)
      delayed_job (>= 3.0, < 5)
      mongoid (>= 3.0, < 8)
      mongoid-compatibility (>= 0.4.0)

I've also tried this on everything from Mongoid 7.0.5 to 7.3.0 and on the Mongo gem up to 2.15.0.alpha.

Latest Rubygems version breaks this gem

Upgraded rubygems, now things are breaking :-(

In Gemfile:
  delayed_job_mongoid depends on
    mongoid (~> 2.0)

  mongoid (2.0.0.beta.20)

The requirement in delayed_job_mongoid should be "> 2.0.0" I'm guessing, not just "> 2.0".

Rubygems breakage sucks :P

jobs:work fails with error "No matching object found"

I've installed the gem and run the create_indexes script on my database, but when I run rake jobs:work it fails with the following output:

rake aborted!
The operation: #<Moped::Protocol::Command
  @length=410
  @request_id=2
  @response_to=0
  @op_code=2004
  @flags=[]
  @full_collection_name="delayed_job_test.$cmd"
  @skip=0
  @limit=-1
  @selector={:findAndModify=>"delayed_backend_mongoid_jobs", :query=>{"run_at"=>{"$lte"=>2013-03-06 17:29:12 UTC}, "failed_at"=>nil, "$or"=>[{"locked_by"=>"host:Ross.local pid:11944"}, {"locked_at"=>nil}, {"locked_at"=>{"$lt"=>2013-03-06 13:29:12 UTC}}]}, :new=>true, :sort=>{"locked_by"=>-1, "priority"=>1, "run_at"=>1}, :update=>{"$set"=>{:locked_at=>2013-03-06 17:29:12 UTC, :locked_by=>"host:Ross.local pid:11944"}}}
  @fields=nil>
failed with error "No matching object found"
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:78:in `block in command'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:573:in `[]'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:573:in `block (3 levels) in flush'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:572:in `map'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:572:in `block (2 levels) in flush'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:129:in `ensure_connected'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:568:in `block in flush'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:583:in `logging'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:567:in `flush'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:556:in `process'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:71:in `command'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/session/context.rb:51:in `block in command'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/session/context.rb:109:in `block in with_node'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/cluster.rb:209:in `block in with_primary'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/node.rb:174:in `ensure_primary'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/cluster.rb:208:in `with_primary'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/session/context.rb:108:in `with_node'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/session/context.rb:50:in `command'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/database.rb:76:in `command'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/session.rb:78:in `command'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/query.rb:256:in `block in modify'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/session.rb:324:in `with'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/moped-1.4.3/lib/moped/query.rb:255:in `modify'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/mongoid-3.1.2/lib/mongoid/contextual/find_and_modify.rb:44:in `result'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/mongoid-3.1.2/lib/mongoid/contextual/mongo.rb:174:in `find_and_modify'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/mongoid-3.1.2/lib/mongoid/contextual.rb:19:in `find_and_modify'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job_mongoid-2.0.0/lib/delayed/backend/mongoid.rb:51:in `reserve'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:258:in `reserve_and_run_one_job'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:187:in `block in work_off'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:186:in `times'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:186:in `work_off'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:151:in `block (4 levels) in start'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:150:in `block (3 levels) in start'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:60:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:60:in `block in initialize'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:65:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:65:in `execute'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:149:in `block (2 levels) in start'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:148:in `loop'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:148:in `block in start'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/plugins/clear_locks.rb:7:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:78:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:60:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:60:in `block in initialize'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:78:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:78:in `block in add'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:65:in `call'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:65:in `execute'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/worker.rb:147:in `start'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/gems/delayed_job-3.0.5/lib/delayed/tasks.rb:9:in `block (2 levels) in <top (required)>'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/bin/ruby_noexec_wrapper:14:in `eval'
/Users/rossbrown/.rvm/gems/ruby-1.9.3-p194@eln/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => jobs:work

This happens with both populated and empty databases, with or without existing delayed_jobs. Any idea what's going on? Am I configuring something incorrectly?

• Ruby 1.9.3
• Rails 3.2.11
• MongoDB shell version: 2.0.1
• Mongoid 3.1.2

Attempted to set a value for 'owner' which is not allowed on the model Delayed::Backend::Mongoid::Job.

hi

i would like to use delayed_job_mongoid as a backend for delayed_job
because i think it can be used on AWS DocumentDB as a backend for delayed_job.

On local

i got this error when running a job

Mongoid::Errors::UnknownAttribute at /biz/company/part_time/branches/6/applicants/10/send_notification_for_schedule_adjustment
message:
  Attempted to set a value for 'owner' which is not allowed on the model Delayed::Backend::Mongoid::Job.
summary:
  Without including Mongoid::Attributes::Dynamic in your model and the attribute does not already exist in the attributes hash, attempting to call Delayed::Backend::Mongoid::Job#owner= for it is not allowed. This is also triggered by passing the attribute to any method that accepts an attributes hash, and is raised instead of getting a NoMethodError.

actually this project use activerecoard for using a default.
i want to separate delayed_job_backend.

is it possible?

here is my related code.

delayed_job.rb

require 'delayed_job'
require 'delayed_job_mongoid'
class Delayed::Job
  # establish_connection :delayed
end

mongoid.yml

defaults: &defaults
  allow_dynamic_fields: true

development:
  <<: *defaults
  # Configure available database clients. (required)
  clients:
    # Defines the default client. (required)
    default:
      # Mongoid can connect to a URI accepted by the driver:
      # uri: mongodb://user:[email protected]:27017/recruiting_bot_development

      # Otherwise define the parameters separately.
      # This defines the name of the default database that Mongoid can connect to.
      # (required).
      database: sogo_career_development
      # Provides the hosts the default client can connect to. Must be an array
      # of host:port pairs. (required)
      hosts:
        - localhost:27017
      options:
        # Note that all options listed below are Ruby driver client options (the mongo gem).
        # Please refer to the driver documentation of the version of the mongo gem you are using
        # for the most up-to-date list of options.
        #
        # Change the default write concern. (default = { w: 1 })
        # write:
        #   w: 1

        # Change the default read preference. Valid options for mode are: :secondary,
        # :secondary_preferred, :primary, :primary_preferred, :nearest
        # (default: primary)
        # read:
        #   mode: :secondary_preferred
        #   tag_sets:
        #     - use: web

        # The name of the user for authentication.
        # user: 'user'

        # The password of the user for authentication.
        # password: 'password'

        # The user's database roles.
        # roles:
        #   - 'dbOwner'

        # Change the default authentication mechanism. Valid options are: :scram,
        # :mongodb_cr, :mongodb_x509, and :plain. Note that all authentication
        # mechanisms require username and password, with the exception of :mongodb_x509.
        # Default on mongoDB 3.0 is :scram, default on 2.4 and 2.6 is :plain.
        # auth_mech: :scram

        # The database or source to authenticate the user against.
        # (default: the database specified above or admin)
        # auth_source: admin

        # Force a the driver cluster to behave in a certain manner instead of auto-
        # discovering. Can be one of: :direct, :replica_set, :sharded. Set to :direct
        # when connecting to hidden members of a replica set.
        # connect: :direct

        # Changes the default time in seconds the server monitors refresh their status
        # via hello commands. (default: 10)
        # heartbeat_frequency: 10

        # The time in seconds for selecting servers for a near read preference. (default: 0.015)
        # local_threshold: 0.015

        # The timeout in seconds for selecting a server for an operation. (default: 30)
        # server_selection_timeout: 30

        # The maximum number of connections in the connection pool. (default: 5)
        # max_pool_size: 5

        # The minimum number of connections in the connection pool. (default: 1)
        # min_pool_size: 1

        # The time to wait, in seconds, in the connection pool for a connection
        # to be checked in before timing out. (default: 5)
        # wait_queue_timeout: 5

        # The time to wait to establish a connection before timing out, in seconds.
        # (default: 10)
        # connect_timeout: 10

        # The timeout to wait to execute operations on a socket before raising an error.
        # (default: 5)
        # socket_timeout: 5

        # The name of the replica set to connect to. Servers provided as seeds that do
        # not belong to this replica set will be ignored.
        # replica_set: name

        # Whether to connect to the servers via ssl. (default: false)
        # ssl: true

        # The certificate file used to identify the connection against MongoDB.
        # ssl_cert: /path/to/my.cert

        # The private keyfile used to identify the connection against MongoDB.
        # Note that even if the key is stored in the same file as the certificate,
        # both need to be explicitly specified.
        # ssl_key: /path/to/my.key

        # A passphrase for the private key.
        # ssl_key_pass_phrase: password

        # Whether to do peer certification validation. (default: true)
        # ssl_verify: true

        # The file containing concatenated certificate authority certificates
        # used to validate certs passed from the other end of the connection.
        # ssl_ca_cert: /path/to/ca.cert

        # Whether to truncate long log lines. (default: true)
        # truncate_logs: true

  # Configure Mongoid specific options. (optional)
  options:
    # Application name that is printed to the mongodb logs upon establishing
    # a connection in server versions >= 3.4. Note that the name cannot
    # exceed 128 bytes. It is also used as the database name if the
    # database name is not explicitly defined. (default: nil)
    # app_name: MyApplicationName

    # Create indexes in background by default. (default: false)
    # background_indexing: false

    # Mark belongs_to associations as required by default, so that saving a
    # model with a missing belongs_to association will trigger a validation
    # error. (default: true)
    # belongs_to_required_by_default: true

    # Raise an exception when a field is redefined. (default: false)
    # duplicate_fields_exception: false

    # Include the root model name in json serialization. (default: false)
    # include_root_in_json: false

    # Include the _type field in serialization. (default: false)
    # include_type_for_serialization: false

    # Whether to join nested persistence contexts for atomic operations
    # to parent contexts by default. (default: false)
    # join_contexts: false

    # Set the Mongoid and Ruby driver log levels when Mongoid is not using
    # Ruby on Rails logger instance. (default: :info)
    # log_level: :info

    # Preload all models in development, needed when models use
    # inheritance. (default: false)
    # preload_models: false

    # Raise an error when performing a #find and the document is not found.
    # (default: true)
    # raise_not_found_error: true

    # Raise an error when defining a scope with the same name as an
    # existing method. (default: false)
    # scope_overwrite_exception: false

    # Use ActiveSupport's time zone in time operations instead of
    # the Ruby default time zone. See the time zone section below for
    # further information. (default: true)
    # use_activesupport_time_zone: true

    # Return stored times as UTC. See the time zone section below for
    # further information. Most applications should not use this option.
    # (default: false)
    # use_utc: false

test:
  clients:
    default:
      database: sogo_career_test
      hosts:
        - localhost:27017
      options:
        read:
          mode: :primary
        max_pool_size: 1

run job

notification.delay(owner: self, priority: -10, queue: "high_priority", run_at: notification_datetime).notify(self) if SmsNotificationLog.where(channel_id: self.id, notification_id: notification.id).size < 1

Invalid index specification on Delayed::Backend::Mongoid::Job:

The instructions says to run:
script/rails runner 'Delayed::Backend::Mongoid::Job.create_indexes'
after running bundle install. However, when i do so, I get this error. I'm using DJ 3.0.3 and DJ_Mongoid 1.0.8.

This is the error I get, anyone seen this or know how to fix this?

Problem:
Invalid index specification on Delayed::Backend::Mongoid::Job: [[:locked_by, -1], [:priority, 1], [:run_at, 1]], {}
Summary:
Indexes in Mongoid are defined as a hash of field name and direction/2d pairs, with a hash for any additional options.
Resolution:
Ensure that the index conforms to the correct syntax and has the correct options.

Identity map not cleared

The Mongoid::IdentityMap should probably be cleared between job runs, as it is for ActiveRecord in:
https://github.com/collectiveidea/delayed_job/blob/5c7628d40c9e2e26ab2076453ab6488df3464730/lib/delayed/backend/base.rb#L92

Support for Mongoid v2.4.0+

I'm not clear on whether the master branch of this gem supports Mongoid 2.4.0+ explicitly. I'm currently evaluating it with 2.4.0, and it seems to be working properly. However, I don't want to get bit down the road... Can someone confirm for me?

#32

What are the requirements for using the version on master?

Move to the mongoid organization?

Looks like things are a bit slow here. We'll gladly take this gem in the mongoid organization, https://github.com/mongoid. At the very least I'd love to help out with merging PRs, closing issues and making a release (dblock at dblock dot org on rubygems).

Compatibility with mongoid 4?

I'm using Rails 4.0.0 with mongoid 4.0.0, and I would love to be able to use delayed_job_mongoid with it also. Though the gem is rails 4 compatible on the master branch, it seems that it's not supporting mongoid 4 yet.

Bundler could not find compatible versions for gem "mongoid":
In Gemfile:
delayed_job_mongoid (>= 0) ruby depends on
mongoid (~> 3.0) ruby

mongoid (4.0.0)

too slow when interact with backend mongoid

Wed Apr 13 09:56:29 [conn2431112] query foobar-account.$cmd ntoreturn:1 command: { findandmodify: "delayed_backend_mongoid_jobs", update: { $set: { locked_at: new Date(1302659785872), locked_by: "delayed_job host:CX160500 pid:1050" } }, sort: { locked_by: -1, priority: 1, run_at: 1 }, query: { $or: [ { locked_by: "delayed_job host:CX160500 pid:1050" }, { locked_at: null }, { locked_at: { $lt: new Date(1302645385872) } } ], failed_at: null, run_at: { $lte: new Date(1302659785872) } } } reslen:90 3430ms
Wed Apr 13 09:59:45 [conn2431122] query foobar-account.$cmd ntoreturn:1 command: { findandmodify: "delayed_backend_mongoid_jobs", query: { failed_at: null, $or: [ { locked_by: "delayed_job host:CX160501 pid:27714" }, { locked_at: null }, { locked_at: { $lt: new Date(1302645584705) } } ], run_at: { $lte: new Date(1302659984705) } }, update: { $set: { locked_at: new Date(1302659984705), locked_by: "delayed_job host:CX160501 pid:27714" } }, sort: { locked_by: -1, priority: 1, run_at: 1 } } reslen:90 938ms

kaminari pagination loaded too late

Delayed::Job.where.page(2) will spew undefined page method because delayed_job_mongoid got loaded then kaminari as can be seen via strategically placed puts statements in

../kaminari-0.12.3/lib/kaminari/railtie.rb
../delayed_job_mongoid-1.0.2/lib/delayed/backend/mongoid.rb

According to rubygems/bundler#107 kaminari will be required first then delayed_job_mongoid when Gemfile is as follow

gem 'rails', '~>3'
gem 'mongoid'
gem 'kaminari'
gem 'delayed_job'
gem 'delayed_job_mongoid'

quickest workaround is to mix in in initializer

Delayed::Job.send(:include, Kaminari::MongoidExtension::Document)

Why does the reserve method search for jobs locked by the worker?

This check makes the query & sort more inefficient (by using both locked_by and locked_at in an $or) and then sorting to bring the locked_by job to the top.

Doing so brings mongodb to a screeching halt (100% locked db with just a handful of workers, all of them locking the db to do a findAndModify). It gets really bad when you have even a tiny bit of load (10-20k jobs).

If possible, moving the "Am I already working on something" check outside of the findAndModify would probably speed things up.

Stack level too deep

I have the following issue with a project using MongoDB, rails 3.1.3 and ruby 1.9.2:

When I use the Thin server, every time delay is called (such as: UserMailer.delay.password_reset(..) ) I get stack level too deep and the website crashes. However, when I start the WEBrick server, it works just fine and emails are sent. I followed the instructions to setup delayed_job for mongoid and rails 3.

Here is the output of the error:

SystemStackError in PasswordResetsController#create

stack level too deep

Full Trace
actionpack (3.1.3) lib/action_dispatch/middleware/reloader.rb:73

Can you please give me some hints why Thin fails and WEBrick works? Is this a problem with delayed_job or with Thin?

Thanks,
Lucian

after upgraded to 3.2.1, getting warning about ActiveSupport::Concern

The InstanceMethods module inside ActiveSupport::Concern will be no longer included automatically. Please define instance methods directly in Delayed::Backend::Mongoid::Job instead. (called from require at /Users/nik/.rvm/gems/ruby-1.9.3-p0@global/gems/bundler-1.0.21/lib/bundler/runtime.rb:68)

Will this be addressed in an upcoming update or is it something I can do myself to correct?

Best,
Nik