ruby-shoryuken / shoryuken Goto Github PK

We are using Shoryuken at my work and need the ability to schedule jobs farther into the future than the 15 minute delay limit imposed by SQS.

My idea is to use DynamoDB for this purpose - and I have created a fork of Shoryuken to work on it. I was wondering if you were interested in such a feature and/or if you have any feedback on it. It looked like it would be more kludgy to build this feature into a separate gem, since it would need to monkey patch Shoryuken and that might make things brittle whenever an updated Shoryuken gem is released.

I see that sns topics are now supported, but I am not sure how they are intended to be used - could someone add some docs for topics?

e.g. how to configure them, how to send to a topic, what needs to be done to subscribe an sqs queue to a topic so workers receive messages

I guess I can try to read the code and come to my own conclusions, but someone else may already be ahead of my understanding.

Is there any reason why there is the limitation of 1 queue = 1 worker?

It seems a bit overkill to have a separate queue for each worker type, especially since shryuken checks the queues serially, which can cause large delays in the case of many queues.

Apologies for opening all of these tickets – I like shoryuken, but need to tailor it to my own purposes. Of course, I am willing to put in the work on some of these tasks, although I may require a little guidance :)

undefined local variable or method 'initialize_logger' for #<Shoryuken::CLI:0x002b96123be308>

In an ElasticBeanstalk application, Web Server instances send messages to SQS, and Worker instances consume SQS messages and process jobs (right?)

Assuming the same codebase is deployed to both Web Server and Worker environments (like Heroku), is there a way to ensure that only Worker instances consume messages and process jobs?

Hey Guys,

I use latest version of gem (shoryuken (1.0.0)):

gem "shoryuken", github: "phstc/shoryuken", branch: "master"

my config/shoryuken.yml

aws:
  access_key_id:      <%= ENV['AWS_ACCESS_KEY_ID'] %>
  secret_access_key:  <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
  region:             <%= ENV['AWS_REGION'] %>
  receive_message:              # See http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/SQS/Queue.html#receive_message-instance_method
    # wait_time_seconds: 1        # The number of seconds to wait for new messages when polling. Defaults to the #wait_time_seconds defined on the queue
    attributes:
      - receive_count
      - sent_at
concurrency: 4,  # The number of allocated threads to process messages. Default 25
delay: 5,        # The delay in seconds to pause a queue when it's empty. Default 0
queues:
  - [<%= ENV['SQS_QUEUE'] %>, 1]

Then I trying to use deliver_later! -> then shoryuken tries to use "mailers" queue, instead of I'm setting in settings.

2.1.5 :017 > MyMailer.notify.deliver_later!
Enqueued ActionMailer::DeliveryJob (Job ID: de473055-f237-401e-b70f-c79e3feff4f8) to Shoryuken(mailers) with arguments: "Users::AuditCertificateRequestMailer", "notify", "deliver_now!", 39, 4
Aws::SQS::Errors::NonExistentQueue: The specified queue does not exist for this wsdl version.

I also tried to fix it by adding custom initializer:
config/initializers/shoryuken.rb

  def parse_config(config_file)
    if File.exist?(config_file)
      YAML.load(ERB.new(IO.read(config_file)).result)
    else
      raise ArgumentError, "Config file #{config_file} does not exist"
    end
  end

  config = parse_config([Dir.pwd, 'config/shoryuken.yml'].join('/')).deep_symbolize_keys
  Shoryuken::EnvironmentLoader.load(config)

Then checking in rails console:

Shoryuken.options
 => {:concurrency=>4, :queues=>[["stagingsqsqueue", 1]], :aws=>{:access_key_id=>"AWS_ACCESS_KEY_ID", :secret_access_key=>"AWS_SECRET_ACCESS_KEY", :region=>"eu-west-1", :receive_message=>{:attributes=>["receive_count", "sent_at"]}}, :delay=>5, :timeout=>8} 

But it same thing. Anyway it tries to use "mailers" queue.

Some ActiveJob outputs:

2.1.5 :008 > ActiveJob::Base.queue_as
 => "default" 
2.1.5 :009 > ActiveJob::Base.queue_name
 => "default" 
2.1.5 :010 > ActiveJob::Base.queue_adapter
 => ActiveJob::QueueAdapters::ShoryukenAdapter 

I also tried to setup queue_name manually for ActiveJob:

2.1.5 :012 > ActiveJob::Base.queue_name = "stagingsqsqueue"
 => "stagingsqsqueue" 
2.1.5 :014 > ActiveJob::Base.queue_name
 => "stagingsqsqueue" 

But same issue.

Thanks for any help 🍻

I have some that we are using at work, for a Rails based project. That project is still using Capistrano 2. Then, another project that we are about to use Shoryuken in (via ActiveJob) is using Capistrano 3. So I will have two flavors shortly. Right now, I have only written tasks for Capistrano 2.

Is there any interest in including Capistrano recipes into Shoryuken, or should I consider putting it into a separate gem?

We've been experiencing high bandwidth usage in production/staging with shoryuken.

My settings are as follows:

• wait_time_seconds: 5
• concurrency: 10
• delay: 5

I have 7 queues, and according to the above settings (confirmed with verbose logging), the queues are being checked every 5 seconds.

In staging (an EC2 instance), I'm seeing upstream usage averaging about 70kb/s with shoryuken, with negligible downstream. I used a 5 min. average to calculate this. Note that during this period, no jobs were executed, as confirmed by the log.

In development (local machine), a 5 min average shows 8kb/s being used by shoryuken, which definitely isn't as bad as on staging, but still higher than I would expect, considering just SQS communication every 5 secs (and why upstream, as opposed to downstream?).

I tried upgrading to the newest version of "aws-sdk-v1", but it had no effect. I also narrowed down the usage to SQS, as the IPs that the data heads to reverse-map to SQS hosts.

Any ideas on what could be causing this strange behavior? 70kb/s average on EC2 is definitely not ideal!

I am getting the following error when using the auto_visibility_timeout option in my worker class:

ERROR: Could not auto extend the message FooWorker/non-interactive-1m/7740b3be-1e0a-4c42-ba49-da69ea1ac19b visibility timeout. Error: undefined method `visibility_timeout=' for #<Aws::SQS::Message:0x007fdf67278d20>

My Gemfile.lock:

shoryuken (1.0.0)
  aws-sdk-core (= 2.0.21)
  aws-sdk-resources (= 2.0.21.pre)
  celluloid (~> 0.16.0)

I use the active job adapter and it seems that the messages are not deleted from the queue after completion.

The auto_delete option is set here, but for some reasons it does not work locally:
https://github.com/phstc/shoryuken/blob/master/lib/shoryuken/extensions/active_job_adapter.rb#L55

I do use
config.active_job.queue_name_prefix = Rails.application.secrets.APP_NAME

in the shoryuken.yml config:

queues:
  - [<%=Rails.application.secrets.APP_NAME + "_default"%>, 1]

and in the job itself:
queue_as :default

(I believe this last of should not have the prefix, otherwise what would the config be good for?)

Thanks,
Julien.

Since everyone will pretty much need this, I think it's worth adding to shoryuken itself.

I've created these for my own project, but they only handle the config options that I had an immediate use for. This ticket is mainly a reminder for myself to abstract it out and create a PR eventually.

If someone wants to take a stab at it before I can find the time, that is welcome (I can share my current rake tasks).

Hi,

First of all thanks a lot for the great work !
From what i understand the gem is currently designed to handle one queue per process, so if i want to listen to n queues i need do start n shoryuken process (right ?)

On our app workers are mainly doing IO intensive job, like database operations. Listening to all queues on a single process would reduce RAM usage a lot and also means fewer process to monitor.
I'm willing to contribute if you're interested in.

Something like sidekiq/testing

Trying to exit the process seems to kill all the threads, but the main process still hangs.

Example output:

^C2014-11-04T10:44:37Z 14299 TID-owmpirlkg INFO: Got INT signal
2014-11-04T10:44:37Z 14299 TID-owmpirlkg INFO: Received INT, will shutdown down
2014-11-04T10:44:45Z 14299 TID-owmpld5lc INFO: No message found for 'development-ernsputer-image'
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Shutting down 25 quiet workers
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
2014-11-04T10:44:45Z 14299 TID-owmqjgkoc INFO: Process died, reason: 
^C

Some of my workers need to have middleware X invoked each time they handle a message, and some don't. At the moment I can only configure the server_middleware chain that is run for every worker. I'm thinking that Shoryuken::Worker should offer a hook for modifying the default chain on a per-worker basis. For example:

class FooWorker
  include Shoryuken::Worker

  shoryuken_options queue: 'foo_queue', airbrake_data: 'some string'

  shoryuken_middleware do |chain|
    chain.insert_before Shoryuken::Middleware::Server::Timing, MyAirbrakeMiddleware
  end
end

I guess when the worker is registering itself with Shoryuken it can cache a new chain if shoryuken_middleware was called, and then the processor would need to pick that up here:

https://github.com/phstc/shoryuken/blob/master/lib/shoryuken/processor.rb#L21

I would like to work on this feature, do you have any feedback about how you would like it to work?

I figure that we would be using the :approximate_receive_count message attribute to drive the retrying of jobs, and then set the message's visibility timeout based on the number of retries. Does this make sense?

Configuration for the retry could be in the shoryuken_options for the worker. Maybe a Hash of Fixnum and/or Range keys that map to visibility timeout values? Middleware would have to scan the Hash to find the first matching key and the assign the visibility timeout. If the retry count is greater than all of the keys in the Hash (Range end or Fixnum), then delete the message?

Just food for thought, really I am just making a bunch of assumptions about how you would like it to work. :-) I am open to anything.

Hey there.

We've built a similar gem https://github.com/musicglue/envoy/tree/simplify to meet some needs in our organisation.

Differences:

• It pushes to sns, not sqs. This is to support fanout pub-sub.
• It has rake tasks to build the sns and sqs infrastructure for you, according to config you define in your initializer:

  config.add_dead_letter_queue 'my_dlq'

  config.add_queue('order_processing') do |queue, subscriptions|
    subscriptions.add 'order_placed', 'OrderWorker'
    subscriptions.add 'order_refunded', 'RefundWorker'
  end

• It generates IAM policies according to the config you have defined.
• It allows you to host custom celluloid actors in-process so you can have long-running jobs:

  config.celluloid.actors += [DailySalesAggregationWorker]

• It uses the latest aws-sdk-core-ruby gem.
• Middlewares for Airbrake, transactional workers + idempotent workers (persisting an application-defined message id in the database when it's processed on a queue and ignoring reprocessing of that message on that queue again).

Drawbacks:

• It doesn't do any kind of load-balancing or queue prioritisation.
• Signal handling is a little bit flakey.
• Requires raw messages in sqs.
• Only me hacking on it. 🚌
• Very patchy test coverage.

Would you be interested in taking any of these ideas in shoryuken?

Is that in the plans?

In Processor.auto_visibility_timeout, there is a queue passed in as the first arg, but this is not used for the visibility_timeout_heartbeat:

    def auto_visibility_timeout(queue, sqs_msg, worker_class)
      if worker_class.auto_visibility_timeout?
        timer = every(worker_class.visibility_timeout_heartbeat) do

Instead, when it gets the value from Worker.extended_visibility_timeout, it uses the queue from the shoryuken options to get the visibility_timeout:

      def extended_visibility_timeout
        Shoryuken::Client.visibility_timeout(get_shoryuken_options['queue'])
      end

However, the actual receiving queue, passed in to auto_visibility_timeout, may well not match the queue from get_shoryuken_options.

I'm always getting that error, it doesn't matter that I create sidekiq, uppercut or shoryuken queues also I created a default queue and no luck.

What I'm trying to accomplish here is to check for a particular queue and after getting messages from there process those messages and then send it back after a successful process of the messages get's done.

Hey Guys,

Is it possible to run shoryuken process for all workers like for sidekiq:

bundle exec sidekiq # Will process all jobs

For example if I run:

bundle exec shoryuken -C config/shoryuken.yml -L log/shoryuken.log

then I see:

2015-03-16T10:51:03Z 11892 TID-agdn8 WARN: No worker supplied for 'staging_mailers'
2015-03-16T10:51:03Z 11892 TID-fyjpc INFO: Starting

For example, if I use ActiveJob#deliver_later! - then these jobs would not be proccessed as I'm not passing path to worker.

Just tested: jobs are adding to SQS queue, but are not process by running:

bundle exec shoryuken -C config/shoryuken.yml -L log/shoryuken.log

Thanks for any help 🍻

how should I start the shoryuken workers on AWS OpsWorks?

It would be nice including OpsWorks deployment instructions / cookbooks in the deployment section ( #66 )

Thanks

What do you think of having the following messages show up only when the "verbose" option is specified:

• Looking for new messages ...
• No message found for ...

They are usually just noise, since they happen all the time, even when no jobs are being processed...

Or perhaps we can make it a config option, in the YML?

What do you think?

In the testing section (https://github.com/phstc/shoryuken/wiki/Testing)
AWS::SQS::ReceivedMessage doesn't work with latest aws-sdk-core (~> 2.0.21) because it has been removed. Please update new guides for testing.

for example:

application.rb

config.active_job.queue_name_prefix = Rails.env

shoryuken.yml

queues:
  - [docker_runner, 6]
  - [default, 2]
  - [image_builder, 1]

got message:

Queue 'docker_runner' does not exist

It would be more cost effective to use long polling rather than short polling, as stated in the SQS docs :

In almost all cases, SQS long polling is preferable to SQS short polling [...] However, if your application
is written to expect an immediate response from a ReceiveMessage call, you may not be able to take
advantage of long polling without some application modifications [...] In such an application, it is
recommended that a single thread be used to process only one queue, allowing for the application to
take advantage of the benefits SQS long polling provides.

When should I use SQS long polling, and when should I use SQS short polling?

What do you think ?

Thx,
Vincent

I'm trying to come up with some structure for multiple workers. Currently it looks like this:

workers/
   some_worker.rb
shoryuken.yml
Gemfile
Gemfile.lock

Now I am thinking where I should put initialization for Rollbar.

configure_server is briefly mentionted in documentation, but still no clue how to use it: https://github.com/phstc/shoryuken/wiki/Middleware

Proposals?

I'm a fan of the easy syntax for adding recurring tasks using https://github.com/tobiassvn/sidetiq with Sidekiq.

e.g.:

class MyRecurrentWorker
  include Shoryuken::Worker
  include Shoryuken::Schedulable

  recurrence { daily }

  def perform
    # do stuff ...
  end
end

Any plans to add something like this or would that be external to shoryuken?

Your gemspec says mit. The license must be LGPL.

I had a use case where I wanted to publish to SNS from a worker. Knowing the worker itself already creates a SQS client, but didn't know if SNS was supported I had to look through the source. Would be nice if it's explicitly documented to show there's support to publish to SNS topic from worker using the same AWS Client. Let me know if I can help with this.

https://github.com/phstc/shoryuken/blob/master/lib/shoryuken/worker.rb#L40

This should be opts || {} , right ?

please correct me if I am wrong.

Instead of manually sleeping :delay seconds for every queue which doesn't have messages why not use Aws::SQS::QueuePoller which is part of aws-sdk? http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/QueuePoller.html
I think it's a great alternative and could save us from maintaining some code, like queue pausing.
I'm trying to port current code to using it, but i don't understand yet if there could be problems if for example Fetcher's fetch method blocked polling and dispatching messages until Aws::SQS::QueuePoller ended for some reason or an exception was raised.

Maybe the creators/maintainers could check this up.

Cheers!
Sebastian

The current master now uses the aws sdk 2 for ruby, which is awesome.

However, when the signature for send_message is now to accept a single hash as an argument:
http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#send_message-instance_method

Here is how perform_async tries to call send_message:

      def perform_async(body, options = {})
        options ||= {}
        options[:message_attributes] ||= {}
        options[:message_attributes]['shoryuken_class'] = {
          string_value: self.to_s,
          data_type: 'String'
        }

        queue = Shoryuken::Client.queues(get_shoryuken_options['queue'])
        queue.send_message(body, options)  # this is the wrong number of args
      end

Instead, the send_message call needs to look like this I think:

      def perform_async(body, options = {})
        options ||= {}
        options[:message_attributes] ||= {}
        options[:message_attributes]['shoryuken_class'] = {
          string_value: self.to_s,
          data_type: 'String'
        }
        options[:message_body] = body

        queue = Shoryuken::Client.queues(get_shoryuken_options['queue'])
        queue.send_message(options)
      end

So when I try to send a message, I get the following error for the send_message on the sqs resource:

Failed to create new job and tasks: missing required parameter params[:message_body]
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/param_validator.rb:24:in `validate!'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/param_validator.rb:9:in `validate!'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/plugins/param_validation.rb:21:in `call'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/plugins/param_conversion.rb:22:in `call'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/request.rb:70:in `send_request'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.21/lib/seahorse/client/base.rb:215:in `block (2 levels) in define_operation_methods'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.0.21.pre/lib/aws-sdk-resources/request.rb:24:in `call'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.0.21.pre/lib/aws-sdk-resources/operations.rb:41:in `call'
        /Users/andrew/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.0.21.pre/lib/aws-sdk-resources/operation_methods.rb:19:in `block in add_operation'

There was a PR yesterday to use role based auth. We'd actually like to use it as-is without looking at your github in our gemfile.

If you could push up master to rubygems with a version bump, all is well =)

If you have a long-running worker processing a job and you SIGINT in the middle of it, you will receive an error from this line:

https://github.com/phstc/shoryuken/blob/master/lib/shoryuken/manager.rb#L212

because there is no such method actual_work_thread. The process is then zombied and has to be SIGTERMed. It looks like this was code taken from an old version of Sidekiq, which now does something different by keeping a hash of processor object ids to threads. See:

sidekiq/sidekiq@06acbd4

Can you not rely on a .yml file to provide the configuration?

I prefer to use ENV to avoid commiting secret key.

First of all, thank you for this useful gem!

Sadly I cannot get it to run and struggle with a quite simple problem.

What I'm trying to do is set up a simple Sinatra server for accepting requests. On request it checks if a file is present, and if it is not, it queues its creation and responds with an intermediate file.

I have put my configuration into shoryuken.yml.

Running bundle exec shoryuken --config shoryuken.yml --require ./models/worker.rb --pidfile tmp/shoryuken.pid succeeds and spawns two workers.

When trying to use Worker.perform_async(params: params) from my Sinatra-initiated request, however, I get an error saying Aws::Errors::MissingRegionError - missing region; use :region option or export region name to ENV['AWS_REGION']:.

This appears reasonable, as this process doesn't know anything about shoryuken.yml, as far as I can tell.

What would be a good way to read this file and pass it to Shoryuken? Am I missing something? Read through all wiki pages, but couldn't figure it out.

Thanks!

The Use Case
I am unifying logs from a variety of sources into a single indexed source (elasticsearch).

The Problem
Parsing the format of the logs from shoryuken into key value pairs is proving to be a problem.
The format of the meta-data is easy to understand and parse, but the fact that the context is optional, and when not present has no placeholder, makes the regex to parse the log a bit heavy.

Log Format

"#{time.utc.iso8601} #{Process.pid} TID-#{Thread.current.object_id.to_s(36)}#{context} #{severity}: #{message}\n"

Regex to parse messages with context

/^(?<time>[^ ]*) (?<pid>[^ ]*) TID-(?<thread_id>[^ ]*) (?<context>[^ ]*) (?<severity>[A-Z]*): (?<message>.*)$/

Regex to parse messages without context

/^(?<time>[^ ]*) (?<pid>[^ ]*) TID-(?<thread_id>[^ ]*) (?<severity>[A-Z]*): (?<message>.*)$/

Regex to handle either

/(?<time>[^ ]*) (?<pid>[^ ]*) TID-(?<tid>[^ ]*)( )?(?<worker_class>.*?)((/(?=[a-z]))| )(?<queue>[^/]*)(/)?(?<sqs_message_id>.*)(?<severity>(INFO|WARN|DEBUG|ERROR)): (?<message>.*)/ 

A Solution
The easy way out is to simply put a space between the TID and the context, the downside is that the there will be a double space if no context is present, the upside is that it makes it much easier to handle it being there some times and not others.

A more involved solution could be to split the manager logging (no context) from the worker logging (server timing middleware, with context) so that they could be moved to write to separate locations without altering source code.

Also, given the fact that the context is always "ActiveJob/#{WorkerClass}/#{queue_name}/#{sqs_msg_id}" I think it might make sense to separate these items

Given the worker:

class SendMailWorker
  include Shoryuken::Worker

  shoryuken_options queue: "send_mail"

  def perform(sqs_msg, body)
    # send email
  end

  # def queue_prefix
  #   if Rails.env.development?
  #     ENV['USER']
  #   else
  #     Rails.env
  #   end
  # end

end

What is the recommended way to do the following:

• In development, prefix the queue with ENV['USER']
• In other environments prefix the queue with ENV['RAILS_ENV']
• Use dynamic queue names with shoryuken.yml

If I upgrade the aws-sdk-resources to the latest version (2.0.29 at this time), the tests don't pass. That happens because Aws::SQS::Queue and Aws::SQS::Message (maybe others) are not defined. I've looked through the aws-sdk docs and it really seems to be the case.

I don't what we should do, cause I'm not sure how one would instantiate queues and messages in this scenario. Could this be an aws-sdk issue?

As far as I can tell, there is no reason that perform should be an instance and not a class method (let me know if I am wrong in assuming this).

My use case is the following:

I have certain workers that I need to keep track of in my DB for UI purposes (tell the user they are processing, failed, and keep track of errors, etc.).

These are AR classes with associated tables, and they can't be instantiated without the proper params due to validations (so I can't just add an instance-level perform method, since Shoryuken would attempt to instantiate it, instead of find it in the DB).

Now, the perform method in my Shoryuken workers simply looks up these instances using a find, and calls a run! method.

If perform was a class method instead, I could put it into my main AR worker class (and include Shoryuken::Worker there), instead of having two classes.

What do you think? Any downsides?

I see in debug DEBUG: Ready: 14, Busy: 11, so even though I set concurrency to 25, only 11 is actually busy. How can I improve performance? So far I can't get even close to performance of previous solution. Even with 4 c3.4xlarge instances pulling from same queue, new messages continue queueing up (and there are a lot of them).

As I am testing a minor addition to shoryuken I've been working on, I can't make a simple test application work as expected using the master branch.

The error is due to the serialization of job in the queue adapter. The error is as described in the title of this issue: the message_body is supposed to be a string, but after inspecting, I saw that job.serialize turns out to be a Hash.

Anyone knows what's going on?

Memory seems to grow slowly but without limit when using shoryuken with rails environment, a single worker and waiting on an empty queue.

Any ideas what could be causing this behaviour?

Is there a sanctioned way to use Shoryuken with a Rails app (having access to its models, etc.)?

I have made my own fork and added a load_rails method to cli.rb (modelled after Sidekiq) which works. I am thinking of creating a PR, so I'm wondering if there is an easier, already-supported way that I may have missed?

I know that amazon has removed some implementations that were crucial to shoryuken.
But, is there no way of updating the amazon sdk version? right now i have request that's failing and works with latest sdk version.
And it's just a pain to use v1.
Amazon sdk's are open source, can't you just copy the code that's missing until amazon adds support?

Thanks,
Sebastian

Since this morning have been getting errors sending strings! My specific question, has the API changed so that message is now supposed to be a hash or something?

Shoryuken::Client.queues("sds-test").send_message( "test"  )


NoMethodError: undefined method `merge' for "test":String
    from /var/app/current/vendor/bundle/gems/shoryuken-1.0.2/lib/shoryuken/queue.rb:23:in `send_message'
    from (irb):4
    from /var/app/current/vendor/bundle/gems/railties-4.1.6/lib/rails/commands/console.rb:90:in `start'
    from /var/app/current/vendor/bundle/gems/railties-4.1.6/lib/rails/commands/console.rb:9:in `start'
    from /var/app/current/vendor/bundle/gems/railties-4.1.6/lib/rails/commands/commands_tasks.rb:69:in `console'
    from /var/app/current/vendor/bundle/gems/railties-4.1.6/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
    from /var/app/current/vendor/bundle/gems/railties-4.1.6/lib/rails/commands.rb:17:in `<top (required)>'
    from bin/rails:8:in `require'
    from bin/rails:8:in `<main>'

I've been going around in circles with the AWS documentation for an EB Ruby web server (Sinatra) + SQS worker.

The shoryuken docs seem to imply that if you have a Rails app, just add config, now you have workers.

Yet, with my Sinatra app, I have to launch a separate process. I've tested that shoryuken runs and works locally and by running eb ssh and connecting to my EB box:

cd /var/app/current
bundle exec shoryuken -r ./config/shoryuken_init.rb -C config/shoryuken.yml

All works fine. But when I tried to configure the workers to be launched by a .ebextensions/xxx.config file, I get grief with a AWSEBAutoscalingGroup error.

• Should I just fork a child process in my config.ru?
• Do I need a separate Worker Tier application created just to run the Workers? If so, how does the webapp launch the workers?

Does anyone have a canonical app/config for deploying a PORO/Sinatra webapp to AWS/SQS?

I'm happy to provide a minimal example if none to date.