sj26 / sidekiq-cloudwatchmetrics

Publish Sidekiq metrics to AWS CloudWatch

Sidekiq CloudWatch Metrics

Runs a thread inside your Sidekiq processes to report metrics to CloudWatch useful for autoscaling and keeping an eye on your queues.

Optimised for Sidekiq Enterprise with leader election, but works everywhere!

Installation

Add this gem to your application’s Gemfile near sidekiq and then run bundle install:

gem "sidekiq"
gem "sidekiq-cloudwatchmetrics"

Usage

Add near your Sidekiq configuration, like in config/initializers/sidekiq.rb in Rails:

require "sidekiq"
require "sidekiq/cloudwatchmetrics"

Sidekiq::CloudWatchMetrics.enable!

By default this assumes you're running on an EC2 instance with an instance role that can publish CloudWatch metrics, or that you've supplied AWS credentials through environment variables that aws-sdk expects. You can also explicitly supply an aws-sdk CloudWatch Client instance:

Sidekiq::CloudWatchMetrics.enable!(client: Aws::CloudWatch::Client.new)

The default namespace for metrics is "Sidekiq". You can configure this with the namespace option:

Sidekiq::CloudWatchMetrics.enable!(client: Aws::CloudWatch::Client.new, namespace: "Sidekiq-Staging")

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/sj26/sidekiq-cloudwatchmetrics.

License

The gem is available as open source under the terms of the MIT License.

sidekiq-cloudwatchmetrics's Issues

Random crashes when deploying application

I'm seeing some random crashes in production when I deploy my app. It only seems to happen for a few of my puma containers, and they are all able to start eventually without crashing. I'm getting the error: "ArgumentError: expected :endpoint to be a HTTP or HTTPS endpoint"

2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | [131] ! Unable to load application: ArgumentError: expected :endpoint to be a HTTP or HTTPS endpoint
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-core-3.178.0/lib/seahorse/client/base.rb:81:in `block in after_initialize'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-core-3.178.0/lib/seahorse/client/plugins/endpoint.rb:39:in `after_initialize': expected :endpoint to be a HTTP or HTTPS endpoint (ArgumentError)
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | bundler: failed to load command: puma (/usr/local/bundle/ruby/2.7.0/bin/puma)
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-core-3.178.0/lib/seahorse/client/base.rb:80:in `each'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-core-3.178.0/lib/seahorse/client/base.rb:80:in `after_initialize'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-core-3.178.0/lib/seahorse/client/base.rb:24:in `initialize'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-cloudwatch-1.78.0/lib/aws-sdk-cloudwatch/client.rb:386:in `initialize'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/aws-sdk-core-3.178.0/lib/seahorse/client/base.rb:102:in `new'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /app/config/initializers/_sidekiq.rb:94:in `<main>'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
2023-09-28T00:05:57Z service/web/web-79df44bcbd-jw2hx 00:05:57 web.1     | 	from /usr/local/bundle/ruby/2.7.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:326:in `block in load'

This is my code in config/initializers/_sidekiq.rb:

if ENV['AWS_CLOUDWATCH_ACCESS_KEY_ID'].present? &&
   ENV['AWS_CLOUDWATCH_ACCESS_KEY_SECRET'].present? &&
   ENV['AWS_CLOUDWATCH_REGION'].present?
  require 'sidekiq/cloudwatchmetrics'
  # Send Sidekiq job metrics to AWS CloudwWtch (used for autoscaling in Convox)
  # https://github.com/sj26/sidekiq-cloudwatchmetrics
  # IAM Role: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html
  # Role: CloudWatchAgentServerPolicy
  cloudwatch_client = Aws::CloudWatch::Client.new(
    region: ENV.fetch('AWS_CLOUDWATCH_REGION', nil),
    credentials: Aws::Credentials.new(
      ENV.fetch('AWS_CLOUDWATCH_ACCESS_KEY_ID', nil),
      ENV.fetch('AWS_CLOUDWATCH_ACCESS_KEY_SECRET', nil)
    )
  )
  Sidekiq::CloudWatchMetrics.enable!(
    client: cloudwatch_client,
    namespace: ENV['AWS_CLOUDWATCH_SIDEKIQ_NAMESPACE'] || 'Sidekiq'
  )
end

There's some kind of weird race condition that I can't figure out. I don't really understand why it crashes and then succeeds later when restarting the container.

It's crashing in this method:

        def after_initialize(client)
          endpoint = client.config.endpoint
          if endpoint.nil?
            msg = "missing required option `:endpoint'"
            raise ArgumentError, msg
          end

          endpoint = URI.parse(endpoint.to_s)
          if URI::HTTPS === endpoint or URI::HTTP === endpoint
            client.config.endpoint = endpoint
          else
            msg = 'expected :endpoint to be a HTTP or HTTPS endpoint'
            raise ArgumentError, msg
          end
        end

If I step through it manually on my local machine:

[2] pry(main)> cloudwatch_client.config.endpoint
#<URI::HTTPS https://monitoring.us-east-1.amazonaws.com>
[3] pry(main)> endpoint = URI.parse(cloudwatch_client.config.endpoint.to_s)
#<URI::HTTPS https://monitoring.us-east-1.amazonaws.com>
[4] pry(main)> URI::HTTPS === endpoint
true

It's very confusing. If my AWS_CLOUDWATCH_REGION env var is unset or an empty string then this code doesn't run at all.

I'll deploy this change and see if I can get some more info:

begin
    cloudwatch_client = Aws::CloudWatch::Client.new(
      region: ENV.fetch('AWS_CLOUDWATCH_REGION', nil),
      credentials: Aws::Credentials.new(
        ENV.fetch('AWS_CLOUDWATCH_ACCESS_KEY_ID', nil),
        ENV.fetch('AWS_CLOUDWATCH_ACCESS_KEY_SECRET', nil)
      )
    )
    Sidekiq::CloudWatchMetrics.enable!(
      client: cloudwatch_client,
      namespace: ENV['AWS_CLOUDWATCH_SIDEKIQ_NAMESPACE'] || 'Sidekiq'
    )
  rescue ArgumentError => e
    Rails.logger.info("Error enabling Sidekiq::CloudWatchMetrics: #{e.message}")
    Sentry.capture_exception(e)

    endpoint = cloudwatch_client.config.endpoint
    Sentry.capture_message("CloudWatch endpoint causing error: '#{endpoint}'")
  end

The value NaN for parameter MetricData.member.11.Value is invalid

I got this error and it contains little detail.

Aws::CloudWatch::Errors::InvalidParameterValue: The value NaN for parameter MetricData.member.11.Value is invalid.

It says that it errors out on member 11, which is probably the Utilization metric.

My guess is that the process["busy"] and process["concurrency"] were zero's - then we have this situation:

0 / 0.to_f
=> NaN

The question is why and when they would report zero's?

Aws::CloudWatch::Errors::InvalidParameterValue: The value NaN for parameter MetricData.member.11.Value is invalid.
-16 non-project frames
1
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/seahorse/client/plugins/raise_response_errors.rb" line 17 in call
2
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb" line 22 in call
3
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/aws-sdk-core/plugins/idempotency_token.rb" line 19 in call
4
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/aws-sdk-core/plugins/param_converter.rb" line 26 in call
5
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/seahorse/client/plugins/request_callback.rb" line 71 in call
6
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/aws-sdk-core/plugins/response_paging.rb" line 12 in call
7
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/seahorse/client/plugins/response_target.rb" line 24 in call
8
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-core-3.114.0/lib/seahorse/client/request.rb" line 72 in send_request
9
File "/home/app/vendor/cache/ruby/2.7.0/gems/aws-sdk-cloudwatch-1.51.0/lib/aws-sdk-cloudwatch/client.rb" line 2928 in put_metric_data
10
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb" line 181 in block in publish
11
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb" line 180 in each
12
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb" line 180 in each_slice
13
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb" line 180 in publish
14
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb" line 64 in run
15
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/util.rb" line 43 in watchdog
16
File "/home/app/vendor/cache/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/util.rb" line 52 in block in safe_thread

Metrics being collected with gaps between each time period

Hi everyone!

I added the sidekiq-cloudwatchmetrics gem to our code and followed the instructions you provided in the Readme. The metrics started to appear on CloudWatch, but there are gaps between each time period, which doesn’t seem to be normal... Ex:

If you see in the image above, we have metrics at 13:27h, between 13:45 to 14h and around 14:34. Aside from these time periods, we don’t have the metrics.

Our app is running on an ECS Fargate Cluster, this is the sidekiq.rb file:

require "sidekiq"
require "sidekiq/cloudwatchmetrics"

Sidekiq::CloudWatchMetrics.enable!

Maybe there is something missing in my config, can you help with that? Thanks!

Feature request - Track the number of jobs processed for each job class

Would it be possible to add metrics for each separate job? I'd like to set up a dashboard where I track the number of processed / errored / failed jobs for a specific job class. Or is there a way I can use this gem to track these metrics as well? Thanks!

Ruby 2.3.5 compatibility

Hey there, would you consider relaxing the ruby version dependency? Not really sure why 2.4 is required.

sidekiq-cloudwatchmetrics incompatible with sidekiq 6.5

Upgrading to sidekiq 6.5.0 / sidekiq-pro 5.5.0 / sidekiq-ent 2.5.0 breaks sidekiq-cloudwatchmetric with the following error.

See sidekiq/sidekiq#5372 for similar issues raised by a number of other gems that depend on sidekiq and require sidekiq/util.

Sidekiq 6 compatibility when using "additional_dimensions" option

Hi Samuel - thanks for what looks like a great Gem.

Would you be able to create a new official 2.* series gem that matches current 'master', with support for sidekiq 6? I think merging https://github.com/sj26/sidekiq-cloudwatchmetrics/pull/9/files and bumping rubygem version to a version number > 2.0.0 should do it.

Thanks for any time you spend on this - really appreciate it. Don't hesitate to let me know if I can help in any way

Context:

We're using Sidekiq 6 - and while it looks like sidekiq-cloudwatchmetrics supports it from version 2.0.0 and above, I get an error when trying to use the additional_dimensions flag.

From what I can see, 2.0.0 doesn't include this commit: 332a43a Version 2.0.0 isn't reflected in this repo's tags list.

Relevent error:

ArgumentError: unknown keyword: :additional_dimensions
/Users/oskar/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb:4

The sidekiq-cloudwatchmetrics-2.0.0/lib/sidekiq/cloudwatchmetrics.rb file contains this, which doesn't use kwargs

    def initialize(client: Aws::CloudWatch::Client.new)
      @client = client
    end

Versions 1.* are limited to sidekiq 5 as per https://github.com/sj26/sidekiq-cloudwatchmetrics/blob/master/sidekiq-cloudwatchmetrics.gemspec#L22 Attempting to force a 1.0 series into my gemfile returned this error:

  In Gemfile:
    sidekiq-cloudwatchmetrics (~> 1) was resolved to 1.2.0, which depends on
      sidekiq (~> 5.0)

    sidekiq-pro was resolved to 5.2.1, which depends on
      sidekiq (>= 6.1.0)

Sidekiq 4 compatibility

Hi there, any reason for making this gem sidekiq 5 only? It looks like you're not really using any 5-only features. Would you consider relaxing the dependency?

The parameter MetricData.member.12.Dimensions.member.1.Value is required. (Aws::CloudWatch::Errors::MissingParameter)

Hey @sj26
this gem is not working with aws-sdk-cloudwatch

Not all metrics being reported

Hey @sj26 ! Hope you are doing great!

Apparently not all of the sidekiq metrics are being sent to CloudWatch. I see that some queues are always zeroed:

While in the logs I see that sidekiq is processing the messages:

The weird thing is that from the logs everything is fine:
2023-08-04T19:12:15.412Z pid=1 tid=39d1 INFO: Publishing Sidekiq CloudWatch Metrics
and metrics for some other queues are being properly sent to CloudWatch, it's like if it was working intermittently.

These are the versions we are using:
sidekiq (6.3.1)
sidekiq-cloudwatchmetrics (2.5.0)

This is how we are initilizing the gem:

require "sidekiq"
require "sidekiq/cloudwatchmetrics"
Sidekiq::CloudWatchMetrics.enable!(namespace: "Sidekiq-" + ENV["ENVIRONMENT"])

Can you help to identify what might be happening? 🙏🏻
Thanks and let me know if you need any additional info!

The parameter MetricData.member.12.Dimensions.member.1.Value is required. (Aws::CloudWatch::Errors::MissingParameter)

Hey @sj26
I have following gems
sidekiq-cloudwatchmetrics (2.6.0)
sidekiq (7.2.0, 7.0.7, 6.4.2)
aws-sdk-cloudwatch (1.81.0, 1.67.0)

and I get the following error on the CloudWatch console.

53
2023-11-01T16:36:01.871-07:00
#<Thread:0x00007f5f4f1d1b30@cloudwatch metrics publisher /usr/local/bundle/gems/sidekiq-6.4.2/lib/sidekiq/util.rb:63 run> terminated with exception (report_on_exception is true):
54
2023-11-01T16:36:01.871-07:00
/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': The parameter MetricData.member.12.Dimensions.member.1.Value is required. (Aws::CloudWatch::Errors::MissingParameter)
55
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
56
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
57
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
58
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
59
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
60
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
61
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
62
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/request.rb:72:in `send_request'
63
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/aws-sdk-cloudwatch-1.67.0/lib/aws-sdk-cloudwatch/client.rb:3365:in `put_metric_data'
64
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-cloudwatchmetrics-2.6.0/lib/sidekiq/cloudwatchmetrics.rb:235:in `block in publish'
65
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-cloudwatchmetrics-2.6.0/lib/sidekiq/cloudwatchmetrics.rb:234:in `each'
66
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-cloudwatchmetrics-2.6.0/lib/sidekiq/cloudwatchmetrics.rb:234:in `each_slice'
67
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-cloudwatchmetrics-2.6.0/lib/sidekiq/cloudwatchmetrics.rb:234:in `publish'
68
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-cloudwatchmetrics-2.6.0/lib/sidekiq/cloudwatchmetrics.rb:77:in `run'
69
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-6.4.2/lib/sidekiq/util.rb:56:in `watchdog'
70
2023-11-01T16:36:01.871-07:00
from /usr/local/bundle/gems/sidekiq-6.4.2/lib/sidekiq/util.rb:65:in `block in safe_thread'

Custom Namespaces

Love the gem, we are using it for autoscaling worker instances based on queue depth.

One challenge that we have encountered is this doesn't really play well with multiple environments, or multiple different types of worker instance since metrics are published to the same namespace.

I'd love a way to customize this on initialization, so for example:

Sidekiq::CloudWatchMetrics.enable!(client: ..., namespace: "Sidekiq-Staging")