dryruby / rack-throttle Goto Github PK

Hi,

I implementing throttling on one of my websites and a client has gone over the threshold and been rate limited. I've increased the throttling limits, however the client is still rate limited a day later.

I tried restarting apache (using Passenger) but still get the error. How do I 'reset' the counter and allow this client to start fresh?

Thanks 😄

I have installed the gem in my Rails object, and trying to customize the response to make it return a application/json I have found out that even if bundle install has installed me the last version 0.4.0, the code is not the same of the repo.

here is the code on my pc:

    ##
    # Outputs an HTTP `4xx` or `5xx` response.
    #
    # @param  [Integer]                code
    # @param  [String, #to_s]          message
    # @param  [Hash{String => String}] headers
    # @return [Array(Integer, Hash, #each)]
    def http_error(code, message = nil, headers = {})
      [code, {'Content-Type' => 'text/plain; charset=utf-8'}.merge(headers),
        [http_status(code), (message.nil? ? "\n" : " (#{message})\n")]]
    end

but the code of this repo is:

    ##
    # Outputs an HTTP `4xx` or `5xx` response.
    #
    # @param  [Integer]                code
    # @param  [String, #to_s]          message
    # @param  [Hash{String => String}] headers
    # @return [Array(Integer, Hash, #each)]
    def http_error(code, message = nil, headers = {})
       contentType = 'text/plain; charset=utf-8'
      if options[:type]
        contentType = options[:type]
      end
      [code, {'Content-Type' => 'text/plain; charset=utf-8'}.merge(headers),
        [http_status(code), (message.nil? ? "\n" : " (#{message})\n")]]
    end

How is this possible?

Is there maybe an error with the versioning?

I may be missing something but it seems strange that

 def allowed?(request)
      t1 = request_start_time(request)
      t0 = cache_get(key = cache_key(request)) rescue nil
      allowed = !t0 || (dt = t1 - t0.to_f) >= minimum_interval
      begin
        cache_set(key, t1)
        allowed
      rescue => e
        # If an error occurred while trying to update the timestamp stored
        # in the cache, we will fall back to allowing the request through.
        # This prevents the Rack application blowing up merely due to a
        # backend cache server (Memcached, Redis, etc.) being offline.
        allowed = true
      end
    end

in interval.rb doesn't call the super implementation in limiter.rb

def allowed?(request)
  case
    when whitelisted?(request) then true
    when blacklisted?(request) then false
    else true # override in subclasses
  end
end

Hi.

We recently upgraded from the dalli gem to mem_cache_store. This silently broke the rack-throttle gem since it expects the cache to support :has_key?, :get, and :set.

To fix it, we applied this monkey patch...

  class RackThrottleMemCacheStore < ActiveSupport::Cache::MemCacheStore
    alias_method :has_key?, :exist?
    alias_method :get, :read
    alias_method :set, :write
  end

  # Limits :max requests per minute by session id
  cache_servers   = Rails.application.config.cache_store[1...-1]
  cache_options   = Rails.application.config.cache_store.last
  memcache_store  = RackThrottleMemCacheStore.new(cache_servers, cache_options)

  Rails.application.config.middleware.use Rack::Throttle::Minute,
                                          max:        60,
                                          key_prefix: :throttle,
                                          cache:      memcache_store

I'm not sure if there's a better way but sharing this tidbit for others who may be running this with dalli and then switched to mem_cache_store.

Thanks for this gem!
Owen

Hi.
Basically as title states,

    config.middleware.use Rack::Throttle::Daily,    max: 1000  # requests
    config.middleware.use Rack::Throttle::Hourly,   max: 100   # requests
    config.middleware.use Rack::Throttle::Minute,   max: 40    # requests
    config.middleware.use Rack::Throttle::Interval, min: 3.0   # seconds
    # NameError: uninitialized constant Rack::Throttle::Second
    # config.middleware.use Rack::Throttle::Second,   max: 1     # requests

config.middleware.use Rack::Throttle::Second is throwing

NameError: uninitialized constant Rack::Throttle::Second

Rails 4.2.7
Ruby 2.3.0

When throttling, it might be more accurate to return the following status code:

429 Too Many Requests

The user has sent too many requests in a given amount of time.

This is apparently intended for use with rate limiting schemes.

I realise that defaulting to this error code would be a backwards incompatible, but I think this should be documented. I can provide a PR updating the README.

Related: #5 (which changed the default status code, rather than simply documenting it)

Under the section of the Readme entitled "Combining various throttling constraints into one overall policy", the following example is provided:

use Rack::Throttle::Daily,    :max => 1000  # requests
use Rack::Throttle::Hourly,   :max => 100   # requests
use Rack::Throttle::Minute,   :max => 60    # requests
use Rack::Throttle::Second,   :max => 1     # requests
use Rack::Throttle::Interval, :min => 3.0   # seconds

The only problem with this approach is that the cache keys for the same second, minute, etc can clash.

For example, consider that you have defined the throttling policy exactly as described above. Now consider the following:

• At 12:11:10 PM, assume that throttle:10.0.2.2:2016-10-05T12:11 (the current minute) is set to 59.
• At 12:11:10 PM, a request comes in from 10.0.2.2, which is allowed, because the max is 60 per minute. The key is now incremented to 60.
• At 12:11:11 PM, a request comes in from 10.0.2.2, which is blocked because the key for the per-second policy is the same as the minute policy -- throttle:10.0.2.2:2016-10-05T12:11 -- which is currently 30, greatly exceeding the maximum of 1 per second.

The same thing happens at 12:12:00 PM, if the per-hour counter is higher than the max for the per-minute counter.

This looks like just a copy & paste error with the cache key for Rack::Throttle::Second.

Ideally, these keys would be distinct.

Grape is not too different from Sinatra and it sits on Rack, it would be awesome if there was support offered for Grape as well :)

Hi

I'm using Rails 4, added custom lib that uses rack-throttle.
So, deploying failing with this notice

"No such middleware to insert after: Rack::Lock"

What is the workaround?
Thanks

Describe the bug

This is a minor bug I noticed today, and it likely won't affect many people. When running solargraph bundle locally, I got an error while building the yard docs for rack-throttle 0.7.0.

No such file or directory @ rb_sysopen - README (Errno::ENOENT)

After doing some digging, I noticed the README in the current published version is a symlink to a file that doesn't exist in the gem (README -> README.md). I couldn't confirm this immediately because of two other minor housekeeping issues:

• Rubygems.org probably needs an update (the gem still points to bendiken/rack-throttle, which correctly forwards to this repo) to reduce confusion
• Published gem versions after 0.5.0 don't appear tagged, or as releases on this repo (this was confusing, coupled with the item above I wasn't sure if there was 2 rack-throttles or not)

To reproduce
Steps to reproduce the behavior:

I was running solargraph bundle on a repository that had the rack-throttle in the Gemfile, but I think this probably affects anyone trying to builds the docs w/ yard

1. Download the build gem from here: https://rubygems.org/downloads/rack-throttle-0.7.0.gem
2. Extract/open view it however you would normally (I just unzipped everything)
3. Observe a README symlink, but no README.md

Expected behavior

This appears to be fixed in latest master, but the the latest published version isn't correct

Additional context

I don't think there's anything I can contribute, PR wise, since it appears fixed in master but I'm happy to help validate after the new version is published. I can certainly point to master for now but it would be nice to see a new release published & the links updated (and versions tagged in github also)

The 0.4.0 version of the gem available at https://rubygems.org/gems/rack-throttle doesn't have per second limit constraint. But in the github repo, it is available. Can you please add a tag for the current version in the github repo so that I can use the github source directly in my project's gemfile?

Hi, was trying to use rack-throttle on Interval 3sec, and it works just fine until you hit the threshold, in which it errors out with:

[2011-02-04 21:00:37] ERROR Rack::Lint::LintError: Response body must respond to each
/Users/kris/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rack-1.2.1/lib/rack/lint.rb:19:in assert' /Users/kris/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rack-1.2.1/lib/rack/lint.rb:513:ineach'
/Users/kris/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rack-1.2.1/lib/rack/handler/webrick.rb:64:in service' /Users/kris/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/webrick/httpserver.rb:111:inservice'
/Users/kris/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/webrick/httpserver.rb:70:in run' /Users/kris/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/webrick/server.rb:183:inblock in start_thread'

I'm seeing what looks like a concurrency issue in a custom Limiter based on rack-throttle.

But I'm not 100% sure, so thought I'd ask issue here.

Specifically, the retrieval and storage of count values in time_window (for example) is not atomic, so it is possible for two simultaneous increments to occur but one of the increments will not be recorded.

This is what I'm seeing in my logs:
I, [2014-08-14T16:49:04.491000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 11/15 => allowed? true
I, [2014-08-14T16:49:05.201000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 12/15 => allowed? true
I, [2014-08-14T16:49:05.799000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 13/15 => allowed? true
I, [2014-08-14T16:49:05.806000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 13/15 => allowed? true
I, [2014-08-14T16:49:07.001000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 14/15 => allowed? true
I, [2014-08-14T16:49:08.630000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 15/15 => allowed? true
I, [2014-08-14T16:49:10.463000 #5547] INFO -- : Rate limiting: 6061dcae08b043d9889169bd9f153fdc-d10de8798db7ced7b5850c6d7000f339:2014-08-14T16 count = 16/15 => allowed? false

The cache is a remote redis, being accessed via Ohm. Less than optimal, but I'm picking up someone else's code.

Also this is running in JRuby, and there are both hourly and daily limits in play.

The ruby threading model is not my forte, so I might be missing something obvious.

my web application has two parts, one for web, one for api.

I want to different interval rate-limiting for the them

api : use Rack::Throttle::Hourly, :max => 100 # requests
web : use Rack::Throttle::Interval, :min => 1.0 # seconds

It seems like the per-second / per-minute / etc. keys that rack-throttle creates in Redis are permanent. We run a high traffic site, which means that we're going to end up with a lot of pollution and wasted space in Redis. Ideally, there should be some way for these keys to expire.

Working on rack-dedos for some apps deployed to render.com (which uses Cloudflare by default), I bumped into a flaw in Rack:

Despite there being code in Rack (at least in Rack 3) which should determine the real client IP of the request not only by looking at the X-Remote-Addr but also at the X-Forwarded-For header, all versions of Rack up to 3.0.7 report the non-public IPs thru request.ip. See these remarks for details.

TL;DR request.ip is not reliable and if rack-throttle is used on an app deployed to render.com and maybe other providers, the throttling will be catastrophic since it's based on the private IP behind the proxy and not the real client IP.

This should of course be fixed on Rack, but since it's such a central piece and most notably Rails is lagging behind on Rack versions (Rails 7 still requires Rack 2), gems which perform traffic shaping will have to deal with it themselves.

Here's how I did it: https://github.com/svoop/rack-dedos/blob/main/lib/rack/dedos/filters/base.rb#L40

Are API requests not being a subject to throttling?

I've created custom limiter, but it is not being hit while running my API specs.

    module RpmLimits
      class LimitPerMinute < ::Rack::Throttle::Minute
        def allowed?(request)
          raise # never happpens
          path_info = Rails.application.routes.recognize_path(request.url) || {}

          path_info[:controller].include?('api') ? super : true
        end
      end
    end

    # application.rb
    config.middleware.use ::RpmLimits::LimitPerMinute, max: 1

When I run other tests exception is thrown each time.

Hello All!

I wanted to open discussion to see if RackAttack could serve the wonderful people using this lovely gem a bit better. https://github.com/kickstarter/rack-attack

It seems to hame more features/functionality/community than here at rack-throttle & I'm thinking we may want to give up the throne to help peoples looking for this functionality & have come here :)

@bendiken @brandonhilkert + any others, thoughts?

Would be swell :-)

In Second#cache_key method, minute directive is missing. This caused some incorrect cache hits when run in same second of same hour.

Current: [super, Time.now.strftime('%Y-%m-%dT%H:%S')].join(':')

Correct: [super, Time.now.strftime('%Y-%m-%dT%H:%M:%S')].join(':')

Is it possible to change the response to be totally custom (JSON, for example) like it is in redis-throttle?

Instead of IP address, I would like to use the user_id to establish the limits.
From the documentation I got this:

"If you wish to instead use a more granular, application-specific identifier such as a session key or a user account name, you need only subclass a throttling strategy implementation and override the #client_identifier method."

However I don't understand how to do it..

Thanks

gem 'rack-throttle', '~> 0.4.2'

when i run i am getting this error, can you please provide solution for this

/data/project/vendor/bundle/ruby/2.2.0/gems/rack-throttle-0.4.2/lib/rack/throttle/time_window.rb:3:in <module:Throttle>': uninitialized constant Rack::Throttle::Limiter (NameError) from /data/project/vendor/bundle/ruby/2.2.0/gems/rack-throttle-0.4.2/lib/rack/throttle/time_window.rb:1:in module:Rack'
from /data/project/vendor/bundle/ruby/2.2.0/gems/rack-throttle-0.4.2/lib/rack/throttle/time_window.rb:1:in <top (required)>' from /data/project/vendor/bundle/ruby/2.2.0/gems/backports-3.6.8/lib/backports/std_lib.rb:9:in require'
from /data/project/vendor/bundle/ruby/2.2.0/gems/backports-3.6.8/lib/backports/std_lib.rb:9:in require_with_backports' from /data/project/lib/app/rate_limiter/request_throttle.rb:1:in <top (required)>'
from /data/project/vendor/bundle/ruby/2.2.0/gems/backports-3.6.8/lib/backports/std_lib.rb:9:in require' from /data/project/vendor/bundle/ruby/2.2.0/gems/backports-3.6.8/lib/backports/std_lib.rb:9:in require_with_backports'
from config.ru:11:in block in <main>' from /data/project/vendor/bundle/ruby/2.2.0/gems/rack-1.5.5/lib/rack/builder.rb:55:in instance_eval'
from /data/project/vendor/bundle/ruby/2.2.0/gems/rack-1.5.5/lib/rack/builder.rb:55:in initialize' from config.ru:1:in new'
from config.ru:1:in <main>' from /data/project/vendor/bundle/ruby/2.2.0/gems/unicorn-4.6.3/lib/unicorn.rb:48:in eval'
from /data/project/vendor/bundle/ruby/2.2.0/gems/unicorn-4.6.3/lib/unicorn.rb:48:in block in builder' from /data/project/vendor/bundle/ruby/2.2.0/gems/unicorn-4.6.3/lib/unicorn/http_server.rb:722:in call'
from /data/project/vendor/bundle/ruby/2.2.0/gems/unicorn-4.6.3/lib/unicorn/http_server.rb:722:in build_app!' from /data/project/vendor/bundle/ruby/2.2.0/gems/unicorn-4.6.3/lib/unicorn/http_server.rb:140:in start'
from /data/project/vendor/bundle/ruby/2.2.0/gems/unicorn-4.6.3/bin/unicorn:126:in <top (required)>' from /data/project/vendor/bundle/ruby/2.2.0/bin/unicorn:23:in load'
from /data/project/vendor/bundle/ruby/2.2.0/bin/unicorn:23:in `