Coder Social home page Coder Social logo

laravel-queue-rate-limit's Introduction

laravel-queue-rate-limit

Current version Monthly Downloads Total Downloads Build Status

Simple Laravel queue rate limiting

Installation

3.* versions are compatible only with Laravel 7+.

$ composer require mxl/laravel-queue-rate-limit

For Laravel 6 use 2.* versions:

$ composer require mxl/laravel-queue-rate-limit "^2.0"

For Laravel 5 use 1.* versions:

$ composer require mxl/laravel-queue-rate-limit "^1.0"

Laravel 5.5+ will use the auto-discovery feature to add MichaelLedin\LaravelQueueRateLimit\QueueServiceProvider::class to providers.

This package is not compatible with older Laravel versions.

Add rate limits to config/queue.php:

'rateLimits' => [
     'mail' => [ // queue name
        'allows' => 1, // 1 job
        'every' => 5 // per 5 seconds
     ]
]

Usage

Make sure that you don't use sync connection when queueing jobs. See default property in config/queue.php.

Run queue worker:

$ php artisan queue:work --queue default,mail

Then push several jobs to default and mail queues:

Mail::queue(..., 'mail');
Mail::queue(..., 'mail');
Mail::queue(..., 'mail');
Mail::queue(..., 'default');
Mail::queue(..., 'default');

You'll see that only mail queue jobs will be rate limited while default queue jobs will run normally.

Disable logging

Extend QueueServiceProvider:

<?php

namespace App\Providers;

class QueueServiceProvider extends \MichaelLedin\LaravelQueueRateLimit\QueueServiceProvider
{
    protected function registerLogger()
    {
        $this->app->singleton('queue.logger', function () {
            return null;
        });
    }
}

Add it to providers array in config/app.php:

<?php

return [
    // ...
    'providers' => [
        // Laravel Framework Service Providers
        // ...
        // Application Service Providers
        // ...
        App\Providers\QueueServiceProvider::class,
        // ...
    ]
];

Maintainers

Other useful Laravel packages from the author

License

See the LICENSE file for details.

laravel-queue-rate-limit's People

Contributors

hettiger avatar laravel-shift avatar mikkoaf avatar mxl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

laravel-queue-rate-limit's Issues

Laravel 7 Support

Dependency issues:

- mxl/laravel-queue-rate-limit v2.0.1 requires illuminate/queue ^6.0 -> satisfiable by laravel/framework[v6.18.0, 6.x-dev], illuminate/queue[6.x-dev, v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.10.0, v6.11.0, v6.12.0, v6.13.0, v6.13.1, v6.14.0, v6.15.0, v6.15.1, v6.16.0, v6.17.0, v6.17.1, v6.18.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0, v6.5.1, v6.5.2, v6.6.0, v6.6.1, v6.6.2, v6.7.0, v6.8.0].
- mxl/laravel-queue-rate-limit v2.0.0 requires illuminate/log ^6.0 -> satisfiable by laravel/framework[v6.18.0, 6.x-dev], illuminate/log[6.x-dev, v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.10.0, v6.11.0, v6.12.0, v6.13.0, v6.13.1, v6.14.0, v6.15.0, v6.15.1, v6.16.0, v6.17.0, v6.17.1, v6.18.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0, v6.5.1, v6.5.2, v6.6.0, v6.6.1, v6.6.2, v6.7.0, v6.8.0].
- mxl/laravel-queue-rate-limit v2.0.1 requires illuminate/log ^6.0 -> satisfiable by laravel/framework[v6.18.0, 6.x-dev], illuminate/log[6.x-dev, v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.10.0, v6.11.0, v6.12.0, v6.13.0, v6.13.1, v6.14.0, v6.15.0, v6.15.1, v6.16.0, v6.17.0, v6.17.1, v6.18.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0, v6.5.1, v6.5.2, v6.6.0, v6.6.1, v6.6.2, v6.7.0, v6.8.0].

Logging Issue

Hello , thanks for this wonderful package . I have issue with the logging, despite creating the QueueServiceProvider in Provider folder and also add it in app.php, i am still getting the logs. is there anything more to do ?

I am using laravel 6.x
Thank you

Laravel 9 Support

There are following dependency issues:

$ composer why-not laravel/framework "^9.0"
mxl/laravel-queue-rate-limit               v3.2.0    requires          illuminate/cache (^7.0||^8.0)
mxl/laravel-queue-rate-limit               v3.2.0    requires          illuminate/contracts (^7.0||^8.0)
mxl/laravel-queue-rate-limit               v3.2.0    requires          illuminate/log (^7.0||^8.0)
mxl/laravel-queue-rate-limit               v3.2.0    requires          illuminate/queue (^7.0||^8.0)

Laravel 10 install error

I'm trying to upgrade a Laravel 9 project to Laravel 10. when I install this package I get:

Problem 1
    - mxl/laravel-queue-rate-limit[v1.0.0, ..., v1.0.2] require illuminate/queue ^5.5 -> found illuminate/queue[v5.5.0, ..., v5.8.36] but these were not loaded, likely because it conflicts with another require.
    - mxl/laravel-queue-rate-limit[v2.0.0, ..., v2.0.1] require illuminate/queue ^6.0 -> found illuminate/queue[v6.0.0, ..., v6.20.44] but these were not loaded, likely because it conflicts with another require.
    - mxl/laravel-queue-rate-limit v3.0.0 requires illuminate/queue ^7.0 -> found illuminate/queue[v7.0.0, ..., v7.30.6] but these were not loaded, likely because it conflicts with another require.
    - mxl/laravel-queue-rate-limit v3.1.0 requires php ^7.3 -> your php version (8.1.0) does not satisfy that requirement.
    - mxl/laravel-queue-rate-limit v3.2.0 requires illuminate/queue ^7.0||^8.0 -> found illuminate/queue[v7.0.0, ..., v7.30.6, v8.0.0, ..., v8.83.27] but these were not loaded, likely because it conflicts with another require.
    - mxl/laravel-queue-rate-limit v3.3.0 requires illuminate/queue ^7.0||^8.0||^9.0 -> found illuminate/queue[v7.0.0, ..., v7.30.6, v8.0.0, ..., v8.83.27, v9.0.0, ..., v9.52.9] but these were not loaded, likely because it conflicts with another require.
    - Root composer.json requires mxl/laravel-queue-rate-limit * -> satisfiable by mxl/laravel-queue-rate-limit[v1.0.0, v1.0.1, v1.0.2, v2.0.0, v2.0.1, v3.0.0, v3.1.0, v3.2.0, v3.3.0].

You can also try re-running composer require with an explicit version constraint, e.g. "composer require mxl/laravel-queue-rate-limit:*" to figure out if any version is installable, or "composer require mxl/laravel-queue-rate-limit:^2.1" if you know which you need.

Laravel 6 please

Tried to create PR for Laravel 6 but get an error on the Worker.
__construct needs a callable $isDownForMaintenance

parent::__construct($manager, $events, $exceptions);

any suggestions?

/**
     * Create a new queue worker.
     *
     * @param  \Illuminate\Contracts\Queue\Factory $manager
     * @param  \Illuminate\Contracts\Events\Dispatcher  $events
     * @param  \Illuminate\Contracts\Debug\ExceptionHandler  $exceptions
     * @param  \callable $isDownForMaintenance
     * @return void
     */
    public function __construct(QueueManager $manager,
                                Dispatcher $events,
                                ExceptionHandler $exceptions,
                                callable $isDownForMaintenance)
    {
        $this->events = $events;
        $this->manager = $manager;
        $this->exceptions = $exceptions;
        $this->isDownForMaintenance = $isDownForMaintenance;
    }

Rate job limiting doesn't seem to work

I configured the rate limit like this:
'rateLimits' => [
'queue' => [
'allows' => 1,
'every' => 60
],

If I understand correctly the job should be staying in the database table "jobs" for 60 seconds before they are taken out of the database job queue? But the Available_at and Created_at always have the same value and the job seems to be taken out very quickly not respecting the rate limit.

I tried to set it to a very long delay but it would not stay in the job table, they are just quickly taken out.

This is what I see in the table.
Payload:
{"uuid":"4eb7a0a5-1404-4186-af37-b04bd2f779d1","displayName":"App\Jobs\PaymentJob","job":"Illuminate\Queue\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"delay":null,"timeout":null,"timeoutAt":null,"data":{"commandName":"App\Jobs\PaymentJob","command":"O:19:"App\Jobs\PaymentJob":9:{s:10:"\u0000*\u0000details";a:1:{s:4:"data";s:9:"something";}s:3:"job";N;s:10:"connection";N;s:5:"queue";N;s:15:"chainConnection";N;s:10:"chainQueue";N;s:5:"delay";N;s:10:"middleware";a:0:{}s:7:"chained";a:0:{}}"}} 0 1617912273 1617912273
Attempts: 0
Reserved_at: null
available_at: 1617912273
created_at: 1617912273

Worker exits when used with --stop-when-empty flag

First of all, thank you for this nice package which is one of its kind when it comes to rate limiting without Redis.

I am not sure whether this is really an issue which can be solved by this package of not. But I want to log this information (even if you prefer to close it as out of scope) here as it will be helpful for other users of the package.

So, I am using worker in a mode where it exits when queue is empty. Like mentioned in laravel doc here https://laravel.com/docs/7.x/queues#running-the-queue-worker

php artisan queue:work --stop-when-empty

Now, let's say we have 10 items in the mail queue and mail queue is rate limited like this :

'rateLimits' => [
     'mail' => [ // queue name
        'allows' => 1, // 1 job
        'every' => 5 // per 5 seconds
     ]
]

Now as soon as worker processes the first job & first job got complete within 2 seconds. Now, we have 9 jobs pending but just because of rate limit, none of job can be picked up within 5 seconds. This sends worker a signal that queue is empty and worker get terminated! Ideally, queue is yet not empty and work should resume the queue after 5 seconds.

Now for those who are using this package along with worker invoked from laravel scheduler like this (I know it should be from supervisor instead, but let's say due to some limitations):

$schedule->command('queue:work --stop-when-empty --timeout=120 --tries=3')->everyMinute();

They will get an impression like rateLimiter is limiting to 1 minute instead of seconds.. This is because worker process get terminated & it get invoked after 1 minutes (or with interval you have set in scheduler). So, that gives you impression like something is wrong with package configuration where every parameter is in minutes instead of seconds :)

Possible work around for this is to set rate limits like this :

'rateLimits' => [
        'default' => [ // queue name
            'allows' => 12, // 12 job
            'every' => 60 // per 1 minute
        ]
    ]

Above will make worker to pick 12 jobs per minutes. However, know that this can result in burst of 12 jobs per few seconds which might or might not work for you based on your scenario. (If API supports leaky bucket, it will work. If API has strict only second based limits, it might not work for you)

Worker Sleep Duration

Jobs in a rate limited queue will be limited by the Worker Sleep Duration even thought the sleep duration is supposed to don't be taken into account when there are multiple available jobs in the pipeline. It's a minor inconvenience in my case. (I've just set the Worker Sleep Duration to 1 second.) However this is an unexpected behavior and it makes it impossible to apply a rate limiting configuration where multiple jobs would run per 1 second.

Limit logging

I am having an issue since I updated Laravel 6 to 9 a few weeks ago.
The laravel logs are being filled with these notices:
[2022-05-19 00:00:01] production.DEBUG: Rate limit is set for queue default
[2022-05-19 00:00:01] production.DEBUG: Rate limit check is passed for queue default
[2022-05-19 00:00:01] production.DEBUG: No available jobs on queue default
And it logs every few seconds.
I don't want to disable logging entirely because then I won't see errors that are occurring.
It's just that I only want an error log so that I know that something is wrong ( and then I can fix it).

Thank you for your package - it has really helped out with my outlook exchange mail connections!

Horizon ignores rate limit

The queue:worker does indeed limit the executes jobs in the queue, but when running horizon these jobs are not limited.

Is this something that should be resolved in horizon?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.