sgrondin / bottleneck Goto Github PK

After rewriting my code I am facing an exception thrown in Bottleneck.js:240:
https://i.imgur.com/1zn0sVh.png

Exception has occurred: TypeError
TypeError: Cannot read property 'apply' of undefined
at Object.wrapped (C:\Users\h9pe\Documents\brawlstats.io-express\node_modules\bottleneck\lib\Bottleneck.js:240:21)
at Timeout._onTimeout (C:\Users\h9pe\Documents\brawlstats.io-express\node_modules\bottleneck\lib\Bottleneck.js:180:34)
at ontimeout (timers.js:469:11)
at tryOnTimeout (timers.js:304:5)
at Timer.listOnTimeout (timers.js:264:5)

The biggest issue here is that I have no idea what call/line is causing to throw that exception. In the past I noticed that this would be thrown when I call schedulePriority with something else than a function as first parameter. However I don't use schedulePriority at all anymore so I am stuck how to figure out my issue. Any ideas?

Hi,
I've got an app that uses Bottleneck. It gets stuck after running the fist batch of requests. Could someone please help? Full description of the problem and app code is posted to stackoverflow

Thanks!

Currently using [email protected] and [email protected] and getting the following issue...

~/node_modules/bottleneck/lib/Bottleneck.js:73
      async disconnect(flush = true) {
            ^^^^^^^^^^
SyntaxError: Unexpected identifier
    at Object.exports.runInThisContext (vm.js:76:16)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (~/node_modules/bottleneck/lib/index.js:3:20)
    at Object.<anonymous> (~/node_modules/bottleneck/lib/index.js:5:4)

Should I expect this module to export logic that can run in my environment, or should I tell babel to have at it? Here's my .babelrc for the curious:

{
    "presets": [[
        "env", {
          "targets": {
            "node": "6.10"
          }
        }
    ]],
    "plugins": [
      ["transform-object-rest-spread", {
        "useBuiltIns": true
      }]
    ]
}

First of all, I'm not sure if this is an issue with Bottleneck or Lolex (used by Sinon to stub setTimeout and friends, usually to speed up tests).

If I install Lolex and then use Bottleneck to schedule two asynchronous operations, only the first of them will execute; the other one will stay pending. Internally, Bottleneck uses setTimeout to schedule, so it looks to me like there's a callback that's not getting executed.

I've created a minimal working example. The same thing happens if I use submit with a callback instead of schedule with a Promise, a well as with Bluebird instead of the native Promise object. Also, the version of Node doesn't seem to matter. Finally, I'm on Windows, but I sure hope that's not the cause.

I've tried to dig into the Bottleneck code but wasn't able to come up with anything useful. I did notice that there's only one clearTimeout and it doesn't get called when I run my example.

Thanks!

When using promises, generic Errors are not super helpful.

Based on the current implementation of schedule for instance, it's not easy to tell if you're dealing with a rejection due to bottleneck's queue filling up or if your promise itself failed.

There are a few ways this could be addressed, I think. One is to set a code property a la node.js errors, which should be sufficient. Another would be to use custom error types.

Thoughts?

I use bottleneck in my project and ran for half a month, got problems I had no idea.First I post an image.

In the plot, X is 50 time points, Y is time spent in second,rectangle is total time spent for each request, and diamond is bottleneck wait time.

Rate limit I set is 3s. Obviously, we can see most points of the set are doing good, but 9~10 of them seems unreasonable that is very high in the imagae, Anyone who have same problem?

Thanks for this. It seems to be perfect (for my simple use case) apart from this wart:

If a callback isn't necessary, you must pass null or an empty function instead. It will not work if you forget to do this.

IMO, this is ugly. It pretty much requires that trailing parameter to be commented/explained away. And I'm not sure how to explain it, since the documentation itself doesn't explain it :-) How about adding a method which doesn't require a callback e.g. schedule i.e. allow:

limiter.submit(fn, null);

to be written as:

limiter.schedule(fn);

Hi
I use the lib to do some throttling, and use 'empty' event to know when can I can put some cool down and retry the procedure.

I tried the once() instead of on(), but it is not supported. also tried to use the return value form limiter.on() but thats the limiter instance.

is there a way to fire the event only once ?

Hey, any idea why this isn't working correctly?

import Promise from 'bluebird'
import Bottleneck from 'bottleneck'

const limiter = new Bottleneck(1, 100)
const example = (v) => Promise.delay(100).then(() => v)
const exampleLimited = (...args) => limiter.schedule.apply(null, [example, ...args])

Hi,

Tried implementing this library for the use case of a remote API that has rate limits.

var limiter = new Bottleneck(3, 1000); // 3 per second

This code actually waits a second before firing off request nr. 2.

I'd like it to be able to do 3 requests per second and only wait if the queue is full, that is when run in a loop it would fire the first three requests immediately, and request 4 would go off 1 sec after request 1, etc.

More complex example use case: (remote API has specs: max 3/sec + 200/hr)

var limiter = new Bottleneck(3, 1000); // 3 per second
var hourlyLimit = new Bottleneck(200, 1000*60*60); // 200 per hour
limiter.chain(hourlyLimit);

This currently fires one and then waits an hour for the next call.

Am I missing something here? Thankful for advice.

In earlier releases, setting highWater param to 0, ensured that the queue is not used. Together with OVERFLOW strategy this helped to make sure that submitted requests are all discarded, if there is a running request. In the last version this is not working anymore.

Currently if promises are removed because the queue becomes too full they aren't rejected.

var limiter = new Bottleneck(1, 100, 3);

limiter.schedule(test).then(console.log).catch(onRejected);
limiter.schedule(test).then(console.log).catch(onRejected);
limiter.schedule(test).then(console.log).catch(onRejected);
limiter.schedule(test).then(console.log).catch(onRejected);
limiter.schedule(test).then(console.log).catch(onRejected);
limiter.schedule(test).then(console.log).catch(onRejected);

function onRejected() {
    console.log("rejected");
}

function test() {
    console.log("starting");
    return new Promise((resolve) => {
        setTimeout(() => {
            resolve("resolved!");
        }, 1000);
    });
}

Here "rejected" is never printed to the console but 4 of the promises are resolved. Might have a look at submitting a PR with this functionality tomorrow.

I am using limiting promises using limiter.schedule(). When running limiter.stopAll() the queue is cleared, but whatever was running is never cleared and seems to just disappear. I am using limiter.nbRunning() to verify.

Am I doing something wrong or is this intended?

Hello!

I was having a hard time getting priorities to work and discovered that priority = this._sanitizePriority(priority); seems to be changing the value of a validly passed priority inside of submitPriority() (which in my application is being called from schedulePriority).

Adding some logging so the function reads like this:

  console.log("TEMP_AJR args[0] in submit = " + arguments[0]);

  priority = arguments[0], task = arguments[1], args = 4 <= arguments.length ? slice.call(arguments, 2, j = arguments.length - 1) : (j = 2, []), cb = arguments[j++];

  console.log("TEMP_AJR pre-sanitize in submit = " + priority);

  priority = this._sanitizePriority(priority);

  console.log("TEMP_AJR sp in submit = " + priority);

logged this:
TEMP_AJR args[0] in submit = 6
TEMP_AJR pre-sanitize in submit = 6
TEMP_AJR sp in submit = 5

I'm a very inexperienced javascript programmer so I expect there's something wrong with how I'm using the library, but maybe there's some actual issue so I figured I'd pass on the note.

Hope this helps!

-Al

I'm running into an error trying to substitute Q as my promise library.

In the README it states that

It's also possible to replace the Promise library used:

var Bottleneck = require("bottleneck");
Bottleneck.Promise = myPromiseLibrary;

var limiter = new Bottleneck(maxConcurrent, minTime, highWater, strategy);

However the following code does not create a Q Promise

var Bottleneck = require('bottleneck');
var Q = require('q');
Bottleneck.Promise = Q.Promise;

var limiter = new Bottleneck(1, 1000);

var p = limiter.schedule(function() {
  return Q.resolve(true);
});

console.log(Q.isPromise(p))  // false

Either this is a bug or I'm not setting this up correctly. Please help?

Hi guys, nice job, thx.

I'm finding the documentation troubling. The gotchas section tells you to read the job options section to read about timeouts, but there is nothing there about them.

Mon Mar 19 2018 11:50:35 GMT+0200 (EET) - error: (node:45466) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 56): Error: Bottleneck limiter (id: 'group-key-nuifj9q6emDxWdXAp') could not find the Redis key it needs to complete this action (key 'b_group-key-nuifj9q6emDxWdXAp_settings'), was it deleted? Note: This limiter is in a Group, it could have been garbage collected.

I get what is happening. But does this mean a group key is one time use? Why doesn't bottleneck just reinitialise keys after garbage collection?

Does stopAll work with .schedule()?

I have a bunch of promises lined up and calling stopAll() throws an uncaught exception

When I run this on node 9.7.1

const Bottleneck = require("bottleneck");

const limiter = new Bottleneck({
  maxConcurrent: 1,
  minTime: 1000
});

limiter.schedule(() => 'im the call')
  .then((result) => { console.log('handle result', result) });

I get

/[...]/node_modules/bottleneck/lib/Bottleneck.js:376
          return task.apply({}, args).then(function (...args) {
                                      ^

TypeError: task.apply(...).then is not a function
    at Object.wrapped ([...]/node_modules/bottleneck/lib/Bottleneck.js:376:39)
    at Timeout._executing.(anonymous function).timeout.setTimeout [as _onTimeout] ([..]/node_modules/bottleneck/lib/Bottleneck.js:233:32)
    at ontimeout (timers.js:466:11)
    at tryOnTimeout (timers.js:304:5)
    at Timer.listOnTimeout (timers.js:267:5)

Is there a known issue with node 9.7.1?

How to use bottleneck with events.

If this is my code which does not have a callback.... how do I rate limit it with bottleneck

    const service = google.drive('v3');
    const downPath = '/path/to/file';
    const dest = fs.createWriteStream(downPath);
    service.files.get({
        auth,
        fileId: fileid,
        alt: 'media',
    }).on('end', () => {
        return callback(true);
    }).on('error', (err) => {
        console.log('error');
    }).pipe(dest);

Use
export = Bottleneck;
instead of
export default Bottleneck;

I got a error in Typescript 2.5.2
Error: Cannot use 'new' with an expression whose type lacks a call or construct signature.

"Bottleneck builds a queue of requests and executes them as soon as possible."
How can I know the queue length? There is no method in "Bottleneck" class to check it.
I think that it's needed because if you issue several "limiter.submit" calls, in the corresponding callback functions is needed to test if it's the end of all requests,

Currently after a stopAll() anything that is queued is removed but dropped isn't fired.

If interrupt is true then should dropped be fired for all running jobs as well?

This issue is a suggestion.

Assuming a function that should be throttled like

var get = function(id) {
   return new Promise(...)
}

It would be nice to be able to do this:

var get = limiter.wrap(function (id) {
  return new Promise(...)
}

Thus, limiter.wrap should return a function. It should pass arguments as received and in the same order to the wrapped function.

Hi, is it possible to do a backoff policy with retry capability? I'm using the 'function wrap' option and was looking for a way to retry a request if the throttling exception occurs, but wait a few seconds before making the follow up request. Thanks

It would be great if redis could be supported for handling multiple nodejs processes (cluster support). For example, if I'm running 4 nodejs instances with a load balancer in front, the bottleneck instance may not work as expected. Let's say there is a large number of hits coming into the site, and you need to ensure that the bottleneck limiter only executes 1 concurrent request at a time every 200ms. With a nodejs cluster, there will be 4 bottleneck limiter instances, each acting independently. So if the load balancer is spreading the load, you might be getting 4 concurrent requests running, instead of the expected 1 concurrent request. With redis support, they could all communication with each other and ensure that only 1 concurrent request is running at one time. I'm using the bottleneck cluster feature for all my work, so it would need to work with that as well. I hope this is explained well enough. Let me know if you need any more details.

I'm wondering if bottleneck is suitable for the following use case.

I have an high throughput API endpoint that receives requests from various sources. I'm using bottleneck.group to rate limit requests per source. Bottleneck does a great job of this.

In doing so, I realise there is a optimisation opportunity. Instead of processing each group request individually, I'd like to process a batch of requests per group. So if the rate limit was 1 request per 10 seconds, in the new design it would be any request that was queued within that 10 second unit.

Is that something I could achieve with bottleneck? Apologies if it's way out of scope for this library.

https://www.npmjs.com/package/@types/bottleneck

https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html

First attempt:

// Type definitions for twit 1.15.0S
// Project: https://github.com/SGrondin/bottleneck
// Definitions by: Romel Gomez <https://github.com/romelgomez>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped


declare module 'bottleneck' {

  namespace Bottleneck {
    export interface Fn {
      (...args: any[]): any
    }
  }

  class Bottleneck {

    /**
     * @see https://github.com/SGrondin/bottleneck#constructor
     */
    constructor(maxNb?: number, minTime?: number, highWater?: number, strategy?: number, rejectOnDrop?: boolean);

    /**
     * @see https://github.com/SGrondin/bottleneck#submit
     */
    submit(fn: Bottleneck.Fn, ...args: any[]);

    /**
     * @see https://github.com/SGrondin/bottleneck#schedule
     */
    schedule(fn: Bottleneck.Fn, ...args: any[]);

    /**
     * @see https://github.com/SGrondin/bottleneck#submitpriority
     */
    submitPriority(priority:number, fn: Bottleneck.Fn, ...args: any[]);

    /**
     * @see https://github.com/SGrondin/bottleneck#schedulepriority
     */
    schedulePriority(priority:number, fn: Bottleneck.Fn, ...args: any[]);

  }

  export = Bottleneck;

}

As title, I have a problem in my project, I have to use priority queue like which in pool. Is it possible to add priority to bottleneck ?

Probably a question for @alexperovich, when importing bottleneck in a Typescript environment, I get compilation errors.

I'm importing like so:

import Bottleneck from 'bottleneck'

then using it:

const limiter = new Bottleneck(10)

When doing a compilation I get an error like:

TSError: ⨯ Unable to compile TypeScript
... File '.../node_modules/bottleneck/bottleneck.d.ts' is not a module.

My tsconfig.json looks like:

{
  "compilerOptions": {
    "lib": ["dom", "dom.iterable", "scripthost", "es2017"],
    "outDir": "./build/",
    "noImplicitAny": true,
    "noUnusedParameters": true,
    "noUnusedLocals": true,
    "module": "commonjs",
    "target": "es6",
    "sourceMap": true,
    "inlineSources": true
  },
  "include": [
    "./**/*.ts"
  ],
  "exclude": [
    "node_modules",
    "ts-node"
  ]
}

I suspect that I'm doing something wrong here but this is a mature code base and haven't run into this with other packages. Any ideas?

Hi All,
Imagine I wanted to write a command line script in Node that needs to be rate-limited across multiple executions, e.g. a Twitter API client. I call it once, the limiter takes note of it, the script terminates. Then I call it again, the limiter is aware of the previous call, checks the limiting etc.

How do I manage this need with this library? E.g. can I "serialize" in some way the status of the library's objects, so that I can save them and re-load them back every time I run the script? I wouldn't want to use any backend component, as in the choice of Redis for classdojo/rolling-rate-limiter, but would be happy with something simpler and slower, such as file system or environment variables.

Thanks.

Just a quick heads up that somewhere between 2.1.0 and 2.2.1 the api was changed so that calls to bottleneck.on() no longer return the bottleneck instance and thus are no longer chainable.

Maybe not a biggie, but probably still worth a mention in the release notes since (for us) this was a breaking change and makes bottlenecks behavior now differ from node's events.EventEmitter.on().

The reservoir option would be a great way to handle situation like GitHub rate limiting.
The API maintain X-RateLimit-Remaining which correspond to the concept of the reservoir, X-RateLimit-Reset which defines when the reservoir will be refilled and X-RateLimit-Limit which determine how much to refill the reservoir.

The implementation could be done like that:

• Create a Bottleneck limiter, setting the reservoir option with the value of X-RateLimit-Remaining
• When the reservoir is exhausted, determine the time to wait until the rate limit reset, wait for that time and refill the reservoir
• The Bottleneck would continue to process the jobs

Currently it seems that when the reservoir is exhausted Bottleneck just stop executing the jobs in the queue without throwing an error nor sending an event.
That makes it really difficult to determine when the reservoir is empty, pause the execution until it's refilled and schedule its refill.

Maybe an event could be sent when the reservoir get exhausted, proving a function to schedule its refill. So when this even would be trigger, one could retrieve the amount and time of the next refill and schedule it.

Another option would be to allow to define the frequency and amount of refills in the Bottleneck constructor. Bottleneck would pause execution when the reservoir get exhausted, wait until the refill schedule, refill of the given amount and continue to process tasks.

This first option is probably more complex to handle but as it's more reactive it would handle better the situation where other things consume some of the available requests in the API. The second option is deterministic, so if someone use some of the available request in the API, there would be a gap between the reservoir size in Bottleneck and the X-RateLimit-Remaining on GitHub.

Hello,
I read that this has been requested in the past and after a very long research I wasn't able to find a rate limiting library backed by redis which would also contain all the features I need, so somehow I am still stuck with bottleneck.

Literally every express project I wrote in the past required clustering. It's a pitty that bottleneck doesn't really support it. Every time I scale my application (up- or down) I am forced to adapt the amount of available requests for my limiters for each instance. Since the available requests usually don't change with my number of instances (because I am using external API services) this is very annoying.

Do you have any recommendations what I could do to solve this? Do you maybe intend to implement cluster support in a major update?

Hi. I use bottleneck to handle API requests that have a rather low limit, therefore my queue can go upwards to 1000-2000 items. The problem I have is that some new tasks will in some circumstances already be in the queue but not completed. Is there a way to check what is in the queue and the arguments, in order to avoid duplicates?

The empty event is called when the nbQueued() drops to 0. Is there another event that will be emitted when the last scheduled Promise is done?

It might be a good idea to release v2.0 in plain javascript. I think it would help to make the code base easier for the majority to contribute to. I know I've personally been put away because I don't know CoffeeScript's syntax at all.

If you are against it, or just plain prefer CoffeeScript, I completely understand! Just a thought.

decaffeinate/decaffeinate can take care of the heavy lifting.

Thanks for this great library!

What's the best way to clear the queue?

The code example for creating a cluster is incorrect. It is missing the "new" keyword on this line:

var cluster = Bottleneck.Cluster(maxConcurrent, minTime, highWater, strategy);

Want some feature in v2? Comment here.

• Make "reject on drop" true by default
• Add support for an options object, but keep the simple argument passing style for the 4 basic arguments (minTime, maxConcurrent, highWater, strategy)
• Rename all the changeX methods to setX
• Make the Promise library an option instead of a Prototype setting. Consider maybe making it an option when requiring the library, i.e. var Bottleneck = require('bottleneck')(myPromiseLibrary);.
• Modern JavaScript (ES2015) rewrite, compiled to ES5 using Babel.
• Support a timeout (with an error if the time is exceeded) for any job, both callbacks and promises. I don't want this in version 1 because it breaks the backwards compatibility.

More to come..

I had global event listeners before adopting a group limiter.

limiter.on('dropped', (dropped) => {
    console.warn("Dropped IC request", dropped)
})

debug && limiter.on('debug', function (message, data) {
    console.log(message, data)
})
``

Now with a group limiter, how do i use this event listeners?

If i create a listener per limiter, who do I clean them up when the linter is garbage collected from the group? (or are linters not gcd?)

Maybe I'm doing something wrong, but I've just updated to v2 in my project and although I have no intention of using clustering, my build is failing with complaints of not being able to resolve redis from node_modules?

Error: ./node_modules/bottleneck/lib/RedisStorage.js
Module not found: Error: Can't resolve 'redis' in 'C:\Users\jamie\Projects\massblock-app\node_modules\bottleneck\lib'
resolve 'redis' in 'C:\Users\jamie\Projects\massblock-app\node_modules\bottleneck\lib'
  Parsed request is a module
  using description file: C:\Users\jamie\Projects\massblock-app\node_modules\bottleneck\package.json (relative path: ./lib)
    Field 'browser' doesn't contain a valid alias configuration
  after using description file: C:\Users\jamie\Projects\massblock-app\node_modules\bottleneck\package.json (relative path: ./lib)
    resolve as module
      looking for modules in C:\Users\jamie\Projects\massblock-app\node_modules
        using description file: C:\Users\jamie\Projects\massblock-app\package.json (relative path: ./node_modules)
          Field 'browser' doesn't contain a valid alias configuration
        after using description file: C:\Users\jamie\Projects\massblock-app\package.json (relative path: ./node_modules)
          using description file: C:\Users\jamie\Projects\massblock-app\package.json (relative path: ./node_modules/redis)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              C:\Users\jamie\Projects\massblock-app\node_modules\redis doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              C:\Users\jamie\Projects\massblock-app\node_modules\redis.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              C:\Users\jamie\Projects\massblock-app\node_modules\redis.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              C:\Users\jamie\Projects\massblock-app\node_modules\redis.json doesn't exist
            as directory
              C:\Users\jamie\Projects\massblock-app\node_modules\redis doesn't exist
[C:\Users\jamie\Projects\massblock-app\node_modules\redis]
[C:\Users\jamie\Projects\massblock-app\node_modules\redis.ts]
[C:\Users\jamie\Projects\massblock-app\node_modules\redis.js]
[C:\Users\jamie\Projects\massblock-app\node_modules\redis.json]
[C:\Users\jamie\Projects\massblock-app\node_modules\redis]
 @ ./node_modules/bottleneck/lib/RedisStorage.js 77:14-30
 @ ./node_modules/bottleneck/lib/Bottleneck.js
 @ ./node_modules/bottleneck/lib/index.js

Any thoughts?

Would be nice for limiter objects to call a callback every time their queue is drained.

Hi folks!

I have a problem with the clustering mode. As you can see at the screenshot jobs has been running 1st, 3rd, 2nd, 4th, but not in sequential order.
There are 2 node processes. Job working time is 1 second.

I keep getting this error with no apparent pattern.

ReplyError: NOSCRIPT No matching script. Please use EVAL.
    at parseError (/Users/gabrielbira/workspace/littledata/queue/node_modules/redis-parser/lib/parser.js:193:12)
    at parseType (/Users/gabrielbira/workspace/littledata/queue/node_modules/redis-parser/lib/parser.js:303:14)
  command: 'EVALSHA',
  args: 
   [ undefined,
     3,
     'b_settings',
     'b_running',
     'b_executing',
     '0',
     '1',
     '1518520777288' ],
  code: 'NOSCRIPT'
}

Hello,
is there a way to get the current queue position / approx remaining time for a request in the schedule?

Hello,
is it also possible to schedule Promises directly instead of functions which return a promise?

I imagined something like this

requestData() {
  const p = new Promise([..])
  return this.limiter.schedule(p)
}

This way I could avoid writing wrapper functions which add a schedule for all my requestXY() functions.

Working with Facebook's Graph API, I stumbled upon a neat 'batch request' feature which basically does what it says and allows you to bundle multiple requests into one. Unfortunately, I'm not sure how not to go over API limits using Bottleneck, since a batch request is counted as all the requests bundled in it. Is there a way to do this? If not, can we have a new feature where we can attribute weights to tasks ? (e.g. tell this library this task is worth 5 tasks)

I note that there is a documented function to stop (and remove all entries in) the queue, but is there a way of temporarily pausing (and unpausing) execution of a currently executing queue without destroying the remaining items in the queue? To be clear, say I populate a queue with tasks that are presently executing. I would like to temporarily suspend execution of the tasks. Perhaps I'll add new tasks whilst the execution of the queue is suspended. Then I would like to be able to have the queue begin executing tasks again, with respect to any changes to the queue during it's suspended phase.

I imagine it's possible to clone the queue prior to calling the function to stop the queue, then repopulating it, but there's probably a less destructive way that I'm missing :) If not - then some tips on how to accomplish this would be wonderfully helpful. Thanks in advance.