Coder Social home page Coder Social logo

jmperez / promise-throttle Goto Github PK

View Code? Open in Web Editor NEW
148.0 4.0 11.0 1.04 MB

A small library to throttle promises. Useful to avoid rate limiting when using REST APIs.

Home Page: https://doxdox.org/jmperez/promise-throttle

License: MIT License

JavaScript 100.00%
promise rest-api throttle-promises throttling rate-limit

promise-throttle's Introduction

Promise Throttle Β  Build Status Coverage Status Greenkeeper badge

This small (~530B minified and compressed) dependency-free library limits promises run per unit of time. Useful for Rest API consumption, which is normally rate-limited to a certain number of requests in a set amount of time.

On Node.js, pass the Promise library you are using to the constructor.

To use, simply add functions to the PromiseThrottle that, once called, return a Promise.

Use

The library can be used either server-side or in the browser.

  var PromiseThrottle = require('promise-throttle');
  /**
   * A function that once called returns a promise
   * @return Promise
   */
  var myFunction = function(i) {
    return new Promise(function(resolve, reject) {
      // here we simulate that the promise runs some code
      // asynchronously
      setTimeout(function() {
        console.log(i + ": " + Math.random());
        resolve(i);
      }, 10);
    });
  };

  var promiseThrottle = new PromiseThrottle({
    requestsPerSecond: 1,           // up to 1 request per second
    promiseImplementation: Promise  // the Promise library you are using
  });

  var amountOfPromises = 10;
  while (amountOfPromises-- > 0) {
    promiseThrottle.add(myFunction.bind(this, amountOfPromises))
      .then(function(i) {
        console.log("Promise " + i + " done");
      });
  }

  // example using Promise.all
  var one = promiseThrottle.add(myFunction.bind(this, 1));
  var two = promiseThrottle.add(myFunction.bind(this, 2));
  var three = promiseThrottle.add(myFunction.bind(this, 3));

  Promise.all([one, two, three])
    .then(function(r) {
        console.log("Promises " + r.join(", ") + " done");
    });

Options

weight

You can specify weight option for each promise to dynamically adjust throttling depending on action "heaviness". For example, action with weight = 2 will be throttled as two regular actions. By default weight of all actions is 1.

  var regularAction = promiseThrottle.add(performRegularCall);
  var heavyAction = promiseThrottle.add(performHeavyCall, {weight: 2});

signal

You can cancel queued promises using an AbortSignal. For this, pass a signal option obtained from an AbortController. Once it is aborted, the promises queued using the signal will be rejected.

If the environment where you are running the code doesn't support AbortController, you can use a polyfill.

  var controller = new AbortController();
  var signal = controller.signal;
  var pt = createPromiseThrottle(10);
  pt.addAll([
    function() {
      return fetch('example.com/a');
    },
    function() {
      return fetch('example.com/b');
    },
    function() {
      ...
    }
  ], {signal: signal});
  ...

  // let's abort the promises
  controller.abort();

You can decide to make only specific promises abortable:

  var controller = new AbortController();
  var signal = controller.signal;
  var pt = createPromiseThrottle(10);
  pt.add(function() { return fetch('example.com/a') });
  pt.add(function() { return fetch('example.com/b') }, {signal: signal});
  pt.add(function() { return fetch('example.com/c') });
  ...

  // let's abort the second one
  controller.abort();

When aborting, the promise returned by add or addAll is rejected with a specific error:

  var controller = new AbortController();
  var signal = controller.signal;
  var pt = createPromiseThrottle(10);
  pt.addAll([
    function() {
      return fetch('example.com/a');
    },
    function() {
      return fetch('example.com/b');
    },
    function() {
      ...
    }
  ], {signal: signal}).catch(function(e) {
    if (e.name === 'AbortError') {
      console.log('Promises aborted');
    }
  });
  ...

  // let's abort the promises
  controller.abort();

Installation

For node.js, install the module with: npm i promise-throttle

If you are using it in a browser, you can use bower: bower install promise-throttle

Development

Install the dependencies using npm install. Run npm start to lint, test and browserify promise-thottle.

Projects using it

See how some projects are using it:

License

MIT

promise-throttle's People

Contributors

acinader avatar dependabot[bot] avatar draperunner avatar greenkeeper[bot] avatar ilyasemenov avatar jmperez avatar joshy avatar mawrkus avatar pyldin601 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

promise-throttle's Issues

An in-range update of coveralls is breaking the build 🚨

The devDependency coveralls was updated from 3.0.4 to 3.0.5.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

coveralls is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 6 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Documentation is missing a bunch of functionality?

Perhaps I'm missing something, but it seems like your examples use a lot of functionality that is missing from your documentation. For example I don't see anything in the docs about createPromiseThrottle, promiseImplementation option, etc. Also your documentation describes requestsPerSecond as a function argument, not an option but your examples show the opposite.

Just confused here maybe...

Abort - takes time to reject pending promises

I am using AbortController to abort my promises. However when aborting, rather than immediate rejection of all pending promises, the library continues to throttle and reject ends up taking more time. E.g if there are 50 promises with throttle of 1 req/sec, you'll end up with 50 seconds for abort to finish.

Below is the code sequence I run. Is there anything I'm missing?

const promiseThrottle = new PromiseThrottle({
  requestsPerSecond: 1,
  promiseImplementation: Promise // the Promise library you are using
});
let controller = new AbortController();
let signal = controller.signal;
let myPromises = [];
myPromises.push[promiseThrottle.add(my_axios_api, { signal })];
myPromises.push[promiseThrottle.add(my_axios_api { signal })];
return Promise.all(myPromises);
// and elsewhere , i loop and poll, if cancel should be aborted
this.cancelPoller = setInterval(() => {
  if (this.shouldCancelUpload()) {
    controller.abort();
  }
}, 1000); //check every 1 second if upload has to be cancelled

another full throttle :)

Just wanted to share a similar undertaking of promises with throttling / load balancing ;) - spex

Came out of need to have one back then, perhaps the same as yours. I appreciate any feedback that you may provide.

Hey dude

Sorry for bothering you here tried to track you down elsewhere.. Im running a group of 100 labels that are creating a daily playlist together.

Ive written a tool to help us update the app as you can imagine its a lot of work otherwise. Everythings going fine with Ajax clientside. But im having issues posting playlist with request in Node

**var options = {
url: 'https://api.spotify.com/v1/users/***************/playlists/***********************/tracks',
headers: {'Authorization': 'Bearer ' + token2,Content-Type': 'application/json'
},
body:idArrayJson, (have also tried data:)
json: true }

request.put (optionsx, function(error, response, body) {console.log (body)})**

at the moment im grabbing the token from the browser. Im getting:

{"error":{"status":400,"message":"JSON body does not adhere to the defined endpoint parameters"}}

and also ive had 'missing payload'

Would really appreciate your help! your tutorials on here have been incredibly useful. I tried a bunch of node libraries and a million things before resorting to contacting you, i hope you dont mind.

Dan DJ Fresh spotify:user:iluvdrumandbass:playlist:1yRrivuEUTh5upqzeZOFBG

An in-range update of promise is breaking the build 🚨

The devDependency promise was updated from 8.0.3 to 8.1.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

promise is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Commits

The new version differs by 7 commits.

  • e9c12a2 feat: support ES2015 iterables in all() and race() (#160)
  • 81b5722 docs: add "Enterprise Support" to README
  • 6d91fa0 Create FUNDING.yml
  • affd5e2 ci: update supported node versions
  • 6272338 adding Promise.race documentation and reference to proper API docs (#155)
  • 44f6d7a Fix typo: optimistically
  • f643fd6 Release 8.0.3

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

How to use it

Hi,

thanks for the library. It seems it is exactly what I need, however I am not sure how to use it in my case. In a promise chain, multiple POST requests are done and then the results are returned
return Promise.all([postRequests]). The POST requests needs to be throttled to 1 request per second, here promise-throttle is needed ;-). How can I achieve this with promise-throttle?
I have tried this:

var PromiseThrottle = require('promise-throttle');
  var myFunction = function(i) {
    return new Promise(function(resolve, reject) {
      setTimeout(function() {
        console.log(new Date() + ", " + i);
      }, 10);
    });
  };
  var promiseThrottle = new PromiseThrottle({
    requestsPerSecond: 1,
  });
var x = promiseThrottle.add(myFunction.bind(this, 1));
var y = promiseThrottle.add(myFunction.bind(this, 2));
var z = promiseThrottle.add(myFunction.bind(this, 3));

Promise.all([x,y,z]).then(data => console.log("Done"));

But 'Done' is never printed. Any idea? If the library needs to be changed and you give me some hints, I will submit a PR. Thanks a lot for your time!
Cheers,
Joshy

.addAll() should return a promise

Hi,
First of all, thx for the library! I'm using it for a web scraper pet project...
A small detail, for the sake of consistency/convenience, I think that .addAll() should return a promise:

const oneTwoThree = promiseThrottle.addAll([
  myFunction.bind(this, 1),
  myFunction.bind(this, 2),
  myFunction.bind(this, 3)
]);

oneTwoThree.then(r => console.log("Promises " + r.join(", ") + " done"));

This should do the trick:

PromiseThrottle.prototype.addAll = function (promises) {
  var addedPromises = promises.map(function(promise) {
    return this.add(promise);
  }.bind(this));

  return Promise.all(addedPromises);
};

I can submit a PR if you agree on the idea.

An in-range update of eslint is breaking the build 🚨

Version 3.19.0 of eslint just got published.

Branch Build failing 🚨
Dependency eslint
Current Version 3.18.0
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As eslint is β€œonly” a devDependency of this project it might not break production or downstream projects, but β€œonly” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this πŸ’ͺ


Status Details
  • ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

  • βœ… coverage/coveralls First build on greenkeeper/eslint-3.19.0 at 100.0% Details

Release Notes v3.19.0
  • e09132f Fix: no-extra-parens false positive with exports and object literals (#8359) (Teddy Katz)
  • 91baed4 Update: allow custom messages in no-restricted-syntax (fixes #8298) (#8357) (Vitor Balocco)
  • 35c93e6 Fix: prevent space-before-function-paren from checking type annotations (#8349) (Teddy Katz)
  • 3342e9f Fix: don't modify operator precedence in operator-assignment autofixer (#8358) (Teddy Katz)
  • f88375f Docs: clarify that no-unsafe-negation is in eslint:recommended (#8371) (Teddy Katz)
  • 02f0d27 Docs: Add soda0289 to Development Team (#8367) (Kai Cataldo)
  • 155424c Fix: ignore empty path in patterns (fixes #8362) (#8364) (alberto)
  • 27616a8 Fix: prefer-const false positive with object spread (fixes #8187) (#8297) (Vitor Balocco)
  • 8569a90 Docs: add note about git's linebreak handling to linebreak-style docs (#8361) (Teddy Katz)
  • 5878593 Chore: fix invalid syntax in no-param-reassign test (#8360) (Teddy Katz)
  • 1b1046b Fix: don't classify plugins that throw errors as "missing" (fixes #6874) (#8323) (Teddy Katz)
  • 29f4ba5 Fix: no-useless-computed-key invalid autofix for getters and setters (#8335) (Teddy Katz)
  • 0541eaf Fix: no-implicit-coercion invalid autofix with consecutive identifiers (#8340) (Teddy Katz)
  • 41b9786 Fix: no-extra-parens false positive with objects following arrows (#8339) (Teddy Katz)
  • 3146167 Fix: eslint.verify should not mutate config argument (fixes #8329) (#8334) (alberto)
  • 927de90 Fix: dot-notation autofix produces invalid syntax for integer properties (#8332) (Teddy Katz)
  • a9d1bea Fix: comma-style autofix produces errors on parenthesized elements (#8331) (Teddy Katz)
  • d52173f Fix: don't generate invalid options in config-rule (#8326) (Teddy Katz)
  • 6eda3b5 Fix: no-extra-parens invalid autofix in for-of statements (#8337) (Teddy Katz)
  • 6c819d8 Fix: dot-notation autofix produces errors on parenthesized computed keys (#8330) (Teddy Katz)
  • 2d883d7 Fix: object-shorthand autofix produces errors on parenthesized functions (#8328) (Teddy Katz)
  • cd9b774 Fix: quotes false positive with backtick option in method names (#8327) (Teddy Katz)
  • d064ba2 Fix: no-else-return false positive for ifs in single-statement position (#8338) (Teddy Katz)
  • 6a718ba Chore: enable max-statements-per-line on ESLint codebase (#8321) (Teddy Katz)
  • 614b62e Chore: update sinon calls to deprecated API. (#8310) (alberto)
  • 0491572 Chore: use precalculated counts in codeframe formatter (#8296) (Vitor Balocco)
  • 8733e6a Chore: Fix incorrect error location properties in tests (#8307) (alberto)
  • c4ffb49 Chore: Fix typos in test option assertions (#8305) (Teddy Katz)
  • 79a97cb Upgrade: devDependencies (#8303) (alberto)
  • e4da200 Upgrade: Mocha to 3.2.0 (#8299) (Ilya Volodin)
  • 2f144ca Fix: operator-assignment autofix errors with parentheses (fixes #8293) (#8294) (Teddy Katz)
  • 7521cd5 Chore: update token logic in rules to use ast-utils (#8288) (Teddy Katz)
  • 9b509ce Chore: refactor space-before-function-paren rule (#8284) (Teddy Katz)
  • ddc6350 Fix: no-param-reassign false positive on destructuring (fixes #8279) (#8281) (Teddy Katz)
  • f8176b3 Chore: improve test coverage for node-event-generator (#8287) (Teddy Katz)
  • 602e9c2 Docs: fix incorrect selector examples (#8278) (Teddy Katz)
Commits

The new version differs by 38 commits .

  • 421aab4 3.19.0
  • 26a4dd5 Build: changelog update for 3.19.0
  • e09132f Fix: no-extra-parens false positive with exports and object literals (#8359)
  • 91baed4 Update: allow custom messages in no-restricted-syntax (fixes #8298) (#8357)
  • 35c93e6 Fix: prevent space-before-function-paren from checking type annotations (#8349)
  • 3342e9f Fix: don't modify operator precedence in operator-assignment autofixer (#8358)
  • f88375f Docs: clarify that no-unsafe-negation is in eslint:recommended (#8371)
  • 02f0d27 Docs: Add soda0289 to Development Team (#8367)
  • 155424c Fix: ignore empty path in patterns (fixes #8362) (#8364)
  • 27616a8 Fix: prefer-const false positive with object spread (fixes #8187) (#8297)
  • 8569a90 Docs: add note about git's linebreak handling to linebreak-style docs (#8361)
  • 5878593 Chore: fix invalid syntax in no-param-reassign test (#8360)
  • 1b1046b Fix: don't classify plugins that throw errors as "missing" (fixes #6874) (#8323)
  • 29f4ba5 Fix: no-useless-computed-key invalid autofix for getters and setters (#8335)
  • 0541eaf Fix: no-implicit-coercion invalid autofix with consecutive identifiers (#8340)

There are 38 commits in total. See the full diff.

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

An in-range update of eslint is breaking the build 🚨

The devDependency eslint was updated from 5.14.0 to 5.14.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).
  • βœ… coverage/coveralls: First build on greenkeeper/eslint-5.14.1 at 100.0% (Details).

Release Notes for v5.14.1
  • 1d6e639 Fix: sort-keys throws Error at SpreadElement (fixes #11402) (#11403) (Krist Wongsuphasawat)
Commits

The new version differs by 3 commits.

  • b2e94d8 5.14.1
  • ce129ed Build: changelog update for 5.14.1
  • 1d6e639 Fix: sort-keys throws Error at SpreadElement (fixes #11402) (#11403)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of sinon is breaking the build 🚨

Version 4.1.6 of sinon was just published.

Branch Build failing 🚨
Dependency sinon
Current Version 4.1.5
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

sinon is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ coverage/coveralls Coverage pending from Coveralls.io Details
  • ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

Commits

The new version differs by 10 commits.

  • 68c37ed Update docs/changelog.md and set new release id in docs/_config.yml
  • cd8ae51 Add release documentation for v4.1.6
  • 29e80be 4.1.6
  • a5c59a5 Update History.md and AUTHORS for new release
  • 0ae60b6 Merge pull request #1653 from mroderick/upgrade-dependencies
  • dcd4191 Upgrade browserify to latest
  • a316f02 Upgrade markdownlint-cli to latest
  • 78ebdb3 Upgrade lint-staged to latest
  • fcf967b Upgrade dependency supports-color
  • 7c3cb4f Enable StaleBot with default configuration (#1649)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

1 request per 5 seconds

Hey,

Thanks for making this! It really simplifies my code. I tried using it for Twitter Rate limiting, but I found the requestsPerSecond approach a bit lacking.
In the Twitter API case, a request needs to be sent every 5 seconds. I tried setting the requestsPerSecond to 0.2, but it didn't work, unfortunately.

Wrong argument type in TypeScript declaration?

Instead of taking a Promise like so:

add<T>(promise: Promise<T>, opts?: QueueOptions): Promise<T>

isn't it correct that add should take a function that returns a Promise<T>?

add<T>(promiseCreator: () => Promise<T>, opts?: QueueOptions): Promise<T>

@IlyaSemenov

unexpected weight behaviour

Hello,

I've implemented PromiseThrottle with 1 call per second and with default weight except one call with a weight of 3. So i was expected that this call has been launched just after the last and the next call to be launched 3 times later. But the only things the weight seems to do is to apply some kind of timeout before it was launched and not before the next call.
To be clear here, a screenshot of devtools network tab where you can see what i mean.

throttle_

Why heaviest call begins 3 seconds after ? And why this is not the next call which get that behaviour ?

thanks

Aborting requests throws a ReferenceError in node.js

Using node.js v8.11.1

Simple example to reproduce:

const PromiseThrottle = require('promise-throttle'),
    promiseThrottle = new PromiseThrottle({
        promiseImplementation: Promise,
        requestsPerSecond: 1
    });

promiseThrottle.add(() => Promise.resolve(), {
    signal: {
        aborted: true
    }
});

This results in an error: UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): ReferenceError: DOMException is not defined

v1.1.2 is not released on npm

For some reason the last release (v.1.1.2) is not exported to npmjs.com, so the version is installed via npm install is 1.1.1

Undefined requestsPerSeconds gives an infinite loop

Several days ago, I've made a typo while configuring a PromiseThrottle on the requestsPerSeconds awaited field. Consequently, the object was configured with an undefined requestsPerSeconds value and my sequence of promise was obviously not fired.

After a small deep diving a bit, I've finally understood that the undefined have been transformed into a NaN while computing the value inc = (1000 / this.requestsPerSecond) * weight. The NaN is propagated in the instruction on below and make the if (the this._execute();) unsolvable.

if (elapsed >= inc) {
    this._execute();
} else {
    // we have reached the limit, schedule a dequeue operation
    setTimeout(function() {
        this.dequeue();
    }.bind(this), inc - elapsed);
}

Is it possible to add some warning or trigger some error in case of undefined or NaN value? I would be happy to send you a PR for that if you need to!

An in-range update of sinon is breaking the build 🚨

The devDependency sinon was updated from 7.3.0 to 7.3.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

sinon is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).
  • βœ… coverage/coveralls: First build on greenkeeper/sinon-7.3.1 at 100.0% (Details).

Commits

The new version differs by 6 commits.

  • 3812a7d Update docs/changelog.md and set new release id in docs/_config.yml
  • 93bef55 Add release documentation for v7.3.1
  • e02c192 7.3.1
  • 8ee1d35 Update CHANGELOG.md and AUTHORS for new release
  • bc53d82 Fix security issues
  • 1a09166 Update @sinonjs/samsam to v3.3.1

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.