mcollina / http-deduplicate

deduplicate multiple http request to a single endpoint

http-deduplicate

Deduplicate multiple http request to a single endpoint.

You have an application composed of multiple HTTP servers, and you need to issue HTTP requests from your frontend to other internal endpoints. So, you call your endpoint for all requests. This module allows you to reduce the traffic and response time by calling a given URL only once at any given interval.

In my tests, http-deduplicate increases the throughput by a 2-5x factor, depending on the application.

Install

npm install http-deduplicate -g

Example

deduplicate(url, function (err, data, cb) {
  // do something async
  cb()
  // cb is optional and can be omitted
})

See example.js for a full example

API

deduplicate(address, callback(err, data [, cb]))

Calls address (as defined inhttp and https only once in a given period, all the other requests will receive the same data.

For extreme optimizations in high load scenarios, you can serve the latest returned value up until all callback (if present) are called.

TODO

• HTTP headers
• HTTP caching (ETag)
• HTTP caching (Last-Modified)

Acknowledgements

http-deduplicate is sponsored by nearForm.

License

MIT

http-deduplicate's Issues

what does this do?

the title of this module has two words that usually intrigue me but i'm not completely sure what this does