mikeal / r2

HTTP client. Spiritual successor to request.

r2

Early in Node.js I wrote an HTTP client library called request. It evolved along with Node.js and eventually became very widely depended upon.

A lot has changed since 2010 and I've decided to re-think what a simple HTTP client library should look like.

This new library, r2, is a completely new approach from request.

• Rather than being built on top of the Node.js Core HTTP library and shimmed for the browser, r2 is built on top of the browser's Fetch API and shimmed for Node.js.
• APIs are meant to be used with async/await, which means they are based on promises.

const r2 = require('r2')

let html = await r2('https://www.google.com').text

Simple JSON support.

let obj = {ok: true}

let resp = await r2.put('http://localhost/test.json', {json: obj}).json

Simple headers support.

let headers = {'x-test': 'ok'}

let res = await r2('http://localhost/test', {headers}).response

Being written to the Fetch API is a huge benefit for browser users.

When running through browserify request is ~2M uncompressed and ~500K compressed. r2 is only 66K uncompressed and 16K compressed.

r2's Issues

Error: Cannot find module 'vinyl-source-stream'

const comms = {
	requestData: async function (method, url, requestBody) {
		return await r2[method](url, requestBody)
	}
}

RequestData results in

Welcome to 3VOT
module.js:457
    throw err;
    ^

Error: Cannot find module 'vinyl-source-stream'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (***/node_modules/r2/tasks/dist.js:10:20)
    at Module._compile (module.js:556:32)
    at Module._extensions..js (module.js:565:10)
    at Object.require.extensions.(anonymous function) [as .js] (***/node_modules/babel-register/lib/node.js:152:7)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)

r2.post not working vs fetch

At my wits end trying to get this POST to return properly.

What else does r2 do to the request vs fetch? Something is wrong. I've tried using different initialization methods and alternate response management to try to get a working result.

Universal variables for all four examples:

const url = 'https://example.com/path';
const params = { example: 'request data', other_data: 'test' };

const body = Object.keys(params).map(key => encodeURIComponent(key) + '=' + encodeURIComponent(params[key])).join('&');

const headers = {
  'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
};

Broken, returns 400 error html:

const r2_example = await r2.post(url, {
  headers,
  body
});
console.log('r2', await r2_example.text);

Broken; alternate configuration, alternate response handling, still returning 400 error in html:

const r2_example_2 = await r2({
  url,
  body,
  method: 'POST',
  headers,
});
console.log('r2_2', await (await r2_example_2.response).text()); // 400 Bad Request

Working example with raw Header/Body

const fetch_example = await fetch(url, {
  method: 'POST',
  headers,
  body
});
console.log('fetch', await fetch_example.text());   // Proper result

Working example with node-fetch.Headers (globals created by r2) and Buffer.from(string).

const fetch_example_with_headers = await fetch(url, {
  method: 'POST',
  headers: new global.Headers(headers),
  body: Buffer.from(body)
});
console.log('fetch_example_with_headers', await fetch_example_with_headers.text()); // Proper result

Consider using functions instead of getter properties

This is most definitely a style preference, but I think many newcomers will be confused by the use of dynamically defined properties that return promises here, and I think they could lead to bugs due to their magicness.

Is there a rationale for preferring:

let html = await r2('https://www.google.com').text

over

let html = await r2('https://www.google.com').text()

Especially considering that the latter aligns with the Fetch API that users might be familiar with?

Since the promises returned here perform I/O, I think it's a stronger signal to the reader to use functions instead of properties.

It's a relatively well accepted maxim in languages that have had property methods for a while (eg, C#) that they should still be inexpensive wrappers to internal state instead of performing any real work – as this goes against users' expectations of what a property is.

Again, totally just a personal preference – but I think it falls well within the principle of least astonishment.

make-fetch-happen instead of node-fetch

Hey,

I want to introduce you make-fetch-happen.

It's similar to node-fetch but more powerful in terms of retries and additional features in the backend side.

Do you think could be a good dependency to be used for this project?

Vulnerability - Upgrade node-fetch version from 2.6.5 to 2.6.7, 3.1.1

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.

Issues with binary files

Hi,

I really wanted to use r2 as a modern alternative to request, but I ran into issues with binary files. When I try to download for example our company logo, it all works fine:

> (async function () { console.log(await require('r2').get('https://wizcorp.jp/images/logo.png').arrayBuffer); console.log('done'); })();

output:

Promise {
  <pending>,
  domain: 
   Domain {
     domain: null,
     _events: 
      { removeListener: [Function: updateExceptionCapture],
        newListener: [Function: updateExceptionCapture],
        error: [Function: debugDomainError] },
     _eventsCount: 3,
     _maxListeners: undefined,
     members: [] } }
> ArrayBuffer { byteLength: 3874 }
done

But on a larger file (not sure if "large" has anything to do with it though), it just sort of hangs there:

(async function () { console.log(await require('r2').get('http://ipv4.download.thinkbroadband.com/5MB.zip').arrayBuffer); console.log('done'); })();

output:

Promise {
  <pending>,
  domain: 
   Domain {
     domain: null,
     _events: 
      { removeListener: [Function: updateExceptionCapture],
        newListener: [Function: updateExceptionCapture],
        error: [Function: debugDomainError] },
     _eventsCount: 3,
     _maxListeners: undefined,
     members: [] } }
> 

Replacing .arrayBuffer with .response does seem to return a response object of some kind, but I don't know how to extract data from that:

> (async function () { console.log(await require('r2').get('http://ipv4.download.thinkbroadband.com/5MB.zip').response); console.log('done'); })();

output:

Promise {
  <pending>,
  domain: 
   Domain {
     domain: null,
     _events: 
      { removeListener: [Function: updateExceptionCapture],
        newListener: [Function: updateExceptionCapture],
        error: [Function: debugDomainError] },
     _eventsCount: 3,
     _maxListeners: undefined,
     members: [] } }
> Response {
  size: 0,
  timeout: 0,
  [Symbol(Body internals)]: 
   { body: 
      PassThrough {
        _readableState: [ReadableState],
        readable: true,
        domain: [Domain],
        _events: [Object],
        _eventsCount: 7,
        _maxListeners: undefined,
        _writableState: [WritableState],
        writable: true,
        allowHalfOpen: true,
        _transformState: [Object] },
     disturbed: false,
     error: null },
  [Symbol(Response internals)]: 
   { url: 'http://ipv4.download.thinkbroadband.com/5MB.zip',
     status: 200,
     statusText: 'OK',
     headers: Headers { [Symbol(map)]: [Object] } } }

Are `makeHeaders` and `caseless` unnecessary?

According to MDN/fetch, the param headers can be a literal object with out being wrapped in Header.
According to RFC, headers' field names are case-insensitive.

So, I think it's safe to remove makeHeaders and caseless in this work.

An in-range update of semantic-release is breaking the build 🚨

Version 15.6.0 of semantic-release was just published.

Branch	Build failing 🚨
Dependency	[semantic-release](https://github.com/semantic-release/semantic-release)
Current Version	15.5.5
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

semantic-release is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

---

Release Notes

v15.6.0

15.6.0 (2018-06-19)

Features

• allow to disable the publish plugin hook (4454d57)

Commits

The new version differs by 1 commits.

• 4454d57 feat: allow to disable the publish plugin hook

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

r2 failed on some pages

When I fetch some page like https://github.com, r2 failed and keep Promise pending.

node.js : v8.9.1
OS: Mac OS 10.12.6
r2: 2.0.0

r2('https://www.google.com').text.then(h => console.log(h)) // OK

r2('https://github.com').text.then(h => console.log(h)) // Failed

Documentation and guide

Hi @mikeal
Great job done, it is generally a great idea to keep your techs up-to-date. The only thing is that I couldn't find any documentations and guid towards using r2 or maybe migrating from request to r2. Are you planning on creating a documentation in the near future? And also, is the version 1.0.0 released or gonna be released soon? I plan on using it in a big project, I think it'd be a good test case with lots of feedback/issues.

Update require-dir (to fix node 8 issues)

node_modules\require-dir\index.js:93
if (!require.extensions.hasOwnProperty(ext)) {
TypeError: require.extensions.hasOwnProperty is not a function

Already fixed in aseemk/requireDir#45. require-dir just needs an update

Potential security vulnerabili

A potential security vulnerability has been disclosed for this repo https://huntr.dev/bounties/b0e92c01-2535-44ec-9808-cbba3a1cf0d7/ 3 months ago .
But i see it still not addressed .
Kindly validate the bug

[TS] r2 typescript declaration file

That would be very helpful in typescript environment.

Clearer error messages in read-only properties set attemps

Currently, failSet does not expose what property you tried to set.

My suggestion:

const failSet = (prop) => () => { throw new Error(`Cannot set read-only property R2.${prop}`) }
...
Object.defineProperty(this, 'json', {
  get: () => this.response.then(resp => resp.clone().json()),
  set: failSet('json')
})
...

this way, the error that will be thrown will allow for quicker and easier debugging, since you'll more easily know what to search for in your code.

String stream support

In https://github.com/mikeal/r2/blob/master/index.js#L14 there is a todo about supporting streams. Would using https://github.com/yoshuawuyts/from2-string work for this project or too large?

r2 doesn't have formData post?

I saw 'TODO: formData API.' comment in index.js.

this means post formData is not yet made?

Is it possible to access connection properties in r2?

With the old request, I can use streams to handle the 'response' event, and get connection properties because the HTTP connection is still open.

@mikeal how is this handled in r2, if it all? I know there's some conceptual overlap between promises and streams so I'm not sure if r2 can do this, can't do this,. or there's a better way of accessing connection properties.

req.on('response', function (res) {
	var certificate = res.req.connection.getPeerCertificate && res.req.connection.getPeerCertificate(true)

Thanks!

Responsible disclosure policy

Hey there!

I belong to an open source security research community, and a member (@ranjit-git) has found an issue, but doesn’t know the best way to disclose it.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

Rename this project

i'm the only r2 in this place

A strange behavior of .json

Hi,

When I receive a heavy json response (115Kb in my case)

const json = await r2.post('my_url', { headers: customHeaders }).json

my process is blocked

but if I write this

const res = await r2.post('my_url', { headers: customHeaders }).response
const json = await res.json()

It's works !

I work with node v9.5.0

can the dependence of node-fetch use the 1.x ?

I find the default dependence of node-fetch is 2.x ,it's a alpha version , and i find in crawling some website (eg. https://www.baidu.com) ,the node-fetch 2.x inner function clone will in error that cannot output any result and not throw any error.
I test in the node-fetch 1.x ，it will ok .

3ks

Which browsers does r2 support?

Basically, the title says it all: Which browsers does r2 support, i.e. on which browser are you able to use it?

Proxy support ?

One thing I liked with request, it's support of env vars for managing proxy behavior (https://github.com/request/request#controlling-proxy-behaviour-using-environment-variables).
It's very convenient, especially if you work under a corporate proxy 😢 , and the production is not.

Is there something like that planned for r2 ?

ReferenceError: error is not defined

after failing trying to run r2 on runkit, i fallback on working on my local machine but i still got an error:

╭─phra at kali in /tmp
╰─λ node -v                                                                                                                                                      0 < 11:54:08
v8.4.0

╭─phra at kali in /tmp
╰─λ cat index.js                                                                                                                                                 0 < 11:54:02
const r2 = require('r2')
const to = require('await-to-ts')
const assert = require('assert')

const URLS = [1,2,3,4,5,6,7,8,9,10].map(i => `https://jsonplaceholder.typicode.com/posts/${i}`)

(async () => {
  const promises = URLS.map(url => r2(url))
  const [err, res] = await to(promises)
  assert.ifError(err)
  res.forEach(console.log.bind(console))
})()⏎                                                                                                                                                                        

╭─phra at kali in /tmp
╰─λ node index.js                                                                                                                                                0 < 11:54:05
/tmp/node_modules/r2/clayconfig.js:5
if(!config.raw.name)	            error( "package.json should have a name value")
                    	            ^

ReferenceError: error is not defined
    at Object.<anonymous> (/tmp/node_modules/r2/clayconfig.js:5:34)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/tmp/node_modules/r2/index.js:15:14)
    at Module._compile (module.js:573:30)

Encoding parameter

Using request, I can add options about encoding: encoding: 'latin1'. Is it possible with r2?

require.extensions is not defined on node v8.x.x

hi,

when i try to use r2 with node v8.x.x i got this error: TypeError: require.extensions.hasOwnProperty is not a function
you can reproduce the issue here: https://runkit.com/phra/r2-require-extensions

Document support browser versions

Please document the support browser and Node versions.

At least for the browsers, I believe this to be very limited given there isn't any employment of a browser-side polyfill here if the Fetch API is not available. I think many people would be surprised by that when coming to the "spiritual successor of request". Granted, that issue could be solved by incorporating something like #27 (using isomorphic-fetch rather than just node-fetch) if desired.

Promise usage

Hi, thanks for the nice library!

In the code you create a resolvable() only to resolve it with an already existing promise which is not something you typically have to do.

You can use fetch as this.response, the setTimeout also isn't necessary as promises take care of reacting on the next tick anyway.

I'll write a PR to illustrate what I mean

Ilegal http header

Hi there,

I'm trying to pass a variable through headers: for example:
let headers = {'x-apikey': apiKey }
but when I run it, the log says that the apiKey value is an ilegal http header this variable (apiKey) return a random string code, but when I pasted the code instead of the variable it runs.
I want to know how pass the variable.

Body cant be an object when x-form-www-urlencoded POST is used

There is room for improvment, in this case body cant be a json obj, and it needs to be manually converted to a string

The automated release is failing 🚨

🚨 The automated release from the master branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here is some links that can help you:

• Usage documentation
• Frequently Asked Questions
• Support channels

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

---

Invalid npm token.

The npm token configured in the NPM_TOKEN environment variable must be a valid token allowing to publish to the registry https://registry.npmjs.org/.

If you are using Two-Factor Authentication, make configure the auth-only level is supported. semantic-release cannot publish with the default auth-and-writes level.

Please make sure to set the NPM_TOKEN environment variable in your CI with the exact value of the npm token.

---

Good luck with your project ✨

Your semantic-release bot 📦🚀

require.extensions.hasOwnProperty is not a function

Hello,
I installed r2 on my computer with nodejs 8.4.0 but when I do a npm start I have this error.

/home/ngoura/projet/testnacim/node_modules/require-dir/index.js:93
            if (!require.extensions.hasOwnProperty(ext)) {
                                    ^

TypeError: require.extensions.hasOwnProperty is not a function
    at requireDir (/home/ngoura/projet/testnacim/node_modules/require-dir/index.js:93:37)
    at Object.<anonymous> (/home/ngoura/projet/testnacim/node_modules/r2/index.js:19:1)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/ngoura/projet/testnacim/app/controllers/proxy.controller.js:11:7)

So I can't use this package 😞

It is not support the typescript

Assuming all JSON payloads are application/json is wrong

This is regarding: https://github.com/mikeal/r2/blob/master/index.js#L73

If current code sees that you are passing json it overrides content-type header to be set to application/json. But application/json is not the only JSON media type. There're plenty of others:

1. HAL http://stateless.co/hal_specification.html
2. JSON API http://jsonapi.org/
3. Siren https://github.com/kevinswiber/siren
4. Hyper http://hyperjson.io/

to name just a few

No package description

It looks so weird

DeprecationWarning: Buffer() is deprecated due to security and usability issues

Due to transitive dependency usage of whatwg-url (v5.0.0) via node-fetch(v2.6.5) I got:

2021-10-11T02:26:37.263Z	ERROR	(node:8) [DEP0005]     DeprecationWarning: Buffer() is deprecated due 
to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
   at showFlaggedDeprecation (buffer.js:194:11)
   at new Buffer (buffer.js:278:3)
   at utf8PercentDecode (/var/task/whatwg-url/lib/url-state-machine.js:105:17)
   at parseHost (/var/task/whatwg-url/lib/url-state-machine.js:400:18)
   at URLStateMachine.parseHostName (/var/task/whatwg-url/lib/url-state-machine.js:851:18)
   at new URLStateMachine (/var/task/whatwg-url/lib/url-state-machine.js:562:44)
   at Object.module.exports.basicURLParse (/var/task/whatwg-url/lib/url-state-machine.js:1258:15)
   at new URLImpl (/var/task/whatwg-url/lib/URL-impl.js:17:27)
   at Object.setup (/var/task/whatwg-url/lib/URL.js:187:17)
   at new URL (/var/task/whatwg-url/lib/URL.js:25:18)

I haven't tested, but upgrading to node-fetch 3.0.0 might come in handy, I guess.

rename package because is already taken on NPM

Hi,

i was losing a lot of time in order to use r2 because i was getting a lot of problems. as @thechriswalker has noted, the r2 package on NPM is another one and so it creates confusion.

imho it's better if we find another name for this package that is free on NPM.

An in-range update of browserify is breaking the build 🚨

The devDependency browserify was updated from 16.3.0 to 16.4.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

browserify is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Commits

The new version differs by 5 commits.

• 8980670 16.4.0
• f871a85 Update changelog.markdown
• 52de2c4 Merge pull request #1916 from browserify/stream-http
• 5dc1bf2 Upgrade stream-http to v3
• 4a5ea7e Add funding.yml

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of semantic-release is breaking the build 🚨

Version 15.9.9 of semantic-release was just published.

Branch	Build failing 🚨
Dependency	semantic-release
Current Version	15.9.8
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

semantic-release is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes

v15.9.9

15.9.9 (2018-08-18)

Bug Fixes

• package: update marked to version 0.5.0 (2f4befe)

Commits

The new version differs by 2 commits.

• 2f4befe fix(package): update marked to version 0.5.0
• cfeaf49 test: use real writable stream

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

chalk is not defined in package.json

hi,

i'm trying to use r2 on runkit but i got this error: Error: Cannot find module 'chalk'
you can reproduce the issue here: https://runkit.com/phra/r2-chalk-missing

Style improvements

Much like #10, consider some ES6ish things:

1. The defineProperty code is unnecessarily verbose. Consider
   i. changing the defineProperty to getters e.g. get json () { return this.response.then(resp => resp.clone().json()) } or
   ii. using ['json', 'blob', ...].forEach((n) => Object.defineProperty(this, n, {get: () => this.response.then(r => r.clone()[n]()}) or
   iii. using defineProperties
2. Define setHeaders(obj = {}) and remove if (opts.headers)
3. change makeBody to a named function so that its name shows up when debugging
4. change setHeader (key, value) { this.setHeaders({ [key]: value }) }
5. have _args return this, then change all the http method methods (get, post, ...) to return this._args(...args)

http2 support?

just became aware of fetch-h2 for http2

are you looking at http2 support?

makeHeaders refactoring

Having to iterate over object to create Headers of them is obsolete.
Headers constructor accepts a plain object and constructs Headers class instance of it.

spec

const makeHeaders = obj => { let h = new Headers() for (let key in obj) { h.append(key, obj[key]) } return h }

becomes

const makeHeaders = obj => new Headers(obj)

What does this library solve that fetch does not?

What's the difference between the two

   ` let info = await r2(url(data[i]['ename']), { headers }).text
     console.log(info) `

No output

   ` let info = await r2(url(data[i]['ename']), { headers }).response
     info = await info.text()
     console.log(info) `

The output is successful

Uri with ':' inside it are misinterpreted.

let data = await r2('https://bibliotheques-specialisees.paris.fr/ark:/73873/pf0000855431/0001/v0001.simple.selectedTab=otherdocs').text;
console.log(data);

outputs nothing because the : is misinterpreted as a port marker, when it's not a port, but

let data = await r2('https://bibliotheques-specialisees.paris.fr/ark\\:/73873/pf0000855431/0001/v0001.simple.selectedTab=otherdocs').text;
console.log(data);

works.

Furthermore the Promise resolves correctly but with an empty .text, thus leaving us without a way to handle it.

Add peer-dependency for isomorphic-fetch or some equal library

Title says all. No mention about anything in the README nor in the package.json (except from node-fetch which is a dependency.

How to access response status code

I'm surprised it's not covered in docs or tests.

r2("http://localhost:3000/500").json
  .catch(e => {
    console.log("e:", e) // not getting there even for 500
  })
  .then(x => {
    console.log("x:", x) // x is response.body here, how to get response.status?
  })

Enable promise dependency injection (just like node-fetch)

node-fetch allows Promise dependency injection (if you like using bluebird for example). It would be great if we could do the same thing with this wrapper.