mikeal / bent

Functional JS HTTP client (Node.js & Fetch) w/ async await

The bodhisattva vow is a form of desire that holds as its object the union of liberation of self and liberation of all others.

-- Asanga (4th century C.E.)

bent's Issues

Exception StatusError does not set name to 'StatusError'

The exception StatusError does not set parent name property to 'StatusError'.

Suggested (in nodejs.js):

class StatusError extends Error {
  constructor (res, ...params) {
    super(...params)

    Error.captureStackTrace(this, StatusError)
    this.name = 'StatusError' // <-- add this
    this.message = `Incorrect statusCode: ${res.statusCode}`
...

Suggestion typescript

I jumped here from Request NodeJS library. That project is now deprecated and I read that you started this new project in order to create a modern javascript library with the latest features. But, what do you think about Typescript? Could be better to refactor the project code to Typescript and continue with that awesome language? What do you think about that?

How to post to a relative URL?

I want to post to "/", but bent throws an error.

I wanted to replace the fetch in the contact form component, that posts to the server (in my case Netlify):

fetch('/', {
  method: 'POST',
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
  body,
})

I don't want to have the domain in the URL because the component would need to know the URL on each domain that it is used on.

POST body is empty

I'm trying to send some JSON via POST like this:

const bent = require('bent');
const postJSON = bent('POST', 'json');
const token = 'xxx';
const user = await postJSON('http://localhost:3000/endpoint', { token });

I'm running a restify-server that is hosting the endpoint - including the bodyParser-plugin:

const restify = require('restify');
const server = restify.createServer();

server.use(restify.plugins.bodyParser({ mapParams: true }));

server.post('/endpoint', (req, res, next) => {
  console.log('endpoint', req.body);
  res.send({a: 'ok!'});
});

server.listen(3000, () => {
  console.log('%s listening at %s', server.name, server.url);
});

But req.body is undefined.

Tried the same with a small expressjs setup - results were the same.

Saw in the test, that you're doing something similar with PUT:

bent/tests/test-basics.js

Lines 141 to 142 in e018c13

	let request = bent('PUT', 'string')
	let str = await request('http://localhost:3000/', {ok: 200})

This seems to work. Any ideas why it isn't working with restify/express?

Custom encodings API

I'd like to add an xml encoding to simplify working with a 3rd party API without having to call the parse method every time. Do you have any plans add custom encodings?

Support for HTTP multipart

Hi.

I haven't seen any reference to multipart requests in your documentation. Is there an elegant way to handle multipart using bent?

Thx.

Error: invalid block type at Zlib.zlibOnError -- Unhandled 'error' event

bent is getting a zlib error and not handling the emitted error event on the stream.

getting this error when calling a service:

events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: invalid block type
    at Zlib.zlibOnError [as onerror] (zlib.js:164:17)
Emitted 'error' event at:
    at errorOrDestroy (internal/streams/destroy.js:107:12)
    at Gunzip.onerror (_stream_readable.js:732:7)
    at Gunzip.emit (events.js:198:13)
    at Zlib.zlibOnError [as onerror] (zlib.js:167:8)

recreating the issue with minimal code I can get a similar error

• To recreate the issue

1. run node bentexampleserver.js (code below)
2. run node benterror.js (code below)
3. you will get this error

      throw er; // Unhandled 'error' event
      ^
Error: incorrect header check
    at Zlib.zlibOnError [as onerror] (zlib.js:164:17)
Emitted 'error' event at:
    at errorOrDestroy (internal/streams/destroy.js:107:12)
    at Gunzip.onerror (_stream_readable.js:732:7)
    at Gunzip.emit (events.js:198:13)
    at Zlib.zlibOnError [as onerror] (zlib.js:167:8)

//bentexampleserver.js

const express = require('express');
const app = express();
const port = 3000;
app.get('/', (req, res) => {
  res.append('content-type', 'application/json');
  res.append('content-encoding', 'gzip');
  res.status(400);
  res.json({ test: true });
});

app.listen(port, () =>
  console.log(`Example app listening at http://localhost:${port}`)
);

//benterror.js

const bent = require('bent');
var request = require('request');
var getJSON = bent('json', 200, 400);

async function test() {
  try {
    let response = await getJSON('http://localhost:3000/');
    console.log({ response });
  } catch (e) {
    console.error('CATCH ERROR', e);
  }
}
test();

Have URL; want to make request

In order to use bent, I think I need to specify a base URL first, then the rest of the URL in the next call.

What if I just have a URL? Does that mean I need to parse it into 2 pieces myself before using bent to make a request?

Support for multiple status codes or the ability to retrieve the failed response

Use case:
I want to provide context sensitive information to the end-user relating to what went wrong and why. The responses from the API might be a 200 for a post request if the result was successful, however, if the API discovered invalid input then a 400 would be returned with information on why validation failed.

Problem:
Bent only seems to support one status code per request.

Investigation:
However, it does look like multiple codes were a consideration at some point. core.js shows the statusCode type as a set and further code checks the set for a matching value. In-fact, changing the code in core.js:29 from:

    } else if (typeof arg === 'number') {
      statusCodes.add(arg)

to:

    } else if (typeof arg === 'number' || Array.isArray(arg)) {
      (Array.isArray(arg) ? arg : [arg]).forEach(a => statusCodes.add(a));

enables the use of multiple status codes.

responseBody in the StatusError object does not get unzipped when the server uses gzip encoding

I noticed when handling StatusError thrown when the server returns an HTTP 400 error, the responseBody promise resolves to a Buffer containing the raw gzipped data returned by the server.

const get = bent('http://www.api.com', 'GET', 'json');
const url = '/endpoint';

try {
  response = await get(url);
} catch (err) {
  err.responseBody.then(body => {
    console.log(body);
  });
}

I would expect the err.responseBody promise to resolve to a buffer (or optionally string?) containing the decompressed data.

Something odd about versions

The latest version on npm is 7.3.1. However package.json on Github shows 7.0.3. Are updates no longer being pushed to Github?

Support for debug requests

Would be great to have a support for debugging requests (logging what was sent/received).
Somthing similar to what package debug offers.

Typescript / types

Have you thought about writing this library in typescript?
Or provide typescript support?

https://www.triplet.fi/blog/three-ways-to-provide-typescript-type-definitions-to-3rd-party-libraries/

It seems to be gaining quite a bit of traction in the javascript community:
https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fm%2F0n50hxv,%2Fm%2F0hjc5m0,%2Fm%2F0h52xr1,es6

Design question

Hi!

I generally like the lean approach bent has and think it's the right direction, but I dislike the parameter guessing.

It makes creating precise Typescript interfaces a hassle and makes the documentation a lot harder. Is there a reason why it can't just accept and object with documented properties?

On big projects it creates unnecessary amount of variation and makes refactoring and finding also harder.

Other than that, I really like it!

Support limit of redirects

In my case, I want to follow no redirects at all, so I'd set that limit to 0.

Error Response Contains Buffer, Not Content

When making a call to a REST API, it is possible for the response to contain an error status code AND contain a valid body with information as to why there was an error.

  return await get(url).catch(async e => {
    let msg = await e.responseBody

The resulting body contains a buffer, but I'm assuming it hasn't been decompressed yet?

Error: Incorrect statusCode: 201

const bentData = bent(myUrl, 'POST', 'json', 200);
const response = await bentData(myKey, myData);
return response;

Error: Incorrect statusCode: 201

I got this error log sometimes.
But, the myData is uploaded completely.
why do I see this error..?

GET request with body omits Content-Length header

I'm trying to send a GET request with JSON body (to an elasticsearch server). Relying on bent to serialize my JSON body results in a missing Content-Length header in the request. This causes my server to ignore the body. Here's some sample code to illustrate the problem and how I work around it. I test locally with nc -l -p 9999 & to see the received request.

const bent = require("bent");
// This code sends GET with proper Content-Type and the expected JSON body
// but there's no Content-Length header, so the server ignores the body
bent("GET", "http://localhost:9999", "json")(`/path`, { some: "body" });

// Here's netcat's dump of the received request
/*
GET /path HTTP/1.1
accept: application/json
Host: localhost:9999
content-type: application/json
Connection: close

{"some":"body"}
*/

---

const bent = require("bent");
// This works OK
const encoded = JSON.stringify({ some: "body" });
const headers = {
  "Content-Length": encoded.length,
  "Content-Type": "application/json"
};
bent("GET", "http://localhost:9999", "json", headers)(`/path`, encoded);

// Here's netcat's dump of the received request

/*
GET /path HTTP/1.1
Content-Length: 15
Content-Type: application/json
accept: application/json
Host: localhost:9999
Connection: close

{"some":"body"}
*/

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

Tests failing

I haven’t been able to get the tests passing after:

1. Clone repo
2. npm i
3. npm test

Error:

  23 passing (16s)
  2 failing

  1) basic 200 ok:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/aral/small-tech/site.js/forks/bent/test/test-basics.js)
      at listOnTimeout (internal/timers.js:549:17)
      at processTimers (internal/timers.js:492:7)

  2) basic json:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/aral/small-tech/site.js/forks/bent/test/test-basics.js)

What was the breaking change in 7?

Curious what the breaking change in v7 was. Is it documented anywhere?

Support for credentials mode

https://fetch.spec.whatwg.org/#concept-request-credentials-mode

test: automate bundle size checking

It would be great if we could get a GitHub action that checked and let us know the browser bundle size on an ongoing basis.

How can I post a url encoded form with bent?

I am trying to convert this requests options object to bent:

var options = {
  method: 'POST',
  url: "some-url",
  headers: {'content-type': 'application/x-www-form-urlencoded'},
  form: {
    foo: 'bar'
  }
};

Is it something like this?

const getJSON = bent('POST', 'json', 200);

const token = await getJSON(
            url,
            {
                 foo: 'bar'
            },
            { 'content-type': 'application/x-www-form-urlencoded' }
        )

Can't get it to work 🤔

Error: Cannot find module './core'

I'm creating a Node socket server. If I require bent I get a compile error Error: Cannot find module './core':

const express = require('express');
const SocketServer = require('ws').Server;
// if I include the below line I get the compile error
// const bent = require('bent');

const port = process.env.PORT || 3333;
const server = express()
  .use((req, res) => {
    res.sendFile(INDEX);

    res.header('Access-Control-Allow-Origin', '*');
    res.header('Access-Control-Allow-Methods', 'GET, OPTIONS');
    res.header('Access-Control-Allow-Headers', 'Content-Type');

    return next();
  })
  .listen(port, () => console.log(`Listening on ${port}`));
const socketServer = new SocketServer({ server });

1.4.0 showing as latest release on GitHub releases page

Not sure why but 1.4.0 is showing as the latest release instead of 7.1.2 here: https://github.com/mikeal/bent/releases

Slice the codebase

First of all, congrats, this project is cool and probably consumed you a lot of time and work.

I suggest as the next step to split the codebase into multiple files, and do some refactoring, to improve project growing, maintenance and ease to new contributors understanding, is OK if I do some work on this?

If is OK, since this project is personal, I'll try to do less opinionated changes possible.

TSLint error when importing

import * as bent from 'bent';

results in error message when compiling

ERROR in [at-loader] ./node_modules/@types/bent/index.d.ts:50:26
TS2304: Cannot find name 'Response'.

installed typescript package is
"@types/bent": "^7.0.1",

Could be fixed by adding the import Response from https

Error object does not contain response headers

Use case

Access to response headers might be required in certain circumstances to retry calls. For example, according to RFC 8555 § 6.5 Replay Protection, ACME servers (e.g., Let’s Encrypt) might reject the nonce sent to the server. When they do, they send a 400 error with a valid nonce in the Replay-Nonce header field. Clients a meant to retry the call using the nonce sent by the server.

To recreate

1. Execute a call that results in a server error using bent
2. Examine the caught error object

What should happen

The returned error object should have a responseHeaders property with the response headers (just like it has a responseBody for the response body.

What actually happens

The headers are missing from the error object.

Error when POSTing JSON and response is blank

If you post JSON to a website that returns an empty response, the following error will be thrown:

SyntaxError: Unexpected end of JSON inputstr""
    at JSON.parse (<anonymous>)
    at ClientRequest.<anonymous> (node_modules\bent\src\nodejs.js:109:24)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

The returned response is completely empty and the status code is 202. 202 has been added as an acceptable status code for the request.

Single-function exports make unit testing difficult

I like the look of this library but have a concern -- is it possible to use with a module-style export, i.e.:

const bent = require('bent');

bent.get('blah');
bent.request('blah');

I know jest is all the rage these days (and jest has ways around this), but many projects use jasmine or mocha/sinon, and it's very difficult to write a reasonable test case for a class that uses bent if it must be called as an exported function. For this same reason I never used the request function pattern either, but always called a function on the module, e.g.:

const request = require('request-promise');

async function easyToTest(url) {
    return request.get(url);
}

describe('#easyToTest', function() {
    it('works', async function() {
        spyOn(request, 'get').and.returnValue(Promise.resolve('OK'));
        expect(await easyToTest('google.com')).toEqual('OK');
        expect(request.get).toHaveBeenCalledWith('google.com');
    });
});

Any thoughts on this pattern?

Access to response headers

Is it possible to gain access to response headers? Looking at proxying a HEAD request, and I need access to the response headers to forward some of them.

How can i use proxy?

Handling of a-gzip compressed content

I am trying to retrieve data using Apple's Appstore Connect API and they return data using a-gzip compression. Zlib is capable of handling it, but since the encoding name is not exactly gzip bent throws an error (which by the way is not caught despite the call being in a try block in my code)

(node:52064) UnhandledPromiseRejectionWarning: TypeError: compression[enc] is not a function

I see two paths to solve this problem, one is adding a key 'agzip' to the compression object defined in nodejs.js and have it pointing to the same function as 'gzip'. Alternatively change the behaviour so that the getResponse method would not try to decompress the content if the encoding/compression is not known (my favourite option).

I can provide a pull-request for either solution, but I would leave the choice of which path to follow to @mikeal

Automatic setting of content-type header overwrites manually-specified header

Bent overwrites a manually-specified content-type header when it detects that a request has a body.

To reproduce

1. Create a POST request
2. Manually pass a headers object with:

   {
     'Content-Type': 'application/jose+json'
   }

3. Carry out the call.

What should happen

Request’s content type should be application/jose+json.

What actually happens

Request’s content type is application/json.

Use case

RFC 8555 requires the content type of calls to be set to application/jose+json.

PS

Thanks for bent, it’s a joy to work with :)

404 response message gets wrapped to 'Incorrect statusCode: ${res.statusCode}'

When I receive 404 response the message gets overwritten to
Incorrect statusCode: ${res.statusCode}

instead of passing the original message.

Could you pass the original message? Why the Incorrect statusCode: wrapping?

It could be fixed by
this.message = res.statusMessage

Unexpected Identifier

When I try to use the sample code from the readme I get an unexpected identifier error for the put call.
Node: v8.1.2

const bent = require('bent')

const put = bent('PUT', 201)
await put('http://site.com/upload', Buffer.from('test'))

SyntaxError: Unexpected identifier
at createScript (vm.js:74:10)
at Object.runInThisContext (vm.js:116:10)
at Module._compile (module.js:533:28)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Function.Module.runMain (module.js:605:10)
at startup (bootstrap_node.js:158:16)
at bootstrap_node.js:575:3

Thanks for your help

oops, packaging issue? can't include core

Failure:

internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module './core'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/Users/srl/src/gp-js-client/node_modules/bent/src/nodejs.js:8:14)

Debug:

$ npm info bent

…
.tarball: https://registry.npmjs.org/bent/-/bent-1.5.14.tgz
…
$ curl https://registry.npmjs.org/bent/-/bent-1.5.14.tgz | tar tvf -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2792  100  2792    0     0   8819      0 --:--:-- --:--:-- --:--:--  8835
-rw-r--r--  0 0      0         774 Oct 26  1985 package/package.json
-rw-r--r--  0 0      0        2289 Oct 26  1985 package/README.md
-rw-r--r--  0 0      0        1395 Oct 26  1985 package/src/browser.js
-rw-r--r--  0 0      0        2756 Oct 26  1985 package/src/nodejs.js

Support keepAlive in Node.js implementation

Most Node.js services heavy in HTTP calls to same destination benefit a lot of having keepAlive on by default, e.g. we had to do it passing agent when using node-fetch:

https://github.com/nearprotocol/nearlib/blob/master/src.ts/utils/web.ts#L24

Is there any way to enable keepAlive when using bent? Also wonder if there is any good reason to not use it by default.

Support to self-signed certificates (UNABLE_TO_VERIFY_LEAF_SIGNATURE)

Hi,

Is it possible to disable the validation of the certificate provided by the server?

Currently, if I access an HTTPS URL that provides a self-signed certificate, I'm getting an error like this:

code:"UNABLE_TO_VERIFY_LEAF_SIGNATURE"
message:"unable to verify the first certificate"
stack: "Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1058:34)
    at TLSSocket.emit (events.js:198:13)
    at TLSSocket._finishInit (_tls_wrap.js:636:8)"

Regards.

Error response body is a still promise

It seems the issue #81, and #91 by extension, is still happening.

Code to reproduce:

const bent = require('bent');

(async () ={
    try {
        await bent('POST', 'json', `https://google.com`)();
    } catch (err) {
        console.error(err);
    }
})();

Outputs:

Promise { pending }

Should bent be split into Node.js and browser versions?

I noticed, while adding responseHeaders to error objects (https://github.com/mikeal/bent/pull/93/files), that the Node and Browser implementations differ in what the responses return. On Browser (unless I’m missing something), I couldn’t see the same details in the response objects as on Node. (For example, no header information.)

Not having looked at depth into the underlying Browser implementation, I’m wondering:

• Should bent be split into two modules (node, browser) with different expectations
• Should the browser and node implementations be made to match perfectly (if possible)

Thoughts?

(This is not a huge issue for me personally and definitely not a hill I wish to die on as I’m only using it in Node but I’m assuming it might be confusing for folks who use it in both Node and Browser if they encounter different behaviour.)

Please feel free to close if not a priority for project goals.

Support for timeout like in axios

See: axios/axios#1503
Being able to pass a timeout to the async call is convenient.

Expose options for underlying http client

Is there a nice way here for exposing the additional configuration options for the underlying http client, such as timeout configuration?

Support for time limit

Could you pls add support for time limit as it was possible with the request because this is important feature and wrapping every bent into a array of promises in order to include setTimeout requires quite a lot of lines of code and makes the code hard to read.

Eg. timeout for one second
async request(url[, body=null, headers={}, timeout=1000])

Bent ignores "username:password" basic auth

The nodejs version of Bent strips credentials included in the URL, forcing users to explicitly construct an Authorization header: https://github.com/mikeal/bent/blob/master/src/nodejs.js#L72-L78

username/password not supported

looks like Basic auth could be supported with a dependency on https://www.npmjs.com/package/basic-authorization-header

Error response body is a promise

It seems the issue #81 is still happening.

Code to reproduce:

const bent = require('bent');

(async () => {
    try {
        await bent('POST', 'json', `https://google.com`)();
    } catch (err) {
        console.error(err);
    }
})();

Outputs:

{ Error: Incorrect statusCode: 405
   ........
  message: 'Incorrect statusCode: 405',
  statusCode: 405,
  responseBody: Promise { <pending> } }

Wrong parsing of Username

It parses an @ symbol in the Username wrong.
It is replaces by %40, when sent to the server.
nodejs.js -> Line 84. parsed.username, parsed.password should be url decoded before passing them on

Access to headers in 'json' mode

Use case

While implementing the ACME (RFC 8555) spec, I need access to both the headers and the body in the POST requests that ACME expects so I cannot use the helpful 'json' return type as that does not include headers. The way I’m handling that is to have a native response object returned and then basically clone what bent itself does using the getBuffer() function and JSON.parse().

So, for example:

async f () {
  // copied from bent
  const getBuffer = stream => new Promise((resolve, reject) => {
    const parts = []
    stream.on('error', reject)
    stream.on('end', () => resolve(Buffer.concat(parts)))
    stream.on('data', d => parts.push(d))
  })

  // assume that url, data, and headers variables are defined earlier
  const prepareRequest = require('bent')
  const request = prepareRequest('POST', url, 200, 201)

  let response, errorBody
  try {
    response = await request('', data, headers)
  } catch (error) {
    errorBody = error.responseBody
  }

  if (errorBody) {
    const errorBodyBuffer = await errorBody
    const errorMessage = errorBodyBuffer.toString('utf-8')
    throw new Error(errorMessage)
  }

  // The response returned is the raw response object. Let’s consume
  // it and return a more relevant response.
  const headers = response.headers
  const responseBodyBuffer = await this.getBuffer(response)
  const body = JSON.parse(responseBodyBuffer.toString('utf-8'))

  const responseObject = {
    headers,
    body
  }

  return responseObject
}

Would it be worthwhile adding additional modes for, e.g., jsonWithHeaders, stringWithHeaders, etc., that follow a similar approach.

Even if not, I wanted to type this up here in case anyone else benefits from it. (Might be an idea to include it in the docs?)