pwnall / node-xhr2

XMLHttpRequest emulator for node.js

XMLHttpRequest Emulation for node.js

This is an npm package that implements the W3C XMLHttpRequest specification on top of the node.js APIs.

Supported Platforms

This library is tested against the following platforms.

• node.js 10
• node.js 12

Keep in mind that the versions above are not hard requirements.

Installation and Usage

The preferred installation method is to add the library to the dependencies section in your package.json.

{
  "dependencies": {
    "xhr2": "*"
  }
}

Alternatively, npm can be used to install the library directly.

npm install xhr2

Once the library is installed, require-ing it returns the XMLHttpRequest constructor.

var XMLHttpRequest = require('xhr2');

The other objects that are usually defined in an XHR environment are hanging off of XMLHttpRequest.

var XMLHttpRequestUpload = XMLHttpRequest.XMLHttpRequestUpload;

MDN (the Mozilla Developer Network) has a great intro to XMLHttpRequest.

This library's CoffeeDocs can be used as quick reference to the XMLHttpRequest specification parts that were implemented.

Features

The following standard features are implemented.

• http and https URI protocols
• Basic authentication according to the XMLHttpRequest specification
• request and response header management
• send() accepts the following data types: String, ArrayBufferView, ArrayBuffer (deprecated in the standard)
• responseType values: text, json, arraybuffer
• readystatechange and download progress events
• overrideMimeType()
• abort()
• timeout
• automated redirection following

The following node.js extensions are implemented.

• send() accepts a node.js Buffer
• Setting responseType to buffer produces a node.js Buffer
• nodejsSet does XHR network configuration that is not exposed in browsers, for security reasons

The following standard features are not implemented.

• FormData
• Blob
• file:// URIs
• data: URIs
• upload progress events
• synchronous operation
• Same-origin policy checks and CORS
• cookie processing

Versioning

The library aims to implement the W3C XMLHttpRequest specification, so the library's API will always be a (hopefully growing) subset of the API in the specification.

Development

The following commands will get the source tree in a node-xhr2/ directory and build the library.

git clone git://github.com/pwnall/node-xhr2.git
cd node-xhr2
npm install
npm pack

Installing CoffeeScript globally will let you type cake instead of node_modules/.bin/cake

npm install -g coffeescript

The library comes with unit tests that exercise the XMLHttpRequest API.

cake test

The tests themselves can be tested by running them in a browser environment, where a different XMLHttpRequest implementation is available. Both Google Chrome and Firefox deviate from the specification in small ways, so it's best to run the tests in both browsers and mentally compute an intersection of the failing tests.

cake webtest
BROWSER=firefox cake webtest

Copyright and License

The library is Copyright (c) 2013 Victor Costan, and distributed under the MIT License.

node-xhr2's Issues

window is not defined

I'm trying to use it inside Chrome extension background.js which is service worker.

global.XMLHttpRequest = require('xhr2');
And it gives me this error.
background.js:7 Uncaught ReferenceError: window is not defined

Support for custom User-Agent

Sometimes it is necessary to set a custom User-Agent header. I don't see any possibility to do so (setRequestHeader restricts to set it). It would be good to set it via global, option or nodejsSet.

Turn off keepalive or re-use existing connections.

Hey, something I use depends on this and I'm making, like, tens of thousands of http calls. Mind turning off keepalive so the OS doesn't freak out and run out of outgoing ports? Or if you want to put more effort, re-use existing connections?

Typings for use with Typescript?

Hello,

is there any chance to have typings?
I could lend a hand to achieve this

Support for XMLHttpRequest.responseXML

I don't see any support for that attribute, however it seems that it is in the spec:
http://www.w3.org/TR/XMLHttpRequest/#the-responsexml-attribute

I think it would be really easy to integrate a DOMParser kind of object and add this extension.
What do you think?

Edit: I would contribute a pull request upon confirmation that this an issue, but I'd also require some hint as to how to integrate the xml parser.

Resolve location url according to RFC #7230

Relative locations are allowed when the status code is a redirect. The location url should resolve to the effective request URI: https://tools.ietf.org/html/rfc7230#section-5.5

The issue:

Request http://example.com/redirect
Response { ... statusCode : 303, headers : { Location: "/redirected" ... } }

Expected:
The new request url should be http://example.com/redirected

Actual:
The new request url is /redirected

Local files support

Just to document here for others. I am using the following code for local files. It works "good enough", but probably one could do smarter things with settings _require and _response to some random values, just to be sure.

fs = require 'fs'

URL_BASE = '/srv/files/'

class LocalFilesXMLHttpRequest extends XMLHttpRequest
  _sendRelative: (data) ->
    fullUrl = url.resolve URL_BASE, @_url.href
    @_url = @_parseUrl fullUrl

    @_sendFile data

  _sendFile: (data) ->
    unless @_method is 'GET'
      throw new NetworkError "The file protocol only supports GET"

    if data? and (@_method is 'GET' or @_method is 'HEAD')
      console.warn "Discarding entity body for #{ @_method } requests"
      data = null
    else
      # Send Content-Length: 0
      data or= ''

    @upload._setData data
    @_finalizeHeaders()

    @_request = null
    @_dispatchProgress 'loadstart'

    if @_sync
      defer = (f) ->
        f()
    else
      defer = process.nextTick

    defer =>
      try
        data = fs.readFileSync @_url.pathname
      catch error
        # TODO: Should set status based on the error
        data = new Buffer "#{ error }"
        status = 404

      @_response = null
      @status = status
      @statusText = http.STATUS_CODES[@status]
      @_totalBytes = data.length
      @_lengthComputable = true

      @_setReadyState XMLHttpRequest.HEADERS_RECEIVED

      @_onHttpResponseData null, data
      @_onHttpResponseEnd null

Unsupported send() data function Writer()

Hello, Currently when building a POST request im getting the below error when attempting to send the payload. After debugging i see if fails during this check :

else if (data.buffer && data.buffer instanceof ArrayBuffer) {
// ArrayBufferView
body = Buffer.alloc(data.byteLength);
offset = data.byteOffset;
view = new Uint8Array(data.buffer);
for (i = k = 0, ref1 = data.byteLength; (0 <= ref1 ? k < ref1 : k > ref1); i = 0 <= ref1 ? ++k : --k) {
body[i] = view[i + offset];
}
this._body = body;

then the following exception occurs

Unsupported send() data function Writer() {
/**
* Current length.
* @type {number}
/
this.len = 0;
/*
* Operations head.
* @type {Object}
/
this.head = new Op(noop, 0, 0);
/*
* Operations tail
* @type {Object}
/
this.tail = this.head;
/*
* Linked forked states.
* @type {Object|null}
*/
this.states = null;

Any idea on what I would need to check or change ?

[DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead

Hi
I worked with Angular Universal and I got a DeprecationWarning. While investigation I found that the warning happened in xhr2
To reproduce I made

git clone git://github.com/pwnall/node-xhr2.git
cd node-xhr2
npm install
npm pack
npm test

And I got a DeprecationWarning

XMLHttpRequest
    level 2 events
      for a successful fetch with Content-Length set
(node:10531) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

I found an article Porting to the Buffer.from()/Buffer.alloc() API

node -v
v10.15.3

I can make a PR
Thank you
P.S. Sorry for my English

Backup plan for coffeedocs?

The library's CoffeeDocs link isn't working, I think the service itself is down.
Is there an easy alternative place to host it?

Angular 2 support

hi, we're evaluating your module to use for Universal Angular 2 but we ran into a small problem. With your module you require the url to have a protocol which means a full url. On the client this is not the case since we're able to use relative paths. Do you think we can support relative urls or did I overlook a feature?

pending PR angular/universal#73

node-xhr2 behind a proxy - food for thoughts

I am working on this topic right now.

Browser's XMLHttpRequest objects do not have an API to set an egress proxy, because everything is cabled by the Browser's internals. Node-speaking, the HTTP & HTTPS Agents are managed by the Browser, not by the XMLHttpRequest root object.

I think it would be good to keep the same principles, by adding a setAgents method (or an agents construction time-option) rather than a setProxy. The HTTP and/or HTTPS agents would be created by the upper layer (dropbox-js in this case), depending on the needs.

Opinion?

"This" is undefined in the onload function

"This" is undefined in the onload function. In browsers it holds the request status.

var xhr = new XMLHttpRequest();
xhr.onload = function(e) {
if (this.status == 200) {
//do stuff
}
}
//open and send request

URGENT! Cannot do any request

When I try to require some URL it give me this error:

_http_agent.js:440
      options = req[kRequestOptions];
                   ^

TypeError: Cannot read property 'Symbol(requestOptions)' of undefined
    at Agent.removeSocket (_http_agent.js:440:20)
    at TLSSocket.onClose (_http_agent.js:367:11)
    at TLSSocket.emit (events.js:326:22)
    at net.js:673:12
    at TCP.done (_tls_wrap.js:563:7)

Synchronous XHR processing not implemented

I'm trying to d a synchronous POST, but I'm getting "Synchronous XHR processing not implemented". code:

      req = new XMLHttpRequest()
      req.open('POST', 'http://localhost:23119/better-bibtex/schomd', false)
      req.send(body)

Non-automatic redirects

Is it possible at all to not automatically follow redirects?

gzip support?

I was getting 404 for some file before I did this

    xhr.open(method, url, true)
    if XMLHttpRequest.prototype._restrictedHeaders
        delete XMLHttpRequest.prototype._restrictedHeaders['accept-encoding']
        xhr.setRequestHeader('Accept-Encoding', 'gzip, deflate')

Now I don't get 404, but the data is obviously garbage. Is there easy way to get unzipped content in this case?

Support synchronous function

The synchronous function is very useful for some scenes.

Is this function support now?

Process hangs if request follows a redirection

var XMLHttpRequest = require('xhr2');

var req = new XMLHttpRequest();
req.addEventListener("load", function () {
  console.log(this.responseText);
});
req.open("GET", "http://news.ycombinator.com");
req.send();

This waits about 45 seconds after printing. Getting https://news.ycombinator.com works fine.

Add Blob support

What would be the level of effort to add Blob support (basically, support for xhr.responseType = 'blob' and xhr.send(Blob), as described in the xhr2 tutorial)?

It seems like Blob support has finally made it into all the browsers except for Opera Mini.

Implement automated redirect following

Relevant XHR specification
https://dvcs.w3.org/hg/xhr/raw-file/tip/Overview.html#infrastructure-for-the-send()-method

This is necessary to fix https://github.com/dropbox/dropbox-js/issues/66

Support for withCredentials attribute in request object

The withCredentials attribute

client . withCredentials
True when user credentials are to be included in a cross-origin request. False when they are to be excluded in a cross-origin request and when cookies are to be ignored in its response. Initially false.

When set: throws an InvalidStateError exception if state is not unsent or opened, or if the send() flag is set.

https://xhr.spec.whatwg.org/#the-withcredentials-attribute

Is it in plan to support withCredentials attribute so, at least to be by default to true, so code from browser can be used in server environment?

nodejs: The url module has expired and the URL class is recommended

for example: url.parse should be replaced with new URL().

When a request is sent in a browser, \{} is not escaped. However, in xhr2, they are escaped as %5C%7B%7D.

Emit progress events

The library doesn't emit progress events for uploading data.
http://www.w3.org/TR/XMLHttpRequest/#make-upload-progress-notifications

The underlying problem is that we use one write call to send the entire data.
https://github.com/pwnall/node-xhr2/blob/master/src/xml_http_request_upload.coffee#L88

I can't work on this right now, but PRs are welcome.

package.json has wrong repository url

The repository url in your package.json is wrong.

Support for responseURL

Available in some browsers these days

Deprecated url.parse() is used - results in malformed request

url.parse is a part of now deprecated Legacy API
https://nodejs.org/api/url.html#url_legacy_url_api

Url parsing differs from a recommended WHATWG: https://nodejs.org/api/url.html#url_percent_encoding_in_urls

In certain cases it results in malformed http requests.

`​​​Discarding entity body for GET requests​​​` logging is surfacing through all my XHR requests

Hi @pwnall

First, thank you for patching xhr in Node! I'm using this as a cross-platform shim, but when I'm in node, I'm getting a bunch of Discarding entity body for GET requests​​​ warnings that are injected throughout my logs. The culprit line is here. Would you mind silencing this by default and allowing users to opt-in to see it? Here are a couple issues I found referencing the same complaint:

• request/request-promise#210
• r0man/cljs-http#94 (comment)

Just goes to show your library is very popular!

XHR2 is automatically parsing JSON

XMLHttpRequest in my browser (e.g. Chrome) just gives me responseText and then I call JSON.parse() on the text to get it into JSON format.

xhr2 appears to automatically detect the content type is JSON and does the parse for me.

This makes my nodejs integration tests incompatible with my browser app. :(

Edit: Nevermind. I re-read the spec. I should be parsing responseText if I want the raw text response. Sorry!