node-site-scrapper's Introduction

All Origins

An open source alternative to AnyOrigin.com with support to gzipped pages and CORS

"Same-Origin-Policy? Not on my watch."

A free and open source javascript clone of AnyOrigin, inspired on Whatever Origin, but with support to gzipped pages.

Use it

Usage is similar to anyorigin and whateverorigin. For example, to fetch the data from http://google.com with jQuery, use this snippet:

$.getJSON('http://api.allorigins.ml/get?url=' + encodeURIComponent('http://google.com'), function(data){
    alert(data.contents);
});

Or via https

$.getJSON('https://api.allorigins.ml/get?url=' + encodeURIComponent('https://google.com'), function(data){
    alert(data.contents);
});

You can also set the response character encoding (charset):

$.getJSON('https://api.allorigins.ml/get?charset=ISO-8859-1&url=' + encodeURIComponent('https://google.com'), function(data){
    alert(data.contents);
});

To get the the raw content (CORS), just add &method=raw

$.get('https://api.allorigins.ml/get?method=raw&url=' + encodeURIComponent('https://google.com'), function(data){
    console.log(data);
});

To use JSONP add &callback=?:

$.getJSON('https://api.allorigins.ml/get?charset=ISO-8859-1&url=' + encodeURIComponent('https://google.com') + '&callback=?')
.done(function (data) {
  console.log(data.contents);
});

On your own server

# Clone the repo
git clone https://github.com/gnuns/AllOrigins

# Install dependencies
cd allorigins
npm install

# Fire it up!
node server.js # the default port is 1458