jonstuebe / product-scraper

Node.js based scraper for ecommerce sites

Product Scraper

Node.js based scraper for ecommerce sites

Deprecated

This repo is no longer being updated. I've made a new generalized site scraper that is not specific to ecommerce sites. Please refer to this new project: new site

Installation

$ npm install product-scraper

Features

• Scrape top ecommerce sites (Amazon, Walmart, Target, BestBuy, Etsy)
• Return basic product information (title, price, details, images, description)
• Easy to use API

API

Simply require the package and initialize with a url and pass a callback function to receive the data.

var scraper = require('product-scraper');

scraper.init('http://www.amazon.com/gp/product/B00X4WHP5E/', function(data){
	console.log(data);
});

Todos

• Need to add ability to run a test to see if markup has changed, and if so disable the store selectors and fallback to the generic scraper.

Store Owner?

If you are a store owner and wondering how to make sure your site is being crawled, using the following attributes on your markup:

[itemprop="title"]
[itemprop="brand"]
[itemprop="description"]
[itemprop="image"]
[itemprop="price"]

Contributing

Currently, this plugin supports a few top ecommerce sites based on alexa ratings. If you want to add any stores, or just have an idea or feature, go ahead and fork this repo and send me a pull request. I'll be happy to take a look when I can and get back to you.

Issues

For any and all issues/bugs, please post a description and code sample to reproduce the problem on the issues page.

License

MIT

product-scraper's Issues

Failing tests

5/5 tests are failing.

Apologies for all the new issues opened, I'm not sure if this project is still active but just wanted to leave some feedback.

Selectors need to be updated

Looks like Best Buy may have updated their product page markup a bit, affecting the current selectors that parse the product info. I didn't really test all e-commerce sites so it's possible some of the others are also affected.

Those seem very rigid and could probably be a bit more generic although there is no guarantee that they won't also break.

Example:

var scraper = require('product-scraper');
 
scraper.init('http://www.bestbuy.com/site/samsung-40-class-40-diag--led-2160p-smart-4k-ultra-hd-tv-dark-titan/5622308.p?skuId=5622308', function(data){
    console.log(data);
});

Response:

{
    title: 'Samsung - 40" Class (40" Diag.) - LED - 2160p - Smart - 4K Ultra HD TV - Dark titan',
    image: 'http://pisces.bbystatic.com//image2/BestBuy_US/images/products/5622/5622308_sa.jpg;maxHeight=210;maxWidth=210',
    details: ['40" screen (measured diagonally from corner to corner)A good size for a living room, or if you want a larger bedroom TV.',
        'Built-in Wi-Fi Smart TV means a huge world of entertainmentStream movies, videos, music, and more using Smart Hub. S Recommendation finds the movies and TV shows you\'ll enjoy most.',
        'ConnectShare MovieWatch videos, play music and view photos from an USB connection without needing to use a PC or special equipment.',
        'Up-scale entertainment to near-4K qualityUHD engine up-scales the picture through a multi-step data analysis that enhances the details of movies, sports, video games, and more.',
        'LED TVs perform well in all lighting conditionsThey also deliver plasma-like deep blacks and rich colors.',
        '2160p resolution for breathtaking HD imagesWatch 4K movies and TV shows at 4x the resolution of Full HD, and upscale your current HD content to gorgeous, Ultra HD-level picture quality.',
        'Wireless ConnectivityProvides fast and reliable streaming with Wi-Fi.',
        '3 HDMI inputs for the best home theater connectionHigh-speed HDMI delivers up to 2160p picture and digital surround sound in one convenient cable. HDMI cable not included.',
        'Manage the viewing experience for your kidsThe built-in V-chip lets you block content based on program ratings and check ratings of unfamiliar programs.',
        'Motion Rate 120Enjoy high-speed action with good motion clarity.',
        '2 USB inputsEasily connect your digital camera, camcorder or other USB device.',
        'Web-based services and content require high-speed Internet service. Some services may require a subscription.'
    ],
    description: 'Enjoy your favorite shows on this Samsung 40-inch 4K Smart TV. The 120Hz refresh rate ensures smooth display, even during complex action scenes, and the 2160p resolution lets you see the smallest details. Multiple ports make it easy to connect gaming systems and media players to this Samsung 40-inch 4K Smart TV.',
    url: 'http://www.bestbuy.com/site/samsung-40-class-40-diag--led-2160p-smart-4k-ultra-hd-tv-dark-titan/5622308.p?skuId=5622308'
}

Notice how there is no price available.

Walmart and React

Looks like Walmart updated their product page and most of it is now driven by JS after the page is loaded. That means cheerio won't be able to parse most of the content since it only reads the initial HTML.

It might be a good idea to consider using something like PhantomJS or SlimerJS