hermit-crab / scrapemate

Scraping assistant tool. Editing and maintaining CSS/XPath selectors across webpages.

ScrapeMate Beta

Scraping assistant tool. Editing and maintaining CSS/XPath selectors across webpages.
Available as a Chrome/Chromium and a Firefox extensions.

Quick way to interactively pick and maintain CSS/XPath selectors and their presets for external use: website testing or scraping.

Features include:

• Element picker (from SelectorGadget).
• List of presets ever worked on.
• Preset JSON editor (convenient for use in conjunction with scrapy or any similar tools). Presets selective export/import.
• Quick preview of the data currently selected.
• Support for ::text / ::attr() css pseudo elements and has-class() xpath function.
• Toggling JavaScript for the tab currently working on.

Planned features: nested selectors, more template testing and possibly actual extraction capabilities. See all: TODO.

Notes

• This was first intended as a bookmarklet but due to various limitations bookmarklet mode has been dropped. You can still check out outdated but somewhat working version at this page.

Screenshots

Build

npm install
npm run build
# this will create ./dist/extension folder
# which you can give to chrome as an unpacked extension
# or to firefox as a temporary extension in about:debugging

Icon credits to: Freepik from www.flaticon.com

scrapemate's Issues

I made a Jupyter notebook that parses ScrapeMate's export files and attempts to scrape the data from the webpage with Pyppeteer

https://github.com/sergeyyurkov1/ScrapeMate_Companion/blob/main/TGStat.ipynb

Please take a look and tell me what you think. If you are interested, this can be made into a full Python package.

Thanks for a great tool.

Bookmarklet support

I was just curious why the bookmarklet support was removed for this. It does seem to be working as expected right?

Is there anything that is not supported in the bookmarklet?

Very good but the UI is ugly