Topic: web-scraping Goto Github

👇 Here are 5073 public repositories matching this topic...

a9t9 / rpa
web-scraping,UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++
User: a9t9
Home Page: https://ui.vision/rpa
automation web-automation selenium-ide visual-recognition ui-tests imacros autohotkey sikulix opencv webassembly

achuthasubhash / complete-life-cycle-of-a-data-science-project
web-scraping,Complete-Life-Cycle-of-a-Data-Science-Project
User: achuthasubhash
feature-engineering unsupervised-learning data-science deep-learning machine-learning exploratory-data-analysis reinforcement-learning eda pytorch supervised-learning

adbar / trafilatura
web-scraping,Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
User: adbar
Home Page: https://trafilatura.readthedocs.io
web-scraping text-extraction nlp html2text news text-mining crawler text-cleaning text-preprocessing article-extractor

alecxe / scrapy-fake-useragent
web-scraping,Random User-Agent middleware based on fake-useragent
User: alecxe
web-scraping python scrapy

alexmathew / scrapple
web-scraping,A framework for creating semi-automatic web content extractors
User: alexmathew
Home Page: http://alexmathew.github.io/scrapple
python css-selector xpath-expression web-scraper web-scraping scrapers scraping scrapy selector extractor

alirezamika / autoscraper
web-scraping,A Smart, Automatic, Fast and Lightweight Web Scraper for Python
User: alirezamika
scraping scraper scrape webscraping crawler web-scraping ai artificial-intelligence python webautomation

apify / crawlee
web-scraping,Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Organization: apify
Home Page: https://crawlee.dev
web-scraping web-crawling npm headless-chrome puppeteer automation apify scraping crawling crawler headless scraper web-crawler javascript nodejs playwright typescript

austinoboyle / scrape-linkedin-selenium
web-scraping,`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
User: austinoboyle
selenium selenium-webdriver linkedin scraping web-scraper web-scraping python scrape scraper

city-bureau / city-scrapers
web-scraping,Scrape, standardize and share public meetings from local government websites
Organization: city-bureau
Home Page: https://cityscrapers.org
web-scraping python open-data scrapy city-scrapers

codingforentrepreneurs / 30-days-of-python
web-scraping,Learn Python for the next 30 (or so) Days.
User: codingforentrepreneurs
Home Page: https://cfe.sh/projects/30-days-python-38
api automation csv fastapi flask jupyter pandas python python3 rest-api selenium selenium-webdriver tutorial web-scraping

crwlrsoft / crawler
web-scraping,Library for Rapid (Web) Crawler and Scraper Development
Organization: crwlrsoft
Home Page: https://www.crwlr.software/packages/crawler
crawling php scraper scraping scraping-websites web-crawler web-crawling web-scraping hacktoberfest crawler

datahenhq / till
web-scraping,DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Organization: datahenhq
Home Page: https://till.datahen.com
web-scraping man-in-the-middle proxy-server mitm scraping crawler scraper

dgtlmoon / changedetection.io
web-scraping,The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
User: dgtlmoon
Home Page: https://changedetection.io
website-monitor website-monitoring change-detection monitoring self-hosted change-alert change-monitoring website-change-monitor url-monitor changedetection website-change-detector website-change-detection website-change-tracker website-change-notification notifications web-scraping restock-monitor website-defacement-monitoring back-in-stock website-watcher

dinubs / coolqlcool
web-scraping,Nextjs server to query websites with GraphQL
User: dinubs
Home Page: https://cool.coolql.cool
graphql schema nextjs javascript web-scraping

evil0ctal / douyin_tiktok_download_api
web-scraping,🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
User: evil0ctal
Home Page: https://douyin.wtf
python pywebio tiktok douyin api scraper fastapi no-watermark online-parsing async douyin-tiktok-api douyin-tiktok-download crawler spider web-scraping aiohttp asgi asyncio tiktok-scraper douyin-scraper

gildas-lormeau / single-file-cli
web-scraping,CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
User: gildas-lormeau
cli deno nodejs single-file web-archiving web-scraper web-scraping

go-rod / rod
web-scraping,A Devtools driver for web automation and scraping
Organization: go-rod
Home Page: https://go-rod.github.io
cdp chrome-headless chrome-devtools chrome-devtools-protocol headless web-scraping automation scraper devtools devtools-protocol

gosom / google-maps-scraper
web-scraping,scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
User: gosom
golang google-maps-scraping web-scraper web-scraping distributed-scraper distributed-scraping google-maps

infinilabs / crawler
web-scraping,🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
Organization: infinilabs
spider crawler lightweight elasticsearch web-crawler crawling web-spider web-scraping scraping

intoli / user-agents
web-scraping,A JavaScript library for generating random user agents with data that's updated daily.
Organization: intoli
user-agent user-agent-spoofer random randomization javascript navigator browsers browser-automation web-scraping

jaebradley / basketball_reference_web_scraper
web-scraping,NBA Stats API via Basketball Reference
User: jaebradley
Home Page: http://jaebradley.github.io/basketball_reference_web_scraper/
basketball-reference python nba web-scraping web-scraper

je-suis-tm / web-scraping
web-scraping,Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
User: je-suis-tm
Home Page: https://je-suis-tm.github.io/web-scraping
sraping scrapper futures futures-historical-data reuters wall-street-journal bloomberg python-web-scraper news-websites financial-times

juancarlospaco / faster-than-requests
web-scraping,Faster requests on Python 3
User: juancarlospaco
Home Page: https://gist.github.com/juancarlospaco/37da34ed13a609663f55f4466c4dbc3e
python python3 python-library http-requests python-requests web-scraping download-file speed curl cython

justmarkham / dat8
web-scraping,General Assembly's 2015 Data Science course in Washington, DC
User: justmarkham
data-science machine-learning scikit-learn data-analysis pandas jupyter-notebook python course linear-regression logistic-regression

lit26 / finvizfinance
web-scraping,Finviz analysis python library.
User: lit26
Home Page: https://finvizfinance.readthedocs.io/en/latest/
web-scraping fundamental-analysis stock-news technical-analysis inside-trader financial-analysis pandas stock-charts fundament screener

lorien / awesome-web-scraping
web-scraping,List of libraries, tools and APIs for web scraping and data processing.
User: lorien
captcha-bypass captcha-recaptcha crawler crawling crawling-framework crawling-python crawling-tool scraping scraping-framework scraping-python scraping-tool spider web-scraping webscraping

lorien / grab
web-scraping,Web Scraping Framework
User: lorien
Home Page: https://grab.readthedocs.io
web-scraping http-client framework python pycurl asynchronous network urllib3 spider crawler

mherrmann / helium
web-scraping,Lighter web automation for Python
User: mherrmann
chrome firefox helium python python3 selenium selenium-python web-automation web-scraping webdriver

oxylabs / amazon-scraper
web-scraping,Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
Organization: oxylabs
Home Page: https://oxylabs.io/products/scraper-api/ecommerce/amazon?utm_source=git-internal&utm_medium=github&utm_campaign=repo&utm_content=oxylabs/amazon-scraper
amazon amazon-api amazon-scraper amazon-scraping e-commerce-api web-scraping csharp javascript python api

php-curl-class / php-curl-class
web-scraping,PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
Organization: php-curl-class
Home Page: https://www.phpcurlclass.com/
api api-client class client curl framework http http-client http-proxy json php php-curl php-curl-library proxy requests restful web-scraper web-scraping web-service xml

platonai / pulsarrpa
web-scraping,Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
User: platonai
web-crawler web-mining data-science web-sql crawler scraper scraping web-scraping data-mining rpa

postmodern / spidr
web-scraping,A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
User: postmodern
spider ruby spider-links crawler web scraper web-scraping web-spider web-crawler web-scraper

programminghistorian / jekyll
web-scraping,Jekyll-based static site for The Programming Historian
Organization: programminghistorian
Home Page: http://programminghistorian.org
programming-historian text-analysis api data-management data-manipulation data-mining pedagogy linked-open-data mapping network-analysis exhibits scraping python dh digital-humanities multi-lingual open-educational-resources open-source r-studio web-scraping

riimuru / gogoanime-api
web-scraping,Anime Streaming, Discovery API made with Cheerio and Express. Uses data from Gogoanime
User: riimuru
anime anime-scraper anime-search anime-list gogoanime gogoanime-api api rest-api streaming web-scraping

roach-php / core
web-scraping,The complete web scraping toolkit for PHP.
Organization: roach-php
Home Page: https://roach-php.dev
crawling php web-scraping

roniemartinez / dude
web-scraping,dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
User: roniemartinez
Home Page: https://roniemartinez.github.io/dude/
python scraping framework playwright scraper xpath css web-scraping beautifulsoup4 parsel

rushter / selectolax
web-scraping,Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
User: rushter
python html5 css modest-engine web-scraping parser

sangaline / wayback-machine-scraper
web-scraping,A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
User: sangaline
Home Page: http://sangaline.com/post/wayback-machine-scraper/
web-scraping wayback-machine wayback-archiver archive-dot-org python command-line-tool

scrapy / scrapy
web-scraping,Scrapy, a fast high-level web crawling & scraping framework for Python.
Organization: scrapy
Home Page: https://scrapy.org
crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python

serpapi / google-search-results-python
web-scraping,Google Search Results via SERP API pip Python Package
Organization: serpapi
python serp-api bing-image google-crawler google-images scraping serpapi web-scraping

shaikhsajid1111 / social-media-profile-scrapers
web-scraping,Fetch user's data across social media
User: shaikhsajid1111
web-scraping web-scraper request python selenium-python facebook-scraper twitter-scraper pinterest reddit-scraper medium-scraper

snooppr / snoop
web-scraping,Snoop — инструмент разведки на основе открытых данных (OSINT world)
User: snooppr
Home Page: https://github.com/snooppr/snoop/releases
osint termux username-search username-checker pentest web-scraping ctf scanner redteam blueteam

spekulatius / phpscraper
web-scraping,A universal web-util for PHP.
User: spekulatius
Home Page: https://phpscraper.de
php php-spider php-crawler scraping-websites php-scraper scraper scraping web-scraper web-scraping beautifulsoup

tidyverse / rvest
web-scraping,Simple web scraping for R
Organization: tidyverse
Home Page: https://rvest.tidyverse.org
html r web-scraping

vida-nyu / ache
web-scraping,ACHE is a web crawler for domain-specific search.
Organization: vida-nyu
Home Page: http://ache.readthedocs.io
web-crawler focused-crawler domain-specific-search web-scraping web-spider web-search hacktoberfest

vprusso / youtube_tutorials
web-scraping,Collection of scripts corresponding to LucidProgramming YouTube tutorials
User: vprusso
Home Page: http://bit.ly/lucidcode
python-tutorial web-scraping youtube-tutorial lucidprogramming ctci-solutions technical-interview python python3 python3-tutorial

x4nth055 / pythoncode-tutorials
web-scraping,The Python Code Tutorials
User: x4nth055
Home Page: https://www.thepythoncode.com
python python3 scapy ethical-hacking network-programming network-security network-analysis python-tutorials tutorials scapy-tutorials

yifeikong / curl_cffi
web-scraping,Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
User: yifeikong
Home Page: https://curl-cffi.readthedocs.io/
curl http-client curl-impersonate http https ja3 ja3-fingerprint tls-fingerprint fingerprinting web-scraping

yusuzech / r-web-scraping-cheat-sheet
web-scraping,Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
User: yusuzech
cheatsheet httr r rselenium rvest scrape-websites web-scraping webscraping

z0m31en7 / uscrapper
web-scraping,Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.
User: z0m31en7
osint python web-scraping website-scraper information-extraction information-gathering osint-python osint-tool reconnaissance webscraping