montoyamoraga / scrapers

scrapers for building your own image databases

scrapers

About

scrapers is a collection of free/libre open-source software written by Aarón Montoya-Moraga.

scrapers is both a tool for building databases and and educational resource for learning scraping.

scrapers is educational because it tries to be heavily documented, clean, and easy to follow.

scrapers performs the scraping in an explicit way, it shows you the browser going through the data, instead of running in the background, thus being very open in the way it works, which can be used for both documentation and live performance.

Technical details

All of these scrapers were written using Python, Selenium, and ChromeDriver.

Contents

• bing-images
• captcha
• google-images
• instagram
• mugshots

Installation and prerrequisites

• Install Python2 and Python3
• Install Homebrew if in Mac
• Install Chromedriver

Acknowledgements

• Thanks to Sam Lavigne and their class Detourning the web.
• Thanks to my classmates at School of Machines that helped me test and prototype some of these scrapers.

License

MIT

scrapers's Issues

google images scraper not working

Today I ran scraper-google-images on:

• macOS Sierra 10.12.6
• Python 3.7.2
• Homebrew 2.0.2
• pip 19.0.3

And received the following error message:

Traceback (most recent call last):
  File "script.py", line 227, in <module>
    scrape(scrapeSubject)
  File "script.py", line 114, in scrape
    scrollUpDown(driver)
  File "script.py", line 201, in scrollUpDown
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  File "/Users/guillermo/Documents/Github/scrapers/scraper-google-images/env/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
    'args': converted_args})['value']
  File "/Users/guillermo/Documents/Github/scrapers/scraper-google-images/env/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Users/guillermo/Documents/Github/scrapers/scraper-google-images/env/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
  (Session info: chrome=72.0.3626.109)
  (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.12.6 x86_64)

create new scraper: cleveland museum of art

http://www.clevelandart.org/art/collection/search

MET publications

https://www.metmuseum.org/art/metpublications/

create new scraper: art institute of chicago

https://www.artic.edu/collection

pulp magazine archive

https://archive.org/details/pulpmagazinearchive

create new scraper: new york public library collection

https://digitalcollections.nypl.org/

Bing Scraper not working

Hey aaron + the internet,

Seems like the bing scraper has broken. I'm on a mac and receive the following error when trying to scrape.

query bing images
script.py:126: DeprecationWarning: use driver.switch_to.frame instead
driver.switch_to_frame("OverlayIFrame")
Traceback (most recent call last):
File "script.py", line 222, in
scrape(scrapeSubject)
File "script.py", line 150, in scrape
popup.click()