Coder Social home page Coder Social logo

petarhristov94 / gpu_crawler Goto Github PK

View Code? Open in Web Editor NEW
1.0 1.0 1.0 755 KB

Extract prices and links of GPUs on German websites

License: Apache License 2.0

Python 100.00%
gpu bot crawler scalp online-retailers selenium beautifulsoup4 requests url python

gpu_crawler's Introduction

GPU_Crawler

When you can't beat them - join them.


This is a python bot that I wrote to monitor the availability of new GPUs in German online retailers during the 2020 pandemic. Since I had troubles with my previous GPU and I had to return it, I was in need of a new one. The whole scalper situation really pissed me off, because pleople that really needed new cards weren't able to get their hands on one, including me. Since I do not need this software anymore, I'm now releasing it to the public.

The bot works as follows: In a given URL, the script makes a request and extracts the website's HTML code. Should that fail (e.g. because of bot protection), it uses Selenium to open a new browser tab and access the webpage (basically doing what a human would) and then extracts its contents with the help of the BeautifulSoup library. Then it searches a user-defined class inside the HTML Code to extract the Name, Price, Availability, Buy Link and Chip Name of each GPU on sale in the current URL. Finally it exports an up to date Excel table to your desktop that is sorted by availability and price. You can also set price thresholds, so that once an affordable GPU is available, Selenium opens the Buy Link and notifies you by playing a sound. (Do note, that you'll require a driver for the webbrowser you're using [e.g. Chrome -> chromedriver.exe]. Google it.)

This is an open source software and is completely free for all uses. Some minor tweaks and changes are required to make it run for your custom case, so that some basic Python programming skills will be necessary. At best you can use this as a reference for your own projects.

When you use this piece of software there is no need to credit me or anything. Please just use this responsibly and only for your own needs. Be contious of others.

Copyright

by Petar Hristov, licensed under Apache License 2.0.

gpu_crawler's People

Stargazers

 avatar

Watchers

 avatar

Forkers

monoxacc

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.