Trump daily is a crawler that crawls the top 25 news contents that contains the top-featured man on the planet, US President Donald Trump. Currently, the crawler crawls from
- CNN
The crawler is built using python 3.6. So you would need to set up Python 3.6. The crawlers can be run as scheduled task (cron job) or can be run manually.
- Create a mysql/mariadb database named 'trumpdaily' in your localhost/hosted server
- Upload the SQL file 'trumpdaily.sql' to create the tables
- Install the required python3.6 libraries
- Run mainCrawler.py to start the crawler
- Connect your website or web application to query the feed from the database and display on the site
The python libraries and the additional tools that are required to set this project are:
| Libraries/Tools | Purpose | Reference |
|---|---|---|
| MySQL | To save the collected feeds | https://www.mysql.com/ |
| Github | Git repository | plugins/github/README.md |
| Selenium 3.141.0 | To render dynamic web pages of CNN | https://pypi.org/project/selenium/ |
| Beautifulsoup4 4.6.3 | To build effective crawler to parse through contents | https://pypi.org/project/beautifulsoup4/ |
| Configparser 3.5.0 | To parse configuration files | https://pypi.org/project/configparser/ |
| time 1.0.0 | Default python library to add delays into twitter scrapping | https://pypi.org/project/time/ |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent