imvictorm / tech-news

Practicing web scraping with python 🕷️

Tech News 📰

Project Context 💡

This project consists in practicing web scraping on the Trybe's blog.

Acquired Knowledge 📖

In this project, I was able to:

• Apply web scraping technics;
• Extract data from HTML content;
• Maintain the extracted data on a NoSQL database.

Main Technologies 🧰

Python	Pytest	Requests	Parsel	Pymongo

Running the application ⚙️

1. Clone the repository and enter it

git clone [email protected]:ImVictorM/Tech-News.git && cd Tech-News

🐋 Running with docker

2. Get the containers running

docker-compose up -d

3. Enter the app container

docker exec -it tech_news bash

4. Install the main package

pip install .

5. Init the app

tech-news-analyzer

🖥️ Running locally

You must have mongodb installed

2. Create the virtual environment

python3 -m venv .venv && source .venv/bin/activate

3. Install the dependencies

python3 -m pip install -r dev-requirements.txt

4. Install the main package

pip install .

5. Init the app

tech-news-analyzer

Testing 🛠️

To run all tests:

python3 -m pytest

Running only one test file:

python3 -m pytest {test_file_path}.py