cynicalanlz / ais__parsing_microservices

Asyncronous microservices used for web urls availability monitoring : python3.5, rabbitmq, asyncio, asyncpg, uvloop

Monitoring microservices*

*docker compose solution lacks synchronization scripts for containers startup ordering, app could be run as follows

docker-compose up -d rabbit
docker-compose up -d pg
docker-compose up -d redis
python backend/web_service.py
python backend/message_db.py
python backend/crawler_service.py

backend/web_service.py - web service for task management + REST api + monitoring data display
backend/crawler_service.py - rabbitmq based service, yields tasks from db via rabbitmq and from rabbitmq queu
backend/message_logger.py - logs messages from ampq to file
backend/message_db.py - makes SQL request to the db based on rabbitmq messages and returns results

Init db

see db_create.sql

Build dockerfile

docker build -f config/Dockerfile -t  ais_rabbitmq

Build ampq docker

docker run -d --log-driver=syslog -e RABBITMQ_NODENAME=my-rabbit --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

Launch backend

cd backend
pip-compile reqs.in && pip install -r reqs.in