This project is a Spring Boot application designed for processing weather data. It reads weather data, converts it to Protocol Buffers (proto) format, publishes it to a Kafka cluster, consumes the Kafka messages, and persists the data into a PostgreSQL database. Additionally, the application provides multiple RESTful endpoints to query weather information, including the most humid cities by day, and the hottest and coldest cities on a daily and monthly basis.
- Data Processing Pipeline: Converts weather data to proto and sends it to a Kafka cluster.
- Database Integration: Reads from Kafka and writes to PostgreSQL.
- RESTful Endpoints: Offers several endpoints to query weather data.
- Grafana Dashboard: Visualizes weather data through a comprehensive dashboard.
- Java 17 or newer
- Maven
- Docker (for Kafka and PostgreSQL containers)
- Grafana
- Clone the repository:
git clone https://github.com/mhhio/mci-bigdata-challenge
- Navigate to the project directory:
cd mci-bigdata-challenge
- Build the docker image:
docker build -t weather-app .
- Start application and other services the Kafka and PostgreSQL services (using Docker):
docker-compose up -d
if you want to run application without docker image please consider postgres url and kafka url in application.properties
To publish data from a source JSON file to Kafka, use the following endpoint:
curl -F "file=@[path/to/json/file]" http://localhost:8080/pipeline/publish
consider file size should be lower than 1GB
- Most Humid Cities Daily:
curl http://localhost:8080/api/weather/top-humid-cities
- Hottest Cities Daily:
curl http://localhost:8080/api/weather/hottest-cities/daily
- Hottest Cities Monthly:
curl http://localhost:8080/api/weather/hottest-cities/monthly
- Coldest Cities Daily:
curl http://localhost:8080/api/weather/coldest-cities/daily
- Coldest Cities Monthly:
curl http://localhost:8080/api/weather/coldest-cities/monthly
To access the Swagger UI for API documentation and testing, navigate to http://localhost:8080/swagger-ui.html in your web browser.
To access Grafana dashboard open Grafana in your browser navigate to http://localhost:3000, login with admin as username and password and in Dashboards menu select Weather Events