Cloud-on-Cloud

Team Member:

Student 1: [Liang Wenjie], [[email protected]], [[email protected]], [https://github.com/lonilow].
Student 2: [Jiang Guanyu], [[email protected]], [[email protected]], [https://github.com/DjangoJungle].
Student 3: [Shen Yipeng], [[email protected]], [[email protected]], [https://github.com/lazyyy77].
Student 4: [Xu Ruikang], [[email protected]], [[email protected]], [https://github.com/xrkkkt].

1. Project Overview

The project of NUS SOC 2024, SWS3004-Cloud Computing with Big Data. The members of our group - group2 - Waffle Rush are: Liang Wenjie, Xu Ruikang, Jiang Guanyu, Shen Yipeng.

1.1 Project Description

This project is for weather informing and forecasting, to create a community and achieve collaborating, preparation-making, resources-sharing and life-saving.

It showcasing the current weather conditions, providing the outdoor tips, predicting the future weather, as well as supporting an on-line weather-chatting-room for information sharing.

1.2 System Architecture

1.3 Key Components

Component 1: Frontend-Part: using vue to design the router/pages, using nginx as web-server and handling the reverse proxy.
Component 2: Backend-Part with Django: response to the data request from the front-end's weather page by selecting and handling information from database.
Component 3: Backend-Part with Java: response to the data request from the front-end's chatbox by selecting\inserting\updating information from database.
Component 4: Backend-Part with Python&ML: calling the api to get the latest weather information while inserting them into the database. Using LSTM/spark to predict the future weather with timestamp.
Component 5: Backend-Part with Kafka: still developing. Creating and dealing with the stream data by compress a long period of time to few seconds.

2. Environment Setup

2.1 Prerequisites

node >= 20.11.0
- https://github.com/coreybutler/nvm-windows/releases/download/1.1.12/nvm-setup.exe
npm >= 10.4.0
- nvm install 20.11.0
Apache Maven >= 3.9.6 https://maven.apache.org/download.cgi
OpenJdk >= 22 https://jdk.java.net/22/
Kafka >= 29.3.10 helm install my-kafka bitnami/kafka --version 29.3.10
Django >= 4.2.3 py -m pip install Django==4.2.3
django-cors-headers pip install django-cors-headers
django-environ >= 0.10.0 pip install "django-environ>=0.10.0"
gunicorn >= 20.0.4 pip install "gunicorn>=20.0.4"

2.2 Installation

The following steps can help you to run the different parts of the project in developing mode.

2.2.1 Step 1: Build the Backend-Weather

Notice: You need to change the database configuration belows to your own. To know more, check the readme in /src/Backend-Weather/readme.md.

weatherApp_backend/weatherApp_backend/settings.py
kube-yaml/mysql-deployment.yaml
kube-yaml/weatherapp-backend-deployment.yaml

python manage.py makemigrations
python manage.py migrate
python manage.py runserver ${your ip address}:${your port}

2.2.2 Step 2: Build the Backend-Information

There will be a more detailed guide in /src/Backend-Information/readme.md

python get_weather.py
python get_warning.py
python get_predict.py

2.2.3 Step 3: Build the Backend-Chatroom

There will be a more detailed guide in /src/Backend-Chatroom/readme.md

mvn clean compile
mvn exec:Java
mvn clean package

2.2.4 Step 4 (Optional): Build the Backend-Kafka

There will be a more detailed guide in /src/Backend-Kafka/readme.md

cd kafka-29.3.8/kafka
helm install -n kafka kafka .

2.2.5 Step 5: Build the Frontend-Release

There will be a more detailed guide in /src/Frontend-Release/readme.md

npm install
npm run dev
npm run build

3. Application Deployment

The following steps can help you to deploy the different parts of the project on k8s with production mode. It directly uses the images create by the dockerfile of each parts.

Please run the commands under the deployment folder:

3.0 AWS

connect to your aws account with AWS CLI.
create an EKS cluster with 2 node groups, whose labels nodegroup 's value equal to FrontendGroup / BackendGroup.

3.1 Service

fsvc.yaml -- LoadBalancer -- expose front-end kubectl apply -f ./1svc/fsvc.yaml
bwsvc.yaml -- NodePort -- expose back-end-weather kubectl apply -f ./1svc/bwsvc.yaml
bcsvc.yaml -- ClusterIP -- expose back-end-chatbox kubectl apply -f ./1svc/bcsvc.yaml
msvc.yaml -- ClusterIP -- expose mysql kubectl apply -f ./1svc/msvc.yaml

3.2 Database

mpv.yaml -- persistent volume kubectl apply -f ./2db/mpv.yaml
mpvc.yaml -- persistent volume claim kubectl apply -f ./2db/mpvc.yaml
mconfig.yaml -- ConfigMap kubectl apply -f ./2db/mconfig.yaml
msecret.yaml -- Secret -- password kubectl apply -f ./2db/msecret.yaml
mysql.yaml -- database kubectl apply -f ./2db/mysql.yaml
exec m-dep pod
- kubectl exec -it ${your sql pod name} -- sh
- execute mysql -u root -p
- pwd = 12345678
- execute create database weather;

3.3 Backend & Database Initialize

bwdep.yaml -- deployment -- backend-weather
- kubectl apply -f ./3be/bwdep.yaml
- kubectl autoscale deployment bw-dep --cpu-percent=50 --min=2 --max=5
- exec bw-dep pods
  - kubectl exec -it ${your sql pod name} -- sh
  - execute python manage.py makemigrations
  - execute python manage.py migrate
bcdep.yaml -- deployment -- backend-chatbox
- kubectl apply -f ./3be/bcdep.yaml
- kubectl autoscale deployment bc-dep --cpu-percent=50 --min=1 --max=5

3.4 Jobs

warning_cj.yaml -- job -- provide warning
- kubectl apply -f ./4cj/warning_cj.yaml
weather_cj.yaml -- job -- provide weather
- kubectl apply -f ./4cj/weather_cj.yaml
predict_cj.yaml -- job -- provide predict
- kubectl apply -f ./4cj/predict_cj.yaml

3.5 Frontend

fdep.yaml -- deployment -- front-end
- kubectl apply -f ./5fe/fdep.yaml
- kubectl autoscale deployment f-dep --cpu-percent=50 --min=2 --max=10

lazyyy77 / cloud-on-cloud Goto Github PK

cloud-on-cloud's Introduction