yjwangtw / iot-system-server

I have built an IoT system to (1) dynamically deploy IoT analytics on gateways to save upload bandwidth as much as possible and (2) dynamically allocate upload bandwidth for each IoT analytic to maximize overall QoS.

IoT-System-Server (Github)

I have built an IoT system to

• dynamically deploy IoT analytics on gateways to save upload bandwidth as much as possible.
• dynamically allocate upload bandwidth for each IoT analytic to maximize overall QoS.

Precisely, my system

• leverages IoT devices, gateways, and cloud servers.
• packages IoT analytics into Docker containers, which are easily deployed and managed.
• manages containers and monitors resources of IoT devices with Kubernetes.

Tools: Python, Docker, Kubernetes, TensorFlow, MQTT, FFserver.

This repository is the server side.

The document is here.

Environment

OS: Ubuntu 18.04.1 LTS Docker: 18.09.7 Kubernetes:

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

File Structure

Implementation/
 |____Experiment/
 |   |____generate_request/
 |   |    |____run.py
 |   |
 |   |____log/
 |   |____parsers/
 |   |____yaml/
 |   |____Algorithm.log
 |   |____cplog.sh
 |   |____delete_all_images.py
 |   |____poisson_run.py
 |   |____qos_measurement_audio.py
 |   |____qos_measurement_yolo_image.py
 |   |____qos_measurement_yolo_result.py
 |   |____registry.sh
 |   |____replace_time.input
 |   |____run.sh
 |   |____SubQoSAudio.sh
 |   |____SubQoSYoloImage.sh
 |   |____SubQoSYoloResult.sh
 |
 |____Gateway/
 |   |____ObjectDector/
 |   |____ObjectDector2/
 |   |____ObjectDector3/
 |   |____ObjectDector4/
 |   |____ObjectDector5/
 |
 |____RunDeploy/
 |   |____analytics.input
 |   |____departure_analytics.txt
 |   |____deploy_measure.py
 |   |____exist.input
 |   |____expect.input
 |   |____images_download.log
 |   |____list.txt
 |   |____replace_time.input
 |   |____terminal.input
 |
 |____RunAllocation/
 |   |____Algo/
 |   |    |____allocate_algo.py
 |   |    |____produce_weights.py
 |   |    |____unweighted_algo.py
 |   |    |____weighted_algo.py
 |   |    |____weights.txt
 |   |
 |   |____QoSknob.txt
 |   |____run.py
 |
 |____docker_daemon.json

Folders and Files

Experiment/: Main experiments

• generate_request/: Input requests and request generations
• log/: Log files
• parsers/: Parsers to parse raw log files
• yaml/: Analytics deployment yaml files
• cplog.sh: Script to copy log files
• poisson_run.py: Main process
• registry.sh: Script to run local registry
• run.sh: Main process

Gateway/: Including Yolo processes

• ObjectDector/~ObjectDector5/: Yolo processes

RunDeploy/: Container deployment process

• deploy_measure.py: Image Download Algorithms:
  • Dynamic programming algorithm ($IDA_D$)
  • $(1-\epsilon)$-approximation algorithm ($IDA_A$)
  • Greedy algorithm ($IDA_G$)

RunAllocation/: Rate allocation process

• run.py: Rate allocation processes
• Algo/: Rate Allocation Algorithms:
  • Our proposed algorithm (RAA)
  • Weighted allocation algorithm (WA)
  • Unweighted allocation algorithm (UA)
• QoSknob.txt: QoS knobs determined by rate allocation algorithms

docker_daemon.json: Local registry setup

Set up

Build Local Registry

Copy Implementation/docker_daemon.json to /etc/docker/daemon.json

cd Implementation
cp docker_daemon.json /etc/docker/daemon.json

Start up

Environment Variables

Set up environment variables in config.env, and then source config.env.

Run Local Registry

• Go to Experiment/

cd Implementation/Experiment/

• Run local registry container

bash registry.sh

Generate Requests

• Go to generate_request/

cd generate_request/

• Run the generating script run.py

python3 run.py [number_of_requests_per_minute] [minimal_departure_time] [maximal_departure_time] [total_experiment_time] [output_file]

output_file is requests_[number_of_requests_per_minute]_[minimal_departure_time]_[maximal_departure_time]_[total_experiment_time]_[version].txt

Example.

python3 run.py 1 1 10 20 requests_1_1_10_20_1.txt

Set up Parameters of Experiments in run.sh

Parameters	Meanings
ida	Image download algorithm (IDA) Ex. DP: $IDA_D$ FPTAS: $IDA_A$ Greedy: $IDA_G$
raa	Rate allocation algorithm (RAA) Ex. RAA: RAA weighted: WA unweighted: UA
lrp	Layer replacement policy (LRP) Ex. LRU, MRU, LFU, MFU
epsilon	Approximation parameter $\epsilon$ of $IDA_A$ Ex. 0.4
alpha	Step size $\alpha$ of RAA Ex. 0.01
total_time	Total experiment time (unit is minute). Ex. 40: for a 40-minute long experiment.
duration	Time slot duration of IDA (unit is minute) Ex. 5
network	Download / upload bandwidth. Ex. optical: 100Mbps / 5Mbps up10: 100Mbps / 10Mbps
request_per_m	Number of requests per minute Ex. 1
alive_time_min	Minimal departure time (unit is minute) Ex. 1
alive_time_max	Maximal departure time (unit is minute) Ex. 10
i	Requests version
date_time	Time stamp

Run Experiments

screen -S run 
bash run.sh > run.log

Check Screen Status

screen -ls

Output should be like this:

There are screens on:
	20519.SubQoSAudio	(08/30/2020 08:51:00 PM)	(Detached)
	20429.SubQoSYoloImage	(08/30/2020 08:50:52 PM)	(Detached)
	20348.SubQoSYoloResult	(08/30/2020 08:50:44 PM)	(Detached)
	19471.run	(08/30/2020 08:49:04 PM)	(Detached)
4 Sockets in /run/screen/S-master.

Parse Log files

Raw Log files

Here is an example

Log files	Information of	Generated by
DP_RAA_LRU_0.4_0.01_1.log	Main process	Experiment/poisson_run.py
deploy_optical_DP_RAA_LRU_0.4_0.01_1.log	IDA	RunDeploy/deploy_measure.py or RunDeploy/greedy_measure.py
allocation_optical_DP_RAA_LRU_0.4_0.01_1.log	RAA	RunAllocation/run.py
Bandwidth_optical_DP_RAA_LRU_0.4_0.01_1.log	Experiment/Bandwidth consumption	Experiment/poisson_run.py
CPU_optical_DP_RAA_LRU_0.4_0.01_1.log	CPU consumption	Experiment/poisson_run.py
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_audio.log	Result of audio classifiers	Experiment/qos_measurement_audio.py
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_yolo.log	Result of object detectors	Experiment/qos_measurement_yolo_result.py

Copy log files to log/

• Edit Parameters in cplog.sh
  • network: network condition. Ex. optical
  • i: version of requests Ex. 1
• Run cplog.sh

bash cplog.sh [algorithm] [target_dir]

• Arguments:
  • algorithm: the name of main process's log file. Ex. DP_RAA_LRU_0.4_0.01_1
  • target_dir: output directory. Ex. log/DP_RAA_MFU_0.4_0.01_1_1_10_20_3GB_optical_Aug_31/
    Notice that ./ will remove log files
• Example.

bash cplog.sh DP_RAA_LRU_0.4_0.01_1 log/DP_RAA_LRU_0.4_0.01_3_1_10_30_3GB_optical_Aug_7/

• Check log directory

ls log/DP_RAA_LRU_0.4_0.01_3_1_10_30_3GB_optical_Aug_7

Output:

...
run.log
allocation_optical_DP_RAA_LRU_0.4_0.01_1.log
Bandwidth_optical_DP_RAA_LRU_0.4_0.01_1.log
CPU_optical_DP_RAA_LRU_0.4_0.01_1.log
deploy_optical_DP_RAA_LRU_0.4_0.01_1.log
DP_RAA_LRU_0.4_0.01_1.log
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_audio.log
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_yolo.log
...

Using Yolo to Calculate ground truth and Parse Log Files

cd log/DP_RAA_LRU_0.4_0.01_3_1_10_30_3GB_optical_Aug_7
bash ../../parsers/parse.sh [log_dir] [name_of_main_process_log_file] [date_time] [yolo_dir] [request_version]

• yolo_dir: the directory of yolo.
  Ex. ../../../Gateway/ObjectDector

Example.

bash ../../parsers/parse.sh ./ DP_RAA_LRU_0.4_0.01_1 0830_2049 ../../../Gateway/ObjectDector 1

Parsed Log Files

Log Files	Information of
accuracy_DP_RAA_LRU_0.4_0.01_1.txt	Overall weighted QoS
deploy_num_DP_RAA_LRU_0.4_0.01_1.txt	Number of analytics containers on the gateway
deploy_num_expect_DP_RAA_LRU_0.4_0.01_1.txt	Expected number of analytics containers on the gateway
down_DP_RAA_LRU_0.4_0.01_1.txt	Download bandwidth consumption
gateway_num_DP_RAA_LRU_0.4_0.01_1.txt	Number of analytics containers on the gateway
request_num_DP_RAA_LRU_0.4_0.01_1.txt	Number of analytics containers on the cloud server and gateway
saved_bw_DP_RAA_LRU_0.4_0.01_1.txt	Saved upload bandwidth
saved_bw_expect_DP_RAA_LRU_0.4_0.01_1.txt	Expected saved upload bandwidth
time_allocate_DP_RAA_LRU_0.4_0.01_1.txt	Running time of rate alloction algorithm
time_ida_DP_RAA_LRU_0.4_0.01_1.txt	Running time of image download algorithm
up_DP_RAA_LRU_0.4_0.01_1.txt	Upload bandwidth consumption