frankdvd / curb-monitor

Curb monitoring system. A prototype.

curb-monitor

Curb monitoring system. A prototype.

1. Setup Envionment

1. Create a new conda environment and activate it

   conda create --name curb
   conda activate curb
   conda install pip
   

2. Install requirement packages

   pip install -r requirements.txt
   

   You might want to install the cuda version of pytorch again using the cmd here if you want to use gpu for pytorch

2. Object Detection Overall Flow

1. Copy image data from s3 to savio

   1. login to savio data transfer node

      ssh [email protected]
      

   2. config rclone

      module load rclone
      rclone config
      

   3. download s3 data to savio using rclone

      rclone copy your_s3_config_name:curbside-data/vid_folder_123 /global/scratch/users/your_savio_username/vid_folder_123
      

      for example

      rclone copy -P s3_frank:curbside-data/all_videos_1 /global/scratch/users/sidali/all_videos_1
      

   4. delete the corrupted video(remove video smaller than 140M)

      cd /global/scratch/users/sidali/all_videos_1
      find . -name "*.mp4" -type 'f' -size -139M -delete
      

   5. Split data into subfolders

      dir_size=1500
      dir_name="video"
      n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1))
      for i in `seq 1 $n`; do mkdir -p "$dir_name$i"; find . -type f -maxdepth 1 | head -n $dir_size | xargs -i mv "{}" "$dir_name$i"; done
      

   6. switch to savio login node

      ssh [email protected]
      

   7. update the data folder path under curb-monitor/savio_jobs/video_detect/production_job_gpu.sh

      module load nano
      nano savio_jobs/video_detect/production_job_gpu.sh
      

   8. go to curb-monitor/scripts/executables and clone the exiftool tool:

      cd scripts/executables
      git clone https://github.com/exiftool/exiftool.git
      

   9. run the sbatch job

      sbatch ./savio_jobs/video_detect/production_job_gpu.sh
      

   10. check your job

       sq
       

   11. cancel your job

       scancel <jobid>