dennewbie / ai_watch_a1

AI Watch A1: Multi-person 3D skeleton detection using Intel RealSense and OpenPose with Kafka support.

AI Watch A1

AI Watch A1 uses one Intel RealSense D435 camera and OpenPose in order to achieve multi-person 3D skeleton detection. Once this task is completed, AI Watch A1 provides support to send each detected skeleton and its joint points' 3D coordinates (room and D435's details have to be specified) via Kafka, in order to let further detached computation possible.

Keynote 📋

• Keynote
• Graduation Day

Demo 💻

• Demo #1
• Demo #2

Docs 📜

• AI Watch A1 documentation was generated using Doxygen
• Bachelor Thesis in Computer Science: PDF TesiOnline

Installation 🚀

Note: Instructions for MacOS with Intel CPU / M1 chip

1. Install RealSense SDK 2.0 and its own dependencies. The following guides are really helpful: build RealSense for macOS Monterey (Intel + Apple Silicon), macOS installation for Intel RealSense SDK.

2. Install OpenPose and its own dependencies. Useful guides: build openpose with/without GPU support for macOS, OpenPose for M1/Intel. In order to properly choose the GPU_MODE during OpenPose installation and/or in order to install OpenPose on other operating systems, check out this page: OpenPose Docs. If you have MacOS with Intel CPU and a GPU that matches OpenPose prerequisites, then you can set GPU_MODE to OPENCL (GPU-accelerated, it is harder to install but provides a faster runtime speed). Otherwise, if you have a MacOS with M1 chip, it's suggested to set GPU_MODE to CPU_ONLY.

CMake Settings for Intel:

CMake Settings for >= M1:

3. Install Apache Kafka, Confluent and their own dependencies. Confluent is not mandatory, but without it, you will have to set up the Kafka environment on your own. Useful guides: how to install Apache Kafka on Mac, how to install Confluent.

4. Run the following command in your terminal:

   git clone --recursive [email protected]:dennewbie/AI_Watch_A1.git
   

   After that, remove all the files named emptyFileForPadding.txt in the cloned folder.

5. From /AI_Watch_A1/src/AI_Watch_A1/ folder, run the following commands in your terminal:

   mkdir build && cd build
   

6. Copy OpenPose's models folder to the project's build folder.

7. Copy OpenPose's BoostConfig.cmake, FindGFlags.cmake and FindGlog.cmake files to the build/cmake/modules/ folder.

8. Update caffe lib path inside CMakeLists.txt. Then run the following commands in your terminal:

   cmake .. && make -j `sysctl -n hw.logicalcpu`
   

9. This step is optional. Run the following command in your terminal:

sudo make install

10. Now let's start the Kafka environment. Run the following commands in a new terminal session located on the parent folder of the confluent folder. Set the environment variable for the Confluent Platform home directory:

    export CONFLUENT_HOME=confluent-7.2.1
    

    Add the Confluent Platform bin directory to your PATH:

    export PATH=$PATH:$CONFLUENT_HOME/bin
    

    Test your installation by running the confluent command:

    confluent --help
    

    Your output should show the available commands for managing Confluent Platform.

    Start Confluent Platform by using the Confluent CLI confluent local services start command. This command starts all of the Confluent Platform components, including Kafka, ZooKeeper, Schema Registry, HTTP REST Proxy for Kafka, Kafka Connect, ksqlDB, and Control Center.

    confluent local services start
    

    Your output should resemble:

    Starting Zookeeper
    Zookeeper is [UP]
    Starting Kafka
    Kafka is [UP]
    Starting Schema Registry
    Schema Registry is [UP]
    Starting Kafka REST
    Kafka REST is [UP]
    Starting Connect
    Connect is [UP]
    Starting KSQL Server
    KSQL Server is [UP]
    Starting Control Center
    Control Center is [UP]
    

    Important #1 📌

    The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, check Confluent website.

    Important #2 📌

    The Confluent CLI requires Java version 1.8 or 1.11. See Confluent versions interoperability.

    Important #3 📌

    In order to improve speed and general performance, you could disable images' showing and change the number of frames captured per second. In this last option, capturing too few frames per second reduces output' quality. Furthermore, it's also possible to reduce OpenPose's network resolution, but a significant drawback is reduced accuracy.

    Important #4 📌

    Place the camera in a way that it can "see" the left and the right wall of the room.

11. Navigate to http://localhost:9021 and create a new topic topic1 with default settings. Now go to /AI_Watch_A1/src/AI_Watch_A1/ and set up your Kafka parameters within the configuration_file.ini file. At this point:

    • if internal OpenPose execution is chosen, then run the following command in your terminal:

      sudo ./AI_Watch_A1.bin --num_gpu 1 --num_gpu_start 2 --image_dir rs_images/rgb --write_json op_output/op --logging_level 255
      

    • if external OpenPose execution is chosen, then set up your OpenPose parameters within the conf.conf file, run the following command in your terminal:

      sudo ./AI_Watch_A1.bin
      

Note that the internal OpenPose execution is suggested.

Known issues ⚠️

1. The module can rarely get stuck on the following invocation located in RealSenseD435Manager::23:

   rs2::pipeline_profile myPipelineProfile = pipelineStream.start(myConfiguration);
   

   This means some errors have occurred, due to the USB connection while starting the camera's environment. In order to fix that, just exit the program, unplug the RealSense camera from the USB cable, and connect it again.

System Structure 🏛

Tools 🛠

• Intel RealSense SDK
• OpenPose
• OpenCV
• OpenCL
• Kafka
• Kafka C/C++ library
• Confluent
• jsoncpp
• Doxygen
• StarUML
• Sketch

Contributing 👨🏻👳🏾‍♂️👩🏼‍🦱👽

In order to contribute to AI Watch A1, please follow the contribution guidelines.

Main Contributors 💫

• Dr. Paola Barra, CS Professor at the University of Naples Parthenope.

License ☢️

AI Watch A1 is licensed under the Apache License, Version 2.0. Copyright 2022. Please, see the license for further details and the /licenses folder for the used libraries' licenses details.

Contacts 🪪

• LinkedIn

Citation 📖

    @report{AIWatchA1,
        author = {Denny Caruso},
        title = {Multi-person 3D skeleton detection with depth cameras​},
        institution = {University of Naples, Parthenope},
        year = {2022}
    }

Supervisor

• Dr. Alessio Ferone, CS Assistant Professor at the University of Naples Parthenope.

Other modules

• B1: anomaly detection + digital twin
• C1: Unity3D rendering

a.a. 2021/2022