introlab / t-top

T-Top is a tabletop robot designed with advanced audio and vision sensors, deep learning perceptual processing and telecommunication capabilities to provide richer interaction modalities and develop higher cognitive abilities from interacting with people.

T-Top

T-Top is a tabletop robot designed with advanced audio and vision sensors, deep learning perceptual processing and telecommunication capabilities to provide richer interaction modalities and develop higher cognitive abilities from interacting with people.

Authors

• Marc-Antoine Maheux (@mamaheux)
• Charles Caya (@chcaya)
• Alexandre Filion (@alexfilion)
• Dominic Létourneau (@doumdi)
• Philippe Warren (@philippewarren)
• Ian-Mathieu Joly (@joli-1801)

Licensing

• Source code files: GPLv3
• Other files: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License

Features

Category	Type	Description
Power	Power Adapter	19 V
	Battery	1x RRC2054-2
	Battery Charger	1x RRC-PMM240
Sensors	Microphone Array	16x xSoundsMicrophones, 1x 16SoundsUSB
	RGB-D Camera	1x Intel RealSense D435i
	Wide Angle Camera	1x Arducam AR0230
	Touchscreen	1x 7 inch 1024x600 capacitive touchscreen
	Current/Voltage	INA220
	Light Sensors	4x Adafruit ALS-PT19
	Buttons	4x buttons
Actuators	Stewart Platform	Displacement range: ±3 cm (x, y and z), ±20° (x and y), ±30° (z). Motor: Dynamixel XL430-W250
	Rotating Base	Displacement range: illimited. Motor: Dynamixel XL430-W250
	Speakers	4x Dayton Audio DMA45-8, 2x MAX9744
	Cooling	2x Noctua NF-A4x20 5V
	Touchscreen	1x 7 inch 1024x600 capacitive touchscreen
	LED	Battery status, volume level, led strip
Network	WiFi	Intel Dual Band Wireless-AC 8265 NGW
	Ethernet	100 Mbps
Processing	Computer	NVIDIA Jetson AGX Xavier Developer Kit or NVIDIA Jetson AGX Xavier Orin Developer Kit
	Motor MCU	Teensy 4.0
	Battery MCU	Teensy LC
Perceptions		SLAM, object detection, person pose estimation, face recognition, sound classification, speaker identification, robot name detection, speech to text, person identification, music beat detection, source source localization, ego noise reduction, vad
Behaviors		Telepresence, emotions, talking, greeting, face following, dancing, exploring, sound following

Repository Structure

• The documentation folder contains the documentation to build and configure T-Top.
• The CAD folder contains the SolidWorks files of the robot. Il also contains the DXF and STL files to cut and print custom parts.
• The firmwares folder contains the firmware for the MCUs.
• The PCB folder contains the KiCad files of the custom PCBs. Il also contains the Gerber files to manufacture the PCBs.
• The ros folder contains the ROS packages to use the robot.
• The tools folder contains the tools to develop and use the robot.

Development Computer Setup

See 01_COMPUTER_CONFIGURATION.md

Papers

• M.-A. Maheux, D. Létourneau, P. Warren, A. M. Panchea, J. M. Robillard and F. Michaud, “‍Designing a Tabletop SAR as an Advanced HRI Experimentation Platform,” in Proceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction (TAHRI 2024), Boulder, CO, USA, 2024
  • YouTube Video
• A. M. Panchea, M.-A. Maheux, P. Warren, D. Létourneau and F. Michaud, “Exploratory Evaluation of a Tabletop Robot With Older Adults,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Boulder, CO, USA, 2024, pp. 817–821.
• M.-A. Maheux, A. M. Panchea, P. Warren, D. Létourneau and F. Michaud, “T-Top, an Open Source Tabletop Robot with Advanced Onboard Audio, Vision and Deep Learning Capabilities,” in Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 2023, pp. 7862-7869.
• M.-A. Maheux, C. Caya, D. Létourneau and F. Michaud, “T-Top, a sar experimental platform,” in Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction, 2022, pp. 904–908.

Sponsor

IntRoLab - Intelligent / Interactive / Integrated / Interdisciplinary Robot Lab

t-top's Issues

T-Top - DNN improvements

• Use AdamW
• Use knowledge distillation
• Object detection
  • Use pretrained yolov7 and yolov7tiny
  • Use yolov7 and yolov7tiny for object descriptor
• Person pose
  • Use efficient net instead of resnet18
  • Optimize the gaussian parameters
  • Increase the resolution of the output
  • BCE divergence for knowledge distillation
  • Training
• Face recognition
  • Improve the face alignment, maybe just a crop
  • Use efficient net instead of OpenFace
  • Use ArcFace instead AM-Softmax (https://openaccess.thecvf.com/content_CVPR_2019/papers/Deng_ArcFace_Additive_Angular_Margin_Loss_for_Deep_Face_Recognition_CVPR_2019_paper.pdf)
  • https://odr.chalmers.se/server/api/core/bitstreams/0f358a79-f2e1-48f3-b591-a4563489cd8f/content section 3.3 for knowledge distillation
  • Test also on a modern dataset like
  • Training
• Audio classification
  • Use AudioSet
  • Use a smaller frame size
  • Use https://arxiv.org/pdf/2102.01243v3.pdf
  • Training
• Speaker identification
  • Reduce the frame duration to 1 sec
  • Use the full size network
  • Use the pretrained network from speechbrain
  • Training

Fix calculate_iou

Move the HBBA Desires and Strategies to another ROS package

Improve audio analysis

• Add the ODAS tracking in the message
• Reduce the interval between analysis
• ODAS frame Z at the base

ROS2 Migration

This is a thread to start thinking about the ROS2 migration for T-Top.

Ideas at this point:

• Use Docker (could be interesting: https://github.com/dusty-nv/jetson-containers)
• Use the ros1_bridge as a first step

Use batch hard triplet mining for the triplet loss in the face recognition training

https://arxiv.org/abs/1703.07737

Use Torch-TensorRT instead of torch2trt for DNN exports

https://github.com/NVIDIA/Torch-TensorRT

Replace Google Text-to-Speech by Piper

• https://github.com/rhasspy/piper
• microsoft/onnxruntime#9371

Add auto-boot jumper for AGX Xavier in documentation

Bounding Box or Semantic Segmentation for sound_object_person_following

Improve perception logger

• Robot state
  • Add the activation of the strategies
• Video analysis
  • Add the class/box confidence of the object
  • Add the blur measure of a face
  • Add how the face is aligned
• Audio Analysis
  • Add ODAS tracking
• Add odas to head TF

Replace Google Speech-to-Text by Whisper

https://github.com/openai/whisper

Remove the usage of the neural network for the Stewart platform

Improve video analyzer

• Object Detection
  • Add the class/box confidence value to the message
• Face descriptor
  • Add a value measuring the blur of the face to the message (std of laplacian filter output)
  • Add a flag indicating how the face is aligned to the message

T-Top Version 4 - Features

• CAD

  • Fit the NVIDIA Jetson AGX Orin
  • Add a LED strip around the base (WS2812B, 30 LEDs / m)
  • Add a LEDs diffuser
  • Add buttons to control the sound level
  • Replace discrete LEDs by a strip (WS2812B, 144 LEDs / m)
  • Add a LED strip to show the sound level (WS2812B, 144 LEDs / m)
  • Add springs to the servo horns to reduce the power consumption
  • Add light sensor
  • Add tie wrap hole on the stewart bottom

• PCB - PSU Control

  • LEDs around the base connected to PCB Control
  • Replace the ISO7521C by a tranceiver (MAX3221E)
  • Button inputs
  • The MCU closes the power so it can send a signal to the computer (example)
  • K7805-3AR3 for 5V
  • Add a EMC compliance circuit
  • Add a light sensor to adjust the LED brightness (ALS-PT19)

• PCB - Replace the OpenCR with a custom board using a Teensy 4.0

  • Use a Schmitt trigger for the limit switch
  • UART for PSU Control (tranceiver MAX3221E)
  • Button inputs (74HC2G14GV)
  • TTL Dynamixel IC: SN74LVC2G241DCTR
  • TTL Dynamixel Connector: 25SH-C-03-TS
  • IMU (ICM-40627)
  • R-78E5.0-0.5 for 5V
  • OpenCR Schematic

• Firmware - psu_control

  • Use this lib for the LED controls
  • Read the buttons to change the sound level
  • Send the current volume to the OpenCR
  • Read LED animation from the OpenCR
  • Max LED brightness 50% max (adjusted with the light sensors)
  • Max audio power: 12 W (battery) or 30 W (PSU)

• Firmware OpenCR

  • Send the volume to ROS
  • Read the LED animation from ROS
  • Send the buttons to ROS
  • Remove ROS and use a custom serial protocol

• Jetson AGX Orin

  • Create a 40W power model aligned with the software when the robot is powered by the battery (Power Estimator)
  • Create a 55W power model aligned with the software when the robot is powered by the power adapter (Power Estimator)
  • RTC Clock battery
  • Create a systemmd service (https://github.com/vasilakisfil/hello.service/blob/master/debian/hello.service, https://fabianlee.org/2017/05/21/golang-running-a-go-binary-as-a-systemd-service-on-ubuntu-16-04/, https://www.freedesktop.org/software/systemd/man/daemon.html) to:
    • Listen shutdown events
    • Change the power mode depending on the powering device (PSU or battery)
    • Listen and send data to ROS
    • Close all roslaunch processes
    • Adjust screen brightness (light sensor)
    • System Tray Icon
      • State of charge
      • Volume up
      • Volume down
      • Close all LEDs
      • Reset torso
      • Reset head

• ROS

  • Update the stewart platform parameters
  • Update the odas configuration
  • Update TF
  • control_panel
    • read the volume from opencr

• Documentation

  • BOM
    • PSU Control PCB
    • Buck-Boost PCB
    • Dynamixel Control PCB
    • 3D printing
    • Laser/CNC
    • Others
  • Assembly
  • TF
  • RS232/UART protocol
  • Electrical block diagram
  • Stewart displacement (root readme)

• Update main readme images

Ideas

• Use only one MCU

Use hydra-torch in dnn_training for the network configurations

https://github.com/pytorch/hydra-torch

ROS packages for object_detection and semantic_segmentation

Great project!

While there is object_detection and semantic_segmentation training code in https://github.com/introlab/t-top/tree/main/tools/dnn_training, these two functionalities are missing in https://github.com/introlab/t-top/tree/main/ros/perceptions.

Can you share your plans on this?

Firmware - opencr