hansonrobotics / tinycog

Small Robot, Toy Robot platform

TinyCog

A collection of speech, vision, and movement functionalities aimed at small or toy robots on embedded systems, such as the Raspberry Pi computer. High level reasoning, language understanding, language gneration and movement planning is provided by OpenCog.

The current hardware platform requires an RPI3 computer, a Pi Camera V2 and a USB Microphone; other sensor/detector components are planned.

The current software functions include face detection, emotion recognition, gesture analysis, speech-to-text and text-to-speech subsystems. All high-level founction is provided by OpenCog, and specifically by the Ghost scripting system -- ghost is able to process sensory input, and provide coordinated chatbot and movement abilities.

Setup

Everything here is meant to run on a rpi3 computer; one can also compile everything on a standard Linux desktop computer.

A fully prepped raspbian image is available here Use xzcat to clone the image as shown here replacing sdx with your device.

    xzcat oc-raspbian-1.img.xz | sudo dd of=/dev/sdx

When you first boot with this image, it automatically logs in to the default user 'oc' and expands the filesystem to occupy the entire root partition and then it reboots by itself.

The default credentials:

    Username: oc
    Password: opencog

The image contains the opencog version at the time of it's building and other libraries such as opencv and dlib (3.2 and 19.7). To see the opencog commit version, pkg-config can be used.

    pkg-config --variable=cogutil opencog    #shows cogutil commit version
    pkg-config --variable=atomspace opencog    #shows opencog commit version
    pkg-config --variable=opencog opencog    #shows opencog commit version

Install

Need to have these whether on desktop or rpi

• Opencv-3.2
• Opencv_contrib-3.2
• Dlib
• Festival
• Guile 2.2
• PocketSphinx On rpi3 only
• Raspicam > v1.5
• WiringPi

Use cmake for building.

    cd to TinyCog dir
    mkdir build
    cd build
    cmake ..  # default build is Debug mode. Set CMAKE_BUILD_TYPE to Release to disapble debug mode. 
    make

Testing

• To test the sensors from the guile shell, run the following from within the build dir which opens up the camera and does a live view of the camera with markings for the sensors.

    $ ./TestDrRoboto.scm

• To test from a video file instead of a camera, run the folloiwng way

    $ ./TestDrRoboto.scm -- <video_file_path>

Running

• Start relex server
• run run.sh

    $ ./run.sh

• A guile shell starts up where you can test behavorial functions with simulated functions.

    guile> (test-set-face-event)

• In another terminal, connect to port 5555 via telnet to input speech

    $ telnet localhost 5555

• You can see response output either from the guile instance above or by telneting to port 6666

    $ telnet localhost 6666

Implementation

Sensors

Sensors are a camera and microphones. The camera for face detection -> face landmark -> facial expression and emotion and also for hand detection -> convexity defects -> fingers count and gestures. The microphone for STT and sound source localization. Some of the sensor programs such as the face, hand and voice activity detection can run on the PI without much stress on the hardware but other functionalities like emotion and and speech recogntion should be implemented as services from a server possibly from singnet. Currently, STT is implemented using pocketsphinx. It's not ideal but can be used for a very limited range of commands and simple conversation.

Act

The robot should also act as well as sense. It must speak and move around. The speech synthesis utilizes festival. The code is in act/audio. Movement was intended to be with SPI communication with the hardware but that has changed. However the spi interface is in comm/spi

Behavior

• Code connecting the behavior module to the STT is found in socket/socket.scm
• Behavior rules are in behavior/
  • Ghost rules that list all syntax found in behavior/test.top
  • For non-speech triggered actions, OpenPsi's dynamic module is used to indirectly trigger ghost rules.

        event ==> phony speech                          (Dynamic OpenPsi rule)
        phony speech ==> speech or some action          (Ghost rule)

* Events and event monitoring functions are in [beavior/events.scm](behavior/events.scm)
* Dynamic OpenPsi rules are found in [behavior/interaction-rules.scm](behavior/interaction-rules.scm)
* Functions/actions used by Ghost rules are found at [behavior/functions.scm](behavior/functions.scm)
* Testing functions in [behavior/test.scm](behavior/test.scm)

• Scheme-C++ binding in dr-roboto/dr-roboto.cpp
• Scratch interface in behavior/scratch

ToDo

• Emotion recognition
• Improve STT
• Ghost rules
• Stories for a specific identity we need the robot to have
• Access to Singnet applications