MagicWand-Gesture-recognition

Real working magic wand

Using an Arduino and a gyroscope/accelerometer as input for gesture recognition, commands can be executed in an almost magical way. It is encased in a 3D-printed housing that looks like Harry Potter's wand.

Functionality:

The accelerometer data is used in the Dynamic Time Warp algorithm to calculate the similarity compared to a previously recorded pattern. If the pattern matches (=below a certain threshold), an action is executed. In my case I wanted to control my LED strips, so I send an IR signal when the gesture is detected.
The serial output is not needed for functionality, but for debugging. It is set up to work with Putty/VSCode and has some nice colors to make debugging easier. (https://xdevs.com/guide/color_serial/)

accelerometer data of one axis; green = sample patter, red = wrong pattern, other = similar to sample pattern

How is a gesture recognized?
As a preparation, the acceleration data from a certain period are stored in an array, the so-called sample pattern. In a second array the current motion data are stored. Now a similarity is to be calculated from these two data. This works by means of Dynamic Time Warping (DTW). DTW tries to create a best possible match of the two time dependent sequences by means of matching, deletion and insertion of data points. For this there is a so-called cost matrix. This contains calculated values, which are then always connected with the smallest neighbor and thus ideally form a diagonal. The area between the calculated path and the diagonal can be used to determine the similarity. Under a certain threshold it is accepted as a gesture

cost matrix visualization; data from pic above
The green diagonal would be if the gesture would match at every point
    -> area = 0 => similarity = 100%.
The blue path is a gesture that is very similar to the sample pattern, which you can see because it is very close to the diagonal
    -> area ≈ 950 -> similarity = ~95%
The red path is the inverse of the sample pattern, which is also clearly visible because it deviates strongly from the diagonal
    -> area ≈ 8500 -> similarity = ~20%
In purple is the area that is calculated to get the similarity.

Components:

ESP32 (DevKitC V4)
(ATTiny85/Arduino Pro Micro)
MPU6050 gyroscope
IR diode
cables

connection sketch

Roadmap/ToDos:

Recognize simple gesture
Recognize multiple gestures
Local constraints in the DTW cost matrix to aboard the calculation if outside of range
Auto-recognition of gestures
Add 3D files
ArduinoProMicro/ATTiny85 compatible
Substract gyro out of acc data!!!!
Serial progress bar for VSCode/Putty (when reccording or calcualting)

Plans:

Prototommy (Prototype 2nd Gen)

3D case plan

Miscellaneous:

Actually it was planned not to use an ESP32 but originally an ATTiny85. However, this is for the development and debugging rather less suitable. The advantage would be that this is much smaller and can be operated longer with a battery. As a compromise I still have an Arduino Pro Micro lying around, maybe the time will find that I can use this, but the memory must be adapted, since this is much smaller than that of the ESP32.
Another idea that just came up, instead of having the Arduino calculate everything, maybe just use it as input and send the data to a more powerful controller like a Raspberry Pi. The Pi then calculates everything and may even be able to do more advanced things due to its higher processing power. The results of the calculation are then converted into an action and sent back to the Arduino, which then performs the given instruction.

flooxo / magicwand-gesture-recognition Goto Github PK