Table of Contents
Nowadays, hand exoskeletons are commonly used for rehabilitation therapy following health accidents such as strokes. Its efficacy in muscle memory training has the potential to be exploited in the fields of music, technology, and rehabilitation therapy.
Our project aims to facilitate muscle memory training of various piano techniques for pianists seeking to strengthen specific muscles and refine pianistic skills with greater efficiency. By using a lightweight hand exoskeleton, we can enforce specific positions at a given rhythm that allows the user to acquire a specific technique more quickly compared to traditional training.
We want the user to be able to input a technique and a specific tempo that they wish to train using an accompanying software tied to the exoskeleton. This then allows the exoskeleton to reproduce the movements of the technique.
At the end of the semester, we have an exoskeleton of a hand that functions as follows:
See the demo of our project here: Demo
Some photos for reference:
It is controlled via a web application, which includes a feature for playing songs with preloaded files. Additionally, the device can record played pieces, aiding the learning process by allowing users to repeat the segments they prefer. Each finger is operated by a separate motor, enabling simultaneous movement of all fingers. This design grants the device flexibility, allowing it to play almost any song desired.
Comprehensive list of elements we used in our project along with needed equipment. To construct the device, one should
-
buy:
- Arduino Uno Board
- Proto Shield ARDUINO V.R3
- Power supply GST60A12-P1J
- Board LM2596
- DC Power Jack PJ-102B
- 4 Bowden cables
- 4 Servo motors DMS15-270
- MIDI keyboard
- Velcro tape
- Sewing kit
- Threads
- Screws
- Small elastic bands
-
have access to:
- 3D printer with TPU filament
- 3D printer with PET filament
- Laser cutting machine
- Plastic board 3mm
- MDF board 3mm
- Driller
- Bunch of different screwdrivers
- Soldering kit
The comprehensive list of all the elements to start:
- 4 adjustable finger wraps
- 4 finger Bowden end place holders
- 4 Bowden cable guide fingers
- 1 Bowden cables guide hand
- 1 Bowden cables guide wrist
- 1 forearm top holder
- 1 motor mounting plate
- 1 forearm top holder with cables passthrough
- 1 Arduino and power supply bottom plate
- 4 forearm attachments
- 4 Bowden cable motor attachments
- 4 Bowden inside board holders
- 2 logo sides
To construct the device, one should first print all the necessary elements using 3D printers:
Then all the necessary elements need to be laser cut:
- On plastic 3mm board: DXF files
Steps to put the device together:
-
Place threads into holes of Bowden cable guides (hand and wrist)
-
Glue Velcro straps to:
- a. Bowden cable guides hand
- b. Bowden cables guides wrist
- c. 4 forearm attachments
- To each of adjustable finger wraps, one should glue:
- a. Finger Bowden end placeholders
- b. Bowden cable guide fingers
-
Screw motors to the motors mounting plate.
-
Screw logo sides to the motors mounting plate.
-
Screw 2 forearm attachments to the forearm top holder which has the cables passthrough.
-
Screw 2 forearm attachments to Arduino and power supply bottom plate
- Solder all the necessary cables according to the scheme Figure 2. For more details about soldering, go to section Electronics.
-
Screw the Arduino board with Proto Shield and all soldered components to Arduino and power supply bottom plate.
-
Pass Bowden cables through:
- a. Bowden cable guides hand
- b. Bowden cables guides wrist
- c. Forearm top holder
-
Adjust the length of the Bowden cables in a way that they fit the placement of the motors.
-
Secure the Bowden cables by placing screws in threads of:
- a. Bowden cable guides hand
- b. Bowden cables guides wrist
-
Solder a bulb at the end of each of the Bowden cables.
- Enclose the soldered bulb in the Bowden cable motor attachments using a screw.
-
Glue all of Bowden inside board holders in correct places on the forearm top holder
-
Connect Bowden cable motor attachments with Bowden inside board holders using elastic bands. Use the reef knot.
-
Attach Bowden cable motor attachments to the motors, so that they point towards the back (it enables half of the circle rotation to the front, and this push a finger down).
-
Screw the motors mounting plate to the forearm top holder.
- Sew Velcro straps to
- a. Bowden cable guides hand
- b. Bowden cables guides wrist
- c. 2 Bowden forearm attachments (on one of the sides of the Forearm top holder)
- Pass the motors’ cables through prepared space on
- a. Forearm top holder
- b. Arduino and power supply bottom plate
-
Attach the round end of the Bowden cable to the Bowden end place holders and force the cable into the finger guides.
-
Attach the logo at the top of the logo sides.
For the electronics, we needed to power 4 6V servomotors so we couldn't use the Arduino 5V power supply. To solve this problem, we used a 12V power supply with a LM2596 voltage regulator to transform the 12V input into 6V. We then connected to 6V input to all the motors voltage input and did the same for the ground. The servomotors need a PWD arduino pin for control so we settle for the pin 3 5 9 and 11 because they don't touch each other so it would be simpler to solder everything. We also used an external arduino connection board to make everything look nicer.
You can see the schematics for the circuit here, the input is missing from the schematics but it goes on the left handside of the potentiometer :
Figure 1: Schematic of the electronic sketch.
Here is the same schematics but the added external board :
Figure 2: Schematic of the electronic sketch.
Step by step guide to connect everything (using a breadboard):
-
solder the potentiometer LM2596 to the connector for the power supply and the other end to cables
-
connect the potentiometer out+ in one line in the breadboard, connect the out- to another line (must be different lines)
-
connect the ground of the arduino to the same line as the out-
-
Connect each motor to the breadboard as follows : red wire goes to out+, brown wire goes to out- and yellow wire is used for later
-
connect the yellow wire of each motor to the arduino on the pin 3 5 9 and 11, each pin correspond to a finger in increasing order (see schematics)
-
You're done !
We are building a Next.js App.
We need to have two terminals to run the frontend (client end) and the backend (server end).
The frontend part is the client of the website, written in the TypeScript programming language.
To run the frontend, please make sure you are in the frontend folder, please run the following command if not:
cd .\frontend\
then
npm install
If you have encountered the problem of concurrently "npm run dev:next" "npm run dev:ws", please run:
npm install concurrently --save-dev
npm run dev
You will reach the welcome page
Now, we can record a piece after clicking on the "Record" blue button to record what we play, then stop recording by clicking on the "Recording ..." grey button, and then save the file. We can also play songs, which are MIDI files, by clicking on "Play MIDI File" button after selecting a file with the "Select a file" button.
The backend is made of a Flask server. It exposes the following functionalities to the WebApp: play a recording on the glove, start to record keypresses from the midi device and list existing recordings. The server listens for incoming HTTP requests. If the request corresponds to one of the two actions: play and record, the server schedules its execution by registering a new task intent in the job queue. A consumer thread periodically pulls the queue for new jobs. Below is the system architecture of the backend.
You will need python3 and pip3 to be installed on your system to run the backend.
The first step is to create a python virtual environment within ./backend:
cd backend
python3 -m venv venv
Enable the virtual environment (Windows):
.\venv\Scripts\activate.bat
Enable the virtual environment (Mac and Linux):
source ./venv/bin/activate
Install the project dependencies:
pip3 install -r requirements.txt
Make sure the Python virtual environment is enabled.
Then you can run the backend server with:
python3 ./src/main.py
For the MIDI Interpreter, we used a python library called Mido to read and play the midifile live. The MIDI Interpreter class has 6 main functions :
-
init : The starting point of the class, it opens the serial port and setup everything that has been given to it
-
write_read(x) : This is the function responsible for communicating with the arduino, both writing to the Serial port and reading from it.
-
set_filename(x)__ : This function is used to switch midifile dynamically without restarting everything
-
preprocess() : This function looks through the midifile and make a list of playable note based on the amount of finger we want to play with, it takes the first note played as a base and uses a bound to reject note that a too far away from it (we can't play with one hand across 2 octaves). Once the file is over or if we have enough note to play, it returns.
-
interpret() : This function is the actual function responsible for playing the midifile, it takes the midifile currently opened in the class and the list of note to play and sends the appropriate message to the arduino to rotate the motor in the right direction.
-
play() : This function executes all sub-functions required to play a midifile.
One of the main challenges in constructing the exoskeleton is determining the correct cable length. We experienced numerous iterations of cutting, soldering, and covering cables before achieving the proper lengths. Incorrect cable lengths prevent effective force transfer from the motors throughout the design, resulting in inadequate finger movement. It's crucial to adjust the cable length to fit the smallest potential user, as it can still be accommodated by larger users with closer placement on the hand.
Another significant issue is that the force is generated by pushing the cables rather than pulling them from the bottom. This makes the cables more susceptible to bending, which posed a considerable problem in our project. Although it may seem repetitive, the efforts to mitigate this issue are essential. The device is secured to the board with elastics that generate force towards the cable covers, and these measures are not redundant. The cables require multiple rigid components to prevent bending and ensure proper finger movement.
Possible improvements for the hardware should include adding control for the thumb. This addition is more complex and requires a different approach compared to the finger attachments.
To incorporate this feature, adjustments are needed in the initial design. Specifically, the wrist and palm stabilizing elements must be modified to accommodate a fifth cable, as there is currently no space for it. However, the motor fixation plate and electronics are already designed to support five motors, so no modifications are needed for these components.
The front-end web developer has developed the following screens:
Dashboard page:
My_Files page:
In the future, we will develop the Computer Vision part. So far, we have the fundamental part of Computer Vision which can track the hand. There is also a piano visualizer that will be combined with the MIDI interpreter and the backend.
There is the My_Files page where the front-end and the back-end developers will combine the two ends. The current version of the My_Files page can see MIDI file information in detail such as the file name, the last used time, and the length. We can select and deselect MIDI files as well. In addition, we can search a file and sort the files. When the client (frontend) and the server (backend) communicate between themselves with websockets, we will be able to upload files from our computer, play and delete the existing MIDI files on the My_Files page.
Distributed under the MIT License. See LICENSE.txt for more information.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent