Project Outline Discussion about capstone2020

[Y1Ck]

1. i will give an outline in this few days; rightnow im trying to consolidate the info/tools/applications/ideas gathered;
2. let's discuss further during this friday meeting;

[nivlekp]

Here is one of my proposals...
More references coming soon...

Real-time translation of Auslan using pose estimation

Motivation

To provide a platform where people using Auslan could participate in conversations with poeple whom cannot use Auslan

Goal

To build an application that would translate sign language into text and voice in real time

Project description and idea of tasks

• Identify a pose/gesture recognition system that would be able to do pose estimation in real-time.
• Figure out a way to recognize different sign gestures (probably using machine learning/neural network)
• Hook the output (gestures recognized) of the above system to a real-time speech generation software.
• If time permits, build an API/GUI for people to map their own gesture.

Reasons to backup

• Not much
• The project goal seems to be meaningful
• By the end of this, we will all be familiar with the basics of machine learning.
• Feel free to approve/disapprove...

[nivlekp]

Here are other couple of examples regarding real-time sign language translation...
https://www.kintrans.com/
and
http://blog.leapmotion.com/giving-deaf-people-a-voice-motionsavvys-real-time-sign-language-translation/
though this one uses a separate sensor.

However the first one seems to be still under development, so not yet widely available to the public yet?
And the second one seems to have died off after 2014, and I couldn't find any further information from them.

Moreover both of these projects (and including the example that you posted) focuses on ASL. So I guess we can focus more on Auslan, though I doubt if the underlying principle of gesture recognition / pose estimation would be very different.

[relientm96]

Some questions/notes that I jotted down during meeting with Jonathan:

• Research based or implementation based? ( I think we have our answer )
• What do we individually want to do after capstone?
• What do we enjoy doing?
• We will be presenting at Endeavor, usually a physical demonstration of project would be more attractive.

Question to follow up from wanting implementation based project:
Are we deciding on creating a system for one application or a robust one that can solve many pose estimation related problems?

[relientm96]

I have a proposal

Real Time Pose Estimation for Embedded Applications

Goal

To come up with a device and application that is small, portable yet powerful enough to do real time pose estimation.

Background Problem

Real Time Pose Estimation can be applicable to many situations but often , these situations are at different locations or are at places that is inconvenient to put a massive cpu/computer at. We want to try to come up with a system or device that ideally can be put into a pocket and be brought around to do real time pose estimation , making it a portable and yet powerful enough to use.

Project Description and idea of tasks

• Choose an implementation of real time pose estimation.
• Design a board or fpga decide to implement chosen real time pose estimation algorithm
• Test it's battery life, power consumption, and accuracy in reading poses in real time
• Make a final board/pcb and an enclosure.

Reasons to backup:

• Jonathan has more knowledge in electronics design so doing by doing an embedded electronics project, he would be able to help us more.
• We still have the research mathematical aspect of this approach as we need to figure out which implementation is best suited, particularly for small embedded systems
• We do not have much time and experience in dealing with AI or machine learning so we would be better off using an implementation online and try to understand what people have one.
• This project combines both analysis and implementation so can be suited for our group's interest?

Let me know what you guys think!

[Y1Ck]

0. embedded machine-vision-system specializing in pose estimation? Somehow, i think your proposal is similar to HDMI-project? correct me if i interpreted wrongly;
1. we certainly can propose this to jonathan and see what's his thought;

my thoughts:
2. this proposal seems more challenging given the limited time we have. I have no confidence.
3. also, as of now, i'm not sure of its value considering that:
- Real-time pose estimation on Android/IOS is already out there (?)
- tablet?
4. i dont think we have much choice but to deal with the algorithm for pose estimation, say machine learning; we still have to justify our choice. so we have to understand them to a certain depth that we could compare and justify it before implementing it;

raw plan (what i have in mind):

1. the first (half?) semester:
   a. to choose the right one and implement it;
   b. improve/build it using the hardware we have, say RGB cameras from jonathan;
   c. iterative feedback; mostly likely it's not gonna work on first hit whether in terms of
   implementation or our choice on the algorithm;

2. winter + second semester + first semester (if we are ahead of time):
   a. we could focus on the interesting application to build on top of it;
   e.g. tsz kiu's idea on using it as a gesture control for music applications?
   jonathan's project on tennis-player-movement-track-and-inference?
   b. i would say that building these applications would take considerably more effort?
   bear in mind that initial choice of the algorithm may not be suitable for the chosen application ...
   c. certainly not limited to software;

3. so i would say 20-30% on research, 70-80% on implementation (applications)

[relientm96]

@Y1Ck

Yeah it is somewhat similar in the sense of creating a board at the end of the day to implement what we have researched on about Pose Estimation.

My proposal stems from that we decided on creating a solution to any possible application instead of choosing a specific application to build a solution using Pose Estimation.

• I thought of using the embedded aspect as way to bring Jonathan into it as we can then ask him about stuff regarding embedded stuff.
• As there are pose estimation stuff done on phones or tablets, they may not necessarily be the most powerful as compared to a specialized board that we can make.
• For development, I personally think that we can focus our implementation using a C/Python program on a board like the RaspberryPi connected to a USB camera. This makes it alot easier for us for the time being as we do not know much about the algorithms and C/Python has support for pose estimation with open source libraries online. If time permits, we can then try to implement it on an FPGA board ( going from high level to low level) .
• Correct me if I'm wrong but from seeing the project timeline that you outlined, do you propose we create the pose estimation solution and tailor it to a specific problem to solve?

[nivlekp]

I like Matthew's @relientm96 proposal.

Jonathan has more knowledge in electronics design so doing by doing an embedded electronics project, he would be able to help us more.

I agree, though anyhow I think we would have to do a lot of work on our own though.

We still have the research mathematical aspect of this approach as we need to figure out which implementation is best suited, particularly for small embedded systems.

This sounds good too, and I am excited for the the challenge with embedded systems.

And on top of the reasons that you've mentioned, your proposal sounds good since the goal is well-defined, and the tasks are well-defined as well (such as choosing the implementation, building the board, testing the battery life, etc.), which means the project would be easier to manage as well.

At the moment I am looking at gesture control in music. I am coming up with a proposal as well, and will post it here either today or tomorrow. My concern at this stage is when dealing with music, the goal could be more open-ended. To work around this problem, I am trying to look at previous works that other people have done, and see if we can do it with pose estimation.

[Y1Ck]

Love it!

[relientm96]

@nivlekp I really like the application that you brought up! Seems like a few people have tried it but looks like a very helpful and cool project if we can get it to work!

Here is a link of a guy explaining about what he had to go through to do it. Has helpful information on what he looked out for !
(He uses some training data + openpose)

https://youtu.be/kqvT2ALe6b8

[relientm96]

I can see that one issue is that a single word may be represented by a series of actions and not just one ( based on watching sample sign language videos on youtube).

This means that we need to analyze not just a frame but multiple frames over time then deduce what word it is.

[nivlekp]

Another rough proposal from me, please feel free to trash it.

Gesture control in music with pose estimation

Background problem

Although gesture control have became a trend since this century, many of them involves wearables.
(TODO: we might need a stronger motivation?)

Here are some examples of gesture control in music:

• MyoSpat
• SoundMorpheus
• gSPAT (Search this word with care please)
  • video

Goal

To give more freedom to the user and make it more accessible, we propose to build a system where users are able to control musical parameters without any wearables but just webcam(s).

Project description and idea of tasks

• Identify a pose/gesture recognition system that would be able to do pose estimation in real-time (OpenCV, OpenPose, etc.)
• More about gesture recognition in music (in real-time)
  • Real-time Gesture Mapping in Pd Environment using Neural Networks
• Identify ways to interact with computer in a musically meaningful way (we can refer to the papers above)
• Mapping strategies (have a look at Tanaka and Wanderley
• Hook the output (gestures recognized) of the above system to an open source digital music software (Pure-data, SuperCollider, etc.)

Reasons to backup

• Not much at the moment apart from that music interests me in general, can lookup for more reasons/motivations if you guys are onboard.

[relientm96]

From our discussion today, I think we are all fairly comfortable with the shortlist of

• Real Time Pose Estimation for Sign Language Translation
• Real Time Pose Estimation in Gesture to Music Mapping

I will close this issue for now as we have narrowed down our scope to these two projects. Please use the Project Framework issue to discuss project proposal stuff.