The tabletennistracker from jainr3

Table Tennis Tracker

Tracks a table tennis ball as a game is being played and attempts to automatically keep score.

Table of Contents

About The Project
- Built With
Getting Started
- Prerequisites
- Installation
Usage
License
Contact
Acknowledgments

About The Project

This project tracks a table tennis ball as it moves around during gameplay. While in professional table tennis, the ball may be hit at much faster speeds, this project aims to track the ball at lower speeds using a regular mobile device camera. Additionally, the performance is highly dependent on the lighting conditions and overall environment setup. There are two ball tracking algorithms that are being used, both of which use a color based image map for orange table tennis balls. The color thresholds for "orange" may need to be adjusted as described in the calibration section. The first algorithm detects the contours of orange blobs as seen here. The second algorithm uses hotbox detection similar to this project.

[Top]

Built With

[Top]

Getting Started

Follow the steps to set up the project locally.

Prerequisites

Android device connected to your PC over USB cable (recommended)
- Can also use Raspberry Pi instead of PC (from testing it seems to be slower)
DroidCam Android Application and Windows Client
- See here for download link
Tripod with phone mount (Optional but recommended)

Installation

Connect android device to PC over a USB cable. To make the connection, first enable developer options on the android device by going to Settings > About Phone and tapping the "Build Number" 7 times in a row. This will enable a new "Developer options" menu that is available under Settings or Settings > System. In the Developer Options menu, enable the USB debugging option. This will allow the PC to read data from the android device.
Now open the DroidCam application on the phone and open it on the PC as well. You might have to allow access to the camera / microphone on the phone side as well as accept any other permissions. After a few seconds, the phone will be automatically detected on the DroidCam Client on the PC as shown below. Be sure to select the "Connect over USB" option as that will provide the best latency versus sending the video stream over the internet. To get better resolution, you might want to enable HD mode by watching an ad on the android device. Also, the PC application should have a setting to increase resolution (you may have to restart the computer).

If you instead want to use a Raspberry Pi, install the Linux distribution of DroidCam. I used a premade script and also had to install adb using the following command:

sudo apt-get install adb

The instructions at the original source for linux doesn't translate exactly to Rasbperry Pi. There was also a small issue with the premade script which is discussed here. Additional references that may be helpful: 1, 2. The Raspberry Pi + DroidCam + Android over USB seemed to give a delayed video feed, so the PC method is recommended. Also if you are using the Raspberry Pi, modify the camera source in the camera.py file as the comments advise.

Next, clone the repository

git clone https://github.com/jainr3/TableTennisTracker.git

Install the required libraries.
```
pip3 install -r requirements.txt
```

[Top]

Usage

Calibration

For best performance, mount the phone on a tripod. Then aim the phone's camera at the table so it is level with the surface and centered at the net. Assume that the humans are out of the camera frame. Also make sure there is enough light in the environment. I had to switch out yellow light bulbs with bulbs that give white light.

Place a couple of ping pong balls around the table. I used orange because it contrasts well with the environment. Take a photo of the setup with the ping pong balls and place it in the img directory as 'calibrate.png'. Now, run the calibration script (source) to figure out the best masking for the color of the ping pong ball. Keep changing the sliders until only the ping pong ball(s) remain in the image. For my setup it looked like the following.
```
python3 calibrate_color.py
```

Now take the number range in the sliders and in camera.py, modify the colorLower and colorUpper tuples. The script should now be calibrated for the environment. To confirm, run the confirmation script using the following command. There may be some noise, as in the case of my example, where a ping pong racket was partially detected.
```
python3 confirm_mask.py
```

Running the flask app

Navigate to the flaskapp folder and run the following command.
```
python3 app.py
```

Then go to the url that appears to view the video stream!

[Top]

Accessing the admin page

Once the application is running, go to the /login page to login as the administrator. The administrator credentials are:
```
Username: admin
Password: admin
```
In the future, a real database can be connected to allow for more roles and to obfuscate these credentials.

After logging in, the video stream will appear along with some controls to manage table tennis games. The admin has the option to start a game, begin recording, add/subtract points, and end a game.

When a game is active, the following options will appear.

Gameplay and automatic scoring

The gameplay is tracked via a state machine (original source).

To start serving, the server should raise the ball within camera's field of vision until it is above a certain height and a chime is heard. Then the server may begin their serve. The rest of the state transitions follow the diagram where the computer attempts to detect bounces, hits, timeouts, and ball side switches. When a point is scored, another sound will be played. The sound functionality seemed to cause delays in frame processing, so was disabled.

Camera Debug Mode

The Camera has a debug mode as indicated by the flag in the Camera class. When set to true, the debug mode will enable hotbox tracking for the ball, display text whenever a bounce or hit is detected, as well as display the state machine's current state on the camera feed.

Future Improvements

There is definitely room for improvement in the project. Some of these areas include:

Lighting condition calibration, Hotbox detection
Higher frame rate camera + processing capabilities to prevent "blur"
Unhandled states in the state machine such as ball hitting the net but still going over

Example of "blur"