Data Cleaning pipeline task by Kopernikus

Step 1. Create and activate the conda environment

conda env create -f environment.yml
conda activate data_cleaning

Step 2. Run the main file with the path to the images

python main.py --dir <path to images>

About the dataset:

This dataset comprises images depicting covered parking lots.
Multiple cameras have been employed to capture diverse perspectives within the dataset.
Each image is labeled with a unique identifier composed of the camera ID and timestamp.
Timestamps are provided in two formats: Unix and Python datetime.
The images encompass various times of the day.
Some images may contain noise, and there are instances where images are None.
Images vary in size.

The entire process is repeated for each unique camera_id to systematically identify and mark images for removal across all cameras.

Each camera provides a distinct viewpoint, so we organize the dataset based on the camera_id. We maintain a dictionary that associates each camera_id with its corresponding set of images.
To facilitate sorting, we convert Python datetime objects to Unix timestamps. This ensures the chronological ordering of the images across all cameras.
We sort the images based on their timestamps. This step is crucial as images captured closer in time are more likely to exhibit similarities.
We define a sliding window mechanism to efficiently compare images. This involves selecting a base image and comparing it with others within a specified window.
To optimize processing, we implement a caching mechanism. This prevents the unnecessary reloading of images within a window, improving overall efficiency.
The first image within a window is chosen as the base image for comparison. Subsequent images within the window are compared against this base image
Images are compared using a scoring mechanism, and if the score exceeds a predefined threshold, the image path is marked for removal.
Remove the images marked for removal from the directory.

The values for min_contour_area and min_score were chosen with experimentation. We chose min_contuor_area as 100 and min_score as 3000.

Starting with conservative values and analysing the results these scores were estimated to be giving good results.

Curated Images: By using deep learning models to selectively capture images based on specific criteria, you can focus on collecting only the relevant data for your use case. For instance, in a parking lot scenario, you can capture images containing only cars. This reduces the amount of irrelevant data and makes data storage and processing more efficient. By selectively capturing and storing only the necessary images, you can significantly reduce storage costs and optimize computational resources.
GDPR Compliant: If required by the law, faces and license plates must be blurred. This will save the company from future legal troubles.
Image naming: Choose a consistent image naming scheme for consistency purposes and avoid any future confusion and errors.
All angles: For a given timestamp image of a parking lot must be taken from all cameras for better scene understanding.