adisingh50 / spectr

CS7643 Final Project. Implementation of SPECTR

SPECTR: Segmentation using Positional Encoding, CNNs, and Transformers

SPECTR by Anshul Ahluwalia, Aditya Singh, and Ranveer Thind

Visual Results (Cityscapes Image and Predicted Output):

Installation

To install SPECTR, we first much create a conda environment with the appropriate libraries.

conda env create -f environment.yml
conda activate spectr-v1

Next, install spectr as a module:

pip install -e .

Training Instructions

1. Download the Cityscapes Dataset onto your local machine and unzip the files: leftImg8bit_trainvaltest.zip and gtFine_trainvaltest.zip.
2. To preprocess the image data and labels, run python spectr/src/cityscapes_preprocess.py. Before running this script, make sure to modify the file paths to be correct for your machine.
3. Modify the hyperparameters defined in spectr/config/config.yaml.
4. Finally, run python spectr/src/lightning_train.py to train the model using the Pytorch Lightning Framework. As training proceeds, a recent copy of your model will be saved in the lightning_logs/checkpoints directory.

Testing Instructions

1. To evaluate the SPECTR model checkpoint on a random training or testing batch, run python spectr/src/lightning_test.py.
   • This will calculate mIoU for the model on the specific batch and dump all visualizations inside the viz folder.
   • You must specify the absolute file path to the pytorch checkpoint model.

spectr's Issues

Resizing semantically segmented labels creates new hex codes which we can't classify

Original Image (1920x1208):

Downsampled Image (480x304):

Notice that in the downsampled image, there are some intermediate pixels that are uniquely generated. Such as the half-yellow/half-pink pixels or the half-blue/half-pink pixels. This is a problem because when we implement our loss function, we won't be able to determine the class of these unique pixels, for their hex codes don't show up in class_list.json.