Inference on a new image about cops

[aabbas90]

I think you can first make the demo.py script work for panoptic deep lab as in https://github.com/facebookresearch/detectron2/blob/3cc9908ba3301785ec946ae7e37d7091fa1d5045/demo/demo.py#L26

Afterwards you can possibly adapt it to use COPS. The code of COPS is organized (and inspired from) panoptic deep lab so my best guess is it should work.

[aabbas90]

Hi @razvanbarbura sorry for late reply. I have added a demo script here:

https://github.com/aabbas90/COPS/blob/camera_ready/affinityNet/run_demo.sh

Feel free to reopen in case you encounter issues.

[aabbas90]

Hi, I am afraid not. What you can possibly do is to have a COCO style dataset with the images of your choice and pass it to eval_coco script.

[razvanbarbura]

Thank you for your fast reply. I will see what I can do.

[razvanbarbura]

Is there any chance to use a script like demo.py from detectron2?

[razvanbarbura]

Hi @aabbas90 , thank you again for your help. I tried to use the run_demo.sh script for cityscapes and I encounter one issue.
The model_0002999.pth is the model that you upload on google drive, and I didn't make a change in the config files.
Here are the changes that I made:

#!/bin/bash

#SBATCH -p gpu20
#SBATCH --ntasks=16
#SBATCH --gres gpu:1
#SBATCH -t 0-11:59:00
#SBATCH --mem=255000
#SBATCH --nodes=1
#SBATCH -o output/coco/logs/%j_eval.out

Make conda available:
. ~/.bashrc_private
eval "$(conda shell.bash hook)"
conda activate cops

MODEL_DIR="./output/cityscapes/fully_differentianle_with_eval"
WEIGHTS="model_0002999.pth"
OUT_FOLDER="model_0002999_results"

CONFIG="panoptic_affinity_fully_differentiable.yaml"
echo ${WEIGHTS}
echo ${MODEL_DIR}/${CONFIG}

python demo.py \
--config-file ${MODEL_DIR}/${CONFIG} \
--input sample_for_demo/input/berlin_000000_000019_leftImg8bit.png \
--output sample_for_demo/result \
MODEL.WEIGHTS ${MODEL_DIR}/${WEIGHTS} \
OUTPUT_DIR ${MODEL_DIR}/${OUT_FOLDER} \
DATALOADER.EVAL_BATCH_SIZE 1 \
DATALOADER.NUM_WORKERS 0 \
MODEL.SAVE_RESULT_IMAGES True \
DATASETS.TEST "('cityscapes_fine_panoptic_val',)"

Here is my folder structure:

And here is the stacktrace with the error:

[07/03 15:38:30 detectron2]: Arguments: Namespace(confidence_threshold=0.5, config_file='./output/cityscapes/fully_differentianle_with_eval/panoptic_affinity_fully_differentiable.yaml', input=['sample_for_demo/input/berlin_000000_000019_leftImg8bit.png'], opts=['MODEL.WEIGHTS', './output/cityscapes/fully_differentianle_with_eval/model_0002999.pth', 'OUTPUT_DIR', './output/cityscapes/fully_differentianle_with_eval/model_0002999_results', 'DATALOADER.EVAL_BATCH_SIZE', '1', 'DATALOADER.NUM_WORKERS', '0', 'MODEL.SAVE_RESULT_IMAGES', 'True', 'DATASETS.TEST', "('cityscapes_fine_panoptic_val',)"], output='sample_for_demo/result', video_input=None, webcam=False)

Traceback (most recent call last):
File "demo.py", line 83, in
demo = VisualizationDemo(cfg)

File "/home/oem/Desktop/COPS/affinityNet/predictor.py", line 35, in init
self.predictor = DefaultPredictor(cfg)

File "/home/oem/anaconda3/envs/COPS/lib/python3.7/site-packages/detectron2/engine/defaults.py", line 282, in init
self.model = build_model(self.cfg)

File "/home/oem/anaconda3/envs/COPS/lib/python3.7/site-packages/detectron2/modeling/meta_arch/build.py", line 22, in build_model
model = META_ARCH_REGISTRY.get(meta_arch)(cfg)

File "/home/oem/Desktop/COPS/affinityNet/panoptic_affinity/panoptic_seg_affinity.py", line 295, in init
self.AFF_EMBED_head = build_aff_embed_branch(cfg, self.backbone.output_shape())

File "/home/oem/Desktop/COPS/affinityNet/panoptic_affinity/panoptic_seg_affinity.py", line 649, in build_aff_embed_branch
return AFF_EMBED_BRANCHES_REGISTRY.get(name)(cfg, input_shape)

File "/home/oem/anaconda3/envs/COPS/lib/python3.7/site-packages/detectron2/config/config.py", line 190, in wrapped
init_func(self, **explicit_args)

File "/home/oem/Desktop/COPS/affinityNet/panoptic_affinity/panoptic_seg_affinity.py", line 699, in init
super().init(input_shape, decoder_channels=decoder_channels, norm=norm, **kwargs)

File "/home/oem/anaconda3/envs/COPS/lib/python3.7/site-packages/detectron2/config/config.py", line 192, in wrapped
init_func(self, *args, **kwargs)

TypeError: init() got an unexpected keyword argument 'in_features'

[aabbas90]

Hi @razvanbarbura,

I think you need to update detectron2 submodule and make sure it points to
https://github.com/aabbas90/detectron2/tree/e96afbad765b27bddf55237bc4a051963690aba0

[razvanbarbura]

Hi @aabbas90 , I made some progress, I do not encounter that error anymore but, unfortunately I've got this one:
Traceback (most recent call last):
File "demo.py", line 93, in <module> predictions, visualized_output = demo.run_on_image(img)

File "/home/oem/Desktop/COPS/affinityNet/predictor.py", line 48, in run_on_image predictions = self.predictor(image)

File "/home/oem/anaconda3/envs/COPS/lib/python3.7/site-packages/detectron2/engine/defaults.py", line 223, in __call__ predictions = self.model([inputs])[0]

File "/home/oem/anaconda3/envs/COPS/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs)

File "/home/oem/Desktop/COPS/affinityNet/panoptic_affinity/panoptic_seg_affinity.py", line 487, in forward image_path = input_per_image['file_name'].split(os.sep)

KeyError: 'file_name'

I used all the libraries that you provided.

[aabbas90]

Hi, you can do MODEL.SAVE_RESULT_IMAGES False in the bash script since the result is going to be saved anyway as you are running the demo.

Specifically I wrote some functionality here

COPS/affinityNet/panoptic_affinity/panoptic_seg_affinity.py

Line 486 in de324db

if self.save_result_images:

to save all network outputs for debugging. If you would like to save these outputs as well then please override image_path in line 487 by something else. The error is occurring because detectron2 is not giving the name of image file during the demo.