Error when doing white-box attack on OneFormer model about oneformer

[praeclarumjj3]

Hi @momo1986, the error arises because you probably do not prepare an input text in your dataset mapper. Are you sure you are using the correct dataset mapper?

OneFormer/oneformer/data/dataset_mappers/oneformer_unified_dataset_mapper.py

Line 372 in 4962ef6

dataset_dict["text"] = text

OneFormer/oneformer/oneformer_model.py

Lines 282 to 283 in 4962ef6

	if self.training:
	texts = torch.cat([self.text_tokenizer(x["text"]).to(self.device).unsqueeze(0) for x in batched_inputs], dim=0)

[momo1986]

Hi, @praeclarumjj3.

Thanks for your reply.

I know that the evaluation dataset should be given the token "text" to enable the training mode.

Is there a demo that "text" field could be edited and set in a correct parameter when "--eval-only" is given.

I wish you are all good.

Best Wishes!
Yours Sincerely,
Momo

[praeclarumjj3]

Hi @momo1986, you would need to create a "text" list inside the evaluation dataset mapper if you want to use it during inference.

OneFormer/oneformer/data/dataset_mappers/dataset_mapper.py

Line 21 in 4962ef6

class DatasetMapper:

You can take a look at the training dataset mapper for reference.

OneFormer/oneformer/data/dataset_mappers/oneformer_unified_dataset_mapper.py

Line 26 in 4962ef6

class OneFormerUnifiedDatasetMapper:

[momo1986]

Hi, I have generated the panoptic file as README.md prompted.

I also set as the training mode.
python train_net_adv.py --num-gpus 1 --config-file configs/cityscapes/convnext/oneformer_convnext_large_bs16_90k.yaml --eval-only MODEL.IS_TRAIN True MODEL.WEIGHTS 250_16_convnext_l_oneformer_cityscapes_90k.pth MODEL.TEST.TASK semantic
However, I obtained the error.

This is the error log.

Traceback (most recent call last):
File "train_net_adv.py", line 438, in
launch(
File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/engine/launch.py", line 84, in launch
main_func(*args)
File "train_net_adv.py", line 419, in main
res = Trainer.test(cfg, model)
File "train_net_adv.py", line 369, in test
results_i = inference_on_dataset(model, data_loader, evaluator)
File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/evaluation/evaluator.py", line 241, in inference_on_dataset
for idx, inputs in enumerate(data_loader):
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/ubuntu/anaconda3/envs/oneformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/data/common.py", line 95, in getitem
data = self._map_func(self._dataset[cur_idx])
File "/home/ubuntu/junyan/Python/segmentation/detectron2/detectron2/utils/serialize.py", line 26, in call
return self._obj(*args, **kwargs)
File "/home/ubuntu/junyan/Python/segmentation/OneFormer/oneformer/data/dataset_mappers/oneformer_unified_dataset_mapper.py", line 301, in call
raise ValueError(
ValueError: Cannot find 'pan_seg_file_name' for panoptic segmentation dataset datasets/cityscapes/leftImg8bit/val/frankfurt/frankfurt_000001_062250_leftImg8bit.png.

I feel sorry for disturing you. Nevertheless, I am looking forward to addressing the problem.

Thanks & Regards!
Yours Sincerely,
Momo

[momo1986]

Looks the pan seg file is existed, how can I link them together?

The file structure is set as recommended.

Thanks & Best wishes!
Regards!
Momo

[momo1986]

Looks like that the model would touch the task of panoptic segmentation when the mode is set to the training model. However, for attacking adversarially in the inference time, evaluation only on semantic segmentation is what we want? Is there any workaround in UI to skip this issue?
Thanks & Regards！
Momo

[praeclarumjj3]

Hi @momo1986, were you able to solve this issue? I am sorry for not being able to reply earlier, been busy.

[praeclarumjj3]

I am closing this issue due to inactivity. Feel free to re-open.