Background Data Missing about poet

[tgjantos]

Dear @z7r7y7 ,

you are confusing two things. The argument synt_background refers to images that you can sample from for synthetic images to replace their background. If you are working with the LMO dataset you have images that are synthetically rendered and thus have no background, i.e. just a black background. For the sake of data augmentation we allowed for the data sampler to sample an image and use it as a background instead of a blank background.

To summarize, the background data for synt_background should be a directory containing random images from which you can sample. For example, I use the validation set of the MSCOCO dataset.

However, you found a bug where it tries to sample from the background images, but it can't if there none provided. I will fix it immediately to raise a warning in that case.

Best,
Thomas

[z7r7y7]

Thank you very much for your response, and the issue regarding the background has been resolved!

While running the code for the object detection model using Mask R-CNN, I encountered another problem in the "poet/models/backbone_maskrcnn.py" file, specifically in lines 98-112, where the variable "img_predictions" is generated containing information about the bounding boxes, labels, and scores.
I noticed that in lines 101-106:
if self.obj_id_map is not None: if cls.item() in self.obj_id_map.keys(): new_cls = self.obj_id_map[cls.item()] box = torch.hstack((box, torch.tensor(new_cls, dtype=torch.float32, device=device))) else: box = torch.hstack((box, cls)) img_predictions.append(box)

when adding the label information (cls) to the boxes, if the "obj_id_map" is not empty but the label does not belong to the "obj_id_map", resulting in a 1x5-dimensional "box" without label information, while in other cases, it is a 1x6-dimensional "box" with label information. This dimension mismatch can lead to an error in torch.hstack in line 112:
img_predictions = torch.stack(img_predictions)
Therefore, I think it may be necessary to add a conditional statement to collect only the box information with dimensions of 1x6.

[tgjantos]

You are definitely right. The model might predict objects that are not in the object_id_map.

With the most recent commit, I implemented it as a skip if the detected object is not of "interest".

Hope it works now and thank you for raising this issue!

Best,
Thomas

[tgjantos]

Closed due to inactivity. Feel free to open the issue again.