ckpt reproduce about dvis

[zhang-tao-whu]

Hi! I tried to retrain DVIS-offline using 4 2080ti GPUs and set the batch size to 4 (ensuring one batch per GPU), and it worked well. Here are the training logs
nohup.txt
and results
stdout - 2023-06-26T102917.565.txt
.

I found that your training log is not normal. After the training is completed, the total_loss should be reduced to around 10. If you haven't changed the configuration file (ensuring that the GPU numbers and batch numbers are same), it may be due to the version of PyTorch. Previously, I also encountered this phenomenon using PyTorch 1.11 on 8 V100 GPUs. If possible, I suggest that you create a new virtual environment and configure a different version of PyTorch (a earlier or newer version). I have run well on PyTorch 1.9.0 (cuda 10.2 or 11.3).

[zhang-tao-whu]

OK. I will retrain DVIS-offline tonight and try to identify the bug. If you could provide me with your training log, that would be great.

[junho-jo]

This is my training log.txt! Please let me know if there is anything strange or something I need to check.

[junho-jo]

As you mentioned, the settings as written in install.md work just fine (torch==1.9.0, cuda==11.1) .
I appreciate for your kind reply!! thx!

[Kunaldargan]

@zhang-tao-whu ,
Thank you for providing the code and installation instruction.
I'm using following version of pytorch with cuda 11.1

Name Version Build Channel
torch 1.9.0+cu111 pypi_0 pypi

while training is completed on a dataset similar to youtube-vis 2019.
In offline training final loss is around 25, In online training final loss is around 5.
During evaluation: mAP is 0.01 for Offline setting and 0.22 for Online setting.

Please review.

[zhang-tao-whu]

Since the segmenter is frozen during the training process of DVIS (tracker and refiner), it is necessary to fine-tune the segmenter using your own dataset.

Please refer to GETTING_STARTED.md. In the 'Training on a new dataset' section of this document, I provide a detailed explanation of how to train DVIS on a completely new dataset. Please follow this process and try again.