Segmentation is not working about depth-anything

[LiheYoung]

Thank you for your feedback. Yes, the dinov2_vit14 should be changed to dinov2_vitl14.

Just now, I re-ran with the latest MMSegmentation (mmseg==1.2.2, mmcv==2.1.0, mmengine==0.10.2). After installation, I follow the four steps listed here to add the DINOv2 backbone to the mmsegmentation working directory.

Then, I can successfully reproduce our results when using the command line below (the mIoU in the image is 84.8% due to single-scale test):

python3 tools/test.py configs/depth_anything/depth_anything_large_mask2former_16xb1_80k_cityscapes_896x896.py cityscapes_vitl_mIoU_86.4.pth

I did not encounter any problem during reproducing. Can you tell me more about the errors you encountered?

[LiheYoung]

Yes, there is no need to install mmseg as a subfolder of Depth Anything. Just copy the necessary files mentioned in the four steps into your mmseg project. We have updated the files under semseg. Please use the latest files. Thank you.

[LiheYoung]

Please try:

pip3 install "mmdet>=3.0.0rc4"

And then re-run the test command.

[LiheYoung]

Hi, sorry for the confusion. We provide the torchhub directory here. Please put it under your working space.

[MrNeRF]

Thank you for you help.

Unfortunately, it does not run with mmsegmentation. I get all the examples running that come with mmsegmentation. So this is not the issue. But when I try depth anything segmentations, it is not working. It starts to crash again.

This line is also wrong:

self.dinov2 = torch.hub.load('torchhub/facebookresearch_dinov2_main', 'dinov2_vit14', source='local', pretrained=False)

There is no dinov2_vit14. It must be dinov2_vitl14 I think. However, it does not integrate properly with mmsegmentation.
Did you use any specific version of mmsegmentation? Is the Readme some essential information missing? I am a bit puzzled why this is not working.

[MrNeRF]

Thank you for your answer!
I guess mmsegmentation can be in its own conda environment? It does not need to be a subfolder of depth anything I assume.

So basically installing an isolated mmsegmentation project and then following the 4 steps?

I will retry it again later tonight. I really appreciate your help. Thank you!

[MrNeRF]

So this is what I get:

Traceback (most recent call last):
File "tools/test.py", line 123, in
main()
File "tools/test.py", line 116, in main
runner = Runner.from_cfg(cfg)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 462, in from_cfg
runner = cls(
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 429, in init
self.model = self.build_model(model)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 836, in build_model
model = MODELS.build(model)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 232, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/home/paja/projects/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 92, in init
self._init_decode_head(decode_head)
File "/home/paja/projects/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 102, in _init_decode_head
self.decode_head = MODELS.build(decode_head)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 232, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/home/paja/micromamba/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/home/paja/projects/mmsegmentation/mmseg/models/decode_heads/mask2former_head.py", line 42, in init
super().init(**kwargs)
TypeError: init() got an unexpected keyword argument 'in_channels'

I am running the following command:

python3 tools/test.py configs/depth_anything/depth_anything_large_mask2former_16xb1_80k_cityscapes_896x896.py cityscapes_vitl_mIoU_86.4.pth

In this repo (depth anything) I am on HEAD (e911178)
In the mmsegmentation repo I am on HEAD (c685fe6767c4cadf6b051983ca6208f1b9d1ccb8)

The versions are:
mmcv==2.1.0, mmengine=0.10.2 (was before on 0.10.3 but has same error)
Obviously mask2former_head.py get some variables it does not like. So that's why I think there is some incompatibility and I am scratching my head how to get this running (without diving fully into the code base to understand all details)

Thank a lot for your help!

[MrNeRF]

Thank you. That did the trick! It runs. Maybe some additional info might be helpful for others to get this running as well.

[LiheYoung]

Thank you for your feedback! I will update the semseg README.

[MACILLAS]

@LiheYoung thank you very much for your response to this issue it was very helpful!

I just wanted to add that there seems to be some issue with mmsegmentation using Pytorch 2, running the test script. I tried both 2.1 and 2.2 with Cudatoolkit 11.8 and ended up installing Pytorch 1.13 with Cudatoolkit 11.7, which worked for me. (My GPU is ampere arch with driver 535...)

Hope this helps someone... :)

[cht619]

在训练seg的时候，有用到提到的一些loss吗