Support for SSD Object Detection Model? about torchdistill

[yoshitomo-matsubara]

Hi @anujdutt9

Thank you for the info.

1. If you use their implementations, you will first need to prepare a model wrapper to align the interface of object detection models in torchvision since the example code in my repo is designed to work with them. To avoid that hassle, you can consider Faster R-CNN models with MobileNetV3 in torchvision that were recently released as pretrained models.
2. KD by Hinton et al. trains student model in end-to-end manner, but this requires the shape of final output from teacher to match that from student, and it would be difficult to apply it to object detectors. While image classifiers always return a fixed-shape output (1 x num_classes) given an image, it may be not always true for object detectors in a sense that the number of bounding boxes predicted by teacher may be different from that by student. On top of that, the object detection task is a combination of regression (bbox coordinates) and classification (for each bbox), and the bboxes predicted by teacher and student may target different objects even if their final output shape matches, that will block knowledge distillation in end-to-end manner.

I'd suggest that you find a few papers about KD specifically for object detection so that you can see

• how the loss functions look,
• from what layers in student/teacher you'll need to extract the input/output in order to compute the losses, and
• whether or not the methods are applicable to your models

[yoshitomo-matsubara]

Hi @anujdutt9

Sure, I can help you use torchdistill for SSD object detectors.

Could you provide URLs for

1. PyTorch models (e.g., PyTorch hub, package, or repo) and
2. (optional) knowledge distillation approaches
   that you want to use?

Basically, torchidistill provides model-agnostic pipelines, thus can collaborate with arbitrary models.
As you might know, this repo contains some official examples for R-CNN object detectors if you're interested in other detection models as well.

[anujdutt9]

Hi, @yoshitomo-matsubara Thanks for your response. Yes sure.

1. To begin with, I am thinking of the SSD model provided by PyTorch here: https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/ as the teacher, and then I can either use this model with some changes or create a smaller version of it like using a smaller backbone and use that as a student.
   An example could be using an SSD with MobileNetv2/3 as the backbone as implemented here: https://github.com/qfgaohao/pytorch-ssd or since the SSD teacher model above uses a ResNet-50 model as the backbone, using a ResNet-18 as a backbone for the student model could work as well.

2. As for the knowledge distillation approaches, I am looking at the general approach as mentioned in the original paper by Geoffery Hinton, to begin with.

Yes, I did look at the existing models, but I am looking at a bit lighter, faster version of the model capable of running on mobile/edge devices, hence SSD or YOLO-like models were my first choice. Thanks