huawei-noah / efficient-ai-backbones

Efficient AI Backbones including GhostNet, TNT and MLP, developed by Huawei Noah's Ark Lab.

Efficient AI Backbones

including GhostNet, TNT (Transformer in Transformer), AugViT, WaveMLP and ViG developed by Huawei Noah's Ark Lab.

• News
• Model zoo
• Citation
• Other versions

News

2023/02/27 The paper of ParameterNet is accepted by CVPR 2024.

2022/12/01 The code of NeurIPS 2022 (Spotlight) GhostNetV2 is released at ./ghostnetv2_pytorch.

2022/11/13 The code of IJCV 2022 G-Ghost RegNet is released at ./g_ghost_pytorch.

2022/06/17 The code of NeurIPS 2022 Vision GNN (ViG) is released at ./vig_pytorch.

2022/02/06 Transformer in Transformer (TNT) is selected as the Most Influential NeurIPS 2021 Papers.

2021/09/18 The extended version of Versatile Filters is accepted by T-PAMI.

2021/08/30 GhostNet paper is selected as the Most Influential CVPR 2020 Papers.

Model zoo

Model	Paper	Pytorch code	MindSpore code
GhostNet	GhostNet: More Features from Cheap Operations. [CVPR 2020]	./ghostnet_pytorch	MindSpore Model Zoo
GhostNetV2	GhostNetV2: Enhance Cheap Operation with Long-Range Attention. [NeurIPS 2022 Spotlight]	./ghostnetv2_pytorch	MindSpore Model Zoo
G-GhostNet	GhostNets on Heterogeneous Devices via Cheap Operations. [IJCV 2022]	./g_ghost_pytorch	MindSpore Model Zoo
TinyNet	Model Rubik’s Cube: Twisting Resolution, Depth and Width for TinyNets. [NeurIPS 2020]	./tinynet_pytorch	MindSpore Model Zoo
TNT	Transformer in Transformer. [NeurIPS 2021]	./tnt_pytorch	MindSpore Model Zoo
PyramidTNT	PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture. [CVPR 2022 Workshop]	./tnt_pytorch	MindSpore Model Zoo
CMT	CMT: Convolutional Neural Networks Meet Vision Transformers. [CVPR 2022]	./cmt_pytorch	MindSpore Model Zoo
AugViT	Augmented Shortcuts for Vision Transformers. [NeurIPS 2021]	./augvit_pytorch	MindSpore Model Zoo
SNN-MLP	Brain-inspired Multilayer Perceptron with Spiking Neurons. [CVPR 2022]	./snnmlp_pytorch	MindSpore Model Zoo
WaveMLP	An Image Patch is a Wave: Quantum Inspired Vision MLP. [CVPR 2022]	./wavemlp_pytorch	MindSpore Model Zoo
ViG	Vision GNN: An Image is Worth Graph of Nodes. [NeurIPS 2022]	./vig_pytorch	-
LegoNet	LegoNet: Efficient Convolutional Neural Networks with Lego Filters. [ICML 2019]	./legonet_pytorch	-
Versatile Filters	Learning Versatile Filters for Efficient Convolutional Neural Networks. [NeurIPS 2018]	./versatile_filters	-
ParameterNet	ParameterNet: Parameters Are All You Need. [CVPR 2024].	./parameternet_pytorch	-

Citation

@inproceedings{ghostnet,
  title={GhostNet: More Features from Cheap Operations},
  author={Han, Kai and Wang, Yunhe and Tian, Qi and Guo, Jianyuan and Xu, Chunjing and Xu, Chang},
  booktitle={CVPR},
  year={2020}
}
@inproceedings{tinynet,
  title={Model Rubik’s Cube: Twisting Resolution, Depth and Width for TinyNets},
  author={Han, Kai and Wang, Yunhe and Zhang, Qiulin and Zhang, Wei and Xu, Chunjing and Zhang, Tong},
  booktitle={NeurIPS},
  year={2020}
}
@inproceedings{tnt,
  title={Transformer in transformer},
  author={Han, Kai and Xiao, An and Wu, Enhua and Guo, Jianyuan and Xu, Chunjing and Wang, Yunhe},
  booktitle={NeurIPS},
  year={2021}
}
@inproceedings{legonet,
  title={LegoNet: Efficient Convolutional Neural Networks with Lego Filters},
  author={Yang, Zhaohui and Wang, Yunhe and Liu, Chuanjian and Chen, Hanting and Xu, Chunjing and Shi, Boxin and Xu, Chao and Xu, Chang},
  booktitle={ICML},
  year={2019}
}
@inproceedings{wang2018learning,
  title={Learning versatile filters for efficient convolutional neural networks},
  author={Wang, Yunhe and Xu, Chang and Chunjing, XU and Xu, Chao and Tao, Dacheng},
  booktitle={NeurIPS},
  year={2018}
}
@inproceedings{tang2021augmented,
  title={Augmented shortcuts for vision transformers},
  author={Tang, Yehui and Han, Kai and Xu, Chang and Xiao, An and Deng, Yiping and Xu, Chao and Wang, Yunhe},
  booktitle={NeurIPS},
  year={2021}
}
@inproceedings{tang2022image,
  title={An Image Patch is a Wave: Phase-Aware Vision MLP},
  author={Tang, Yehui and Han, Kai and Guo, Jianyuan and Xu, Chang and Li, Yanxi and Xu, Chao and Wang, Yunhe},
  booktitle={CVPR},
  year={2022}
}
@inproceedings{han2022vig,
  title={Vision GNN: An Image is Worth Graph of Nodes}, 
  author={Kai Han and Yunhe Wang and Jianyuan Guo and Yehui Tang and Enhua Wu},
  booktitle={NeurIPS},
  year={2022}
}
@article{tang2022ghostnetv2,
  title={GhostNetV2: Enhance Cheap Operation with Long-Range Attention},
  author={Tang, Yehui and Han, Kai and Guo, Jianyuan and Xu, Chang and Xu, Chao and Wang, Yunhe},
  journal={arXiv preprint arXiv:2211.12905},
  year={2022}
}

Other versions of GhostNet

This repo provides the TensorFlow/PyTorch code of GhostNet. Other versions and applications can be found in the following:

0. timm: code with pretrained model
1. Darknet: cfg file, and description
2. Gluon/Keras/Chainer: code
3. Paddle: code
4. Bolt inference framework: benckmark
5. Human pose estimation: code
6. YOLO with GhostNet backbone: code
7. Face recognition: cavaface, FaceX-Zoo, TFace

efficient-ai-backbones's Issues

#将ghostnet训练place365分类效果不够理想的问题#

@iamhankai 您好，我尝试了将ghostnet网络在place365数据集上进行微调，在迭代了100个epoch之后，Acc@1: 35.% ,Acc@5: 62.%。
部分超参数设置如下：
batch_size = 256;
loss: CrossEntropyLoss();
optimizer: SGD();
lr_init=0.1 & Sets the learning rate to the initial LR decayed by 10 every 30 epochs

先只对最后的classifier训练，后面又对整个网络进行了训练，得到的两者的结果都很相近。
请问您能给一些在place365数据集上训练时的建议和帮助吗？
非常感谢！

线性的变换 为什么d*d呢 那不就仍是卷积？卷积核是d*d？

some questions about the network structure?

Hi! Thank you for your codes!
There are some differences in your code. In your paper, after AvgPooling, the features maps go through a conv1x1 and a fc layer to output the result, but in your pytorch version, the feature maps go through two fc layers, and in your tensorflow version, the feature maps seems to go through two conv1x1(fully convolution network), which one can produce the highest accuracy and reach the accuracy written in your paper?
Thank you!

Could you please tell us some details about GhostNet-BCD settings?

I read the paper Model Rubik’s Cube: Twisting Resolution, Depth and Width for TinyNets
I have interest to reproduce this on GhostNet-(X), could you please share us the details of r,d,c setting compare to GhostNet-A?

why not use huawei disout?

if use huawei disout,What is the effect？

Bloated model

Hi,
I am using Ghostnet backbone for training YoloV3 model in Tensorflow, but I am getting a bloated model. The checkpoint data size is approx. 68MB, but the checkpoint given here is of approx 20MB https://github.com/huawei-noah/ghostnet/blob/master/tensorflow/models/ghostnet_checkpoint.data-00000-of-00001

I am also training EfficientNet model with YoloV3 and that seems to be working fine, without any bloated size.

Could anyone or the author please confirm if this is the correct architecture or anything seems weird?
I have attached the Ghostnet architecture file out of the code.

Thanks.
ghostnet_model_arch.txt

Actual Inference Speed

What kind of ARM-based mobile phone you use testing the actual inference speed and the model runned on CPU or GPU?

GhostNet architecture

Hi,

According to the paper, you follow MobileNetV3 and replace bottleneck with G-bottleneck.
However, from the code, I think GhostNet is architecturally different from MobileNetV3.

For example, there is a layer in GhostNet with 48 #exp and 24 #out, which I couldn't find in MobileNetV3. Also, for the last few layers, MobileNetV3 has two 960 #exp and 160 #out layers but GhostNet has four of them. Moreover, in MobileNetV3 there are two 120 #exp and 40 #out layers while GhostNet only has one.

Can you explain why is this the case? How do you come up with this final architecture?
Have you tried directly replacing all the bottlenecks in MobileNetV3 to Ghost module without altering the base architecture?

Thanks,
Rudy

About the training cost

Hi, thanks for sharing such interesting work,

I find the proper inference information (e.g., FLOPs and Params) in the paper, and am wondering about the training cost instead, could you share the training cost comparison with other CNN architectures, like MobileNet-V3 structure you follow?

How to use it?

我已经下载了这个文档，但是不知道怎么用，还有就是那个pytorch版本为什么比这个tensorflow版本内容少那么多，刚入门不太懂，这个文档下载下来以后可以做一些迁移学习吗？刚入门的同学能用明白吗？

I have troubles when i using the pre-trained parameters, any one can help?

Hi, very thanks for your working, @iamhankai

but i have troubles when i using the pre-trained parameters, the result of the end_points['Conv2d_18'] was [[[[0,0,0...,0,0]]]] and the front vectors were allright. could u tell me how to solving this? thanks again!
(tf==1.14.0,python==3.6, tensorpack==0.10.1)

how to train ghostnet?

how to train ghostnet in my dataset?

关于论文思路感觉就是在残差链接concat输出的特征图上采用了多个固定权重的滑窗操作

具体的激活函数是作者设计，这个设计就很有讲究了，这些激活函数其实是类似于边缘检测算子或者高斯核，而不是非线性变换

Undefined name 'DepthConv' in myconv2d.py

flake8 testing of https://github.com/huawei-noah/ghostnet on Python 3.8.1

$ flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

./myconv2d.py:23:1: F822 undefined name 'DepthConv' in __all__
__all__ = ['DepthConv', 'GhostModule']
^
1     F822 undefined name 'DepthConv' in __all__
1

https://flake8.pycqa.org/en/latest/user/error-codes.html

Why did you exclude EfficientNetB0 from Accuracy-Latency chart?

@iamhankai Hi,

Great work!

1. Why did you exclude EfficientNetB0 (0.390 BFlops - 76.3% Top1) from Accuracy-Latency chart?

2. Also what mini_batch_size did you use for training GhostNet?

Any change to release the code for Object Detection task

First of all, thanks for the great work.
This is related to #15 and #17
On section 4.2.2, this paper demostrates Ghostnet's briliant ability of computation cost reduce in object detection task. Unfortunely, there seems no related code in the repostory. It would be great if you can release it in the near future.

will you release pretrained ghostnet0.5 model for pytorch?

kernel size in primary convolution of Ghost module

Hi,
It is said in your paper that the primary convolution in Ghost module can have customized kernel
size, which is a major difference from existing efficient convolution schemes. However, it seems that in this code all the kernel size of primary convolution in Ghost module are set to [1, 1], and the kernel set in _CONV_DEFS_0 are only used in blocks of stride=2. Is it set intentionally?

How can I get same performance as in paper when applying GhostModule on ResNet-50?

Hi, very thanks for your working, @iamhankai

I'm interested in this GhostModule design, so I substitude the GhostModule in ghostnet for vanilla convolution in ResNet-50. With s=2, paper shows distinguished performance as 75.0% Top1-Acc and 2.2G FLOPs. But I trained this model only to gain a 73.7% Top1-Acc and 2.4G FLOPs. Could you give me relevant training setting file or remind me of some special training details? Thanks again!

How can I get same performance as in paper when applying GhostModule on ResNet-50?

Hi, very thanks for your working, @iamhankai

I'm interested in this GhostModule design, so I substitude the GhostModule in ghostnet for vanilla convolution in ResNet-50. With s=2, paper shows distinguished performance as 75.0% Top1-Acc and 2.2G FLOPs. But I trained this model only to gain a 73.7% Top1-Acc and 2.4G FLOPs. That makes me very upset. Could you give me relevant training setting file or remind me of some special training details? Thanks again!

Faster R-CNN use GhostNet

I replaced the backbone network of Faster R-CNN with GhostNet and modified the structure of GhostNet, so I did not use the pre-training file you provided. This makes my model fail to converge. Can you share your experience in setting the hyperparameters? thank you

idea similar to dense-net?

To my understanding, suppose input feature map with shape (C, H, W),
for ordinary convolution:
output shape: (D, H, W), here suppose output D channels

while for ghost op, suppose ratio=2:
1. first, with ordinary convolution, input (C, H, W), output (D / 2, H, W),
2. second, still with ordinary convolution, input (D / 2, H, W), output (D / 2, H, W)
3. concatenate the above 2 output, then get final output (D, H, W)
here omit batch normalization, relu etc. for simplicity. The above three ops final reach almost the same effect with Densenet, though the motivation is different.

Please correct me if I have any misunderstanding.

作者大佬，有没有Ghost-VGG16模型的代码啊？

Pretrained model can't be loaded?

what's the version of your Pytorch? when I load the pretrained model,it will exits error:
_pickle.UnpicklingError: invalid load key, '
'.

On the method of feature visualization

Thank you for the open source code. Here I would like to ask what method is used to visualize the feature map in figure 1 of the paper. I look forward to your reply！

#寻求帮助#如何提升ghostnet在place365数据集上的效果问题？

@iamhankai 您好，我尝试了将ghostnet网络在place365数据集上进行微调，在迭代了100个epoch之后，Acc@1: 35.% ,Acc@5: 62.%。
部分超参数设置如下：
batch_size = 256;
loss: CrossEntropyLoss();
optimizer: SGD();
lr_init=0.1 & Sets the learning rate to the initial LR decayed by 10 every 30 epochs

先只对最后的classifier训练，后面又对整个网络进行了训练，得到的两者的结果都很相近。
请问您能给一些在place365数据集上训练时的建议和帮助吗？
非常感谢！

Replace Conv2d in my network, however it becomes slower, why?

Above all, thanks for your great work! It really inspires me a lot! But now I have a question.

I replace all the Conv2d operations in my network except the final ones, the model parameters really becomes much more less.
However, when testing, I found that the average forward time decreases a lot by the replacement (from 428FPS down to 354FPS).
So, is this a normal phenomenon? Or is this because of the concat operation?

Can I receive the pretrained ghostnet model with width ratio 1.3

Hello,

As per the paper, the validation accuracy on imagenet dataset of ghostnet is higher than mobilenet-v3 (75.7%). However, the actual experiment accuracy from this repo is 74% (lower than mobilenet-v3) .
The model ghostnet implemented at this repo is for width_ratio 1.0. Can I receive the pretrained ghostnet model with width ratio 1.3 (ghostnet 1.3x has 75.7% accuracy as per the paper)

EfficientNet-B0 performance do not matched its original paper

Hi, cheers for your amazing art!

I noticed that EfficientNet-B0 seems to have accuracy higher than 77%, but only about 76.3% according to Figure 6.
I wonder what causes this difference, is that the EfficientNet result could't be reproducible?

thanks for your kindly guidance.

yolov3和Darknet結合：Darknet5003中的Conv2d全部替換成Ghost Module

您好，我想問一下使用的預訓練的權重文件應該是yolov3還是Ghostnet的呢，期待您的回復！

GhostNet is way much slower than mobilenetV1?

When I test Ghostnet (drop all SE-modules) and mobilenetv1 on caffe and TensorRT, it shows that Ghostnet is 2 times slower than mobilenet. Is it because there are too many convs in ghostnet model?

MAC and parameter count on Ghostnet

I have tried to implement Ghostnet myself on Tensorflow and I get about 3.7M parameters and about 138M MAC. Can you confirm that I should expect 5.2M parameters and 141M MAC?

读论文和程序遇到的一些疑惑。期待您的解惑，谢谢。

您好，最近我读完这篇论文，我想请教几个问题1.程序中使用Ghost Bottleneck完成标准卷积和廉价变换。文中说Ghost module就可以完成(是在程序中一个Ghost Module第一卷积是标准卷积，第二卷积中去掉激活函数么)。2.程序中的恒等变换指的是满足条件的shortcut么。期待您的解惑，谢谢。

关于对线性变换的一些问题

作者您好，首先恭喜这篇工作中了CVPR
我对Ghost模块的理解是，先对原始特征图A做常规的卷积得到特征图B，再对特征图B做分组卷积得到其他特征图C，最后特征图B,C concat到一起进入到下一层
我想的是您文中说的这部分是个线性变换，但是您还是在对特征图B分组卷积后进行了一次Relu激活，加入激活函数不就是打破了线性吗，这不应该是一个线性变换。
望作者能解答一下

Is there detection code?

I saw that there was detection results on the paper.

For the proper re-production of detection results, I'd like to refer your code... but seems not included in this repository.

Is there any other repository for the detection? or not available?

why video memory increase

Excuse me, why I used your GhostModule, but the video memory increased. I simply experimented, turning a channel number of 64 into a channel number of 128 through convolution, and the program showed increased video memory. Is the number of channels too small to fit, or the number of ghostmodules is not enough, thank you for your answer.

resnet50 + ghost module

ghostnet在 resnet50中的那个实验，是怎么把ghost module结合进去的呀，cheap conv用的是depth conv吗

No speed up after using GhostModule!!

Thanks you guys' work a lot firstly.
I replaced the BasicBlock in ResNet18 with the GhostModule,but the inference processs cost more time after using it with GPU.It worked as the paper shows with CPU. Is this reasonable??

Can more pretrained model(TF) be released?

I find that in this repo, there is only a pretrained TF model with width ratio 1. Can more pretrained models with diverse width ratio(such as 0.75, 0.5, etc..) to be provided?
Thanks.

Can I only replace the Conv2d in my net with GhostModule rather than use the GhostNet?

Can I only replace the Conv2d in my net with GhostModule rather than use the GhostNet?

replace conv2d with GhostModule in crnn， the test accuracy decrease...

the crnn repo i used：https://github.com/Sierkinhane/crnn_chinese_characters_rec
I replace conv2d with GhostModule in the cnn part，then train with my data， same epoch（100）， the test accuracy is lower than the original conv2d（0.9824 vs 0.9879）， Is this normal？

Is there a ghostnet caffe model?

Is there a ghostnet caffe model?

converted to darknet2caffe but not working properly in nnie 3519a,3559a.

What should I do if I want to test ghostnet or tinynet in 3519a or 3559a?

The question of formula (5) in the thesis

Formula (5) is as follows:

I think there is an error in the calculation of the parameters of the depth separable convolution part of the denominator. The correct formula should be as follows:

kernel size in GhostModule

Good Work! Thanks for sharing.
Why is the kernel size in primary_conv 1 and this in cheap_operation 3(dw) ? If you have tried setting the kernel size in primary 3 and in cheap_operation 1(dw)? It seems that the latter has less FLOPs. (Although I guess this adjustment may lead to worse behavier.)

Different levels of computational complexity

Hi,
Thanks for your explicit work.
May I ask how to set the arguments to realize different levels of computational complexity, like 0.5×, 1.0×, 1.3×?

Done

How does the Ghostnet train?

Hello, how does the Ghostnet train? Can you provide the source code for the training model? Will you open the training code in future?

ratio is always 1，the stride slice do nothing

after train by myself，I convert to tflite。found the strideslice don‘t change the channel，then I found in the Ghostmodel channel_mult is always equal ratio-1(2-1=1),so filters-init_channels=0.the strideslice is useless

if i want use ghostnet to resnet-50,i just need replace nn.Conv2d() with GhostModule?

ghostnet is more efficient for object detection？

论文的实验结果中，ghostnet在分类任务中用和mobilenetv3相当的FLOPs做到几乎同样的准确率，而在检测任务中用更加少的FLOPs就可以做到和mobilenetv3同样的准确率。请问这个现象怎么解释？

分类：

Model	FLOPs	Top-1 Acc (%)
MobileNetV3 Large 1.0×	219	75.2
GhostNet 1.3×	226	75.7

检测：

Backbone	Detection Framework	Backbone FLOPs	mAP
MobileNetV3 1.0×	Faster R-CNN	219M	26.9%
GhostNet 1.1×	Faster R-CNN	164M	26.9%