sovrasov / flops-counter.pytorch Goto Github PK
View Code? Open in Web Editor NEWFlops counter for convolutional networks in pytorch framework
License: MIT License
Flops counter for convolutional networks in pytorch framework
License: MIT License
Can you also support upsampling layer which is used in encoder-decoder models? Thank you.
Lack of support for depthwise separable convolution
If a module contain two types of operations:
The results output by ptflops will be misleading. Since in the calculation of full module's flops:
It just counts all flops of modules() output.
But to output per layer flops:
it ignores flops which a module's parent is in supported modules.
This mismatch cause the final results pretty confusing.
I think you should at least make these two types of calculation consistent.
flops, params = get_model_complexity_info(net, input_res=(3, 256, 512), input_constructor=prepare_input, as_strings=True, print_per_layer_stat=True)
input_res的格式是什么?channel,row,col?还是channel,col,row?
For now, I see that the GMac of each block is displayed, but for the number of parameters, only the total number is shown in the end. How can I know the exact number of parameters of each block? Would you please give me some advice? Thank you!
Hello, thank you for sharing your code. Recently I'm trying to design new architectures based on Mobilenet v2, but I found the Macc calculated by your project is different from it in the paper.
In the paper:
It should be 585 MMac and 6.9M paremeters for MobilNet V2(1.4x).
However,
I get :
By the way, I get the different result by Lyken17's code
code I used:
from torchvision.models import mobilenet_v2
model = mobilenet_v2(width_mult=1.4)
from ptflops import get_model_complexity_info
flops, params = get_model_complexity_info(model, (3,224,224,), as_strings=True, print_per_layer_stat=True,)
print('{:<30} {:<8}'.format('Computational complexity: ', flops))
print('{:<30} {:<8}'.format('Number of parameters: ', params))
Could you tell me the reason for the difference? Thanks.
When installing using pip with python 3.7, the following error occurs:
ERROR: Package 'dataclasses' requires a different Python: 3.7.9 not in '>=3.6, <3.7'
Then I try to run the source code directly, and it works well with python 3.7 and python 3.6.9.
It seems that it's ok to remove the dataclasses dependency to better support python 3.7+
After using this package, my model output is changed a little bit!
My model is 3D U-Net, and the input is a 3D image patch, the output is a 3d patch.
It took me some time to debug this and finally found that the problem is from using this package!
print_per_layer_stat=True, ignore_modules=[torch.nn.BatchNorm1d,torch.nn.BatchNorm2d,torch.nn.ReLU])
File "/usr/local/lib/python3.7/dist-packages/ptflops-0.6.2-py3.7.egg/ptflops/flops_counter.py", line 46, in get_model_complexity_info
File "/usr/local/lib/python3.7/dist-packages/ptflops-0.6.2-py3.7.egg/ptflops/flops_counter.py", line 141, in print_model_with_flops
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1134, in repr
extra_repr = self.extra_repr()
File "/usr/local/lib/python3.7/dist-packages/ptflops-0.6.2-py3.7.egg/ptflops/flops_counter.py", line 121, in flops_repr
ZeroDivisionError: float division by zero
Hey, it's a nice tool
However, I am wondering whether the return of get_model_complexity_info is correct.
Let's assume all calculations are in floating point.
1MACs = 2OPs
MAC = Mult + Add
FLOPS = 2MACS(roughly equivalent, especially those models set up by all convolutional layers)
Thus, in your code:
The returns show flops_count, params_count
However, all printing result from the function flops_to_string are in Mac
The return type is not consistent with the example as well
The units in flops_to_string are supposed to be flops instead of mac. Otherwise, half the flops to macs. Could you please check it out again?
Hey, Thanks for your package it is quite useful!
I have one small issue and that is that the models I use have a __str__
method defined on them, which prints a shorter version of the modules and thus the __repr__
is not used. This causes your print_per_layer_stat
argument not to work correctly.
In order to fix this, you could replace Line 144:
print(repr(model), file=ost)
This will force to use the __repr__
of the module, independent of whether a __str__
has been defined or not.
As I understand it, in PyTorch, Linear
is the name for a fully-connected layer. Its forward pass is a matrix multiplication of an input tensor of shape [B, M, K] by a weight tensor of shape [B, K, N], with an output with shape of [B, M, K]. The number of MACs (not counting any bias terms or other things) is B*M*N*K
. However, when I was using flops-counter.pytorch
in my work, I noticed that flops-counter.pytorch
's MAC estimates for Linear layers seemed really low relative to my hand-calculations. I think I found the issue. The current code (below) only does B*K*N
, and it does not account for the M dimension.
def linear_flops_counter_hook(module, input, output):
input = input[0]
batch_size = input.shape[0]
module.__flops__ += int(batch_size * input.shape[1] * output.shape[1])
The following is my proposed fix. At least on my particular models, it produces results that match both my intuition and my calculations. If it would be of use, I would be happy to submit the following as a pull request.
def linear_flops_counter_hook(module, input, output):
input = input[0]
M = input.shape[-2]
K = input.shape[-1]
N = output.shape[-1]
batch_size = input.shape[0]
module.__flops__ += int(batch_size * M*N*K)
Why use the GMACs?GMACs is different with GFLOPs.
When I am running code, I encounter the following warnings:
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleLinear ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleLinear ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleLinear ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleLinear ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleConv2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleBatchNorm2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleAdaptiveAvgPool2d ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleLinear ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleLinear ptflops can affect your code!
Warning: variables __flops__ or __params__ are already defined for the moduleReLU ptflops can affect your code!
How to avoid the warnings?
Does this already defined warning affect my code?
MACs: Multiply-Add cumulation?
FLOPs: Floating Point Operations?
Are they same? What is their relationship?
from the last table only saw MACs and params, is that FLOPs doesn't needed a record? While some paper still report their FLOPs/G for comparasion.
How to compute FLOPS for LSTM model for skeleton data (NTU-RGBD)? Error is due to input_res parameter as for skeleton data what should be this value.
I have two networks as following, I thought they should result in the same MACC and parameters but got different results. Any suggestion?
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torch.backends.cudnn as cudnn
import torchvision
import torchvision.transforms as transforms
import os
import re
from models import *
from ptflops import get_model_complexity_info
import torchvision.models as models
from tensorboardX import SummaryWriter
class NetOne(nn.Module):
def __init__(self):
super(NetOne, self).__init__()
self.conv1 = nn.Conv2d(3, 24, kernel_size=3, bias=False, groups= 1)
self.conv2 = nn.Conv2d(24, 24, kernel_size=3, bias=False, groups= 24)
def forward(self, x):
out = self.conv2((self.conv1(x)))
return out
class NetTwo(nn.Module):
def __init__(self):
super(NetTwo, self).__init__()
self.conv1 = nn.Conv2d(3, 24, kernel_size=3, bias=False, groups= 1)
self.conv2 = nn.Conv2d(24, 12, kernel_size=3, bias=False, groups=12)
self.conv3 = nn.Conv2d(24, 12, kernel_size=3, bias=False, groups=12)
def forward(self, x):
out = self.conv1(x)
out1 = self.conv2(out)
out2 = self.conv3(out)
out = torch.cat([out1,out2], 1)
return out
if __name__ == "__main__":
# Check the first network
net = NetOne()
with torch.cuda.device(0):
#net = models.densenet161()
flops, params = get_model_complexity_info(net, (3, 32, 32), as_strings=True, print_per_layer_stat=True)
print('Flops: ' + flops)
print('Params: ' + params)
# Check the second network
net = NetTwo()
with torch.cuda.device(0):
#net = models.densenet161()
flops, params = get_model_complexity_info(net, (3, 32, 32), as_strings=True, print_per_layer_stat=True)
print('Flops: ' + flops)
print('Params: ' + params)
And the results that I got is
NetOne(
0.001 GMac, 8e+05 MACs,
(conv1): Conv2d(0.001 GMac, 6e+05 MACs, 3, 24, kernel_size=(3, 3), stride=(1, 1), bias=False)
(conv2): Conv2d(0.0 GMac, 2e+05 MACs, 24, 24, kernel_size=(3, 3), stride=(1, 1), groups=24, bias=False)
)
Flops: 0.75 MMac
Params: 864
NetTwo(
0.001 GMac, 9e+05 MACs,
(conv1): Conv2d(0.001 GMac, 6e+05 MACs, 3, 24, kernel_size=(3, 3), stride=(1, 1), bias=False)
(conv2): Conv2d(0.0 GMac, 2e+05 MACs, 24, 12, kernel_size=(3, 3), stride=(1, 1), groups=12, bias=False)
(conv3): Conv2d(0.0 GMac, 2e+05 MACs, 24, 12, kernel_size=(3, 3), stride=(1, 1), groups=12, bias=False)
)
Flops: 0.92 MMac
Params: 1.08 k `
Thank you
I really appreciate this flops counter. Would you guys plan to support the calculation with FP16?
The efficientnet paper insists efficientnet-b0 is 0.39 Gflops, but the measurement is 00.2Gflops = (2*0.01 GMac)
% python
Python 3.6.5 (default, Nov 11 2019, 18:04:50)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ptflops import get_model_complexity_info
>>> from efficientnet_pytorch import EfficientNet
>>> network = EfficientNet.from_pretrained("efficientnet-b0")
Loaded pretrained weights for efficientnet-b0
>>> get_model_complexity_info(network, (3, 224, 224), print_per_layer_stat=False)
('0.01 GMac', '5.29 M')
今天大半天都被这个flops_counter带偏了,不管做什么尝试,我的shufflenetv2和mobilenetv2的loss 始终降不下来,而acc 就和随机猜得的一样,
global CUSTOM_MODULES_MAPPING = custom_modules_hooks
^
SyntaxError: invalid syntax
The flops of ConvTranspose2d operation maybe not correct?
It should be calculated as same as Conv2d: K_w*K_h*C_in*C_out*O_w*O_h
when group = 1.
However your implement is K_w*K_h*C_in*C_out*I_w*I_h
as follows:
Hi, I have a training script with structure as :
model=Net()
flops, params = get_model_complexity_info(model, (3, 256, 256), as_strings=True, print_per_layer_stat=True)
print('Flops: ' + flops)
print('Params: ' + params)
training
torch.save(model)
Then when I tried to load from above saved checkpoint using torch.load('checkpoint.pth'), it occurs into an error:
AttributeError: 'Net' object has no attribute 'start_flops_count'.
Do you have any idea why I get this error and how can i solve it?
Hi, I run the torch.nn.MultiheadAttention model and find it is 0 MACs. The simplified code is shown as follows. Could anyone give me a hand? The pytorch version is 1.8.1.
import torch
import torch.nn as nn
from ptflops import get_model_complexity_info
class NN(nn.Module):
def __init__(self, in_size, nhead):
super().__init__()
self.net = nn.MultiheadAttention(in_size, nhead)
def forward(self, x):
x = x.transpose(0, 1)
x = self.net(x, x, x)[0]
x = x.transpose(0, 1)
return x
model = NN(512, 4)
model.eval()
with torch.no_grad():
macs, params = get_model_complexity_info(
model, (580, 512), as_strings=True, print_per_layer_stat=True, verbose=True
)
print('{:<30} {:<8}'.format('Computational complexity: ', macs))
print('{:<30} {:<8}'.format('Number of parameters: ', params))
Error message:
~/anaconda3/envs/torch1.8/lib/python3.7/site-packages/ptflops/flops_counter.py in flops_repr(self)
122 flops_to_string(accumulated_flops_cost,
123 units=units, precision=precision),
--> 124 '{:.3%} MACs'.format(accumulated_flops_cost / total_flops),
125 self.original_extra_repr()])
126
ZeroDivisionError: float division by zero
Hello, the tool you supply is very useful, I wanna ask is there a similar tool, which can evaluate the params and the flops of a tensorflow model, such as xx.tflite or xx.pb
flops_model.eval()
will return NoneType
object.
Until I decompose the line 15 into:
flops_model.eval()
flops_model.start_flops_count()
the program processes properly.
please could you answer my questions
Q1- can we computing flops for a model without training the model? is there any relation between flops and training? can training affect flops? when flops can be computed?
I am asking this question because I defined a model and then I computed the flops and here is the results.
Computational complexity: 0.03 GMac
Number of parameters: 2.24 M
Q2- if we want to have flops in million, we should multiply 0.03 * 1000 ? if yes then for this case the computational complexity is 30.0 million.
Q3- what I understand from your code, Mac is flops, am I right ?
Thank you
Seems like you add bias addition ops to the total FLOPs count for convolutional and RNN modules, but it looks like it is not the case for linear layers. Is that something to be addressed?
Sorry, I found another problem. When I calculate the 3D model's params, the answer is different from my own results. My own code is
def count_param(model):
param_count = 0
for param in model.parameters():
param_count += param.view(-1).size()[0]
return param_count
A 3D model's param calculated by my code is 166.21M and by your code is 160.64 M
Can anyone help me regarding how to use this code for 10 class (other than Imagenet) classification with our custom architechture ?
This tool is very useful, but there is still a lack of functionality in the benchmark.
so, could u add benchmark mode support?
Just like:
code
flops, params = get_model_complexity_info(net, (batch, 3, 224, 224), mode='layer', loop=1000, as_strings=True, print_per_layer_stat=True)
output
(conv3d): Conv3d(x GMac, x% average MACs, x average TFLOPS, x average fps, x, x, kernel_size=[x, x, x], stride=(x, x, x), bias=True)
or
(conv3d): Conv3d(x GMac, x% average MACs, x average TFLOPS, x average ms, x, x, kernel_size=[x, x, x], stride=(x, x, x), bias=True)
It would be nice to have a dictionary containing each layer/operations' GMac and Params that would be useful to plot and be able to see network bottlenecks.
Hi, first of all thank you so much, that's a really helpful repo!
I'm now working with some pretrained BERT models, so I'm wondering if this flop_counter works for all types of BERT models? I noticed here's an example using BertForSequenceClassification, with input_res=(2, 128). But when I try some other BERT models, the input_res could be the big problem. If there any baseline for BERT model like you mentioned (similar to image networks) in README? Or is there any lookup table? Thank you very much!
Can GMac to be negative
I build custome model with almost 22 layers and I get -14.23 GMac
Hi @sovrasov ,
Why does the conv_flops_counter_hook
not consider the stride of the convolutions? Can you please add support for that as it significantly affects the MAC count?
Hello,
Does anyone encounter this issue? A negative flops.
C:\Anaconda3\envs\pytorch\lib\site-packages\ptflops\flops_counter.py:285: RuntimeWarning: overflow encountered in long_scalars
overall_conv_flops = conv_per_position_flops * active_elements_count
- Flops: -27913.09 MMac
- Params: 556.1 k
The tested network includes Convolution, Batchnorm, and Relu only.
Thank you
Hi,
This is a great repo! :-)
But can you add functionality to compute the addition count and multiplication count separately?
Thanks!
Traceback (most recent call last):
File "/home/xiaofei/my_project/SiamTrackers/compute_params/flops-counter.pytorch/sample.py", line 35, in
net = pt_modelsargs.model
File "/home/xiaofei/anaconda3/envs/siamtracker/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'x'
thank you
Hi, thanks for your tools. However, why did the result of GMac is negativae number when I calculate my own model? This is my code and output.
model =ResUNet(layers=[2,3,4,6,3])
macs, params = get_model_complexity_info(model,(1,256,256), as_strings=True,
print_per_layer_stat=True,verbose=True)
print('{:<30} {:<8}'.format('Computational complexity: ', macs))
print('{:<30} {:<8}'.format('Number of parameters: ', params))
Computational complexity: -52.1 GMac
Number of parameters: 97.23 M
I have fit a model with pixelshuffle operation to this flops calculation.
However, it cannot calculate the flops in pixelshuffle layer.
PixelShuffle(0.0 M, 0.000% Params, 0.0 GMac, 0.000% MACs, upscale_factor=2)
Can anyone help?
To get the number of FLOPs and params, use "get_model_complexity_info". But I get an Error, Illegal instruction (core dumped).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.