Comments (4)
- The M sub-matrices have their own codebook, thus M codebooks should be stored.
- To learn D and B, you only need unlabeled images from the training subset. The optimization process is to approximate the original network's each layer's activation, so there is no need for category labels. However, if you want to fine-tune the network, then the labelled training data is required.
- The inner products are only computed when you have an input image to the network. It is input-dependent. Different input images will produce different look-up tables.
from quantized-cnn.
-
then there must be sth wrong in my understanding, please correct me.
Each sub-matrix has own codebook, then just set the codeword in the codebook same with the submatrix. in another word, there is only one word in the codebook, and the word is exactly the submatrix.
And there is M codebooks need to be stored, instead of the whole weight matrix. but does the size of M codebooks smaller than the original matrix? -
so the computation cost of the product should also be included in the FLOP? and why do we need the look-up table? Every input will result in different product.
Thanks very much for your prompt reply :)
from quantized-cnn.
-
The codebook is a collection of codewords, and each codeword is a vector, not a matrix.
Let us consider the first fully-connected layer in AlexNet, which takes a 9216-D vector (9216=256*6*6) as input, and outputs a 4096-D vector. Here, we let the number of subspace dimensions to be 4, so the number of subspaces, i.e. M, equals to 9216/4=2304. So we split the 9216x4096 weighting matrix, i.e. W, into 2304 sub-matrices, each of size 4x4096. For each sub-matrix, we learn a codebook of size 4xK, which consists of K codewords, each of which is a 4-D vector. Note that K is much smaller than 4096. Then we use this codebook to quantize the sub-matrix, i.e. each column in the sub-matrix is approximated (or replaced) by a codeword selected from this codebook. -
The computation cost of inner products is included in the FLOPs.
Continue with the above example. For a 9216-D input vector, we split it into 2304 sub-vectors, each of 4-D. For each sub-vector, its inner products with all the 4096 column vectors in the corresponding sub-matrix is needed. Since the sub-matrix is quantized with the codebook, we can compute a look-up table of K elements, which are the sub-vector's inner products with all codewords in that codebook. So we can reduce the number of inner product computations from 4096 to K. The look-up table contains all the 4096 inner products we needed to compute the layer response.
from quantized-cnn.
Oh, I see! each codeword is a vector, not a matrix
Thanks very much for your explanation!!
from quantized-cnn.
Related Issues (20)
- Other Model Architectures HOT 1
- Error correction and fine-tune questions HOT 2
- Not able to read binary file HOT 13
- help HOT 1
- help
- help
- OpenVML installation HOT 3
- Quantizeing the Fully-connected Layer--->look-up table HOT 2
- https://github.com/jiaxiang-wu/caffe-QCNN HOT 2
- the onedrive file not exist( ILSVRC12.227x227.IMG and AlexNet/Bin.Files ) HOT 2
- how to do the quantization about inputs and weights ? HOT 14
- question about your paper HOT 1
- How to apply your framework on Android platform
- 如何实现Error Correction HOT 1
- 您的这篇工作十分有趣,请问有pytorch版本的实现吗? HOT 3
- Is QCNN suitable to fully convolution net(FCN)? HOT 2
- Program crashed right before ends: Double release buffer in ~CaffeEva::~CaffeEva(void) HOT 1
- The result is weird. HOT 17
- implement issues about error correction for fully connect layer ? HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quantized-cnn.