htqin / bibert Goto Github PK

Dear authors,

Great thanks for the opensource code. I wonder how you calculate the size of the models?
Actually, I have a very fundamental question the BERT-base model (fp32) is
embedding_param = 23835648
num_param = 85526016
(embedding_param + num_param) / 1e6 * 4 =437.4MB

instead of 418MB?

for n, p in model.items():
    if 'Norm' in n:
        continue
    if len(p.size()) ==2 :
        if 'embedding' in n:
            embedding_param += p.numel()
        else:
            num_param += p.numel()

It would be great if you can clarify this. (similar repohttps://github.com/huawei-noah/Pretrained-Language-Model/issues/184)

Hi Haotong,
Thanks for your excellent work! However, I am curious why the weights have scaling_factors while the activations don't in modules like QuantizeLinear ( in modeling_quant.py). Previous work like IR-NET has the same implementation.
Also, none of the operands calculates the scaling factor for matrix multiplication in the Attention module (such as attn@value). Why?

Looking forward to your reply!

Thanks for sharing the source code. I have a question about Bi-Attention structure.
When calculating the attention score * value, you binarize the attention score to 0 or 1 in the source code. In the paper, I noticed that you proposed a new bitwise operation to support computation between the binarized attention weight bool(A) and the binarized value during inference. But in the source code, I didn't find this part of the code.
Are the attention scores in the training and testing phases in the source code binarized to 0 and 1?

is there trick can solve the problem or it is a mistake?

Was Two Stage Knowledge Distillation used as in BinaryBERT in Table 7 (https://arxiv.org/pdf/2012.15701.pdf) to get these results?

您好，想问一个问题，论文里量化 1-1-1 模型的存储空间占用是13.4M，为什么我跑出来还是418M呢？怎样才能实现全1 bit位的模型，将模型最终的大小压缩到13.4M呢？

Hi, haotong, it is nice to see such a great work.

I am trying to re-implement BIBert with the code provided in the ICLR 2022 supplementary materials. I used the suggested DynaBert as teahcer model, and I gained the same teacher accuracy (32-bit baseline acc). However, I could not gain the same performance on GLUE with this code. For example, the acc is only 0.13/0.08 for STS-B, 0.833 for SST-2, which is far lower than the acc reported in table 2.

May you help me with this issue? Does the provided code in supplementary materials is already sufficient for re-implementation? Or does the acc-mismatch problem come from wrong hyper-parameter setting (I did not modify the code and I use the suggested dybnabert as teacher and pretrained model)?

Many Thanks!