luozhouyang / deepse

Sentence Embeddings using Deep Nerual Networks in PRODUCTION!

ZhouYang Luo

Deep Learning and NLP enthusiast.

deepse's Issues

ValueError: operands could not be broadcast together with shapes

环境

硬件环境：GPU P100 、linux
软件环境一：TF 2.7 、 transformer-keras 0.4.9
软件环境二：TF2.4 ,trainsformer-keras 0.3.1

数据集

• 大小：训练集2048条，验证集512条
• 格式：unsup格式

超参：

• batch_size ：[4,32,64,80]均无效
• epoch:1
• mode:unsup

现象：

1、数据正常读取，bert能加载，但是在model.fit报错了

debug信息如下
2021-11-10 10:39:52,766 WARNING utils.py 80] Gradients do not exist for variables ['bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?
975/Unknown - 398s 378ms/step - acc: 0.9998 - loss: 0.0051Traceback (most recent call last):
File "run_simcse.py", line 79, in
save_weights_only=False)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/keras/utils/generic_utils.py", line 896, in update
self._values[k][0] += v * value_base
ValueError: operands could not be broadcast together with shapes (64,) (27,) (64,)

我尝试方法

更改数据集大小为batch_size的整数倍，无法解决
davidADSP/GDL_code#63

损失函数需要再求一个batch平均值

`loss = tf.keras.losses.sparse_categorical_crossentropy(labels, cosine, from_logits=True)`    

DeepSE/deepse/simcse.py

Line 109 in a7c47c5

loss = tf.keras.losses.sparse_categorical_crossentropy(labels, cosine, from_logits=True)

这里的loss实际上是对所有的损失求和，需要再做一个reduce_mean操作

麻烦那个small.txt能提供一下吗？