saiwaiyanyu / bi-lstm-crf-ner-tf2.0

Named Entity Recognition (NER) task using Bi-LSTM-CRF model implemented in Tensorflow 2.0(tensorflow2.0 +)

bi-lstm-crf-ner-tf2.0

Named Entity Recognition (NER) task using Bi-LSTM-CRF model implemented in Tensorflow2.0.

Requirements

• python >3.6
• tensorflow==2.0.0
• tensorflow-addons==0.6.0

data

data example

１	B-TIME
９	I-TIME
９	I-TIME
７	I-TIME
年	E-TIME
，	O
是	O
中	B-LOC
国	E-LOC
发	O
展	O
历	O
史	O
上	O
非	O
常	O
重	O
要	O
的	O
很	O
不	O
平	O
凡	O
的	O
一	O
年	O
。	O
end

Usage

train

$ # pip install requirement.txt
$ python3 train.py

...
[-INFO-] 2019-12-05 21:11:15,037 24300 epoch   1, step 575, loss  5.0533 , accuracy --
[-INFO-] 2019-12-05 21:11:34,002 24300 epoch   1, step 576, loss  6.2023 , accuracy --
[-INFO-] 2019-12-05 21:11:52,543 24300 epoch   1, step 577, loss  4.3899 , accuracy --
[-INFO-] 2019-12-05 21:12:11,175 24300 epoch   1, step 578, loss  3.1313 , accuracy --
[-INFO-] 2019-12-05 21:12:29,661 24300 epoch   1, step 579, loss  6.4625 , accuracy --
[-INFO-] 2019-12-05 21:12:48,233 24300 epoch   1, step 580, loss  5.5159 , accuracy --
[-INFO-] 2019-12-05 21:12:48,325 24300 model saved
...

predict

$ python3 predict.py


input: ****总书记、国家主席***发表１９９８年新年讲话

[
    {
        "end": 4,
        "words": "****",
        "type": "ORG",
        "begin": 1
    },
    {
        "end": 15,
        "words": "***",
        "type": "PER",
        "begin": 13
    },
    {
        "end": 22,
        "words": "１９９８年",
        "type": "TIME",
        "begin": 18
    }
]

bi-lstm-crf-ner-tf2.0's Issues

报错 ValueError: tf.function-decorated function tried to create variables on non-first call.

ValueError: tf.function-decorated function tried to create variables on non-first call.

trainer error

总是报这个错，很奇怪
Traceback (most recent call last):
File "/Applications/anaconda3/envs/ner_tf2/lib/python3.6/logging/config.py", line 565, in configure
handler = self.configure_handler(handlers[name])
File "/Applications/anaconda3/envs/ner_tf2/lib/python3.6/logging/config.py", line 738, in configure_handler
result = factory(**kwargs)
File "/Applications/anaconda3/envs/ner_tf2/lib/python3.6/logging/handlers.py", line 202, in init
BaseRotatingHandler.init(self, filename, 'a', encoding, delay)
File "/Applications/anaconda3/envs/ner_tf2/lib/python3.6/logging/handlers.py", line 57, in init
logging.FileHandler.init(self, filename, mode, encoding, delay)
File "/Applications/anaconda3/envs/ner_tf2/lib/python3.6/logging/init.py", line 1032, in init
StreamHandler.init(self, self._open())
File "/Applications/anaconda3/envs/ner_tf2/lib/python3.6/logging/init.py", line 1061, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/shf/PycharmProjects/bi-lstm-crf-ner-tf2.0/log/error.log'

Why you didn't use model.fit?

I noticed that you implemented the training process by your customer functions train_one_step and evaluate_one_step. Is there a particular reason that you did not use model.fit?

修改hidden_num为1024或者256都会报错tensor 不匹配

hidden_num从512修改为1024后,
报错:
[-INFO-] 2021-01-21 16:53:14,307 20534 train. line:32 hidden_num:1024, vocab_size:4733, label_size:14
Traceback (most recent call last):
File "train.py", line 77, in
loss, logits, text_lens = train_one_step(text_batch, labels_batch)
File "train.py", line 47, in train_one_step
logits, text_lens, log_likelihood = model(text_batch, labels_batch,training=True)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in call
outputs = self.call(cast_inputs, *args, **kwargs)
File "/mnt/python/tensorflow2/bi-lstm-crf-ner-tf2.0/model.py", line 35, in call
logits = self.dense(self.biLSTM(inputs))
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/layers/wrappers.py", line 531, in call
return super(Bidirectional, self).call(inputs, **kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 964, in call
self._maybe_build(inputs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2416, in _maybe_build
self.build(input_shapes) # pylint:disable=not-callable
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/layers/wrappers.py", line 685, in build
self.forward_layer.build(input_shape)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/layers/recurrent.py", line 573, in build
self.cell.build(step_input_shape)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 316, in wrapper
output_shape = fn(instance, input_shape)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2324, in build
caching_device=default_caching_device)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 577, in add_weight
caching_device=caching_device)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 724, in _add_variable_with_custom_getter
name=name, shape=shape)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 791, in _preload_simple_restoration
checkpoint_position=checkpoint_position, shape=shape)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 75, in init
self.wrapped_value.set_shape(shape)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1107, in set_shape
(self.shape, shape))
ValueError: Tensor's shape (300, 2048) is not compatible with supplied shape (300, 4096)

这个输出维度,应该是可以自己定义的吧?

predict 不能进行预测，求大佬讲解一下

这是输出的日志记录，代码没有动过
input:我是范冰冰 dataset: [[ 131 2067 3239 2234 2234]] logits [[[-0.72535753 4.161972 -2.9507422 -3.1868236 -3.2397275 -3.269389 -3.0774546 -3.2384624 -3.2812405 -2.9897215 -3.1217413 -3.6321018 -3.1315987 -3.0792222 ] [-2.8050935 4.9421215 -2.3537152 -2.844575 -2.8227744 -2.926732 -2.4319122 -2.8924425 -2.7267509 -2.4868493 -2.7142 -3.325798 -2.5980093 -2.7135816 ] [-0.40876862 1.085415 -1.0298109 -1.1841301 -1.3309526 -1.5018826 -1.0938866 -1.1450801 -1.3572048 -1.1883298 -1.3080059 -1.4665012 -1.1049844 -1.3270234 ] [ 0.03211104 0.3558294 -0.6737744 -0.6955142 -0.8401693 -0.9766958 -0.69395465 -0.74898857 -0.8396182 -0.77023965 -0.86173874 -0.88923264 -0.68174887 -0.8035339 ] [ 0.13178386 -0.05146006 -0.43199152 -0.3135421 -0.44383863 -0.5618567 -0.37870246 -0.36361524 -0.4879145 -0.49257016 -0.50506824 -0.50199145 -0.37561214 -0.4440806 ]]] text_lens [5] viterbi_path,va: [1, 1, 1, 1, 1] 15.173229 paths: [[1, 1, 1, 1, 1]] [id2tag[id] for id in paths[0]]: ['O', 'O', 'O', 'O', 'O'] entities_result: [] json_result: []

why does loss have negative numbers

I use my own data

[-INFO-] 2020-08-18 17:03:03,229 31422  train.<module> line:80  epoch 22, step 880, loss -102.3357 , accuracy 0.8097
[-INFO-] 2020-08-18 17:03:08,052 31422  train.<module> line:80  epoch 23, step 900, loss -55.5771 , accuracy 0.8884
[-INFO-] 2020-08-18 17:03:12,845 31422  train.<module> line:80  epoch 23, step 920, loss -109.6482 , accuracy 0.8704
[-INFO-] 2020-08-18 17:03:17,668 31422  train.<module> line:80  epoch 24, step 940, loss 6.0771 , accuracy 0.9116
[-INFO-] 2020-08-18 17:03:22,638 31422  train.<module> line:80  epoch 24, step 960, loss -25.0925 , accuracy 0.9362
[-INFO-] 2020-08-18 17:03:27,504 31422  train.<module> line:80  epoch 25, step 980, loss 3.5970 , accuracy 0.9642
[-INFO-] 2020-08-18 17:03:32,433 31422  train.<module> line:80  epoch 25, step 1000, loss -180.6551 , accuracy 0.8611
[-INFO-] 2020-08-18 17:03:37,352 31422  train.<module> line:80  epoch 26, step 1020, loss -185.5950 , accuracy 0.9019
[-INFO-] 2020-08-18 17:03:42,195 31422  train.<module> line:80  epoch 26, step 1040, loss -26.1356 , accuracy 0.9035
[-INFO-] 2020-08-18 17:03:47,002 31422  train.<module> line:80  epoch 27, step 1060, loss -60.3808 , accuracy 0.9388
[-INFO-] 2020-08-18 17:03:52,013 31422  train.<module> line:80  epoch 27, step 1080, loss -368.6034 , accuracy 0.7799
[-INFO-] 2020-08-18 17:03:56,875 31422  train.<module> line:80  epoch 28, step 1100, loss -136.8763 , accuracy 0.8971
[-INFO-] 2020-08-18 17:04:01,815 31422  train.<module> line:80  epoch 28, step 1120, loss 4.8511 , accuracy 0.9637
[-INFO-] 2020-08-18 17:04:06,698 31422  train.<module> line:80  epoch 29, step 1140, loss -145.3242 , accuracy 0.8888
[-INFO-] 2020-08-18 17:04:11,561 31422  train.<module> line:80  epoch 29, step 1160, loss -140.1508 , accuracy 0.9304
[-INFO-] 2020-08-18 17:04:16,441 31422  train.<module> line:80  epoch 30, step 1180, loss -191.4668 , accuracy 0.9232
[-INFO-] 2020-08-18 17:04:21,178 31422  train.<module> line:80  epoch 30, step 1200, loss -193.8787 , accuracy 0.8962
[-INFO-] 2020-08-18 17:04:26,084 31422  train.<module> line:80  epoch 31, step 1220, loss -158.3943 , accuracy 0.9505
[-INFO-] 2020-08-18 17:04:30,918 31422  train.<module> line:80  epoch 31, step 1240, loss -118.8575 , accuracy 0.8672
[-INFO-] 2020-08-18 17:04:35,754 31422  train.<module> line:80  epoch 32, step 1260, loss -367.0381 , accuracy 0.7739
[-INFO-] 2020-08-18 17:04:40,663 31422  train.<module> line:80  epoch 32, step 1280, loss 4.8814 , accuracy 0.9808
[-INFO-] 2020-08-18 17:04:40,716 31422  train.<module> line:84  model saved
[-INFO-] 2020-08-18 17:04:45,581 31422  train.<module> line:80  epoch 33, step 1300, loss -83.2353 , accuracy 0.9721
[-INFO-] 2020-08-18 17:04:50,272 31422  train.<module> line:80  epoch 33, step 1320, loss -386.9019 , accuracy 0.7725
[-INFO-] 2020-08-18 17:04:55,229 31422  train.<module> line:80  epoch 34, step 1340, loss -42.3887 , accuracy 0.9648
[-INFO-] 2020-08-18 17:05:00,131 31422  train.<module> line:80  epoch 34, step 1360, loss 4.1282 , accuracy 0.9734
[-INFO-] 2020-08-18 17:05:05,001 31422  train.<module> line:80  epoch 35, step 1380, loss -139.4714 , accuracy 0.9339
[-INFO-] 2020-08-18 17:05:09,776 31422  train.<module> line:80  epoch 35, step 1400, loss 2.0760 , accuracy 0.9880
[-INFO-] 2020-08-18 17:05:09,833 31422  train.<module> line:84  model saved
[-INFO-] 2020-08-18 17:05:14,682 31422  train.<module> line:80  epoch 36, step 1420, loss -151.9778 , accuracy 0.9138
[-INFO-] 2020-08-18 17:05:19,641 31422  train.<module> line:80  epoch 36, step 1440, loss -94.4556 , accuracy 0.8666
[-INFO-] 2020-08-18 17:05:24,617 31422  train.<module> line:80  epoch 37, step 1460, loss -201.3645 , accuracy 0.9187
[-INFO-] 2020-08-18 17:05:29,423 31422  train.<module> line:80  epoch 37, step 1480, loss -142.1119 , accuracy 0.9106
[-INFO-] 2020-08-18 17:05:34,226 31422  train.<module> line:80  epoch 38, step 1500, loss -349.6097 , accuracy 0.8306
[-INFO-] 2020-08-18 17:05:39,113 31422  train.<module> line:80  epoch 38, step 1520, loss 2.9786 , accuracy 0.9374

error in tf 2.3 and tfa 11.2

i tried on my environment with tensorflow 2.3 and tensorflow-addons 11.2, encountered exceptions. do you have plan to upgrade the code?

screenshot pasted below:
: Invalid argument: slice index 0 of dimension 0 out of bounds.
Traceback (most recent call last):
File "train.py", line 79, in
accuracy = get_acc_one_step(logits, text_lens, labels_batch)
File "train.py", line 59, in get_acc_one_step
viterbi_path, _ = tf_ad.text.viterbi_decode(logit[:text_len], model.transition_params)
File "C:\Python37\lib\site-packages\tensorflow_addons\text\crf.py", line 362, in viterbi_decode
trellis[0] = score[0]
File "C:\Python37\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "C:\Python37\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1024, in _slice_helper
name=name)
File "C:\Python37\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "C:\Python37\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1196, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "C:\Python37\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 10320, in strided_slice
_ops.raise_from_not_ok_status(e, name)
File "C:\Python37\lib\site-packages\tensorflow\python\framework\ops.py", line 6843, in raise_from_not_ok_status

six.raise_from(core._status_to_exception(e.code, message), None)

File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 0 of dimension 0 out of bounds. [Op:StridedSlice] name: strided_slice/

thanks
sanyorke

这样应该没有办法训练transition_params吧？最后输出的transition_params是不合理的。

为什么修改hidden_num会报维度不匹配的错误？请问如何修改这些超参数

test.txt在划水啊

train.py里面在线评估没用到
predict.py里面离线预测也没用到
得用用吧～

test.txt并未使用

train.py 训练model的时候，text.txt应该用来在线评估的，
但是好像并未使用这个测试集，这是为了加快模型的训练速度吗？

model.py里面transition_params的trainable状态是否应该改为True

否则转移状态矩阵怎么更新呢

predict 提示使用tf.function

tensorflow:7 out of the last 7 calls to <function _make_execution_function..distributed_function at 0x0000019E6F4A59D8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

但是我使用@tf.function的时候，提示 tnesor 没有numpy

感觉只用acc作为指标有点问题

这种分类问题，使用precision 和 recall 作为 metrics 会比较好；假如训练数据中绝大多数都是O，那么acc一开始就会很高，不能准确反应模型的能力啊

请问为什么accuracy = 0.5？

现在pip无法下载addons0.6.0，使用最新addons+tf2.0.0出现错误

Traceback (most recent call last):
File "C:/Users/87124/PycharmProjects/bilstm-crf-ner-tf2.0/model.py", line 8, in
import tensorflow_addons as tf_ad
File "C:\Users\87124.conda\envs\tensorflow2.0.0\lib\site-packages\tensorflow_addons_init_.py", line 21, in
from tensorflow_addons import activations
File "C:\Users\87124.conda\envs\tensorflow2.0.0\lib\site-packages\tensorflow_addons\activations_init_.py", line 17, in
from tensorflow_addons.activations.gelu import gelu
File "C:\Users\87124.conda\envs\tensorflow2.0.0\lib\site-packages\tensorflow_addons\activations\gelu.py", line 26, in
@tf.keras.utils.register_keras_serializable(package="Addons")
AttributeError: module 'tensorflow_core.keras.utils' has no attribute 'register_keras_serializable'

accuracy 1.0

is problem at calculate loss
debugged accuracy ,feeling it's ok