alibaba-edu / simple-effective-text-matching

Source code of the ACL2019 paper "Simple and Effective Text Matching with Richer Alignment Features".

RE2

This is the original Tensorflow implementation of the ACL 2019 paper Simple and Effective Text Matching with Richer Alignment Features. Pytorch implementation: https://github.com/alibaba-edu/simple-effective-text-matching-pytorch.

Quick Links

• About
• Setup
• Usage

Simple and Effective Text Matching

RE2 is a fast and strong neural architecture for general purpose text matching applications. In a text matching task, a model takes two text sequences as input and predicts their relationship. This method aims to explore what is sufficient for strong performance in these tasks. It simplifies or omits many slow components which are previously considered as core building blocks in text matching. It achieves its performance by a simple idea, which is keeping three key features directly available for inter-sequence alignment and fusion: previous aligned features (Residual vectors), original point-wise features (Embedding vectors), and contextual features (Encoder output).

RE2 achieves performance on par with the state of the art on four benchmark datasets: SNLI, SciTail, Quora and WikiQA, across tasks of natural language inference, paraphrase identification and answer selection with no or few task-specific adaptations. It has at least 6 times faster inference speed compared with similarly performed models.

The following table lists major experiment results. The paper reports the average and standard deviation of 10 runs and the results can be easily reproduced. Inference time (in seconds) is measured by processing a batch of 8 pairs of length 20 on Intel i7 CPUs. The computation time of POS features used by CSRAN and DIIN is not included.

Model	SNLI	SciTail	Quora	WikiQA	Inference Time
BiMPM	86.9	-	88.2	0.731	0.05
ESIM	88.0	70.6	-	-	-
DIIN	88.0	-	89.1	-	1.79
CSRAN	88.7	86.7	89.2	-	0.28
RE2	88.9±0.1	86.0±0.6	89.2±0.2	0.7618 ±0.0040	0.03~0.05

Refer to the paper for more details of the components and experiment results.

Setup

• install python >= 3.6 and pip
• pip install -r requirements.txt
• install Tensorflow 1.4 or above (the wheel file for Tensorflow 1.4 gpu version under python 3.6 can be found here)
• Download GloVe word vectors (glove.840B.300d) to resources/

Data used in the paper are prepared as follows:

SNLI

• Download and unzip SNLI (pre-processed by Tay et al.) to data/orig.
• Unzip all zip files in the "data/orig/SNLI" folder. (cd data/orig/SNLI && gunzip *.gz)
• cd data && python prepare_snli.py

SciTail

• Download and unzip SciTail dataset to data/orig.
• cd data && python prepare_scitail.py

Quora

• Download and unzip Quora dataset (pre-processed by Wang et al.) to data/orig.
• cd data && python prepare_quora.py

WikiQA

• Download and unzip WikiQA to data/orig.
• cd data && python prepare_wikiqa.py
• Download and unzip evaluation scripts. Use the make -B command to compile the source files in qg-emnlp07-data/eval/trec_eval-8.0. Move the binary file "trec_eval" to resources/.

Usage

To train a new text matching model, run the following command:

python train.py $config_file.json5

Example configuration files are provided in configs/:

• configs/main.json5: replicate the main experiment result in the paper.
• configs/robustness.json5: robustness checks
• configs/ablation.json5: ablation study

The instructions to write your own configuration files:

[
    {
        name: 'exp1', // name of your experiment, can be the same across different data
        __parents__: [
            'default', // always put the default on top
            'data/quora', // data specific configurations in `configs/data`
            // 'debug', // use "debug" to quick debug your code  
        ],
        __repeat__: 5,  // how may repetitions you want
        blocks: 3, // other configurations for this experiment 
    },
    // multiple configurations are executed sequentially
    {
        name: 'exp2', // results under the same name will be overwritten
        __parents__: [
            'default', 
            'data/quora',
        ],
        __repeat__: 5,  
        blocks: 4, 
    }
]

To check the configurations only, use

python train.py $config_file.json5 --dry

Citation

Please cite the ACL paper if you use RE2 in your work:

@inproceedings{yang2019simple,
  title={Simple and Effective Text Matching with Richer Alignment Features},
  author={Yang, Runqi and Zhang, Jianhai and Gao, Xing and Ji, Feng and Chen, Haiqing},
  booktitle={Association for Computational Linguistics (ACL)},
  year={2019}
}

License

RE2 is under Apache License 2.0.

simple-effective-text-matching's Issues

Training model on Quora dataset

Hi all,

I ran into an OOM error while trying to train the model, on a 1-GPU workstation (GTX 1080TI). A snip of the error log can be seen below. Has anyone been successful in training this model with similar hardware?
Thanks in advance!

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[512,98,200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: block-1/align/sub_1/mul_5 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](block-1/align/sub_1/mul_2, block-1/align/sub_1/add_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[Node: stack_8/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6954_stack_8", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Edit: Decreasing batch size to 64 (default was 512) fixed this issue. I will try to see what's the maximum batch size that the 1080TI can handle.

ValueError

hello, I run the train.py. But a mistake has occurred at map_, mrr = [float(s[-6:]) for s in stdout.strip().split('\n')]. ValueError:could not convert string to float.

关于evaluate方式的问题

您好，代码中使用的evaluate的方式我理解是通过加载回参数，重新实例化了一个model对象进行的。请问为什么不用restore或是读取meta保存的图结构的方式呢？

assert gnorm >= 0, 'encounter nan in gradients

Hello

I get the following error when I try to train the model with my own training data.
What could be the error

Question about the code of "Augmented Residual Connections"

如果block数量大于3，比如代码实现第四个block时encoder的输入是前三个block输出之和，而不是公式中的前两个之和。

sqrt在0处不可导，不用添加一个eps保持数值稳定吗？

simple-effective-text-matching/src/modules/__init__.py

Line 35 in 0a42712

w_norm = tf.sqrt(tf.reduce_sum(tf.square(w), axis=list(range(len(shape) - 1)), keep_dims=True))

w_norm = tf.sqrt(tf.maximum(tf.reduce_sum(tf.square(w), axis=list(range(len(shape) - 1)), keep_dims=True), eps)) 

Can you show me the format of model_path, the specific model file like the python evaluate.py model_path data_file

Can you show me the format of model_path, the specific model file like the python evaluate.py model_path datafile. I always get the error ‘tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for ./models/quora/benchmark-4/checkpoint-best’. What is the suffix of the model file

How I can Obtain Text Vectors

Hello

How i can get left and right text vectors before concatenation.

self.prediction(a, b, dropout_keep_prob)
I want to get "a" and "b" --> vectors

def __call__(self, a, b, dropout_keep_prob, name='prediction'):
      x = self._features(a, b)
      with tf.variable_scope(name):
          x = tf.nn.dropout(x, dropout_keep_prob)
          x = dense(x, self.args.hidden_size, activation=tf.nn.relu, name='dense_1')
          x = tf.nn.dropout(x, dropout_keep_prob)
          x = dense(x, self.args.num_classes, activation=None, name='dense_2')
          return x

If I change this code to return directly a and b is that enough
Or is there another way without of changing any code

the accuracy

The accuracy I got is far from the accuracy provided in the paper. Maybe the parameters have not been adjusted well? Thank you for your answer！

MAP MRR

Wikiqa数据集，计算ＭＡＰ时出错了：
Traceback (most recent call last):tes 10 loss: 0.2441 lr: 0.0010 gnorm: 0.1420
File "train.py", line 48, in
main()
File "train.py", line 31, in main
states = trainer.train()
File "/media/xuezzz/xueee/simple-effective-text-matching-master/src/trainer.py", line 66, in train
score, dev_stats = model.evaluate(sess, dev_batches)
File "/media/xuezzz/xueee/simple-effective-text-matching-master/src/model.py", line 187, in evaluate
stats.update(metricsmetric)
File "/media/xuezzz/xueee/simple-effective-text-matching-master/src/utils/metrics.py", line 80, in ranking
map_, mrr = [float(s[-6:]) for s in stdout.strip().split('\n')]
File "/media/xuezzz/xueee/simple-effective-text-matching-master/src/utils/metrics.py", line 80, in
map_, mrr = [float(s[-6:]) for s in stdout.strip().split('\n')]
ValueError: could not convert string to float:
你知道是哪的问题嘛？谢谢！