jcyk / seqgen

Source code for our EMNLP19 paper "Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework"

seqgen

a retrieval-enhanced sequence generation framework

Source code for our EMNLP19 paper "Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework"

Our dataset is released here under the name Retrieval Generation Chat.

download link:Retrieval_Generation_Chat.zip

code is tested with python==3.6.8 and torch==1.2.0

Run Demo

run_demo.sh with details explained in deploy.py. you can query the demo in the following ways:

1. query + retrievals => skeletons + responses http://0.0.0.0:8080/query_retrievals?query=q&retrievals=r1;;;r2;;;r3 where q is a single query, r1, r2 and r3 are multiple retrieval responses (also support single retrieval response), seperated by ;;;.

   return format: {'skeletons':[s1, s2, s3], "responses":[re1, re2, re3]}

2. query + responses => ranks http://0.0.0.0:8080//query_responses?query=q&responses=r1;;;r2;;;r3 where q is a single query, r1, r2 and r3 are multiple response candidates seperated by ;;;.

   return format: {'rank':[ra1, ra2, ra3]} correspond to teh rank of r1, r2, r3 (start from 0)

3. query + skeleton => response http://0.0.0.0:8080/query_skeleton?query=q&skeleton=s where q is a single query, s is a single skeleton, the blanks in the skeleton is indicated by ;;; , e.g., 今天(today);;;上课(in class);;;表扬(praise);;;.

   return format: {'respones':r}

Note the demo requires pretrianed neural masker, generator, and ranker. To obtain your own models, please refer to the following instructions.

Train from scratch

Prepare Data and Vocab

1. preprocess_zh/segment.py --train_file_path train.txt this will result in a preprocessed data (train.txt_processed) file and a vocab file (vocab).
2. In our experiments, we use the same vocab for both query and response sides (i.e., vocab_src = vocab_tgt = vocab).
3. We provide the stopwords list used in our experiments in data/stopword

Train and Test

(You should check every shell scripts for data path config. The default hyper-parameter settings in this repo are used in our experiments.)

1. Masker ranker/train_masker.sh and ranker/generate_skeleton.sh

2. Generator train.sh and work.sh

3. Ranker ranker/train_ranker.sh and ranker/score.sh

Citation

If you find the code useful, please cite our paper.

@inproceedings{cai-etal-2019-retrieval,
    title = "Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework",
    author = "Cai, Deng  and
      Wang, Yan  and
      Bi, Wei  and
      Tu, Zhaopeng  and
      Liu, Xiaojiang  and
      Shi, Shuming",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1195",
    doi = "10.18653/v1/D19-1195",
    pages = "1866--1875",
}

Contact

For any questions, please drop an email to Deng Cai.

seqgen's Issues

UnicodeDecodeError when Prepare Data and Vocab

There is UnicodeDecodeError when I run the segment.py;
Actually, I find this error is caused by the data and the code is ok. For example, see the line 87.
And there are valid lines:5495620, error lines:12109, total lines:5507729.

Please check the dataset.

File "preprocess_zh/segment.py", line 119, in
lines = [ x.decode('utf8') for x in open(where).readlines() ]
File "/home/user/xxxx/anaconda3/envs/tf2sks/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 3696: invalid continuation byte

lines = []
err_line_ids = []
with open(where, "rb") as fp:
    for ii, x in enumerate(fp, 1):
        try:
            lines.append(x.decode('utf8'))
        except:
            err_line_ids.append(ii)
            # pdb.set_trace()    
print("valid lines:{}, error lines:{}, total lines:{}".format(len(lines), len(err_line_ids), len(lines)+len(err_line_ids)))

Code questions

question1: ranker/data.py中的第67行 tgt = ListsToTensor([x[1] for x in data], vocab_src), 对tgt的序列号为什么使用src的vocab呢?
question2:文献1689页的第一段表明训练Skeleton-guided Response Generator时,对ground-truth response的若干随机操作得到Skeleton. 对应的代码是不是data.py(不是ranker文件夹中的)? data.py是对ref_tgt进行随机操作的,ref_tgt不是检索得到的回复么?
谢谢指导!

Problems with data sets and loss functions

Hello, I have some problems when studying the code:

1. The difference between the two data sets, when to use train.txt and when to use train.retrieval. txt.
2. There are two losses in masker_ranker.py. What is the function of baseline_mse loss?It doesn't seem to be reflected in your paper.

train.txt 与模型输入不匹配

根据readme里的说法，模型使用的是train.txt，而其他模型使用的是train.retrieval.txt。
但是train.txt 每一行是query-response的二元组，在你的代码中默认输入格式都是四元组，这导致不能载入train.txt里的数据。
请问应该怎么解决？

can not find some files

There is no vocab_src, vocab_tgt, and stop_words_file. Can you provide them, at least stop_words_file. Thank you very much.