dialogue-utterance-rewriter's People
Forkers
auscenery henryflee luciencho derekkk everwind simons2017 zhangyanbo2007 colinsongf haojiepan1 vincentyua katherinelyx itjune chapzq77 shenyong123 siviltaram linxiyao caoxu915683474 cosecant-csc adamszq mandyyang1989 youly172 nonstopfor zhuoerfeng leichen9 qsong4 xrosliang sumhncku paopao6 gaoyiyeah gdh756462786 luomuqinghan rileyshe xinbingzhe littlehappi sz128 smj0 neoql lianzhaoy yale1417 wxc1884 bellyfat fancyerii 520jefferson cxjtju zzw-mathdialogue-utterance-rewriter's Issues
Where is Lambda in the paper ?
Code
请问代码要什么时候公开呢
no data
Hello, I try my best but I can not find the data, would you mind share the data again?
Code?
When is the code going to be available?
已复现,但指标有些对不上,问题请教下
你好,
冒昧的打扰您了,我正在做多轮对话中的改写问题,阅读了您的“Improving Multi-turn Dialogue Modelling with Utterance ReWriter”,深有启发。下载了您的代码,并跑通了结果。但有几个问题想请教您。
1. 您是完全基于github中给出的两万条正样本做训练和预测吗
2. 您的负样本(不需要改写的)是怎么构造的呢,我是通过将正样本中的summarization替换掉current,来构造负样本的,不知道可行不
3. 您训练集和验证集数据有多少呢。我是按照1比3划分的,训练集正负样本共3万条,评测集正负样本共1万条,正负样本比例1:1
4. 您训练集和验证集正负样本比例是1:1吗
5. 我基于pointer-network(LSTM的base网络,没使用coverage机制)的实验结果为,
bleu1: 0.836, bleu2: 0.807, bleu4: 0.746,em_score: 0.449 rouge_1: 0.915, rouge_2: 0.840, rouge_l: 0.865
除了em(exact match)和您给出数据差不多之外,其他两个好得有点离谱了,想请教下
1)您的bleu和rouge怎么计算的呢,不知道是不是我们计算方式不一致。我的计算代码贴在后面
from nltk.translate.bleu_score import sentence_bleu
from nltk.translate.bleu_score import SmoothingFunction
from rouge import Rouge
smoothing_function = SmoothingFunction().method4
class Metrics(object):
def __init__(self):
pass
@staticmethod
def bleu_score(references, candidates):
"""
计算bleu值
:param references: 实际值, list of string
:param candidates: 验证值, list of string
:return:
"""
# 遍历计算bleu
bleu1s = []
bleu2s = []
bleu3s = []
bleu4s = []
for ref, cand in zip(references, candidates):
ref_list = [list(ref)]
cand_list = list(cand)
bleu1 = sentence_bleu(ref_list, cand_list, weights=(1, 0, 0, 0), smoothing_function=smoothing_function)
bleu2 = sentence_bleu(ref_list, cand_list, weights=(0.5, 0.5, 0, 0), smoothing_function=smoothing_function)
bleu3 = sentence_bleu(ref_list, cand_list, weights=(0.33, 0.33, 0.33, 0), smoothing_function=smoothing_function)
bleu4 = sentence_bleu(ref_list, cand_list, weights=(0.25, 0.25, 0.25, 0.25), smoothing_function=smoothing_function)
# print ("ref: %s, cand: %s, bleus: %.3f, %.3f, %.3f, %.3f"
# % (ref, cand, bleu1, bleu2, bleu3, bleu4))
bleu1s.append(bleu1)
bleu2s.append(bleu2)
bleu3s.append(bleu3)
bleu4s.append(bleu4)
# 计算平均值
bleu1_average = sum(bleu1s) / len(bleu1s)
bleu2_average = sum(bleu2s) / len(bleu2s)
bleu3_average = sum(bleu3s) / len(bleu3s)
bleu4_average = sum(bleu4s) / len(bleu4s)
# 输出
print "average bleus: bleu1: %.3f, bleu2: %.3f, bleu4: %.3f" % (bleu1_average, bleu2_average, bleu4_average)
return (bleu1_average, bleu2_average, bleu4_average)
@staticmethod
def em_score(references, candidates):
total_cnt = len(references)
match_cnt = 0
for ref, cand in zip(references, candidates):
if ref == cand:
match_cnt = match_cnt + 1
em_score = match_cnt / (float)(total_cnt)
print "em_score: %.3f, match_cnt: %d, total_cnt: %d" % (em_score, match_cnt, total_cnt)
return em_score
@staticmethod
def rouge_score(references, candidates):
"""
rouge计算,NLG任务语句生成,词语的recall
https://github.com/pltrdy/rouge
:param references: list string
:param candidates: list string
:return:
"""
rouge = Rouge()
# 遍历计算rouge
rouge1s = []
rouge2s = []
rougels = []
for ref, cand in zip(references, candidates):
ref = ' '.join(list(ref))
cand = ' '.join(list(cand))
rouge_score = rouge.get_scores(cand, ref)
rouge_1 = rouge_score[0]["rouge-1"]['f']
rouge_2 = rouge_score[0]["rouge-2"]['f']
rouge_l = rouge_score[0]["rouge-l"]['f']
# print "ref: %s, cand: %s" % (ref, cand)
# print 'rouge_score: %s' % rouge_score
rouge1s.append(rouge_1)
rouge2s.append(rouge_2)
rougels.append(rouge_l)
# 计算平均值
rouge1_average = sum(rouge1s) / len(rouge1s)
rouge2_average = sum(rouge2s) / len(rouge2s)
rougel_average = sum(rougels) / len(rougels)
# 输出
print "average rouges, rouge_1: %.3f, rouge_2: %.3f, rouge_l: %.3f" \
% (rouge1_average, rouge2_average, rougel_average)
return (rouge1_average, rouge2_average, rougel_average)
if __name__ == '__main__':
references = ["腊八粥喝了吗", "我的机器人女友好好看啊", "那长沙明天天气呢"]
candidates = ["腊八粥喝了吗", "机器人女友好好看啊", '长沙明天呢']
# decode
references = [ref.decode('utf-8') for ref in references]
candidates = [cand.decode('utf-8') for cand in candidates]
# 计算metrics
# Metrics.bleu_score(references, candidates)
# Metrics.em_score(references, candidates)
Metrics.rouge_score(references, candidates)
corpus.txt语料中有部分问题
那你认识张琪吗 额是的 她长什么样 A
诗乃怎么样 诗乃是不错 哪里不错 A
哆啦a梦 蓝梦岛金银岛 是新出来的电影 哆啦a梦
下面这句不知道是不是有问题:
为啥遇到什么事了 分手了 安啦你一定会找到那个对的人的 分手了很难过
code?
ACL不会核实有没有代码的吗?留了一个代码链接,却没有代码,第一次遇见,作者什么时候能公布代码?能给个时间吗?
代码很多问题
基于LSTM其实也也无所谓的
(1)数据没有负样本;
(2)代码问题太多,跑不通;
(3)即使改的能跑通后,发现数据格式和代码中写的对不上。
最新数据集上的效果指标能否贴一下
如题,由于数据集和论文中不一样,重现模型的效果没法对比,可以把最新数据集上的效果报告给出来吗?
这和论文里说的不一样啊,基于transformer的呢?
给了个阉割版的?
Cannot decode as normal
Hi, the authors of Dialogue Rewrite. I'm trying to reproduce your results on the latest dataset, but I'm stuck in the decoding phase when predicting the final results. I'm wondering why the decoding max dec step should by set to1? If not so, the code will raise AssertException; Otherwise, the result will only produce 1 word in the result.txt
.
Looking forward to your response.
Best
code
8月21日晴
依旧没有等到code,伤心
No data vocab.txt
no data about vocab.txt
数据和代码呢?
论文不是说马上就会公开吗?
关于负样本的数据
您好,非常感谢提供代码和正样本的数据!我只用正样本跑了一下,所以推断的时候基本每句话都会修改。所以能否提供另外的2万负样本以供复现和研究?再次感谢!
Dataset
I'm looking for a similar dataset as yours in English. Do you have such information?
Thank you
为什么LSTM和transformer的效果差这么多?
Transformer是将上文和当前query拼接后进行编码,而LSTM是将两者分别编码,这样比较是不是不太合适?有没有尝试过将拼接后的句子用LSTM编码?个人感觉这种简短的句子,性能差别不会这么大
Transformer based training details
Thanks for your paper and the code released. I am confused with some details about transformer-based training:
- Did you try using the pre-trained language model to initialize encoder or decoder or just replace lstm with tranformer framework? Hints were in the code but not in the paper.
- What are the missing hparams when you train transformer-based model? such like warmup rate, decay rate?
- My summarization with LSTM without turn embedding, always decodes duplicated words, probably due to some word are more "important", does turn embedding play an important role in this problem?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.