The deepnlg from thiagocf05

thiagocf05 / deepnlg Goto Github PK

View Code? Open in Web Editor NEW

A systematic comparison between pipeline and end-to-end architectures in the RDF-to-text task

Python 1.03% PLSQL 81.64% Jupyter Notebook 0.63% Lex 16.56% Shell 0.15%

deepnlg's People

Contributors

Stargazers

Watchers

deepnlg's Issues

Why reference length differ?

Hi, thanks for sharing the work :) When I check the evaluation part, I found the refer_len differ with different methods. In my opinion, the ref_len should be the same, I wonder what makes them different here?

Thanks!

Final: 
Set:  test
Approach: pipeline
Model:  rand
b'BLEU = 41.68, 76.8/51.1/34.0/22.6 (BP=1.000, ratio=1.133, hyp_len=19903, ref_len=17563)\n'
----------
Final: 
Set:  test
Approach: pipeline
Model:  major
b'BLEU = 43.82, 77.1/52.9/36.3/24.9 (BP=1.000, ratio=1.071, hyp_len=33747, ref_len=31517)\n'
----------
Final: 
Set:  test
Approach: pipeline
Model:  transformer
b'BLEU = 51.68, 83.0/59.5/44.0/32.8 (BP=1.000, ratio=1.026, hyp_len=30942, ref_len=30160)\n'
----------
Final: 
Set:  test
Approach: pipeline
Model:  rnn
b'BLEU = 50.55, 81.3/58.3/43.0/32.1 (BP=1.000, ratio=1.026, hyp_len=32557, ref_len=31718)\n'
----------
Final: 
Set:  test
Approach: end2end
Model:  transformer
b'BLEU = 31.88, 61.1/37.1/26.2/19.1 (BP=0.977, ratio=0.977, hyp_len=38993, ref_len=39898)\n'
----------
Final: 
Set:  test
Approach: end2end
Model:  rnn
b'BLEU = 33.49, 58.1/37.3/27.5/21.1 (BP=1.000, ratio=1.042, hyp_len=42722, ref_len=41009)\n'
----------

Recommend Projects