thiagocf05 / neuralreg

Referring Expression Generation using Neural Networks

NeuralREG

This project provides the data and models described on the ACL 2018 paper "NeuralREG: An end-to-end approach to referring expression generation" (available here).

NeuralREG models

NeuralREG+Seq2Seq

Seq2Seq version. To train and evaluate the model, you may update the variable paths in the script and run the following command:

python seq2seq.py --dynet-gpu

NeuralREG+CAtt

Concatenative attention version. To train and evaluate the model, you may update the variable paths in the script and run the following command:

python3 attention.py --dynet-gpu

NeuralREG+HierAtt

Hierarcical attention version. To train and evaluate the model, you may update the variable paths in the script and run the following command:

python hierattention.py --dynet-gpu

Data

WebNLG

The original and delexicalized versions of the WebNLG corpus used in our experiments.

Referring Expressions

Training, development and test referring expressions sets and vocabularies. This is the official data used to train and evaluate the models. It was extracted from WebNLG/ using the command:

python preprocessing.py [IN_PATH] [OUT_PATH] [STANFORD_PATH]

Baselines

OnlyNames

OnlyNames baseline. The model may be executed by the following command:

python2.7 only_names.py

Castro Ferreira et al.

This baseline is an adaptation of the model described in this paper. The model may be executed by the following commands:

python2.7 reg_train.py
python2.7 reg_main.py

Evaluation

eval/

Automatic evaluation scripts to extract information about the referring expression collection (corpus.py), to obtain the results depicted in the paper (evaluation.py) and to test statistical significance (statistics.R)

humaneval/

Human evaluation scripts to obtain results depicted in the paper (stats.py) and to test statistical significance (statistics.R)

Citation

@InProceedings{ferreiraetal2018b,
  author = 	"Castro Ferreira, Thiago
		and Moussallem, Diego
		and K{\'a}d{\'a}r, {\'A}kos
		and Wubben, Sander
		and Krahmer, Emiel",
  title = 	"NeuralREG: An end-to-end approach to referring expression generation",
  booktitle = 	"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
  year = 	"2018",
  publisher = 	"Association for Computational Linguistics",
  pages = 	"1959--1969",
  location = 	"Melbourne, Australia",
  url = 	"http://aclweb.org/anthology/P18-1182"
}

Author: Thiago Castro Ferreira

Date: 15/12/2017 (Updated on June 3rd 2019)

neuralreg's Issues

Make the code operational / python 3 version / requirements

Hi Thiago,
This is very interesting work, but it is unfortunately very hard (dare I say impossible) to run.

• Can you please include a requirements.txt file (command pipreqs .)
• Can you please fix the small python3 incompatibility issues (print, and cPickle)

---

• The reason I say "impossible" is the fact that when I try to do the first step in the README:

python2.7 preprocessing.py

The following lines exist:

sys.path.append('../')
sys.path.append('~/workspace/stanford_corenlp_pywrapper')

Which unless running on your machine, probably won't have all dependencies.

NeuralREG applied to visual dataset?

Thank you for your implementation, I have a question regarding the application of your framework NeuralREG. is it applicable to visual dataset like COCO or RefCOCO where the input is an image + target region? If so, would you please provide an example / instruction on applying NeuralREG to such type of data?

thank you so much in advance!