skywalker023 / pragmatic-consistency

🤖 Code for our EMNLP 2020 paper: "Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness"

Pragmatic Self-Consciousness
for Improving Persona Consistency in Dialogues

Official PyTorch implementation of our EMNLP paper:
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness. EMNLP, 2020 [Paper]

• TL;DR: Inspired by social cognition and pragmatics, we model public self-consciousness in existing dialogue agents with an imaginary listener to improve consistency. Compared to previous works, our method does not require additional consistency-related labels or training.

Earlier version of this work was also accepted at ICLR 2020 Bridging AI and Cognitive Science (BAICS) workshop as an oral presentation.

Reference

If you use the materials in this repository as part of any published research, we ask you to cite the following paper:

@inproceedings{Kim:2020:selfc,
  title={Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness},
  author={Kim, Hyunwoo and Kim, Byeongchang and Kim, Gunhee},
  booktitle={EMNLP},
  year=2020
}

Have any question?

Please contact Hyunwoo Kim at [email protected].

Implementation

System Requirements

• Python 3.6.8
• Pytorch 1.6.0
• CUDA 10.1 supported GPU with at least 12GB memory
• See environment.yml for details

Environment setup

Our code is built on the ParlAI framework.
We recommend you create a conda environment as follows

conda env create -f environment.yml

and activate it with

conda activate pragmatic-consistency

Running Experiments

Self-conscious Blender for its persona

Dialogue NLI

python eval_dnli.py --conscious-target self -t tasks.teachers:SelfConsciousDialogueTeacher --model agents.selfconscious_blender:SelfConsciousBlenderAgent --fp16 false

PersonaChat

python eval_personachat.py --conscious-target self -t tasks.teachers:SelfConsciousDialogueTeacher --model agents.selfconscious_blender:SelfConsciousBlenderAgent --batchsize 48 --fp16 false

Self-conscious Blender for its context

Dialogue NLI

python eval_dnli.py --conscious-target context -t tasks.teachers:ContextConsciousDialogueTeacher --model agents.selfconscious_blender:SelfConsciousBlenderAgent --fp16 false

PersonaChat

python eval_personachat.py --conscious-target context -t tasks.teachers:ContextConsciousDialogueTeacher --model agents.selfconscious_blender:SelfConsciousBlenderAgent --batchsize 48 --fp16 false

💡 In case you want to run the evaluation with vanilla Blender as is, set the --conscious-target to none.

Acknowledgements

We would like to thank Reuben Cohn-Gordon, Sean Welleck, Junhyug Noh and Jiwan Chung for their valuable comments. We also thank the anonymous reviewers for their thoughtful suggestions on this work.

This research was supported by Brain Research Program by National Research Foundation of Korea (NRF) (2017M3C7A1047860), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-01772, Video Turing Test, No. 2019-0-01082, SW StarLab), and Creative Pioneering Researchers Program through Seoul National University.

License

This repository is MIT licensed. See the LICENSE file for details.

pragmatic-consistency's Issues

[C-score 관련]

안녕하세요

Blender 90M 모델에 대해 정량적 실험 결과가 논문에 기재된 결과와 유사하게 잘 복원되는지를 확인하는 도중에 한 가지 질문이 생겼습니다.

논문에 기재된 0.85 (Blender S0 에 해당) C-score 와 달리, 제가 직접 실행해보니 C-score 가 음수 값이 산출되고 있습니다. 실험에 사용한 데이터셋은 convai2:normalized valid dataset 에 대해 정량적 점수를 구하고 있습니다. (다른 ppl, f1 값은 유사하게 산출되고 있습니다) 모델은 동일하게 zoo:blender/blender_90M/model 을 사용하고 있습니다.

혹시, 제가 놓친 부분이 있거나 잘못한 부분이 있을까요?

감사합니다.

Baseline hyperparameters& datasets

Hi! Could you specify what hyperparameters you use when finetuning the Blender model on PersonaChat? (I did not see that specified in your paper). To be more specific, what's the parlai comment you use to finetune the model? We tried the following commend but did not obtain the same results:
parlai train_model -m transformer/generator --delimiter ‘\n’ -t personachat:self_original -bs 4 -mf zoo:blender/blender_90M/model -tblog True -stim 1000 -lr 0.0001

How to interact with the model?

Hello, thank you for the great work. I wander how to interact with the provided checkpoint via CLI? Could you provide a script to do it?

Experiment datasets

Hi,
One more question. In your experiments, did you use the original or the revised version of PersonaChat? Or both? (I see a both setting in Parlai's doc but not sure what that implies)