renll / c-pmi

[ACL 2023 DialDoc] C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation

C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation

Liliang Ren, Mankeerat Sidhu, Qi Zeng, Revanth Gangi Reddy, Heng Ji, ChengXiang Zhai

University of Illinois at Urbana-Champaign

Updates

1. [Aug. 28th] Fixed an issue on incomplete context and achieved better average scores! Details in the following table:

Metrics	Interesting	Fluent	Engaging	Specific	Relevant	Correct	Appro.	Und.	Avg.
FED + C-PMI-SYM	48.4	16.6	36.9	28.0	10.5	14.8	17.9	10.7	23.0
FED + C-PMI	48.2	17.6	37.0	28.7	12.8	17.6	18.1	11.1	23.9

Abstract

We propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative loglikelihood-based scorer with our proposed CPMI scorer, we achieve a relative 60.5% higher Spearman correlation on average for the FED evaluation metric.

Code Structure

The implementation of C-PMI is quite simple and only needs a few lines of code. Running the jupyter notebook, c-pmi.ipynb, will reproduce the experiment results in our paper. Our C-PMI and C-PMI-SYM scorer are defined as the function MI_score_turn_pmi and the function MI_score_turn_sympmi respectively in the notebook.

Citation

If you find our work useful, please consider citing:

@article{ren2023cpmi,
  title   = {C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation},
  author  = {Liliang Ren and Mankeerat Sidhu and Qi Zeng and Revanth Gangi Reddy and Heng Ji and ChengXiang Zhai},
  year    = {2023},
  journal = {arXiv preprint arXiv: 2306.15245}
}

Contact

Liliang Ren ([email protected])

c-pmi's Issues

The first turn is not in the context

in function MI_score_turn

  lpx = - get_avg_nll(text + sep + m)
  response = text.split(sep)[-1]
  context = sep.join(text.split(sep)[1:-1])   # hree

the text starts with "<|endoftext|> " (without leading space)
the split function will return ["<|endoftext|> turn0...", "turn1", ...]
so the first turn will not be in the context