Summarization metrics about nlp-progress

[sebastianruder]

Hey @ellej16, could you be more explicit how Retention Ratio is used as a metric? In Hassel (2004), it is only defined as information in Summary / information in Full Text.

[sebastianruder]

Cool. Given your description, it seems that Answer Recall Average is a lot more expensive to evaluate, particularly on a large scale, as you require human answers for every text. I think that's similar to human evaluation vs. BLEU in Machine Translation and arguably is the main reason why automatic metrics are preferred.

[ellej16]

Tagging summarization.md contributors , sorry for the bother!
@jfsantos @shashiongithub @FredRodrigues @sebastianruder

[ellej16]

Hi @sebastianruder !
On a past research we used Answer Recall Average [mani 2002] to define information is in the summary, via answering certain questions based on the full text. A respondent is tasked beforehand to create the said Q&As on the full text (information in Full text)

[ellej16]

Thank you very much for your time and insight on this one!

Also, huge thanks for this repository (Glad to see summarization still having research interest!)