This is a past event
Reassessing the state of the art in multi-document news summarization and its evaluation
In this talk I will survey the state-of-the-art techniques for multi-document news summarization and automatic evaluation of summarization. I will outline three open problems with significant implications for future progress in the field. First, I will discuss the ability to control the level of specificity in the summary, an aspect of summarization in which the operation of systems still differs dramatically from that of people performing the same task. Human authored summaries are more general than the original documents while machine summaries are markedly more specific, calling for the need to incorporate measures of specificity in approaches for content selection and sentence compression. Next, I will present two relatively recent findings in summarization evaluation, namely that the best systems perform equally well, with only a few significant differences among them according to manual evaluation and that the combined content of their summaries forms an informative gold-standard such that automatic evaluation can be performed with high accuracy using only that gold-standard. Taken together these finings indicate an untapped potential for system combination in summarization. Finally, I will present a recent assessment of the accuracy of ROUGE metrics which draws attention to the need for more careful use of automatic evaluation. Using ROUGE to compare systems leads to incorrect conclusion in one in three comparisons. Moreover, there is a considerable difference in the ability of different ROUGE variants to support correct inferences about the statistical significance of differences between systems.
Bio:Ani Nenkova is an Assistant Professor of Computer and Information Science at the University of Pennsylvania. Her main areas of research are automatic text summarisation, affect recognition and text quality. She obtained her PhD degree in Computer Science from Columbia University in 2006. She then spent a year and a half as a postdoctoral fellow at Stanford University before joining Penn in Fall 2007. Ani was a member of the editorial board of Computational Linguistics (2009--2011) and has served as an area chair/senior program committee member for ACL, NAACL, AAAI and IJCAI. Her research is sponsored by NSF, NIH and Google.
- Speaker
- Prof Ani Nenkova
- Hosted by
- Advaith Siddharthan
- Venue
- MT6