TY - JOUR T1 - Task-based evaluation of text summarization using Relevance Prediction JF - Information Processing & Management Y1 - 2007 A1 - Hobson,Stacy President A1 - Dorr, Bonnie J A1 - Monz,Christof A1 - Schwartz,Richard KW - Relevance prediction KW - Summarization evaluation KW - Summary usefulness AB - This article introduces a new task-based evaluation measure called Relevance Prediction that is a more intuitive measure of an individual’s performance on a real-world task than interannotator agreement. Relevance Prediction parallels what a user does in the real world task of browsing a set of documents using standard search tools, i.e., the user judges relevance based on a short summary and then that same user—not an independent user—decides whether to open (and judge) the corresponding document. This measure is shown to be a more reliable measure of task performance than LDC Agreement, a current gold-standard based measure used in the summarization evaluation community. Our goal is to provide a stable framework within which developers of new automatic measures may make stronger statistical statements about the effectiveness of their measures in predicting summary usefulness. We demonstrate—as a proof-of-concept methodology for automatic metric developers—that a current automatic evaluation measure has a better correlation with Relevance Prediction than with LDC Agreement and that the significance level for detected differences is higher for the former than for the latter. VL - 43 SN - 0306-4573 UR - http://www.sciencedirect.com/science/article/pii/S0306457307000234 CP - 6 M3 - 10.1016/j.ipm.2007.01.002 ER -