%0 Journal Article %J Information Processing & Management %D 2007 %T Task-based evaluation of text summarization using Relevance Prediction %A Hobson,Stacy President %A Dorr, Bonnie J %A Monz,Christof %A Schwartz,Richard %K Relevance prediction %K Summarization evaluation %K Summary usefulness %X This article introduces a new task-based evaluation measure called Relevance Prediction that is a more intuitive measure of an individual’s performance on a real-world task than interannotator agreement. Relevance Prediction parallels what a user does in the real world task of browsing a set of documents using standard search tools, i.e., the user judges relevance based on a short summary and then that same user—not an independent user—decides whether to open (and judge) the corresponding document. This measure is shown to be a more reliable measure of task performance than LDC Agreement, a current gold-standard based measure used in the summarization evaluation community. Our goal is to provide a stable framework within which developers of new automatic measures may make stronger statistical statements about the effectiveness of their measures in predicting summary usefulness. We demonstrate—as a proof-of-concept methodology for automatic metric developers—that a current automatic evaluation measure has a better correlation with Relevance Prediction than with LDC Agreement and that the significance level for detected differences is higher for the former than for the latter. %B Information Processing & Management %V 43 %P 1482 - 1499 %8 2007/11// %@ 0306-4573 %G eng %U http://www.sciencedirect.com/science/article/pii/S0306457307000234 %N 6 %R 10.1016/j.ipm.2007.01.002