In order to create successful evaluations of information retrieval technology, researchers generally make simplifying assumptions. For example, it is standard in the community to assume that relevance is binary -- a document is or is not relevant -- even though everyone recognizes that relevance is actually context dependent and that it is often true that some items are "more relevant" than others. However, making the assumption that relevance can only be true or false has resulted in clearer evaluations that have successfully advanced the state of the art over several decades.
We are interested in relaxing -- or at least changing -- some assumptions that are made in the evaluation of topic tracking in the context of the Topic Detection and Tracking (TDT) experiments and of information filtering in the context of the Text REtrieval Conference (TREC) workshops. The common evaluation models for both tasks abstract the user out of the evaluation in one of two extreme ways: TDT topic tracking pretends that the user never interacts with the system after it has been started, and TREC information filtering imagines that the user is always readily available to make relevance assessments. Both evaluation models assume the user is always available to make a judgment, and do not allow for user fatigue. None of those assumptions is realistic for most users, so it is fruitful to investigate what happens when they are changed to be less simplistic.
In this talk I will compare the TDT and TREC evaluation models and discuss their differences and similarities. I will then extend the evaluation framework for TDT topic tracking to incorporate those more realistic user-centered issues. I will demonstrate that tracking can be done in a realistic interactive setting with minimal impact on tracking cost and with substantial reduction in required interaction.
For the colloquium series schedule, see the UMD Computational Linguistics Colloquium Series web page at http://umiacs.umd.edu/~resnik/cl_colloquium/. If you are interested in meeting with the speaker, please contact Philip Resnik (resnik@umiacs.umd.edu).