Promoting Insight-Based Evaluation of Visualizations: From Contest to Benchmark Repository

TitlePromoting Insight-Based Evaluation of Visualizations: From Contest to Benchmark Repository
Publication TypeJournal Articles
Year of Publication2008
AuthorsPlaisant C, Fekete J-D, Grinstein G
JournalVisualization and Computer Graphics, IEEE Transactions on
Pagination120 - 134
Date Published2008/02//jan
ISBN Number1077-2626
KeywordsComputer-Assisted;Software;Software Validation;User-Computer Interface;, Factual;Evaluation Studies as Topic;Image Interpretation, InfoVis contest;benchmark repository;information visualization;data visualisation;Algorithms;Benchmarking;Computer Graphics;Databases

Information visualization (InfoVis) is now an accepted and growing field, but questions remain about the best uses for and the maturity of novel visualizations. Usability studies and controlled experiments are helpful, but generalization is difficult. We believe that the systematic development of benchmarks will facilitate the comparison of techniques and help identify their strengths under different conditions. We were involved in the organization and management of three InfoVis contests for the 2003, 2004, and 2005 IEEE InfoVis Symposia, which requested teams to report on insights gained while exploring data. We give a summary of the state of the art of evaluation in InfoVis, describe the three contests, summarize their results, discuss outcomes and lessons learned, and conjecture the future of visualization contests. All materials produced by the contests are archived in the InfoVis benchmark repository.