| |
In October 2008, Bonnie Dorr, and her students Matthew Snover and Nitin Madnani (in collaboration with Rich Schwartz at BBN Technologies) participated in the first ever NIST Metric MATR workshop -- a workshop coordinated by NIST to evaluate and compare automatic machine translation evaluation metrics. Their submission, TERp (Translation Edit Rate plus), was evaluated by its ability to automatically predict the quality of a translation by correlating scores from the metric with human judgments of translation quality.
TERp was one of the top performing metrics at the workshop, and had the highest absolute correlation, as measured by the Pearson correlation coefficient, with human judgments in 9 of the 45 test conditions---more than any other metric. In addition, in 33 of the 45 test conditions, TERp was statistically indistinguishable from the top metric---again more than any other metric. Overall, TERp was consistently one of the best performing metrics in the workshop.
State of the art, baseline evaluation metrics from IBM, Carnegie Mellon University, the University of Maryland, and the National Institute of Standards and Technology were also included in the evaluation. Novel automatic machine translation evaluation metrics were also submitted by researchers from Carnegie Mellon University, RWTH Aachen University, the University of Washington, Columbia University, Stanford University, the University of Southern California, the Information Sciences Institute, the National University of Singapore and several others---a total of 39 submitted metrics. |
