Evaluating Basic Text Meaning Representations Produced
by the OntoSem Semantic Analyzer
Sergei Nirenburg
Department of Computer Science and
Institute for Language and Information Technologies
I will describe an evaluation regimen developed for the text meaning representations (TMRs) produced by the general purpose text analyzer OntoSem. The TMRs are formulated in a metalanguage informed by an ontology developed specifically to support natural language processing. OntoSem uses a combination of knowledge-based and stochastic constraints for making heuristic preferences in the space of potential textual analyses. Static constraints are stored in Ontosem lexicons, the ontology and the fact repository of remembered instances of ontological concepts. Dynamic constraints are generated by pre-semantic processing stages, such as morphology and syntax, as well as by semantic processing itself. The initial evaluation has concentrated on a subset of material produced by the analyzer and recorded in the TMRs, specifically, on the “who did what to whom” basic propositional semantic content. Therefore, at the semantic level we measured only the quality of word sense disambiguation and the correctness of semantic dependency structures produced by OntoSem. The goal of evaluation is to determine the quality of TMRs but even more importantly to assign blame for various classes of errors, thus suggesting directions for improvement of both knowledge resources and processors. I will describe the OntoSem processing environment, the evaluation regime itself and results from our first evaluation effort.
About the
Speaker:
For the
colloquium series schedule, see the UMD Computational http://www.umiacs.umd.edu/research/CLIP/colloq/. If you are interested in meeting with the
speaker, please contact Doug <http://www.glue.umd.edu/~oard/> Oard (oard@umiacs.umd.edu <mailto:oard@umiacs.umd.edu> ).