===============================================
Pourpre scoring script for definition questions
===============================================
by Dina Demner-Fushman and Jimmy Lin

Version 1.0, released 05/29/2005


For a description of what Pourpre actually does, please refer to:

    Jimmy Lin and Dina Demner-Fushman. Automatically Evaluating Answers
    to Definition Questions.  Technical Report
    LAMP-TR-119/CS-TR-4695/UMIACS-TR-2005-04, University of Maryland,
    College Park, February 2005.

Few caveats:

- Although our post-hoc study has shown that Pourpre is better than
  Rouge/Bleu for automatically evaluating definition questions, it has
  never been tested on new research systems.  Your mileage may vary.

- Definition QA evaluation in its current state is very
  coarse-grained, and there is a lot of noise in the F-measures.  It
  is unclear how this truly impacts the diagnostic capabilities of
  Pourpre (or any metric, for that matter).  See Ellen Voorhees' TREC
  overview papers for more about this.

If you find this package helpful (e.g., for hill-climbing), please
feel free to share your experiences with us.

-------------------------------------------------------------------------------

There are two scripts included in this package:
  pourpre-1.0i.pl: generates scores based of term idf values
  pourpre-1.0c.pl: generates scores based on simple term counts

  use the -h option to display full usage info

-------------------------------------------------------------------------------

sample invocation:

   $ ./pourpre-1.0c.pl -s samplerun.trec2003.bbn2003c -r nuggets.trec2003 
   BBN2003C	0.35103827192815

   $ ./pourpre-1.0c.pl -s samplerun.trec2004.run12 -r nuggets.trec2004 
   RUN-12	0.30918089803617

-------------------------------------------------------------------------------
Distribution Information

Pourpre scoring script for definition questions
Copyright (C) 2005, Dina Demner-Fushman and Jimmy Lin

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, please visit 
http://www.gnu.org/copyleft/gpl.html

See licesnse.txt for full details

-------------------------------------------------------------------------------