Notes from April 2-3, 2004 Meeting (not included in Committee Reports below)

1. Feedback from Annotators

• Summary of annotators' feedback on the first round of annotation:
  • Greatest confusion on assignment of theta rules; examples don't cover all the cases; daughter nodes of verb sometimes include elements that should not receive theta roles (or occasionally do not include those that should [?]).
  • Not always sure when to assign DummyConcept. [problem may be local]
  • Assignment of open-ended number of Concepts not entirely satisfactory; options considered:
    • Rate each concept as Yes, No, Possible
    • Order concepts
    • Weight concepts
    • Put limit on number of concepts asssigned by any one annotator and allow reconciliation process to smooth things out.
• See Notes from Reconciliation phone call.
  • Collapse steps 2 and 3, but keep voting secret.
  • Consider use of adjudicator, who not only would reconcile differences after final voting, but who would also indicate source of difference
• See Notes from Consistency checking phone call.
  • Near synonymy: What counts?
    • Look at Hirst & Edmonds paper, Near-synonymy and lexical choice, Computational Linguistics 28/2 (June 2002): 105-144.
    • Guidance should be included in IL1 manual
    • This could provide the basis for the panel at the ACL Text Meaning workshop
  • Alignment tool for IL1 consistency checking needed

3. Ontology

• Ed is looking for an ontology czar
• Processes in developing the onotology include:
  • Merge Mikrokosmos concepts into matching WordNet concepts
  • Prune overly specific concepts
  • Add concepts not found during IL1 annotation
  • Merge Catvar into the ontology

• Procedure Committee, April 2-3, 2004
• Content Committee, April 2-3, 2004