Evaluation getting off to a start, but there are issues. Some issues with kappa are being investigated

Re-allocation of funds to travel? No one but CMU seems to be having problems as of yet.

Annotators - what are they doing this week?
Columbia and MITRE catching up - MITRE will be caught up today, Columbia needs next week
UMD needs next week
NMSU has started reconciliation

intra-annotator (re-named consistency checking)is ONE annotator looking at two texts from same source, and comparing annotations.
Lori suggested doing reconciliation of same annot. on SAME text, done a month or so apart, without consulatation of first annotation.
Should we do inter- or intra- annot. agreement?
What order should we do the two?
Should everyone do both? Should some sites do one and some the other?

Research Hypothesis: Inter-annotator reconciliation will raise IA numbers. Intra-annotator reconciliation will raise IA numbers. If we do both, that will raise IA more.

Teruko suggested calling intra-annotator reconciliation "consistency checking" to avoid confusion of terms.

There was much discussion of the feasibility of doing consistency checking; two main concerns were the task load/time of annotators, and the lack of support in the current tools for doing this.

Teruko and Owen both expressed concerns about annotators with stronger personality dominating the reconciled annotation.

Should we conflate site annotations (2->1)? Or should we have the annotators categorize what are real ambiguities, reasons for these, and where they could actually reconcile/merge annotations.

Procedures will be developed offline, but the goals are the following:
Consistency checking will raise inter-site IA.
Intra-site K will be 1. We will see if it raises K across sites.
[Discussion unfinished.....]

In Gold Standard, do we annotate ambiguity?

Follow-on TASKS for this week
3 sites will discuss reconciliation: NMSU, ISI, CMU - discuss by e-mail
3 sites will discuss consistency checking: Columbia, MITRE, UMD + Lori