Scoring, transducing, reporting

In addition to iteratively preparing corpora using "tag a little, learn a little", MAT has tools for assessing your progress, converting corpora from one annotation structure to another, and reporting on the content of corpora.

Scoring

We saw the scorer, MATScore, briefly in Tutorial 4. The scorer can be used to compare an annotator's data with a gold standard, or to compare two annotators with each other. The scorer uses a very general and configurable comparison algorithm to determine a best match among the annotations in the document pairs, and then uses a separate scoring configuration to determine which label aggregations and decompositions to display. The output is an Excel- or OpenOffice-compatible spreadsheet.

The <similarity_profile> element in your task XML file allows you to specify, for a given true label, which dimensions of the annotation are used for comparison; what comparison method is used; and how the comparison value should be weighted in computing the overall similarity. The similarity and score algorithm provides a default comparison for use when no comparison is specified for a label. You can define multiple similarity profiles in your task, and choose the one you want using the --similarity_profile option of MATScore.

The <score_profile> element in your task XML file allows you to specify how label scores should be decomposed (e.g., compute separate subscores for annotations with particular attribute values) or aggregated (e.g., compute a separate subscore for all spanned annotations). You can define multiple scoring profiles in your task, and choose the one you want using the --score_profile option of MATScore.

Details on how to define the profiles can be found here.

The scorer itself, in addition to using each of these profile types, also allows you to collapse labels using equivalence classes, and to ignore particular labels.

Finally, the scorer is one of the MAT tools which does not require a task; you can separately tell it which annotations count as content annotations for scoring, and which count as token annotations for token-based scoring. In this case, the default similarity and score profiles will be used. You can use the MAT scorer, then, to compare documents you can read using a MAT reader (e.g., inline XML), even if you haven't configured a task.

Transducing

The transducer tool, MATTransducer, will convert documents from any known readable format (e.g., inline XML) to any known writeable format (e.g., MAT-JSON). The available core readers and writers are described here; it's possible that your task will have defined its own readers. Like the scorer, the transducer does not require a task.

In addition to format conversion, the transducer can convert the structure of your annotations using a declarative XML format. For instance, let's say that you've been given data which contains annotations with the ENAMEX label, and a TYPE attribute whose values are PER, ORG, LOC. And you'd prefer for these annotations to be PERSON, ORGANIZATION and LOCATION labels. You can define this document conversion in XML as follows:

<instructions>
  <labels source="ENAMEX">
    <promote_attr source="TYPE"/>
  </labels>
  <labels source="PER">
    <map target="PERSON"/>
  </labels>
  <labels source="ORG">
    <map target="ORGANIZATION"/>
  </labels>
  <labels source="LOC">
    <map target="LOCATION"/>
  </labels>
</instructions>

With this declarative format, you can

• promote attributes to labels, and demote labels to attributes
• convert annotations from spanned to spanless
• map labels
• apply operations to labels which bear particular attribute-value pairs
• discard and map attributes
• discard annotations if particular attributes are null
• map attribute values

and a range of other things.

Reporting

The reporting tool, MATReport, will generate concordance-style spreadsheets listing all the annotations and their descriptions, with their left and right contexts. For spanless annotations, the tool will show the span of text implied by the spanned annotations linked to the spanless annotations (e.g., relation arguments), and indicate the arguments in-line. You can optionally choose to suppress the left and right concordance context.

The reporting tool also provides the option of generating per-label spreadsheet reports, which have the advantage of showing each label attribute as a separate column. You can use this mode as a quick and dirty CSV export for document annotations.

Like the other two tools describe here, the reporter does not require a task.