In addition to iteratively preparing corpora using "tag a little, learn a little", MAT has
tools for assessing your progress, converting corpora from one
annotation structure to another, and reporting on the content of
We saw the scorer, MATScore, briefly
in Tutorial 4. The scorer can be
used to compare an annotator's data with a gold standard, or to
compare two annotators with each other. The scorer uses a very
general and configurable comparison
algorithm to determine a best match among the annotations in
the document pairs, and then uses a separate scoring configuration
to determine which label aggregations and decompositions to
display. The output is an Excel-
or OpenOffice-compatible spreadsheet.
element in your task XML file allows you to specify, for a given
true label, which dimensions of the annotation are used for
comparison; what comparison method is used; and how the comparison
value should be weighted in computing the overall similarity. The
similarity and score algorithm provides a default comparison for
use when no comparison is specified for a label. You can define
multiple similarity profiles in your task, and choose the one you
want using the --similarity_profile option of MATScore.
element in your task XML file allows you to specify how label
scores should be decomposed (e.g., compute separate subscores for
annotations with particular attribute values) or aggregated (e.g.,
compute a separate subscore for all spanned annotations). You can
define multiple scoring profiles in your task, and choose the one
you want using the --score_profile option of MATScore.
The scorer itself, in addition to using each of these profile
types, also allows you to collapse labels using equivalence
classes, and to ignore particular labels.
Finally, the scorer is one of the MAT tools which does not
require a task; you can separately tell it which annotations count
as content annotations for scoring, and which count as token
annotations for token-based scoring. In this case, the default
similarity and score profiles will be used. You can use the MAT
scorer, then, to compare documents you can read using a MAT reader (e.g., inline
XML), even if you haven't configured a task.
The transducer tool, MATTransducer,
will convert documents from any known readable format (e.g.,
inline XML) to any known writeable format (e.g., MAT-JSON). The
available core readers and writers are described here; it's possible that
your task will have defined its own
readers. Like the scorer, the transducer does not require a
In addition to format conversion, the transducer can convert the
structure of your annotations using a declarative XML format. For
instance, let's say that you've been given data which contains
annotations with the ENAMEX label, and a TYPE attribute whose
values are PER, ORG, LOC. And you'd prefer for these annotations
to be PERSON, ORGANIZATION and LOCATION labels. You can define
this document conversion in XML as follows:
With this declarative format, you can
and a range of other things.
The reporting tool, MATReport, will
generate concordance-style spreadsheets listing all the
annotations and their descriptions, with their left and right
contexts. For spanless annotations, the tool will show the span of
text implied by the spanned annotations linked to the spanless
annotations (e.g., relation arguments), and indicate the arguments
in-line. You can optionally choose to suppress the left and right
The reporting tool also provides the option of generating
per-label spreadsheet reports, which have the advantage of showing
each label attribute as a separate column. You can use this mode
as a quick and dirty CSV export for document annotations.
Like the other two tools describe here, the reporter does not
require a task.