Scoring engine

Description

The scoring engine compares two tagged files, or two directories of tagged files. Typically, one input is the hypothesis (an automatically tagged file) and the other is the reference (a gold-standard tagged file). But this tool can be used to compare any two inputs.

For a description of the pairing and scoring algorithm, see here. For a description of the scorer output, see here.

Usage

Unix:

% $MAT_PKG_HOME/bin/MATScore

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd

Usage: MATScore [options]

Core options

Filtering annotations with annotation sets

The interaction between specified, included or excluded annotation sets and the annotations present is subtle. In some cases, annotation sets define attributes which refer to labels which are defined in other annotation sets. In these cases, the scoring algorithm ensures that only the labels and attributes which are made available by the included sets are scored. The result might be that some annotation attributes might be ignored in the pairing algorithm, and some annotations might not be compared on the basis of their labels and spans. These restrictions interact appropriately with the similarity and score profiles.

The scorer will not report on annotation labels designated processable="no".

Hypothesis options

Reference options

Score output options

Note that all the CSV files created by the scorer are in UTF-8 encoding.

When the user requests confidence data via the --compute_confidence_data option, the scorer produces 1000 alternative score sets. Each score set is created by making M random selections of file scores from the core set of M file scores. (This procedure will naturally have multiple copies of some documents and no copies of others in each score set, which is the source of the variation for this computation.) The scorer then computes the overall metrics for each alternative score set, and computes the mean and variance over the 1000 instances of each of the precision, recall, and fmeasure metrics. This "sampling with replacement" yields a more stable mean and variance.

Other options

The readers referenced in the --file_type and --ref_file_type options may introduce additional options, which are described here. These additional options must follow the --file_type and --ref_file_type options. The options for the reference file types are all prepended with a ref_ prefix; so for instance, to specify the --xml_input_is_overlay option for xml-inline reference documents, use the option --ref_xml_input_is_overlay.

Examples

Example 1

Let's say you have two files, /path/to/ref and /path/to/hyp, which you want to compare. The default settings will print a table to standard output.

Unix:

% $MAT_PKG_HOME/bin/MATScore --file /path/to/hyp --ref_file /path/to/ref

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd --file c:\path\to\hyp --ref_file c:\path\to\ref

Example 2

Let's say that instead of printing a table to standard output, you want to produce CSV output with embedded Excel formulas (the default formula type), and you want all three spreadsheets.

Unix:

% $MAT_PKG_HOME/bin/MATScore --file /path/to/hyp --ref_file /path/to/ref \
--csv_output_dir $PWD --details --by_token

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd --file c:\path\to\hyp --ref_file c:\path\to\ref \
--csv_output_dir %CD% --details --by_token

This invocation will not produce any table on standard output, but will leave three files in the current directory: bytag_excel.csv, bytoken_excel.csv, and details.csv.

Example 3

Let's say you have two directories full of files. /path/to/hyp contains files of the form file<n>.txt.json, and /path/to/ref contains files of the form file<n>.json. You want to compare the corresponding files to each other, and you want tag and token scoring, but not details, and you intend to view the spreadsheet in OpenOffice.

Unix:

% $MAT_PKG_HOME/bin/MATScore --dir /path/to/hyp --ref_dir /path/to/ref \
--ref_fsuff_off '.txt.json' --ref_fsuff_on '.json' \
--csv_output_dir $PWD --csv_formula_output oo --by_token

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd --dir c:\path\to\hyp --ref_dir c:\path\to\ref \
--ref_fsuff_off ".txt.json" --ref_fsuff_on ".json" \
--csv_output_dir %CD% --csv_formula_output oo --by_token

For each file in /path/to/hyp, this invocation will prepare a candidate filename to look for in /path/to/ref by removing the .txt.json suffix and adding the .json suffix. The current directory will contain bytag_oo.csv and bytoken_oo.csv.

Example 4

Let's say that you're in the same situations as example 3, but you want confidence information included in the output spreadsheets:

Unix:

% $MAT_PKG_HOME/bin/MATScore --dir /path/to/hyp --ref_dir /path/to/ref \
--ref_fsuff_off '.txt.json' --ref_fsuff_on '.json' \
--csv_output_dir $PWD --csv_formula_output oo --by_token --compute_confidence_data

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd --dir c:\path\to\hyp --ref_dir c:\path\to\ref \
--ref_fsuff_off ".txt.json" --ref_fsuff_on ".json" \
--csv_output_dir %CD% --csv_formula_output oo --by_token --compute_confidence_data

Example 5

Let's say that you're in the same situation as example 3, but your documents contain lots of tags, but you're only interested in scoring the tags listed in the "Named Entity" task. Furthermore, you're going to import the data into a tool other than Excel, so you want the values calculated for you rather than having embedded equations:

Unix:

% $MAT_PKG_HOME/bin/MATScore --dir /path/to/hyp --ref_dir /path/to/ref \
--ref_fsuff_off '.txt.json' --ref_fsuff_on '.json' \
--csv_output_dir $PWD --csv_formula_output literal --by_token --task "Named Entity"

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd --dir c:\path\to\hyp --ref_dir c:\path\to\ref \
--ref_fsuff_off ".txt.json" --ref_fsuff_on ".json" \
--csv_output_dir %CD% --csv_formula_output literal --by_token --task "Named Entity"

Example 6

Let's say you're in the same situation as example 3, but your reference documents are XML inline documents, and are of the form file<n>.xml. Do this:

Unix:

% $MAT_PKG_HOME/bin/MATScore --dir /path/to/hyp --ref_dir /path/to/ref \
--ref_fsuff_off '.txt.json' --ref_fsuff_on '.xml' \
--csv_output_dir $PWD --csv_formula_output oo --by_token --ref_file_type xml-inline

Windows native:

> %MAT_PKG_HOME%\bin\MATScore.cmd --dir c:\path\to\hyp --ref_dir c:\path\to\ref \
--ref_fsuff_off ".txt.json" --ref_fsuff_on ".xml" \
--csv_output_dir %CD% --csv_formula_output oo --by_token --ref_file_type xml-inline

Note that --ref_fsuff_on has changed, in addition to adding the --ref_file_type option.