Experiment engine XML reference

The XML format for the experiment files (see MATExperimentEngine) is described in this document. Use cases are described here. Click here for a split-screen view.

Element hierarchy

<experiment>
<binding>
<corpora>
<partition>
<fixed_partition>
<crossvalidation>
<size>
<corpus>
<pattern>
<prep>
<workspace_corpora>
<workspace_corpus>
<partition>
<size>
<fixed_partition>
<crossvalidation>
<model_sets>
<build_settings>
<iterator>
<corpus_settings>
<iterator>
<model_set>
<training_corpus>
<runs>
<run_settings>
<prep_args>
<score_args>
<args>
<iterator>
<run>
<test_corpus>

<experiment>

The toplevel element in the file. Note that none of the five child elements are obligatory; the experiment XML can be used simply to build corpora, or to build models, without performing any experimental runs, if, for instance, you want to build a model or corpus to be used in multiple experiments.

Attributes

Children

<binding> (of <experiment>)

This element allows the user to define global bindings which can be referred to in any other element of the experiment XML file (except the attributes of the <experiment> element itself, and the <binding> elements). These bindings can be referred to either in XML attributes or in text within XML elements. The pattern for each binding is $(...). The experiment directory, whether provided via the dir attribute of the <experiment> element or on the command line, is provided as EXP_DIR; the pattern directory, if provided by the --pattern_dir command line argument to MATExperimentEngine, is provided as PATTERN_DIR.

Attributes

<corpora> (of <experiment>)

Describes corpora to be used in the experiment. This element may be repeated; the intention is that a single <corpora> element will correspond to a shared set of preprocessing instructions.

The corpora may be local, in which case a set of patterns should be provided, or remote, in which case the source_corpus_dir attribute should be provided. Remote corpora are used directly unless one or more of the processing tags are specified (<partition>, <fixed_partition>, <prep>, <crossvalidation>). In this case, the specified processing steps are added or redone locally, on a separate copy of the corpus. For instance, if the remote corpus is split into test and train, but not preprocessed, and the <prep> tag is specified here, the corpus documents will be postprocessed here, and the remote split will be preserved. If the remote corpus is preprocessed and split, but the local <partition> tag specifies that the corpus type is "train", the remote corpus preprocessing will be preserved, but locally the split will be ignored. If the remote corpus contains enough patterns for 300 documents, but max_size remotely is 100 and max_size locally is 200, the local max_size will be used; this is possible because all the documents are preprocessed by default when a corpus is prepared, regardless of max_size, and the order of documents (after an initial randomization) is preserved from remote corpus to local copy.

Note that inside the experiment engine, MAT uses the MAT JSON document format exclusively. Therefore, if you want to provide documents which are in a different format which MAT also understand (e.g., XML inline), you must use the <prep> tag to convert the documents to MAT JSON format.

Attributes

Children

<partition> (of <corpora>)

Specifies a proportional partition of the sister corpora specified with the <corpus> tag. May be repeated. If there are no instances of this element or <fixed_partition>, the corpus has no partitions. The proportional partitions segment the entire corpus, so the fraction values are normalized to shares of the corpus. If you want just a 10th of the corpus, for instance, you must divide the corpus into two partitions at a ratio of 9:1 and ignore the larger slice.

Each <corpora> element is may have either proportional or fixed partitions, or be designated as a crossvalidation corpus, but not more than one of these.

Attributes

<fixed_partition> (of <corpora>)

Specifies a fixed partition of the sister corpora specified with the <corpus> tag. May be repeated. If there are no instances of this element or <partition>, the corpus has no partitions. These partitions do not segment the entire corpus. If you want a fixed partition to encompass "everything else", use the special "remainder" value as described below.

Each <corpora> element is may have either proportional or fixed partitions, or be designated as a crossvalidation corpus, but not more than one of these.

Attributes

<crossvalidation> (of <corpora>)

Indicates that the corpus is a crossvalidation corpus, and allows the specification of the number of crossvalidation folds.

A crossvalidation corpus is used by the experiment engine for crossvalidation. Internally, it is split into <n> partitions, where <n> is the number of folds. A model which is trained against a crossvalidation corpus actually generates <n> models, where the training data for each model <i> is the crossvalidation corpus with partition <i> removed. When the model is used in a run, the run must also be applied to the crossvalidation corpus, and each model <i> is applied to partition <i>. The aggregated results are treated as a single run.

Each <corpora> element is may have either proportional or fixed partitions, or be designated as a crossvalidation corpus, but not more than one of these.

Attributes

<size> (of <corpora>)

Specifies the size properties of the sister corpora specified with the <corpus> tag. If this tag is present, and the sister corpus has the source_corpus_dir attribute set, the specified values will override those in the source corpus.

This element is not compatible with <crossvalidation>.

Attributes

<corpus> (of <corpora>)

A corpus is specified either by a set of patterns, or by a reference to another corpus (via source_corpus_dir). The documents specified by a set of patterns are randomly reordered before any subsequent processing is performed (e.g., split, preprocess). If source_corpus_dir is present, patterns are ignored.

Attributes

Children

<prep> (of <corpora>)

This element houses the arguments to the MATEngine command to use to preprocess the corpora. You might use this command to take documents which have been deidentified and resynthesize fillers for the deidentified regions.

Attributes

<workspace_corpora> (of <experiment>)

In order to run experiments against workspaces, we've introduced a new element which allows you to specify a collection of corpora to be drawn from a workspace, restricted by various dimensions of the workspace contents. Each <workspace_corpora> element establishes a context for a  workspace, by describing a subset of eligible workspace files, and within that context you can define <workspace_corpus> elements which are ultimately transformed into the same objects which the <corpus> elements correspond to.

Each workspace corpus set must have a single trainable step in the workspace's workflow which each document has reached. If the workflow has more than one trainable step, you must specify the "step" attribute.

Attributes

Children

<workspace_corpus> (of <workspace_corpora>)

Each <workspace_corpus> element defines a set of files, similar to a <corpus> element. Because its common context is a workspace, we've chosen to allow the partition, crossvalidation and size information to be specified independently for each of these elements, rather than establishing them in the common workspace context (as <corpora> does for <corpus>). Within the workspace context, you can further restrict each corpus in the same way as you restrict the context itself, and also identify a special, unique corpus as containing the remainder of the files in the workspace context. Note that you can't further filter the workspace's workflow step; this is specifiable only in the parent element.

Attributes

Children

The semantics of these elements are identical to the semantics of these elements in the scope of the <corpora> element.

<partition> (of <workspace_corpus>)

See <partition> of <corpora> for details.

<fixed_partition> (of <workspace_corpus>)

See <fixed_partition> of <corpora> for details.

<crossvalidation> (of <workspace_corpus>)

See <crossvalidation> of <corpora> for details.

<size> (of <workspace_corpus>)

See <size> of <corpora> for details.

<model_sets> (of <experiment>)

Each experiment also can contain a number of model sets. A model set is a sequence of models built out of the same corpus, with successively larger numbers of training inputs. This iterative capability does not have to be used, but is available if the user wants to track the change in performance relative to the number of training documents.

Attributes

Children

<build_settings> (of <model_sets>)

In order to run an experiment, you must have declared  the appropriate <model_config> in your task.xml file.

While the training engine you're most likely to use is the jCarafe engine, your task may have multiple trainable steps and multiple trainable engines, and you'll need to ensure that the proper one is chosen. If you have multiple trainable steps, you must specify the trainable step explicitly using the "trainable_step" attribute.

Attributes

Children

<iterator> (of <build_settings>)

It is possible to iterate through a set of values for the model builder using an iterator. For instance, the default jCarafe engine allows you to customize the degree of L1 regularization (see the jCarafe documentation for details). You might want to build a series of models exploring the effects of L1 regularization values from 0.0 through 2.0, at increments of .2. Or, you might want to vary the number of training iterations the engine performs from 10 to 150 at increments of 10.

You can specify multiple iterators, and you'll get the cross-product of the settings. The iterator mechanism is flexible enough that you can build iterators which depend on the last model built, or iterators which specify their own model builder class (if, for example, they need to do some extensive computation on the training corpus before they train).

Corpus setting iterators and build setting iterators are both applied to the model, and the cross-product of the possible values is used. The corpus setting iterators are applied first.

The build settings support two built-in iterators, which can be configured using the <iterator> element.

Attributes

Here are the attributes and values available for the "value" iterator:

Here are the attributes and values available for the "increment" iterator:

<corpus_settings> (of <model_sets>)

In addition to the model builder itself, you can configure properties of the training corpus as well. At the moment, the only property of the training corpus you can configure is its size, and this is mostly in service of the iterator over corpus size. When  you specify the size of a corpus, it will truncate the training corpus to the specified length.

Attributes

Children

<iterator> (of <corpus_settings>)

It is possible to iterate through a set of values for the model corpus using an iterator. Right now, the only available iterator is the "corpus_size" iterator (although you can also define your own if you need to).

Corpus setting iterators and build setting iterators are both applied to the model, and the cross-product of the possible values is used. The corpus setting iterators are applied first.

Attributes

Here are the attributes and values available for the "corpus_size" iterator:

<model_set> (of <model_sets>)

Attributes

Children

<training_corpus> (of <model_set>)

May be repeated. Specifies the training corpora to use in building this model set. Each corpus is referred to by name, and an optional partition name. Note that the model will not use a document path more than once, so if you specify multiple corpora, and the corpora overlap, only one instance of each path will be used.

If the corpus is a crossvalidation corpus, there can be only one training corpus, and its partitions may not be referred to. In addition, you may not specify the corpus size in the corpus settings, or use the corpus size iterator.

Attributes

<runs> (of <experiment>)

The experiment also can have a set of runs. The runs in each <runs> element share a set of run settings. Whenever the experiment is run, each <run> is scored, whether or not it's been scored before. This is a convenient way of reviewing the scores after an experiment is finished.

The scores produced by the experiment engine reflect only the annotations created, and attributes set, by the step which the model for the run was built for. It is equivalent to passing the annotation sets for that step as the --content_annotation_sets option to MATScore.

Attributes

Children

<run_settings> (of <runs>)

Children

<args> (of <run_settings>)

This element houses the arguments to the MATEngine command to perform the experiment runs.

Attributes

<prep_args> (of <run_settings>)

This element houses the arguments to the MATEngine command to to use to prepare the annotated documents for the experiment runs. By default, the documents are converted to raw documents, but if instead you want to just undo a step and leave them as MAT JSON documents, you can use this element to achieve that.

Attributes

<score_args> (of <run_settings>)

These flags control the behavior of the scorer. They are not yet generally processed the way <prep_args> and <args> are; they're individually defined and handled, for the moment.

Attributes

<iterator> (of <run_settings>)

It is possible to iterate through a set of values for the run using an iterator. For instance, the default jCarafe engine allows you to customize the recall/precision bias (see the jCarafe documentation for details). You might want to build a series of models exploring the effects of recall/precision bias values from -2.0 through 2.0, at increments of .5.

You can specify multiple iterators, and you'll get the cross-product of the settings. The iterator mechanism is flexible enough that you can build your own iterators if you need them.

The run settings support two built-in iterators, which can be configured using the <iterator> element.

Attributes

Here are the attributes and values available for the "value" iterator:

Here are the attributes and values available for the "increment" iterator:

<run> (of <runs>)

Attributes

Children

<test_corpus> (of <run>)

May be repeated. One or more test corpora (and possibly partitions of corpora) to use in this run.

Attributes