OntoNotes: Large Scale Multi-Layer, Multi-Lingual,  Distributed  Annotation

Abstract

The OntoNotes project has annotated a large corpus comprising various genres in three languages with syntax, predicate argument structure, word senses, named entities and within document coreference. An important goal of the project was to ensure that each layer of annotation had reasonably high inter-annotator agreement ($\sim$90%). The multiple layers of annotation were developed asynchronously across multiple annotation sites. In this case study, we focus on the mechanics of the annotation process rather than on the annotations themselves. We first describe the data representation challenges, and present the developed representation. We then discuss the requirements for managing the data logistics, and, finally, describe some particular challenges pertaining to specific annotation layers.

Defining, Tracking and Reporting Annotation Errors

This section gives the details of the error code scheme used to track errors in the proposition and sense layers, making clear the many distinctions that turned out to be important to track and giving an example of how these complexities can be dealt with. Table 6 shows the structure of the component sub-fields, and Table 7 lists the specific codes with their meaning and severity levels.

Table 6 Nomenclature of the error codes

Table 7 Descriptions of error codes used to identify problems in proposition annotation—including ones for the TB/PB merge cases

For the sense and proposition data, we periodically tracked the status of the coverage errors by creating a html report. Figure 4 shows a screenshot of the report for the proposition layer.

Following is the description of the information presented in the table.

• 1. Corpus—This is the sub-corpus that the row represents. It shows intermediate names used for various subcorpora.

• 2. Total Taggable—These are the total number of taggable instances for that particular corpus. In case of the web data, this was exactly the number of pointers in the repository as the web data annotation started after this procedure was put in place. In case of older data, for example, WSJ, we may not have added blank pointers to represent the holes in annotation yet, so this number would tend to be somewhat greater (if not equal) to the number of tagged instances in the respective .prop files in the repository.

• 3. Ready to Release—These are the annotated instances that have passed the consistency checks in the automatic database build process and can be considered ready to be released if we were to prepare an OntoNotes release at the time the report was built—so only “gold” or “adjudicated” instances will be counted in this category. These also included cases, that, according to the knowledge built in the API, we think were correct to begin with or had been successfully automatically mapped on to newer versions of trees. This category was used to track coverage.

• 4. Completed Annotation—These are the total instances that have been adjudicated.

• 5. Awaiting Correction—These represented the total number of instances that had been flagged by the build process as having at least one error or warning in them.

• 6. Awaiting Correction – Details by Error code—From a reporting point of view, it is not very important to get to the details of how many instances have been flagged by what types of errors, but form an operational perspective this information can be very useful. So, this was a “hidden” column that can be shown or re-hidden by using the buttons at the bottom of the table. When shown, this column shows all the error codes that have been encountered in loading the data, and under each error code will be listed the instances for each corpora that got identified.The list of error codes is a subset of the potential error-codes for the data; i.e., if a particular error-type is not encountered in all of the data for a particular language, then there won’t be the error code column here, but that implicitly means that there were no instances identified for that error code, and not that the error code does not exist. Owing to the “one instance can have multiple-errors” rule, the number in all the error codes columns do not sum to the total number of instances awaiting correction. Therefore, to mitigate that, we tried to also generate a “normalized” version of the instances that belong to an error code column. The normalization is done using a “severity” measure that we have tentatively assigned to each error. The measure was roughly proportional to “minutes required to correct an error of this type—on average”. So, if an instance was tagged with multiple errors, then the error that required the most number of minutes would be assigned to that particular instance, and we would increment the number in that error code column by one, and no other error code column will have any increment for that instance. This ensured that when we are done with the table, that the total of all the instances under each error code column sums to the total numbers that are awaiting correction. One downside was that someone has to decide what the exact “severity” values are. Another interesting case that had to be considered is the error code 14051 which represents a successful copy to newer trees—is not really an error code, and that it falls in the category of—“we expect do a random check on these instances, and once it seems satisfactory change all these to error code 00000”. Therefore, in order to give a realistic estimate of the “release coverage” we count just this one category in addition to the ones that have a 00000 error code as been covered. In order to be able to match the error code with the description easily, we hyperlinked the error codes in the column heading. Clicking on this would highlight the appropriate error code along with the description in the list below.

Fig. 4

Screen capture of the proposition report table seen after a successful release build. Only some of the error columns are shown owing to space limitations

• 7. Partially Annotated—Since it is not always the case that the all the data in the pipeline would have been adjudicated at the end of each month, we will also have other columns to represent the data at intermediate levels in the pipeline—single-annotated, double-annotated, pre-gold, etc. This column represents the total data in this category. We will process all the instances in these categories in through the build, but will only merge the ones that have been adjudicated completely.