Time Interval Annotations
This document provides an introduction to performing time interval annotations in Platform for Situated Intelligence Studio. It explains how to construct your own annotation schemas and definitions, and how to create annotation streams. The tutorial is structured in the following sections:
- Overview: explains the capabilities of the annotations system in PsiStudio.
- Unrestricted Annotation Schema: explains the structure of unrestricted annotation schemas.
- Finite Annotation Schema: explains the structure of finite annotation schemas.
- Annotation Definition: explains the structure of an annotation definition JSON document.
- Creating an Annotation Stream: explains how to create a new annotation stream in PsiStudio.
- Annotation Editing: explains how to add, edit, and delete annotations in PsiStudio.
- Saving Changes: explains how to save your changes after editing.
Platform for Situated Intelligence Studio (or PsiStudio) enables the creation of time interval annotations, that is, annotated temporal events with specified start and end times. Time interval annotations may contain any number of tracks, and each track may use its own annotation schema. for example, the image below illustrates temporal annotations performed to identify utterances, together with their transcript (in the first track), and valence (in the second track).
Psistudio supports both unrestricted annotation schemas, in which the set of possible values an annotation might take is infinite - such as the transcriptions above, as well as finite annotation scehmas, where the set of possible values an annotation may take is known a priori - such as the valence above, which can have one of three possible values: positive, negative, or neutral.
An unrestricted annotation schema is one where no constraints are imposed on the possible values that an instance of the annotation schema may take. Annotation schemas can be defined in .json files, and an example of an unrestricted annotation schema object useful for performing transcriptions is shown below.
{
"$type": "Microsoft.Psi.Data.Annotations.AnnotationSchema`1[[System.String, mscorlib]], Microsoft.Psi.Data",
"Name": "TranscriptSchema",
"IsFiniteAnnotationSchema": false,
"DefaultValue": "Hello World",
"Metadata": {
"BorderColor": "LightSkyBlue",
"FillColor": "Cyan",
"TextColor": "Blue",
"BorderWidth": 1.0
}
}The $type property specifies the type of the annotation schema, all PsiStudio annotation schemas must derive from the generic Microsoft.Psi.Data.Annotations.AnnotationSchema<T>. In this case our schema is of type Microsoft.Psi.Data.Annotations.AnnotationSchema<string>, which is to say, each value in this annotation schema will be of type System.String. Since this is an unrestricted annotation schema, instances of this annotation schema may contain any System.String.
Next, the Name property specifies the name of this annotation schema definition.
The Default Value property indicates the default value that a newly created instance of this schema should take. In this case, whenever we add a new annotation in PsiStudio its initial value will be "Hello World".
The Metadata object allows us to specify several visual properties, including border color, border width, fill color, and text color of the time interval annotations that are created using this schema.
Finally, the IsFiniteAnnotationSchema property specifies whether or not this annotation schema is finite. In the next section we discuss how finite annotation schemas differ from unrestricted schemas.
A finite annotation schema is one where the set of valid values an annotation may take is finite and known a priori. An example of a finite annotation schema definition used to characterize the valence of an utterance as positive, negative, or neutral is shown below:
{
"$type": "Microsoft.Psi.Data.Annotations.FiniteAnnotationSchema`1[[System.String, mscorlib]], Microsoft.Psi.Data",
"Name": "ValenceSchema",
"IsFiniteAnnotationSchema": true,
"DefaultValue": "Neutral",
"Metadata": {
"BorderColor": "LightGray",
"FillColor": "DarkGray",
"TextColor": "White",
"BorderWidth": 1.0
},
"SchemaValues": [{
"Value": "Negative"
},
{
"Value": "Neutral"
},
{
"Value": "Positive",
"Metadata": {
"BorderColor": "Red",
"FillColor": "Red",
"TextColor": "White",
"BorderWidth": 1.0
}
}
]
}This JSON is very similar to the unrestricted annotation schema we defined in the previous section, but with a few differences.
The $type property for a finite annotation schema must derive from Microsoft.Psi.Data.Annotations.FiniteAnnotationSchema<T>, and in this case our schema is of type Microsoft.Psi.Data.Annotations.FiniteAnnotationSchema<string>.
The IsfiniteAnnotationSchema flag is set to true for a finite annotation schema.
Most importantly, the SchemaValues object defines all the possible values for an instance of a finite annotation schema. An instance of this schema may only take on one of the values defined by the SchemaValues object. In this case we have three possible (or valid) values; Negative, Neutral, and Positive. Note also that we have defined the color scheme for this finite annotation schema with the Metadata object in the same way as we did before - this specifies the default visual properties an annotation will use. Notice that we can also define a Metadata object for each of the individual values. This metadata will override the default. In this case, for the annotations where the valence is Positive, we've defined a separate color scheme so that positive values will stand out when we visualize the annotations stream in PsiStudio, as shown below.
An annotation definition is the complete definition of an annotation scheme and can contain any number of tracks contained by a single annotation instance. The structure of an annotation definition JSON file is shown below:
{
"Name": "<Annotation Definition Name>",
"SchemaDefinitions": [{
"Name": "<TrackName1>",
"Schema": <AnnotationSchema1>
}, {
"Name": "<TrackName2>",
"Schema": <AnnotationSchema2>
},
...
{
"Name": "<TrackNameN>",
"Schema": <AnnotationSchemaN>
},
]
}
Generally, the top-level Name property would be the same as the filename of the JSON file on disk. This is followed by the SchemaDefinitions collection, and each object in this collection has a track name followed by the annotation schema defined for the track.
We're going to create a two track annotation definition named TestDefinition in a file called TestDefinition.pad (all annotation definition files should have the .pad extension). The first track will be named Transcript and will use the TranscriptSchema we defined above. The second track will be named Valence and will use the ValenceSchema we defined above.
Putting everything together we end up with a complete annotation definition as follows:
{
"Name": "TestDefinition",
"SchemaDefinitions": [{
"Name": "Transcript",
"Schema": {
"$type": "Microsoft.Psi.Data.Annotations.AnnotationSchema`1[[System.String, mscorlib]], Microsoft.Psi.Data",
"Name": "TranscriptSchema",
"IsFiniteAnnotationSchema": false,
"DefaultValue": "Hello World",
"Metadata": {
"BorderColor": "LightSkyBlue",
"FillColor": "Cyan",
"TextColor": "Blue",
"BorderWidth": 1.0
}
}
},
{
"Name": "Valence",
"Schema": {
"$type": "Microsoft.Psi.Data.Annotations.FiniteAnnotationSchema`1[[System.String, mscorlib]], Microsoft.Psi.Data",
"Name": "ValenceSchema",
"IsFiniteAnnotationSchema": true,
"DefaultValue": "Neutral",
"Metadata": {
"BorderColor": "LightGray",
"FillColor": "DarkGray",
"TextColor": "White",
"BorderWidth": 1.0
},
"SchemaValues": [{
"Value": "Negative"
},
{
"Value": "Neutral"
},
{
"Value": "Positive",
"Metadata": {
"BorderColor": "Red",
"FillColor": "Red",
"TextColor": "White",
"BorderWidth": 1.0
}
}
]
}
}
]
}When PsiStudio starts up, it scans a subfolder of the Windows MyDocuments special folder (usually located in c:\Users\<User>\Documents\PsiStudio\AnnotationDefinitions) for annotation definition files, i.e. files with the .pad extension, and will automatically load into memory any that it finds. If you've previously run PsiStudio at least once then this directory will already exist, but if not you should create it now.
Next, paste the above annotation definition into a text editor such as Notepad and save the file to the above directory with the filename TestDefinition.pad.
Since PsiStudio only scans this directory at startup, if you already have PsiStudio running you should shut it down and restart so that this annotation definition is loaded.
Now we need an existing store to annotate against. For this example we're going to use the store we created in the Psi Studio sample, so start up PsiStudio and load that store. Drag the Audio stream into the visualization canvas, and then drag the Voice Activity stream into the same panel, as we're going to be annotating this audio stream using the voice activity detector messages to easily set the edges of our annotation events. PsiStudio should now look like this:
Now click the Create Annotations Stream button to launch the Add Annotation Stream dialog, it will look like the image below. There is a very important notice at the top of this dialog that should be explained. When \psi writes streams to a store, all the messages from all of the streams are written in an interleaved fashion, as they arrive, to maximize throughput. As a consequence however, when PsiStudio needs to save the changes to a single stream to the store, it must read and then rewrite ALL the streams in the entire store. If the store is large or contains many streams, this save operation could take a very long time. For this reason, we recommend that you store your annotation streams in their own store (which generally will be small in size.)
In the Stream Name field, write Annotations, then select the Create in new partition option button. The dialog will now change to look like the image below.
Give your new store a name and a path and then click OK to close the dialog and continue. You should see a new store in the dataset named AnnotationsTest containing a single annotations stream named Annotations.
If you now drag the Annotations stream into an empty area of the visualization canvas you should see a new timeline panel containing the Annotations stream. We haven't added any annotations yet, so the timeline panel will be blank.
To add a time interval annotation, we first create a selection that will define the start and end times of the annotation event. If necessary, use the mouse scroll wheel to zoom in to an interesting area of the streams. Then, while holding the SHIFT key, left-click where you want the annotation event to start; a green Selection Start marker will be dropped where you clicked. Now, while again holding the SHIFT key, right-click where you want the annotation to end; a red Selection End marker will be dropped where you clicked.
Now, in the timeline panel that contains the empty annotations stream, right click somewhere within the selection region and from the context menu select Add Annotation. A new annotation will be created between the start and end markers, with the default values in each track as specified by the schema.
Repeat the above steps a couple more times to create several annotations. Your stream should now look like the image below.
If you wish to set the start and end times of your annotations exactly on corresponding message times of one of the stores in the stream, then you can do the following. First use the mouse scroll wheel to zoom into an area of the timeline where the voice activity value transitions from 0 to 1. Now, click on the Voice Activity stream in the Layout tab to display its properties on the right of PsiStudio and change the Marker Style property to Circle. This will add a small marker to the voice activity plot for each message in the stream. Finally, right-click on the Voice Activity stream in the Layout tab and from the context menu select Snap to Stream. Notice that the icon for this stream now changes to indicate that we are snapping the cursor to this stream. If you then move the mouse over the voice activity stream in the timeline panel you should see that the timeline cursor now snaps from one message to the next as you move the mouse.
Currently all of our annotations have a Transcript value of Hello World and a Valence value of Neutral, so we need to start setting the correct values. Click on one of the annotations and its properties will be displayed in the Properties tab on the right. Notice that the StartTime and EndTime properties are displayed for the annotation as well as the current values for each track. Recall that the Transcript track is an unrestricted string annotation, so its value can be edited via a simple textbox. Changing the text of this property will be immediately reflected in the annotation itself. The Valence property, however, is a finite annotation, so it is rendered as a dropdown combobox. If you click on the current value you can select the appropriate value from the list of possible values in the dropdown. Recall also that we used a custom Metadata scheme for the Positive value of Valence, and setting the value to positive results in that track of the annotation instance taking on this color scheme. You can also right-click directly on an annotation and set the value from the resulting context menu.
You can also adjust the start or end time of a time interval annotation after you have created it. Hover the mouse over one of the edges of the annotation and the mouse cursor will change to indicate the annotation instance can be resized. You can then click and drag the edge of the annotation to a new time (if the cursor is set to snap to the messages in a certain stream, it will do so throughout moving and adjusting the edge of the annotation). Time interval annotations are constrained to be non-overlapping, so you can drag the edge of an annotation instance no further than the edge of an adjacent annotation instance.
Once you've dragged the edge of an annotation instance so that it touches the edge of an adjacent annotation instance, these two edges become "locked" to each other. If you subsequently use the mouse to drag this common edge, then the edges of both annotation instances will be dragged so that they stay locked together. If you wish to "unlock" these two annotation edges, hold down the ALT key before dragging and they will separate.
To delete an annotation, right-click on it and select Delete Annotation from the context menu.
If you have made any changes to an annotation stream, then in the Datasets tab an asterisk will appear next to the annotation stream that was editied, and also next to the partition that contains the annotation stream that was edited to denote that there are unsaved changes. Recall that the partition represents the physical store on disk, so the partition name is decorated with an asterisk to indicate that the entire store will be rewritten when you save.
To save your changes to disk, right-click on the Annotations Test partition node in the Datasets tab and from the context menu select Save Changes. A progress dialog will apprear while the store is rewritten with the changes you have made and will close once the save operation has completed.