3. The Review Ratings Window
The Review Ratings window allows users to view and compare previously saved annotation files, as well as watch media files without recording ratings. The Review Ratings window opens to display a mostly blank window with some rectangular panes, some buttons, and a menu bar.
To open a media file, users should select Open Media File from the Media menu and select the desired audio or video file using the prompt. This will display the first frame of the media file in the black pane. Users can then press the Play button to watch the media file. Left-clicking on the rating axes at the bottom of the screen will jump/seek the media file to the corresponding time in the media file.
To load one or more annotation files, users should select Add Annotation File(s) from the Annotations menu or left-click the Add Annotations button. More than one annotation file can be loaded at once by selecting multiple files in the prompt (e.g., by holding the Ctrl or Shift keys). This operation can be performed before, after, or without opening a media file. In order to be viewed simultaneously, all loaded annotation files must have the same length (i.e., sampling rate and duration).
Once loaded, the name(s) of the annotation file(s) will appear on the upper right pane with a colored index number next to it. The index number corresponds to the number in the boxplot displayed in the summary pane. The color corresponds to the color of annotation series displayed in the time-series plots in the rating axis. Previously added annotation files can be removed by selecting them in the upper right pane and selecting Remove Selected File/All Files from the Annotations menu or left-clicking the Remove Selected/All Files buttons.
Users can left-click the Show/Hide Mean Plot button to toggle the mean plot display. When toggled on, the mean of all loaded annotation series will be displayed in red and all previously loaded annotation series will be changed to light grey. Users can export this mean series by selecting Export Mean Series from the Annotations menu.
The Analyze Ratings button opens a new window that displays information about each loaded annotation file's Mean and Standard Deviation (SD) for each dimension. The overall reliability for each dimension is also displayed. Various Intraclass Correlation Coefficients are used as estimates of inter-rater agreement and reliability (McGraw and Wong, 1996), while various RWG scores are used as estimates of inter-rater agreement in the presence of restricted variance (James, Demaree, and Wolf, 1984; LeBreton et al., 2003).
The Review Ratings window before any media or annotation files are loaded
The Review Ratings window with five annotation files loaded
The Review Ratings window with the mean plot of five annotation files
The Analyze Ratings window for the five example annotation files