6 Video, audio, mocap

This example shows how it is possible to use MGT for the combined analysis of visualization of video data and video and audio. As opposed to the previous example, this requires using the MGT data structure for handling synchronization of the data streams.

Reading in multiple data files

We start by reading in three files from the same recording session: video (.mp4), audio (.wav) and motion capture data (.tsv):

mg = mgread('pianist.avi','pianist.wav','pianist.tsv');

This leads to the structure mg containing the following information:

These are three very different types of files, and they, therefore, have to be treated differently inside of Matlab. The aim is that MGT takes care of most of these differences to work on the MGT structure directly.

Trimming the structure

The benefit of using an MGT structure is that it will handle the trimming of all related files and data:

mgseg = mgreadsegment(mg,10,30);

Here mgseg is a new (and complete) structure used for further processing.

mgsegmo = mgmotion(mgseg,'Diff','Regular',0.1);

Plotting together

The function mgvideoplot plots the horizontal and vertical motiongram combined with mocapgram over time.

mgvideoplot(mgsegmo);

Next, we can investigate the audio recording. mgwaveplot plots the waveform of audio, root mean square and their spectrum.

mgwaveplot(mgsegmo);

Periodicity

Next, the periodicity can be investigated from the quantity of motion:

[per,ac,eac,lag] = mgautocor(mgsegmo,'video');

The autocorrelation function for the data shows periodicities:

plot(lag, ac), xlabel('Period / secs')

The enhanced autocorrelation function for the same data looks like this:

plot(lag, eac), xlabel('Period / secs')

Statistics

If we want to look at some features of the motion image, function mgstatistics can generate first-order and second-order features:

features1 = mgstatistics(mgsegmo,'Video','FirstOrder');
features2 = mgstatistics(mgsegmo,'Video','SecondOrder');