Options and Parameters

Contents

1. Tracking Parameters
2. Output Options
   • Output Fields
   • Post-Processing
3. Converting pixel units to mm
4. Camera Calibration
5. Vignette Correction
6. Video Recording
7. Tracking Performance

Overview

MARGO understands that the setup and goals of tracking can vary greatly from instance to instance. For that reason, many tools have been included to offer flexibility and customization in how objects are tracked and what data is output. Tracking quality can often be improved by understanding how to utilize MARGO's parameters and utilities.

Tracking Parameters

Many of tracking parameters are used to track, identify, or sort ROIs and tracked objects. These parameters can be adjusted from the main tracking console or under Options > Tracking.

• auto estimate traces per ROI - toggles automatic estimation of the number of objects to track in each ROI during reference initialization (multitrack mode only). Note: once the trace number is estimated during referencing, it is fixed during tracking.

• adjust difference image contrast - adjusts the luminance of the background subtraction image to improve the range of functional image segregation parameters. This operation saturates the top and bottom 1% of pixel values in the image and may dramatically increase the level of background pixel noise. Note: for the option to work as intended, toggle it before background reference initialization and noise sampling.

• background color - defines the expected color (light or dark) of the background or tells MARGO to attempt to automatically detect the background color during reference initialization. In practice, this parameter determines whether the current image is subtracted from the background reference image (best for dark objects on a bright background) or the background reference image is subtracted from the current image (best for light objects on a dark background).

• detection mode - sets the mode by which ROIs are defined (grid or auto). See ROI detection for more details

• dilation radius - sets the radius (pixels) of the structuring element used to dilate the thresholded difference image. Dilation helps fill in gaps between above threshold pixels that, ideally, should belong to the same blob. This is particular helpful in under-illuminated images where tracked individuals are split into multiple nearby blobs. Increasing this value may help stitch nearby pixel islands into the same blob.

• distance threshold - maximum allowed distance to the center of an ROI. This parameter is used to identify which ROI to assign blob centroids. This parameter is only used when distance centroid sort mode is selected. Note: toggling the distance threshold radio button will display a circle showing the range of inclusion for each ROI.

• enable noise detection - toggles sampling of above threshold pixels in the thresholded background subtraction image. With noise detection enabled, the distribution of above threshold pixels during tracking will be constantly compared to clean reference distribution acquired during noise sampling. Deviation of the above threshold pixel distribution from the reference sample greater than skip frame &sigma or reset background &sigma will result in skipping of single frames or automatic reacquisition of the background reference image, respectively.

• erosion radius - sets the radius (pixels) of the structuring element used to erode the thresholded difference image. Erosion helps separate blob elements that, ideally, should not belong to the same blob. This is particularly helpful in images where nearby tracked individuals are merged into a single blob. Increasing this value may help separate nearby objects into the serparate blobs.

• fixed number of traces per ROI - sets the maximum number of trackable objects in each ROI (multitrack mode only). Setting the number of traces only fixes the upper limit on how many objects can be tracked in any given ROI and does not guarantee that any specific number of objects will be tracked on any given frame. MARGO will track as many valid objects appear in the ROI up to the maximum allowed number. If more objects than the maximum appear, precedence will be given to objects with the longest lifespan (i.e. appearing the most consistently frame to frame).

• frames to sample - sets the number of frames to sample the number of above threshold pixels from threshold background subtraction image during noise sample.

• interfram speed thresh - maximum allowed distance traveled per second. Each update in an object's position is time-stamped. Every frame, MARGO starts with unassigned blobs in a difference image. To assemble blob positions into traces over time, tracked objects are assigned to an ROI. The speed threshold excludes centroids in any given frame that have moved too far from the last recorded centroid for the paired ROI. This parameter helps prevent the centroid from jumping too far from frame to frame. Note: toggling the speed threshold radio button after ROI detection and referencing will display a measured rolling average speed and standard deviation for each ROI to help with parameter setting.

• min/max blob area - these parameters set lower and upper bounds that blobs in the threshold image will be tracked. Objects in the image outside of this range are excluded from tracking in the current frame. Note: toggling the area threshold radio button after ROI detection and referencing will display a measured rolling average area and standard deviation for each ROI to help with parameter setting. This will also display concentric circles of area equal to the specified bounds.

• missing trace noise - defines the noise sampling behavior for apparently empty ROIs. By default, MARGO will estimate the amount of noise that should be generated by ROIs with no tracked objects by bootstrap resampling from the distrbution of ROIs with tracked objects. ROIs may appear empty for two reasons: 1.) because the ROI is actually empty 2.) a stationary object has been referenced into the background. The second reason can generate undersampled noise distributions if many tracked objects are referenced into the background during reference initialization and noise sampling but move into the foreground during acquisition. Recommended behavior is to resample when few or no ROIs are empty and do nothing when many ROIs are empty.

• reference computation - sets the function (mean or median) used to calculate a background reference image from a stack of images.

• reference mode - sets the mode for background reference image calculation (live or video). Live mode allows for continuous updating of the background reference image at fixed time intervals or when difference image noise exceeds the limits set during noise sampling. Video referencing mode is only available in offline tracking of videos and constructs a background reference image by randomly sampling frames from the entire video.

• ROI clustering tolerance - sets the maximum number of standard deviations in vertical distance for adjacent ROIs to be assigned to the same ROI. To automatically assign numbers to ROIs when using auto ROI detection mode, MARGO first sorts ROIs by their vertical center of mass and measures the vertical distance between adjacent ROIs. Any pair of ROIs that are within the maximum number of standard deviations of one another are said to belong to the same row. Decreasing this value will make vertical clustering of ROIs more stringent. Increasing the value will cluster ROIs that are more vertically separated into the same row of ROIs. Note: this parameter is only used during automatic ROI detection and does not apply to grid mode detection.

• sorting criteria - sets mode by which centroids are assigned to ROIs. Distance and bounds sorting modes requires centroids to be within the specified distance of the center or fall within the bounds of an ROI, respectively. Centroids not meeting the requirement are not eligible for assignment to that ROI. Distance mode is most useful with round shaped ROIs. Bounds mode is useful for ROIs that have an asymmetric width and height.

• target acquisition rate - sets the maximum acquisition rate of tracking in Hz. Adjusting this parameter only ensures that tracking will not exceed this rate. The maximum achievable rate will depend on computer hardware, image resolution, number of objects tracked and centroid sorting mode. For more information on optimizing acquisition rate, see improving tracking performance.

• tracking mode - sets the tracking mode (single or multitrack) to allow a single or multiple objects to be tracked in each ROI. Individual identities are not maintained in multitrack mode. Note: parameters under the Multi-tracking panel are disabled while the tracking mode is set to single.

• vignette gaussian weight - sets the multiplicative weight of a gaussian mask applied to the entire image during ROI detection (only used for auto vignette correction). Detecting all ROIs in an image of varying luminance can be challenging. Once MARGO knows which parts of the image are foreground and which are background, a vignette filter can easily be calculated. Before foreground and background are distinguished, some assumptions can help smooth the luminance profile of the image. Many camera lenses and luminance sources create images that are slightly brighter in the center and dimmer at the edges. By default, MARGO assumes a gaussian profile of global luminance in the image and applies a filter to the image to smooth the global luminance. A lower weight will reduce the smoothing applied to the image.

• vignette gaussian sigma - sets the standard deviation of the above vignette correction gaussian as a fraction of the image height.

Output Options

The outputs menu allows user to configure raw data outputs and post-processing and can be accessed under Options > outputs. The outputs menu contains two main panels: 1) Output Fields where the user can select additional blob features to output as raw data files and 2) Post-processing where the user can toggle post-processing and select which operations are performed or which features are calculated in post-processing.

Output fields

Each of the following fields can be output to raw data binary files during acquisition. The data can be access through margo's ExperimentData object methods. Alternatively, the dimensions (N = number of frames, M = number of ROIs) and data type of the data are provided for users who want to directly access the via the binary data files.

Name	Description	Dimensions	Data Type
Speed	speed of each object in each frame in pixels s-1 (default) or mm s-1	N x M	Single
Area	size of each object in each frame in pixels2 (default) or mm2	N x M	Single
Orientation	angle between the major axis of the fitted ellipse of each object and the x-axis in each frame in degrees (-90 to 90)	N x M	Single
Weighted Centroid	centroid (x,y) of each object in each frame weighted by the luminance of each pixel in the background subtraction image	N x 2 x M	Single
Major Axis Length	length (pixels or mm) of the major axis of ellipses fitted to each object in each frame	N x M	Single
Minor Axis Length	length (pixels or mm) of the minor axis of ellipses fitted to each object in each frame	N x M	Single

Post-processing

Disclaimer: the following post-processing options are included as an unofficial and unsupported feature of MARGO. We use them for our own analysis and have included them in the event that others may find them useful.

• disable - toggles all post-processing.

• handedness - calculates ROI circling handedness index (-1 to 1), which quantifies the relative abundance of clockwise and counter-clockwise circling relative to the center of the ROI. Result stored in expmt.meta.handedness.

• parse movement bouts - parses speed data of each object into discreet bouts defined by a starting and stopping index for each movement bout .Result stored in expmt.data.speed.bouts.

• model/correct lens distortion - models the fisheye lens distortion as a second order polynomial and regresses out the distortion from the raw speed data. The result is the model residuals stored in a raw data binary file, ../raw data/regressed_speed.bin.

• bootstrap speed - constructs a bootstrapped null model by constructing simulated speed traces. Traces are generated by resampling parsed movement bouts with replacement from a shared pool of all bouts up to the number of frames in the observed data. Result stored in expmt.data.speed.bs.

• speed sliding average - calculates a moving average of speed data by sub-sampling over a sliding window, useful for capturing slow movement dynamics over longer timescales. Result stored in expmt.meta.Circadian.

• parse ceiling/floor bouts - uses the area of each object on each frame to classify individual frames for each object as occuring on the floor or ceiling of the arena. Because this method uses area as the splitting criteria, there must be a sufficient difference in the apparent size of the object when on the ceiling vs the floor.

Name	Description	Dimensions	Data Type
Speed	speed of each object in each frame in pixels s-1 (default) or mm s-1	N x M	Single
Heading Direction	angle (radians) of the heading direction of each object in each frame (-&pi to &pi)	N x M	Single
Radius	radial position (0-1) of each object in each frame, equivalent to the normalized radial polar coordinate relative to the ROI center	N x M	Single
Four-Quadrant Arctangent	angle (radians) between each object and the center of its ROI in each frame (-&pi to &pi), equivalent to the angular polar coordinate relative to the ROI. Reported angles are relative to the positive x-axis with the center of the ROI defined as the origin.	N x M	Single

Converting pixel units to mm

By default MARGO measures distances in pixel units. A tool is included to allow calculating a pixel/millimeter conversion factor by drawing a line over a known distance in the image (e.g. the diameter of an arena or length of the platform). To estimate a pixel to mm unit conversion factor:

1. Open the unit conversion tool under Options > distance scale

2. Enter the size of the known target length in millimeters

3. Click Draw a new line, then click and drag a line over the target length in the camera preview window

4. If necessary, reposition the line end points and select Update to calculate a new conversion factor

5. Close distance scale window to accept the millimeter per pixel unit conversion

For more accurate calibration, ensure the following:

• behavioral platform is perpendicular to the camera

• conversion factor is reacquired if the camera adjusted

• camera is calibrated to correct for fisheye lens distortion

Camera calibration

Camera parameter objects output by MATLAB's camera calibrator app can be used to correct fisheye lens distortion in images. To use camera calibration in MARGO, follow the instructions in MATLAB's tutorial to calculate and export camera parameters to correct images. Save the exported object to ../margo/hardware/camera_calibration/ in a .mat file. Create the camera_calibration directory if necessary. MARGO will automatically look for camera parameter objects in the specified directory upon initialization. Once the camera parameter object is exported and saved, toggle calibration under Hardware > camera > use calibration.

Vignette correction

Variation in baseline luminance across images can make it difficult to cleanly separate out tracked objects and ROIs with a single threshold value. MARGO uses vignette correction to increase the dynamic range over which objects can be separated from the background. To achieve this, the software employs two different strategies to smooth the baseline illumination of all ROIs in the image.

• Gaussian correction is utilized in auto ROI detection mode prior to initial detection of ROIs. Uneven illumination occurs most commonly as vignetting (ie. light fall-off) at the edges of image (fig 5.4). MARGO uses the assumption of a gaussian profile of vignetting centered on the image to smooth out the illumination. The width and weighting of the gaussian can be adjusted under tracking parameters. Figure 5.4 illustrates how correcting vignetting can allow all ROIs in the image to be cleanly separated from the background.

• ROI-based correction can only be employed after ROI detection because it relies on using ROI definitions to pick out dim ROIs in the image. Using this information, MARGO generates a subtraction matrix from the reference image. This matrix reduces all bright areas of image to the brightness of the dim ROI. This has the advantage of not assuming any particular shape to the variation in luminance across the image and often results in more even illumination than gaussian correction. A sample dim ROI can be picked manually by selecting Tracking > vignetting. Click and drag in the camera preview window to select a region of the image at the target brightness.

Video recording

Raw video data can be streamed to disk simultaneous with tracking. Video recording options can be accessed under Options > video. Videos will be output to the same selected save directory as all other data outputs. Video recording must be configured prior to tracking.

• record video - toggles video recording

• enable compression - toggles moderate compression of recorded videos

• image data source - sets the source of the image data to save each frame (e.g. raw image or background subtraction image)

• enable video sub-sampling - allows frames of output video to be saved at a different rate (between 0Hz and the acquisition rate) by sub-sampling the image source

• output video index - toggles outputting a raw data file that records which frames during acquisition were sub-sampled for video recording. The output binary data file will be N x 1, where N is the raw number of frames during acquisition and the value at each position indicates whether or not the frame was output to video (frame recorded = 1, frame not recorded = 0).

• sub-sampling rate - defines the target rate of the output sub-sampled video. Margo will attempt to approximate the sub-sampling rate as closely as possible by sub-sampling frames that are acquired at the acquisition rate.

Uncompressed video files are useful for software packages that train behavioral classifiers on the raw pixel data such as the Janelia Automatic Animal Behavior Annotator (JAABA). Videos can additionally be used for manual annotation behaviors or fed back into MARGO or other tracking programs for independent validation of traces. We find that videos can generally be significantly compressed without substantial impact on tracking performance.

Tracking performance

Low frame rates are often sufficient to capture behavior and can help reduce files sizes and data processing. But closed-loop stimulus delivery may demand acquisition rates that far exceed these rates. MARGO's real-time tracking excels at applications requiring tight closed-loop feedback. MARGO can often achieve high acquisition rates with some optimization. The following is a list of things that affect tracking speed roughly ordered by overall importance:

• image resolution

• camera frame rate

• display update mode (disabling the under Display > none increases tracking speed)

• number of ROIs/objects tracked

• tracking mode (multitrack mode typically slower than single)

• low target rate set under Tracking > tracking parameters

• background pixel noise

• blob dilation/erosion (disabled by setting dilation/erosion radius = 0)

• grid ROI detection mode

• unoptimized minimum and maximum area thresholds