visual-perception-functions.Rmd

---
title: "Estimating visual perceptions from tracking data"
author: "Eric R. Press" 
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Estimating visual perceptions from tracking data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteDepends{ggplot2}
  %\VignetteDepends{magrittr}
  %\VignetteEncoding{UTF-8}
---
```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Introduction

Studies of visually guided locomotion in birds, insects or even humans often
involve data gathered from motion capture technologies such as Optitrack's
[(Motive)](https://optitrack.com/software/motive/), or the Straw Lab's
[(Flydra)](https://github.com/strawlab/flydra). For these experiments, it is
important to understand how visual stimuli influence behavior. Although is it
not possible to measure how subjects directly perceive visual stimuli, it is
possible to use motion capture data to calculate estimates of stimulus
properties as they are perceived by the subject. With the tools available in
pathviewr, researchers in ethology or psychology may provide analyses of both
stimulus and response under natural locomotor conditions.
 
*Inherent to estimations of visual perceptions, we make several assumptions,
which will be discussed below. We welcome suggestions and aim to address any
assumptions that limit the accuracy of these estimates*.
  
To bridge the gap between objective measures of subject position and estimates
of subjective stimulus perception, we can begin by calculating the angle that a
visual pattern subtends on the subject's retina: the visual angle (θ). Visual
angles can be used to calculate aspects of image motion such as the rate of
visual expansion [(Dakin *et al.*,
2016)](https://www.pnas.org/doi/10.1073/pnas.1603221113/). For a detailed
review of different forms of visual motion and how they're processed by the
brain, see [(Frost, 2010)](https://www.karger.com/Article/Abstract/314284).
  
Visual angles can be calculated provided there is information about the physical
size of the pattern and its distance from the subject's retina. Because
researchers can control or measure the size of a pattern, and we can calculate
the distance between the subject and pattern using motion capture data, we can
further calculate the visual angle produced by patterns in the visual scene.
Therefore, we first need to calculate the distance from the subject's retina to
the pattern.
  
*Currently, we assume the subject's gaze is directly frontal or lateral to the
face, in effect estimating image properties at single points in the frontal and
lateral fields of view, respectively. We currently calculate distances to the
center of the subject's head rather than the position of the retina; future
versions of `pathviewr` will include features addressing these limitations, such
as including head orientation information and eye position relative to the
center of the subject's head*.
  
```{r, echo=FALSE, out.width="100%", fig.cap="Visual angles can be calculated using the size of a visual pattern (`stim_param`) and the distance to the pattern. Larger patterns at shorter distances produce larger visual angles. For dot stimuli, visual angles can be calculated independent of stimulus orientation."}
knitr::include_graphics("https://raw.githubusercontent.com/ropensci/pathviewr/dc728871873b92dc8412bdcb5ef58031457f6788/images/stim_param_angle.jpeg")
```

## Loading packages
To begin, let's load `pathviewr` and a few of the packages in `tidyverse`.

```{r package_loading, message=FALSE, warning=FALSE}
library(pathviewr)
library(ggplot2)
library(magrittr)
```


## Data preparation
Data objects must be prepared as described in [the Data Import and Cleaning
vignette](https://docs.ropensci.org/pathviewr/articles/data-import-cleaning.html)
pipeline prior to their use with these functions. For a detailed description of
the importing and cleaning functions, and when to use them, please see the
linked vignette.

Let's work with a few example datasets included in the package.
`pathviewr_motive_example_data.csv` is a `.csv` file exported from Motive.
`pathviewr_flydra_example_data.mat` is a `.mat` file exported from Flydra. For
more coarse-grained data cleaning tasks, `pathviewr` contains an all-in-one
cleaning function `clean_viewr`. We will use this function for the following
examples.


```{r}
## Import motive data set
motive_data <- # import
  read_motive_csv(
    system.file("extdata", "pathviewr_motive_example_data.csv",
                package = 'pathviewr')
  )

## Clean motive data set
motive_full <-
  motive_data %>%
  clean_viewr(
    relabel_viewr_axes = TRUE,
    gather_tunnel_data = TRUE,
    trim_tunnel_outliers = TRUE,
    standardization_option = "rotate_tunnel",
    select_x_percent = TRUE,
    desired_percent = 50,
    rename_viewr_characters = FALSE,
    separate_trajectories = TRUE,
    max_frame_gap = "autodetect",
    get_full_trajectories = TRUE,
    span = 0.95
  )
```
```{r}
## Import flydra data set
flydra_data <- 
  read_flydra_mat(
    system.file("extdata", "pathviewr_flydra_example_data.mat",
                package = 'pathviewr'),
    subject_name = "birdie_wooster")

## Clean flydra data set
flydra_full <- 
  flydra_data %>%
  clean_viewr(
    relabel_viewr_axes = FALSE,
    gather_tunnel_data = FALSE,
    trim_tunnel_outliers = FALSE,
    standardization_option = "redefine_tunnel_center",
    length_method = "middle",
    height_method = "user-defined",
    height_zero = 1.44,
    get_velocity = FALSE,
    select_x_percent = TRUE,
    desired_percent = 60,
    rename_viewr_characters = FALSE,
    separate_trajectories = TRUE,
    get_full_trajectories = TRUE
  )
```



## Add experiment information with `insert_treatments()`
Now that our objects have been cleaned, we must use `insert_treatments()` to add
information about the experiments, including relevant properties of the visual
stimulus and experimental tunnel that are necessary for calculating visual
perceptions.

*`pathviewr` currently supports rectangular (box) or V-shaped experimental
tunnels, though we are interested in including additional tunnel
configurations*.

If you would like to request any additional features such as other tunnel configurations, please create an issue [(here)](https://github.com/ropensci/pathviewr/issues/new/choose)

#### V-shaped tunnel example
The data within `motive_full` were collected from birds flying through a 3m long
V-shaped tunnel in which the height of the origin `(0,0,0)` was set to the
height of the perches that were 0.3855m above the vertex, which was angled at
90 degree. The visual stimuli on the positive and negative walls of the tunnel (where
`position_width` values > 0 and < 0, respectively) were stationary dot-fields.
Each dot was of size 0.05m in diameter. The visual stimuli on the positive and
negative end walls of the tunnel (where `position_length` > 0 and < 0,
respectively) were dot fields with dots 0.1m in diameter. This treatment was
referred to as `"latB"`.

Therefore we will use the following code:
```{r}
motive_treatments <- 
  motive_full %>% 
  insert_treatments(tunnel_config = "v",
                    perch_2_vertex = 0.3855,
                    vertex_angle = 90,
                    tunnel_length = 2,
                    stim_param_lat_pos = 0.05,
                    stim_param_lat_neg = 0.05,
                    stim_param_end_pos = 0.1,
                    stim_param_end_neg = 0.1,
                    treatment = "latB")
names(motive_treatments)
```
`motive_treatments` now has the variables `tunnel_config`, `perch_2_vertex`,
`vertex_angle`, `tunnel_length`, `stim_param_lat_pos`, `stim_param_lat_neg`,
`stim_param_end_pos`, and `stim_param_end_neg` which are needed to calculate
visual angles. The variable `treatment` has also been included and all of this
information has been stored in the object's metadata.


#### Box-shaped tunnel example
The data within `flydra_full` were collected from birds flying in a 1 x 3m
rectangular tunnel (a box). The visual stimuli were the same as in the motive
example.
```{r}
flydra_treatments <-
  flydra_full %>%
  insert_treatments(tunnel_config = "box",
                    tunnel_width = 1,
                    tunnel_length = 3,
                    stim_param_lat_pos = 0.05,
                    stim_param_lat_neg = 0.05,
                    stim_param_end_pos = 0.1,
                    stim_param_end_neg = 0.1,
                    treatment = "latB")
```
`flydra_treatments` similarly has the variables `tunnel_config`, `tunnel_width`,
`tunnel_length`, `stim_param_lat_pos`, `stim_param_lat_neg`,
`stim_param_end_pos`, `stim_param_end_neg` and `treatment`.



## Calculating visual angles
### Start by calculating distances to visual stimuli 

To estimate the visual angles perceived by the subject is it moves through the
tunnel, we first need to calculate the distance between the subject and the
visual stimuli. For this, we will use `calc_min_dist_v` or `calc_min_dist_box`
depending on the configuration of the experimental tunnel. These functions
calculate the minimum distance between the subject and the surface displaying a
visual pattern, therefore maximizing the visual angles.  

For V-shaped tunnels, several internal calculations are required and can be
added to the output object with `simplify_output = FALSE`. Otherwise, the
minimum distance are computed to the lateral walls and the end wall to which the
subject is facing.

```{r motive_min_dist_pos, fig.height=4, fig.width=7}
motive_min_dist <- 
  motive_treatments %>% 
  calc_min_dist_v(simplify_output = FALSE)

## Display minimum distances to the positive lateral walls 
## Viewpoint is from the end of the tunnel
motive_min_dist %>% 
  ggplot(aes(x = position_width, y = position_height)) +
  geom_point(aes(color = min_dist_pos), size = 2, shape = 1) +
  coord_fixed() +
  theme_classic() +
  geom_segment(aes(x = 0,         # positive wall
                   y = -0.3855,
                   xend = 0.5869,
                   yend = 0.2014)) +
  geom_segment(aes(x = 0,         # negative wall
                   y = -0.3855,
                   xend = -0.5869,
                   yend = 0.2014))
```

```{r flydra_min_dist_end, fig.height=4, fig.width=7}
flydra_min_dist <- 
  flydra_treatments %>% 
  calc_min_dist_box()

## Display minimum distances to the end walls
## Viewpoint is from above the tunnel
flydra_min_dist %>% 
  ggplot(aes(x = position_length, y = position_width)) +
  geom_point(aes(color = min_dist_end), size = 2, shape = 1) +
  coord_fixed() +
  theme_classic() +
  geom_segment(aes(x = -1,         # negative wall
                   y = -0.5,
                   xend = 1,
                   yend = -0.5)) +
  geom_segment(aes(x = -1,         # positive wall
                   y = 0.5,
                   xend = 1,
                   yend = 0.5))
```


### Now get visual angles 

```{r motive_vis_angle_pos, fig.height=4, fig.width=7}
motive_vis_angle <- 
  motive_min_dist %>% 
  get_vis_angle()

## Visualize the angles produced from stimuli on the positive wall
## Viewpoint is from the end of the tunnel
motive_vis_angle %>% 
  ggplot(aes(x = position_width, y = position_height)) +
  geom_point(aes(color = vis_angle_pos_deg), size = 2, shape = 1) +
  coord_fixed()+
  theme_classic() +
  geom_segment(aes(x = 0,         # positive wall
                   y = -0.3855,
                   xend = 0.5869,
                   yend = 0.2014)) +
  geom_segment(aes(x = 0,         # negative wall
                   y = -0.3855,
                   xend = -0.5869,
                   yend = 0.2014))
```

Notice larger visual angles as the subject approaches the positive wall. 

```{r flydra_vis_angle_end, fig.height=4, fig.width=7}
flydra_vis_angle <- 
  flydra_min_dist %>% 
  get_vis_angle()

## Visualize the angles produced by stimuli on the end walls
## Viewpoint is from above the tunnel
flydra_vis_angle %>% 
  ggplot(aes(x = position_length, y = position_width)) +
  geom_point(aes(color = vis_angle_end_deg), size = 2, shape = 1) +
  coord_fixed() +
  theme_classic() +
  geom_segment(aes(x = -1,        # negative wall
                   y = -0.5,
                   xend = 1,
                   yend = -0.5)) +
  geom_segment(aes(x = -1,        # positive wall
                   y = 0.5,
                   xend = 1,
                   yend = 0.5))
```

Notice larger visual angles as the subject approaches the end wall that it is
moving towards.

## Calculating spatial frequency
With visual angles, we can now determine the spatial frequency of a visual
pattern as it is perceived by the subject. Spatial frequency refers to the size
of the pattern in visual space. It's often reported as the number of cycles of a
visual pattern per 1 degree of the visual field (cycles/degree). Here, we will define
a cycle length as the length used for the `stim_param`. For a given distance
from the subject, a larger visual pattern produces a smaller spatial frequency,
whereas a smaller visual pattern produces a larger spatial frequency.

To calculate the spatial frequency of the visual stimuli as perceived by the
subject some distance from the stimuli, we will use `get_sf()`.

```{r motive_sf_pos, fig.height=4, fig.width=7}
motive_sf <- 
  motive_vis_angle %>%
  get_sf()

## Visualize the spatial frequency of the stimulus on the positive wall 
## point is from the end of the tunnel
motive_sf %>% 
  ggplot(aes(x = position_width, y = position_height)) +
  geom_point(aes(color = sf_pos), size = 2, shape = 1) +
  coord_fixed()+
  theme_classic() +
  geom_segment(aes(x = 0,         # positive wall
                   y = -0.3855,
                   xend = 0.5869,
                   yend = 0.2014)) +
  geom_segment(aes(x = 0,         # negative wall
                   y = -0.3855,
                   xend = -0.5869,
                   yend = 0.2014))
```

Notice the spatial frequency increases the further the subject recedes from
positive wall.

```{r flydra_sf_end, fig.height=4, fig.width=7}
flydra_sf <- 
  flydra_vis_angle %>% 
  get_sf()

## Visualize the spatial frequency of the stimulus on the end walls
## Viewpoint is from above the tunnel
flydra_sf %>% 
  ggplot(aes(x = position_length, y = position_width)) +
  geom_point(aes(color = sf_end), size = 2, shape = 1) +
  coord_fixed() +
  theme_classic() +
  geom_segment(aes(x = -1,        # negative wall
                   y = -0.5,
                   xend = 1,
                   yend = -0.5)) +
  geom_segment(aes(x = -1,        # positive wall
                   y = 0.5,
                   xend = 1,
                   yend = 0.5))
```

Notice the spatial frequency decreases as the subject approaches the end wall
that it is moving towards.

### Stay tuned as additional features for image motion estimation are coming soon!