2.towardsERPs.tex

\section{Towards ERPs as a metric for haptic immersion}

The goal of haptic devices is to render realistic sensory feedback that mimics the sensory experience a user would normally perceive when interacting with the real world. The simple case that we examine here is of \textit{touching an object}. Imagine grasping a cup of coffee on the breakfast table: as we reach out for the cup, our visual system provides ongoing feedback about the position of the arm and hand relative to the cup on the table, while proprioceptive feedback from muscles and joints provides information about the relative position of the hand and the strength of our grasp. Combined with the motor plan, the sensory feedback is used to compare what is \textit{effectively happening} in the environment with what was \textit{predicted to happen}~\cite{clark_whatever_2013}. When making contact with the cup, the visual and proprioceptive feedback are integrated with haptic feedback providing information about the contact with the object. In the case where all sensory information channels provide consistent feedback, the action would be successful (and the coffee can be enjoyed). However, in the case of a \textit{mismatch} in the incoming information, attention has to be directed to this mismatch so that the action can be corrected in real-time~\cite{savoie_visuomotor_2018}. It is precisely this idea -- that the brain has evolved to optimize motor behavior based on detected sensory mismatches -- that inspired us to investigate brain responses to sensory mismatches as a potential metric for haptic immersion.

To investigate these prediction errors in VR, an electroencehalogram (EEG) can be used. An EEG measures the electrical activity of cortical neurons in the human brain with high temporal resolution~\cite{nunez_electric_2006}. Transient sensory events (e.g., haptic feedback from touching a coffee mug) evoke event-related potentials (ERPs) in the ongoing oscillatory activity of the brain, reflecting sensory and cognitive processing of incoming stimuli~\cite{luck_introduction_2014}. An ERP is a stereotyped response comprised of a series of positive and negative deflections. One specific component of the ERP is the \textbf{prediction error negativity} (PEN)---a negative potential that occurs from 100 to 200 ms whenever a deviation from the predicted state of the environment is detected~\cite{savoie_visuomotor_2018}. We propose utilizing this prediction error (highlighted in Figure\ref{main_fig_diff_erp}) as a metric for haptic immersion, a potential immersion indicator that does not require subjective interpretation or interrupting the user, which results in breaking the immersive experience. Furthermore, as we will discuss, this metric can be used in realtime to continuously adapt an environment depending on the users prediction of and actual state of the environment.

To actualize this proposal, we conducted a user study in which we measured the brain activity of 11 participants using a 64 channel EEG system. During the VR experience, participants touched different virtual objects with each touch being accompanied by feedback via the incremental combination of popular feedback modalities: (1) visual feedback, (2) tactile (via vibration) + visual feedback, and (3) force feedback (via EMS) + tactile + visual feedback. To provoke the participant's brain into processing the experience of an unrealistic VR interaction, we provided the haptic feedback prematurely in 25\% of the trials. When comparing these ERPs to the ERPs from realistic interactions, we found that the amplitude of the PEN increased, indicating that we can successfully detect error processing hinting at a loss in immersion, without interrupting the VR experience. Furthermore, we found that this error prediction systematically covaried with the number of feedback channels. Before detailing our experiment, we will leverage the HCI and neuroscience literature to ground our ERP-based approach.