Meta issue: eye-tracking todos

[scott-huberty]

I'm going to open a meta-issue here to keep track of the features proposed in my GSoC, plus the other work that has cropped up during my work so far. I think at this point the work will extend beyond august but I'm happy to continue working on this after my GSoC finishes.

Maintenance

• Per BUG: Some raw classes use attributes rather than ._raw_extras #11825, make sure that the RawEyeLink doesn't have unwanted private attributes attached to it.
• make resampling safe for eyetracking data: finish [WIP] ENH: resampling with annotations #11408 (Update: hopefully solved with ENH: Add polyphase resampling #12268)
• refactor raw.plot to map a sensor's FIFF_UNIT to a sensible scaling.

Eyetracking features

• gaze heatmap: finish ENH: eyetracking plot_heatmap function #11798
• Unit conversion helper function (i.e. pixels to radians) ( Function to convert eyegaze units to radians #12237 )
• pupil deconvolution ( Feature: Pupil Deconvolution (port from pyeparse) #12505 )

[larsoner]

#11408 is probably going to be tough... I'd suggest doing that one last. In theory the "only" problem with resampling while ignoring annotations (like in main) is that artifacts can spread a bit. Hopefully it's not too bad for eyetracking data -- have you tried on your data and it's bad @scott-huberty ?

refactor raw.plot to map a sensor's FIFF_UNIT to a sensible scaling.

This one will also require changing a lot of plotting functions. So I would also put this down the queue a bit, especially given that the workaround of passing scalings isn't too onerous in the meantime.

So I would recommend first _raw_extras would be a quick one and worthwhile, then the new features. Then the more systematic stuff... but open to other ideas or changing based on your personal preference as well @scott-huberty

[scott-huberty]

So I would recommend first _raw_extras would be a quick one and worthwhile, then the new features. Then the more systematic stuff... but open to other ideas or changing based on your personal preference as well @scott-huberty

works for me!

Hopefully it's not too bad for eyetracking data -- have you tried on your data and it's bad @scott-huberty ?

Yeah - I've run into issues resampling eyetracking data while using mne.preprocessing.realign_raw.

tl;dr - in practice, realign_raw always resamples other (in my experience), even though the two raw objects in the example below have the same sfreq (1000 Hz). If the RawEyelink object is other, and has NaN values within BAD_ACQ_SKIP periods, resample, which isn't annotation aware, will spread the nan values throughout the signal.

For example you can run this code below:

Note that the really simple work around is just to make sure the raw_eeg is other (that's what we do in our eyetracking tutorial), so that the eeg is what gets resampled and everything is fine. I just bring this up because at some point someone might experience an issue while trying to resample eyetracking data.

import mne
from mne.datasets.eyelink import data_path
from mne.preprocessing.eyetracking import read_eyelink_calibration

# Get sample data
et_fpath = data_path() / "sub-01_task-plr_eyetrack.asc"
eeg_fpath = data_path() / "sub-01_task-plr_eeg.mff"

# Load Raws
raw_et = mne.io.read_raw_eyelink(et_fpath, create_annotations=["blinks", "saccades"])
raw_eeg = mne.io.read_raw_egi(eeg_fpath, preload=True, verbose="warning")

# interpolate blinks
mne.preprocessing.eyetracking.interpolate_blinks(raw_et,
                                                 buffer=(0.05, 0.2),
                                                 interpolate_gaze=True)

# Get events
et_events = mne.find_events(raw_et, min_duration=0.01, shortest_event=1, uint_cast=True)
event_dict = {"Flash": 2}
eeg_events = mne.find_events(raw_eeg, stim_channel="DIN3")

# Convert event onsets from samples to seconds
et_din_times = et_events[:, 0] / raw_et.info["sfreq"]
eeg_din_times = eeg_events[:, 0] / raw_eeg.info["sfreq"]

# Align the data
mne.preprocessing.realign_raw(raw_eeg, raw_et, eeg_din_times, et_din_times) # EDIT. had parameters in wrong order

# Add EEG channels to the eye-tracking raw object
raw_et.add_channels([raw_eeg], force_update_info=True)
%matplotlib qt
raw_et.plot(scalings=dict(eyetrack=1e3))

[larsoner]

If the RawEyelink object is other, and has NaN values within BAD_ACQ_SKIP periods, resample, which isn't annotation aware, will spread the nan values throughout the signal.

Ahh right nan will propagate... for now a workaround would be to do interpolation of any channels with NaN before resampling.

[britta-wstnr]

Wait, but does that mean that blinks always have to be interpolated before combining raws, @larsoner ?

[larsoner]

Yes until the resampling PR is updated. Which is less than ideal but if we add an option to keep the BAD_blink annotations even after interpolation it gets less ugly because that way you can interpolate just to make resample happy and still omit those segments from your data

[britta-wstnr]

but if we add an option to keep the BAD_blink annotations even after interpolation it gets less ugly because that way you can interpolate just to make resample happy and still omit those segments from your data

Yeah, that would be good! Needs thorough documentation then as well I think as this is somewhat of a "hack" until resampling works - and might not be an intuitive approach to everyone (e.g. me 🙃 )

[scott-huberty]

And just one note that I think I mentioned earlier - if the Eye-tracking raw object has the same sfreq as the EEG raw, or a lower sfreq than EEG raw, users can just pass the EEG raw as the second argument to realign_raw (the other keyword argument), like:

mne.preprocessing.realign_raw(ET_raw, EEG_raw)

This way the resample function will be called on the EEG raw, sidestepping the issue of blink periods periods in the eye-tracking object. In this case, you don't have to interpolate blinks before calling realign_raw.

I am happy to improve the documentation for this. Maybe we can add it to the tutorial, @britta-wstnr what do you think?

P.S. sorry I couldn't make it to office hours today, I had a meeting at the same time!

[britta-wstnr]

Hi @scott-huberty - sorry I missed this in my notifications!!
I think a tutorial about this would be great, did you mean adding it here: https://mne.tools/stable/auto_tutorials/preprocessing/90_eyetracking_data.html#align-the-eye-tracking-data-with-eeg-the-data ?
I feel like a NOTE or WARNING box about the resampling there plus some tips and tricks how to deal with it would be great - which we can adapt then once there is progress on the resampling.