Timing with the Bonsai Shaders Library

[swilshin]

We're running into an odd timing issue with the Bonsai Shaders Library and was hoping someone might have a solution. The timing between our arduino and our render drift out of synch.

Use Case:
We want to present a sequence of grating (black and white lines moving atan angle in a window, one grating every 2 seconds) while at the same time setting a pin on an arduino to high for two seconds then low for two seconds, repeating. The rising edge on the pin on the arduino would represent the onset of a new four second stimulus of two grating, and the falling edge the halfway point.

Problem:
The visual stimuli is from BonVision library and the TTL pulse on an arduino is created using the Bonsai Arduino library. Script is depicted below:

We create a window to do the rendering in and load the relevant resources. Then we load a collection of parameters for our grating from a CSV file, do a little processing and make a grating sequence. The top set of nodes render the grating while the second line of nodes controls the state of a pin on the arduino.
Everything except the second group of nodes seems to be working correctly. We create our set of gratings and render them onto the screen, they all look good. The LED we have hooked up to our pin on the Arduino lights up at the right time to start with, some temporal variance is to be expected since the arduino wont respond instantly over USB, but at the start of the sequence performance is pretty decent (it is not visiually possible to see a drift between the LED attached to the arduino and the onset of a change in the grating being rendered). The problem is that the pin on the arduino drifts out of synch with the presented grating over time. Over about 20 seconds it goes from being visually in synch, to being about 100-200ms out of synch. The blinking of the LED consistently drifts behind the presented visual stimulus.
The python node in our script is nothing fancy, it just takes the output from the Timer node (which isn't the regular Timer node but rather the one from the shaders library which should be synched to the rendering) and does this with it:
@returns(bool)
def process(t):
tTotal = 4.*100
tOff = 2.*100
return (( t % tTotal ) < tOff)
The timer node is set to output with a period of 10ms, so this causes our LED to come on for two seconds, then go off for two seconds according to the clock in the renderer, which should match the time our grating are being presented. Visualisers confirm that all of these nodes output what is expected, just with an increasing delay between what is rendered, and when our LED turns on and off.
We have tried pulling the elapsed time directly from the RenderFrame node instead and using an Accumulate node to count the time this way, but we get the same drift (although we expected there would be issues there using the TimeStep.ElapsedRealTime value since we want to know where in the sequence of gratings we are, not how much actual time has passed).
The issue is that between the GratingSequence node and the RenderFrame node any details of what is being rendered have been abstracted away so we have no way, beyond timing and frame count, of knowing what is being rendered to the screen, and none of the "times" that we have access to (any of the three in the RenderFrame node and the one in the Timer (Shader) node) seem to correspond to how far through the render we are.
Is there some other way we should be doing this? Is there another way to emit events associated with what is being rendered in the BonVision package? Which clock is supposed to be synched with what is being presented in a window? How would we get access to it so we can use it to drive the ardunio pin?

Previous Searches:
We haven't turned up any similar posts in our searches:
Closest I could find was looking at timing without the arduino component:
https://github.com/orgs/bonsai-rx/discussions/1203
The proposed solutions seem to require access to details about how the render is being performed that are not accessible using the BonVision grating sequence node, and besides, we would like to know the origin of the drift so we can avoid it entirely if possible.

Bonsai ver. 2.8.1
All installed packages up to date as of this posting

[RoboDoig]

Hi @swilshin,

I suspect the reason for the drift is that the Timer used is not directly linked to the stimulus time in your workflow. You have a Timer for generating the LED signal, and GratingSequence internally also has a Timer for setting the duration of the trial. Although these timers are both linked to the render time, they are not necessarily in sync with each other. If they were both reset each trial the offsets would be minimal, but the Timer that drives the LED state is running continuously in parallel so it will accumulate timing drift relative to the internal Timer of GratingSequence.

Although the rendering logic of GratingSequence is abstracted away in the BonVision package, you can make modifications to it by ungrouping it (right-click >> Ungroup or Ctrl+Shift+G). This could allow you to link LED state updates directly to the start/stop of each trial. As an example, here is a modified version of GratingSequence where a boolean is triggered at the start and end of each grating trial (within CreateObservable in the GratingSequence workflow):

As you mentioned, the LED will still have some timing variability relative to the stimulus onset/offset due to USB communication time but should not drift over time. I'm not sure what you're using the LED for in your use case, but one approach for screen synchronisation is to render e.g. a colored quad at the corner of the screen, tied to some rendering event (RenderFrame update or start of a new stimulus) and record this with a photodiode attached to the screen.

[theskymallard]

Thanks Robo, just wanted to confirm that following along with your approach let us fix our issue, so if anyone else runs into a similar problem they should consult your comment. As you say, the Timers were not in synch.

[abailey20]

Hi @RoboDoig

I'm replying on behalf of my colleague @swilshin. Your suggestion helped us with our issue, thanks very much!

There was a slight issue with the first/last reactive nodes as they threw an error because there was 'no sequence detected' if the script wasn't run within a timeframe of two seconds.

For what we are using it for this isn't an issue but I wanted to make it known to anyone else wanting to apply your suggestions.

Again, thanks very much!