Matplotlib can't handle more than 499 data points

[giulioungaretti]

Bug from Thorvald at qdev.

Steps to reproduce

# VNA power scan:
v1.spec_mode(0)
v1.bandwidth(500)
v1.avg(5)

a = 6.61064e9;
v1.start(a-20e6)
v1.stop(a+20e6)
v1.npts(101)

loop = qc.Loop(v1.power.sweep(-45,-55,1)).each(v1.trace)
data = loop.get_data_set();
plot=qc.MatPlot(array, subplots=(1,2))
plot.add(array, subplot=2)
loop.with_bg_task(plot.update, 0.0005).run()

Expected behaviour

The plot should show the data

Actual behaviour

The plot only show the data if a row has less than 499 points

System

windows

qcodes branch
master

qcodes commit
3475e49

[nataliejpg]

nb (from Thorvald) this was after a MatPlot update 1.5.3 although hadn't tried with this many points before the update). On downgrading to 1.5.1 still broken...

[jenshnielsen]

Is this specific to one instrument.

I can't reproduce it doing something simple with a mock instrument (Slightly modified from the base tutorial producing a 600 time 650 plot in the end.

%matplotlib nbagg
import matplotlib.pyplot as plt
from pprint import pprint
import time
import numpy as np
import qcodes as qc

# spawn doesn't like function or class definitions in the interpreter
# session - had to move them to a file.
from toymodel import AModel, MockGates, MockSource, MockMeter, AverageGetter, AverageAndRaw

# now create this "experiment", note that all these are instruments 
model = AModel()
gates = MockGates('gates', model=model)
source = MockSource('source', model=model)
meter = MockMeter('meter', model=model)

# The station is a container for all instruments that makes it easy 
# to log meta-data
station = qc.Station(gates, source, meter)

# it's nice to have the key parameters be part of the global namespace
# that way they're objects that we can easily set, get, and slice
c0, c1, c2, vsd = gates.chan0, gates.chan1, gates.chan2, source.amplitude

loop = qc.Loop(c1[-30:30:0.1], 0.01).loop(c0[-30:35:.1], 0.00001).each(
    meter.amplitude, # first measurement, at c2=0 -> amplitude_0 bcs it's action 0
    qc.Task(c2.set, 1), # action 1 -> c2.set(1)
    meter.amplitude, # second measurement, at c2=1 -> amplitude_2 bcs it's action 2
    qc.Task(c2.set, 0)
    )
data = loop.get_data_set(name='2D_test')

plot = qc.MatPlot(data.meter_amplitude_0, cmap=plt.cm.hot, figsize=(12, 4.5), subplots=(1, 2))
plot.add(data.meter_amplitude_2, cmap=plt.cm.hot, subplot=2)
data2 = loop.with_bg_task(plot.update, plot.save,  0.0000001).run()

[nataliejpg]

nope, we get it with alazar and VNA measurements on two different computers (I don't have great confidence in the version control on either computer but they used different install methods so that's at least 2 data points :P). Both live plot and plotting old data. Didn't try it out with the tutorial notebook but I can try running the above code when I have a moment and see how it looks.

[jenshnielsen]

That would be great. I am happy to test out the instrument too just need to hook it up or use the lab computer

[nataliejpg]

I think you'll have trouble getting hold of either of those instruments to be honest :P I guess it could be something to do with the 'hard parameter' aspect of it but that seems a bit obscure. PyQt plot has previously had a 'nonlinearity of setpoints' issue but its not obvious to me that it's linked. I'll let you know when I've tested the tutorial. It will late afternoon/tomorrow morning

[jenshnielsen]

Investigating this a bit more the issue seems to be that the frequency setpoint array being filed with NaNs

In the case of 559 the frequency array is filled with NaNs rather than the actual frequency after the first 3 measurements. I bet this is the reason for #397 too

[jenshnielsen]

I.e. do something like:

import qcodes as qc
import matplotlib.pyplot as plt
loaded_data = qc.load_data("VNA_power_set_frequency.dat")
fig, ax = plt.subplots(1,2)
ax[0].plot(loaded_data.frequency[0])
ax[1].plot(loaded_data.frequency[3])
fig.tight_layout()

and notice that the second plot is empty because all the data is Not a Number (NaN)

[nataliejpg]

yh that was something I noticed which sometimes happens he VNA driver (and promptly forgot about), it usually doesn't matter because its a setpoint array that we make up so will be the same array for each trace but I also have no clear idea at all why it sometimes doesnt woek considering that the driver should set these arrays for every trace... I have some doubts that it's the same as #397 because that can be 'solved' by increasing the limit for nonlinearity a bit as @MerlinSmiles suggested which I wouldn't expect to allow it to plot if it was because of NaN problems, also if it was that then why would it be ok live plotting it but not loading it in afterwards? I can look around briefly for a data set without NaNs though and see if it can plot that though as a check for that being the reason for this bug

[jenshnielsen]

This can be reproduces relatively simply with a mock instrument

%matplotlib nbagg
import matplotlib.pyplot as plt
import numpy as np
import qcodes as qc
from toymodel import AModel, MockGates, MockSource, MockMeter, AverageAndRaw, ArrayGetter

# now create this "experiment", note that all these are instruments 
model = AModel()
gates = MockGates('gates', model=model)
source = MockSource('source', model=model)
meter = MockMeter('meter', model=model)

# The station is a container for all instruments that makes it easy 
# to log meta-data
station = qc.Station(gates, source, meter)

# it's nice to have the key parameters be part of the global namespace
# that way they're objects that we can easily set, get, and slice
c0, c1, c2, vsd = gates.chan0, gates.chan1, gates.chan2, source.amplitude
loop = qc.Loop(c0.sweep(0,20,0.1), delay=0.001).each(ArrayGetter(meter.amplitude, c2[-10:10:0.2], 0.001))
data = loop.get_data_set(name='testsweep')
plot = qc.MatPlot()
plot.add(data.meter_amplitude)
_ = loop.with_bg_task(plot.update).run()
plot2 = qc.MatPlot()
plot2.add(data.meter_amplitude)

However it only happens with the background task updating a plot. Removing the background task or replacing it with a dumme task seems to fix the issue.

[jenshnielsen]

So the problem is that the update function calls dataset.sync which ends up rereading the datafile from disk. This overwrites the dataset which already has all the hard arraylike setpoints defined with the on disk version which only has the setpoint arrays for already measured values. I think this can be work around by passing a dummy update function to plot.add.

Something like

def dummy_sync():
    pass

%matplotlib nbagg
import matplotlib.pyplot as plt
import numpy as np
import qcodes as qc
from toymodel import AModel, MockGates, MockSource, MockMeter, AverageAndRaw, ArrayGetter

# now create this "experiment", note that all these are instruments 
model = AModel()
gates = MockGates('gates', model=model)
source = MockSource('source', model=model)
meter = MockMeter('meter', model=model)

# The station is a container for all instruments that makes it easy 
# to log meta-data
station = qc.Station(gates, source, meter)

# it's nice to have the key parameters be part of the global namespace
# that way they're objects that we can easily set, get, and slice
c0, c1, c2, vsd = gates.chan0, gates.chan1, gates.chan2, source.amplitude
loop = qc.Loop(c0.sweep(0,20,0.1), delay=0.001).each(ArrayGetter(meter.amplitude, c2[-10:10:0.2], 0.001))
data = loop.get_data_set(name='testsweep')
plot = qc.MatPlot()
plot.add(data.meter_amplitude, updater=dummy_sync)
_ = loop.with_bg_task(plot.update).run()
plot2 = qc.MatPlot()
plot2.add(data.meter_amplitude)

does not reproduce the issue

[nataliejpg]

@jenshnielsen, thanks for this. Do you have ideas for a 'non workaround' fix?

[jenshnielsen]

We agreed to remove the sync from the default update function. I will make that change today and we can push a new release with that fix soon.