custom visualization callback for dichroism scans

[prjemian]

visualize (plot) energy and (phase retarder fast rotation) PZT scans for polarization direction with this sequence in the primary data stream

1. set energy
2. move phase retarder positioning motor
3. acquire at phase retarder pzt +
4. acquire at phase retarder pzt -
5. acquire at phase retarder pzt -
6. acquire at phase retarder pzt +

Process this group as a single event in a separate stream and visualize from that stream

Both streams are saved to the databroker and available during and after the experiment.

[prjemian]

The X axis could be energy, magnetic field, positioner, or other (as selected by user)

[prjemian]

This aspect is an optional part of a dichroism scan. It's not easy to preset default parameters so this should be avoided.

The processed stream should report the polarization, not the PZT value. The polarization as used today is left or right (not a float value).

[prjemian]

see phase_retarders branch

[prjemian]

Is the inner scan the one_dichro_step()?

ipython-polar/profile_bluesky/startup/instrument/plans/local_scans.py

Lines 19 to 37 in fe86149

	def one_dichro_step(detectors, step, pos_cache, take_reading=trigger_and_read):
	"""
	Inner loop for dichro scans.
	Parameters
	----------
	detectors : iterable
	devices to read
	step : dict
	mapping motors to positions in this step
	pos_cache : dict
	mapping motors to their last-set positions
	take_reading : plan, optional
	function to do the actual acquisition ::
	def take_reading(dets, name='primary'):
	yield from ...
	Callable[List[OphydObj], Optional[str]] -> Generator[Msg], optional
	Defaults to `trigger_and_read`
	"""

[prjemian]

@gfabbris Can you make a sketch of the desired visualization? Nothing fancy, just to give an idea what & how to show.

[gfabbris]

The plot itself is not too complicated. It'd be something like the one below, with the XANES on the top panel and the XMCD in the bottom.

This is how you calculate the XANES and XMCD if you are reading from the database:

def load_dichro(scan,xcol='monochromator_energy',monitor='Ion Ch 4',detector='Ion Ch 5',
                **kwargs):
    
    """Load dichro XMCD of one scan
    """
    
    table =db[scan].table()
    size = table.shape[0]//4
    
    x = np.array(table[xcol]).reshape(size,4)
    absorption = np.log(np.array(table[monitor])/np.array(table[detector])).reshape(size,4)
    
    x = x.mean(axis=1)
    xanes = absorption.mean(axis=1)
    xmcd = absorption[:,[0,3]].mean(axis=1) - absorption[:,[1,2]].mean(axis=1)
    
    return x,xanes,xmcd

Besides plotting, the callback would ideally save the processed data to another stream.

[danielballan]

Hello, @gfabbris. I am leading an effort to build better visualization widgets for Bluesky. I mentioned to @prjemian that I would like to try them on some data at APS, and he suggested that your problem would be a good test case for my work.

A primary goal is to make it easy to perform transformations on the data on the way to plotting, exactly as in your example above. (We have separate but related on saving the processed data. I'd like to set that aside at first, but we could incorporate that too, later.) The widgets are designed to fit into web-based applications, Jupyter, and desktop applications (Qt). Pete mentioned that visualization in Jupyter might be of particular interest to you at this point. Is that right?

If you are willing to be a "test user" then I think the best first step would be to share with me some data that I can use to build a small example. We have a command-line tool for packing up scans into a "portable databroker" or sorts. I'd ask you install that tool and export some scans. For example, to send scans 135, 136, 137 plotted in your example above:

pip install databroker-pack
databroker-pack mongodb_config --copy-external --query '{"scan_id": {"$in": [135, 136, 137]}}' exported_scans
# Be careful about the use of single-quotes and double-quotes above; they matter.

This will create a directory exported_scans which you may share with me by any convenient means---email (if it's small), Dropbox, Google Drive, etc. My work email is [email protected]; my Google account is associated with [email protected].

---

@prjemian I got mongodb_config in the command above from

ipython-polar/profile_bluesky/startup/instrument/framework/initialize.py

Line 33 in b115a0c

db = databroker.Broker.named('mongodb_config')

which looks like the name of the databroker of interest here. Please correct me if I'm wrong.

[prjemian]

That's right. In this repo, the file is ~/.config/databroker/mongodb_config.yml which is found by this code as mongodb_config .

[prjemian]

I see databroker-pack is a conda package from the nsls2forge channel. Same version 0.3.0.

[prjemian]

@gfabbris : Note that other search criteria are possible, using the mongodb-query language.

[danielballan]

Just chiming in to provide a heartbeat on this.

Our new visualization tools have been fast-moving but are now "annealing" into a stable-ish form. @AbbyGi and I put most of today in working this example and exploring different approaches (with some help from @tacaswell). I expect to report back sometime next week with a working example ready for use and feedback.

[danielballan]

By configuring ophyd devices differently, we can make thing easier for downstream code (visualization, analysis, ...). The metadata reported by the scan of the monochrometer_energy looks great:

 'hints': {'dimensions': [[['monochromator_energy'], 'primary']]},

but the two-motor scan over temperature is giving us imprecise metadata:

 'hints': {'dimensions': [[['lakeshore 360_loop1_readback',
                            'lakeshore 360_loop1_setpointRO',
                            'lakeshore 360_loop2_readback',
                            'lakeshore 360_loop2_setpointRO'],
                           'primary']]},

Bluesky's intention is that that should include just one item per axis, "the thing to plot against". The setpoints should be omitted from that list, then, and we should only have the readbacks. That is, I want this to say:

 'hints': {'dimensions': [[['lakeshore 360_loop1_readback',
                            'lakeshore 360_loop2_readback'],
                           'primary']]},

This can be accomplished by changing this Kind here

ipython-polar/profile_bluesky/startup/instrument/devices/lakeshore336.py

Line 27 in 88c7217

kind=Kind.hinted)

from Kind.hinted to Kind.normal. Only the readback, the thing to plot against, should be hinted. Likewise here:

ipython-polar/profile_bluesky/startup/instrument/devices/lakeshore340.py

Line 25 in 88c7217

kind=Kind.hinted)

I see the same issue in another scan:

 'hints': {'dimensions': [[['magnet_6T_field_readback',
                            'magnet_6T_field_setpoint'],

It's not immediately clear to me where in the profile that is configured.

We can work around this by adding some logic to the visualization code ("If there are a pair of keys that start the same but end in _readback and _setpoint, use the _readback one and ignore the _setpoint one.") but it would obviously be cleaner if we could record the metadata better at acquisition time.

[gfabbris]

Sounds good. I will go through the devices and clean up the hints.

[danielballan]

Does this look on the right track?

[gfabbris]

Yeah, looks great!

[AbbyGi]

Hello @gfabbris and @prjemian. @danielballan and I have finished up a working example to use, located in a gist here. There's a README included to guide you through using it. Please give it a try and let us know what you think!

[gfabbris]

@AbbyGi and @danielballan, thanks!! I tested at the beamline computer and it works well for the saved data. We are in shutdown now, x-rays will be back in a couple of weeks and I will test the live data part then.

I'd automate a plot like this to happen only for dichro scans. We already have a decorator that handles some setup that is needed for this type of scans, and @prjemian mentioned the bluesky.preprocessors.subs_decorator. So, I think this might work:

ipython-polar/profile_bluesky/startup/instrument/plans/local_preprocessors.py

Line 189 in 0751fe7

# Start plot subscription

Does that look ok to you? Perhaps there is a better way to automate this plot?

[danielballan]

Great to hear. Thanks again for taking the time. We look forward to hearing how live visualization goes.

Off the top of my head, I think defining the model and the view inside a preprocessor like you have sketched there presents some difficulties. Defining them inside of the processors makes it awkward to access them from other parts of the code. It may be important to have access model to do things like close all the figures

model.figures.clear()

clear the runs

model.runs.clear()

or manually add some saved scans alongside your live ones.

model.add_run(db[172])

It can also be useful to access the view to save the figures, for example.

# We should add a convenience method for this snippet....
for figure_view in view.figures.values():
    mpl_figure = figure_view.figure  # a normal matplotlib.figure.Figure object
    mpl_figure.savefig(f"{figure_view.model.title}.png")

My hypothesis is that it will be best to subscribe model semi-permanently to RE

model = AutoXanes()
RE.subscribe(stream_documents_to_runs(model))
view = QtFigures(model)
view.show()

and make model smart enough to only act when it sees a scan that is applicable and ignore all others.

A second reason for going this path is that subs_decorator firmly ties the plotting code to the data acquisition code. With that approach, they must run together on the same machine and in the same process. With the RE.subscribe(...) approach, there is a natural path to someday moving that subscription to a separate process or a separate machine, such that the data flows like RE --> network --> subscriber. This has several advantages. If live visualization/processing workloads becomes significant, moving them to a separate process and/or machine will ensure that data acquisition is not slowed down; each can proceed at their own pace. Likewise, any errors or hangups in the visualization/processing will not interfere with data acquisition.

That's my thinking at this moment, but I think anything is worth trying at this point as long as we stay flexible and open-minded. This is somewhat new territory for the collaboration, and we'll be learning together as we go.

[gfabbris]

Yeah, it will be good to have access to model. I'll look into this.

[gfabbris]

Yesterday I went to the lab and turned on some detectors so that I could test the live plotting without x-rays. It works, but I had to make one change, the downsampled function now is:

def downsampled(x):
    if array(x).size%4 == 0:
        return array(x).reshape(-1, 4).mean(axis=1)
    else:
        return None

ValueError happens if this is not done because it will try to run the reshape(-1,4) at every event and (when the number of points is not a multiple of 4).

THIS DOESN'T WORK... SEE BELOW

[gfabbris]

I forgot to say that the way that the plot is automated now is by watching a 'scan_type': 'dichro' flag in the md['hints'].

ipython-polar/profile_bluesky/startup/instrument/callbacks/dichro_plot.py

Line 58 in 9474588

scan_type = run.metadata["start"]["hints"].pop('scan_type', None)

ipython-polar/profile_bluesky/startup/instrument/plans/local_scans.py

Line 97 in 9474588

md = {'hints': {'scan_type': 'dichro'}}

Is this better?

[gfabbris]

Actually, I did some extra tests, and the solution above doesn't quite work. The problem is how to make the plotter only operate when the number of points is a multiple of 4.

[gfabbris]

I managed to make it work by doing this:

def xanes(monitor, detector):
    
    if array(detector).size < 4:
        return 0
    
    rng = 4*(array(detector).size//4)
    absorption = log(array(monitor)[:rng] / array(detector)[:rng]).reshape(-1, 4)
    return absorption.mean(axis=1)

So when the number of points is not a multiple of 4 it will just replot the last time that it was.

[gfabbris]

But I also realized another issue. The XANES/XMCD plot rendering is not automatically update when a new point is available. But it does update if I resize the screen (note that I'm just plotting random numbers at every step because the detector are reading zero...). I suppose this is just some Qt settings?

plot.mov

[danielballan]

We are just catching up on this, as @AbbyGi and I have been in trainings 9-5 all week.

---

Is this better?

Yes, this looks great.

ipython-polar/profile_bluesky/startup/instrument/callbacks/dichro_plot.py

Line 58 in 9474588

scan_type = run.metadata["start"]["hints"].pop('scan_type', None)

Exactly what we intended. A minor suggestion: use get(...) instead of pop(...) so that you can access the value without removing it from the document. Editing the contents of the document like that might have unexpected consequences.

---

I managed to make it work by doing this:

def xanes(monitor, detector):
    
    if array(detector).size < 4:
        return 0
    
    rng = 4*(array(detector).size//4)
    absorption = log(array(monitor)[:rng] / array(detector)[:rng]).reshape(-1, 4)
    return absorption.mean(axis=1)

I think that rather than changing xanes I would consider adding the < 4 check to handle_new_stream as well.

    def handle_new_stream(self, run, stream_name):
       ...
       if run.primary.to_dask()[self._detector].size < 4:
            # Nothing to do;l this is too short.
            return

I prefer that approach because

(1) It keep the control of over whether we plot together in one place.
(2) It leaves the xanes function as a self-contained thing that always tries to produce this result, rather than sometimes returning 0. That kind of return type instability can lead to confusing bugs, and makes things harder to reason about.

---

Thanks for providing a clear demonstration of the failure to automatically update in Qt. Yes, this must be some detail of Qt that we are not getting quite right. We will consult our local Qt experts and get back to you on this.

[tacaswell]

If it is redrawing on resize then we know the Qt mainloop is running like we expect and processing Qt event (good!), however we are clearly losing the draw_idle calls someplace. The code looks right to me, I suspect we are going to need some print-style debugging to get to the bottom of this. My current theory is that the Qt Trampoline is not working the way we expect so the callbacks are still running on the background thread, but that is based on intuition rather than evidence.

---

As an aside, is the previously plotted data jumping around like it does expected?

[danielballan]

At your convenience, @gfabbris, would you please try to run the following and let us know if it live-updates properly. This will help us isolate whether the problem is a general one with your installation or something specific about the XANES plot.

ipython --gui=qt

In [1]: %run -m bluesky_widgets.examples.ipy_qt_figures

In [2]: motor.delay = 0.5

In [3]: RE(plan())

[gfabbris]

It turns out, the plot update is fine. The issue is that the plot starts in a small window (that I didn't realize was plotting the data), and because I used our beamline setup, a separate window also plots the data, but this one doesn't update correctly (note that the qt_test.py is the same as the ipy_qt_figures but it doesn't run the plan):

qttest.mov

A couple of questions:

• Is there a way to start the screen in a larger size? Or rescale when the scan starts?
• Can we prevent this duplicated screen to come up? This also happens if I run the XANES/XMCD plot callback.

[gfabbris]

@tacaswell - It was jumping around because I was just generating a new set of random numbers at every event. I didn't remember I could use the ophyd.sim stuff...

@danielballan - Adding the size check to the handle_new_stream function doesn't work, because (as I understand it) this function just runs once in the beginning of the scan. But I agree with your point that it'd be better to not send anything to the plotter (instead of zero or the same data as the previous point). Any other ideas on how to do this?

[gfabbris]

In my early discussions with @prjemian about this, we considered creating a new stream that would hold the processed XANES/XMCD. Maybe that's a good solution? I believe the callback would look somewhat like this: https://blueskyproject.io/bluesky/callbacks.html#secondary-event-stream

[danielballan]

Is there a way to start the screen in a larger size?

Yes. For whatever reason, it does not initialize small for me, so I will rely on use to test my proposed solution. Please see instructions at bluesky/bluesky-widgets#98.

Can we prevent this duplicated screen to come up?

My guess is that you are doing

%matplotlib qt5

and/or

import matplotlib.pyplot as plt
plt.ion()

either of which turn on "interactive plotting," which causes matplotlib to eagerly display figures in stand-alone windows rather than letting us place them intentionally. (Does that sound accurate to you, @tacaswell?) The fix would be to find wherever you are running that in your profile and remove it, or else override it with plt.ioff() before attempting to use bluesky-widgets.

Any other ideas on how to do this?

Ah, you are correct of course. I'll circle back later with a better suggestion.

[danielballan]

we considered creating a new stream that would hold the processed XANES/XMCD. Maybe that's a good solution?

This is a useful discussion. I have responded in a new issue, #75, to avoid entangling it with the plotting discussion here.

[gfabbris]

Yeah, we do turn on interactive plotting during startup... I'll use plt.ioff().

[danielballan]

It looks like we should be able to be compatible with plt.ion() by keeping our figures separate from the generic matplotlib figures that aren't managed by us (by QtFigures). Will work on this....

FYI, I just released bluesky-widgets v0.0.7 on PyPI with the sizing fix, so you can go back to running on a standard release if you like.

[danielballan]

Action items from discussion today:

• Provide an example of a DerivedSignal that down-samples.
• Provide an example of a plan that using trigger_and_read everything Nth (i.e. fourth) point to record data into a separate stream.
• Build a "main" AutoPlotter that dispatches out to other AutoPlotter(s) and is configurable by the user. (It might make sense to add UI elements to enables this configuration from the UI itself --- no coding required.)
• Include a generic BestEffortAutoPlotter (successor to the old bluesky.callbacks.best_effort.BestEffortCallback) is this, which should keep quiet if any more specific AutoPlotters like the dichro one have something to say.

[danielballan]

Just chiming in to say, attention in the group has recently been elsewhere, but this is still a going concern. We are doing related work this week, should have some concrete to report back in a couple weeks.

[gfabbris]

Sounds good. I plan to test the current XMCD plotter in an experiment this week. I did some minor tweaks to it. One thing that I found particularly useful is the ability to start a new plot (using the same scanning positioner) without closing the old one (but this may not be the best way to do this).

ipython-polar/profile_bluesky/startup/instrument/callbacks/dichro_plot.py

Line 84 in 2182904

def new_plot(self, x_name=None):