HDF5 from Bruker

[Fred-Ad]

Hi,

First, thanks for this amazing work, that permits to see bayond EBSP.
However, I've got a problem with a HDF5 export with Esprit 2.2 (Bruker software).
when I plot
iq = s.get_image_quality(); s_iq = hs.signals.Signal2D(iq); s.plot(navigator=s_iq, scalebar=False)

The result is not as expected. I attached the result of IQ map by KikuchiPy and the CI map plotted by Mtex.
I think that kikuchipy read EBSP in the wrong order (X before Y or the inverse). Is it a possibility ? Do you have an idea to help me.

Thanks a lot

Best regards

Fred

[hakonanes]

Hi @Fred-Ad!

Thanks for reaching out and asking this question. I agree that the IQ map looks strange, and this might well be due to a bug in kikuchipy...

Which version of kikuchipy are you running (import kikuchipy; print(kikuchipy.__version__))? If you have v0.3, you can get an average neighbour dot product map, which should give us an even better view of whether kikuchipy reads the patterns in the wrong order.

I don't have access to Bruker Esprit v2 myself, and while I wrote the Bruker HDF5 reader in kikuchipy, I tested it against only one file, the one made available on Zenodo by Angus Wilkinson and David M Collins. If the reader indeed reads your patterns in the wrong order, do you think you could send me your dataset, so we make sure that the reader can read your dataset?

Lastly, I would like to mention, for those not so familiar with very early EBSD work, that the image quality calculation implemented in kikuchipy (and e.g. EMsoft) uses the method of Krieger Lassen (page 87 in his PhD thesis found here), i.e. it is not the same as the one provided by commercial packages, like EDAX TSL, which relies on the Hough transform.

[hakonanes]

Just to be sure, I redownloaded and read in the mentioned Bruker HDF5 dataset from Wilkinson and Collins, and computed maps of the mean intensity, image quality, and average neighbour dot product:

That version listed in the dataset is "Esprit 2.X". What is the Esprit version of your dataset, @Fred-Ad?

[Fred-Ad]

Hi and thanks for your answer.

It's the only dataset I have, so I can't test with another. Indeed exporting hdf5 is a paid option for Esprit (!!!), and this is the map I sent to them to test their hdf5 output, before buying. I work with Esprit 2.2 but I don't know what version they used to generate the hdf5 from my bcf file.

The result of the dot product is weird too :

One other source of problem in that my dataset is :

• On BrukerEsprit, when you have your SEM image, you can choose a ROI where you acquire the EBSD map. For this map, I used a 169x103 px map in the 800x600 px SEM image.
• However, when I export indexed data in .ctf for mtex, ther is a 800x600px data file, with all values equal to 0 except for the 169x103 indexed px (cf. image)
  
• It's the same with the hdf5 file. For example the value ['/Scan 1/EBSD/Header/NCOLS'] is 800 and ['/Scan 1/EBSD/Header/NROWS'] is 600, but the number of patterns is 17407 (169x103).
• At first, kikuchipy gave me an error, so I changed manually NCOLS and NROWS in the hdf5 file with their true values. It worked, but perhaps It is the source of the problem.

PS1 : I use Kikuchipy 0.3.1.
PS2 : I had another error in kikuchipy because I didn't acquired the static background, but I commented the appropriate line to be able to work on my file.

[hakonanes]

Thanks for looking more into this, @Fred-Ad.

* It's the same with the hdf5 file. For example the value `['/Scan 1/EBSD/Header/NCOLS']` is 800 and  `['/Scan 1/EBSD/Header/NROWS']` is 600, but the number of patterns is 17407 (169x103).
* At first, kikuchipy gave me an error, so I changed manually `NCOLS` and `NROWS` in the hdf5 file with their true values. It worked, but perhaps It is the source of the problem.

I think you're right in that this is the source of the problem. We should be able to read patterns from such a dataset where only patterns within an ROI is included.

I would be very greatful if you could send me the data, so that I can update the reader to read your data correctly! You can email me at the university email [email protected].

PS1 : I use Kikuchipy 0.3.1.
PS2 : I had another error in kikuchipy because I didn't acquired the static background, but I commented the appropriate line to be able to work on my file.

I'm 99.9% sure the error you reference here is fixed in #331. This change, and other bug fixes, are slated for a patch release (available via pip/anaconda) 0.3.3 on this coming Sunday (0.3.2, containing a few bug fixes unrelated to this discussion, was released in the start of Feb). I hope that updating the Bruker reader, when debugging with your dataset, goes smoothly, so that we can include a Bruker reader update in that release as well. Then, you should be able to update kikuchipy next week and read your data!

[Fred-Ad]

I can send my dataset. Can you give me your email or a way to send it ?

Thanks a lot

[hakonanes]

Great, my uni email is [email protected]

[Fred-Ad]

I'm on vacation tomorrow and doesn't have the dataset at home. So I'll send it on monday.

Thanks again for your help and this very usefull work.

[hakonanes]

Sounds good, I've postponed the patch release one week to try to get a fix to this in.

[Fred-Ad]

I sent you the hdf5 file this morning. Did you receive it ?

[hakonanes]

Yes, I received it, thank you! Indeed, there are some issues with the Bruker reader. I'm working on fixing those, thanks to your dataset!

[hakonanes]

Finally, I figured out what the "Scan 1/EBSD/SEM/IX" and "Scan 1/EBSD/SEM/IY" arrays show, i.e. the (flattened) indices of patterns in the final 2D map into the 1D array of patterns in "Scan 1/EBSD/Data/RawPatterns". With this fix, I could now do:

>>> import matplotlib.pyplot as plt
>>> import kikuchipy as kp
>>> s = kp.load("/home/hakon/phd/data/epTest-coeur.hdf5")
>>> static_bg = s.mean(axis=(0, 1))  # Average all patterns to get a background
>>> static_bg.change_dtype(s.data.dtype)  # Revert to 8-bit unsigned integer
>>> s.remove_static_background(static_bg=static_bg.data)
>>> s.remove_dynamic_background()
>>> iq = s.get_image_quality()
>>> adp = s.get_average_neighbour_dot_product_map()
>>> fig, ax = plt.subplots(ncols=2)
>>> ax[0].imshow(iq)
>>> ax[0].set_title("Image quality")
>>> ax[0].axis("off")
>>> ax[1].imshow(adp)
>>> ax[1].set_title("Average neighbour dot product")
>>> ax[1].axis("off")
>>> fig.tight_layout()
>>> fig.savefig("/home/hakon/phd/data/iq_adp.png"), bbox_inches="tight", pad_inches=0, dpi=150)

to produce this

I'll get this fix in now, and then release this as part of a v0.3.3 patch.

[hakonanes]

The fix mentioned in the previous comment is implemented in #339, and will be released as part of v0.3.3 shortly!

[hakonanes]

@Fred-Ad, I just published kikuchipy v0.3.3 on PyPI, could you try pip install kikuchipy==0.3.3, and try to load your dataset again?

[Fred-Ad]

Hi Hakon,
I can load the dataset and produce the s.plot_virtual_bse_intensity(roi)
That's a good point, thanks !
However, there is an error when trying vbse_rgb_img = vbse_gen.get_rgb_image(r=red, g=green, b=blue)
this gives me this error :

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-18-7637d6229fcb> in <module>
----> 1 vbse_rgb_img = vbse_gen.get_rgb_image(r=red, g=green, b=blue)

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\kikuchipy\generators\virtual_bse_generator.py in get_rgb_image(self, r, g, b, percentiles, normalize, alpha, dtype_out, **kwargs)
    149             for roi in rois:
    150                 if isinstance(roi, tuple):
--> 151                     roi = self.roi_from_grid(roi)
    152                 roi_image = self.signal.get_virtual_bse_intensity(roi).data
    153                 if isinstance(roi_image, Array):

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\kikuchipy\generators\virtual_bse_generator.py in roi_from_grid(self, index)
    251         min_col = cols[min(index[:, 1])]
    252         max_col = cols[max(index[:, 1])] + cols[1]
--> 253         min_row = rows[min(index[:, 0])]
    254         max_row = rows[max(index[:, 0])] + rows[1]
    255 

IndexError: index 7 is out of bounds for axis 0 with size 6

Perhaps it si due to the count of columns and row, as

vbse_gen.grid_shape = (10, 10)
red = [(7, 1), (8, 1), (8, 2), (9, 1), (9, 2)]
green = [(8, 4), (8, 5), (9, 4), (9, 5)]
blue = [(7, 8), (8, 7), (8, 8), (9, 7), (9, 8)]
p = vbse_gen.plot_grid(
    rgb_channels=[red, green, blue],
    visible_indices=True,  # Default
    pattern_idx=(0, 0),  # Default is (0, 0)
)
p

gives the following image, with a problem with the squares coordinates. I tried with the Ni datadat, no problem. So it is only due to the hdf5 Bruker file.

Sorry to bother you. Hope this is not a big deal.

Hope you enjoyed the EBSD2021 RMS congress, I saw you were connected. A lot of news and advanced in the EBSD world... and a lot more to do in the next years !

Regards
Fred

[hakonanes]

Thanks for checking, glad you could read your patterns correctly!

I suggest we continue discussing this issue in a separate issue #346, is that OK with you? I find that having separate issues for separate problems helps in keeping track of what's wrong where etc.

Yeah, the RMS meeting had many interesting talks, and gave a nice overview of what EBSD work people are doing, at least what they're doing in Britain. As you say, lots do to in the next years, and hopefully some of that can be done in kikuchipy!

[Fred-Ad]

It's the same with vbse = s.get_virtual_bse_intensity(roi) that gives me this - long - error :

---------------------------------------------------------------------------
TraitError                                Traceback (most recent call last)
<ipython-input-6-a39e5bb3612f> in <module>
----> 1 vbse = s.get_virtual_bse_intensity(roi)
      2 vbse

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\kikuchipy\signals\ebsd.py in get_virtual_bse_intensity(self, roi, out_signal_axes)
   1494         """
   1495         vbse = roi(self, axes=self.axes_manager.signal_axes)
-> 1496         vbse_sum = self._get_sum_signal(vbse, out_signal_axes)
   1497         vbse_sum.metadata.General.title = "Virtual backscatter electron image"
   1498         vbse_sum.set_signal_type("VirtualBSEImage")

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\kikuchipy\signals\ebsd.py in _get_sum_signal(signal, out_signal_axes)
   1443     @staticmethod
   1444     def _get_sum_signal(signal, out_signal_axes: Optional[List] = None):
-> 1445         out = signal.sum(signal.axes_manager.signal_axes)
   1446         if out_signal_axes is None:
   1447             out_signal_axes = list(

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in sum(self, axis, out, rechunk)
   3648         if axis is None:
   3649             axis = self.axes_manager.navigation_axes
-> 3650         return self._apply_function_on_data_and_remove_axis(
   3651             np.sum, axis, out=out, rechunk=rechunk)
   3652     sum.__doc__ %= (MANY_AXIS_PARAMETER, OUT_ARG, RECHUNK_ARG)

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in _apply_function_on_data_and_remove_axis(self, function, axes, out, **kwargs)
   3594             ar_axes = ar_axes[0]
   3595 
-> 3596         s = out or self._deepcopy_with_new_data(None)
   3597 
   3598         if np.ma.is_masked(self.data):

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in _deepcopy_with_new_data(self, data, copy_variance, copy_navigator, copy_learning_results)
   2329                 del self.learning_results
   2330             self.models._models = DictionaryTreeBrowser()
-> 2331             ns = self.deepcopy()
   2332             ns.data = data
   2333             return ns

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in deepcopy(self)
   4847         :py:meth:`~hyperspy.signal.BaseSignal.copy`
   4848         """
-> 4849         return copy.deepcopy(self)
   4850 
   4851     def change_dtype(self, dtype, rechunk=True):

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\copy.py in deepcopy(x, memo, _nil)
    151             copier = getattr(x, "__deepcopy__", None)
    152             if copier is not None:
--> 153                 y = copier(memo)
    154             else:
    155                 reductor = dispatch_table.get(cls)

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in __deepcopy__(self, memo)
   4813 
   4814     def __deepcopy__(self, memo):
-> 4815         dc = type(self)(**self._to_dictionary())
   4816         if isinstance(dc.data, np.ndarray):
   4817             dc.data = dc.data.copy()

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\kikuchipy\signals\ebsd.py in __init__(self, *args, **kwargs)
     99         optional input parameters.
    100         """
--> 101         Signal2D.__init__(self, *args, **kwargs)
    102 
    103         self._detector = kwargs.pop(

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\_signals\signal2d.py in __init__(self, *args, **kw)
    318 
    319     def __init__(self, *args, **kw):
--> 320         super().__init__(*args, **kw)
    321         if self.axes_manager.signal_dimension != 2:
    322             self.axes_manager.set_signal_dimension(2)

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in __init__(self, data, **kwds)
   2172             self.learning_results = LearningResults()
   2173             kwds['data'] = data
-> 2174             self._load_dictionary(kwds)
   2175             self._plot = None
   2176             self.inav = SpecialSlicersSignal(self, True)

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\signal.py in _load_dictionary(self, file_data_dict)
   2438         if 'axes' not in file_data_dict:
   2439             file_data_dict['axes'] = self._get_undefined_axes_list()
-> 2440         self.axes_manager = AxesManager(
   2441             file_data_dict['axes'])
   2442         if 'metadata' not in file_data_dict:

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\axes.py in __init__(self, axes_list)
    782             obj : The AxesManager that the event belongs to.
    783             """, arguments=['obj'])
--> 784         self.create_axes(axes_list)
    785         # set_signal_dimension is called only if there is no current
    786         # view. It defaults to spectrum

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\axes.py in create_axes(self, axes_list)
    969             axes_list.sort(key=lambda x: x['index_in_array'])
    970         for axis_dict in axes_list:
--> 971             self._append_axis(**axis_dict)
    972 
    973     def _update_max_index(self):

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\axes.py in _append_axis(self, *args, **kwargs)
   1032 
   1033     def _append_axis(self, *args, **kwargs):
-> 1034         axis = DataAxis(*args, **kwargs)
   1035         axis.axes_manager = self
   1036         self._axes.append(axis)

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\hyperspy\axes.py in __init__(self, size, index_in_array, name, scale, offset, units, navigate)
    262         self.high_index = self.size - 1
    263         self.low_index = 0
--> 264         self.index = 0
    265         self.update_axis()
    266         self.navigate = navigate

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\traits\trait_types.py in _validate(self, object, name, value)
   1819                 pass
   1820 
-> 1821         self.error(object, name, value)
   1822 
   1823     def _typed_value(self, value, low, high):

D:\Python\WPy64-3880\python-3.8.8.amd64\lib\site-packages\traits\base_trait_handler.py in error(self, object, name, value)
     72             The proposed new value for the attribute.
     73         """
---> 74         raise TraitError(
     75             object, name, self.full_info(object, name, value), value
     76         )

TraitError: The 'index' trait of a DataAxis instance must be 0 <= a number <= -1, but a value of 0 <class 'int'> was specified.

[hakonanes]

As above, I opened a separate issue. Thanks for reporting this!

[Fred-Ad]

I'd love helping, but I'm new in Python. Just know coding in mtex in Matlab.
If I can do something more, just tell me !

[hakonanes]

Just reporting these issues are helping making the package better! We have a contributing guide, which is nice to look over if you would like to contribute code, which can also be documentation (Jupyter notebook user guide), not just functionality!

Since we solved the original issue of reading the patterns correctly from a Bruker h5ebsd file with a region of interest, and since we have started the two separate issues to continue our troubleshooting there, do you think you can mark this discussion as resolved? E.g. marking #329 (comment) as the answer, since this shows the correct maps?