SpecWriterCallback file can't be plotted by newplot or SPEC's splot

[prjemian]

SPEC files produced by SpecWriterCallback cannot be plotted by newplot or SPEC's splot. Problems: newplot first complains about missing #P lines, splot complains the file is not in spec format.

However, PyMCA plots the data properly. So does NeXpy (although it complained the default data was not indicated after import).

Check these packages to see how they read SpecWriterCallback data files:

• newplot - Too old, can't read SPEC data files after 2006
• splot - reports #MD custom control lines as errors
• PyMCA - reads OK, plots OK
• NeXpy - imports OK, plots OK
• silx - converts to HDF5 OK, plots OK

[prjemian]

example SPEC data file as written:

20190704-163109.dat.txt

Produces this console report using NewPlot 2.75 in MS Windows:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\PYTHON23\Lib\lib-tk\Tkinter.py", line 1345, in __call__
  File "c:\NewplotBuilder\Specplot.py", line 302, in openfile
  File "c:\NewplotBuilder\WSource.py", line 57, in openfile
  File "c:\NewplotBuilder\SPSSelect.py", line 2241, in __init__
  File "c:\NewplotBuilder\SPSSelect.py", line 764, in create_file_widget
  File "c:\NewplotBuilder\SpecFileWidget.py", line 262, in open
  File "c:\NewplotBuilder\SpecFileWidget.py", line 636, in select
  File "c:\NewplotBuilder\SpecFileWidget.py", line 680, in refresh
  File "c:\NewplotBuilder\SPSSelect.py", line 1093, in file_scan_select
IndexError: column out of bounds

No earthly idea what situation triggered this message.

[prjemian]

NewPlot was last released in 2006. No useful diagnostics. Has problems with SPEC files produced by SPEC after 2006. Too old for us to try for compatibility.

[prjemian]

Looking through the splot source code, it is expecting to find #O (or #o) and #P control lines. Apparently the code creates an error message if there are "no motor names". The code also checks if the names are consistent for all scans from a header block. These control lines are not defined now.

It will be a challenge to produce the header lines since the motor names might change for each scan. Or, a new header block could be created for each scan... That's ugly.

[prjemian]

Also, splot catches this error message and informs the user that perhaps the file is not in spec format without explaining what triggered this conclusion.

[prjemian]

Start splot with -d option to get debug info

[prjemian]

Trying with a test data file (most data lines removed for brevity):

#F tuning.dat
#E 1550619537
#D Tue Feb 19 17:38:57 2019
#C BlueSky  user = mintadmin  host = mint-vm

#S 5  TuneAxis.tune()
#D Tue Feb 19 17:12:23 2019
#C Tue Feb 19 17:12:23 2019.  plan_type = generator
#C Tue Feb 19 17:12:23 2019.  uid = 957d83c1-9f11-47df-936b-ce14a396cd76
#MD md = {'activity': 'TuneAxis development and testing', 'peak_model': 'pseudo Voigt', 'peak_scale': 100000.0, 'peak_center': -1.3940973681450914, 'peak_sigma': 0.025534621641250733, 'peak_eta': 0.2993015167776747, 'peak_bkg': 0.0045580721055284755}
#MD tune_md = {'width': 2.5, 'initial_position': -1.5, 'time_iso8601': '2019-02-19 17:12:23.523994'}
#MD tune_parameters = {'num': 30, 'width': 2.5, 'initial_position': -1.5, 'peak_choice': 'cen', 'x_axis': 'm1', 'y_axis': 'spvoigt'}
#N 30
#L Epoch_float  m1  m1_user_setpoint  spvoigt  Epoch
1.4876151084899902 -2.75 -2.75 466.4182216464095 1
1.763300895690918 -2.66 -2.663793103448276 467.98001092446935 2
2.0559957027435303 -2.58 -2.5775862068965516 469.67695329594545 2
2.3592529296875 -2.49 -2.4913793103448274 472.047288907075 2
2.6607282161712646 -2.41 -2.405172413793103 474.7040562212433 3
#C Tue Feb 19 17:11:03 2019.  num_events_primary = 5
#C Tue Feb 19 17:11:03 2019.  num_events_PeakStats = 1
#C Tue Feb 19 17:11:03 2019.  exit_status = aborted

Here is my report of the errors found by the internal splot routine that reads the file:

scan: 5
command: TuneAxis.tune()
date: Tue Feb 19 17:12:23 2019
==== ==========================================
line error message                             
==== ==========================================
10   erroneous data / unknown header line (M)  
11   erroneous data / unknown header line (M)  
12   erroneous data / unknown header line (M)  
15   erroneous data / wrong number of columns  
16   erroneous data / wrong number of columns  
17   erroneous data / wrong number of columns  
18   erroneous data / wrong number of columns  
19   erroneous data / wrong number of columns  
--   no motor names                            
==== ==========================================

[prjemian]

Clearly, splot will never be able to read SPEC data files that include custom, multi-character control lines such as #MD. This is evident from the control line parser which only examines the first character of control lines to decide the function to handle that line. @jkirchman: This would make trouble for any of our SPEC users that write files with custom control lines and wish to plot those data using splot.

Still, it is disturbing that the splot reader has reported erroneous data / wrong number of columns in every data line. We'll work to resolve that, and work with the splot developer to accept custom control lines.

[prjemian]

Can also see if silx can read data files from our SpecWriterCallback.

[prjemian]

In SPEC data files, #N indicates the number of columns of data (not the number of expected data lines). Changing 30 to 5 in the above data example removes those error messages.

[prjemian]

try silx (see also prjemian/spec2nexus#141):

(base) mintadmin@mint-vm:~$ conda create -y -n silx python=3 silx -c conda-forge
...
(base) mintadmin@mint-vm:~$ conda activate silx
(silx) mintadmin@mint-vm:~$ cd /tmp
(silx) mintadmin@mint-vm:/tmp$ cp <apstools>/docs/source/resources/spec_tunes.dat ./
(silx) mintadmin@mint-vm:/tmp$ ipython
Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from silx.io.convert import convert                                                                                         

In [2]: convert("spec_tunes.dat", "spec_tunes.hdf5")                                                                                

Wow! That was easy. NeXPy can plot that (with a little help since the HDF5 files are not written with any NXentry/NXdata groups - data is in NXentry/NXcollection instead).

Here is the plot from silx view:

[prjemian]

With #203 in progress, this issue can be closed now.

[prjemian]

For completeness sake, here's the NeXPy plot: