Example for FLASH nexus conversion/metadata collection #420

rettigl · 2024-06-18T20:59:40Z

This PR is intended for discussing necessary/missing changes to the FLASH metadata structure for conversion to nexus.

rettigl

Comments on the changes I did and TODOs

sed/config/NXmpes_config-HEXTOF.json

coveralls · 2024-07-05T17:06:23Z

Pull Request Test Coverage Report for Build 9811460900

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 91.888%

Totals
Change from base Build 9811469473:	0.0%
Covered Lines:	6457
Relevant Lines:	7027

💛 - Coveralls

zain-sohail

Dima sent me the updated nexus config file so I am just putting it here

sed/config/NXmpes_config-HEXTOF.json

coveralls · 2024-07-10T18:35:09Z

Pull Request Test Coverage Report for Build 11784465431

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 91.942%

Totals
Change from base Build 11784242100:	0.04%
Covered Lines:	6492
Relevant Lines:	7061

💛 - Coveralls

rettigl · 2024-10-02T20:15:26Z

@kutnyakhov @zain-sohail Running the current state (rebased to master) outside maxwell results in the following warnings during saving:

Using mpes reader to convert the given files:  
• ../sed/config/NXmpes_config-HEXTOF.json
Could not find value for key /ENTRY[entry]/entry_identifier with value @attrs:metadata/_id.
Tried prefixes: [('@attrs', 'metadata/_id')].
Could not find value for key /ENTRY[entry]/USER[user0]/name with value @attrs:metadata/pi/last_name.
Tried prefixes: [('@attrs', 'metadata/pi/last_name')].
Could not find value for key /ENTRY[entry]/USER[user0]/affiliation with value @attrs:metadata/pi/institute.
Tried prefixes: [('@attrs', 'metadata/pi/institute')].
Could not find value for key /ENTRY[entry]/USER[user0]/email with value @attrs:metadata/pi/email.
Tried prefixes: [('@attrs', 'metadata/pi/email')].
Could not find value for key /ENTRY[entry]/USER[user1]/name with value @attrs:metadata/leader/last_name.
Tried prefixes: [('@attrs', 'metadata/leader/last_name')].
Could not find value for key /ENTRY[entry]/USER[user1]/affiliation with value @attrs:metadata/leader/institute.
Tried prefixes: [('@attrs', 'metadata/leader/institute')].
Could not find value for key /ENTRY[entry]/USER[user1]/email with value @attrs:metadata/leader/email.
Tried prefixes: [('@attrs', 'metadata/leader/email')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/projection with value @attrs:metadata/scientificMetadata/collectioncolumn/projection.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/projection')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/lens_mode with value @attrs:metadata/scientificMetadata/collectioncolumn/lens_mode.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/lens_mode')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/extractor_voltage with value @attrs:metadata/scientificMetadata/collectioncolumn/extractor_voltage/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/extractor_voltage/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/extractor_voltage/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/extractor_voltage/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/extractor_voltage/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/extractor_current with value @attrs:metadata/scientificMetadata/collectioncolumn/extractor_current/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/extractor_current/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/extractor_current/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/extractor_current/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/extractor_current/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/shape with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_shape.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_shape')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/size with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_size/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_size/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/size/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_size/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_size/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/POSITIONER[fa_m1]/value with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_x/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_x/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/POSITIONER[fa_m1]/value/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_x/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_x/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/POSITIONER[fa_m2]/value with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_y/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_y/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/field_aperture/POSITIONER[fa_m2]/value/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/field_aperture_y/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/field_aperture_y/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/contrast_aperture/size with value @attrs:metadata/scientificMetadata/collectioncolumn/contrast_aperture_size/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/contrast_aperture_size/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/COLLECTIONCOLUMN[collectioncolumn]/contrast_aperture/size/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/contrast_aperture_size/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/contrast_aperture_size/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/ENERGYDISPERSION[energydispersion]/pass_energy with value @attrs:metadata/scientificMetadata/collectioncolumn/tof_voltage/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/tof_voltage/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/ENERGYDISPERSION[energydispersion]/pass_energy/@units with value @attrs:metadata/scientificMetadata/collectioncolumn/tof_voltage/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/collectioncolumn/tof_voltage/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/amplifier_type with value @attrs:metadata/scientificMetadata/detector/amplifier_type.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/amplifier_type')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/detector_type with value @attrs:metadata/scientificMetadata/detector/detector_type.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/detector_type')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/sensor_pixels with value @attrs:metadata/scientificMetadata/detector/sensor_pixels.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/sensor_pixels')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/amplifier_bias with value @attrs:metadata/scientificMetadata/detector/amplifier_bias/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/amplifier_bias/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/amplifier_bias/@units with value @attrs:metadata/scientificMetadata/detector/amplifier_bias/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/amplifier_bias/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/amplifier_voltage with value @attrs:metadata/scientificMetadata/detector/amplifier_voltage/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/amplifier_voltage/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/amplifier_voltage/@units with value @attrs:metadata/scientificMetadata/detector/amplifier_voltage/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/amplifier_voltage/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/detector_voltage with value @attrs:metadata/scientificMetadata/detector/detector_voltage/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/detector_voltage/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR[detector]/detector_voltage/@units with value @attrs:metadata/scientificMetadata/detector/detector_voltage/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/detector/detector_voltage/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/name with value @attrs:metadata/scientificMetadata/source/name.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/name')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/probe with value @attrs:metadata/scientificMetadata/source/probe.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/probe')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/type with value @attrs:metadata/scientificMetadata/source/type.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/type')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/frequency with value @attrs:metadata/scientificMetadata/source/repetition_rate/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/repetition_rate/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/frequency/@units with value @attrs:metadata/scientificMetadata/source/repetition_rate/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/repetition_rate/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_probe]/incident_energy with value @attrs:metadata/scientificMetadata/source/photon_energy/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/photon_energy/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_probe]/incident_energy/@units with value @attrs:metadata/scientificMetadata/source/dispersion/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/dispersion/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_probe]/incident_energy_dispersion with value @attrs:metadata/scientificMetadata/source/dispersion/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/dispersion/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_probe]/incident_energy_dispersion/@units with value @attrs:metadata/scientificMetadata/source/dispersion/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/source/dispersion/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_wavelength with value @attrs:metadata/scientificMetadata/laser/wavelength/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/wavelength/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_wavelength/@units with value @attrs:metadata/scientificMetadata/laser/wavelength/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/wavelength/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_energy with value @attrs:metadata/scientificMetadata/laser/incident_energy/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/incident_energy/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_energy/@units with value @attrs:metadata/scientificMetadata/laser/incident_energy/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/incident_energy/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/pulse_duration with value @attrs:metadata/scientificMetadata/laser/pulse_duration/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/pulse_duration/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/pulse_duration/@units with value @attrs:metadata/scientificMetadata/laser/pulse_duration/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/pulse_duration/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_polarization with value @attrs:metadata/scientificMetadata/laser/incident_polarization/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/incident_polarization/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_polarization/@units with value @attrs:metadata/scientificMetadata/laser/incident_polarization/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/incident_polarization/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/pulse_energy with value @attrs:metadata/scientificMetadata/laser/pulse_energy/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/pulse_energy/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/pulse_energy/@units with value @attrs:metadata/scientificMetadata/laser/pulse_energy/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/laser/pulse_energy/unit')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/MANIPULATOR[manipulator]/type with value @attrs:metadata/scientificMetadata/manipulator/type.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/manipulator/type')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/MANIPULATOR[manipulator]/sample_bias_voltmeter/value with value @attrs:metadata/scientificMetadata/manipulator/sample_bias/value.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/manipulator/sample_bias/value')].
Could not find value for key /ENTRY[entry]/INSTRUMENT[instrument]/MANIPULATOR[manipulator]/sample_bias_voltmeter/value/@units with value @attrs:metadata/scientificMetadata/manipulator/sample_bias/unit.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/manipulator/sample_bias/unit')].
Could not find value for key /ENTRY[entry]/SAMPLE[sample]/name with value @attrs:metadata/scientificMetadata/sample/sample_name.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/sample/sample_name')].
Could not find value for key /ENTRY[entry]/SAMPLE[sample]/preparation_description with value @attrs:metadata/scientificMetadata/sample/preparation_description.
Tried prefixes: [('@attrs', 'metadata/scientificMetadata/sample/preparation_description')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/DISTORTION[distortion]/symmetry with value @attrs:metadata/momentum_correction/correction/rotsym.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/correction/rotsym')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/DISTORTION[distortion]/original_centre with value @attrs:metadata/momentum_correction/correction/pcent.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/correction/pcent')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/DISTORTION[distortion]/original_points with value @attrs:metadata/momentum_correction/correction/pouter.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/correction/pouter')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/DISTORTION[distortion]/cdeform_field with value @attrs:metadata/momentum_correction/correction/cdeform_field.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/correction/cdeform_field')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/DISTORTION[distortion]/rdeform_field with value @attrs:metadata/momentum_correction/correction/rdeform_field.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/correction/rdeform_field')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/depends_on with value @attrs:metadata/momentum_correction/registration/depends_on.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/depends_on')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_x] with value @attrs:metadata/momentum_correction/registration/trans_x/value.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_x/value')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_x]/@transformation_type with value @attrs:metadata/momentum_correction/registration/trans_x/type.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_x/type')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_x]/@units with value @attrs:metadata/momentum_correction/registration/trans_x/units.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_x/units')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_x]/@vector with value @attrs:metadata/momentum_correction/registration/trans_x/vector.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_x/vector')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_x]/@depends_on with value @attrs:metadata/momentum_correction/registration/trans_x/depends_on.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_x/depends_on')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_y] with value @attrs:metadata/momentum_correction/registration/trans_y/value.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_y/value')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_y]/@units with value @attrs:metadata/momentum_correction/registration/trans_y/units.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_y/units')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_y]/@transformation_type with value @attrs:metadata/momentum_correction/registration/trans_y/type.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_y/type')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_y]/@vector with value @attrs:metadata/momentum_correction/registration/trans_y/vector.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_y/vector')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[trans_y]/@depends_on with value @attrs:metadata/momentum_correction/registration/trans_y/depends_on.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/trans_y/depends_on')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[rot_z] with value @attrs:metadata/momentum_correction/registration/rot_z/value.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/rot_z/value')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[rot_z]/@units with value @attrs:metadata/momentum_correction/registration/rot_z/units.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/rot_z/units')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[rot_z]/@transformation_type with value @attrs:metadata/momentum_correction/registration/rot_z/type.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/rot_z/type')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[rot_z]/@offset with value @attrs:metadata/momentum_correction/registration/rot_z/offset.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/rot_z/offset')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[rot_z]/@vector with value @attrs:metadata/momentum_correction/registration/rot_z/vector.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/rot_z/vector')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/REGISTRATION[registration]/TRANSFORMATIONS[transformations]/AXISNAME[rot_z]/@depends_on with value @attrs:metadata/momentum_correction/registration/rot_z/depends_on.
Tried prefixes: [('@attrs', 'metadata/momentum_correction/registration/rot_z/depends_on')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/CALIBRATION[kx_calibration]/scaling with value @attrs:metadata/momentum_calibration/calibration/kx_scale.
Tried prefixes: [('@attrs', 'metadata/momentum_calibration/calibration/kx_scale')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/CALIBRATION[kx_calibration]/offset with value @attrs:metadata/momentum_calibration/calibration/x_center.
Tried prefixes: [('@attrs', 'metadata/momentum_calibration/calibration/x_center')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/CALIBRATION[kx_calibration]/calibrated_axis with value @attrs:metadata/momentum_calibration/calibration/kx_axis.
Tried prefixes: [('@attrs', 'metadata/momentum_calibration/calibration/kx_axis')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/CALIBRATION[ky_calibration]/scaling with value @attrs:metadata/momentum_calibration/calibration/ky_scale.
Tried prefixes: [('@attrs', 'metadata/momentum_calibration/calibration/ky_scale')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/CALIBRATION[ky_calibration]/offset with value @attrs:metadata/momentum_calibration/calibration/y_center.
Tried prefixes: [('@attrs', 'metadata/momentum_calibration/calibration/y_center')].
Could not find value for key /ENTRY[entry]/PROCESS[process]/CALIBRATION[ky_calibration]/calibrated_axis with value @attrs:metadata/momentum_calibration/calibration/ky_axis.
Tried prefixes: [('@attrs', 'metadata/momentum_calibration/calibration/ky_axis')].
The entry '/ENTRY[entry]/definition/@version' (value: v2022.07.post1.dev1490+gd3c5237d) should not be changed by the reader. The provided version 'None' is kept. We assume you know what you are doing.
WARNING: The data entry corresponding to /ENTRY[entry]/USER[user0]/name is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/USER[user0]/affiliation is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/USER[user1]/name is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/USER[user1]/affiliation is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/type is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_probe]/incident_energy is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/INSTRUMENT[instrument]/beamTYPE[beam_pump]/incident_energy is required and hasn't been supplied by the reader.
WARNING: The required group, /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYSER[electronanalyser]/DETECTOR, hasn't been supplied.
WARNING: The data entry corresponding to /ENTRY[entry]/INSTRUMENT[instrument]/MANIPULATOR[manipulator]/sample_bias_voltmeter/value is required and hasn't been supplied by the reader.
WARNING: The data entry corresponding to /ENTRY[entry]/SAMPLE[sample]/name is required and hasn't been supplied by the reader.
WARNING: Field /ENTRY[entry]/SAMPLE[sample]/temperature/temperature_sensor written without documentation.
WARNING: Field /ENTRY[entry]/SAMPLE[sample]/gas_pressure/pressure_gauge written without documentation.
WARNING: Field /ENTRY[entry]/PROCESS[process]/energy_calibration/coefficients written without documentation.
WARNING: Field /ENTRY[entry]/PROCESS[process]/energy_calibration/fit_function written without documentation.
WARNING: Field /ENTRY[entry]/PROCESS[process]/energy_calibration/original_axis written without documentation.
WARNING: Field /ENTRY[entry]/PROCESS[process]/energy_calibration/calibrated_axis written without documentation.
The output file generated: runs44824-27.nxs.

I.e. a huge number of metadata is not being found.

Running the same on Maxwell results in an error when trying to load the metadata file:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[46], line 3
      1 from sed.core.config import load_config
      2 import numpy as np
----> 3 metadata = load_config(path + "/44824_20230324T060430.json")
      5 # Fix metadata
      6 metadata["scientificMetadata"]["Laser"]["wavelength"]["value"] = float(metadata["scientificMetadata"]["Laser"]["wavelength"]["value"][:-2])

File ~/sed/sed/core/config.py:159, in load_config(config_path)
    157 config_file = Path(config_path)
    158 if not config_file.is_file():
--> 159     raise FileNotFoundError(
    160         f"could not find the configuration file: {config_file}",
    161     )
    163 if config_file.suffix == ".json":
    164     with open(config_file, encoding="utf-8") as stream:

FileNotFoundError: could not find the configuration file: /asap3/flash/gpfs/pg2/2023/data/11019101/raw/hdf/offline/fl1user3/44824_20230324T060430.json

kutnyakhov · 2024-10-10T09:53:43Z

@rettigl for FLASH data we are using the SciCat catalogue for the metadata. In order to get them you need to get your personal SciCat Token from Settings and add/update it into your config file under metadata: section as scicat_token:
or add that info manually:

config_override['metadata'] = {}
config_override['metadata'] = {
    'scicat_url': 'https://scicat-flash-test1.desy.de/api/v3',
    'scicat_token': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJfaWQiOiI2Mzc0YmYyMmE4NWQ0N2QwMGE5MWRlN2MiLCJ1c2VybmFtZSI6Imt1dG55YWtkIiwiZW1haWwiOiJkbXl0cm8ua3V0bnlha2hvdkBkZXN5LmRlIiwiYXV0aFN0cmF0ZWd5IjoibG9jYWwiLCJpZCI6IjYzNzRiZjIyYTg1ZDQ3ZDAwYTkxZGU3YyIsInVzZXJJZCI6IjYzNzRiZjIyYTg1ZDQ3ZDAwYTkxZGU3YyIsImlhdCI6MTcyODU1MDc5NCwiZXhwIjoxNzI4NTkzOTk0fQ.Hqy6ICvBBMLMq8KwCLf-5tRsA-BdWGfMCR9mjWv4ldU'
}

Also, as discussed during last meeting, we still have problems that some user info is missing and IT guys are working to solve this problem for future, but for now we have to also read additionally beamtime metadata file:

f = open('/asap3/flash/gpfs/pg2//2023/data/11019101/beamtime-metadata-11019101.json', "r")
beamtime_metadata = json.loads(f.read())

beamtime_metadata_contact = {'leader': beamtime_metadata['leader'], 'pi': beamtime_metadata['pi']}

and accordingly, add this info while reading data

sp = SedProcessor(
    runs=[44824,44825,44826,44827],
    config=config_override,
    system_config=config_file,
    metadata=beamtime_metadata_contact,
    collect_metadata=True
)

rettigl · 2024-10-10T10:03:31Z

for FLASH data we are using the SciCat catalogue for the metadata. In order to get them you need to get your personal SciCat Token from Settings and add/update it into your config file under metadata: section as scicat_token:
or add that info manually:

For general usage that is fine, but for the example we need to provide a solution that works without this, i.e. put the scicat metadata into a file and provide it with the example data. This exists in the zenodo upload, but not in the data on the maxwell cluster, thus it fails there.

Also, as discussed during last meeting, we still have problems that some user info is missing and IT guys are working to solve this problem for future, but for now we have to also read additionally beamtime metadata file:

This file again is not part of the upload on zenodo, so again it would fail if you are not on maxwell

Even with these additions, there are a lot of metadata not being found by the reader, because the information in the config file is wrong. This should be fixed for this example, that is my point. If some data are not available for this, let's just remove the respective entries and add them later once these data are available.

rettigl · 2024-10-25T14:52:00Z

@zain-sohail @kutnyakhov I removed everything that does not exist here, and corrected to reflect the paths in the metadata files.
The example still fails on maxwell because the metadata json files are not present there. I suggest to copy them there (I cannot do it), to make this work for now. If more metadata are readily available, this can still be added later.

tutorial/4_hextof_workflow.ipynb

rettigl commented Jun 18, 2024

View reviewed changes

rettigl mentioned this pull request Jun 18, 2024

Add nexus definitions for hextof in the repo #417

Closed

zain-sohail assigned kutnyakhov Jun 19, 2024

rettigl linked an issue Jul 2, 2024 that may be closed by this pull request

Add nexus definitions for hextof in the repo #417

Closed

rettigl force-pushed the flash_to_nexus branch from 76bb1a0 to 4856e81 Compare July 5, 2024 17:00

zain-sohail reviewed Jul 10, 2024

View reviewed changes

sed/config/NXmpes_config-HEXTOF.json Outdated Show resolved Hide resolved

sed/config/NXmpes_config-HEXTOF.json Outdated Show resolved Hide resolved

zain-sohail requested a review from kutnyakhov July 16, 2024 16:54

rettigl deleted the branch main July 18, 2024 07:21

rettigl closed this Jul 18, 2024

rettigl reopened this Jul 18, 2024

rettigl force-pushed the flash_to_nexus branch from bc7ddc6 to f312e94 Compare July 18, 2024 10:35

rettigl changed the base branch from flash_to_nexus to main July 18, 2024 10:36

zain-sohail approved these changes Jul 26, 2024

View reviewed changes

rettigl and others added 6 commits October 2, 2024 20:59

original version of FLASH NXmpes config

fe9c57e

add example and fix various fields

81d5c4b

update pynxtools version

c43952c

add paths for monochomator dispersion

7a3f067

update pynxtools

c76dc9f

updates from dima

f1e11da

rettigl force-pushed the flash_to_nexus_fixes branch from 682d3d7 to f1e11da Compare October 2, 2024 19:58

rettigl added 2 commits October 25, 2024 16:29

update config file

4790410

Merge remote-tracking branch 'origin/main' into flash_to_nexus_fixes

970b2bf

zain-sohail reviewed Nov 10, 2024

View reviewed changes

tutorial/4_hextof_workflow.ipynb Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/main' into flash_to_nexus_fixes

494c41e

add meta_path

fc818a3

rettigl merged commit 4c4fada into main Nov 11, 2024
6 checks passed

rettigl deleted the flash_to_nexus_fixes branch November 11, 2024 18:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example for FLASH nexus conversion/metadata collection #420

Example for FLASH nexus conversion/metadata collection #420

rettigl commented Jun 18, 2024

rettigl left a comment

coveralls commented Jul 5, 2024 •

edited

Loading

zain-sohail left a comment

coveralls commented Jul 10, 2024 •

edited

Loading

rettigl commented Oct 2, 2024

kutnyakhov commented Oct 10, 2024

rettigl commented Oct 10, 2024

rettigl commented Oct 25, 2024

Example for FLASH nexus conversion/metadata collection #420

Example for FLASH nexus conversion/metadata collection #420

Conversation

rettigl commented Jun 18, 2024

rettigl left a comment

Choose a reason for hiding this comment

coveralls commented Jul 5, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9811460900

Details

💛 - Coveralls

zain-sohail left a comment

Choose a reason for hiding this comment

coveralls commented Jul 10, 2024 • edited Loading

Pull Request Test Coverage Report for Build 11784465431

Details

💛 - Coveralls

rettigl commented Oct 2, 2024

kutnyakhov commented Oct 10, 2024

rettigl commented Oct 10, 2024

rettigl commented Oct 25, 2024

coveralls commented Jul 5, 2024 •

edited

Loading

coveralls commented Jul 10, 2024 •

edited

Loading