Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Energy calibration bias shift #411

Merged
merged 23 commits into from
Jul 16, 2024
Merged

Conversation

rettigl
Copy link
Member

@rettigl rettigl commented Jun 2, 2024

Shift energy calibration by bias voltages, and apply shift of dataset bias upon adding energy column.
Unify dataframe channels config

closes #399
closes #138

@zain-sohail
Copy link
Member

I understand that you want to merge to v1 feature branch because these are all breaking changes?

@rettigl
Copy link
Member Author

rettigl commented Jun 13, 2024

I understand that you want to merge to v1 feature branch because these are all breaking changes?

Yes, they reqire configuration changes

@coveralls
Copy link
Collaborator

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9553353548

Details

  • 76 of 87 (87.36%) changed or added relevant lines in 3 files are covered.
  • 3 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.01%) to 91.952%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 51 60 85.0%
Files with Coverage Reduction New Missed Lines %
sed/binning/numba_bin.py 3 87.74%
Totals Coverage Status
Change from base Build 9522245431: -0.01%
Covered Lines: 6490
Relevant Lines: 7058

💛 - Coveralls

sed/loader/mpes/loader.py Outdated Show resolved Hide resolved
sed/config/mpes_example_config.yaml Outdated Show resolved Hide resolved
sed/config/mpes_example_config.yaml Outdated Show resolved Hide resolved
sed/loader/mpes/loader.py Outdated Show resolved Hide resolved
sed/config/mpes_example_config.yaml Show resolved Hide resolved
@rettigl
Copy link
Member Author

rettigl commented Jun 19, 2024

This code here unfortunately performs rather poorly. The generation of the per-file data columns alone make the computation ~20% or so slower, adding the energy offset per electron multiplies the binning time by a factor of almost 4!

@coveralls
Copy link
Collaborator

coveralls commented Jun 19, 2024

Pull Request Test Coverage Report for Build 9587172660

Details

  • 81 of 99 (81.82%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.05%) to 92.009%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/calibrator/energy.py 17 18 94.44%
sed/core/processor.py 6 9 66.67%
sed/loader/mpes/loader.py 58 72 80.56%
Totals Coverage Status
Change from base Build 9522245431: 0.05%
Covered Lines: 6494
Relevant Lines: 7058

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 19, 2024

Pull Request Test Coverage Report for Build 9587913234

Details

  • 87 of 99 (87.88%) changed or added relevant lines in 3 files are covered.
  • 3 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.01%) to 91.952%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 62 72 86.11%
Files with Coverage Reduction New Missed Lines %
sed/binning/numba_bin.py 3 87.74%
Totals Coverage Status
Change from base Build 9522245431: -0.01%
Covered Lines: 6490
Relevant Lines: 7058

💛 - Coveralls

@rettigl rettigl force-pushed the energy_calibration_bias_shift branch from 5f336a8 to 98c4e8c Compare June 19, 2024 21:00
@coveralls
Copy link
Collaborator

coveralls commented Jun 19, 2024

Pull Request Test Coverage Report for Build 9588080384

Details

  • 87 of 99 (87.88%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.08%) to 91.995%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 62 72 86.11%
Totals Coverage Status
Change from base Build 9588067031: 0.08%
Covered Lines: 6493
Relevant Lines: 7058

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 22, 2024

Pull Request Test Coverage Report for Build 9628853930

Details

  • 87 of 99 (87.88%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.08%) to 91.995%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 62 72 86.11%
Totals Coverage Status
Change from base Build 9588067031: 0.08%
Covered Lines: 6493
Relevant Lines: 7058

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 22, 2024

Pull Request Test Coverage Report for Build 9628937920

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.04%) to 91.956%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9588067031: 0.04%
Covered Lines: 6493
Relevant Lines: 7061

💛 - Coveralls

@rettigl rettigl mentioned this pull request Jun 23, 2024
12 tasks
@rettigl rettigl force-pushed the energy_calibration_bias_shift branch from 0004113 to c347bad Compare June 23, 2024 19:25
@coveralls
Copy link
Collaborator

coveralls commented Jun 23, 2024

Pull Request Test Coverage Report for Build 9635909706

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.04%) to 91.894%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9635884851: 0.04%
Covered Lines: 6462
Relevant Lines: 7032

💛 - Coveralls

@rettigl rettigl requested a review from zain-sohail June 23, 2024 19:36
@coveralls
Copy link
Collaborator

coveralls commented Jun 23, 2024

Pull Request Test Coverage Report for Build 9635950194

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 4 files are covered.
  • 3 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.02%) to 91.88%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 70 80 87.5%
Files with Coverage Reduction New Missed Lines %
sed/binning/numba_bin.py 3 87.5%
Totals Coverage Status
Change from base Build 9635919510: 0.02%
Covered Lines: 6461
Relevant Lines: 7032

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 23, 2024

Pull Request Test Coverage Report for Build 9636169386

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.04%) to 91.894%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9635919510: 0.04%
Covered Lines: 6462
Relevant Lines: 7032

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 23, 2024

Pull Request Test Coverage Report for Build 9636247350

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.08%) to 91.894%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9636221019: 0.08%
Covered Lines: 6462
Relevant Lines: 7032

💛 - Coveralls

@rettigl rettigl mentioned this pull request Jun 24, 2024
@rettigl rettigl force-pushed the energy_calibration_bias_shift branch from 2aaf8b5 to 54f9673 Compare June 24, 2024 20:56
@coveralls
Copy link
Collaborator

coveralls commented Jun 24, 2024

Pull Request Test Coverage Report for Build 9652315474

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.04%) to 91.897%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/core/processor.py 7 9 77.78%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9652279283: 0.04%
Covered Lines: 6464
Relevant Lines: 7034

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 25, 2024

Pull Request Test Coverage Report for Build 9659018433

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.1%) to 91.914%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/calibrator/energy.py 25 27 92.59%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9652390026: 0.1%
Covered Lines: 6468
Relevant Lines: 7037

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jun 30, 2024

Pull Request Test Coverage Report for Build 9733951548

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.05%) to 92.511%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/calibrator/energy.py 25 27 92.59%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9733946272: 0.05%
Covered Lines: 6905
Relevant Lines: 7464

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jul 2, 2024

Pull Request Test Coverage Report for Build 9757007395

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.05%) to 92.511%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/calibrator/energy.py 25 27 92.59%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9756916343: 0.05%
Covered Lines: 6905
Relevant Lines: 7464

💛 - Coveralls

@coveralls
Copy link
Collaborator

coveralls commented Jul 2, 2024

Pull Request Test Coverage Report for Build 9759204395

Details

  • 104 of 116 (89.66%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.05%) to 92.511%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/calibrator/energy.py 25 27 92.59%
sed/loader/mpes/loader.py 70 80 87.5%
Totals Coverage Status
Change from base Build 9756916343: 0.05%
Covered Lines: 6905
Relevant Lines: 7464

💛 - Coveralls

@zain-sohail zain-sohail mentioned this pull request Jul 2, 2024
11 tasks
@coveralls
Copy link
Collaborator

coveralls commented Jul 8, 2024

Pull Request Test Coverage Report for Build 9847465611

Details

  • 159 of 169 (94.08%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.1%) to 92.599%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/calibrator/energy.py 25 27 92.59%
sed/loader/mpes/loader.py 80 88 90.91%
Totals Coverage Status
Change from base Build 9845601339: 0.1%
Covered Lines: 6969
Relevant Lines: 7526

💛 - Coveralls

Copy link
Member

@zain-sohail zain-sohail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried to review but currently, I don't understand the internals of the calibration methods enough to review this properly. I have tried because I wanted to adapt these for the side band correction but failed to understand.
Honestly, It'd be great if a small guide on the internals can be made. We can then also put it in user documentation.
If that's too much, if you could explain why you made these changes, I should be able to review based on that.

The only thing I noticed currently was that the tutorials still have the ref_id (in methods find_bias_peaks etc.) whereas the code only uses the ref_energy now.

@rettigl
Copy link
Member Author

rettigl commented Jul 8, 2024

I have tried to review but currently, I don't understand the internals of the calibration methods enough to review this properly. I have tried because I wanted to adapt these for the side band correction but failed to understand. Honestly, It'd be great if a small guide on the internals can be made. We can then also put it in user documentation. If that's too much, if you could explain why you made these changes, I should be able to review based on that.

The only thing I noticed currently was that the tutorials still have the ref_id (in methods find_bias_peaks etc.) whereas the code only uses the ref_energy now.

Thank you for your effort. I will for now comment on the changes to explain them better. If I find time, I can also make a more general documentation item out of this.

sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/core/dfops.py Show resolved Hide resolved
sed/loader/mpes/loader.py Show resolved Hide resolved
@rettigl
Copy link
Member Author

rettigl commented Jul 8, 2024

For using the calibration function for features from sidebands, do the following:

  1. Extract the TOF values of the sidebands (e.g. TOF=[t0, t1, t2, t3, t4, t5, t6]).
  2. Generate the corresponding energies, shifted by the bias voltage of your measurement, e.g. energies=[-3hv, -2hv, -hv, 0, hv, 2hv, 3hv] - (maybe +) bias
  3. Calculate the energy of the first side band, e.g. E_corelevel - 3hv
  4. Provide these values to ec.calibrate(ref_energy=reference, landmarks=TOF, biases=energies, ...)
    This should work, I think.

Copy link
Member

@zain-sohail zain-sohail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few minor comments. Thanks for explaining! Helped a lot. Could be a start to the calibration guide later.

sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
sed/calibrator/energy.py Show resolved Hide resolved
@rettigl rettigl merged commit cb7c97a into v1_feature_branch Jul 16, 2024
1 of 2 checks passed
@rettigl rettigl deleted the energy_calibration_bias_shift branch July 16, 2024 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Offset energy calibration by sample bias Unify axis naming and definition
3 participants