-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
export coefficients to json file #58
Conversation
Next step is the user interaction. I would add another section into the pygac config which gives the possibility to add the path to a user defined |
Being able to just override a small subset of coefficients would be a nice feature! Users focused on traceability can still download an entire set of coefficients via DOI or so. I'm ok with the config section. Can you please also make this an argument of |
Good idea @sfinkens. |
Regarding the coeffs names, I would also vote for having a documentation page explaining each coeff and the formulas they are applied in. |
pygac/calibration.py
Outdated
key: cls.parse(custom.get(key) or default.get(key)) | ||
for key in cls.fields |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
key: cls.parse(custom.get(key) or default.get(key)) | |
for key in cls.fields | |
**default, **custom |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In principle this take default values from default
and uses custom
coeffs when availble
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only available for python 3.5+, so it won't survive the CI checks for python 2.7... The backward compatible way I found is:
spacecraft_coeffs = default.copy()
spacecraft_coeffs.update(custom)
In my opinion, we can do without python 2 compatibility, but I do not know if it is also your intention.
And again a feature missing in python 2.... In ancient times, the |
With the last commits, the user can provide different default coefficients via the config file
If coeffs_file is not given, it will fallback to the pygac defaults in This covers the three use cases:
So far the implementation of this feature... Now, we should decide for a good naming convention of these parameters. |
Great job. Thanks a lot @carloshorn for this adaptation. |
Nice work @carloshorn ! The |
Having in mind that I wanted to avoid list types in the json to give every item a specific name, I came up with the following naming convention, which makes use of the hierarchical json structure. "metopb": {
"channel_1": {
"dark_count": 39.7,
"gain_high_s0": 0.166,
"gain_high_s1": 2.019,
"gain_high_s2": -0.201,
"gain_low_s0": 0.055,
"gain_low_s1": 2.019,
"gain_low_s2": -0.201,
"gain_switch": 501.12
},
"channel_2": {
"dark_count": 40.0,
"gain_high_s0": 0.183,
"gain_high_s1": 1.476,
"gain_high_s2": -0.137,
"gain_low_s0": 0.061,
"gain_low_s1": 1.476,
"gain_low_s2": -0.137,
"gain_switch": 500.82
},
"channel_3a": {
"dark_count": 40.3,
"gain_high_s0": 0.201,
"gain_high_s1": 1.748,
"gain_high_s2": -0.033,
"gain_low_s0": 0.029,
"gain_low_s1": 1.748,
"gain_low_s2": -0.033,
"gain_switch": 501.32
},
"channel_3b": {
"centroid_wavenumber": 2664.3384,
"radiance_correction_c0": 0.0,
"radiance_correction_c1": 1.0,
"radiance_correction_c2": 0.0,
"space_radiance": 0.0,
"to_eff_blackbody_intercept": 0.9970158319134996,
"to_eff_blackbody_slope": 1.7711318
},
"channel_4": {
"centroid_wavenumber": 933.71521,
"radiance_correction_c0": 5.44,
"radiance_correction_c1": 0.89848,
"radiance_correction_c2": 0.00046964,
"space_radiance": -4.98,
"to_eff_blackbody_intercept": 0.9986240957209157,
"to_eff_blackbody_slope": 0.51860807
},
"channel_5": {
"centroid_wavenumber": 839.72764,
"radiance_correction_c0": 3.84,
"radiance_correction_c1": 0.93751,
"radiance_correction_c2": 0.00025239,
"space_radiance": -3.4,
"to_eff_blackbody_intercept": 0.9988311677674785,
"to_eff_blackbody_slope": 0.40059787
},
"date_of_launch": 2012.77,
"thermometer_1": {
"c0": 276.6194,
"c1": 0.050919,
"c2": 1.471e-06,
"c3": 0.0,
"c4": 0.0
},
"thermometer_2": {
"c0": 276.6511,
"c1": 0.050892,
"c2": 1.489e-06,
"c3": 0.0,
"c4": 0.0
},
"thermometer_3": {
"c0": 276.6597,
"c1": 0.050845,
"c2": 1.521e-06,
"c3": 0.0,
"c4": 0.0
},
"thermometer_4": {
"c0": 276.3685,
"c1": 0.050992,
"c2": 1.482e-06,
"c3": 0.0,
"c4": 0.0
}
} Even if some coefficients have the same prefix, I think that two levels provide a good readability. The only thing that I would like to change is the launch date format. A fraction of year is not very intuitive. I would prefer a classical timestamp like Edit: Maybe someone can come up with a better name for to_eff_blackbody, which should point out that this coefficients should be used for the transformation from thermometer temperature to effective black body temperature. as defined in (7.1.2.4-3) KLM Users Guide. |
Looks good! for the thermometer coeffs, why not skip |
I would prefer to keep Lines 495 to 499 in 2d7fbaf
If we default them to zero in the code, I would not see the point of having a default value file. |
Good point, I agree, thank you for explaining this :) |
I will start the implementation of the new structure tomorrow. I would also like to put more comment lines in the calibration functions and maybe rename some inner variables, but it should neither chnage the results, nor should it change the interface (function argument names). Are you okay with adding it into this PR? |
Go ahead. |
Even comment lines can break python 2.7.... |
Cleanup and finalisation plan: if we agree on the coefficients, I would remove the |
@carloshorn Looks and sounds very good! Thanks! |
No... I simply forgot it... |
Nice! I would prefer the optional correction for Sun-earth-distance in the |
I think it's good if the coefs are identical, so I'd say keep the rounding. I don't have an opinion about the scaled/isotropic radiance thing :) |
I agree with keeping the rounding. Is the scaled radiance being output at the moment to hdf files? |
Yes.
In pygac 1.2 (CLARA-A2 L1C data) the scaled radiance output was corrected for changing Sun-Earth-distance, but not anymore in the updated pygac Version for the test CLARA-A3 L1C data. |
Last thing to do are tests:
Anything I forgot? |
I implemented the tests into |
@sfinkens, you once mentioned that in order to use coefficients of older PyGAC versions, you cherry-picked them from previous commits. If wanted, you could tell me which commits you used, and which version tag I should give them, so I can produce json files for them and add them together with their md5 hash. |
@carloshorn That's very kind of you, but there is no need to do this for our past activities. That's frozen anyway. But for the future it will be very helpful! Haven't looked at the tests in detail, buy your proposal sounds good. And a separate test case is a good choice in my opinion! |
Great work @carloshorn ! I agree with @sfinkens for both points. |
Alright, I hope that's it. In summary:
If the user passes an official PyGAC defaults file (currently there is only one), the code will recognise it. At runtime, it is possible to check the calibration file version using the class method Furthermore, it is possible to change individual coefficients by passing the satellite specific changes to the my_calibration= {"channel_1": {"dark_count": 43.21}}
filepath = "/path/to/file"
Reader = get_reader_class(filepath)
reader = Reader(calibration=my_calibration)
reader.read(filepath) In addition, the functions of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice, I can't find anything but a single typo 😄 And so much documentation for the calibration! Love it!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for making the coeffs configurable!
In favour of giving more control over the calibration coefficients to the user, this PR will export the coefficients from
calibration.py
to an external data file.Closes #62