Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix start/end time properties of hrit_jma reader #2851

Merged
merged 6 commits into from
Jul 19, 2024

Conversation

sfinkens
Copy link
Member

@sfinkens sfinkens commented Jul 5, 2024

Fix start/end_time properties of hrit_jma reader to return datetime objects (currently integers).

Reason: datetime64.astype(dt.datetime) doesn't always return a datetime object. (see numpy/numpy#20351).

>>> np.datetime64("2000-01-01T00:00:00.000001").astype(dt.datetime)  # microsecond precision
datetime.datetime(2000, 1, 1, 0, 0, 0, 1)
>>> np.datetime64("2000-01-01T00:00:00.000000001").astype(dt.datetime)  # nanosecond precision
946684800000000001

I've updated usages of datetime64.astype(dt.datetime) in other readers, too. Although it doesn't seem to be a problem there, because they use micro/millisecond precision.

  • Tests added

@sfinkens sfinkens requested review from mraspaud and djhoese as code owners July 5, 2024 11:03
Copy link

codecov bot commented Jul 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.95%. Comparing base (440925d) to head (4011541).
Report is 497 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2851      +/-   ##
==========================================
- Coverage   95.96%   95.95%   -0.02%     
==========================================
  Files         366      366              
  Lines       53593    53619      +26     
==========================================
+ Hits        51433    51452      +19     
- Misses       2160     2167       +7     
Flag Coverage Δ
behaviourtests 4.05% <4.76%> (+<0.01%) ⬆️
unittests 96.05% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Jul 5, 2024

Pull Request Test Coverage Report for Build 9807301260

Details

  • 23 of 24 (95.83%) changed or added relevant lines in 7 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.002%) to 96.069%

Changes Missing Coverage Covered Lines Changed/Added Lines %
satpy/readers/hrit_jma.py 2 3 66.67%
Totals Coverage Status
Change from base Build 9765409550: 0.002%
Covered Lines: 51670
Relevant Lines: 53784

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jul 5, 2024

Pull Request Test Coverage Report for Build 9807923958

Details

  • 24 of 24 (100.0%) changed or added relevant lines in 7 files are covered.
  • 8 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.01%) to 96.056%

Files with Coverage Reduction New Missed Lines %
satpy/tests/utils.py 2 93.16%
satpy/tests/reader_tests/gms/test_gms5_vissr_l1b.py 3 98.67%
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 3 97.18%
Totals Coverage Status
Change from base Build 9765409550: -0.01%
Covered Lines: 51663
Relevant Lines: 53784

💛 - Coveralls

satpy/readers/viirs_vgac_l1c_nc.py Outdated Show resolved Hide resolved
satpy/utils.py Outdated Show resolved Hide resolved
@coveralls
Copy link

Pull Request Test Coverage Report for Build 9971081072

Details

  • 21 of 21 (100.0%) changed or added relevant lines in 6 files are covered.
  • 8 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.01%) to 96.058%

Files with Coverage Reduction New Missed Lines %
satpy/tests/utils.py 2 93.16%
satpy/tests/reader_tests/gms/test_gms5_vissr_l1b.py 3 98.67%
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 3 97.18%
Totals Coverage Status
Change from base Build 9842330086: -0.01%
Covered Lines: 51681
Relevant Lines: 53802

💛 - Coveralls

@sfinkens sfinkens requested a review from djhoese July 17, 2024 09:56
Comment on lines -76 to -80
def dt64_to_datetime(self, dt64):
"""Conversion of numpy.datetime64 to datetime objects."""
if isinstance(dt64, np.datetime64):
return dt64.astype(dt.datetime)
return dt64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are you able to straight remove this with no other changes in this file? Was nothing else using this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was only used once in the tests

Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had one question for something that was confusing me, otherwise looks good from what I can tell.

Are there any warnings generated when using numpy 2 and timezone aware python datetime objects?

now = dt.datetime.now(dt.timezone.utc)
np.datetime64(now)

^ produces a warning.

@sfinkens
Copy link
Member Author

Are there any warnings generated when using numpy 2 and timezone aware python datetime objects?

No I don't see any. In this case I'm going the other way round from datetime64 to datetime. Since datetime64 has no timezone support I wouldn't expect warnings. There are other timezone related warnings, though. For example:

/home/runner/work/satpy/satpy/satpy/cf/attrs.py:212: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    _history_create = "Created by pytroll/satpy on {}".format(datetime.datetime.utcnow())

@djhoese
Copy link
Member

djhoese commented Jul 18, 2024

Yeah in another PR we should fix those. I thought Panu had started but maybe that was just the "dt" import change. Anyway...should you maybe add a UTC timezone to these datetimes? Or maybe we'll remember to do that in the future utcnow PR? Probably not. 🤷‍♂️

@pnuu
Copy link
Member

pnuu commented Jul 18, 2024

In #2787 I just swapped from datetime import datetime to import datetime as dt. I wasn't yet sure which way we are going with timezone aware/unaware datetimes. One hindrance with aware is the aforementioned Numpy datetime64 that do not support timezones.

@pnuu
Copy link
Member

pnuu commented Jul 18, 2024

Oh, and as a reference the utcnow() issue: #2752

@sfinkens
Copy link
Member Author

Oh, and as a reference the utcnow() issue: #2752

Oh, I forgot about that, thanks!

@sfinkens
Copy link
Member Author

sfinkens commented Jul 18, 2024

should you maybe add a UTC timezone to these datetimes?

Maybe it makes more sense to change them all at once when we decided on the way forward?

@pnuu
Copy link
Member

pnuu commented Jul 18, 2024

I also think the datetime aware stuff should happen in "one go" so that nothing breaks in any of the comparisons etc.

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud mraspaud merged commit f60a579 into pytroll:main Jul 19, 2024
18 of 19 checks passed
@sfinkens sfinkens deleted the fix-dt64-conversion branch October 18, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants