Refactor steps in blending code #453

sidekock · 2025-01-20T20:27:12Z

…4 for the refactoring

Co-authored-by: mats-knmi <[email protected]>

….velocity_perturbations = [] in __initialize_random_generators

…to come later

…xed seed assingments

* Use seed for all rng to make a test run completely deterministic * fix probmatching test and some copy paste oversights * Add test for vel_pert_method * Change the test so that it actually runs the lines that need to be covered

…workers at the same time

…as the master

codecov · 2025-01-20T20:37:42Z

Codecov Report

Attention: Patch coverage is 72.22222% with 10 lines in your changes missing coverage. Please review.

Project coverage is 84.31%. Comparing base (a7dae54) to head (4fb784e).

Files with missing lines	Patch %	Lines
pysteps/nowcasts/steps.py	72.22%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #453      +/-   ##
==========================================
+ Coverage   84.26%   84.31%   +0.05%     
==========================================
  Files         160      160              
  Lines       13067    13250     +183     
==========================================
+ Hits        11011    11172     +161     
- Misses       2056     2078      +22

Flag	Coverage Δ
unit_tests	`84.31% <72.22%> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

sidekock · 2025-01-20T20:41:08Z

Both visual tests and assert statements give the same conclusion: the output is exactly the same!

sidekock · 2025-01-20T20:51:19Z

@dnerini The codecov check is failing but the only things it is actuary failing on in the deepcopys I have done in the code and on the timing which does not seem worth it to write tests for as this is a very basic subtraction. Would it be able to disable it for these things or how does this work? I know you where able to do it for the re-factoring of the steps nowcasting.

RubenImhoff

Hi @sidekock and @mats-knmi,

Thanks a lot for your work here! I'm sorry I was not around the past days to answer some questions and think along. However, this looks great! With the large number of changes, it is becoming difficult to check everything thoroughly, so at least I'm happy to see that it gives exactly the same results. I think a to do for a new PR could be updating the documentation, but that is probably easier and cleaner once this PR is pushed.

RubenImhoff · 2025-01-21T08:20:33Z

pysteps/blending/steps.py

-      To further reduce memory usage, both this array and the ``velocity_models`` array
-      can be given as float32. They will then be converted to float64 before computations
-      to minimize loss in precision.
+# TODO: compare old and new version of the code, run a benchmark to compare the two


Is this TODO still necessary? I can imagine we'd like to have this done prior to merging this PR.

dnerini · 2025-01-21T12:21:05Z

@dnerini The codecov check is failing but the only things it is actuary failing on in the deepcopys I have done in the code and on the timing which does not seem worth it to write tests for as this is a very basic subtraction. Would it be able to disable it for these things or how does this work? I know you where able to do it for the re-factoring of the steps nowcasting.

@sidekock we can simply ignore the failing check and merge whenever you feel it's ready

sidekock and others added 30 commits November 18, 2024 11:28

Refactored all names in the steps blending code from old to new

2066f14

Made some name changes but test still do not pass

72d0fbc

Fixed naming changes, now the tests pass

1ce563e

Built the rough scaffolding for the blending class

fbe551b

Refactored untill no rain case

46a93e5

Added code to estimation of ar parameters of radar

1eede39

Next go, start with forecast loop #7

a18f1f6

Added some uniformity between nowcast and blending steps. Now at # 8.…

8d16c11

…4 for the refactoring

Small changes since prev commit

88df97d

All code is tranfered. Last part of the main loop needs to be refactored

7ee0020

Everything is refactored, no test ran as of yet

f387981

Old forecast function is updated to fit newly refactored code

760c185

Removed old code which is no longer used

8d8905a

6 more tests that fail

d6249f5

All tests pass, still need to fix TODOs

38702b3

Updated gitignore

5ff1713

Cleanup of params and state dataclasses, next step: better typing

d999501

Cleanup of params and state dataclasses, now all tests pass

ed20ecc

Added correct typing to all parts of params and state

701e726

Ready for pull request

b9de511

Made changes for Codacy review

38ed195

Added aditional tests which currently fail in master branch

32b656f

Update .gitignore

4fe9f78

Co-authored-by: mats-knmi <[email protected]>

Used the __zero_precip_time in __zero_precipitation_forecast()

b31d55c

Changed typing hints to python 3.10+ version

cc02593

Added comments back to the State dataclass

4e4a148

Changed the self.__state.velocity_perturbations = [] to self.__params…

0f4e037

….velocity_perturbations = [] in __initialize_random_generators

Added code changes as suggested by Ruben, comments and documentation …

9f413aa

…to come later

Added frozen functionality to dataclasses, removed reset_state and fi…

c72d953

…xed seed assingments

Added frozen dataclass to nowcast

00f057b

sidekock and others added 6 commits December 19, 2024 17:21

The needed checks are done for this TODO so it can be removed

1b82512

Use the seed in all rng in blending code (#449)

47ab6c3

* Use seed for all rng to make a test run completely deterministic * fix probmatching test and some copy paste oversights * Add test for vel_pert_method * Change the test so that it actually runs the lines that need to be covered

Removed deepcopy of worker_state. The state is now accessable to all …

48187c4

…workers at the same time

Update to probmatching comments to keep in track with main

9b216a7

Fix for multithreading issue, this produces exactly the same results …

561e7ac

…as the master

New commit for new pr

4fb784e

sidekock added the documentation label Jan 20, 2025

sidekock requested review from ladc, RubenImhoff and mats-knmi January 20, 2025 20:27

sidekock self-assigned this Jan 20, 2025

mats-knmi approved these changes Jan 20, 2025

View reviewed changes

RubenImhoff approved these changes Jan 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor steps in blending code #453

Refactor steps in blending code #453

sidekock commented Jan 20, 2025

codecov bot commented Jan 20, 2025 •

edited

Loading

sidekock commented Jan 20, 2025

sidekock commented Jan 20, 2025

RubenImhoff left a comment

RubenImhoff Jan 21, 2025

dnerini commented Jan 21, 2025

Refactor steps in blending code #453

Are you sure you want to change the base?

Refactor steps in blending code #453

Conversation

sidekock commented Jan 20, 2025

codecov bot commented Jan 20, 2025 • edited Loading

Codecov Report

sidekock commented Jan 20, 2025

sidekock commented Jan 20, 2025

RubenImhoff left a comment

Choose a reason for hiding this comment

RubenImhoff Jan 21, 2025

Choose a reason for hiding this comment

dnerini commented Jan 21, 2025

codecov bot commented Jan 20, 2025 •

edited

Loading