Feature: Generators #866

joernweissenborn · 2021-10-15T23:33:05Z

This PR salvages the generators from the project PR.

This is kind of in conflict with the work of @jsnel but we should use this PR to marry everything.

My approach of generators is more "dump", as the generating functions should return model dicts instead of models, I'am not sure why, but I think this will be useful.

Also this generators no't have any state, they are boilerplate replacement and I think all what is needed.

The usage will become more clear in a follow up PR where I add projects. This will also enable all the features the current Generator achieves with state fullness.

More details to follow

Change summary

Created glotaran.project.generators package

Checklist

✔️ Passing the tests (mandatory for all PR's)
👌 Closes issue (mandatory for ✨ feature and 🩹 bug fix PR's)
🧪 Adds new tests for the feature (mandatory for ✨ feature and 🩹 bug fix PR's)
📚 Adds documentation of the feature

Closes issues

closes #XXXX

github-actions · 2021-10-15T23:33:18Z

👈 Launch a binder notebook on branch joernweissenborn/pyglotaran/feature/project/generators

github-actions · 2021-10-15T23:38:58Z

Benchmark is done. Checkout the benchmark result page.
Benchmark differences below 5% might be due to CI noise.

Benchmark diff v0.5.1 vs. main

Parametrized benchmark signatures:

BenchmarkOptimize.time_optimize(index_dependent, grouped, weight)


All benchmarks:

       before           after         ratio
     [96b42630]       [19439efc]
     <v0.5.1>                   
       61.8±0.9ms       62.0±0.3ms     1.00  BenchmarkOptimize.time_optimize(False, False, False)
        95.9±30ms        68.3±40ms    ~0.71  BenchmarkOptimize.time_optimize(False, False, True)
       61.3±0.5ms       62.3±0.8ms     1.02  BenchmarkOptimize.time_optimize(False, True, False)
       65.9±0.9ms        88.3±30ms    ~1.34  BenchmarkOptimize.time_optimize(False, True, True)
         73.4±1ms       75.3±0.9ms     1.03  BenchmarkOptimize.time_optimize(True, False, False)
        79.6±30ms       79.4±0.5ms     1.00  BenchmarkOptimize.time_optimize(True, False, True)
       74.7±0.3ms       75.0±0.8ms     1.00  BenchmarkOptimize.time_optimize(True, True, False)
        80.0±20ms        86.4±30ms     1.08  BenchmarkOptimize.time_optimize(True, True, True)
             203M             199M     0.98  IntegrationTwoDatasets.peakmem_optimize
       2.02±0.05s       1.97±0.03s     0.98  IntegrationTwoDatasets.time_optimize

Benchmark diff main vs. PR

Parametrized benchmark signatures:

BenchmarkOptimize.time_optimize(index_dependent, grouped, weight)


All benchmarks:

       before           after         ratio
     [bde9b19c]       [19439efc]
       62.5±0.8ms       62.0±0.3ms     0.99  BenchmarkOptimize.time_optimize(False, False, False)
        67.5±20ms        68.3±40ms     1.01  BenchmarkOptimize.time_optimize(False, False, True)
       62.4±0.3ms       62.3±0.8ms     1.00  BenchmarkOptimize.time_optimize(False, True, False)
       66.2±0.6ms        88.3±30ms    ~1.33  BenchmarkOptimize.time_optimize(False, True, True)
       74.6±0.7ms       75.3±0.9ms     1.01  BenchmarkOptimize.time_optimize(True, False, False)
        82.7±30ms       79.4±0.5ms     0.96  BenchmarkOptimize.time_optimize(True, False, True)
       72.9±0.9ms       75.0±0.8ms     1.03  BenchmarkOptimize.time_optimize(True, True, False)
        79.8±10ms        86.4±30ms     1.08  BenchmarkOptimize.time_optimize(True, True, True)
             204M             199M     0.97  IntegrationTwoDatasets.peakmem_optimize
       1.95±0.05s       1.97±0.03s     1.01  IntegrationTwoDatasets.time_optimize

codecov · 2021-10-15T23:40:47Z

Codecov Report

Merging #866 (19439ef) into main (bde9b19) will decrease coverage by 0.0%.
The diff coverage is 94.8%.

@@           Coverage Diff           @@
##            main    #866     +/-   ##
=======================================
- Coverage   85.4%   85.3%   -0.1%     
=======================================
  Files         87      90      +3     
  Lines       4872    4857     -15     
  Branches     920     912      -8     
=======================================
- Hits        4163    4147     -16     
- Misses       558     561      +3     
+ Partials     151     149      -2

Impacted Files	Coverage Δ
...lotaran/deprecation/modules/examples/sequential.py	`81.8% <81.8%> (ø)`
glotaran/project/generators/generator.py	`90.4% <90.4%> (ø)`
glotaran/__init__.py	`100.0% <100.0%> (ø)`
glotaran/builtin/io/yml/yml.py	`88.6% <100.0%> (-2.0%)`	⬇️
glotaran/deprecation/deprecation_utils.py	`100.0% <100.0%> (ø)`
glotaran/project/generators/__init__.py	`100.0% <100.0%> (ø)`
.../testing/simulated_data/parallel_spectral_decay.py	`100.0% <100.0%> (ø)`
...esting/simulated_data/sequential_spectral_decay.py	`100.0% <100.0%> (ø)`
glotaran/testing/simulated_data/shared_decay.py	`100.0% <100.0%> (ø)`
glotaran/project/result.py	`81.3% <0.0%> (-4.0%)`	⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bde9b19...19439ef. Read the comment docs.

sonarqubecloud · 2021-10-16T13:07:14Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
4 Code Smells

No Coverage information
6.2% Duplication

sonarqubecloud · 2021-12-03T12:43:55Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
21.9% Duplication

s-weigand

Mostly nitpicky things

Please move

glotaran/testing/parallel_spectral_decay.py -> glotaran/testing/simulated_data/parallel_spectral_decay.py
glotaran/testing/sequential_spectral_decay.py -> glotaran/testing/simulated_data/sequential_spectral_decay.py

That way we can add a file glotaran/testing/simulated_data/common.py and share common values (e.g. SIMULATION_COORDINATES, PARAMETER_YML), which will reduce code duplication.

As for disabling the quick-start integration test (ca82e91), we should revert that commit and properly deprecate glotaran.examples that is what glotaran.deprecation.modules is for.

glotaran/project/generators/generator.py

glotaran/project/generators/test/test_genenerate_decay_model.py

glotaran/project/generators/generator.py

glotaran/deprecation/modules/test/test_project_result.py

glotaran/testing/parallel_spectral_decay.py

glotaran/testing/sequential_spectral_decay.py

glotaran/testing/parallel_spectral_decay.py

s-weigand · 2022-01-16T18:37:10Z

@joernweissenborn Don't forget to add the changes to the changelog. (@jsnel we still need to add the changes from #860)

sonarqubecloud · 2022-01-20T13:20:30Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
18.5% Duplication

for more information, see https://pre-commit.ci

Co-authored-by: Sebastian Weigand <[email protected]>

for more information, see https://pre-commit.ci

glotaran.testing.sequential_spectral_decay -> glotaran.testing.simulated_data.sequential_spectral_decay

…ackage

s-weigand · 2022-01-20T21:27:11Z

I made the generators keyword only and added a TypedDict for the arguments.
@joernweissenborn please check that you are ok with this change.

s-weigand

From my side it is fine now 🚀

This allow showinging the correct new usage while importing the functionality from somewhere else.

sourcery-ai · 2022-01-21T01:09:16Z

Sourcery Code Quality Report

✅ Merging this PR will increase code quality in the affected files by 0.10%.

Quality metrics	Before	After	Change
Complexity	1.12 ⭐	1.03 ⭐	-0.09 👍
Method Length	49.65 ⭐	49.40 ⭐	-0.25 👍
Working memory	6.22 🙂	6.20 🙂	-0.02 👍
Quality	79.41% ⭐	79.51% ⭐	0.10% 👍

Other metrics	Before	After	Change
Lines	3171	2668	-503

Changed files	Quality Before	Quality After	Quality Change
glotaran/init.py	91.44% ⭐	88.07% ⭐	-3.37% 👎
glotaran/builtin/io/folder/test/test_folder_plugin.py	85.18% ⭐	87.00% ⭐	1.82% 👍
glotaran/builtin/io/yml/yml.py	78.43% ⭐	77.86% ⭐	-0.57% 👎
glotaran/builtin/io/yml/test/test_save_model.py	94.31% ⭐	94.31% ⭐	0.00%
glotaran/builtin/io/yml/test/test_save_result.py	83.82% ⭐	87.74% ⭐	3.92% 👍
glotaran/builtin/io/yml/test/test_save_scheme.py	86.35% ⭐	86.35% ⭐	0.00%
glotaran/deprecation/deprecation_utils.py	72.44% 🙂	70.38% 🙂	-2.06% 👎
glotaran/deprecation/modules/test/test_changed_imports.py	91.59% ⭐	91.73% ⭐	0.14% 👍
glotaran/deprecation/modules/test/test_model_model.py	93.30% ⭐	95.01% ⭐	1.71% 👍
glotaran/deprecation/modules/test/test_parameter_parameter_group.py	90.24% ⭐	90.24% ⭐	0.00%
glotaran/deprecation/modules/test/test_project_result.py	94.31% ⭐	95.02% ⭐	0.71% 👍
glotaran/deprecation/modules/test/test_project_scheme.py	75.33% ⭐	74.59% 🙂	-0.74% 👎
glotaran/deprecation/test/test_deprecation_utils.py	87.71% ⭐	87.86% ⭐	0.15% 👍
glotaran/deprecation/test/dummy_package/init.py	94.05% ⭐	89.49% ⭐	-4.56% 👎
glotaran/deprecation/test/dummy_package/deprecated_module_attribute.py	87.99% ⭐	82.07% ⭐	-5.92% 👎
glotaran/model/test/test_model.py	72.61% 🙂	72.68% 🙂	0.07% 👍
glotaran/project/test/test_result.py	80.10% ⭐	93.62% ⭐	13.52% 👍

Here are some functions in these files that still need a tune-up:

File	Function	Complexity	Length	Working Memory	Quality	Recommendation
glotaran/deprecation/deprecation_utils.py	check_qualnames_in_tests.deprecate_dict_entry	11 🙂	135 😞	14 😞	47.35% 😞	Try splitting into smaller methods. Extract out complex expressions
glotaran/builtin/io/yml/yml.py	YmlProjectIo.save_model	15 🙂	106 🙂	13 😞	48.93% 😞	Extract out complex expressions
glotaran/model/test/test_model.py	test_fill	0 ⭐	243 ⛔	7 🙂	62.26% 🙂	Try splitting into smaller methods
glotaran/model/test/test_model.py	test_items	0 ⭐	214 ⛔	7 🙂	64.02% 🙂	Try splitting into smaller methods
glotaran/deprecation/deprecation_utils.py	check_qualnames_in_tests.module_attribute.deprecate_submodule	6 ⭐	123 😞	8 🙂	64.34% 🙂	Try splitting into smaller methods

Legend and Explanation

The emojis denote the absolute quality of the code:

⭐ excellent
🙂 good
😞 poor
⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.

Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!

sonarqubecloud · 2022-01-21T01:09:44Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

s-weigand

Now that glotaran.examples is properly deprecated and all integration tests run again it should be good to merge.

jsnel

Looks good to me, just some minor cosmetic changes requested (not all of them captured by the comments), but I will pick that up myself and then give it the green light.

glotaran/builtin/io/yml/test/test_save_scheme.py

glotaran/deprecation/deprecation_utils.py

glotaran/deprecation/modules/test/test_changed_imports.py

glotaran/testing/simulated_data/shared_decay.py

jsnel

Can be merged in now! 🎉

Note to @glotaran/maintainers: we should revisit #815 after merging in this PR.

jsnel · 2022-01-21T23:05:09Z

Update, last messages was written under the assumption I merged in some changes, which I didn't. Migrated those changes to a separate follow up PR:
#965

Improve spelling, fix typos. Specific changes: - Use parameters instead of parameter where it makes sense. - Avoid using wildcards when importing from module

joernweissenborn requested review from jsnel and s-weigand as code owners October 15, 2021 23:33

sourcery-ai bot mentioned this pull request Oct 15, 2021

Feature/project/generators (Sourcery refactored) #867

Closed

joernweissenborn force-pushed the feature/project/generators branch 2 times, most recently from 9d8debf to f6aca9a Compare October 16, 2021 11:57

joernweissenborn requested a review from a team as a code owner October 16, 2021 12:53

joernweissenborn mentioned this pull request Oct 19, 2021

✨ Add simple decay megacomplexes #860

Merged

4 tasks

This was referenced Oct 24, 2021

[WIP] Feature Projects #769

Closed

{dontmerge]Scraping result #823

Closed

❌ [TEST] Refactor/refactor project after 807 #825

Closed

❌ [TEST-PR] Wip/projects against d2b297 #829

Closed

joernweissenborn force-pushed the feature/project/generators branch from 2159970 to f0f7715 Compare December 2, 2021 13:56

joernweissenborn changed the title ~~Feature/project/generators~~ Feature: Generators Dec 2, 2021

joernweissenborn force-pushed the feature/project/generators branch from 37bf142 to 9123bba Compare December 3, 2021 12:02

jsnel force-pushed the feature/project/generators branch from db3ed0b to 733e4c3 Compare January 16, 2022 11:48

s-weigand requested changes Jan 16, 2022

View reviewed changes

s-weigand added the Type: Enhancement Feature requests label Jan 16, 2022

joernweissenborn and others added 6 commits January 20, 2022 14:58

Added model generators

e19e096

Added spectral decay model generator

61b09df

Refactored examples

3d2c723

Fix test

c546759

Changed result tests to use examples

fbace9f

[pre-commit.ci] auto fixes from pre-commit.com hooks

3eb4508

for more information, see https://pre-commit.ci

pre-commit-ci bot and others added 8 commits January 20, 2022 14:58

[pre-commit.ci] auto fixes from pre-commit.com hooks

e9897f0

for more information, see https://pre-commit.ci

[pre-commit.ci] auto fixes from pre-commit.com hooks

62c73df

for more information, see https://pre-commit.ci

Added naming consitency.

b702f63

Refactored test simulations.

b3ad3ac

Prevent using positional arguments in generators.

0c454de

Co-authored-by: Sebastian Weigand <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

272d4cc

for more information, see https://pre-commit.ci

Updated changelog.

1293547

Partially reverted suggestions from code review.

05c1b5b

joernweissenborn force-pushed the feature/project/generators branch from 7fb9ae5 to 05c1b5b Compare January 20, 2022 13:59

🩹📚 Fix changed import in docs

af866cd

glotaran.testing.sequential_spectral_decay -> glotaran.testing.simulated_data.sequential_spectral_decay

s-weigand force-pushed the feature/project/generators branch from 2bba71d to af866cd Compare January 20, 2022 19:24

s-weigand added 2 commits January 20, 2022 20:48

🩹 Added __init __.py so glotaran.testing.simulated_data is a proper p…

a904d12

…ackage

👌 Made arguments for generators keyword only

348d115

s-weigand previously approved these changes Jan 20, 2022

View reviewed changes

🧪 Reactivated quick-start integration test

5e2306e

s-weigand dismissed their stale review via 5e2306e January 20, 2022 21:37

✨ Added module_load_overwrite arg to deprecation functions for modules

72d6b21

This allow showinging the correct new usage while importing the functionality from somewhere else.

sourcery-ai bot mentioned this pull request Jan 21, 2022

Feature: Generators (Sourcery refactored) #962

Closed

s-weigand and others added 2 commits January 21, 2022 02:04

🚧🗑️ Properly deprecated 'glotaran.examples'

554a99e

♻️ Refactored by Sourcery

19439ef

s-weigand force-pushed the feature/project/generators branch from 8f9a656 to 19439ef Compare January 21, 2022 01:09

s-weigand approved these changes Jan 21, 2022

View reviewed changes

jsnel reviewed Jan 21, 2022

View reviewed changes

jsnel approved these changes Jan 21, 2022

View reviewed changes

jsnel merged commit 09de6b2 into glotaran:main Jan 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Generators #866

Feature: Generators #866

joernweissenborn commented Oct 15, 2021 •

edited by jsnel

Loading

github-actions bot commented Oct 15, 2021

github-actions bot commented Oct 15, 2021 •

edited

Loading

codecov bot commented Oct 15, 2021 •

edited

Loading

sonarqubecloud bot commented Oct 16, 2021

sonarqubecloud bot commented Dec 3, 2021

s-weigand left a comment

s-weigand commented Jan 16, 2022

sonarqubecloud bot commented Jan 20, 2022

s-weigand commented Jan 20, 2022 •

edited

Loading

s-weigand left a comment

sourcery-ai bot commented Jan 21, 2022

sonarqubecloud bot commented Jan 21, 2022

s-weigand left a comment

jsnel left a comment

jsnel left a comment

jsnel commented Jan 21, 2022

Feature: Generators #866

Feature: Generators #866

Conversation

joernweissenborn commented Oct 15, 2021 • edited by jsnel Loading

Change summary

Checklist

Closes issues

github-actions bot commented Oct 15, 2021

github-actions bot commented Oct 15, 2021 • edited Loading

codecov bot commented Oct 15, 2021 • edited Loading

Codecov Report

sonarqubecloud bot commented Oct 16, 2021

sonarqubecloud bot commented Dec 3, 2021

s-weigand left a comment

Choose a reason for hiding this comment

s-weigand commented Jan 16, 2022

sonarqubecloud bot commented Jan 20, 2022

s-weigand commented Jan 20, 2022 • edited Loading

s-weigand left a comment

Choose a reason for hiding this comment

sourcery-ai bot commented Jan 21, 2022

Sourcery Code Quality Report

Legend and Explanation

sonarqubecloud bot commented Jan 21, 2022

s-weigand left a comment

Choose a reason for hiding this comment

jsnel left a comment

Choose a reason for hiding this comment

jsnel left a comment

Choose a reason for hiding this comment

jsnel commented Jan 21, 2022

joernweissenborn commented Oct 15, 2021 •

edited by jsnel

Loading

github-actions bot commented Oct 15, 2021 •

edited

Loading

codecov bot commented Oct 15, 2021 •

edited

Loading

s-weigand commented Jan 20, 2022 •

edited

Loading