Snakemake upgrade #471

Alputer · 2024-10-08T11:01:33Z

No description provided.

tiborsimko · 2024-10-23T08:22:23Z

reana_commons/snakemake.py

+    ) as snakemake_api:
+        try:
+            workflow_api = snakemake_api.workflow(
+                resource_settings=ResourceSettings(nodes=300),


For the nodes settings, we already have SNAKEMAKE_MAX_PARALLEL_JOBS environmental variable that could be used instead of the hard-coded value of 300, even though this might "fully' work only when the validation would be moved to the server-side.

Still, you could preventively introduce in reana_commons/config.py something like:

SNAKEMAKE_MAX_PARALLEL_JOBS = int(os.getenv("SNAKEMAKE_MAX_PARALLEL_JOBS", "300")) """Snakemake maximum number of jobs that can run in parallel."""

which might be better that hard-coding the value of 300.

(Similarly as we do in reana_workflow_engine_snakemake/config.py.)

tiborsimko · 2024-10-23T08:23:27Z

setup.py

-        # see https://github.com/snakemake/snakemake/issues/2657
-        "snakemake @ git+https://github.com/mdonadoni/snakemake.git@cea31624976989ad0645eb2e1751260d32259506",  # branch `7.32.4-python3.12`
-        "pulp>=2.7.0,<2.8.0",
+        "snakemake==8.23.0",


There is already Snakemake 8.24.0 out.

tiborsimko · 2024-10-23T08:25:40Z

reana_commons/snakemake.py

+                workdir=workdir,
+            )
+            # Try to create dag and it throws an error in case of a failure.
+            # Seems to be enough but might not be an extensive validation.


Perhaps amend the docstring to:

Seems to be enough for the first validation. We may move to snakemake --dry-run when the validation process will be fully moved to the server-side.

…ahub#471)

codecov · 2024-10-23T12:12:42Z

Codecov Report

Attention: Patch coverage is 63.41463% with 15 lines in your changes missing coverage. Please review.

Project coverage is 41.79%. Comparing base (564c027) to head (3b80320).
Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
reana_commons/snakemake.py	68.42%	12 Missing ⚠️
reana_commons/config.py	0.00%	2 Missing ⚠️
reana_commons/version.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #471      +/-   ##
==========================================
- Coverage   44.23%   41.79%   -2.45%     
==========================================
  Files          28       28              
  Lines        1978     2010      +32     
==========================================
- Hits          875      840      -35     
- Misses       1103     1170      +67

Files with missing lines	Coverage Δ
reana_commons/version.py	`0.00% <0.00%> (ø)`
reana_commons/config.py	`0.00% <0.00%> (ø)`
reana_commons/snakemake.py	`30.09% <68.42%> (-60.32%)`	⬇️

…ahub#471)

tiborsimko · 2024-10-28T09:18:06Z

tests/test_snakemake.py


 from reana_commons.snakemake import snakemake_load


+@pytest.mark.xfail(
+    sys.version_info >= (3, 11),
+    reason="Test expted to fail for python versions 3.11 and above as we currently return only empty dictionary in snakemake_load function for python >= 3.11. Development is blocked until we move validation to the server side.",


Typo: "expected" to fail

You can also remove the last part "Development is blocked until we move validation to the server side" to make the reason shorter. We'll revamp this anyway as part of the server-side validation sprint.

tiborsimko · 2024-10-28T09:20:55Z

.github/workflows/ci.yml

@@ -124,7 +124,7 @@ jobs:
      - name: Install system dependencies
        run: |
          sudo apt-get update -y
-          sudo apt-get install python3-dev graphviz libgraphviz-dev pkg-config
+          sudo apt-get install python3-dev graphviz libgraphviz-dev pkg-config uuid-dev


This is necessary to make the CI pass for the Snakemake v8 upgrade, so I would just squash the two functional commits together under the heading refactor(snakemake): upgrade snakemake 7 to snakemake 8 (#471) and add only the separate release commit on top.

tiborsimko · 2024-10-28T09:21:43Z

reana_commons/snakemake.py



 def snakemake_validate(
    workflow_file: str, configfiles: List[str], workdir: Optional[str] = None
 ):
-    """Validate Snakemake workflow specification.
+    """Validate snakemake workflow."""


Cosmetics: Validate Snakemake workflow. (note upppercase)

tiborsimko · 2024-10-28T09:23:37Z

reana_commons/snakemake.py

+def snakemake_validate_v7(
+    workflow_file: str, configfiles: List[str], workdir: Optional[str] = None
+):
+    """Old snakemake validate version for python versions < 3.11, it is needed since snakemake8 dropped support for python < 3.11.


Snakemake 7 workflow validation function, necessary for Python versions < 3.11.

(to make the docstring shorter)

tiborsimko · 2024-10-28T09:24:34Z

reana_commons/snakemake.py

+def snakemake_validate_v8(
+    workflow_file: str, configfiles: List[str], workdir: Optional[str] = None
+):
+    """Seems to be enough for the first validation. We may move to snakemake --dry-run when the validation process will be fully moved to the server-side.


Snakemake 8 workflow validation function for Python versions >= 3.11.

Note that we may move to using snakemake --dry-run when the validation process will be fully moved to the server side.

(to make the headline shorter, whilst the comment goes to the docstring body.)

tiborsimko · 2024-10-28T09:25:56Z

reana_commons/snakemake.py

+
+
+def snakemake_load(workflow_file: str, **kwargs: Any):
+    """Load snakemake specification."""


Cosmetics: Load Snakemake specification. (note uppercase)

tiborsimko

Works nicely for the full Snakemake test matrix 👍

Alputer added the worfkflow/snakemake label Oct 8, 2024

Alputer requested review from tiborsimko and mdonadoni October 8, 2024 11:01

Alputer self-assigned this Oct 8, 2024

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 15, 2024

refactor(snakemake): convert old snakemake api to new api (reanahub#471)

20a01a4

Alputer force-pushed the snakemake-upgrade branch from a4ab1b0 to 20a01a4 Compare October 15, 2024 09:43

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 15, 2024

refactor(snakemake): convert old snakemake api to new api (reanahub#471)

46bd3b6

Alputer force-pushed the snakemake-upgrade branch from 20a01a4 to 46bd3b6 Compare October 15, 2024 10:32

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 15, 2024

refactor(snakemake): convert old snakemake api to new api (reanahub#471)

c243ae1

Alputer force-pushed the snakemake-upgrade branch from 46bd3b6 to c243ae1 Compare October 15, 2024 10:49

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): convert old snakemake api to new api (reanahub#471)

ade46fe

Alputer force-pushed the snakemake-upgrade branch from c243ae1 to 88cc0dd Compare October 23, 2024 08:10

tiborsimko reviewed Oct 23, 2024

View reviewed changes

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

dfa4283

Alputer force-pushed the snakemake-upgrade branch from 88cc0dd to dfa4283 Compare October 23, 2024 09:40

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

804e084

Alputer force-pushed the snakemake-upgrade branch from dfa4283 to 804e084 Compare October 23, 2024 09:44

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

e23bba8

Alputer force-pushed the snakemake-upgrade branch from 804e084 to e23bba8 Compare October 23, 2024 09:58

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

ci(actions): update github actions dependencies to build xrootd (rean…

6ab9917

…ahub#471)

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

ce3a418

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

ci(actions): update github actions dependencies to build xrootd (rean…

5d54f47

…ahub#471)

Alputer force-pushed the snakemake-upgrade branch from 6ab9917 to 5d54f47 Compare October 23, 2024 12:02

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

84818db

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

ci(actions): update github actions dependencies to build xrootd (rean…

b10656e

…ahub#471)

Alputer force-pushed the snakemake-upgrade branch from 5d54f47 to 939f501 Compare October 23, 2024 12:08

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

4d98bf3

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 23, 2024

ci(actions): update github actions dependencies to build xrootd (rean…

ab341d2

…ahub#471)

Alputer force-pushed the snakemake-upgrade branch from 939f501 to 150afd2 Compare October 23, 2024 16:32

tiborsimko reviewed Oct 28, 2024

View reviewed changes

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 28, 2024

refactor(snakemake): upgrade snakemake 7 to snakemake 8 (reanahub#471)

58b5643

Alputer force-pushed the snakemake-upgrade branch from 150afd2 to 794c79b Compare October 28, 2024 09:39

Alputer added a commit to Alputer/reana-commons that referenced this pull request Oct 28, 2024

refactor(snakemake): add support for Snakemake 8 (reanahub#471)

cc7f721

Alputer force-pushed the snakemake-upgrade branch from 794c79b to 7b19310 Compare October 28, 2024 10:13

Alputer added 2 commits October 28, 2024 11:22

feat(snakemake): add support for Snakemake 8 (reanahub#471)

cac0dcc

chore(master): release 0.95.0a5

3b80320

Alputer force-pushed the snakemake-upgrade branch from 7b19310 to 3b80320 Compare October 28, 2024 10:23

tiborsimko approved these changes Oct 28, 2024

View reviewed changes

tiborsimko merged commit 3b80320 into reanahub:master Oct 28, 2024
14 checks passed

github-actions bot mentioned this pull request Oct 21, 2024

chore(master): release 0.9.10 #450

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snakemake upgrade #471

Snakemake upgrade #471

Alputer commented Oct 8, 2024

tiborsimko Oct 23, 2024

tiborsimko Oct 23, 2024

tiborsimko Oct 23, 2024

codecov bot commented Oct 23, 2024 •

edited

Loading

tiborsimko Oct 28, 2024

tiborsimko Oct 28, 2024

tiborsimko Oct 28, 2024

tiborsimko Oct 28, 2024

tiborsimko Oct 28, 2024

tiborsimko Oct 28, 2024

tiborsimko Oct 28, 2024

tiborsimko left a comment



		def snakemake_load(workflow_file: str, **kwargs: Any):
		"""Load snakemake specification."""

Snakemake upgrade #471

Snakemake upgrade #471

Conversation

Alputer commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 23, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiborsimko left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 23, 2024 •

edited

Loading