🐛 unarchiving_utils are now based on 7zip cli #6959

GitHK · 2024-12-12T15:20:26Z

What do these changes do?

When a new style dynamic service has to fetch an input port containing a zip archive, it will try to unzip it.
If it's not able to open it (see issue here), the service would fail to start.

Important changes:

♻️ dropped used options from archiving and unarchiving
🎨 created drop-in implementation using 7zip cli for archiving and unarchiving
🐛 7zip alo addresses Deflate64 zip-files not supported by dynamic-sidecar powered services #2843

Related issue/s

fixes Deflate64 zip-files not supported by dynamic-sidecar powered services #2843

How to test

Dev-ops checklist

No ENV changes or I properly updated ENV (read the instruction)

…64-zip

codecov · 2024-12-12T15:32:19Z

Codecov Report

Attention: Patch coverage is 79.81651% with 44 lines in your changes missing coverage. Please review.

Project coverage is 86.84%. Comparing base (68f0b24) to head (7041eee).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6959      +/-   ##
==========================================
- Coverage   87.02%   86.84%   -0.18%     
==========================================
  Files        1621     1286     -335     
  Lines       63993    54997    -8996     
  Branches     2035     1161     -874     
==========================================
- Hits        55687    47764    -7923     
+ Misses       7972     7021     -951     
+ Partials      334      212     -122

Flag	Coverage Δ
integrationtests	`64.58% <16.00%> (-10.42%)`	⬇️
unittests	`84.95% <79.81%> (-1.02%)`	⬇️

Components	Coverage Δ
api	`∅ <ø> (∅)`
pkg_aws_library	`93.49% <ø> (ø)`
pkg_dask_task_models_library	`∅ <ø> (∅)`
pkg_models_library	`∅ <ø> (∅)`
pkg_notifications_library	`∅ <ø> (∅)`
pkg_postgres_database	`∅ <ø> (∅)`
pkg_service_integration	`∅ <ø> (∅)`
pkg_service_library	`73.53% <88.08%> (+0.30%)`	⬆️
pkg_settings_library	`∅ <ø> (∅)`
pkg_simcore_sdk	`85.38% <ø> (ø)`
agent	`96.82% <ø> (ø)`
api_server	`90.13% <ø> (ø)`
autoscaling	`96.09% <ø> (ø)`
catalog	`90.57% <ø> (ø)`
clusters_keeper	`99.48% <ø> (ø)`
dask_sidecar	`91.26% <ø> (ø)`
datcore_adapter	`93.18% <ø> (ø)`
director	`76.40% <ø> (ø)`
director_v2	`90.68% <ø> (-0.74%)`	⬇️
dynamic_scheduler	`97.19% <ø> (ø)`
dynamic_sidecar	`89.76% <16.00%> (+0.01%)`	⬆️
efs_guardian	`90.12% <ø> (ø)`
invitations	`93.44% <ø> (ø)`
osparc_gateway_server	`∅ <ø> (∅)`
payments	`92.66% <ø> (ø)`
resource_usage_tracker	`89.62% <ø> (+0.05%)`	⬆️
storage	`89.54% <ø> (ø)`
webclient	`∅ <ø> (∅)`
webserver	`84.41% <ø> (-0.03%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68f0b24...7041eee. Read the comment docs.

odeimaiz

👍

sanderegg

What is the idea behing this change?

Also I was wondering:

the dask-sidecar checks the input file mime type. if it is set to zip, then it does not uncompress

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py

…64-zip

sanderegg

Please read my additional comments. I don't undersatnd how you wanna present this error to the user.

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py

packages/service-library/src/servicelib/archiving_utils.py

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py

…64-zip

sonarqubecloud · 2024-12-20T12:24:26Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

GitHK · 2024-12-20T12:29:36Z

@giancarloromeo @matusdrobuliak66 @sanderegg
I've rewritten the entire logic here. please review it as if it were a new.
Thanks a lot

sanderegg

thanks, but please let's look at some of the comments. we can have a quick chat.

sanderegg · 2024-12-20T13:16:56Z

packages/service-library/src/servicelib/archiving_utils/_errors.py

@@ -0,0 +1,4 @@
+class ArchiveError(Exception):


not using OsparcMixinError??

packages/service-library/src/servicelib/archiving_utils/_interface_7zip.py

sanderegg · 2024-12-20T13:20:30Z

packages/service-library/src/servicelib/archiving_utils/_interface_7zip.py

+class ArchiveInfoParser:
+    def __init__(self) -> None:
+        self.total_bytes: NonNegativeInt | None = None
+        self.file_count: NonNegativeInt | None = None
+
+    async def parse_chunk(self, chunk: str) -> None:
+        # search for ` NUMBER bytes ` -> set byte size
+        if self.total_bytes is None and (match := re.search(_TOTAL_BYTES_RE, chunk)):
+            self.total_bytes = int(match.group(1))
+
+        # search for ` NUMBER files` -> set file count
+        if self.file_count is None and (match := re.search(_FILE_COUNT_RE, chunk)):
+            self.file_count = int(match.group(1))
+
+    def get_parsed_values(self) -> tuple[NonNegativeInt, NonNegativeInt]:
+        if self.total_bytes is None:
+            msg = f"Unexpected value for {self.total_bytes=}. Should not be None"
+            raise ArchiveError(msg)
+
+        if self.file_count is None:
+            msg = f"Unexpected value for {self.file_count=}. Should not be None"
+            raise ArchiveError(msg)
+
+        return (self.total_bytes, self.file_count)
+


you use pydantic types in a non pydantic class.
Why not use at a dataclass? or pydantic at this point.

sanderegg · 2024-12-20T13:30:20Z

services/clusters-keeper/requirements/_base.txt

@@ -187,7 +187,7 @@ h11==0.14.0
    #   uvicorn
 httpcore==1.0.7
    # via httpx
-httpx==0.28.0
+httpx==0.28.1


why is httpx updated here?

sanderegg · 2024-12-20T13:30:31Z

services/autoscaling/requirements/_test.txt

@@ -93,7 +93,7 @@ httpcore==1.0.7
    # via
    #   -c requirements/_base.txt
    #   httpx
-httpx==0.28.0
+httpx==0.28.1


why is httpx updated here?

sanderegg · 2024-12-20T13:30:42Z

services/clusters-keeper/requirements/_test.txt

@@ -112,7 +112,7 @@ httpcore==1.0.7
    # via
    #   -c requirements/_base.txt
    #   httpx
-httpx==0.28.0
+httpx==0.28.1


sanderegg · 2024-12-20T13:31:32Z

services/dynamic-sidecar/requirements/_base.txt

@@ -328,11 +328,11 @@ opentelemetry-instrumentation-fastapi==0.49b2
    # via -r requirements/../../../packages/service-library/requirements/_fastapi.in
 opentelemetry-instrumentation-httpx==0.49b2
    # via -r requirements/../../../packages/service-library/requirements/_fastapi.in
-opentelemetry-instrumentation-logging==0.48b0
+opentelemetry-instrumentation-logging==0.49b2


there are a few updates I don't quite get why they come here. are you sure about these?

sanderegg · 2024-12-20T13:32:17Z

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py

    start_time = time.perf_counter()

    settings: ApplicationSettings = get_settings()
    PORTS: Nodeports = await node_ports_v2.ports(
        user_id=settings.DY_SIDECAR_USER_ID,
        project_id=ProjectIDStr(settings.DY_SIDECAR_PROJECT_ID),
-        node_uuid=NodeIDStr(settings.DY_SIDECAR_NODE_ID),
+        node_uuid=TypeAdapter(NodeIDStr).validate_python(settings.DY_SIDECAR_NODE_ID),


NodeIDStr is just a string. not sure you need to use the typeadapter here.

move archive to port instead of failing

97fd911

GitHK self-assigned this Dec 12, 2024

GitHK added the a:dynamic-sidecar dynamic-sidecar service label Dec 12, 2024

Merge remote-tracking branch 'upstream/master' into pr-osparc-deflate…

29533e9

…64-zip

GitHK added bug buggy, it does not work as expected t:maintenance Some planned maintenance work and removed bug buggy, it does not work as expected labels Dec 12, 2024

GitHK marked this pull request as ready for review December 13, 2024 05:47

GitHK requested review from mguidon, giancarloromeo, pcrespov, odeimaiz, matusdrobuliak66, sanderegg and mrnicegyu11 December 13, 2024 05:47

odeimaiz reviewed Dec 13, 2024

View reviewed changes

sanderegg reviewed Dec 13, 2024

View reviewed changes

Andrei Neagu added 3 commits December 13, 2024 10:45

using log context and rename

ca87d0b

Merge remote-tracking branch 'upstream/master' into pr-osparc-deflate…

7f3e41e

…64-zip

refactor to use less generic error UnsupportedArchiveFormat

be58b0c

GitHK requested a review from sanderegg December 13, 2024 10:15

sanderegg reviewed Dec 13, 2024

View reviewed changes

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py Outdated Show resolved Hide resolved

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py Outdated Show resolved Hide resolved

renaming error

53e0728

pcrespov reviewed Dec 13, 2024

View reviewed changes

packages/service-library/src/servicelib/archiving_utils.py Outdated Show resolved Hide resolved

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/nodeports.py Outdated Show resolved Hide resolved

Andrei Neagu added 5 commits December 16, 2024 13:47

replace unarchiver

65b5291

remove unused

e6d118d

remove comment

2a2f917

refactor 7zip

56d7fe1

fixed broken install

9c82bc5

Merge remote-tracking branch 'upstream/master' into pr-osparc-deflate…

e282c95

…64-zip

GitHK changed the title ~~🐛 Replace unarchiving function~~ 🐛 unarchiving_utils are now based on 7zip cli Dec 19, 2024

Andrei Neagu added 13 commits December 19, 2024 15:16

removes unused repro-zipfile

7279155

refactor imports

571a411

restructure model

da30d13

simplify outputs parsing

9ca1090

install path is now fixed to avoid using old system versions

97bdff1

fixed 7z installation

280b0d3

enhanced 7zip interface

6759dc7

added arm64 target install

4a5a3a6

fixed lines scanning parsing

21c7f0a

added edge case test

a58849f

Merge remote-tracking branch 'upstream/master' into pr-osparc-deflate…

f5ae500

…64-zip

refactor

abc45ef

refactor

82413f1

GitHK added the a:webserver issue related to the webserver service label Dec 20, 2024

Andrei Neagu added 7 commits December 20, 2024 11:54

rename

cbf7533

mypy

5739963

making flatter

9495ac1

revert change

d693eeb

adding missing

7e5b967

remove unused

05ab2f1

remove useless

7041eee

GitHK requested review from sanderegg, giancarloromeo and matusdrobuliak66 December 20, 2024 12:28

giancarloromeo approved these changes Dec 20, 2024

View reviewed changes

sanderegg reviewed Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 unarchiving_utils are now based on 7zip cli #6959

🐛 unarchiving_utils are now based on 7zip cli #6959

GitHK commented Dec 12, 2024 •

edited

Loading

codecov bot commented Dec 12, 2024 •

edited

Loading

odeimaiz left a comment

sanderegg left a comment

sanderegg left a comment

sonarqubecloud bot commented Dec 20, 2024

GitHK commented Dec 20, 2024

sanderegg left a comment

sanderegg Dec 20, 2024

sanderegg Dec 20, 2024

sanderegg Dec 20, 2024

sanderegg Dec 20, 2024

sanderegg Dec 20, 2024

sanderegg Dec 20, 2024

sanderegg Dec 20, 2024

🐛 unarchiving_utils are now based on 7zip cli #6959

Are you sure you want to change the base?

🐛 unarchiving_utils are now based on 7zip cli #6959

Conversation

GitHK commented Dec 12, 2024 • edited Loading

What do these changes do?

Related issue/s

How to test

Dev-ops checklist

codecov bot commented Dec 12, 2024 • edited Loading

Codecov Report

odeimaiz left a comment

Choose a reason for hiding this comment

sanderegg left a comment

Choose a reason for hiding this comment

sanderegg left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 20, 2024

Quality Gate passed

GitHK commented Dec 20, 2024

sanderegg left a comment

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

sanderegg Dec 20, 2024

Choose a reason for hiding this comment

GitHK commented Dec 12, 2024 •

edited

Loading

codecov bot commented Dec 12, 2024 •

edited

Loading