Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 archiving_utils creates deterministic zip archives #6472

Merged
merged 11 commits into from
Oct 3, 2024

Conversation

GitHK
Copy link
Contributor

@GitHK GitHK commented Sep 30, 2024

What do these changes do?

When creating a zip archive its hash would not be the same if ran twice on the same input files. Reasons:

  • files were not added in the same order (even though python sorting is deterministic in the same interpreter process, if the interpreter is restart this is no longer guaranteed)
  • datetimes are also used in zip archives which complicates thing even more

These changes make it so that the produced archive will be deterministic. This allows different python interpreters to produce the same archive out of the same files (if their content does not change).

Related issue/s

How to test

Dev-ops checklist

@GitHK GitHK self-assigned this Sep 30, 2024
@GitHK GitHK added bug buggy, it does not work as expected a:sidecar issue related with the sidecar worker service a:services-library issues on packages/service-libs labels Sep 30, 2024
@GitHK GitHK added this to the MartinKippenberger milestone Sep 30, 2024
Copy link

codecov bot commented Sep 30, 2024

Codecov Report

Attention: Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.

Project coverage is 87.9%. Comparing base (cafbf96) to head (a3834d9).
Report is 614 commits behind head on master.

Files with missing lines Patch % Lines
.../service-library/src/servicelib/archiving_utils.py 84.2% 3 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6472      +/-   ##
=========================================
+ Coverage    84.5%   87.9%    +3.3%     
=========================================
  Files          10    1215    +1205     
  Lines         214   53197   +52983     
  Branches       25    1096    +1071     
=========================================
+ Hits          181   46769   +46588     
- Misses         23    6237    +6214     
- Partials       10     191     +181     
Flag Coverage Δ
integrationtests 64.6% <ø> (?)
unittests 85.5% <84.2%> (+0.9%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../service-library/src/servicelib/archiving_utils.py 83.2% <84.2%> (ø)

... and 1209 files with indirect coverage changes

@GitHK GitHK changed the title 🐛 deterministic zip archives 🐛 archiving_utils creates deterministic zip archives Oct 1, 2024
@GitHK GitHK marked this pull request as ready for review October 1, 2024 09:00
Copy link
Contributor

@matusdrobuliak66 matusdrobuliak66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. thanks!

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx!
When @giancarloromeo reviews it, since @sanderegg is not here, i can force-merge it when you tell me

Copy link

sonarqubecloud bot commented Oct 3, 2024

@pcrespov pcrespov merged commit 840fdf5 into ITISFoundation:master Oct 3, 2024
57 checks passed
@GitHK GitHK deleted the pr-osparc-reporducible-zipfile branch October 3, 2024 14:19
mrnicegyu11 pushed a commit to mrnicegyu11/osparc-simcore that referenced this pull request Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:services-library issues on packages/service-libs a:sidecar issue related with the sidecar worker service bug buggy, it does not work as expected
Projects
None yet
Development

Successfully merging this pull request may close these issues.

make zipping in simcore-sdk deterministic
3 participants