Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛Storage: fix database locking #6005

Merged

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Jun 27, 2024

What do these changes do?

This PR aims at removing all unbounded database connection locks in storage

Details

  • in s3 client copy_objects_recursively removed deletion in case of error as this could delete fils in an uncontrolled way
  • typing improvements in aws-library
  • better logging in storage

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added the a:storage issue related to storage service label Jun 27, 2024
@sanderegg sanderegg added this to the South Island Iced Tea milestone Jun 27, 2024
@sanderegg sanderegg self-assigned this Jun 27, 2024
Copy link

codecov bot commented Jun 27, 2024

Codecov Report

Attention: Patch coverage is 89.80892% with 16 lines in your changes missing coverage. Please review.

Project coverage is 87.7%. Comparing base (cafbf96) to head (1a31807).
Report is 310 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6005      +/-   ##
=========================================
+ Coverage    84.5%   87.7%    +3.2%     
=========================================
  Files          10    1330    +1320     
  Lines         214   55249   +55035     
  Branches       25    1368    +1343     
=========================================
+ Hits          181   48501   +48320     
- Misses         23    6465    +6442     
- Partials       10     283     +273     
Flag Coverage Δ
integrationtests 64.8% <ø> (?)
unittests 85.7% <89.8%> (+1.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
packages/aws-library/src/aws_library/s3/_client.py 96.7% <100.0%> (ø)
...src/simcore_service_storage/handlers_simcore_s3.py 100.0% <100.0%> (ø)
...s/aws-library/src/aws_library/s3/_error_handler.py 76.1% <60.0%> (ø)
packages/service-library/src/servicelib/utils.py 91.6% <80.0%> (ø)
...rage/src/simcore_service_storage/simcore_s3_dsm.py 89.8% <91.1%> (ø)

... and 1312 files with indirect coverage changes

@sanderegg sanderegg force-pushed the storage/fix-database-locking branch 3 times, most recently from 622d65c to 4b84a0d Compare June 28, 2024 13:37
@sanderegg sanderegg marked this pull request as ready for review July 1, 2024 10:17
@sanderegg sanderegg requested a review from pcrespov as a code owner July 1, 2024 10:17
@sanderegg sanderegg force-pushed the storage/fix-database-locking branch from 49c416c to b822993 Compare July 1, 2024 10:17
@sanderegg sanderegg requested review from GitHK and bisgaard-itis July 1, 2024 10:17
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx a lot! I am looking forward to check this fix!

Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

services/storage/tests/conftest.py Outdated Show resolved Hide resolved
@sanderegg sanderegg force-pushed the storage/fix-database-locking branch from b822993 to d0bd3a3 Compare July 1, 2024 11:24
@sanderegg sanderegg force-pushed the storage/fix-database-locking branch from 017ef9f to 1a31807 Compare July 1, 2024 12:07
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool. Just a few comments/questions from my side

@sanderegg sanderegg merged commit 430f229 into ITISFoundation:master Jul 1, 2024
67 checks passed
@sanderegg sanderegg deleted the storage/fix-database-locking branch July 1, 2024 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:storage issue related to storage service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

storage remove database locking while uploading/copying
4 participants