Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix: make listing of S3 objects faster in storage #2329

Merged

Conversation

sanderegg
Copy link
Member

What do these changes do?

calling aiobotocore.list_objects_v2 method with a Prefix is several order of magnitudes faster when Prefix is ending with a /

e.g.:
looking for files in of project 0000802f-1e81-551f-8ab2-769c941aac66:

client.list_objects_v2(Prefix="0000802f-1e81-551f-8ab2-769c941aac66") # this is SLOWWWWWWWWWW
client.list_objects_v2(Prefix="0000802f-1e81-551f-8ab2-769c941aac66/") # this is FASTTTTTTTTTT

for example on master, the first call takes 12 seconds to complete... while the second call takes a few ms...

Related issue/s

How to test

Checklist

@sanderegg sanderegg added the a:storage issue related to storage service label May 12, 2021
@sanderegg sanderegg added this to the Chinchilla milestone May 12, 2021
@sanderegg sanderegg requested review from GitHK, pcrespov and mguidon May 12, 2021 07:15
@sanderegg sanderegg self-assigned this May 12, 2021
@sanderegg sanderegg changed the title list_objects_v2 is much faster with a / at the end of Prefix Bugfix: make listing of S3 objects faster in storage May 12, 2021
@codecov
Copy link

codecov bot commented May 12, 2021

Codecov Report

Merging #2329 (a9a9605) into master (e0b4343) will decrease coverage by 0.0%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #2329     +/-   ##
========================================
- Coverage    71.7%   71.7%   -0.1%     
========================================
  Files         506     506             
  Lines       19925   19926      +1     
  Branches     1950    1950             
========================================
- Hits        14303   14288     -15     
- Misses       5146    5160     +14     
- Partials      476     478      +2     
Flag Coverage Δ
integrationtests 61.9% <ø> (-0.1%) ⬇️
unittests 67.0% <100.0%> (-0.1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ervices/storage/src/simcore_service_storage/dsm.py 70.2% <100.0%> (-2.4%) ⬇️
...ce_webserver/resource_manager/garbage_collector.py 74.8% <0.0%> (-2.8%) ⬇️
.../director/src/simcore_service_director/producer.py 60.8% <0.0%> (+0.2%) ⬆️

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

subtle! 👍

Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤯

Copy link
Contributor

@KZzizzle KZzizzle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow

@sanderegg sanderegg merged commit 33cf7a0 into ITISFoundation:master May 12, 2021
@sanderegg sanderegg deleted the bugfix/storage_timeout_issue branch May 12, 2021 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:storage issue related to storage service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants