Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add storage client copy move methods & fix threading when downloading blobs #153

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

JanusAsmussen
Copy link
Contributor

No description provided.

@github-actions
Copy link

Coverage

Coverage Report
FileStmtsMissCoverMissing
proteus/connectors/service_bus
   __init__.py110%3
   _connector.py17170%4–40
proteus/logs
   _base.py963168%73–76, 92, 110, 257–307
proteus/logs/handlers
   datadog_api_handler.py711480%54, 74–76, 86–97, 108, 131–132, 135
proteus/metrics
   _base.py16160%4–71
proteus/metrics/providers
   datadog_provider.py37370%4–91
proteus/security/clients
   _azure_client.py472645%27, 40–43, 53–56, 59, 68–95, 98, 102–108
proteus/storage/blob
   azure_storage_client.py1095252%46–48, 61–62, 65–78, 81, 90–98, 104, 121–122, 131, 145–150, 156–158, 173–183, 192–204, 212–227
   base.py34585%84–89, 132–134
proteus/storage/database
   azure_sql.py32320%5–109
   odbc.py701381%76–86, 96, 107–108, 126, 132–134, 160, 179–180
proteus/storage/models
   azure.py411173%32, 35–36, 39–40, 55–58, 65, 68–69, 72
   hive.py44440%4–112
   local.py14193%26
proteus/storage/secrets
   __init__.py110%5
   _base.py10100%5–29
   azure_secret_client.py16160%4–35
proteus/utils
   _common.py22768%35–46, 57
tests
   test_logs.py48296%83–84
TOTAL108233669% 

Tests Skipped Failures Errors Time
35 0 💤 0 ❌ 0 🔥 5.114s ⏱️

Copy link
Contributor

@george-zubrienko george-zubrienko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be done in an easier manner. Check comments

destination_blob = self._get_blob_client(blob_path=destination_blob_path)

destination_blob.start_copy_from_url(source_blob.url)
return destination_blob
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you should return here what start_copy_from_url returns

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do that, but it only returns a dictionary of current copy properties.

If we'd like to check progress, we need the blob_client object

destination_blob.start_copy_from_url(source_blob.url)
return destination_blob

def copy_blob(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd suggest using example from MS here

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/samples/blob_samples_common.py#L178

And remove async parameter, as this operation is always async.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the MS example, it doesn't seem they wait for the operation to complete?

Can we be certain the operation is complete when copied_blob.get_blob_properties() is called?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants