Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactored table azure_storage_blob to stream results per container and removed Go routine from the List function to resolve the inefficient memory usage issue Closes #854 #855

Merged
merged 1 commit into from
Nov 28, 2024
Merged

Refactored table azure_storage_blob to stream results per container and removed Go routine from the List function to resolve the inefficient memory usage issue Closes #854 #855

merged 1 commit into from
Nov 28, 2024

Conversation

ParthaI
Copy link
Contributor

@ParthaI ParthaI commented Nov 27, 2024

Integration test logs

Logs 
Add passing integration test logs here

Example query results

Results 
> select * from azure_storage_blob where resource_group = 'new-rg' and storage_account_name = 'tfstatetest93' limit 5
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------+------------+-------------+-------------+------------>
| name                                                                                                                                                               | storage_account_name | container_name           | type       | is_snapshot | access_tier | creation_ti>
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------+------------+-------------+-------------+------------>
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=20/h=03/m=00/PT1H.json | tfstatetest93        | insights-logs-auditevent | AppendBlob | false       |             | 2024-11-20T>
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=20/h=04/m=00/PT1H.json | tfstatetest93        | insights-logs-auditevent | AppendBlob | false       |             | 2024-11-20T>
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=19/h=22/m=00/PT1H.json | tfstatetest93        | insights-logs-auditevent | AppendBlob | false       |             | 2024-11-20T>
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=20/h=00/m=00/PT1H.json | tfstatetest93        | insights-logs-auditevent | AppendBlob | false       |             | 2024-11-20T>
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=19/h=20/m=00/PT1H.json | tfstatetest93        | insights-logs-auditevent | AppendBlob | false       |             | 2024-11-20T>
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------+------------+-------------+-------------+------------>

Time: 5.8s. Rows returned: 0. Rows fetched: 5. Hydrate calls: 10.

Scans:
  1) azure_storage_blob.azure: Time: 5.8s. Fetched: 5. Hydrates: 10. Quals: resource_group=new-rg, storage_account_name=tfstatetest93. Limit: 5.

> select * from azure_storage_blob where resource_group = 'new-rg' and storage_account_name = 'tfstatetest93'
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------->
| name                                                                                                                                                                                                    | storage_account_name | container_name                            >
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------->
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=21/h=18/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=20/h=20/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=20/h=21/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=21/h=00/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=21/h=12/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=19/h=20/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=22/h=21/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=23/h=19/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=20/h=14/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=21/h=03/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent                  >
| resourceId=/SUBSCRIPTIONS/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/RESOURCEGROUPS/DEMO/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/TETS56/y=2024/m=11/d=23/h=15/m=00/PT1H.json                                      | tfstatetest93        | insights-logs-auditevent

@ParthaI
Refactored table_azure_storage_blob to stream results per container a…
705c58c 
…nd removed Go routine from the List function Closes #854
@ParthaI ParthaI changed the title Refactored table_azure_storage_blob to stream results per container and removed Go routine from the List function to resolve the inefficient memory usage issue Closes #854 Refactored table azure_storage_blob to stream results per container and removed Go routine from the List function to resolve the inefficient memory usage issue Closes #854 Nov 27, 2024
@ParthaI ParthaI requested a review from misraved November 27, 2024 07:38
@ParthaI ParthaI self-assigned this Nov 27, 2024
@ParthaI ParthaI linked an issue Nov 27, 2024 that may be closed by this pull request
Inefficient memory usage for table_azure_storage_blob #854
Closed
@misraved misraved merged commit 7123e9d into main Nov 28, 2024
1 check passed
@misraved misraved deleted the issue-854 branch November 28, 2024 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@misraved misraved misraved approved these changes

Assignees

@ParthaI ParthaI

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Inefficient memory usage for table_azure_storage_blob
2 participants
@ParthaI @misraved