`azurerm_data_factory_dataset_parquet` - Support `azure_blob_fs_location` #23261

bianyifan · 2023-09-14T08:15:30Z

Acceptance test:

$ make acctests SERVICE='datafactory' TESTARGS='-run=TestAccDataFactoryDatasetParquet_blobFS' TESTTIMEOUT=60m
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
TF_ACC=1 go test -v ./internal/services/datafactory -run=TestAccDataFactoryDatasetParquet_blobFS -timeout 60m -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccDataFactoryDatasetParquet_blobFS
=== PAUSE TestAccDataFactoryDatasetParquet_blobFS
=== RUN   TestAccDataFactoryDatasetParquet_blobFSDynamicPath
=== PAUSE TestAccDataFactoryDatasetParquet_blobFSDynamicPath
=== CONT  TestAccDataFactoryDatasetParquet_blobFS
=== CONT  TestAccDataFactoryDatasetParquet_blobFSDynamicPath
--- PASS: TestAccDataFactoryDatasetParquet_blobFS (185.55s)
--- PASS: TestAccDataFactoryDatasetParquet_blobFSDynamicPath (273.66s)
PASS
ok      github.com/hashicorp/terraform-provider-azurerm/internal/services/datafactory   273.677s

stephybun

Thanks for this PR @bianyifan.

I left some minor comments in-line. We also need to update the docs for this resource, but once that's done we can take another look through.

Thanks!

internal/services/datafactory/data_factory_dataset_parquet_resource.go

…y-dataset-parquet-azure-blob-fs

stephybun

There's a test failure for the delimited text data set which uses the same expand function

------- Stdout: -------
=== RUN   TestAccDataFactoryDatasetDelimitedText_blobFS
=== PAUSE TestAccDataFactoryDatasetDelimitedText_blobFS
=== CONT  TestAccDataFactoryDatasetDelimitedText_blobFS
------- Stderr: -------
panic: interface conversion: interface {} is nil, not bool
goroutine 681 [running]:
github.com/hashicorp/terraform-provider-azurerm/internal/services/datafactory.expandDataFactoryDatasetAzureBlobFSLocation(0xc00381c7e0?)
  /opt/teamcity-agent/work/3337027aeff310bf/internal/services/datafactory/data_factory.go:429 +0x585
github.com/hashicorp/terraform-provider-azurerm/internal/services/datafactory.expandDataFactoryDatasetLocation(0x7d26160?)
  /opt/teamcity-agent/work/3337027aeff310bf/internal/services/datafactory/data_factory.go:361 +0xb5
github.com/hashicorp/terraform-provider-azurerm/internal/services/datafactory.resourceDataFactoryDatasetDelimitedTextCreateUpdate(0x0?, {0x66c8bc0?, 0xc005ec8000?})
  /opt/teamcity-agent/work/3337027aeff310bf/internal/services/datafactory/data_factory_dataset_delimited_text_resource.go:345 +0x4a5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0x7d46980?, {0x7d46980?, 0xc0046a67e0?}, 0xd?, {0x66c8bc0?, 0xc005ec8000?})
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:766 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0032ee700, {0x7d46980, 0xc0046a67e0}, 0xc003af56c0, 0xc0039a7500, {0x66c8bc0, 0xc005ec8000})
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:909 +0xa7e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc0038eaf00, {0x7d46980?, 0xc0046a6720?}, 0xc003ae7310)
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/grpc_provider.go:1060 +0xe8d
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0050275e0, {0x7d46980?, 0xc004539ef0?}, 0xc0039fae70)
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:859 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x7116ce0?, 0xc0050275e0}, {0x7d46980, 0xc004539ef0}, 0xc0039fae00, 0x0)
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:467 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001785860, {0x7d57a60, 0xc002c26ea0}, 0xc004557680, 0xc001f596b0, 0xc8ac138, 0x0)
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/google.golang.org/grpc/server.go:1376 +0xdd2
google.golang.org/grpc.(*Server).handleStream(0xc001785860, {0x7d57a60, 0xc002c26ea0}, 0xc004557680, 0x0)
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/google.golang.org/grpc/server.go:1753 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/google.golang.org/grpc/server.go:998 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
  /opt/teamcity-agent/work/3337027aeff310bf/vendor/google.golang.org/grpc/server.go:996 +0x18c

bianyifan · 2023-09-20T06:06:09Z

Thanks for catching this! Just updated to make data_factory_dataset_delimited_text supports dynamic expressions in azure_blob_fs_location as well.

$ make acctests SERVICE='datafactory' TESTARGS='-run=TestAccDataFactoryDatasetDelimitedText_blobFS' TESTTIMEOUT=60m
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
TF_ACC=1 go test -v ./internal/services/datafactory -run=TestAccDataFactoryDatasetDelimitedText_blobFS -timeout 60m -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccDataFactoryDatasetDelimitedText_blobFS
=== PAUSE TestAccDataFactoryDatasetDelimitedText_blobFS
=== CONT  TestAccDataFactoryDatasetDelimitedText_blobFS
--- PASS: TestAccDataFactoryDatasetDelimitedText_blobFS (241.38s)
PASS
ok      github.com/hashicorp/terraform-provider-azurerm/internal/services/datafactory   241.397s
$ make acctests SERVICE='datafactory' TESTARGS='-run=TestAccDataFactoryDatasetParquet_blobFS' TESTTIMEOUT=60m
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
TF_ACC=1 go test -v ./internal/services/datafactory -run=TestAccDataFactoryDatasetParquet_blobFS -timeout 60m -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccDataFactoryDatasetParquet_blobFS
=== PAUSE TestAccDataFactoryDatasetParquet_blobFS
=== RUN   TestAccDataFactoryDatasetParquet_blobFSDynamicPath
=== PAUSE TestAccDataFactoryDatasetParquet_blobFSDynamicPath
=== CONT  TestAccDataFactoryDatasetParquet_blobFS
=== CONT  TestAccDataFactoryDatasetParquet_blobFSDynamicPath
--- PASS: TestAccDataFactoryDatasetParquet_blobFS (188.82s)
--- PASS: TestAccDataFactoryDatasetParquet_blobFSDynamicPath (261.98s)
PASS
ok      github.com/hashicorp/terraform-provider-azurerm/internal/services/datafactory   261.995s

stephybun

Thanks @bianyifan, tests are looking green. LGTM 👍

<Actions> <action id="4a39167e811ac038e4a588362092472c27cfbe9e4929ae61d035f708a093a669"> <h3>Bump Terraform `azurerm` provider version</h3> <details id="1d9343c012f5434ac9fe8a98135bae3667b399259be16d9b14302ea3bd424a24"> <summary>Update Terraform lock file</summary> <p>"hashicorp/azurerm" updated from "3.73.0" to "3.74.0" in file ".terraform.lock.hcl"</p> <details> <summary>3.74.0</summary> <pre>Changelog retrieved from:
	https://github.com/hashicorp/terraform-provider-azurerm/releases/tag/v3.74.0
NOTES:

* `azurerm_synapse_sql_pool` - users that have imported `azurerm_synapse_sql_pool` resources that were created outside of Terraform using an `LRS` storage account type will need to use `ignore_changes` to avoid the resource from being destroyed and recreated.

FEATURES:

* **New Resource**: `azurerm_arc_resource_bridge_appliance` ([#23108](hashicorp/terraform-provider-azurerm#23108 **New Resource**: `azurerm_data_factory_dataset_azure_sql_table` ([#23264](hashicorp/terraform-provider-azurerm#23264 **New Resource**: `azurerm_function_app_connection` ([#23127](https://github.com/hashicorp/terraform-provider-azurerm/issues/23127))

ENHANCEMENTS:

* dependencies: updating to `v0.20230918.1115907` of `github.com/hashicorp/go-azure-sdk` ([#23337](hashicorp/terraform-provider-azurerm#23337 dependencies: downgrading to `v1.12.5` of `github.com/rickb777/date` ([#23296](hashicorp/terraform-provider-azurerm#23296 `mysql`: updating to use API Version `2022-01-01` ([#23320](hashicorp/terraform-provider-azurerm#23320 `azurerm_app_configuration` - support for the `replica` block ([#22452](hashicorp/terraform-provider-azurerm#22452 `azurerm_bot_channel_directline` - support for `user_upload_enabled`, `endpoint_parameters_enabled`, and `storage_enabled` ([#23149](hashicorp/terraform-provider-azurerm#23149 `azurerm_container_app` - support for scale rules ([#23294](hashicorp/terraform-provider-azurerm#23294 `azurerm_container_app_environment` - support for zone redundancy ([#23313](hashicorp/terraform-provider-azurerm#23313 `azurerm_container_group` - support for the `key_vault_user_identity_id` property for Customer Managed Keys ([#23332](hashicorp/terraform-provider-azurerm#23332 `azurerm_cosmosdb_account` - support for MongoDB connection strings ([#23331](hashicorp/terraform-provider-azurerm#23331 `azurerm_data_factory_dataset_delimited_text` - support for the `dynamic_file_system_enabled`, `dynamic_path_enabled`, and `dynamic_filename_enabled` properties ([#23261](hashicorp/terraform-provider-azurerm#23261 `azurerm_data_factory_dataset_parquet` - support for the `azure_blob_fs_location` block ([#23261](hashicorp/terraform-provider-azurerm#23261 `azurerm_monitor_diagnostic_setting` - validation to ensure either `category` or `category_group` are supplied in `enabled_log` and `log` blocks ([#23308](hashicorp/terraform-provider-azurerm#23308 `azurerm_network_interface` - support for the `auxiliary_mode` and `auxiliary_sku` properties ([#22979](hashicorp/terraform-provider-azurerm#22979 `azurerm_postgresql_flexible_server` - increased the maximum supported value for `storage_mb` ([#23277](hashicorp/terraform-provider-azurerm#23277 `azurerm_shared_image_version` - support for the `replicated_region_deletion_enabled` and `target_region.exclude_from_latest_enabled` properties ([#23147](hashicorp/terraform-provider-azurerm#23147 `azurerm_storage_account` - support for setting `domain_name` and `domain_guid` for `AADKERB` ([#22833](hashicorp/terraform-provider-azurerm#22833 `azurerm_storage_account_customer_managed_key` - support for cross-tenant customer-managed keys with the `federated_identity_client_id`, and `key_vault_uri` properties ([#20356](hashicorp/terraform-provider-azurerm#20356 `azurerm_web_application_firewall_policy` - support for the `rate_limit_duration`, `rate_limit_threshold`, `group_rate_limit_by`, and `request_body_inspect_limit_in_kb` properties ([#23239](https://github.com/hashicorp/terraform-provider-azurerm/issues/23239))

BUG FIXES:

* Data Source: `azurerm_container_app_environment`: fix `log_analytics_workspace_name` output to correct value ([#23298](hashicorp/terraform-provider-azurerm#23298 `azurerm_api_management_api` - set the `service_url` property when importing the resource ([#23011](hashicorp/terraform-provider-azurerm#23011 `azurerm_app_configuration` - prevent crash by nil checking the encryption configuration ([#23302](hashicorp/terraform-provider-azurerm#23302 `azurerm_app_configuration_feature` - update `percentage_filter_value` to accept correct type of float ([#23263](hashicorp/terraform-provider-azurerm#23263 `azurerm_container_app` - fix an issue with `commands` and `args` being overwritten when using multiple containers ([#23338](hashicorp/terraform-provider-azurerm#23338 `azurerm_key_vault_certificate` - fix issue where certificates couldn't be recovered anymore ([#23204](hashicorp/terraform-provider-azurerm#23204 `azurerm_key_vault_key` - the ForceNew when `expiration_date` is removed from the config file ([#23327](hashicorp/terraform-provider-azurerm#23327 `azurerm_linux_function_app` - fix a bug in setting the storage settings when using Elastic Premium plans ([#21212](hashicorp/terraform-provider-azurerm#21212 `azurerm_linux_web_app` - fix docker app stack update ([#23303](hashicorp/terraform-provider-azurerm#23303 `azurerm_linux_web_app` - fix crash in auto heal expansion ([#21328](hashicorp/terraform-provider-azurerm#21328 `azurerm_linux_web_app_slot` - fix docker app stack update ([#23303](hashicorp/terraform-provider-azurerm#23303 `azurerm_linux_web_app_slot` - fix crash in auto heal expansion ([#21328](hashicorp/terraform-provider-azurerm#21328 `azurerm_log_analytics_solution` - fix bug where the resource wasn't handling successful creation on subsequent applies ([#23312](hashicorp/terraform-provider-azurerm#23312 `azurerm_management_group_subscription_association` - fix bug to correctly mark resource as gone if not found during read ([#23335](hashicorp/terraform-provider-azurerm#23335 `azurerm_mssql_elasticpool` - remove check that prevents `license_type` from being set for certain skus ([#23262](hashicorp/terraform-provider-azurerm#23262 `azurerm_servicebus_queue` - fixing an issue where `auto_delete_on_idle` couldn't be set to `P10675199DT2H48M5.4775807S` ([#23296](hashicorp/terraform-provider-azurerm#23296 `azurerm_servicebus_topic` - fixing an issue where `auto_delete_on_idle` couldn't be set to `P10675199DT2H48M5.4775807S` ([#23296](hashicorp/terraform-provider-azurerm#23296 `azurerm_storage_account` - prevent sending unsupported blob properties in payload for `Storage` account kind ([#23288](hashicorp/terraform-provider-azurerm#23288 `azurerm_synapse_sql_pool` - expose `storage_account_type` ([#23217](hashicorp/terraform-provider-azurerm#23217 `azurerm_windows_function_app` - fix a bug in setting the storage settings when using Elastic Premium plans ([#21212](hashicorp/terraform-provider-azurerm#21212 `azurerm_windows_web_app` - fix docker app stack update ([#23303](hashicorp/terraform-provider-azurerm#23303 `azurerm_windows_web_app_slot` - fix docker app stack update ([#23303](https://github.com/hashicorp/terraform-provider-azurerm/issues/23303))

DEPRECATIONS:

* `azurerm_application_gateway` - deprecate `Standard` and `WAF` skus ([#23310](hashicorp/terraform-provider-azurerm#23310 `azurerm_bot_channel_web_chat` - deprecate `site_names` in favour of `site` block ([#23161](hashicorp/terraform-provider-azurerm#23161 `azurerm_monitor_diagnostic_setting` - deprecate `retention_policy` in favour of `azurerm_storage_management_policy` ([#23260](https://github.com/hashicorp/terraform-provider-azurerm/issues/23260))


</pre> </details> </details> </action> </Actions> --- <table> <tr> <td width="77"> <img src="https://www.updatecli.io/images/updatecli.png" alt="Updatecli logo" width="50" height="50"> </td> <td> <p> Created automatically by <a href="https://www.updatecli.io/">Updatecli</a> </p> <details><summary>Options:</summary> <br /> <p>Most of Updatecli configuration is done via <a href="https://www.updatecli.io/docs/prologue/quick-start/">its manifest(s)</a>.</p> <ul> <li>If you close this pull request, Updatecli will automatically reopen it, the next time it runs.</li> <li>If you close this pull request and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made.</li> </ul> <p> Feel free to report any issues at <a href="https://github.com/updatecli/updatecli/issues">github.com/updatecli/updatecli</a>.<br /> If you find this tool useful, do not hesitate to star <a href="https://github.com/updatecli/updatecli/stargazers">our GitHub repository</a> as a sign of appreciation, and/or to tell us directly on our <a href="https://matrix.to/#/#Updatecli_community:gitter.im">chat</a>! </p> </details> </td> </tr> </table> Co-authored-by: Jenkins Infra Bot (updatecli) <[email protected]>

github-actions · 2024-05-13T02:12:15Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

bianyifan added 3 commits September 14, 2023 15:13

data_factory_dataset_parquet support azure_blob_fs_location

f88432c

support read azure_blob_fs_location

607973a

add acc test

2dbefda

bianyifan marked this pull request as ready for review September 14, 2023 08:15

github-actions bot added service/data-factory size/L labels Sep 14, 2023

bianyifan changed the title ~~data_factory_dataset_parquet - Support azure_blob_fs_location~~ azurerm_data_factory_dataset_parquet - Support azure_blob_fs_location Sep 14, 2023

stephybun requested changes Sep 19, 2023

View reviewed changes

github-actions bot added the waiting-response label Sep 19, 2023

bianyifan added 2 commits September 19, 2023 15:49

ExactlyOneOf and doc

37811b3

Merge remote-tracking branch 'remotes/upstream/main' into data-factor…

6a277e5

…y-dataset-parquet-azure-blob-fs

github-actions bot added documentation and removed waiting-response labels Sep 19, 2023

bianyifan requested a review from stephybun September 19, 2023 08:03

stephybun requested changes Sep 19, 2023

View reviewed changes

github-actions bot added the waiting-response label Sep 19, 2023

also update azure_blob_fs_location for dataset_delimited_text

314baa1

bianyifan requested a review from stephybun September 20, 2023 06:05

github-actions bot added size/XL and removed waiting-response size/L labels Sep 20, 2023

stephybun added the enhancement label Sep 20, 2023

stephybun approved these changes Sep 20, 2023

View reviewed changes

stephybun merged commit 7e2a425 into hashicorp:main Sep 20, 2023
23 checks passed

github-actions bot added this to the v3.74.0 milestone Sep 20, 2023

bianyifan deleted the data-factory-dataset-parquet-azure-blob-fs branch September 20, 2023 07:11

stephybun added a commit that referenced this pull request Sep 20, 2023

Update for #23239 #23261 #23331 #22452 #23264

b21ae25

github-actions bot locked as resolved and limited conversation to collaborators May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`azurerm_data_factory_dataset_parquet` - Support `azure_blob_fs_location` #23261

`azurerm_data_factory_dataset_parquet` - Support `azure_blob_fs_location` #23261

bianyifan commented Sep 14, 2023

stephybun left a comment

stephybun left a comment

bianyifan commented Sep 20, 2023

stephybun left a comment

github-actions bot commented May 13, 2024

azurerm_data_factory_dataset_parquet - Support azure_blob_fs_location #23261

azurerm_data_factory_dataset_parquet - Support azure_blob_fs_location #23261

Conversation

bianyifan commented Sep 14, 2023

stephybun left a comment

Choose a reason for hiding this comment

stephybun left a comment

Choose a reason for hiding this comment

bianyifan commented Sep 20, 2023

stephybun left a comment

Choose a reason for hiding this comment

github-actions bot commented May 13, 2024

`azurerm_data_factory_dataset_parquet` - Support `azure_blob_fs_location` #23261

`azurerm_data_factory_dataset_parquet` - Support `azure_blob_fs_location` #23261