`azurerm_databricks_workspace` - Upgrade Databricks API to the 2021-04-01-preview version #12543

WodansSon · 2021-07-10T03:26:03Z

NOTE: This PR cannot be merged until the service team removes the subscription level feature flag from Azure excluding all subscriptions who have not registered the feature flag within their subscription from calling this API version, else it would break all customers currently using the Terraform Databricks resource.

ADDITIONAL: Service team has indicated that this version of the API will be callable by all subscriptions on the 26th of July. So I have moved this into the v2.70.0 Release.

ADDITIONAL ADDITIONAL: Removing the feature flag for the API is taking longer than expected to roll out and is now estimated to be complete by Friday 7/30/2021. So I have moved this into the v2.71.0 Release.

Dependencies: Upgrade Databricks API to the 2021-04-01-preview version
New Data Source: azurerm_databricks_workspace_private_endpoint_connection

Added fields:

public_network_access_enabled
network_security_group_rules_required
load_balancer_backend_address_pool_id
nat_gateway_name
public_ip_name
storage_account_name
storage_account_sku_name
vnet_address_prefix
public_subnet_network_security_group_association_id
private_subnet_network_security_group_association_id

Data Source fields:

workspace_id
private_endpoint_id
connections block
connections block fields:
- workspace_private_endpoint_id
- name
- status
- description
- action_required

Examples:

Added an example configuration file on how to correctly configure Databricks Workspace with Private Link Endpoint.

fixes #11158
fixes #7799

azurerm/internal/services/databricks/databricks_workspace_resource.go

…urce.go Co-authored-by: kt <[email protected]>

…raform-providers/terraform-provider-azurerm into e-databricks-upgrade-private

katbyte

Thanks @WodansSon - just a couple comments left inline

katbyte · 2021-07-20T22:51:55Z

azurerm/internal/services/databricks/databricks_workspace_resource.go

+						"load_balancer_backend_pool_name": {
+							Type:         pluginsdk.TypeString,
+							ForceNew:     true,
+							Optional:     true,
+							AtLeastOneOf: workspaceCustomParametersString(),
+						},
+
+						"load_balancer_id": {


can you get the name from the id?

No, I cannot get the backend name from the load balancer but I can get the load balancer ID from the backend pool id. 🤔

Fixed... I have removed the two custom parameters(e.g. load_balancer_backend_pool_name and load_balancer_id) and replaced it with a single top level parameter load_balancer_backend_address_pool_id. Since this field is not real, I had to do some magic behind the curtains to get it to work and inject the values into the custom parameters struct of the ARM call. That said, it appears to be correct since my local test cases are working, azure portal and the state file appears to be correct as well.

katbyte · 2021-07-20T22:52:18Z

azurerm/internal/services/databricks/databricks_workspace_resource.go

+				Default:  true,
+			},
+
+			"require_network_security_group_rules": {


should we be consistant with the tense?

Suggested change

"require_network_security_group_rules": {

"network_security_group_rules_required": {

katbyte · 2021-07-20T22:52:45Z

azurerm/internal/services/databricks/databricks_workspace_resource.go

+							AtLeastOneOf: workspaceCustomParametersString(),
+						},
+
+						"storage_account_sku_name": {


can we get this from the storage account or is it auto created from the name above:

No, this is auto created by the Databricks RP, it has a default value of standard_GRS I think. This is offered as an override so you can control sku of the auto generated storage account.

WodansSon · 2021-07-30T22:25:52Z

katbyte

Thanks @WodansSon - LGTM 🍰

github-actions · 2021-08-06T03:16:13Z

This functionality has been released in v2.71.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2021-09-06T02:08:24Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

Upgrade databricks API to 2021-04-01-preview

83e89be

WodansSon added enhancement documentation dependencies service/databricks labels Jul 10, 2021

WodansSon added this to the v2.68.0 milestone Jul 10, 2021

github-actions bot added the size/XL label Jul 10, 2021

WodansSon added 5 commits July 9, 2021 20:28

Fix documentation lint error

6ef7d2b

Expose new fields in API

71cf895

Fix lint error

c57dd87

Fix new line lint error...

e3b5948

Stable removes depends_on requirement

d5cc1f6

WodansSon added 2 commits July 13, 2021 12:55

add nil checks incase resource is in failed state

463ebd1

Stable new validation checks

4f5e0f7

WodansSon modified the milestones: v2.68.0, v2.69.0 Jul 15, 2021

katbyte requested changes Jul 15, 2021

View reviewed changes

azurerm/internal/services/databricks/databricks_workspace_resource.go Outdated Show resolved Hide resolved

azurerm/internal/services/databricks/databricks_workspace_resource.go Outdated Show resolved Hide resolved

WodansSon and others added 6 commits July 16, 2021 16:41

Update azurerm/internal/services/databricks/databricks_workspace_reso…

4d63970

…urce.go Co-authored-by: kt <[email protected]>

Update azurerm/internal/services/databricks/databricks_workspace_reso…

649d63a

…urce.go Co-authored-by: kt <[email protected]>

Fully working private link

e4cfa27

Merge branch 'e-databricks-upgrade-private' of https://github.com/ter…

d857180

…raform-providers/terraform-provider-azurerm into e-databricks-upgrade-private

Add datasource, example, and documentation

e879cfe

Ignore custom params not returned by the ARM GET

45fe834

WodansSon changed the title ~~[WIP]azurerm_databricks_workspace - Upgrade Databricks API to the 2021-04-01-preview version~~ azurerm_databricks_workspace - Upgrade Databricks API to the 2021-04-01-preview version Jul 17, 2021

Update names in example

6a9f729

WodansSon modified the milestones: v2.69.0, v2.70.0 Jul 20, 2021

katbyte requested changes Jul 20, 2021

View reviewed changes

WodansSon added 4 commits July 20, 2021 21:44

Update require_network_security_group_rules name

d77bea6

Update name in example

ef62428

PR feedback, added examples and updated docs

4dbba4a

rename test to remove test case ambiguity

e21492b

WodansSon modified the milestones: v2.70.0, v2.71.0 Jul 29, 2021

WodansSon added 2 commits July 30, 2021 15:05

Update tests and documentation

b6a0027

Fix misspelling in documentation

4e4d52f

katbyte approved these changes Jul 30, 2021

View reviewed changes

WodansSon merged commit 252dde4 into master Jul 30, 2021

WodansSon deleted the e-databricks-upgrade-private branch July 30, 2021 23:02

WodansSon added a commit that referenced this pull request Jul 30, 2021

updating to include #12543

fd58215

tombuildsstuff mentioned this pull request Aug 19, 2021

Changelog? #13056

Closed

nfx mentioned this pull request Aug 20, 2021

[ISSUE] Azure authentication stopped working with azurerm v2.71.0+ databricks/terraform-provider-databricks#781

Closed

github-actions bot locked as resolved and limited conversation to collaborators Sep 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`azurerm_databricks_workspace` - Upgrade Databricks API to the 2021-04-01-preview version #12543

`azurerm_databricks_workspace` - Upgrade Databricks API to the 2021-04-01-preview version #12543

WodansSon commented Jul 10, 2021 •

edited

Loading

katbyte left a comment

katbyte Jul 20, 2021

WodansSon Jul 21, 2021

WodansSon Jul 22, 2021 •

edited

Loading

katbyte Jul 20, 2021

WodansSon Jul 21, 2021

katbyte Jul 20, 2021

WodansSon Jul 21, 2021

WodansSon commented Jul 30, 2021

katbyte left a comment

github-actions bot commented Aug 6, 2021

github-actions bot commented Sep 6, 2021

	"require_network_security_group_rules": {
	"network_security_group_rules_required": {

azurerm_databricks_workspace - Upgrade Databricks API to the 2021-04-01-preview version #12543

azurerm_databricks_workspace - Upgrade Databricks API to the 2021-04-01-preview version #12543

Conversation

WodansSon commented Jul 10, 2021 • edited Loading

Added fields:

Data Source fields:

connections block fields:

Examples:

katbyte left a comment

Choose a reason for hiding this comment

katbyte Jul 20, 2021

Choose a reason for hiding this comment

WodansSon Jul 21, 2021

Choose a reason for hiding this comment

WodansSon Jul 22, 2021 • edited Loading

Choose a reason for hiding this comment

katbyte Jul 20, 2021

Choose a reason for hiding this comment

WodansSon Jul 21, 2021

Choose a reason for hiding this comment

katbyte Jul 20, 2021

Choose a reason for hiding this comment

WodansSon Jul 21, 2021

Choose a reason for hiding this comment

WodansSon commented Jul 30, 2021

katbyte left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 6, 2021

github-actions bot commented Sep 6, 2021

`azurerm_databricks_workspace` - Upgrade Databricks API to the 2021-04-01-preview version #12543

`azurerm_databricks_workspace` - Upgrade Databricks API to the 2021-04-01-preview version #12543

WodansSon commented Jul 10, 2021 •

edited

Loading

`connections` block fields:

WodansSon Jul 22, 2021 •

edited

Loading