Support autoscaling Cosmos DB databases and containers #7773

marc-sensenich · 2020-07-16T02:42:58Z

Adds support for enabling autoscaling Cosmos DB databases and containers

Depends on fix for issue #7825 for testing
Depends on #7597 (merged in v2.20.0)
Closes #6164
Closes #7315

Example Usage

resource "azurerm_cosmosdb_sql_database" "autoscale" {
  account_name        = "myaccount"
  name                = "mydb"
  resource_group_name = "myrg"

  autoscale_settings {
    max_throughput = 4000
  }
}

Test Status

Run with commit 324d259

Passing

--- PASS: TestAccAzureRMCosmosDbCassandraKeyspace_basic (1330.26s)
--- PASS: TestAccAzureRMCosmosDbCassandraKeyspace_complete (1525.71s)
--- PASS: TestAccAzureRMCosmosDbCassandraKeyspace_update (1582.38s)
--- PASS: TestAccAzureRMCosmosDbCassandraKeyspace_autoscale (1778.28s)
--- PASS: TestAccAzureRMCosmosGremlinDatabase_requiresImport (1376.95s)
--- PASS: TestAccAzureRMCosmosGremlinDatabase_basic (1422.04s)
--- PASS: TestAccAzureRMCosmosGremlinDatabase_complete (1563.30s)
--- PASS: TestAccAzureRMCosmosGremlinDatabase_autoscale (1735.37s)
--- PASS: TestAccAzureRMCosmosDbGremlinGraph_requiresImport (1466.30s)
--- PASS: TestAccAzureRMCosmosDbGremlinGraph_customConflictResolutionPolicy (1498.55s)
--- PASS: TestAccAzureRMCosmosDbGremlinGraph_basic (1548.04s)
--- PASS: TestAccAzureRMCosmosDbGremlinGraph_indexPolicy (1610.39s)
--- PASS: TestAccAzureRMCosmosDbGremlinGraph_autoscale (1739.45s)
--- PASS: TestAccAzureRMCosmosDbGremlinGraph_update (1777.58s)
--- PASS: TestAccAzureRMCosmosDbMongoCollection_basic (1413.85s)
--- PASS: TestAccAzureRMCosmosDbMongoCollection_throughput (1624.28s)
--- PASS: TestAccAzureRMCosmosDbMongoCollection_autoscale (1744.72s)
--- PASS: TestAccAzureRMCosmosDbMongoDatabase_complete (1302.42s)
--- PASS: TestAccAzureRMCosmosDbMongoDatabase_basic (1307.51s)
--- PASS: TestAccAzureRMCosmosDbMongoDatabase_autoscale (1618.07s)
--- PASS: TestAccAzureRMCosmosDbSqlDatabase_basic (1350.69s)
--- PASS: TestAccAzureRMCosmosDbSqlDatabase_update (1500.18s)
--- PASS: TestAccAzureRMCosmosDbSqlDatabase_autoscale (1728.33s)
--- PASS: TestAccAzureRMCosmosDbSqlContainer_complete (1426.93s)
--- PASS: TestAccAzureRMCosmosDbSqlContainer_basic (1427.35s)
--- PASS: TestAccAzureRMCosmosDbSqlContainer_update (1572.95s)
--- PASS: TestAccAzureRMCosmosDbSqlContainer_autoscale (1734.78s)
--- PASS: TestAccAzureRMCosmosDbTable_basic (1380.36s)
--- PASS: TestAccAzureRMCosmosDbTable_autoscale (1788.62s)
--- PASS: TestAccAzureRMCosmosDbTable_update (1918.18s)

Failing

These failing tests are related to #7825

--- FAIL: TestAccAzureRMCosmosDbMongoCollection_complete (1459.93s)
--- FAIL: TestAccAzureRMCosmosDbMongoCollection_withIndex (1502.19s)
--- FAIL: TestAccAzureRMCosmosDbMongoCollection_update (1639.81s)

marc-sensenich · 2020-07-21T11:05:56Z

azurerm/internal/services/cosmos/common/throughput.go

+
+func CheckForChangeFromAutoscaleAndManualThroughput(d *schema.ResourceData) error {
+	if d.HasChange("throughput") && d.HasChange("autoscale_settings") {
+		return fmt.Errorf("Switching between autoscale and manual provisioned throughput is not supported at this time.")


From https://docs.microsoft.com/en-us/azure/cosmos-db/autoscale-faq#can-i-enable-autoscale-on-an-existing-database-or-a-container

Yes. You can also switch between autoscale and standard (manual) provisioned throughput as needed. Currently, for all APIs, you can only use the Azure portal to do these operations.

Based on this and no clear documented API for toggling back and forth, I've decided to not allow this operation

There are some documented endpoints for migrating between autoscale and manual throughput (e.g. https://docs.microsoft.com/en-us/rest/api/cosmos-db-resource-provider/2020-04-01/tableresources/migratetabletoautoscale) however I couldn't find SDK support, so this seems fine 👍

simongottschlag · 2020-08-05T05:39:42Z

How is this PR going? 👍

nadworny · 2020-08-06T09:47:44Z

Does this PR also apply for azurerm_cosmosdb_mongo_database or only sql?

marc-sensenich · 2020-08-06T14:59:05Z

@simongottschlag this PR is complete and just awaiting review from an approver based on their priorities

@nadworny this PR covers all Cosmos APIs that are supported by Terraform today

WodansSon

@marc-sensenich, thank you so much for submitting this PR. It's looking really good I just left a few minor comments that I would like your input on. Once the changes are made this will LGTM! 🚀

WodansSon · 2020-08-14T22:55:42Z

azurerm/helpers/validate/cosmos.go

@@ -42,3 +42,24 @@ func CosmosThroughput(v interface{}, k string) (warnings []string, errors []erro

 	return warnings, errors
 }
+
+func CosmosMaxThroughput(v interface{}, k string) (warnings []string, errors []error) {
+	value := v.(int)


Can we validate the type here to avoid a panic if the wrong type is passed?

Suggested change

value := v.(int)

v, ok := i.(int)

if !ok {

errors = append(errors, fmt.Errorf("expected type of %q to be int", k))

return

}

WodansSon · 2020-08-14T22:56:59Z

azurerm/helpers/validate/cosmos.go

@@ -42,3 +42,24 @@ func CosmosThroughput(v interface{}, k string) (warnings []string, errors []erro

 	return warnings, errors
 }
+
+func CosmosMaxThroughput(v interface{}, k string) (warnings []string, errors []error) {


can we change v to be i to be consistent with the rest of the provider code?

Suggested change

func CosmosMaxThroughput(v interface{}, k string) (warnings []string, errors []error) {

func CosmosMaxThroughput(i interface{}, k string) (warnings []string, errors []error) {

WodansSon · 2020-08-14T23:01:30Z

azurerm/helpers/validate/cosmos.go

+func CosmosMaxThroughput(v interface{}, k string) (warnings []string, errors []error) {
+	value := v.(int)
+
+	if value < 4000 {


Can we switch all of the value variables to be v as well?

Suggested change

if value < 4000 {

if v < 4000 {

marc-sensenich · 2020-08-15T14:37:10Z

@WodansSon I've made the requested changes in 7684202.

marc-sensenich · 2020-08-24T21:00:18Z

@WodansSon or @katbyte are there any further changes you'd like to see in this PR?

joshatcaper · 2020-09-02T22:50:16Z

Hey @tombuildsstuff it looks like this is tagged for v2.26.0 and that seems to be due tomorrow. Is this going to make the merge window for that? I have a lot of use cases that could benefit from this and I'm really hoping we can start making use of this.

Co-authored-by: Tom Bamford <[email protected]>

…ce.go Co-authored-by: Tom Bamford <[email protected]>

Co-authored-by: Tom Bamford <[email protected]>

marc-sensenich · 2020-09-11T19:03:33Z

@manicminer I've addressed your most recent comments. I'll run the full test-suite for Cosmos DB this evening and update this PR with the results

marc-sensenich · 2020-09-12T11:17:02Z

@manicminer Cosmos DB database and collection acceptance tests have been run from 324d259 and are passing as noted in the initial PR message

manicminer · 2020-09-14T08:08:27Z

Test results:

Autoscaling:

All (3 unrelated failures):

jonnekleijer · 2020-09-15T20:50:12Z

@marc-sensenich

Hi Marc, thanks for doing this great work! Does this mean we can already use this functionality in Azure? I was looking at the documentation and there the autoscale option was not (yet) added.

marc-sensenich · 2020-09-16T09:43:49Z

@jonnekleijer this functionality will be available in the AzureRM provider in the yet to be released 2.28.0 version

ghost · 2020-09-17T17:47:02Z

This has been released in version 2.28.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.28.0"
}
# ... other configuration ...

ghost · 2020-10-14T14:50:26Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

ghost added the size/XXL label Jul 16, 2020

marc-sensenich marked this pull request as draft July 16, 2020 02:43

marc-sensenich mentioned this pull request Jul 16, 2020

Support for cosmos db autopilot container #6164

Closed

ghost added size/M size/L size/XL documentation and removed size/XXL size/M size/L labels Jul 20, 2020

marc-sensenich marked this pull request as ready for review July 21, 2020 10:52

marc-sensenich commented Jul 21, 2020

View reviewed changes

favoretti mentioned this pull request Jul 28, 2020

Add autoscale_throughput to cosmosdb #7932

Closed

WodansSon added the service/cosmosdb label Aug 14, 2020

WodansSon added this to the v2.24.0 milestone Aug 14, 2020

WodansSon requested a review from katbyte August 14, 2020 22:46

WodansSon requested changes Aug 14, 2020

View reviewed changes

tombuildsstuff modified the milestones: v2.24.0, v2.25.0 Aug 20, 2020

tombuildsstuff requested a review from WodansSon August 20, 2020 09:53

tombuildsstuff modified the milestones: v2.25.0, v2.26.0 Aug 27, 2020

tombuildsstuff modified the milestones: v2.26.0, v2.27.0 Sep 3, 2020

marc-sensenich and others added 19 commits September 11, 2020 13:26

Support Cosmos DB Autoscaling in Gremlin Graphs

ef96502

Support Cosmos DB Autoscaling in Mongo Collections

ad78641

Support Cosmos DB Autoscaling in Tables

627ec69

Support Cosmos DB Autoscaling in Cassandra Keyspaces

a2eb0fe

fixup! Support Cosmos DB Autoscaling in Mongo Collections

9ea59de

fixup! Support Cosmos DB Autoscaling in Tables

d6697a8

fixup! Support Cosmos DB Autoscaling in Gremlin Graphs

886eefb

fixup! Support Cosmos DB Autoscaling in Mongo Collections

7836979

Add documentation for Cosmos DB autoscale_settings

c07c48b

fixup! Add documentation for Cosmos DB autoscale_settings

f50c2db

fixup! fixup! Add documentation for Cosmos DB autoscale_settings

97b4f34

fixup! Add documentation for Cosmos DB autoscale_settings

aa28074

Update CosmosMaxThroughput validator to align with conventions

2f2629c

fixup! Update CosmosMaxThroughput validator to align with conventions

1d980ed

Update azurerm/internal/services/cosmos/cosmosdb_table_resource.go

063cebe

Co-authored-by: Tom Bamford <[email protected]>

Update azurerm/internal/services/cosmos/cosmosdb_gremlin_graph_resour…

e6dd978

…ce.go Co-authored-by: Tom Bamford <[email protected]>

Update azurerm/internal/services/cosmos/common/throughput.go

b7d0d99

Co-authored-by: Tom Bamford <[email protected]>

Update azurerm/internal/services/cosmos/common/throughput.go

464d931

Co-authored-by: Tom Bamford <[email protected]>

Address PR comments from 2020/09/11

a25ae40

fixup! Address PR comments from 2020/09/11

324d259

manicminer merged commit 0c2aab3 into hashicorp:master Sep 14, 2020

manicminer added a commit that referenced this pull request Sep 14, 2020

Update changelog for #7773

0b84e14

ghost locked as resolved and limited conversation to collaborators Oct 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support autoscaling Cosmos DB databases and containers #7773

Support autoscaling Cosmos DB databases and containers #7773

marc-sensenich commented Jul 16, 2020 •

edited

Loading

marc-sensenich Jul 21, 2020

manicminer Sep 11, 2020

simongottschlag commented Aug 5, 2020

nadworny commented Aug 6, 2020

marc-sensenich commented Aug 6, 2020 •

edited

Loading

WodansSon left a comment

WodansSon Aug 14, 2020

WodansSon Aug 14, 2020

WodansSon Aug 14, 2020

marc-sensenich commented Aug 15, 2020

marc-sensenich commented Aug 24, 2020

joshatcaper commented Sep 2, 2020

marc-sensenich commented Sep 11, 2020

marc-sensenich commented Sep 12, 2020

manicminer commented Sep 14, 2020

jonnekleijer commented Sep 15, 2020

marc-sensenich commented Sep 16, 2020 •

edited

Loading

ghost commented Sep 17, 2020

ghost commented Oct 14, 2020

-	value := v.(int)
+	v, ok := i.(int)
+	if !ok {
+		errors = append(errors, fmt.Errorf("expected type of %q to be int", k))
+		return
+	}

	func CosmosMaxThroughput(v interface{}, k string) (warnings []string, errors []error) {
	func CosmosMaxThroughput(i interface{}, k string) (warnings []string, errors []error) {

Support autoscaling Cosmos DB databases and containers #7773

Support autoscaling Cosmos DB databases and containers #7773

Conversation

marc-sensenich commented Jul 16, 2020 • edited Loading

Example Usage

Test Status

Passing

Failing

marc-sensenich Jul 21, 2020

Choose a reason for hiding this comment

manicminer Sep 11, 2020

Choose a reason for hiding this comment

simongottschlag commented Aug 5, 2020

nadworny commented Aug 6, 2020

marc-sensenich commented Aug 6, 2020 • edited Loading

WodansSon left a comment

Choose a reason for hiding this comment

WodansSon Aug 14, 2020

Choose a reason for hiding this comment

WodansSon Aug 14, 2020

Choose a reason for hiding this comment

WodansSon Aug 14, 2020

Choose a reason for hiding this comment

marc-sensenich commented Aug 15, 2020

marc-sensenich commented Aug 24, 2020

joshatcaper commented Sep 2, 2020

marc-sensenich commented Sep 11, 2020

marc-sensenich commented Sep 12, 2020

manicminer commented Sep 14, 2020

jonnekleijer commented Sep 15, 2020

marc-sensenich commented Sep 16, 2020 • edited Loading

ghost commented Sep 17, 2020

ghost commented Oct 14, 2020

marc-sensenich commented Jul 16, 2020 •

edited

Loading

marc-sensenich commented Aug 6, 2020 •

edited

Loading

marc-sensenich commented Sep 16, 2020 •

edited

Loading