Merge branch 'refs/heads/main' into task-resource-v1-readiness

# Conflicts: # pkg/acceptance/bettertestspoc/README.md # pkg/acceptance/bettertestspoc/assert/resourceassert/gen/resource_schema_def.go
Snowflake-Labs · Nov 4, 2024 · 90404b9 · 90404b9
2 parents ee9583d + 9fd8f88
commit 90404b9
Show file tree

Hide file tree

Showing 251 changed files with 22,926 additions and 1,526 deletions.
diff --git a/CREATING_ISSUES.md b/CREATING_ISSUES.md
@@ -202,7 +202,20 @@ resource "snowflake_grant_privileges_to_account_role" "grant_on_procedure" {
   account_role_name = snowflake_account_role.name
   on_schema_object {
     object_type = "PROCEDURE"
-    object_name = "\"${snowflake_database.database.name}\".\"${snowflake_schema.schema.name}\".\"${snowflake_procedure.procedure.name}(NUMBER, VARCHAR)\""
+    object_name = "\"${snowflake_database.database.name}\".\"${snowflake_schema.schema.name}\".\"${snowflake_procedure.procedure.name}\"(NUMBER, VARCHAR)"
+  }
+}
+```
+
+If you manage the procedure in Terraform, you can use `fully_qualified_name` field:
+
+```terraform
+resource "snowflake_grant_privileges_to_account_role" "grant_on_procedure" {
+  privileges        = ["USAGE"]
+  account_role_name = snowflake_account_role.name
+  on_schema_object {
+    object_type = "PROCEDURE"
+    object_name = snowflake_procedure.procedure_name.fully_qualified_name
   }
 }
 ```

diff --git a/MIGRATION_GUIDE.md b/MIGRATION_GUIDE.md
@@ -7,6 +7,117 @@ across different versions.
 > [!TIP]
 > We highly recommend upgrading the versions one by one instead of bulk upgrades.
 
+## v0.97.0 ➞ v0.98.0
+
+### snowflake_streams data source changes
+New filtering options:
+- `like`
+- `in`
+- `starts_with`
+- `limit`
+- `with_describe`
+
+New output fields
+- `show_output`
+- `describe_output`
+
+Breaking changes:
+- `database` and `schema` are right now under `in` field
+- `streams` field now organizes output of show under `show_output` field and the output of describe under `describe_output` field.
+
+Please adjust your Terraform configuration files.
+
+### *(behavior change)* Provider configuration rework
+On our road to v1, we have decided to rework configuration to address the most common issues (see a [roadmap entry](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/ROADMAP.md#providers-configuration-rework)). We have created a list of topics we wanted to address before v1. We will prepare an announcement soon. The following subsections describe the things addressed in the v0.98.0.
+
+#### *(behavior change)* changed behavior of some fields
+For the fields that are not deprecated, we focused on improving validations and documentation. Also, we adjusted some fields to match our [driver's](https://github.com/snowflakedb/gosnowflake) defaults. Specifically:
+- Relaxed validations for enum fields like `protocol` and `authenticator`. Now, the case on such fields is ignored.
+- `user`, `warehouse`, `role` - added a validation for an account object identifier
+- `validate_default_parameters`, `client_request_mfa_token`, `client_store_temporary_credential`, `ocsp_fail_open`,  - to easily handle three-value logic (true, false, unknown) in provider's config, type of these fields was changed from boolean to string. For more details about default values, please refer to the [changes before v1](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/CHANGES_BEFORE_V1.md#default-values) document.
+- `client_ip` - added a validation for an IP address
+- `port` - added a validation for a port number
+- `okta_url`, `token_accessor.token_endpoint`, `client_store_temporary_credential` - added a validation for a URL address
+- `login_timeout`, `request_timeout`, `jwt_expire_timeout`, `client_timeout`, `jwt_client_timeout`, `external_browser_timeout` - added a validation for setting this value to at least `0`
+- `authenticator` - added a possibility to configure JWT flow with `SNOWFLAKE_JWT` (formerly, this was upported with `JWT`); the previous value `JWT` was left for compatibility, but will be removed before v1
+
+### *(behavior change)* handling copy_grants
+Currently, resources like `snowflake_view`, `snowflake_stream_on_table`, `snowflake_stream_on_external_table` and `snowflake_stream_on_directory_table`  support `copy_grants` field corresponding with `COPY GRANTS` during `CREATE`. The current behavior is that, when a change leading for recreation is detected (meaning a change that can not be handled by ALTER, but only by `CREATE OR REPLACE`), `COPY GRANTS` are used during recreation when `copy_grants` is set to `true`. Changing this field without changes in other field results in a noop because in this case there is no need to recreate a resource.
+
+### *(new feature)* recovering stale streams
+Starting from this version, the provider detects stale streams for `snowflake_stream_on_table`, `snowflake_stream_on_external_table` and `snowflake_stream_on_directory_table` and recreates them (optionally with `copy_grants`) to recover them. To handle this correctly, a new computed-only field `stale` has been added to these resource, indicating whether a stream is stale.
+
+### *(new feature)* snowflake_stream_on_directory_table and snowflake_stream_on_view resource
+Continuing changes made in [v0.97](#v0960--v0970), the new resource `snowflake_stream_on_directory_table` and `snowflake_stream_on_view` have been introduced to replace the previous `snowflake_stream` for streams on directory tables and streams on views.
+
+To use the new `stream_on_directory_table`, change the old `stream` from
+```terraform
+resource "snowflake_stream" "stream" {
+  name     = "stream"
+  schema   = "schema"
+  database = "database"
+
+  on_stage    = snowflake_stage.stage.fully_qualified_name
+
+  comment = "A stream."
+}
+```
+
+to
+
+```terraform
+resource "snowflake_stream_on_directory_table" "stream" {
+  name     = "stream"
+  schema   = "schema"
+  database = "database"
+
+  stage             = snowflake_stage.stage.fully_qualified_name
+
+  comment = "A stream."
+}
+```
+
+To use the new `stream_on_view`, change the old `stream` from
+```terraform
+resource "snowflake_stream" "stream" {
+  name     = "stream"
+  schema   = "schema"
+  database = "database"
+
+  on_view    = snowflake_view.view.fully_qualified_name
+
+  comment = "A stream."
+}
+```
+
+to
+
+```terraform
+resource "snowflake_stream_on_view" "stream" {
+  name     = "stream"
+  schema   = "schema"
+  database = "database"
+
+  view             = snowflake_view.view.fully_qualified_name
+
+  comment = "A stream."
+}
+```
+
+Then, follow our [Resource migration guide](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/docs/technical-documentation/resource_migration.md).
+
+### *(new feature)* Secret resources
+Added a new secrets resources for managing secrets.
+We decided to split each secret flow into individual resources.
+This segregation was based on the secret flows in CREATE SECRET. i.e.:
+- `snowflake_secret_with_client_credentials`
+- `snowflake_secret_with_authorization_code_grant`
+- `snowflake_secret_with_basic_authentication`
+- `snowflake_secret_with_generic_string`
+
+
+See reference [docs](https://docs.snowflake.com/en/sql-reference/sql/create-secret).
+
 ## v0.96.0 ➞ v0.97.0
 
 ### *(new feature)* snowflake_stream_on_table, snowflake_stream_on_external_table resource
@@ -35,7 +146,7 @@ resource "snowflake_stream" "stream" {
 
 to
 
-```
+```terraform
 resource "snowflake_stream_on_table" "stream" {
   name     = "stream"
   schema   = "schema"
@@ -48,6 +159,7 @@ resource "snowflake_stream_on_table" "stream" {
 }
 ```
 
+
 Then, follow our [Resource migration guide](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/docs/technical-documentation/resource_migration.md).
 
 ### *(new feature)* new snowflake_service_user and snowflake_legacy_service_user resources
@@ -83,7 +195,7 @@ resource "snowflake_user" "service_user" {
   lifecycle {
     ignore_changes = [user_type]
   }
-  
+
   name         = "Snowflake Service User"
   login_name   = "service_user"
   email        = "[email protected]"

diff --git a/Makefile b/Makefile
@@ -65,7 +65,7 @@ test: test-client ## run unit and integration tests
 	go test -v -cover -timeout=45m ./...
 
 test-acceptance: ## run acceptance tests
-	TF_ACC=1 SF_TF_ACC_TEST_CONFIGURE_CLIENT_ONCE=true TEST_SF_TF_REQUIRE_TEST_OBJECT_SUFFIX=1 go test -run "^TestAcc_" -v -cover -timeout=90m ./...
+	TF_ACC=1 SF_TF_ACC_TEST_CONFIGURE_CLIENT_ONCE=true TEST_SF_TF_REQUIRE_TEST_OBJECT_SUFFIX=1 go test -run "^TestAcc_" -v -cover -timeout=120m ./...
 
 test-integration: ## run SDK integration tests
 	TEST_SF_TF_REQUIRE_TEST_OBJECT_SUFFIX=1 go test -run "^TestInt_" -v -cover -timeout=45m ./...
@@ -76,6 +76,9 @@ test-architecture: ## check architecture constraints between packages
 test-client: ## runs test that checks sdk.Client without instrumentedsql
 	SF_TF_NO_INSTRUMENTED_SQL=1 SF_TF_GOSNOWFLAKE_LOG_LEVEL=debug go test ./pkg/sdk/internal/client/... -v
 
+test-object-renaming: ## runs tests in object_renaming_acceptance_test.go
+	TEST_SF_TF_ENABLE_OBJECT_RENAMING=1 go test ./pkg/resources/object_renaming_acceptace_test.go -v
+
 test-acceptance-%: ## run acceptance tests for the given resource only, e.g. test-acceptance-Warehouse
 	TF_ACC=1 TF_LOG=DEBUG SF_TF_ACC_TEST_CONFIGURE_CLIENT_ONCE=true go test -run ^TestAcc_$*_ -v -timeout=20m ./pkg/resources
 

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -1,5 +1,115 @@
 # Our roadmap
 
+## (25.10.2024) Project state overview
+
+### Goals
+
+Since the last update we have focused on:
+
+* [Reducing the feature gap](#reducing-the-feature-gap) (focusing on the Snowflake essential GA resources)
+* Redesigning identifiers (check [\#3045](https://github.com/Snowflake-Labs/terraform-provider-snowflake/discussions/3045) and [identifiers_rework_design_decisions](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/docs/technical-documentation/identifiers_rework_design_decisions.md))
+* Reworking the provider's configuration (the doc/discussion will be shared when ready)
+* Researching the object renaming in our provider (the doc will be shared when ready)
+
+These steps were all needed to get us closer to the first stable version of the provider which... is really close. In the next 1-2 months we want to:
+
+* [Wrap up the functional scope](#wrap-up-the-functional-scope) (not all the objects will be declared stable, more details below)
+* [Prepare for the V1 release](#prepare-for-the-v1-release)
+* [Prepare some basic performance benchmarks](#prepare-some-basic-performance-benchmarks) (especially, after a few major changes to the resources logic)
+* [Improve/update the documentation](#improveupdate-the-documentation)
+* [Run a closed early adopter program](#run-a-closed-early-adopter-program) to verify the readiness of the provider to enter a stable V1
+
+If there won't be any major obstacles or critical issues we aim to release V1 on **December 9th**. To better understand its scope, please check the ["What is V1?"](#what-is-v1) section.
+
+#### Reducing the feature gap
+
+During the last six months, we have been tackling objects from the [essential](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/ESSENTIAL_GA_OBJECTS.MD) and [remaining](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/REMAINING_GA_OBJECTS.MD) object lists. We’ve been aligning the implementation, adding missing attributes, and fixing known issues of the chosen objects (full list below). We had to make design decisions that sometimes were not only dictated by our engineering assessments but also by the limitations of Terraform and the underlying [SDKv2](https://developer.hashicorp.com/terraform/plugin/sdkv2). The main decisions are listed inside the repository in the [Design decisions before v1](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/CHANGES_BEFORE_V1.md#design-decisions-before-v1) (we will validate if all essential ones are present there before releasing V1).
+
+#### Wrap up the functional scope
+
+It’s about finishing the redesign of objects we want to declare stable. This mainly affects tables and accounts, but it also involves small alterations in other objects (which will be listed in the migration guide as usual).
+
+As shown [below](#which-resources-will-be-declared-stable), all but one of the [essential](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/ESSENTIAL_GA_OBJECTS.MD) objects and a few of the [remaining](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/REMAINING_GA_OBJECTS.MD) objects made the cut.
+
+#### Prepare for the V1 release
+
+This is mainly cleaning up the repository but also activities around the release:
+
+* removing deprecated resources
+* marking the resources as [preview features](#preview-resourcesdatasources)
+* removing deprecated attributes
+* potentially renaming some configuration options
+* summarizing migration guidelines between v0.x.x and v1.0.0
+
+#### Prepare some basic performance benchmarks
+
+During the resources redesign we introduced multiple changes that may affect the performance. Namely:
+
+* more SQL statements are run (`SHOW`, `DESCRIBE`, and `SHOW PARAMETERS` when needed)
+* the state we save is bigger because of the `show_output`, `describe_output`, and `parameters`.
+
+We observed that our customers tend to have lots of objects in single terraform deployments. This leads to longer planning and execution times. To be able to guide “what is too much”, we need to perform tests with more objects on our end.
+
+#### Improve/update the documentation
+
+We greatly improved the docs and the transparency of the project. However, there are still topics that need our attention (e.g. adding a migration guide directly to the [registry](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs), adding missing design decisions like granting ownership, or adding more guides \- similar to [identifiers rework](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/guides/identifiers) \- like importing existing infrastructure into Snowflake).
+
+#### Run a closed early adopter program
+
+We planned V1 to be as close as possible to the latest 0.x.x version before the V1 release. However, some changes/migrations are still expected. To improve confidence, we have decided to provide early V1 binaries to early adopters. We are still actively recruiting customers; please reach out to your Snowflake Account Manager at the earliest if you would like to participate. The program runs from mid-November to mid-December.
+
+#### What is V1?
+
+The first major version, V1, marks the first step in getting to GA by providing stable versions to customers who use the provider. We hope to have all our current customers migrate to V1. The provider's Product and Engineering teams will be available for migration or any other questions, as we believe this migration is key in preparing our customers for seamless GA adoption.
+
+From the engineering point of view, the provider will be in the stable version, but it will still stay in the Snowflake-Labs GitHub organization. We plan to change that and move it to the official snowflakedb org so that it gets the official Snowflake support. This will be a necessary step to reach the GA.
+
+#### Which resources will be declared stable
+
+Check [this list](v1-preparations/LIST_OF_STABLE_RESOURCES_FOR_V1.md) for details.
+
+#### Preview resources/datasources
+
+On our road to V1, we went through the resources, starting with the most used ones. We did not cover all of them (as described above). Because of that, in the newest [v0.97.0](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/0.97.0/docs) version of the provider, we have multiple resources that were not redesigned/fixed.
+
+We discussed two main options: removing them from 1.0.0 or marking them as preview features. We were mostly worried that removing resources would prevent the majority of our users from migrating to the stable version. On the other hand, we know they are not ready so we don’t want to declare them as stable.
+
+After consideration, we decided to leave them as preview features that need to be **explicitly enabled by the user**. This way, we are not reducing the provider's functionality between v0.x.x and v1.0.0 and leave the possibility to use them while accepting the limitations they have. However, these resources will be subject to change after V1. They should be treated as [Snowflake Preview Features](https://docs.snowflake.com/en/release-notes/preview-features) so changes to their schemas (breaking changes included\!) may be introduced even without bumping the major version of the provider.
+
+#### “Attachment” resources clarification
+
+During our road to V1 we tried to limit the number of resources needed to be configured in order to manage the given Snowflake object correctly. Because of that, we moved [Snowflake parameters](https://docs.snowflake.com/en/sql-reference/parameters) handling directly to the given object’s resource (check [this](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/CHANGES_BEFORE_V1.md#snowflake-parameters)). We did that to other types of properties too (e.g. we changed the logic for public keys handling in the [snowflake_user](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/0.95.0/docs/resources/user#rsa_public_key) resource, so that [snowflake_user_public_keys](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/0.94.1/docs/resources/user_public_keys) is no longer compatible with it).
+
+Still, these “attachment” objects serve a specific use case (i.e. the main object is not managed by Terraform but part of the object may be). It opened a question for the future not only because of the aforementioned use case but also because of a wider perspective on the default resource behavior. For example, a resource monitor can be attached to a warehouse only by a user with an ACCOUNTADMIN role (check [\#3019](https://github.com/Snowflake-Labs/terraform-provider-snowflake/issues/3019)). Some of our users would like to provision warehouses separately from assigning resource monitors but the caveat here is that leaving the resource monitor empty in the resource config will currently remove any assigned resources. Handling this would require adding a separate attachment resource and allowing a conditional change in behavior for empty assignments in the main object.
+
+The topic is wide. For the V1, we decided to keep most of the attachment resources as [preview features](#preview-resourcesdatasources) and we will discuss the need for handling the use cases described in this section as a separate topic after V1.
+
+#### Which resources will be left as preview features
+
+Check [this list](v1-preparations/LIST_OF_PREVIEW_FEATURES_FOR_V1.md) for details.
+
+#### Which resources will be removed
+
+Check [this list](v1-preparations/LIST_OF_REMOVED_RESOURCES_FOR_V1.md) for details.
+
+#### Roadmap short after V1
+
+Right after V1, we would like to focus on helping all of you with the migration. We will prioritize it so we encourage you to approach us with any issues you might have.
+
+In the meantime, if we have enough time, we want to prioritize redesigning the object marked as preview features. Currently, stages and shares open the list.
+
+#### Next year priorities
+
+This is only a general overview of the next year and may be subject to change:
+
+* Graduate out of Snowflake-Labs into the official snowflakedb organization
+* GA of the Snowflake Terraform Provider
+* Research performance improvements (optimize Snowflake invocations)
+* Grants improvements
+* Redesign remaining GA objects
+* Design transition to the [plugin framework](https://developer.hashicorp.com/terraform/plugin/framework)
+* Introduce Terraform modules
+
 ## (05.05.2024) Roadmap Overview
 
 ### Goals