Challenges and Best Practices for Upgrading Terraform Snowflake Provider to 1.0.0

[santhoshreddych]

Hello Team,

I’m looking to learn and understand how other teams are managing their Terraform Snowflake provider upgrades.

We are currently on version 0.92.0 and working towards upgrading to 1.0.0. Even upgrading one version at a time still poses same challenges with deprecated resources.

I’ve reviewed the Migration Guide and the Resource Migration Guide. While these resources are helpful to some extent, the suggested approaches don’t seem to scale well at an enterprise level. So far, the upgrade process has been quite challenging, especially when dealing with deprecated resources.

We use multiple submodules to dynamically create Snowflake objects and grants. Below are some of the key challenges we’ve encountered during the upgrade:

• Generating Terraform imports as described in 3.1.2 for every resource is highly error-prone and nearly impossible to manage.

• We attempted the import approach mentioned in 3.2.1. However, the current Terraform version we use (1.4.2) does not support for_each in import blocks. Upgrading Terraform to 1.7.2 revealed that import blocks are not supported in child modules, which has left us at a roadblock.

I’d love to hear how others are approaching this upgrade and managing these challenges. Any insights or suggestions would be greatly appreciated!

Few of the deprecated resources we currently use:

snowflake_role --> snowflake_account_role
snowflake_role_grants --> snowflake_grant_account_role
snowflake_schema_grant --> snowflake_grant_privileges_to_account_role

[sfc-gh-asawicki]

Hey @santhoshreddych. Thanks for reaching out to us.

From your question I understand that your concerns come mostly from the grant resources migrations. This migration is unfortunately not trivial.

The guidelines you reference were meant to mainly clarify how to approach the no-downtime migration properly. They were not meant to promote any method over the others. We hoped that they would be enough to craft proper automation on the user side (because of the wide variety of setups our users have). Also, we hoped that almost a year (the first new grant resources were added in v0.85.0) would be enough to tackle the migration from the old grant resources. Unfortunately, more and more users are approaching us with grant resource migration problems.

What we can recommend now is to:

• check the already existing issues and our answers (e.g. #2707, #3187, issues connected with grant_privileges_to_account_role - you can check other labels too)
• while preparing automation for imports for new grant resources, have a look at the implementation for the resource identifiers:
  • grant_privileges_to_database_role_identifier.go
  • grant_privileges_to_share_identifier.go
  • grant_privileges_to_account_role_identifier.go
  • grant_privileges_identifier_commons.go
  • grant_ownership_identifier.go
• split the deployments for upgrade into smaller ones, and preferably, logically, so that grant resources are upgraded separately
• consider migration directly to the newest version:
  • we still recommend version-by-version migration following our migration guide, however...
  • ... sometimes it may be easier to upgrade, especially from the older provider versions, by just removing old resources from the state and importing them into the desired version
  • in this case, it is essential that the migration guide and the docs for the newest version are studied thoroughly before

All the recommendations above still require automation on the user side. We can't completely eliminate the need for manual work, but because of the increasing number of requests, we will discuss internally how we can ease the migration using our limited resources.

Our current idea would be to create simple scripts that can be reused or that can at least serve as a base/inspiration for the user-side migration automation. They may contain (remember, these are still just ideas and not the features we guarantee):

• printing the desired config for new grants based on the old grant resource input
• printing the desired config for new grants based on the output from the Snowflake query
• generating proper import statements (specifically generating correct identifiers)

We are currently discussing the scope for the next months in the provider. We will make sure we discuss the above ideas next week. The high V1 adoption is our biggest objective before reaching GA, and grant resources migration looks like the biggest blocker for our customers, so we may be able to make the ideas above a high-priority topic.

If you have any other ideas on how we can help you with the migration, please don't hesitate to share them.

[santhoshreddych]

Thank you so much, @sfc-gh-asawicki, for setting the right expectation that the migration is not trivial and requires some level of automation. This reassures me that I’m on the right path and gives me confidence to move forward. Hopefully, it will all lead to the right outcome! :)

Could you provide some guidance on how the import statement can be dynamically constructed, particularly the ID part? For example, generating the ID seems to be quite dynamic and challenging, especially for snowflake_grant_privileges_to_account_role with ~4,000 assets.

[sfc-gh-asawicki]

You can check how ID is created programmatically here: grant_privileges_to_account_role.go#L1142.

Basically, it should be based on old configs and Snowflake output as I hinted here:

• printing the desired config for new grants based on the old grant resource input
• printing the desired config for new grants based on the output from the Snowflake query
• generating proper import statements (specifically generating correct identifiers)

We will discuss this week what we can provide to ease this migration and when, I will get back to you in this thread.