Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Resource for External Volumes #3106

Merged
merged 19 commits into from
Oct 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1704a50
feat: Add External Volume Resource
jdoldis Sep 24, 2024
afbd4a0
Clean up and call out existing issue
jdoldis Sep 29, 2024
fd4062e
Review feedback part 1
jdoldis Oct 14, 2024
f143a26
Generate show output assertions and update acceptance tests to use these
jdoldis Oct 14, 2024
afe0235
Add V1 candidate note
jdoldis Oct 14, 2024
2304ae4
Add custom external volumes assert for storage locations at index, and
jdoldis Oct 15, 2024
1f190e6
Update helpers file with changes suggested in review comments
jdoldis Oct 20, 2024
68c104d
Reduce number of test files and move list of valid storage providers
jdoldis Oct 20, 2024
337f545
Remove schema version, add defer for temp location cleanup and clean …
jdoldis Oct 20, 2024
4f89a9d
Rename longestCommonPrefix and use DeepEqual for StorageLocationsEqual
jdoldis Oct 20, 2024
2c09c7d
Order schema gen list alphabetically
jdoldis Oct 20, 2024
15b8738
Use pre defined temp storage location name
jdoldis Oct 21, 2024
90f2c88
Remove defer as it's not currently working
jdoldis Oct 21, 2024
6029e87
Finish revert of defer change
jdoldis Oct 21, 2024
ff43f8a
Bring back SNOW-999142 TODO, rename function and fix typo
jdoldis Oct 22, 2024
8ac4af1
Set FullyQualifiedNameAttributeName and move functions to external_vo…
jdoldis Oct 22, 2024
184b06c
Merge remote-tracking branch 'upstream/main'
sfc-gh-jmichalak Oct 25, 2024
b2836c7
Resolve conflicts
sfc-gh-jmichalak Oct 25, 2024
c4af02d
Merge remote-tracking branch 'upstream/main'
sfc-gh-jmichalak Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions docs/resources/external_volume.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
page_title: "snowflake_external_volume Resource - terraform-provider-snowflake"
subcategory: ""
description: |-
Resource used to manage external volume objects. For more information, check external volume documentation https://docs.snowflake.com/en/sql-reference/commands-data-loading#external-volume.
---

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

V1 candidate note is missing, a new template has to be created in the templates/resources directory for external volume (you can copy-paste the schema.md.tmpl one, but change the migration guide link header to the correct one).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have added that without a migration guide link for now as I'm not sure when we'll get it released, will leave the comment as unresolved so we don't forget.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, we don't want this note here. This object is not officially supported and requires a follow-up on our side. After this follow-up (also with an entry in the migration guide), we can add a note about the release candidate.

We can merge it now, but this note shouldn't be present in the next release.

!> **V1 release candidate** This resource was reworked and is a release candidate for the V1. We do not expect significant changes in it before the V1. We will welcome any feedback and adjust the resource if needed. Any errors reported will be resolved with a higher priority. We encourage checking this resource out before the V1 release.

# snowflake_external_volume (Resource)

Resource used to manage external volume objects. For more information, check [external volume documentation](https://docs.snowflake.com/en/sql-reference/commands-data-loading#external-volume).



<!-- schema generated by tfplugindocs -->
## Schema

### Required

- `name` (String) Identifier for the external volume; must be unique for your account. Due to technical limitations (read more [here](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/docs/technical-documentation/identifiers_rework_design_decisions.md#known-limitations-and-identifier-recommendations)), avoid using the following characters: `|`, `.`, `(`, `)`, `"`
- `storage_location` (Block List, Min: 1) List of named cloud storage locations in different regions and, optionally, cloud platforms. Minimum 1 required. The order of the list is important as it impacts the active storage location, and updates will be triggered if it changes. Note that not all parameter combinations are valid as they depend on the given storage_provider. Consult [the docs](https://docs.snowflake.com/en/sql-reference/sql/create-external-volume#cloud-provider-parameters-cloudproviderparams) for more details on this. (see [below for nested schema](#nestedblock--storage_location))

### Optional

- `allow_writes` (String) Specifies whether write operations are allowed for the external volume; must be set to TRUE for Iceberg tables that use Snowflake as the catalog. Available options are: "true" or "false". When the value is not set in the configuration the provider will put "default" there which means to use the Snowflake default for this value.
- `comment` (String) Specifies a comment for the external volume.

### Read-Only

- `describe_output` (List of Object) Outputs the result of `DESCRIBE EXTERNAL VOLUME` for the given external volume. (see [below for nested schema](#nestedatt--describe_output))
- `fully_qualified_name` (String) Fully qualified name of the resource. For more information, see [object name resolution](https://docs.snowflake.com/en/sql-reference/name-resolution).
- `id` (String) The ID of this resource.
- `show_output` (List of Object) Outputs the result of `SHOW EXTERNAL VOLUMES` for the given external volume. (see [below for nested schema](#nestedatt--show_output))

<a id="nestedblock--storage_location"></a>
### Nested Schema for `storage_location`

Required:

- `storage_base_url` (String) Specifies the base URL for your cloud storage location.
- `storage_location_name` (String) Name of the storage location. Must be unique for the external volume. Do not use the name `terraform_provider_sentinel_storage_location` - this is reserved for the provider for performing update operations. Due to technical limitations (read more [here](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/docs/technical-documentation/identifiers_rework_design_decisions.md#known-limitations-and-identifier-recommendations)), avoid using the following characters: `|`, `.`, `(`, `)`, `"`
- `storage_provider` (String) Specifies the cloud storage provider that stores your data files. Valid values are (case-insensitive): `GCS` | `AZURE` | `S3` | `S3GOV`.

Optional:

- `azure_tenant_id` (String) Specifies the ID for your Office 365 tenant that the allowed and blocked storage accounts belong to.
- `encryption_kms_key_id` (String) Specifies the ID for the KMS-managed key used to encrypt files.
- `encryption_type` (String) Specifies the encryption type used.
- `storage_aws_role_arn` (String) Specifies the case-sensitive Amazon Resource Name (ARN) of the AWS identity and access management (IAM) role that grants privileges on the S3 bucket containing your data files.

Read-Only:

- `storage_aws_external_id` (String) External ID that Snowflake uses to establish a trust relationship with AWS.


<a id="nestedatt--describe_output"></a>
### Nested Schema for `describe_output`

Read-Only:

- `default` (String)
- `name` (String)
- `parent` (String)
- `type` (String)
- `value` (String)


<a id="nestedatt--show_output"></a>
### Nested Schema for `show_output`

Read-Only:

- `allow_writes` (Boolean)
- `comment` (String)
- `name` (String)
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,11 @@ var allStructs = []SdkObjectDef{
ObjectType: sdk.ObjectTypeTask,
ObjectStruct: sdk.Task{},
},
{
IdType: "sdk.ExternalVolumeObjectIdentifier",
ObjectType: sdk.ObjectTypeExternalVolume,
ObjectStruct: sdk.ExternalVolume{},
},
{
IdType: "sdk.SchemaObjectIdentifier",
ObjectType: sdk.ObjectTypeSecret,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
package resourceassert

import (
"fmt"
"strconv"

"github.com/Snowflake-Labs/terraform-provider-snowflake/pkg/acceptance/bettertestspoc/assert"
)

func (e *ExternalVolumeResourceAssert) HasStorageLocationLength(len int) *ExternalVolumeResourceAssert {
jdoldis marked this conversation as resolved.
Show resolved Hide resolved
e.AddAssertion(assert.ValueSet("storage_location.#", strconv.FormatInt(int64(len), 10)))
return e
}

func (e *ExternalVolumeResourceAssert) HasStorageLocationAtIndex(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: we can pass here a slice of a custom struct with these fields. But it's okay like this.

index int,
expectedName string,
expectedStorageProvider string,
expectedStorageBaseUrl string,
expectedStorageAwsRoleArn string,
expectedEncryptionType string,
expectedEncryptionKmsKeyId string,
expectedAzureTenantId string,
) *ExternalVolumeResourceAssert {
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.storage_location_name", strconv.Itoa(index)), expectedName))
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.storage_provider", strconv.Itoa(index)), expectedStorageProvider))
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.storage_base_url", strconv.Itoa(index)), expectedStorageBaseUrl))
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.storage_aws_role_arn", strconv.Itoa(index)), expectedStorageAwsRoleArn))
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.encryption_type", strconv.Itoa(index)), expectedEncryptionType))
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.encryption_kms_key_id", strconv.Itoa(index)), expectedEncryptionKmsKeyId))
e.AddAssertion(assert.ValueSet(fmt.Sprintf("storage_location.%s.azure_tenant_id", strconv.Itoa(index)), expectedAzureTenantId))
return e
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions pkg/acceptance/check_destroy.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,9 @@ var showByIdFunctions = map[resources.Resource]showByIdFunc{
resources.ExternalTable: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.ExternalTables.ShowByID)
},
resources.ExternalVolume: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.ExternalVolumes.ShowByID)
},
resources.FailoverGroup: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.FailoverGroups.ShowByID)
},
Expand Down
55 changes: 35 additions & 20 deletions pkg/acceptance/helpers/external_volume_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package helpers

import (
"context"
"fmt"
"testing"

"github.com/Snowflake-Labs/terraform-provider-snowflake/pkg/sdk"
Expand All @@ -21,39 +20,55 @@ func NewExternalVolumeClient(context *TestClientContext, idsGenerator *IdsGenera
}
}

func (c *ExternalVolumeClient) exec(sql string) error {
ctx := context.Background()
_, err := c.context.client.ExecForTests(ctx, sql)
return err
func (c *ExternalVolumeClient) client() sdk.ExternalVolumes {
return c.context.client.ExternalVolumes
}

// TODO(SNOW-999142): Use SDK implementation for External Volume once it's available
// TODO(SNOW-999142): Switch to returning *sdk.ExternalVolume. Need to update existing acceptance tests for this.
func (c *ExternalVolumeClient) Create(t *testing.T) (sdk.AccountObjectIdentifier, func()) {
jdoldis marked this conversation as resolved.
Show resolved Hide resolved
t.Helper()
ctx := context.Background()

id := c.ids.RandomAccountObjectIdentifier()
err := c.exec(fmt.Sprintf(`
create external volume %s
storage_locations =
(
(
name = 'my-s3-us-west-2'
storage_provider = 's3'
storage_base_url = 's3://my_example_bucket/'
storage_aws_role_arn = 'arn:aws:iam::123456789012:role/myrole'
encryption=(type='aws_sse_kms' kms_key_id='1234abcd-12ab-34cd-56ef-1234567890ab')
)
);
`, id.FullyQualifiedName()))
kmsKeyId := "1234abcd-12ab-34cd-56ef-1234567890ab"
storageLocations := []sdk.ExternalVolumeStorageLocation{
{
S3StorageLocationParams: &sdk.S3StorageLocationParams{
Name: "my-s3-us-west-2",
StorageProvider: "S3",
StorageAwsRoleArn: "arn:aws:iam::123456789012:role/myrole",
StorageBaseUrl: "s3://my_example_bucket/",
Encryption: &sdk.ExternalVolumeS3Encryption{
Type: "AWS_SSE_KMS",
KmsKeyId: &kmsKeyId,
},
},
},
}

req := sdk.NewCreateExternalVolumeRequest(id, storageLocations)
err := c.client().Create(ctx, req)
require.NoError(t, err)

_, showErr := c.client().ShowByID(ctx, id)
require.NoError(t, showErr)

return id, c.DropFunc(t, id)
jdoldis marked this conversation as resolved.
Show resolved Hide resolved
}

func (c *ExternalVolumeClient) Alter(t *testing.T, req *sdk.AlterExternalVolumeRequest) {
t.Helper()
ctx := context.Background()
err := c.client().Alter(ctx, req)
require.NoError(t, err)
}

func (c *ExternalVolumeClient) DropFunc(t *testing.T, id sdk.AccountObjectIdentifier) func() {
t.Helper()
ctx := context.Background()

return func() {
err := c.exec(fmt.Sprintf(`drop external volume if exists %s`, id.FullyQualifiedName()))
err := c.client().Drop(ctx, sdk.NewDropExternalVolumeRequest(id).WithIfExists(true))
require.NoError(t, err)
}
}
Loading
Loading