Expose HCL2 native typing (cty) in SDKv2

[paultyng]

Problem Statement (current situation)

The SDK historically defined types via its own "type system" (e.g. schema.TypeString, schema.TypeInt, schema.TypeBool, ...). Values are then internally serialized on both sides (Terraform's core and Terraform SDK/provider).

Prior to 0.12 Terraform didn't have a proper type system. 0.12 uses cty as the main way to express types and serialize data accordingly. cty is already used by SDK under the hood, but it was not exposed to providers yet to retain compatibility.

The existing system limits us due to the way it used to serialize data prior to 0.12 (mostly as strings) and makes it very difficult to change behaviour of existing types, fix bugs in that area or introduce new types as all the data need to go through pre-0.12 shim layers.

Proposal

Using cty as the main interface for declaring types in the SDK is the first step towards getting rid of the wrapper which effectively became unnecessary shim translating between cty and "its own type system" in 0.12.

Caveats

This is effectively protocol 5 only feature because cty was introduced in protocol 5. However provider's interface is SDK's schema structs, so we need to explore how to best introduce this feature and consider what impact it will have on 0.11 users, where Terraform talks protocol 4 (i.e. no cty).

Ideally we'd want to prevent even launching a plugin that opted into such proto5-only feature, but the exact implementation and possibilities are TBD.

---

There is a number of type-related features that would benefit from schema defining types more directly through cty:

• Support the pseudo dynamic type #248
• Allow providers' schema to define named HCL blocks #220
• Add support for maps with non-primitive types (rich / object map support) #62

[paddycarver]

Given our design constraints for v2 (not removing the shims) is this still a goal, @paultyng? Or is this better targeted for the HCL2-native SDK redesign project?

[paultyng]

I think we need to definitively answer whether we can circumvent the shims in an opt-in manner, but that may take the form of using the mux. Without an HCL2 native "framework" though, there is still value in offering direct access to native types here.

[jakubpech]

Hi, can I please ask for a small update regarding the possibilities to use HCL2 native typing? We still need to differentiate between
the zero value and nil in our vmware sd-wan (velocloud) provider and have "temporary" dirty workarounds from 2019.

We have the situations, where:
advertise = true >> advertise IP ranges
advertise = false >> no not advertise
advertise not mentioned in .tf (advertise = nil) >> use setting from the parent profile

So once we set "advertise" to true or false to override the parent setting, we can never go back as removing the "advertise" parameter from child .tf will only set it to false (default go value).

We're only the customer so do not have any power to solve it on the API side.

[eddycharly]

Any news on this ?
Is there a framework other than plugin sdk v2 that gives more control when writing providers ?

[magodo]

@eddycharly You might want to take a look at: https://github.com/hashicorp/terraform-plugin-framework

[bflad]

A good solution here, as mentioned above, would be to check out terraform-plugin-framework, which was built from the ground up with more intuitive (given the current state of Terraform data handling) and customizable data types. It is inching closer to becoming the recommended method for developing providers.

For folks writing providers in this SDK that are interested in directly accessing native configuration, plan, and state values in CRUD functions without the other SDK data handling, helper/schema.ResourceData in recent versions now has GetRawConfig, GetRawPlan, and GetRawState methods available. This functionality is experimental should be considered only for advanced use cases. There is not currently any documentation for how to interact with this type of cty data, but it is available as a stopgap solution until folks can migrate over to the new SDK.

The maintainers are now actively focused on developing the new SDK, so while this stopgap solution may be less than ideal due to its complexity, it is more likely better that prioritization efforts continue in that project rather than here so the ecosystem can move forward. In that regard, I'm going to close this issue out. If there are questions around using this functionality or how to iteratively migrate providers to the new framework, please raise them in the HashiCorp Discuss forums. If there is a specific bug report or feature request relating to the cty data available, please open a targeted new issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.