Question: How to evaluate unknown functions without errors?

[wata727]

Recently namespaced functions were merged. I understand that this will be used for Terraform's provider-contributed functions in the future.

TFLint, which I maintain, supports evaluation of HCL expressions. All functions were implemented, so all function evaluation was available. But I'm concerned that when provider-contributed functions become available, TFLint won't be able to support function evaluation since it won't know about all the functions (with static analysis, the providers won't be accessible).

The smartest idea is to walk through all expressions in advance, gather all function calls, and generate pseudo-functions that returns cty.DynamicVal. In native syntax this is possible, but in JSON syntax this is not possible as it does not provide a walker. See also #543

Given the above, it seems like you have two options:

• Provide a way to gather all function calls in JSON syntax
• Provide a fallback function when evaluating unknown functions
  • Imagine adding a function field like FallbackUnknownFunction to hcl.EvalContext

Honestly, I don't like the 2nd option, but it's the only solution I can think of at the moment. What do you think? Is there any other better way?

[apparentlymart]

Hi @wata727,

Indeed, this new capability is for supporting extensible function namespaces, and Terraform's provider-contributed functions are likely to be the first user of it.

The exact details of how this is going to work in Terraform itself are still in progress; the PR you linked to was just a proof-of-concept to show that there was at least one viable way to implement it, but it had various problems in the implementation details and so the final detailed design will probably be a little different.

It does seem possible that Terraform itself might need to analyze expressions to learn which functions are being called too, so that it can make sure the provider plugins are available at the right time. In the proof-of-concept I avoided that by starting up a new plugin instance for each function call, but that's a very high-overhead approach that the final design will presumably try to avoid.

Given that, it's possible that a subsequent HCL API change will add a function similar to the current Expression.Variables for finding the functions called in a particular expression using static analysis, which would then encapsulate the tree walk you've mentioned.

Even if there isn't an explicit API for this, it should still be possible to implement outside of HCL in a similar way to what you mentioned, by relying on some of the quirks of the JSON syntax to do something similar to what Expression.Variables does internally for JSON:

• Use json.IsJSONExpression to determine if a particular expression is from the JSON syntax. If not, the remaining steps are irrelevant.
• Use Expression.Value with a nil EvalContext, which will therefore interpret the JSON value as a literal without any prior HCL parsing. If the result is a cty.String then it will have raw template sequences embedded in it, because no template parsing or evaluation occurs with a nil evaluation context.
• For any part of the result that is a known value of type cty.String, call hclsyntax.ParseTemplate on it to obtain the native syntax representation of the template inside.
• Analyze the resulting native syntax expression tree as normal.

If Terraform itself ends up needing to do something like this then I will advocate for it to be added as a built-in HCL feature so that applications like yours can use it too, but I also wanted to mention how it's possible to do analysis of JSON expressions already today, just with some (annoying) extra steps, because the JSON syntax is itself really just an extra layer on top of the native syntax expression/template languages.

[wata727]

Thank you for answering my question, @apparentlymart!

Given that, it's possible that a subsequent HCL API change will add a function similar to the current Expression.Variables for finding the functions called in a particular expression using static analysis, which would then encapsulate the tree walk you've mentioned.

This is good news. It exactly meets my requirements.

Also, I appreciate you sharing how I can achieve this with existing functionality. By using these, it seems possible to perform static analysis of provider-contributed functions without any problems.

Closing this issue due to resolve my question. Thank you again!