Add support for Terraform 0.12's Rich Value Types

[metmajer]

Prerequisites

Put an x into the box that applies:

• This issue describes a bug.
• This issue describes a feature request.

For more information, see the Contributing Guidelines.

Description

With the upcoming v0.12, Terraform will introduce the concept of Rich Value Types. These rich value types will introduce breaking changes to terraform-docs.

[metmajer]

This is going to be a beast. Interested in a challenge @khos2ow? :-)

[khos2ow]

Yes, let's do this, I can make some time in this madness of schedule I'm in!

[Acen]

FYI Terraform has now been updated to 0.12. https://www.terraform.io/upgrade-guides/0-12.html

[metmajer]

Yes, we're aware.

[antonbabenko]

While there is no native support for Terraform 0.12 in terraform-docs, there is an alternative solution which involves awk script (kudos to @osterman and @cytopia for making this).

I have updated pre-commit-terraform to v1.16.0 where this is implemented.

See this pull request to terraform-aws-vpc module for example.

[metmajer]

I am all with you: the missing support for Terraform 0.12 is temporary. However, if we just make it work somehow by throwing away existing functionality, the solution may work for some, but not for many others. In this case, we're not talking about an easy transition and HashiCorp has already warned us about using Terraform 0.12's public APIs which may change without warning. Integrating with HCL 2.0 directly would be the official way, but is far from straight forward.

Please also keep in mind that, while we do our best giving this project the ongoing support it needs, we do this in our spare time. At the moment, we are deeply involved in projects to make our living. Again, this doesn't mean that we don't hear you.

In the meantime, please see @antonbabenko's suggestion above. If he, as a maintainer of many Terraform community modules and a long-time user of terraform-docs can live with this temporary solution, you will hopefully too for the time being.

#113

[metmajer]

@moatra has brought up the idea of using the hashicorp/terraform-config-inspect module in #113. The nice thing about this approach is that it provides an abstraction over HCL and HCL2 parsing of Terraform modules and could become an ideal integration layer for us if we can address the following, current shortcomings:

• Variables are not parsed in the order they are defined. This is important as some users use terraform-docs to automatically populate forms that trigger the deployment of a Terraform module.
• Lead comments for a module, as well as comments for variables and outputs are not supported. While not a widely adopted feature, we'll need this to be backwards compatible.
• The fact that the library only supports the parsing of a single module is not an issue and is in line with the mission of terraform-docs (although multi-module support has historically been supported in some places, this should be avoided).

I'll investigate and prepare feature requests for this library as needed. Let me know if you want to join me @khos2ow @moatra @sjauld @antonbabenko ?

[sjauld]

I'm happy to take a ticket or two on this one.

…

On Mon, 1 Jul 2019 at 07:33, Martin Etmajer ***@***.***> wrote: @moatra <https://github.com/moatra> has brought up the idea of using the hashicorp/terraform-config-inspect <https://github.com/hashicorp/terraform-config-inspect> module in #113 <#113>. The nice thing about this approach is that it provides an abstraction over HCL and HCL2 parsing of Terraform modules and could become an ideal integration layer for us if we can address the following, current shortcomings: - Variables are not parsed in the order they are defined. This is important as some users use terraform-docs to automatically populate forms that trigger the deployment of a Terraform module. - Lead comments for a module, as well as comments for variables and outputs are not supported. While not a widely adopted feature, we'll need this to be backwards compatible. - The fact that the library only supports the parsing of a single module is not an issue and is in line with the mission of terraform-docs (although multi-module support has historically been supported in some places, this should be avoided). I'll investigate and prepare pull requests for this library as needed. Let me know if you want to join @khos2ow <https://github.com/khos2ow> @moatra <https://github.com/moatra> @sjauld <https://github.com/sjauld> @antonbabenko <https://github.com/antonbabenko> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB6Z4N3KALC4OE5I2PKNESDP5EREFANCNFSM4FWW5N5Q> .

[moatra]

I've submitted a few PRs to the terraform-config-inspect module. While digging around, it looks like the module exposes SourcePos fields with filename and line number on resource, data, variable, output, and module blocks.

Variables are not parsed in the order they are defined.

It should be possible to use line numbers to do a manual sort. What's the current behavior for variables defined in different files, though?

Lead comments for a module, as well as comments for variables and outputs are not supported.

It probably won't be too onerous to add a second parsing pass looking for comments above their positions.

If you're okay with maintaining a custom parser for comments, then I don't think any upstream changes are necessary.

[CarpathianUA]

+1! Waiting for 12 support.

[eredi93]

Same

[szymon3]

Guys, check this for TF 0.12.x Docs: https://hub.docker.com/r/cytopia/terraform-docs

[cytopia]

Guys, check this for TF 0.12.x Docs: https://hub.docker.com/r/cytopia/terraform-docs

Dockerized version

The GitHub repository for that mentioned Docker image image can be found here: https://github.com/cytopia/docker-terraform-docs

What it does, is to internally modify *.tf files of Terraform >= 0.12.x to a syntax that works with Terraform <= 0.11.x and simply runs terraform-docs over it.

The magic happens by this awk script: https://github.com/cytopia/docker-terraform-docs/blob/master/data/terraform-docs.awk

---

Two most well-known projects

There are two other repositories available I know of where I submit the latest changes of the awk script to:

• terraform-docs should not fail if complex types contain 'description' keyword cloudposse/build-harness#174 (terraform-docs.awk originally developed here)
• terraform_docs failing on complex types which contains "description"  antonbabenko/pre-commit-terraform#65 (mentioned above in this issue: Add support for Terraform 0.12's Rich Value Types #62 (comment))

[eredi93]

the awk hack "works" but is not ideal. for example the usage comment is now being ignored (which i really liked), also complex types are not being added to the docs, it adds object

what we have is better than no docs but i still would like a full on support for this project

[cytopia]

usage comment is now being ignored

@eredi93 can you give an example.

[eredi93]

i have this:

/**
 * ## IAM Role
 *
 * Module usage:
 *
 * ```
 * module "iam_role" {
 *   source = "[email protected]:org/repo.git//modules/iam/role"
 *
 *   name   = "${var.name}"
 *   policy = "${var.policy}"
 *   mfa    = true
 * }
 * ```
 */

on top of my main.tf in my IAM role module
but in my README i have only ## Inputs and ## Outputs

when i use terraform-docs and TF 0.11 i have my intro at the beginning

## IAM Role

Module usage:

```
module "iam_role" {
  source = "[email protected]:intercom/terraform-modules.git//modules/iam/role"

  name   = "${var.name}"
  policy = "${var.policy}"
  mfa    = true
}
```


## Inputs

| Name | Description | Type | Default | Required |

.......

## Outputs

| Name | Description |
|------|-------------|

...........

[rvodden]

@khos2ow do you need a hand? My team are about to embark on a big 0.12 migration and I'm sure that we can give some Dev effort to this, seeing as how much easier it will make documentation. Just let us know what I can do.

[khos2ow]

That's awesome @rvodden . What I was thinking is to finalize some of the outstanding issues and cut a release (v0.7.0) and then merge #113 to the interim branch on the main repo, as HEAD has diverged from that PR and rebasing is gonna be difficult, and then eventually working on multiple v0.8.0-rc versions trying to iterate over that PR to finalize 0.12 support.

[rvodden]

Makes sense. Feels like the most value we can add is to look at getting some of that re-base done ahead of time. I'll have a look tomorrow. Obviously feel free to suggest another direction if you think we could be more helpful in other places :-)

[tbugfinder]

@rvodden @khos2ow
My understanding is that terraform-docs and hcltemplate have the same requirement in regards to getting to the description. Maybe both projects could benefit from enhancing the same library (cty - https://github.com/zclconf/go-cty)?

apparentlymart/hcltemplate#1

[marcuscaisey]

@eredi93

the awk hack "works" but is not ideal. for example the usage comment is now being ignored (which i really liked), also complex types are not being added to the docs, it adds object

what we have is better than no docs but i still would like a full on support for this project

I've written a Python script which wraps terraform-docs and converts the input much like @cytopia awk script does and then runs terraform-docs on those converted files. My conversion of files preserves all lines (so would preserve your usage comments) and also preserves the type information for object types as long as they don't contain any object types themselves.

Any options/arguments you pass to terraform-docs12 will be passed through to the underlying terraform-docs call.

[Aankhen]

@marcuscaisey, out of curiosity, would you be okay with putting something based on that in terraform-docs?

EDIT:
Nevermind, sorry, I just saw the PR for support for 0.12 anyway.

[khos2ow]

FYI to the folks on this thread I've picked up #113 and working on rebasing and fixing the issues, I should be having a workable PR by the end of week and my plan is to potentially cut a release sometime before end of year. Thanks for your patience and support, we will definitely have support for Terraform 0.12 by start of 2020!

[smutel]

Better than the gifts sent by 🎅

[khos2ow]

I just merged #113 🎉, closing this long-standing issue!

I'm gonna cut a RC soon, please watch for the new release and test it out with your configs and report back any potential issues you have found.