Rewrite Validate Command (#341)

* Initial validate implementation

* Add function tests to TestFileMismatchCheck

* Add MixedDirectoryCheck and refactor relevant tests

* Add functions and guides to glob pattern

* Update README.md

* Update documentationDirGlob to pass tests

* Add copyright headers

* Resolve linting errors

* Update README.md with a link to provider documentation guidelines

* Resolve linting errors

* Add changelog entries

* Add `NumberOfFilesCheck` to validate

* Make file paths OS-agnostic

* Fix file path in test error assertion

* Update Go version to `1.21`

* Change wording to past tense on changelog entries

* Remove unused file

* Remove `filecheck` interface

* Refactor `schemamd` and `functionmd` packages to `internal/`

* Remove unused method

* Revert "Refactor `schemamd` and `functionmd` packages to `internal/`"

This reverts commit a1b156e.

* Remove go compatability section

* Switch to `yuin/goldmark` for frontmatter parsing

* Remove unused test

* Move global variables to top of file

* Refactor `testcases` to use `map[string]struct`

* Refactor to use `t.tempDir()`

* Fix documentationGlobPattern and add more debug and info logs

* Add negative scenario acceptance tests

.changes/unreleased/FEATURES-20240305-135426.yaml

@@ -0,0 +1,5 @@
+kind: FEATURES
+body: 'validate: Added support for Provider-defined Function documentation to all checks'
+time: 2024-03-05T13:54:26.307742-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-135726.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `InvalidDirectoriesCheck` which checks for valid provider documentation
+  folder structure'
+time: 2024-03-05T13:57:26.273538-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-135933.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `MixedDirectoriesCheck` which throws an error if both legacy
+  documentation and registry documentation are found'
+time: 2024-03-05T13:59:33.741601-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-140106.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `NumberOfFilesCheck` which checks the number of provider
+  documentation files against the registry limit'
+time: 2024-03-05T14:01:06.742843-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-140234.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `FileSizeCheck` which checks the provider documentation file 
+  size against the registry limit'
+time: 2024-03-05T14:02:34.112782-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-140346.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `FileExtensionCheck` which checks for valid provider documentation
+  file extensions'
+time: 2024-03-05T14:03:46.816256-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-140451.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `FrontMatterCheck` which checks the YAML frontmatter of provider
+  documentation for missing required fields or invalid fields'
+time: 2024-03-05T14:04:51.781688-05:00
+custom:
+  Issue: "341"

.changes/unreleased/FEATURES-20240305-140622.yaml

@@ -0,0 +1,6 @@
+kind: FEATURES
+body: 'validate: Added `FileMismatchCheck` which checks the names/number of provider
+  documentation files against the provider schema'
+time: 2024-03-05T14:06:22.168518-05:00
+custom:
+  Issue: "341"

README.md

@@ -52,7 +52,7 @@ Available commands are:
                 the generate command is run by default
     generate    generates a plugin website from code, templates, and examples
     migrate     migrates website files from either the legacy rendered website directory (`website/docs/r`) or the docs rendered website directory (`docs/resources`) to the tfplugindocs supported structure (`templates/`).
-    validate    validates a plugin website for the current directory
+    validate    validates a plugin website
        
 ```
 
@@ -81,6 +81,11 @@ Usage: tfplugindocs generate [<args>]
 $ tfplugindocs validate --help
 
 Usage: tfplugindocs validate [<args>]
+
+    --provider-dir <ARG>       relative or absolute path to the root provider code directory; this will default to the current working directory if not set                                                              
+    --provider-name <ARG>      provider name, as used in Terraform configurations                                                                                                                                                
+    --providers-schema <ARG>   path to the providers schema JSON file, which contains the output of the terraform providers schema -json command. Setting this flag will skip building the provider and calling Terraform CLI    
+    --tf-version <ARG>         terraform binary version to download. If not provided, will look for a terraform binary in the local environment. If not found in the environment, will download the latest version of Terraform  
 ```
 
 `migrate` command:
@@ -145,6 +150,22 @@ Otherwise, the provider developer can set an arbitrary description like this:
     // ...
 ```
 
+#### Validate subcommand
+
+The `validate` subcommand can be used to validate the provider website documentation against the [Terraform Registry's provider documentation guidelines](https://developer.hashicorp.com/terraform/registry/providers/docs) and provider documentation best practices. The current checks in the `validate` command are:
+
+| Check                     | Description                                                                                                                                                                         |
+|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `InvalidDirectoriesCheck` | Checks for valid subdirectory structure and throws an error if an invalid Terraform Provider documentation subdirectory is found.                                                   |
+| `MixedDirectoriesCheck`   | Throws an error if both legacy documentation (`/website/docs`) and registry documentation (`/docs`) are found.                                                                      |
+| `NumberOfFilesCheck`      | Throws an error if the number of files in a directory is larger than the registry limit.                                                                                            |
+| `FileSizeCheck`           | Throws an error if the documentation file is above the registry storage limit.                                                                                                      |
+| `FileExtensionCheck`      | Throws an error if the extension of the given file is not a valid registry documentation extension.                                                                                 |
+| `FrontMatterCheck`        | Checks the YAML frontmatter of documentation for missing required fields or invalid fields.                                                                                         |
+| `FileMismatchCheck`       | Throws an error if the names/number of resources/datasources/functions in the provider schema does not match the names/number of files in the corresponding documentation directory |
+
+All check errors are wrapped and returned as a single error message to stderr.
+
 #### Migrate subcommand
 
 The `migrate` subcommand can be used to migrate website files from either the legacy rendered website directory (`website/docs/r`) or the docs 

cmd/tfplugindocs/main_test.go

@@ -54,3 +54,11 @@ func Test_SchemaJson_MigrateAcceptanceTests(t *testing.T) {
 		Dir: "testdata/scripts/schema-json/migrate",
 	})
 }
+
+func Test_SchemaJson_ValidateAcceptanceTests(t *testing.T) {
+	t.Parallel()
+
+	testscript.Run(t, testscript.Params{
+		Dir: "testdata/scripts/schema-json/validate",
+	})
+}

cmd/tfplugindocs/testdata/scripts/schema-json/validate/framework_provider_error_file_mismatch.txtar

@@ -0,0 +1,127 @@
+# Copyright (c) HashiCorp, Inc.
+# SPDX-License-Identifier: MPL-2.0
+
+# Run of tfplugindocs validate command with a misnamed file
+[!unix] skip
+! exec tfplugindocs validate --provider-name=terraform-provider-scaffolding --providers-schema=schema.json
+stderr 'Error executing command: validation errors found:'
+stderr 'matching resource for documentation file \(resource2.md\) not found, file is extraneous or incorrectly named'
+stderr 'missing documentation file for resource: scaffolding_example'
+
+-- docs/data-sources/example.md --
+---
+subcategory: "Example"
+page_title: "Example: example_thing"
+description: |-
+  Example description.
+---
+# Data Fields
+
+Name: {{.Name}}
+Type: {{.Type}}
+-- docs/resources/resource2.md --
+---
+subcategory: "Example"
+page_title: "Example: example_thing"
+description: |-
+  Example description.
+---
+# Data Fields
+
+Name: {{.Name}}
+Type: {{.Type}}
+-- schema.json --
+{
+  "format_version": "1.0",
+  "provider_schemas": {
+    "registry.terraform.io/hashicorp/scaffolding": {
+      "provider": {
+        "version": 0,
+        "block": {
+          "attributes": {
+            "endpoint": {
+              "type": "string",
+              "description": "Example provider attribute",
+              "description_kind": "markdown",
+              "optional": true
+            }
+          },
+          "description": "Example provider",
+          "description_kind": "markdown"
+        }
+      },
+      "resource_schemas": {
+        "scaffolding_example": {
+          "version": 0,
+          "block": {
+            "attributes": {
+              "configurable_attribute": {
+                "type": "string",
+                "description": "Example configurable attribute",
+                "description_kind": "markdown",
+                "optional": true
+              },
+              "defaulted": {
+                "type": "string",
+                "description": "Example configurable attribute with default value",
+                "description_kind": "markdown",
+                "optional": true,
+                "computed": true
+              },
+              "id": {
+                "type": "string",
+                "description": "Example identifier",
+                "description_kind": "markdown",
+                "computed": true
+              }
+            },
+            "description": "Example resource",
+            "description_kind": "markdown"
+          }
+        }
+      },
+      "data_source_schemas": {
+        "scaffolding_example": {
+          "version": 0,
+          "block": {
+            "attributes": {
+              "configurable_attribute": {
+                "type": "string",
+                "description": "Example configurable attribute",
+                "description_kind": "markdown",
+                "optional": true
+              },
+              "id": {
+                "type": "string",
+                "description": "Example identifier",
+                "description_kind": "markdown",
+                "computed": true
+              }
+            },
+            "description": "Example data source",
+            "description_kind": "markdown"
+          }
+        }
+      },
+      "functions": {
+        "example": {
+          "description": "Given a string value, returns the same value.",
+          "summary": "Echo a string",
+          "return_type": "string",
+          "parameters": [
+            {
+              "name": "input",
+              "description": "Value to echo.",
+              "type": "string"
+            }
+          ],
+          "variadic_parameter": {
+            "name": "variadicInput",
+            "description": "Variadic input to echo.",
+            "type": "string"
+          }
+        }
+      }
+    }
+  }
+}

cmd/tfplugindocs/testdata/scripts/schema-json/validate/framework_provider_error_mixed_directories.txtar

@@ -0,0 +1,127 @@
+# Copyright (c) HashiCorp, Inc.
+# SPDX-License-Identifier: MPL-2.0
+
+# Run of tfplugindocs validate command with mixed directory structure
+[!unix] skip
+! exec tfplugindocs validate --provider-name=terraform-provider-scaffolding --providers-schema=schema.json
+stderr 'Error executing command: validation errors found:'
+stderr 'mixed Terraform Provider documentation directory layouts found, must use only legacy or registry layout'
+
+-- website/docs/d/example.html.md --
+---
+subcategory: "Example"
+layout: "example"
+page_title: "Example: example_thing"
+description: |-
+  Example description.
+---
+# Data Fields
+
+Name: {{.Name}}
+Type: {{.Type}}
+-- docs/data-sources/example.md --
+---
+subcategory: "Example"
+page_title: "Example: example_thing"
+description: |-
+  Example description.
+---
+# Data Fields
+
+Name: {{.Name}}
+Type: {{.Type}}
+-- schema.json --
+{
+  "format_version": "1.0",
+  "provider_schemas": {
+    "registry.terraform.io/hashicorp/scaffolding": {
+      "provider": {
+        "version": 0,
+        "block": {
+          "attributes": {
+            "endpoint": {
+              "type": "string",
+              "description": "Example provider attribute",
+              "description_kind": "markdown",
+              "optional": true
+            }
+          },
+          "description": "Example provider",
+          "description_kind": "markdown"
+        }
+      },
+      "resource_schemas": {
+        "scaffolding_example": {
+          "version": 0,
+          "block": {
+            "attributes": {
+              "configurable_attribute": {
+                "type": "string",
+                "description": "Example configurable attribute",
+                "description_kind": "markdown",
+                "optional": true
+              },
+              "defaulted": {
+                "type": "string",
+                "description": "Example configurable attribute with default value",
+                "description_kind": "markdown",
+                "optional": true,
+                "computed": true
+              },
+              "id": {
+                "type": "string",
+                "description": "Example identifier",
+                "description_kind": "markdown",
+                "computed": true
+              }
+            },
+            "description": "Example resource",
+            "description_kind": "markdown"
+          }
+        }
+      },
+      "data_source_schemas": {
+        "scaffolding_example": {
+          "version": 0,
+          "block": {
+            "attributes": {
+              "configurable_attribute": {
+                "type": "string",
+                "description": "Example configurable attribute",
+                "description_kind": "markdown",
+                "optional": true
+              },
+              "id": {
+                "type": "string",
+                "description": "Example identifier",
+                "description_kind": "markdown",
+                "computed": true
+              }
+            },
+            "description": "Example data source",
+            "description_kind": "markdown"
+          }
+        }
+      },
+      "functions": {
+        "example": {
+          "description": "Given a string value, returns the same value.",
+          "summary": "Echo a string",
+          "return_type": "string",
+          "parameters": [
+            {
+              "name": "input",
+              "description": "Value to echo.",
+              "type": "string"
+            }
+          ],
+          "variadic_parameter": {
+            "name": "variadicInput",
+            "description": "Variadic input to echo.",
+            "type": "string"
+          }
+        }
+      }
+    }
+  }
+}