Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Authentication signature from unknown issue when installing registry.opentofu.org/datadog/datadog v3.39.0 #3179

Closed
1 of 2 tasks
sombrerosheep opened this issue Jun 3, 2024 · 9 comments
Assignees
Labels
bug Something isn't working

Comments

@sombrerosheep
Copy link

Describe the bug

We have an application deployment using terragrunt which changed some providers to use the open-tofu registry; I'm not too thrilled that this is now the default especially with existing statefiles and may represent some confusion below as to what version we expected versus what we used. It seems this update happened a while ago but stopped working mid-last week. No updates to terragrunt or our terraform configuration had been made to provoke this; it seemed to have surfaced out of nowhere. The pull-request this was first seen in was for a NodeJS dependency update.

The full error:


│ Error: Failed to install provider

│ Error while installing registry.opentofu.org/datadog/datadog v3.39.0:
│ authentication signature from unknown issuer

Per the referenced open-tofu issue, we were able to resolve this by manually migrating the provide back to the one located at registry.terraform.io using the tofu state replace-provider... command. Attempting to use terragrunt or terraform would result in errors fetching the provider with the error below.

Steps To Reproduce

terraform {
  required_version = ">= 1.7.1"

  required_providers {
    datadog = {
      source = "DataDog/datadog"
    }
  }
}

Expected behavior

A clear and concise description of what you expected to happen.

We had an existing state (from terragrunt). While performing dependency updates for application maintenance, our terraform-plan began to fail with the error: Error while installing registry.opentofu.org/datadog/datadog v3.39.0: authentication signature from unknown issuer

Nice to haves

  • Terminal output
CI output
  • cd terraform
  • terragrunt init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
Downloading <redacted_internal_module> for <redacted_internal_module>...

  • <redacted_internal_module> in <redacted_internal_module>

Initializing provider plugins...

  • Reusing previous version of hashicorp/null from the dependency lock file
  • Reusing previous version of hashicorp/archive from the dependency lock file
  • Finding latest version of registry.opentofu.org/hashicorp/archive...
  • Finding latest version of registry.opentofu.org/hashicorp/aws...
  • Reusing previous version of hashicorp/time from the dependency lock file
  • Reusing previous version of hashicorp/aws from the dependency lock file
  • Reusing previous version of datadog/datadog from the dependency lock file
  • Finding latest version of registry.opentofu.org/hashicorp/local...
  • Finding latest version of registry.opentofu.org/hashicorp/time...
  • Reusing previous version of hashicorp/local from the dependency lock file
  • Finding latest version of registry.opentofu.org/hashicorp/null...
  • Finding latest version of registry.opentofu.org/datadog/datadog...
  • Installing registry.opentofu.org/hashicorp/archive v2.4.2...
  • Installed registry.opentofu.org/hashicorp/archive v2.4.2 (self-signed, key ID 0C0AF313E5FD9F80)
  • Installing hashicorp/time v0.11.1...
  • Installed hashicorp/time v0.11.1 (signed by HashiCorp)
  • Installing datadog/datadog v3.39.0...
  • Installed datadog/datadog v3.39.0 (signed by a HashiCorp partner, key ID FB70BE941301C3EA)
  • Installing registry.opentofu.org/hashicorp/time v0.11.2...
  • Installed registry.opentofu.org/hashicorp/time v0.11.2 (self-signed, key ID 0C0AF313E5FD9F80)
  • Installing hashicorp/archive v2.4.2...
  • Installed hashicorp/archive v2.4.2 (signed by HashiCorp)
  • Installing registry.opentofu.org/hashicorp/aws v5.51.1...
  • Installed registry.opentofu.org/hashicorp/aws v5.51.1 (self-signed, key ID 0C0AF313E5FD9F80)
  • Installing hashicorp/aws v5.51.1...
  • Installed hashicorp/aws v5.51.1 (signed by HashiCorp)
  • Installing registry.opentofu.org/hashicorp/local v2.5.1...
  • Installed registry.opentofu.org/hashicorp/local v2.5.1 (self-signed, key ID 0C0AF313E5FD9F80)
  • Installing hashicorp/local v2.5.1...
  • Installed hashicorp/local v2.5.1 (signed by HashiCorp)
  • Installing registry.opentofu.org/hashicorp/null v3.2.2...
  • Installed registry.opentofu.org/hashicorp/null v3.2.2 (self-signed, key ID 0C0AF313E5FD9F80)
  • Installing registry.opentofu.org/datadog/datadog v3.39.0...
  • Installing hashicorp/null v3.2.2...
  • Installed hashicorp/null v3.2.2 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

│ Error: Failed to install provider

│ Error while installing registry.opentofu.org/datadog/datadog v3.39.0:
│ authentication signature from unknown issuer

time=2024-05-30T12:16:37Z level=error msg=terraform invocation failed in /drone/src/terraform
time=2024-05-30T12:16:37Z level=error msg=1 error occurred:

  • [/drone/src/terraform] exit status 1
  • Screenshots

Versions

  • Terragrunt version: 0.58.13 per alpine/terragrunt
  • OpenTofu/Terraform version: We thought terraform 1.8.4 but according to terragrunt 0.58.13, tofu 1.7.x?
  • Environment details (Ubuntu 20.04, Windows 10, etc.): Docker Image alpine/terragrunt:1.8.4

Additional context

Carrying over from opentofu/registry#630 as recommended by the Open-Tofu team. Tagging @ZachGoldberg as requested!

@sombrerosheep sombrerosheep added the bug Something isn't working label Jun 3, 2024
@ZachGoldberg
Copy link
Contributor

Thanks for filing @sombrerosheep! The team is going to take a look at this shortly.

@yhakbar yhakbar self-assigned this Jun 3, 2024
@yhakbar
Copy link
Collaborator

yhakbar commented Jun 3, 2024

Hey @sombrerosheep ,

I'm sorry that was your experience when upgrading to a new version of Terragrunt. The error that you got did not clearly indicate what was happening, and we can take further action to make it easier to understand why this happened.

The reason you received that error is that the authentication signature you currently have in your .terraform.lock.hcl file does not line up with the authentication signature for registry.opentofu.org/datadog/datadog (as the authentication signature was generated for the registry.terraform.io/datadog/datadog provider and the registry.opentofu.org registry does not contain GPG keys for the provider). Although they are effectively the same binary built with the same source code, for licensing reasons, OpenTofu does not fetch from registry.terraform.io.

As of version v0.57.12 OpenTofu is the default tool Terragrunt uses when both Terraform and OpenTofu are installed on a system.

To explicitly decide to have Terragrunt use Terraform going forward, you'll have to set terragrunt-tfpath explicitly.

The simplest way to do that is with the environment variable TERRAGRUNT_TFPATH like so:

TERRAGRUNT_TFPATH='terraform' terragrunt plan

If you would like to fix your .terraform.lock.hcl file so that it will still work with OpenTofu, you can run the following:

terragrunt init -upgrade

That will update the entries in your .terraform.lock.hcl file to align with the values corresponding to those of registry.opentofu.org instead of registry.terraform.io. With that, you won't get the error from OpenTofu on the signature mismatch.

@sombrerosheep
Copy link
Author

@yhakbar We are starting to see this on more repositories so I'd like to clarify the best way to resolve this issue and get things stable using terraform again.

Should removing .terraform.lock.hcl and running a terragrunt init -upgrade resolve the provider signature issue?

Having never had this issue when using terraform, you can understand why were now cautious letting terragrunt use open-tofu. Once we are past the provider signature issue, if we wanted everything pointed back at terraform, we would need to:

  • terragrunt state replace-provider ... for each provider (in each environment).
  • set TERRAGRUNT_TFPATH to terraform
  • set TERRAGRUNT_PROVIDER_CACHE_REGISTRY_NAMES to registry.terraform.io

Or would the two variables + a terragrunt init -upgrade update the state providers too?

@sombrerosheep
Copy link
Author

@yhakbar we also have another repository with this error. They are not committing a .terraform.lock.hcl file at all. I don't believe the state tracks that kind of information so it would appear there may be an issue with registry.opentofu.org?

@yhakbar
Copy link
Collaborator

yhakbar commented Jun 6, 2024

You should not need to set TERRAGRUNT_PROVIDER_CACHE_REGISTRY_NAMES unless you are using the provider cache. Are you leveraging provider caching?

Either of:

  • Setting TERRAGRUNT_TFPATH to terraform
  • Not having OpenTofu installed

Will ensure that Terragrunt uses Terraform instead of OpenTofu.

In the repositories that are committing their lock files, as long as you haven't modified your lock files, you do not have to use terragrunt init -upgrade. You only need to do that if you're actively trying to change your .terraform.lock.hcl file so that it's compatible with the OpenTofu provider while using OpenTofu to back Terragrunt.

If your state file is now tracking registry.opentofu.org/datadog/datadog, you'll want to run the following, keeping the instructions above for ensuring that Terraform is backing Terragrunt instead of OpenTofu in mind:

 terragrunt state replace-provider 'registry.opentofu.org/datadog/datadog' 'registry.terraform.io/datadog/datadog'

@sombrerosheep
Copy link
Author

Thanks for the info! We are not using provider caching.

When we run terragrunt state replace-provider ... this error is presented. We are unable to run terragrunt init either due to the same error.

We had a successful run on June 3, and on June 4 started seeing this. The state had not been updated since May 26, 2024 which was bumping alpine/terragrunt:1.8.3 to tag 1.8.4. This change bumped terraform from 1.7.1 to 1.8.4 and changed all providers to use registry.opentofu.org.

From May 26 until this began failing on June 3, we had 3 successful deployments; all involving application code, not infrastructure.

Is there anything else we can do to get our application back to a stable state?

@sombrerosheep
Copy link
Author

sombrerosheep commented Jun 6, 2024

@yhakbar We were finally able to get past this using the --terragrunt-no-auto-init with the replace-provider command.

Now my question is whether or not we need to worry about this happening again? Is there any control we have for making sure that terraform and its registries are used?

@yhakbar
Copy link
Collaborator

yhakbar commented Jun 6, 2024

@sombrerosheep

What might help you get to a stable state is to go to an empty directory and experiment with switching back and forth between Terraform and OpenTofu to see what it does to your lock file, and your state file.

# ./empty-dir/terragrunt.hcl
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  required_providers {
    datadog = {
      source = "DataDog/datadog"
    }
  }
}
EOF

}

If you have both OpenTofu and Terragrunt installed on your system, and you run the following:

TERRAGRUNT_TFPATH='tofu' terragrunt plan

You'll see the following in your .terraform.lock.hcl file:

# This file is maintained automatically by "tofu init".
# Manual edits may be lost in future updates.

provider "registry.opentofu.org/datadog/datadog" {
  version = "3.39.0"
  hashes = [
    # Some hashes
  ]
}

After that, run the following to see the error that you get:

$ TERRAGRUNT_TFPATH='terraform' terragrunt plan
╷
│ Error: Inconsistent dependency lock file
│
│ The following dependency selections recorded in the lock file are
│ inconsistent with the current configuration:
│   - provider registry.terraform.io/datadog/datadog: required by this configuration but no version is selected
│
│ To update the locked dependency selections to match a changed
│ configuration, run:
│   terraform init -upgrade
╵

Not an insurmountable problem! You can then run the following to explicitly change the provider listed in your lock file:

$ TERRAGRUNT_TFPATH='terraform' terragrunt init -upgrade

You can see how your lock file changes as a consequence:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/datadog/datadog" {
  version = "3.39.0"
  hashes = [
    # Some hashes
  ]
}

Now, I don't have a DataDog account, so I can't test out a demo of the same thing, but for state, but I do have access to an AWS account, so I can show you that.

# ./other-empty-dir/terragrunt.hcl
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}
EOF

}

generate "sts_get_caller_identity" {
  path      = "main.tf"
  if_exists = "overwrite"
  contents  = <<EOF
data "aws_caller_identity" "current" {}
EOF
}

Like before, run a terragrunt apply:

$ TERRAGRUNT_TFPATH='tofu' terragrunt apply

Notice what the .terraform.lock file and the terraform.tfstate files look like:

# .terraform.lock.hcl
# This file is maintained automatically by "tofu init".
# Manual edits may be lost in future updates.

provider "registry.opentofu.org/hashicorp/aws" {
  version = "5.52.0"
  hashes = [
    # Some hashes
  ]
}
# cat terraform.tfstate
{
  "version": 4,
  "terraform_version": "1.7.1",
  "serial": 5,
  "lineage": "A UUID",
  "outputs": {},
  "resources": [
    {
      "mode": "data",
      "type": "aws_caller_identity",
      "name": "current",
      "provider": "provider[\"registry.opentofu.org/hashicorp/aws\"]",
      "instances": [
        # Some instances
      ]
    }
  ],
  "check_results": null
}

Similarly, trigger an error by switching to Terraform:

$ TERRAGRUNT_TFPATH='terraform' terragrunt apply
╷
│ Error: Inconsistent dependency lock file
│
│ The following dependency selections recorded in the lock file are
│ inconsistent with the current configuration:
│   - provider registry.terraform.io/hashicorp/aws: required by this configuration but no version is selected
│
│ To update the locked dependency selections to match a changed
│ configuration, run:
│   terraform init -upgrade
╵

You'll notice up above, that the provider field in the state file included registry.opentofu.org/hashicorp/aws. Let's fix that by running that replace command:

$ TERRAGRUNT_TFPATH='terraform' terragrunt state replace-provider 'registry.opentofu.org/hashicorp/aws' 'registry.terraform.io/hashicorp/aws'
Terraform will perform the following actions:

  ~ Updating provider:
    - registry.opentofu.org/hashicorp/aws
    + registry.terraform.io/hashicorp/aws

Changing 1 resources:

  data.aws_caller_identity.current

Do you want to make these changes?
Only 'yes' will be accepted to continue.

Enter a value: yes

Successfully replaced provider for 1 resources.

You'll also notice that the .terraform.lock.hcl file has the provider block set to registry.opentofu.org/hashicorp/aws. Let's fix that by running the following:

$ TERRAGRUNT_TFPATH='terraform' terragrunt init -upgrade

Take a look at your terraform.tfstate file and your .terraform.lock.hcl file and see how they change how you would expect, having the registry.opentofu.org/hashicorp/aws entries replaced with registry.terraform.io/hashicorp/aws.

If you want, you can even go the other way to prove to yourself that you can flip back and forth.

After doing that experimentation, take a look at the lock and state files for the resources that are throwing errors. Are any of them in a configuration you don't expect?

You use the same techniques you used here to repair them, and ensure that TERRAGRUNT_TFPATH is set such that you are explicit about whether you're using OpenTofu or Terraform.

@yhakbar
Copy link
Collaborator

yhakbar commented Jun 6, 2024

@sombrerosheep

Now my question is whether or not we need to worry about this happening again? Is there any control we have for making sure that terraform and its registries are used?

In addition to setting the environment variable I show above, and not having OpenTofu installed, you can also set the terraform_binary value in your terragrunt.hcl files so that you're defining as part of your configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants