Terraform rejects expired signatures when downloading providers through registry

[josemgom]

Terraform Version

terraform version
Terraform v1.6.0
on linux_amd64

### Terraform Configuration Files

```cat versions.tf
terraform {
  required_providers {
    bigip = {
      source = "f5networks/bigip"
    }
    phpipam = {
      source = "lord-kyron/phpipam"
    }
  }
  required_version = "< 2"
}

Debug Output

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding hashicorp/null versions matching "< 4.0.0"...
- Finding f5networks/bigip versions matching "< 2.0.0"...
- Finding lord-kyron/phpipam versions matching "< 1.4.0"...
- Finding hashicorp/dns versions matching "< 4.0.0"...
- Installing lord-kyron/phpipam v1.3.8...
- Installed lord-kyron/phpipam v1.3.8 (self-signed, key ID AFA614F0D7D39AB0)
- Installing hashicorp/dns v3.3.2...
- Installed hashicorp/dns v3.3.2 (signed by HashiCorp)
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)
- Installing f5networks/bigip v1.19.0...

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
╷
│ Error: Failed to install provider
│
│ Error while installing f5networks/bigip v1.19.0: error checking signature: openpgp: key expired
╵

Expected Behavior

Terraform init must work as expected

Actual Behavior

Terraform init fails throwing a key expired error.

Steps to Reproduce

terraform init

Additional Context

No response

References

No response

[Zeneixe]

Same error with the Statuscake provider StatusCakeDev/terraform-provider-statuscake#150

[NyanKiyoshi]

Based on my investigations, this may have been caused by this commit 1fe57d4. It appears that terraform wasn't checking whether or not the PGP signature was expired when using golang.org/x/crypto/openpgp but github.com/ProtonMail/go-crypto may be doing it automatically on-behalf of terraform.

This may have been an accidental breaking change.

We can verify why statuscake mentioned-above is failing (#33984 (comment)):

$ curl https://registry.terraform.io/v1/providers/StatusCakeDev/statuscake/2.2.0/download/linux/amd64 | jq -r '.signing_keys.gpg_public_keys[0].ascii_armor' | gpg --show-keys -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4851    0  4851    0     0   2944      0 --:--:--  0:00:01 --:--:--  2948
pub   rsa4096 2021-01-28 [SC] [expired: 2022-01-28]
      A4FFCEB73654749457FCA6684E916C2959C4EAA9
uid                      StatusCake <[email protected]>
sub   rsa4096 2021-01-28 [E] [expired: 2022-01-28]

GPG says the key was expired on 2022-01-28 thus the maintainer would need to renew their key. Same issue appears here: cyrilgdn/terraform-provider-postgresql#357 (comment)

There is also this point at ProtonMail/go-crypto#21 (comment) that could relate to this issue:

Here's a current list of changes:

• [...]
• Detect expired signatures; though leaving it up to the caller whether they want to reject them

[apparentlymart]

Thanks for sharing that, @NyanKiyoshi. I agree that seems like a plausible explanation.

I think then we will have a decision to make about whether not having been checking those expirations before qualifies as a (retroactively-discovered) security problem that is now fixed per the security exception in our 1.x compatibility promises. To decide that I think we'll need to evaluate the risk tradeoff of continuing to allow expired keys.

If the decision is that we want to continue allowing expired keys for some transitional period to give time for provider developers to renew their keys then it seems like (from what you shared) it should be possible to do that.

Another possibility is for Terraform to check whether the key was expired at the time the provider was published, rather than at the time the provider is being installed. That seems the most pragmatic to me since these signatures are a point-in-time assertion made by the author when publishing and not something we expected folks to have to constantly redo. However, I'm not sure off the top of my head whether the provider registry protocol provides enough metadata to know when a provider was published, and indeed it might be wrong for Terraform CLI to trust the registry about when the publication happened.

I think to decide what's best here we'll need to seek advice from our product security team. We'll post more updates once we have a clearer idea of what the situation is and what options are available.

Thanks again!

[marcispauls]

by security best practices - the software (in this case terraform provider) developer should take care of his code including dependencies and including to keep them up2date. if some of software package dependencies are not kept up2date it means the software vendor should replace the dependency with safer and with dependency which follows security best practices.

[adeturner]

Workaround use 1.5.7

- name: "Setup Terraform"
  uses: hashicorp/setup-terraform@v2
  with:
    terraform_version: 1.5.7

terraform {
  required_version = "= 1.5.7"
  required_providers {
    ....
  }
}

[invidian]

Also, it seems Terraform Registry does not actually allow updating or removing GPG keys, so I assume, even if signing keys are refreshed, they will still not reach end users?

[lra]

FTR, the statuscake provider is now fixed: StatusCakeDev/terraform-provider-statuscake#150

[nikolay]

@apparentlymart You own the registry. You knew how many providers would break. Yet, you released a version that broke the environments of all those who venture into using your latest product. This is not a good attitude towards paying customers if you don't care about the rest.

You can also display in the registry warnings about expired keys! You must revert and make this opt-in, not twist-arm.

[nikolay]

See how this should be handled in a dialog with the community: opentofu/opentofu#673

[crw]

Hi @nikolay, thanks for this feedback. Please be mindful of the community guidelines when posting, in particular:

Be considerate.
Be respectful.
Be professional.

A bit of history that may not be clear from the above conversation. This change is an unintended consequence of migrating from the deprecated golang.org/x/crypto/openpgp to the maintained github.com/ProtonMail/go-crypto/openpgp in #33406. This change was advocated by @rolandshoemaker in #33131 (comment) as well as by @MasonM in an earlier PR (#32056). The team was slow to make this change, as we are generally risk-averse when it comes to updating dependencies. This was meant to be a transparent change. In this case, per this issue, there was the unintended consequence of the more-strict checking of PGP signature expiration.

With regards to remediation: the currently proposed PR (#34004) would take the following steps:

1. Revert to the previous behavior (without reverting the new opengpg dependency)
2. Warn the user that a signature is expired (this is a new warning)

As always, any PR that is not merged and not yet included in a released version is not guaranteed to be the final word on an eventual solution.

Apologies in advance if I made a mistake in any of the details above, I will correct as the issue progresses.

[scottmchargue-eb]

@josemgom , @apparentlymart , @crw : Still receiving errors for Error while installing confluentinc/confluent v1.54.0: error checking signature: openpgp: key expired

[crw]

@scottmchargue-eb the next release would have any changes merged after 1.6.0. I would expect this next release, 1.6.1, to happen next week. You will continue to see these errors in 1.6.0. In the meantime, the workaround is to revert to using a 1.5.x version release, latest on that branch is 1.5.7.

[dghubble]

As a provider maintainer, it's still unclear to me whether there is a signing problem I need to address (poseidon/ct) or whether I should instruct all the folks asking about this to wait for Terraform v1.6.1. We have an up-to-date public key published to Ubuntu Keyservers, which should sync to a keyserver Terraform uses.

Things that would help provider maintainers:

• Terraform CLI providers subcommand to show signing keys public key (it's only discoverable on first install)
• Terraform Registry shows a warning to maintainers (I'm still not sure whether Terraform is cool with our public key or not)
• Documentation on Terraform's upstream keyserver and details about how signatures are checked (e.g. pubkey is valid now, pubkey was valid at signing, etc to @apparentlymart's points)
• Proactive warnings when there are changes to provider signing requirements

This stuff matters because a single provider being inadvertently considered expired blocks entire workspaces (and often companies)

[jbardin]

Hi @dghubble,

Sorry if it was unclear, there is no change, or plan to change the current release signing workflow for providers. This was an unanticipated side-effect from an upstream dependency change and will be patched in v1.6.1. It's possible that there could be future enhancements to key management in the registry, but that would be done outside of Terraform. When that time comes Terraform could participate in communicating the changes of course, though that is more often a last resort, since for the majority of providers the end users are not the one who can directly make changes to the providers.

[camprr]

I have also come across this problem for the Algolia provider (https://github.com/philippe-vandermoere/terraform-provider-algolia) it seems to use the keys from the user to determine this?: https://github.com/philippe-vandermoere.gpg
In this case there are two keys, one of which is valid, the other not (email address changed for privacy reasons):

pub   rsa4096 2020-08-24 [SC] [expired: 2022-08-24]
      107EA55DF1555EA4081CDE6C16BFE72B4C53A795
uid           [ expired] Philippe VANDERMOERE <[email protected]>

pub   rsa3072 2022-07-11 [SC]
      3529C14492874491B5CF501E8A87738F1BF20F68
uid           [ unknown] Philippe VANDERMOERE <[email protected]>
sub   rsa3072 2022-07-11 [E]

Downgrading to 1.5.7 solved the problem.

It seems that Terraform is finding the expired key and stopping, it does not 'see' the second, valid key, or maybe the 'unknown' status is causing problems?

[josemgom]

I have also come across this problem for the Algolia provider (https://github.com/philippe-vandermoere/terraform-provider-algolia) it seems to use the keys from the user to determine this?: https://github.com/philippe-vandermoere.gpg In this case there are two keys, one of which is valid, the other not (email address changed for privacy reasons):

pub   rsa4096 2020-08-24 [SC] [expired: 2022-08-24]
      107EA55DF1555EA4081CDE6C16BFE72B4C53A795
uid           [ expired] Philippe VANDERMOERE <[email protected]>

pub   rsa3072 2022-07-11 [SC]
      3529C14492874491B5CF501E8A87738F1BF20F68
uid           [ unknown] Philippe VANDERMOERE <[email protected]>
sub   rsa3072 2022-07-11 [E]

Downgrading to 1.5.7 solved the problem.

It seems that Terraform is finding the expired key and stopping, it does not 'see' the second, valid key, or maybe the 'unknown' status is causing problems?

Please note that Terraform 1.6.1 was released couple of days ago an you should be able to use that version without facing that error.

[camprr]

Sorry for polluting the thread, not 'aware' that v1.6.1 was released. Doh.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.