[Bug]: Terraform apply command freezes during AWS provider initialization #39523

law · 2024-09-27T20:38:43Z

Terraform Core Version

1.5.7

AWS Provider Version

5.69.0

Affected Resource(s)

n/a

Expected Behavior

'terraform apply' should continue, and ask me for confirmation before applying changes

Actual Behavior

When running `terraform apply, the process freezes during the initialization of the AWS provider. The command does not complete and requires manual termination.

Relevant Error/Panic Output Snippet

2024-09-27T13:46:40.034-0600 [DEBUG] provider.terraform-provider-aws_v5.69.0_x5: assertion failed [arm_interval().contains(address)]: code fragment does not contain the given arm address
2024-09-27T13:46:40.034-0600 [DEBUG] provider.terraform-provider-aws_v5.69.0_x5: (CodeFragmentMetadata.cpp:48 instruction_extents_for_arm_address)

Terraform Configuration Files

https://gist.github.com/law/62b9c75214c18a015c37f16285a13ba4

Steps to Reproduce

Run TF_LOG=debug terraform apply (or terragrunt init) when using the above provider
Observe that the .terraform/providers/registry.terraform.io/hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5 process hangs indefinitely

Debug Output

https://gist.github.com/law/5271d0e0cd052d438a194eb50c11da63

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

The text was updated successfully, but these errors were encountered:

github-actions · 2024-09-27T20:38:59Z

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

law · 2024-09-27T20:48:14Z

I forgot to add, reverting to provider 5.68.0 works like a champ.

alexpodr · 2024-09-27T22:06:43Z

I have the same issue with v5.69.0.
And with v5.68.0 - all Ok.

UPD: Chip - Apple M1 Pro

elaigor · 2024-09-30T08:28:36Z

What we found for our team is that only x86 provider did not work as it should. Mac with intel or arm doesn't matter, if you use 5.69.0_x86 (with Rosseta on arm) there will be some kind of problems.

jotasixto · 2024-09-30T14:58:13Z

I have the same issue with v5.69.0(hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5) and chip Apple M3 with Rosetta 2.

JnMik · 2024-09-30T18:42:57Z

Same issue on Apple M3

LozanoMatheus · 2024-09-30T19:53:04Z

It seems to be an issue only with Apple M chips, the version v5.69.0 works fine on my Linux/amd64.

Also, it works fine when I use the same version in the CDKTF. Quite odd.

cailen · 2024-09-30T21:58:58Z

I was trying to figure this out for 3 hours today what it was. Was convinced it was a provider and here we are. Thanks for whoever reported this.

cailen · 2024-10-01T11:22:55Z

Looks like these projects have had similar errors recently:

SST: assertion failed ... code fragment does not contain the given arm address sst/sst#4422
Pulumi: Provider v6.42.0+ crashing on Apple Silicon when running with Rosetta2 pulumi/pulumi-aws#4190

cailen · 2024-10-01T11:28:12Z

It looks like the darwin_arm64 version is not being properly downloaded?

cailen · 2024-10-01T11:44:45Z

This led me down quite the rabbit hole wondering why on Apple M2 I am only getting amd64. It turns out (probably during one of the many upgrades from 0.11/0.12 darwin_arm64-unsupported versions), I may have set the architecture to amd64 and it downloaded not only the old version but the version we upgrade to (1.5.7) in amd64.

% export TFENV_ARCH=arm64

% tfenv install 1.5.7
Terraform v1.5.7 is already installed

% terraform version      
Terraform v1.5.7
on darwin_amd64
+ provider registry.terraform.io/datadog/datadog v3.44.1
+ provider registry.terraform.io/hashicorp/aws v5.68.0
+ provider registry.terraform.io/hashicorp/http v3.4.5
+ provider registry.terraform.io/hashicorp/null v3.2.3
+ provider registry.terraform.io/hashicorp/random v3.6.3
+ provider registry.terraform.io/hashicorp/time v0.12.1
+ provider registry.terraform.io/mongodb/mongodbatlas v1.20.0

Your version of Terraform is out of date! The latest version
is 1.9.5. You can update by downloading from https://www.terraform.io/downloads.html

% tfenv uninstall 1.5.7
Uninstall Terraform v1.5.7
Terraform v1.5.7 is successfully uninstalled

% terraform version    
version '1.5.7' is not installed (set by /repo/.terraform-version). Installing now as TFENV_AUTO_INSTALL==true
Installing Terraform v1.5.7
Downloading release tarball from https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_darwin_arm64.zip
######################################################################################################################## 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_SHA256SUMS
Not instructed to use Local PGP (/opt/homebrew/Cellar/tfenv/3.0.0/use-{gpgv,gnupg}) & No keybase install found, skipping OpenPGP signature verification
Archive:  /var/folders/6w/zd8qsqzn1r7g4b00h3rk79v80000gp/T/tfenv_download.XXXXXX.9YGFyexnCK/terraform_1.5.7_darwin_arm64.zip
  inflating: /opt/homebrew/Cellar/tfenv/3.0.0/versions/1.5.7/terraform  
Installation of terraform v1.5.7 successful. To make this your default version, run 'tfenv use 1.5.7'
Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.68.0


Your version of Terraform is out of date! The latest version
is 1.9.5. You can update by downloading from https://www.terraform.io/downloads.html

terraform init -upgrade   

Initializing the backend...
Upgrading modules...
...

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 2.23.0, >= 3.35.0, >= 4.0.0, >= 4.10.0, ~> 5.0"...

Terraform has been successfully initialized!

And now a darwin arm64 copy is showing up:

If using the amd64 version, I've always had issues where, like the other PRs I've linked, I've had to set GODEBUG=asyncpreemptoff=1

jotasixto · 2024-10-01T12:28:51Z

This led me down quite the rabbit hole wondering why on Apple M2 I am only getting amd64. It turns out (probably during one of the many upgrades from 0.11/0.12 darwin_arm64-unsupported versions), I may have set the architecture to amd64 and it downloaded not only the old version but the version we upgrade to (1.5.7) in amd64.

% export TFENV_ARCH=arm64

% tfenv install 1.5.7
Terraform v1.5.7 is already installed

% terraform version      
Terraform v1.5.7
on darwin_amd64
+ provider registry.terraform.io/datadog/datadog v3.44.1
+ provider registry.terraform.io/hashicorp/aws v5.68.0
+ provider registry.terraform.io/hashicorp/http v3.4.5
+ provider registry.terraform.io/hashicorp/null v3.2.3
+ provider registry.terraform.io/hashicorp/random v3.6.3
+ provider registry.terraform.io/hashicorp/time v0.12.1
+ provider registry.terraform.io/mongodb/mongodbatlas v1.20.0

Your version of Terraform is out of date! The latest version
is 1.9.5. You can update by downloading from https://www.terraform.io/downloads.html

% tfenv uninstall 1.5.7
Uninstall Terraform v1.5.7
Terraform v1.5.7 is successfully uninstalled

% terraform version    
version '1.5.7' is not installed (set by /repo/.terraform-version). Installing now as TFENV_AUTO_INSTALL==true
Installing Terraform v1.5.7
Downloading release tarball from https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_darwin_arm64.zip
######################################################################################################################## 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_SHA256SUMS
Not instructed to use Local PGP (/opt/homebrew/Cellar/tfenv/3.0.0/use-{gpgv,gnupg}) & No keybase install found, skipping OpenPGP signature verification
Archive:  /var/folders/6w/zd8qsqzn1r7g4b00h3rk79v80000gp/T/tfenv_download.XXXXXX.9YGFyexnCK/terraform_1.5.7_darwin_arm64.zip
  inflating: /opt/homebrew/Cellar/tfenv/3.0.0/versions/1.5.7/terraform  
Installation of terraform v1.5.7 successful. To make this your default version, run 'tfenv use 1.5.7'
Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.68.0


Your version of Terraform is out of date! The latest version
is 1.9.5. You can update by downloading from https://www.terraform.io/downloads.html

terraform init -upgrade   

Initializing the backend...
Upgrading modules...
...

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 2.23.0, >= 3.35.0, >= 4.0.0, >= 4.10.0, ~> 5.0"...

Terraform has been successfully initialized!

And now a darwin arm64 copy is showing up:

If using the amd64 version, I've always had issues where, like the other PRs I've linked, I've had to set `GODEBUG=asyncpreemptoff=1`

I’m using an Apple M3 with tfenv configured to the amd64 architecture in my TFENV_ARCH environment variable. This is because, in the projects I work on with my team, not everyone uses Apple M3; most of them are on Linux. To stay aligned with the rest of the team, I always configure my Terraform binary to work in amd64. Therefore, in my case, I’m on Apple M3 and using Rosetta 2. This issue only started happening with the latest version of the AWS provider (5.69.0). With version 5.68.0, everything works perfectly, only get the error that @law mentioned with version 5.69.0.

@cailen I tried running the command with the environment variable you suggested (GODEBUG=asyncpreemptoff=1) along with TF_LOG="debug", as shown below:

TF_LOG="debug" GODEBUG=asyncpreemptoff=1 terraform plan

While this reduced the number of log lines, the issue persists and fails at the same elements. Below is an excerpt from the logs:

...
2024-10-01T14:33:27.861+0200 [DEBUG] created provider logger: level=debug
2024-10-01T14:33:27.861+0200 [INFO]  provider: configuring client automatic mTLS
2024-10-01T14:33:27.869+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5]
2024-10-01T14:33:27.876+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5 pid=51108
2024-10-01T14:33:27.876+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5
2024-10-01T14:34:27.881+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.69.0/darwin_amd64/terraform-provider-aws_v5.69.0_x5 pid=51108 error="signal: killed"
...
2024-10-01T14:34:33.100+0200 [DEBUG] created provider logger: level=debug
2024-10-01T14:34:33.100+0200 [INFO]  provider: configuring client automatic mTLS
2024-10-01T14:34:33.108+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/darwin_amd64/terraform-provider-template_v2.2.0_x4 args=[.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/darwin_amd64/terraform-provider-template_v2.2.0_x4]
2024-10-01T14:34:33.113+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/darwin_amd64/terraform-provider-template_v2.2.0_x4 pid=52965
2024-10-01T14:34:33.113+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/darwin_amd64/terraform-provider-template_v2.2.0_x4
2024-10-01T14:34:34.598+0200 [INFO]  provider.terraform-provider-template_v2.2.0_x4: configuring server automatic mTLS: timestamp=2024-10-01T14:34:34.598+0200
2024-10-01T14:34:34.638+0200 [DEBUG] provider.terraform-provider-template_v2.2.0_x4: plugin address: address=/var/folders/s4/sp8pl52s6ynbzwm57pq5gmkh0000gn/T/plugin458334256 network=unix timestamp=2024-10-01T14:34:34.638+0200
2024-10-01T14:34:34.638+0200 [DEBUG] provider: using plugin: version=5
2024-10-01T14:34:34.692+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2024-10-01T14:34:34.692+0200 [DEBUG] No provider meta schema returned
2024-10-01T14:34:34.696+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/darwin_amd64/terraform-provider-template_v2.2.0_x4 pid=52965
2024-10-01T14:34:34.696+0200 [DEBUG] provider: plugin exited
...

cailen · 2024-10-01T13:23:42Z

@jotasixto The binaries are all made with the same code. I would think you could use the native copies (darwin arm64) to run anything locally on your computer and the other users could use darwin amd64 or whatever other flavor without issue. We do this where I work. Some are still on older Intel Macs. We also run things via Github Actions using Linux Amd64. I'm not at all discounting that it is broken, but you may be better off using the native version for your system unless there is no compatible one (like for very old versions of Terraform).

jotasixto · 2024-10-01T13:35:59Z

@jotasixto The binaries are all made with the same code. I would think you could use the native copies (darwin arm64) to run anything locally on your computer and the other users could use darwin amd64 or whatever other flavor without issue. We do this where I work. Some are still on older Intel Macs. We also run things via Github Actions using Linux Amd64. I'm not at all discounting that it is broken, but you may be better off using the native version for your system unless there is no compatible one (like for very old versions of Terraform).

@cailen Unfortunately, some of our legacy projects (which we are currently working on updating) use version 3 of the AWS provider, which doesn't have a compiled version for ARM to download. Therefore, I am unable to work with them locally on my machine. This is why I also have the Terraform binary configured to use darwin_amd64

cailen · 2024-10-01T13:47:57Z

@jotasixto makes sense then! Are you sure they don't have ARM copies though? From 3.30.0 there are ARM versions for Darwin. https://releases.hashicorp.com/terraform-provider-aws/3.30.0/. Maybe you are stuck using a version less than 3.30.0, but it may be worth trying to upgrade to the latest 3.x version if you can. The ARM version of Terraform also runs a lot faster.

jotasixto · 2024-10-01T14:20:32Z

@jotasixto makes sense then! Are you sure they don't have ARM copies though? From 3.30.0 there are ARM versions for Darwin. https://releases.hashicorp.com/terraform-provider-aws/3.30.0/. Maybe you are stuck using a version less than 3.30.0, but it may be worth trying to upgrade to the latest 3.x version if you can. The ARM version of Terraform also runs a lot faster.

@cailen I apologize for the confusion earlier. I was replying from my phone at the time and recalling from memory, as it had been a few months since I last worked on this issue. Now that I’ve had the chance to check it again on my laptop, I can confirm that the problem was actually related to HashiCorp providers and not AWS providers.

╷
│ Error: Incompatible provider version
│
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
╵

╷
│ Error: Incompatible provider version
│
│ Provider registry.terraform.io/hashicorp/local v1.4.0 does not have a package available for your current platform, darwin_arm64.
│
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
╵

╷
│ Error: Incompatible provider version
│
│ Provider registry.terraform.io/hashicorp/null v2.1.2 does not have a package available for your current platform, darwin_arm64.
│
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
╵

I should have verified this before responding.

However, this issue is causing a similar kind of blockage in legacy projects, as I can't remove the providers without following the proper migration process. Additionally, using the darwin_arm64 binary in my case is not a viable solution to work on the migration of these projects.

Thank you for your understanding, and I appreciate your suggestion!

claytonolley · 2024-10-01T16:46:24Z

Since Terraform Cloud runners are all x64, I lock my arch to that anyway on my M macs... helps with the lock file hashes as well. Maybe this is overkill but until this issue it's been working well for me.

LozanoMatheus · 2024-10-01T17:36:09Z

Looks like these projects have had similar errors recently:

SST: assertion failed ... code fragment does not contain the given arm address sst/ion#696

Pulumi: Provider v6.42.0+ crashing on Apple Silicon when running with Rosetta2 pulumi/pulumi-aws#4190

If I'm not mistaken, in the past Pulumi use terraform providers in the end, not sure if they still do it for some cases like this. If so, could be it. I'm not sure about stt 🤔

It seems to be an issue only with Apple M chips, the version v5.69.0 works fine on my Linux/amd64.

Also, it works fine when I use the same version in the CDKTF. Quite odd.

And about this, I just double checked and that CDKTF was using v5.59.0, I guess I was a bit too sleepy at that time 😅 . I got the same issue with the version v5.69.0.

cailen · 2024-10-02T00:17:29Z

I just unplugged my laptop for the day and noticed how hot my macbook is. Turns out these bad AWS providers from this morning were still running!

mmadrono · 2024-10-04T11:29:09Z

I have the same problem, I think, with the provider version v5.70.0 and earlier, I use a MAC IOS 15.0.1 - Chip Apple M2 - terraform version 0.14.11 and 1.9.7 and there is no way to run terraform without problems:

and with this error all the time, and if it does not give this error the process hangs, increasing the consumption of cpu and memory without success, and it is necessary to proceed to kill the process with what entails the blockade in dynamoDB and it is impossible to deploy infra in AWS and it is fundamental for my work.

rmayore · 2024-10-06T11:03:59Z

Same issue
Macbook Pro M1
Terraform v1.5.7
Provider v5.70

Downgrade to v5.68 fixed the issue. Also I want to emphasize on a comment above on how hard the problematic v5.70 providers hit your CPU and memory

cailen · 2024-10-07T12:10:52Z

I find it crazy that this has not been triaged yet. What is going on, maintainers? @marcosentino @breathingdust @justinretzolk

mars64 · 2024-10-07T20:28:53Z

Just chiming in: back-pinning to v5.68 also seems to have worked for me.

Macbook Pro M3
Terraform v1.8.5
Provider v5.70

And the 5.70 provider binary was indeed running in the background at 90+ usage.

ewbankkit · 2024-10-07T20:56:25Z

Could anyone who is experiencing this with v5.69.0 or v5.70.0 see if they have the same problem with v5.65.0 or v5.66.0?
These are all versions compiled with Go 1.23 whereas v5.67.0 and v5.68.0 were compiled with Go 1.22.
Thanks.

mmadrono · 2024-10-23T10:20:11Z

after more than two weeks we still have the same problem, which is being a real problem for those of us who deploy infra every day, please help us.

JnMik · 2024-10-23T15:19:05Z

GODEBUG=asyncpreemptoff=1 helped me get around the issue for now. Thanks for sharing @justinretzolk
Did you try it out @mmadrono ?

mmadrono · 2024-10-23T15:26:04Z

@JnMik > GODEBUG=asyncpreemptoff=1 me ayudó a solucionar el problema por ahora. Gracias por compartir ¿Lo probaste ?
I have not tried it, but I know what it is and where and how to configure it, forgive my ignorance, what do I have to do, but isn't there a solution or revision or a new version of the provider that solves the problems that we have been having since version 5.69.0?

JnMik · 2024-10-23T15:28:51Z

As you can see in this thread, the issue is not yet resolved, no.

Just put the variable in front of your terraform apply and see if it works

export GODEBUG=asyncpreemptoff=1
terraform apply

mmadrono · 2024-10-24T06:25:54Z

As you can see in this thread, the issue is not yet resolved, no.

Just put the variable in front of your terraform apply and see if it works
export GODEBUG=asyncpreemptoff=1
terraform apply

@JnMik Thank you very much for trying to help me, but it doesn't work for me, same problem and same error:

mmadrono · 2024-10-28T07:21:54Z

With version 5.73.0 also gives problems, and the same failure.

mmadrono · 2024-11-04T06:58:30Z

With version 5.74.0 also gives problems, and the same failure.

mmadrono · 2024-11-10T00:02:37Z

With version 5.75.0 also gives problems, and the same failure:

mmadrono · 2024-11-11T09:11:42Z

@bschaatsbergen Thanks for your answer, but the link that you have provided me does not tell me anything, I do not know if it is a problem or solution or if it has to do with the problem that we are having all those who manage terraform with a Mac with an apple chip. We have been more than a month with this problem, and after 6 or more versions of the provider, we still have the same problem, just tell you that it is an important problem for people who deploy infrastructure in AWS every day, and I am giving me the feeling that this problem is not being taken with the importance it requires. I just ask you please to solve it as soon as possible, because this depends on our work and day to day.

mmadrono · 2024-11-12T09:02:12Z

With version 5.75.1 also gives problems, and the same failure:

thomaspeitz · 2024-11-12T10:55:05Z

I am not sure why, but it happens a lot less frequently if I put terraform into debugging log mode.

TF_LOG=debug terraform

But I am maybe a special case because I need to deploy for a customer with an older terraform version.

Maybe some timing differences (through log output) already help to make the problem less severe.

justinretzolk · 2024-11-12T15:52:24Z

Hey @mmadrono 👋 Due to the nature of this issue, we don't expect that it will be resolved in the short term. This seems to be a bug in Rosetta2, so we're keeping an eye out for any updates that might address it, and will update this thread accordingly so that everyone following is notified. If your setup differs significantly from what's been laid out in the thread so far, it may be worth opening a new Issue so that we can address that separately. In the meantime, releases will continue to be effected, so you don't need to worry about letting us know that you're continuing to see the same behavior when a new version is released.

I noticed that you're using Terragrunt and that your screenshots seem to indicate that you're running darwin_amd64 binaries rather than the native darwin_arm64 builds. I'm not familiar enough with Terragrunt to know if that's a setting, or might be a result of caching, but if you've not done that intentionally, switching to the darwin_arm64 binary will resolve your issue. If you are intentionally running darwin_amd64 binaries, you may need to take additional steps in order to use the workarounds mentioned above. Noting my familiarity, I'll leave the specifics around working with Terragrunt to someone else with more experience.

@thomaspeitz thanks for the addition information!

mmadrono · 2024-11-13T08:31:28Z

Hey @mmadrono 👋 Due to the nature of this issue, we don't expect that it will be resolved in the short term. This seems to be a bug in Rosetta2, so we're keeping an eye out for any updates that might address it, and will update this thread accordingly so that everyone following is notified. If your setup differs significantly from what's been laid out in the thread so far, it may be worth opening a new Issue so that we can address that separately. In the meantime, releases will continue to be effected, so you don't need to worry about letting us know that you're continuing to see the same behavior when a new version is released.

I noticed that you're using Terragrunt and that your screenshots seem to indicate that you're running darwin_amd64 binaries rather than the native darwin_arm64 builds. I'm not familiar enough with Terragrunt to know if that's a setting, or might be a result of caching, but if you've not done that intentionally, switching to the darwin_arm64 binary will resolve your issue. If you are intentionally running darwin_amd64 binaries, you may need to take additional steps in order to use the workarounds mentioned above. Noting my familiarity, I'll leave the specifics around working with Terragrunt to someone else with more experience.

@thomaspeitz thanks for the addition information!

Thanks for your answer, and for the time spent, but tell you that the version of terraform that is used in my organization is 0.14.11 and this version is not Darwin_arm64 architecture, there is only amd64, so using the binary terraform in this architecture. there is no other. But I have never had any problems with this binary until now, well until provider version 5.69.0 and later.

gvoss · 2024-11-13T15:09:48Z

I am also not convinced this issue is solely related to rosetta as I have explained already my workflow does not include any Apple CPUs and I am both facing the same issue and benefit from the same workaround. The only thing I can think of is perhaps if the root cause is actually related to virtualization in some way and that a rosetta fix in golang or this module may also fix what we're experiencing on AWS.

justinretzolk · 2024-11-13T22:26:46Z

Thanks for the data point on why you're using the darwin_amd64 builds @mmadrono!

@gvoss: Just to confirm, since you're the only report we've seen that wasn't tied to MacOS/Rosetta: are you receiving the same error, or just similar behavior? To save you searching this thread, the relevant error was:

2024-09-27T13:46:40.034-0600 [DEBUG] provider.terraform-provider-aws_v5.69.0_x5: assertion failed [arm_interval().contains(address)]: code fragment does not contain the given arm address
2024-09-27T13:46:40.034-0600 [DEBUG] provider.terraform-provider-aws_v5.69.0_x5: (CodeFragmentMetadata.cpp:48 instruction_extents_for_arm_address)

If it's similar behavior without that exact error, you'll want to open a new issue so that we can look into that separately. If it's the same error and you're able to provide any debug logging, that would be quite helpful.

claytonolley · 2024-11-13T22:30:09Z

@justinretzolk for another datapoint, some older providers don't have arm builds still and we'd rather not compile them ourselves (this is less of an issue lately), however we also feel that since Terraform Cloud uses amd runners, we might as well run that locally just in case. This also allows us to align our lock files better with TFC runners.

ohookins · 2024-11-14T00:03:36Z

@justinretzolk for another datapoint, some older providers don't have arm builds still and we'd rather not compile them ourselves (this is less of an issue lately), however we also feel that since Terraform Cloud uses amd runners, we might as well run that locally just in case. This also allows us to align our lock files better with TFC runners.

Indeed, that's the situation for us. We have to use the AMD64 version of the GCP provider, and that forces us to use AMD64 for all providers, when on ARM Mac hardware.

gvoss · 2024-11-14T14:34:24Z

@gvoss: Just to confirm, since you're the only report we've seen that wasn't tied to MacOS/Rosetta: are you receiving the same error, or just similar behavior? To save you searching this thread, the relevant error was:
2024-09-27T13:46:40.034-0600 [DEBUG] provider.terraform-provider-aws_v5.69.0_x5: assertion failed [arm_interval().contains(address)]: code fragment does not contain the given arm address
2024-09-27T13:46:40.034-0600 [DEBUG] provider.terraform-provider-aws_v5.69.0_x5: (CodeFragmentMetadata.cpp:48 instruction_extents_for_arm_address)
If it's similar behavior without that exact error, you'll want to open a new issue so that we can look into that separately. If it's the same error and you're able to provide any debug logging, that would be quite helpful.

I originally created #39627 but it was triaged and closed to defer to this issue. It sounds like perhaps the workaround is the same but maybe these two issues differ?

nina-nee · 2024-11-18T15:28:44Z

I have an interesting data-point for y'all. I've got v.5.76.0 sources from GitHub, and compiled with go 1.23.2 twice: natively on MacOS, and cross-compiling for Darwin amd64 from Ubuntu (GOOS=darwin GOARCH=amd64 go build). I started the provider binary in a loop on my M2 Mac.

~~The natively compiled version didn't show any issues at all, but cross-compiled version crashes regularly with (CodeFragmentMetadata.cpp:48 instruction_extents_for_arm_address).~~

~~Thus, if you experience this issue, you might consider compiling the provider from sources yourself.~~

~~I suppose it makes sense to investigate what is happening with Go and Darwin amd64 cross-compilation. It would be nice if someone could replicate my results and confirm them.~~

Never mind, in my experiments I accidentally compiled provider for arm64 instead of amd64. So, no new information: darwin-amd64 crashes, darwin-arm64 works as expected.

Edit 2: I have compiled v5.68.0 with GOOS=darwin GOARCH=amd64 go build on Darwin using golang v1.23 (as opposed to v1.22), and it hangs with code fragment does not contain the given arm address. That very strongly implicates an issue between golang and Rosetta2.

lancedikson · 2024-11-19T12:17:05Z

Encountered the same issue, but don't know how I can help. Downgraded to 5.68 and it seems working. Please let me know if you'd like any additional details.

aorozco-g · 2024-11-27T08:58:21Z

Same here with M3, with 5.68 works fine.

alexw23 · 2024-11-27T18:53:24Z

Had this issue and seems tfenv defaults to amd64 even if you’re running on arm64 (I.e Apple Silicon).

To fix this, add the following to ~/.zshrc (or equivalent):

export TFENV_ARCH=arm64

Then run the following commands to reinstall:

tfenv uninstall 1.10.0
tfenv use 1.10.0

sc1f · 2024-12-05T16:50:32Z

An additional data point - I have been encountering this issue for a while now starting with AWS provider v5.69.0. We are using ASDF for version management of Terraform and Terragrunt and it turns out that ASDF automatically installs the amd64 version of Terraform (and then installs the arm64 version of Terragrunt, surprisingly). Forcing ASDF to install the arm64 version of Terraform seems to have fixed my issues with "timeout while waiting for plugin to start`:

asdf uninstall terraform <version>
ASDF_HASHICORP_OVERWRITE_ARCH=arm64 asdf install terraform <version>

vladiceanu · 2024-12-23T14:40:00Z

An additional data point - I have been encountering this issue for a while now starting with AWS provider v5.69.0. We are using ASDF for version management of Terraform and Terragrunt and it turns out that ASDF automatically installs the amd64 version of Terraform (and then installs the arm64 version of Terragrunt, surprisingly). Forcing ASDF to install the arm64 version of Terraform seems to have fixed my issues with "timeout while waiting for plugin to start`:
asdf uninstall terraform <version>
ASDF_HASHICORP_OVERWRITE_ARCH=arm64 asdf install terraform <version>

this one worked for me, thank you

mmadrono · 2024-12-23T16:06:23Z

An additional data point - I have been encountering this issue for a while now starting with AWS provider v5.69.0. We are using ASDF for version management of Terraform and Terragrunt and it turns out that ASDF automatically installs the amd64 version of Terraform (and then installs the arm64 version of Terragrunt, surprisingly). Forcing ASDF to install the arm64 version of Terraform seems to have fixed my issues with "timeout while waiting for plugin to start`:
asdf uninstall terraform <version>
ASDF_HASHICORP_OVERWRITE_ARCH=arm64 asdf install terraform <version>

do you know if this works with terraform version 0.14.11?
https://releases.hashicorp.com/terraform/0.14.11/
since there is only that version for MAC:

jacoor · 2025-01-17T13:36:33Z

Confifmed

# this possibly fixes https://github.com/hashicorp/terraform-provider-aws/issues/39523
export TFENV_ARCH=arm64

and reinstall terraform using tfenv solved this.
I am now on AWS provider 5.84

law added the bug Addresses a defect in current functionality. label Sep 27, 2024

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Sep 27, 2024

cailen mentioned this issue Oct 2, 2024

[Bug]: terraform-provider-aws v5.69.0 Fails to Load #39558

Closed

ewbankkit added regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. go Pull requests that update Go code labels Oct 7, 2024

terraform-aws-provider bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Oct 7, 2024

bschaatsbergen mentioned this issue Nov 10, 2024

fix: improve Terraform download error handling tfversion/tfversion#58

Open

justinretzolk added the upstream Addresses functionality related to the cloud provider. label Nov 12, 2024

albertsj1 mentioned this issue Dec 20, 2024

feat: Add Log Anomaly Detection Support terraform-aws-modules/terraform-aws-cloudwatch#74

Merged

3 tasks

ewbankkit removed their assignment Dec 29, 2024

[Bug]: Terraform apply command freezes during AWS provider initialization #39523

[Bug]: Terraform apply command freezes during AWS provider initialization #39523

Comments

law commented Sep 27, 2024

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

github-actions bot commented Sep 27, 2024

Community Note

law commented Sep 27, 2024

alexpodr commented Sep 27, 2024 • edited Loading

elaigor commented Sep 30, 2024

jotasixto commented Sep 30, 2024 • edited Loading

JnMik commented Sep 30, 2024

LozanoMatheus commented Sep 30, 2024 • edited Loading

cailen commented Sep 30, 2024

cailen commented Oct 1, 2024 • edited Loading

cailen commented Oct 1, 2024

cailen commented Oct 1, 2024 • edited Loading

jotasixto commented Oct 1, 2024 • edited Loading

cailen commented Oct 1, 2024

jotasixto commented Oct 1, 2024

cailen commented Oct 1, 2024 • edited Loading

jotasixto commented Oct 1, 2024

claytonolley commented Oct 1, 2024

LozanoMatheus commented Oct 1, 2024

cailen commented Oct 2, 2024

mmadrono commented Oct 4, 2024 • edited Loading

rmayore commented Oct 6, 2024 • edited Loading

cailen commented Oct 7, 2024

mars64 commented Oct 7, 2024

ewbankkit commented Oct 7, 2024

mmadrono commented Oct 23, 2024

JnMik commented Oct 23, 2024

mmadrono commented Oct 23, 2024 • edited Loading

JnMik commented Oct 23, 2024 • edited Loading

mmadrono commented Oct 24, 2024

mmadrono commented Oct 28, 2024

mmadrono commented Nov 4, 2024 • edited Loading

mmadrono commented Nov 10, 2024

mmadrono commented Nov 11, 2024

mmadrono commented Nov 12, 2024

thomaspeitz commented Nov 12, 2024 • edited Loading

justinretzolk commented Nov 12, 2024

mmadrono commented Nov 13, 2024 • edited Loading

gvoss commented Nov 13, 2024

justinretzolk commented Nov 13, 2024

claytonolley commented Nov 13, 2024

ohookins commented Nov 14, 2024

gvoss commented Nov 14, 2024

nina-nee commented Nov 18, 2024 • edited Loading

lancedikson commented Nov 19, 2024

aorozco-g commented Nov 27, 2024

alexw23 commented Nov 27, 2024 • edited Loading

sc1f commented Dec 5, 2024 • edited Loading

vladiceanu commented Dec 23, 2024

mmadrono commented Dec 23, 2024 • edited Loading

jacoor commented Jan 17, 2025

alexpodr commented Sep 27, 2024 •

edited

Loading

jotasixto commented Sep 30, 2024 •

edited

Loading

LozanoMatheus commented Sep 30, 2024 •

edited

Loading

cailen commented Oct 1, 2024 •

edited

Loading

cailen commented Oct 1, 2024 •

edited

Loading

jotasixto commented Oct 1, 2024 •

edited

Loading

cailen commented Oct 1, 2024 •

edited

Loading

mmadrono commented Oct 4, 2024 •

edited

Loading

rmayore commented Oct 6, 2024 •

edited

Loading

mmadrono commented Oct 23, 2024 •

edited

Loading

JnMik commented Oct 23, 2024 •

edited

Loading

mmadrono commented Nov 4, 2024 •

edited

Loading

thomaspeitz commented Nov 12, 2024 •

edited

Loading

mmadrono commented Nov 13, 2024 •

edited

Loading

nina-nee commented Nov 18, 2024 •

edited

Loading

alexw23 commented Nov 27, 2024 •

edited

Loading

sc1f commented Dec 5, 2024 •

edited

Loading

mmadrono commented Dec 23, 2024 •

edited

Loading