-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hcl parser escape behavior is broken, cannot use variables or functions with content that requires escaping #6359
Comments
Backslashes are REALLY messing up parsing. "console/target/layout" = "${replace(var.nlog_console_target_layout, "/\\\\/", "\\")}" If I escape the backslash in the 3rd replace argument, it's a "parse error: syntax error" If I DON'T escape the backslash in the 3rd replace argument, then it becomes an empty string |
Here's some debug info I grabbed using terraform source + delve to understand the backslash syntax error issue. It appears to be a very low level parsing error.
|
For completeness I also tried converting my template to JSON and seeing if it's just an issue with the HCL parser. It's not!
{
"provider": {
"consul": {
"address": "${var.consul_server_address}",
"scheme": "${var.consul_scheme}",
"datacenter": "${var.consul_datacenter}"
}
},
"resource": {
"consul_key_prefix": {
"nlog_config": {
"token": "${var.consul_server_token}",
"path_prefix": "${replace(concat(var.consul_prefix, \"/\"), \"////\", \"/\")}${var.nlog_config_name}/",
"subkeys": {
"console/enabled": "${var.nlog_console_enabled}",
"console/logger/minlevel": "${var.nlog_console_logger_minlevel}",
"console/logger/name": "${var.nlog_console_logger_name}",
"console/target/layout": "${replace(var.nlog_console_target_layout, \"\\\\\\\\\", \"\\\")}",
"filelog/enabled": "${var.nlog_filelog_enabled}",
"filelog/logger/minlevel": "${var.nlog_filelog_logger_minlevel}",
"filelog/logger/name": "${var.nlog_filelog_logger_name}",
"filelog/target/fileName": "${var.nlog_filelog_target_fileName}",
"filelog/target/layout": "${var.nlog_filelog_target_layout}",
"kinesis/enabled": "${var.nlog_kinesis_enabled}",
"kinesis/logger/minlevel": "${var.nlog_kinesis_logger_minlevel}",
"kinesis/logger/name": "${var.nlog_kinesis_logger_name}",
"kinesis/target/environment": "${var.nlog_kinesis_target_environment}",
"kinesis/target/index": "${var.nlog_kinesis_target_index}",
"kinesis/target/maxQueueSize": "${var.nlog_kinesis_target_maxQueueSize}",
"kinesis/target/region": "${var.nlog_kinesis_target_region}",
"kinesis/target/stream": "${var.nlog_kinesis_target_stream}",
"kinesis/target/system": "${var.nlog_kinesis_target_system}",
"ravendb/enabled": "${var.nlog_ravendb_enabled}",
"ravendb/logger/minlevel": "${var.nlog_ravendb_logger_minlevel}",
"ravendb/logger/name": "${var.nlog_ravendb_logger_name}"
}
}
}
}
}
It's literally impossible for me to insert a single |
@phinze is there any additional information I can gather to attempt to help resolve this issue? The parse and escape behavior is SO BROKEN that I cannot trust terraform for production use as it stands. Here's some more examples showing why this is a big problem: 1
If I don't escape the backslashes in the JSON file, terraform complains of an However, if I do escape it, terraform KEEPS both backslashes! Note here proving that it's parsing the escape sequences in some cases: 2
Output includes unicode characters:
However, backslashes aren't the only escape sequences that get 'validated' but not actually 'parsed', e.g.: 3
Note how each 'escape sequence' is kept, rather than being parsed into its actual value. Now, if you noticed my above example using the Let's try it out:
Hmm, well it does convert it to a backslash, except it again still keeps the 'escaping' backslash, too! Now that you have a taste of JSON, what about HCL? Does it work the same? Let's try the above examples in HCL, instead: 1
Well, that's basically what we would expect. We put in two backslashes and we received two backslashes. Since we're not in JSON anymore and we don't need to escape a backslash, let's see if we can do just a single backslash: 1a
Nope, terraform still treating them as escape characters. 2
Same as in JSON, terraform is parsing the backslash and following 3
Just like JSON, these values are accepted as valid escape sequences, but don't actually get parsed into anything, and you're left with the literal. 4
Again, just like in HCL, parsed as an escape sequence, but left with the escaping backslash. I also discovered some weird behavior between HCL and JSON regarding the replace() method (due to backslash parsing), but I'll add that detail in a separate comment. |
If this doesn't prove that backslashes are breaking things, I dunno what will... 1
2
|
Hi @eedwardsdisco - thanks for putting together all these examples! Definitely multiple bugs in here to be sorted out. One of us dig in and keep you posted as we make progress. |
Getting bit again by this parsing issue in yet ANOTHER way. I seem to be unable to insert provisioner "remote-exec" {
inline = [
"sudo tee ${var.chef_client_tmp_path}/knife.sh > /dev/null <<KNIFESCRIPT",
"#!/bin/bash",
"knife serve -c ${var.chef_client_tmp_path}/client.rb --chef-zero-host ${aws_instance.chef_server.private_ip} > ${var.chef_client_tmp_path}/knife-serve.out 2> ${var.chef_client_tmp_path}/knife-serve.err < /dev/null & echo \"$!\" > ${var.chef_client_tmp_path}/pid",
"KNIFESCRIPT",
"sudo chmod 700 ${var.chef_client_tmp_path}/knife.sh"
]
} Notably, this section near the end of the long command string: When terraform parses it and runs it in the .sh file that it creates on the target server, the echo contains an empty string (e.g. I've tried escaping it as per #4747 Related? Edit: I found a workaround for this specific issue by double-escaping the backslash:
|
I hate to bump on this, but the low level parsing library is a pretty core part of Terraform and I'm stumped on how to fix this (I'm not enough of a wizard to figure it out, I tried...). Who did the majority work on implementing the new HCL parser a few versions ago? I believe this is when the problem was introduced, and I'm not sure who else can figure it out. |
Fixes hashicorp/terraform#6359 by unquoting a double backslash within braces to a single backslash.
Fixes hashicorp/terraform#6359 and several related issues by unescaping a double backslash within braces to a single backslash. This bumps into the bit of HIL that still hangs out in HCL - for values that aren't interpolated, like Terraform's variable defaults - users were unable to get a backslash to show up within `${}`. That's because `${}` is handled specially to allow for, e.g., double quotes inside of braces. Here, we add `\\` as a special cased escape (along with `\"`) so users can get backslashes in these scenarios by doubling them up.
@phinze you're my hero! |
Pulls in hashicorp/hcl#125 Which fixes #6359
😀 👍 Thanks for all the documentation on this issue @eedwardsdisco! I believe I've addressed a common denominator bug that allows workarounds for nearly all of the examples mentioned here. I'm going to file breakout issues for the remaining things from this backlog that are problematic. |
Yup I looked at your commits and saw you used some of my example problems as test cases, much appreciated! Super excited about this fix. |
Terraform Version0.7.2 it appears I'm having this problem in the latest version
This will produce the following error: main.tf: At 2:31: illegal char escape Changing this to escape the backslash seems to keep both
|
@phinze is this a regression? I was hoping this case was fixed as I have a project I've been blocked on finishing due to this parsing issue. |
This is still present in terraform v10.4 This issue seems to keep resurfacing |
Outputs
`Terraform v0.11.2
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
Terraform Version
0.6.15, 0.6.16
Affected Resource(s)
Expected Behavior
Actual Behavior
${}
, and throwing a "cannot contain interpolations" error\:
, and throwing an "illegal char escape" error\:
is included in a .tfvars file within a valueSteps to Reproduce
Unescaped Backslash in Variable (Error)
TF code:
terraform validate:
Escaped Backslash in Variable (Escape Character Not Removed)
TF code:
consul key value (should not contain extra backslash):
Un-Escaped Interpolation Syntax in Variable
TF code:
terraform validate:
Dollar Escaped Interpolation Syntax in Variable
TF code:
terraform validate:
Backslash Escaped Interpolation Syntax in Variable
TF code:
terraform validate:
Specifying Value in .tfvars file instead, un-escaped
vars.tfvars
terraform plan
Specifying Value in .tfvars file instead, escaped
vars.tfvars
consul key value (should not contain extra backslash):
Notes
It appears that the values specified in 'default' aren't always getting ignored for interpolation parsing. I have multiple variables that contain
${}
and they don't always cause the 'cannot contain interpolation syntax' error.It also appears that the backslash is getting treated as an escape character, but if I escape the backslash with another backslash, both backslashes are kept rather than it actually being escaped.
I suspect this is a problem with validation.
References
The text was updated successfully, but these errors were encountered: