Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hcl parser escape behavior is broken, cannot use variables or functions with content that requires escaping #6359

Closed
eedwardsdisco opened this issue Apr 26, 2016 · 17 comments · Fixed by #7048

Comments

@eedwardsdisco
Copy link

eedwardsdisco commented Apr 26, 2016

Terraform Version

0.6.15, 0.6.16

Affected Resource(s)

  • consul_key_prefix
  • core ?

Expected Behavior

  • Terraform treats default values of variable as 'raw' and does not interpolate or parse them.
  • Terraform allows a variable value from a .tfvars file to contain backslashes, or a mechanism to escape them properly

Actual Behavior

  • Terraform is attempting to parse the 'default' value of a variable containing ${}, and throwing a "cannot contain interpolations" error
  • Terraform is attempting to parse the 'default' value of a variable containing \:, and throwing an "illegal char escape" error
  • Terraform errors if \: is included in a .tfvars file within a value

Steps to Reproduce

Unescaped Backslash in Variable (Error)

TF code:

variable "nlog_console_target_layout" {
    type = "string"
    default = "${date:format=MM/dd HH\:mm\:ss} ${logger}|${message}"
}

terraform validate:

$ terraform validate
Error loading files Error parsing /<snip>/variables.tf: At 65:40: illegal char escape

Escaped Backslash in Variable (Escape Character Not Removed)

TF code:

variable "nlog_console_target_layout" {
    type = "string"
    default = "${date:format=MM/dd HH\\:mm\\:ss} ${logger}|${message}"
}

consul key value (should not contain extra backslash):

${date:format=MM/dd HH\\:mm\\:ss} ${logger}|${message}

Un-Escaped Interpolation Syntax in Variable

TF code:

variable "nlog_filelog_target_fileName" {
    type = "string"
    default = "MyApp.${level}.log"
}

terraform validate:

$ terraform validate
Error validating: 1 error(s) occurred:

* Variable 'nlog_filelog_target_fileName': cannot contain interpolations

Dollar Escaped Interpolation Syntax in Variable

TF code:

variable "nlog_filelog_target_fileName" {
    type = "string"
    default = "MyApp.$${level}.log"
}

terraform validate:

$ terraform validate
Error validating: 1 error(s) occurred:

* Variable 'nlog_filelog_target_fileName': cannot contain interpolations

Backslash Escaped Interpolation Syntax in Variable

TF code:

variable "nlog_filelog_target_fileName" {
    type = "string"
    default = "MyApp.\${level}.log"
}

terraform validate:

$ terraform validate
Error loading files Error parsing /<snip>/variables.tf: At 89:26: illegal char escape

Specifying Value in .tfvars file instead, un-escaped

vars.tfvars

nlog_console_target_layout = "${date:format=MM/dd HH\:mm\:ss} ${logger}|${message}"

terraform plan

$ terraform plan -var-file=vars.tfvars consul_nlog_config-v0.0.7 
invalid value "vars.tfvars" for flag -var-file: Error parsing vars.tfvars: At 14:58: illegal char escape

Specifying Value in .tfvars file instead, escaped

vars.tfvars

nlog_console_target_layout = "${date:format=MM/dd HH\\:mm\\:ss} ${logger}|${message}"

consul key value (should not contain extra backslash):

${date:format=MM/dd HH\\:mm\\:ss} ${logger}|${message}

Notes

It appears that the values specified in 'default' aren't always getting ignored for interpolation parsing. I have multiple variables that contain ${} and they don't always cause the 'cannot contain interpolation syntax' error.

It also appears that the backslash is getting treated as an escape character, but if I escape the backslash with another backslash, both backslashes are kept rather than it actually being escaped.

I suspect this is a problem with validation.

References

@eedwardsdisco
Copy link
Author

Backslashes are REALLY messing up parsing.

"console/target/layout" = "${replace(var.nlog_console_target_layout, "/\\\\/", "\\")}"

If I escape the backslash in the 3rd replace argument, it's a "parse error: syntax error"

If I DON'T escape the backslash in the 3rd replace argument, then it becomes an empty string ""

@eedwardsdisco
Copy link
Author

Here's some debug info I grabbed using terraform source + delve to understand the backslash syntax error issue.

It appears to be a very low level parsing error.

(dlv) > [bk55017056] github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).Parse.func1() /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:50 (hits goroutine(1):1 total:1) (PC: 0x3136c8)
�[34m    45:    �[0m// Parse returns the fully parsed source and returns the abstract syntax tree.
�[34m    46:    �[0mfunc (p *Parser) Parse() (*ast.File, error) {
�[34m    47:    �[0m    f := &ast.File{}
�[34m    48:    �[0m    var err, scerr error
�[34m    49:    �[0m    p.sc.Error = func(pos token.Pos, msg string) {
�[34m=>  50:    �[0m        scerr = &PosError{Pos: pos, Err: errors.New(msg)}
�[34m    51:    �[0m    }
�[34m    52:    �[0m
�[34m    53:    �[0m    f.Node, err = p.objectList()
�[34m    54:    �[0m    if scerr != nil {
�[34m    55:    �[0m        return nil, scerr
(dlv) print msg
"illegal char escape"
(dlv) stack
 0  0x00000000003136c8 in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).Parse.func1
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:50
 1  0x0000000000548d93 in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner.(*Scanner).err
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner/scanner.go:586
 2  0x000000000054890d in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner.(*Scanner).scanEscape
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner/scanner.go:520
 3  0x000000000054883b in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner.(*Scanner).scanString
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner/scanner.go:493
 4  0x000000000054792c in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner.(*Scanner).Scan
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/scanner/scanner.go:166
 5  0x0000000000312d54 in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).scan
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:376
 6  0x0000000000311787 in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).object
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:243
 7  0x000000000031075b in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).objectItem
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:160
 8  0x000000000030fb30 in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).objectList
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:70
 9  0x0000000000311dcf in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).objectType
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:273
10  0x0000000000310447 in github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser.(*Parser).objectItem
    at /Users/ethanedwards/Documents/dev/golang/src/github.com/hashicorp/terraform/vendor/github.com/hashicorp/hcl/hcl/parser/parser.go:165
(dlv) 

@eedwardsdisco
Copy link
Author

For completeness I also tried converting my template to JSON and seeing if it's just an issue with the HCL parser.

It's not!

\\ still results in a syntax error...

{
  "provider": {
    "consul": {
      "address": "${var.consul_server_address}",
      "scheme": "${var.consul_scheme}",
      "datacenter": "${var.consul_datacenter}"
    }
  },

  "resource": {
    "consul_key_prefix": {
      "nlog_config": {
        "token": "${var.consul_server_token}",
        "path_prefix": "${replace(concat(var.consul_prefix, \"/\"), \"////\", \"/\")}${var.nlog_config_name}/",
        "subkeys": {
          "console/enabled": "${var.nlog_console_enabled}",
          "console/logger/minlevel": "${var.nlog_console_logger_minlevel}",
          "console/logger/name": "${var.nlog_console_logger_name}",
          "console/target/layout": "${replace(var.nlog_console_target_layout, \"\\\\\\\\\", \"\\\")}",
          "filelog/enabled": "${var.nlog_filelog_enabled}",
          "filelog/logger/minlevel": "${var.nlog_filelog_logger_minlevel}",
          "filelog/logger/name": "${var.nlog_filelog_logger_name}",
          "filelog/target/fileName": "${var.nlog_filelog_target_fileName}",
          "filelog/target/layout": "${var.nlog_filelog_target_layout}",
          "kinesis/enabled": "${var.nlog_kinesis_enabled}",
          "kinesis/logger/minlevel": "${var.nlog_kinesis_logger_minlevel}",
          "kinesis/logger/name": "${var.nlog_kinesis_logger_name}",
          "kinesis/target/environment": "${var.nlog_kinesis_target_environment}",
          "kinesis/target/index": "${var.nlog_kinesis_target_index}",
          "kinesis/target/maxQueueSize": "${var.nlog_kinesis_target_maxQueueSize}",
          "kinesis/target/region": "${var.nlog_kinesis_target_region}",
          "kinesis/target/stream": "${var.nlog_kinesis_target_stream}",
          "kinesis/target/system": "${var.nlog_kinesis_target_system}",
          "ravendb/enabled": "${var.nlog_ravendb_enabled}",
          "ravendb/logger/minlevel": "${var.nlog_ravendb_logger_minlevel}",
          "ravendb/logger/name": "${var.nlog_ravendb_logger_name}"
        }
      }
    }
  }
}
e:Error loading config: Error loading consul_nlog_config-v0.0.8/consul_nlog_config.tf.json: Error reading config for consul_key_prefix[nlog_config]: parse error: syntax error

It's literally impossible for me to insert a single \ character.

@eedwardsdisco
Copy link
Author

Suspect related to #4052 and #5550

@eedwardsdisco
Copy link
Author

@phinze is there any additional information I can gather to attempt to help resolve this issue?

The parse and escape behavior is SO BROKEN that I cannot trust terraform for production use as it stands.

Here's some more examples showing why this is a big problem:

1

{
  "resource": {
    "template_file": {
      "parse_test": {
        "template": "this \\string \\contains \\backslashes"
      }
    }
  }
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this \\string \\contains \\backslashes"

If I don't escape the backslashes in the JSON file, terraform complains of an illegal char escape, so it's at least considering any backslash in JSON to be an escape character.

However, if I do escape it, terraform KEEPS both backslashes!

Note here proving that it's parsing the escape sequences in some cases:

2

{
  "resource": {
    "template_file": {
      "parse_test": {
        "template": "this \u1234string \u8273contains \u0123backslashes"
      }
    }
  }
}

Output includes unicode characters:

o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this ሴstring 艳contains ģbackslashes"

However, backslashes aren't the only escape sequences that get 'validated' but not actually 'parsed', e.g.:

3

{
  "resource": {
    "template_file": {
      "parse_test": {
        "template": "this \r string \n contains \t backslashes"
      }
    }
  }
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this \r string \n contains \t backslashes"

Note how each 'escape sequence' is kept, rather than being parsed into its actual value.

Now, if you noticed my above example using the \uABCD syntax where it was properly converting into the character, could I just use the unicode value for a backslash and get past my syntax error?

Let's try it out:
4

{
  "resource": {
    "template_file": {
      "parse_test": {
        "template": "this \u005c string \u005c contains \u005c backslashes"
      }
    }
  }
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this \\ string \\ contains \\ backslashes"

Hmm, well it does convert it to a backslash, except it again still keeps the 'escaping' backslash, too!

Now that you have a taste of JSON, what about HCL? Does it work the same? Let's try the above examples in HCL, instead:

1

resource "template_file" "parse_test" {
    template = "this \\string \\contains \\backslashes"
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this \\string \\contains \\backslashes"

Well, that's basically what we would expect. We put in two backslashes and we received two backslashes.

Since we're not in JSON anymore and we don't need to escape a backslash, let's see if we can do just a single backslash:

1a

resource "template_file" "parse_test" {
    template = "this \string \contains \backslashes"
}
e:Error loading files Error parsing /Users/ethanedwards/Documents/dev/packages/consul_nlog_config-edwards/parse_test/test.tf: At 2:28: illegal char escape

Nope, terraform still treating them as escape characters.

2

resource "template_file" "parse_test" {
    template = "this \u1234string \u8273contains \u0123backslashes"
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this ሴstring 艳contains ģbackslashes"

Same as in JSON, terraform is parsing the backslash and following uABCD as a valid escape sequence and then replacing it with the unicode character.

3

resource "template_file" "parse_test" {
    template = "this \r string \n contains \t backslashes"
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this \r string \n contains \t backslashes"

Just like JSON, these values are accepted as valid escape sequences, but don't actually get parsed into anything, and you're left with the literal.

4

resource "template_file" "parse_test" {
    template = "this \u005c string \u005c contains \u005c backslashes"
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "this \\ string \\ contains \\ backslashes"

Again, just like in HCL, parsed as an escape sequence, but left with the escaping backslash.

I also discovered some weird behavior between HCL and JSON regarding the replace() method (due to backslash parsing), but I'll add that detail in a separate comment.

@eedwardsdisco
Copy link
Author

If this doesn't prove that backslashes are breaking things, I dunno what will...

1

variable "parse_var" {
    default = "this \\ string \\ contains \\ backslashes"
}

resource "template_file" "parse_test" {
    template = "${ replace(var.parse_var, "\", "test") }"
}
o:+ template_file.parse_test
    rendered: "" => "<computed>"
    template: "" => "testttesthtestiteststest test\\test teststestttestrtestitestntestgtest test\\test testctestotestntestttestatestitestnteststest test\\test testbtestatestctestkteststestltestateststesthtesteteststest"

2

variable "parse_var" {
    default = "this \\ string \\ contains \\ backslashes"
}

resource "template_file" "parse_test" {
    template = "${ replace(var.parse_var, "\\", "test") }"
}
e:Error loading config: Error loading /Users/ethanedwards/Documents/dev/packages/consul_nlog_config-edwards/parse_test/test.tf: Error reading config for template_file[parse_test]: parse error: syntax error

@phinze
Copy link
Contributor

phinze commented Apr 28, 2016

Hi @eedwardsdisco - thanks for putting together all these examples! Definitely multiple bugs in here to be sorted out. One of us dig in and keep you posted as we make progress.

@eedwardsdisco
Copy link
Author

eedwardsdisco commented May 3, 2016

Getting bit again by this parsing issue in yet ANOTHER way. I seem to be unable to insert $! when using remote-exec.

provisioner "remote-exec" {
    inline = [
        "sudo tee ${var.chef_client_tmp_path}/knife.sh > /dev/null <<KNIFESCRIPT",
        "#!/bin/bash",
        "knife serve -c ${var.chef_client_tmp_path}/client.rb --chef-zero-host ${aws_instance.chef_server.private_ip} > ${var.chef_client_tmp_path}/knife-serve.out 2> ${var.chef_client_tmp_path}/knife-serve.err < /dev/null & echo \"$!\" > ${var.chef_client_tmp_path}/pid",
        "KNIFESCRIPT",
        "sudo chmod 700 ${var.chef_client_tmp_path}/knife.sh"
    ]
}

Notably, this section near the end of the long command string: & echo \"$!\"

When terraform parses it and runs it in the .sh file that it creates on the target server, the echo contains an empty string (e.g. & echo "").

I've tried escaping it as per $$! as well as \$!, and the double-dollar sign didn't work, while the backslash is considered an invalid escape character.

#4747 Related?

Edit:

I found a workaround for this specific issue by double-escaping the backslash:

& echo \\$!

@eedwardsdisco
Copy link
Author

I hate to bump on this, but the low level parsing library is a pretty core part of Terraform and I'm stumped on how to fix this (I'm not enough of a wizard to figure it out, I tried...).

Who did the majority work on implementing the new HCL parser a few versions ago? I believe this is when the problem was introduced, and I'm not sure who else can figure it out.

@eedwardsdisco eedwardsdisco changed the title unable to set consul key with backslash or dollar sign from input variable hcl parser escape behavior is broken, cannot use variables or functions with content that requires escaping May 23, 2016
phinze added a commit to hashicorp/hcl that referenced this issue Jun 6, 2016
Fixes hashicorp/terraform#6359 by unquoting a
double backslash within braces to a single backslash.
phinze added a commit to hashicorp/hcl that referenced this issue Jun 7, 2016
Fixes hashicorp/terraform#6359 and several
related issues by unescaping a double backslash within braces to a
single backslash.

This bumps into the bit of HIL that still hangs out in HCL - for values
that aren't interpolated, like Terraform's variable defaults - users
were unable to get a backslash to show up within `${}`.

That's because `${}` is handled specially to allow for, e.g., double
quotes inside of braces.

Here, we add `\\` as a special cased escape (along with `\"`) so users
can get backslashes in these scenarios by doubling them up.
@eedwardsdisco
Copy link
Author

@phinze you're my hero!

phinze added a commit that referenced this issue Jun 7, 2016
@phinze
Copy link
Contributor

phinze commented Jun 7, 2016

😀 👍

Thanks for all the documentation on this issue @eedwardsdisco! I believe I've addressed a common denominator bug that allows workarounds for nearly all of the examples mentioned here. I'm going to file breakout issues for the remaining things from this backlog that are problematic.

@eedwardsdisco
Copy link
Author

Yup I looked at your commits and saw you used some of my example problems as test cases, much appreciated! Super excited about this fix.

@levenaux
Copy link

Terraform Version

0.7.2

it appears I'm having this problem in the latest version

resource "template_file" "parse_test" {
    template = "this \string \contains \backslashes"
}

This will produce the following error: main.tf: At 2:31: illegal char escape

Changing this to escape the backslash seems to keep both

resource "template_file" "parse_test" {
    template = "this \\string \\contains \\backslashes"
}
template_file.parse_test: Creating...
  rendered: "" => "<computed>"
  template: "" => "this \\string \\contains \\backslashes"

@eedwardsdisco
Copy link
Author

@phinze is this a regression?

I was hoping this case was fixed as I have a project I've been blocked on finishing due to this parsing issue.

@mrproper
Copy link

mrproper commented Sep 8, 2017

This is still present in terraform v10.4
variable "uris" { "default" = "/v1/foo/\\d+/bar" }
without the \ doing \d+ complains in illegal char
with the \ it applies the value to the resource with a literal \

This issue seems to keep resurfacing

@Spechal
Copy link

Spechal commented Mar 15, 2018

variable "partition_key" { default = "$${topic()}" }

Outputs $${topic()} ... should output ${topic()}

variable "partition_key" { default = "\${topic()}" }
Outputs: illegal char escape

variable "partition_key" { default = "\$${topic()}" }
Outputs invalid char

variable "partition_key" { default = "${topic()}" }
Outpus: Error: variable "partition_key": default may not contain interpolations

`Terraform v0.11.2

  • provider.archive v0.1.0
  • provider.aws v1.11.0
  • provider.null v1.0.0
  • provider.template v1.0.0
  • provider.tls v1.1.0`

@ghost
Copy link

ghost commented Apr 4, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants