Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using comments in the add_field section causes syntax error #137

Open
Kvrepo opened this issue May 31, 2019 · 1 comment
Open

Using comments in the add_field section causes syntax error #137

Kvrepo opened this issue May 31, 2019 · 1 comment

Comments

@Kvrepo
Copy link

Kvrepo commented May 31, 2019

In Logstash 6.3.0 I've noticed the following phenomenon:

I have configuration A and configuration B. I run them with the following command:

/usr/share/logstash/bin/logstash -f /home/user/A.cf --path.data="/srv"

/usr/share/logstash/bin/logstash -f /home/user/B.cf --path.data="/srv"

Configuration A:

input {
    stdin {}
}

filter {

    mutate {

        add_field => {
            # Comment 1
            "some_field_1" => "some value"
            
            # Comment 2
            "some_field_2" => "some value"
        }

    }

}

output {
    stdout {}
}

Configuration A yields the following error:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-31 12:58:29.124 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-31 12:58:30.131 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.0"}
[ERROR] 2019-05-31 12:58:31.253 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 14, column 13 (byte 188) after filter {\n\n    mutate {\n\n        add_field => {\n            # Comment 1\n            \"some_field_1\" => \"some value\"\n            \n            # Comment 2\n            ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

Configuration B:

input {
    stdin {}
}

filter {

    mutate {

        add_field => {
            # Comment 1
            "some_field_1" => "some value"
            
            "some_field_2" => "some value"
        }

    }

}

output {
    stdout {}
}

Configuration B yields no errors:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-31 12:59:49.383 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-31 12:59:50.726 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.0"}
[INFO ] 2019-05-31 12:59:59.327 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-05-31 12:59:59.594 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x395ce82 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2019-05-31 12:59:59.730 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-05-31 13:00:00.123 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9603}
Test
{
      "@timestamp" => 2019-05-31T11:00:04.690Z,
    "some_field_1" => "some value",
    "some_field_2" => "some value",
         "message" => "Test",
        "@version" => "1",
            "host" => "some.host.com"
}

The only difference between the two configurations is this part:

A:

        add_field => {
            # Comment 1
            "some_field_1" => "some value"
            
            # Comment 2
            "some_field_2" => "some value"
        }

B:

        add_field => {
            # Comment 1
            "some_field_1" => "some value"
            
            "some_field_2" => "some value"
        }

Am I right to assume that you can only use comments at the "beginning" of an add_field section, but not in the middle? Is it a bug or a feature?

Of course the obvious solution to this issue is avoiding these kinds of comments, but I wanted to gather some plus information about this before moving onward.

Udate: We have examined the issue further and come to the conclusion that this issue is present in every plugin which has a hash as a parameter (split, translate etc.).

@voilaviola
Copy link

This problem persists in version 7.8.1 with rename plugin. The following configurations will trigger a syntax error :

mutate {
    rename => {
        "column1" => "field_name" # Comment
    }
}

And

mutate {
    rename => {
        "column1" => "field_name" 
        # Comment
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants