Environment variables sometimes aren't working?

[jordansissel]

I'm not sure what causes this, but it's easy to reproduce. Reproduced only with Elasticsearch output but haven't tried other plugins.

for openers:

• Environment variables seem to work fine, and we have tests for them.
• Except in this one case:

% ES_HOST=localhost bin/logstash -e 'output { elasticsearch { hosts => "${ES_HOST}" } }'
[2017-02-13T02:01:56,198][ERROR][logstash.agent           ] fetched an invalid config {:config=>"output { elasticsearch { hosts => \"${ES_HOST}\" } }input { stdin { type => stdin } }", :reason=>"bad URI(is not URI?): ${ES_HOST}"}

The specific part of the error for focus: bad URI(is not URI?): ${ES_HOST}"}

I tested other settings (heartbeat_path) and env vars are working there.

[pigri]

+1

[ValentinOdier]

Hi,

We have this issue too and we think it's a bug that appeared between logstash 5.1.1 and logstash 5.2.2

Using logstash 5.2.2

15:22:34.214 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@hosts = [http://elasticsearch:9200]
15:22:34.214 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@index = "datascience-testing"
15:22:34.214 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@flush_size = 250
15:22:34.214 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@id = "cbf411562ac0e019174e618a8f7c7ad6fdbb21a3-2"
15:22:34.214 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@enable_metric = true
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain id=>"plain_c60ee176-dc79-4533-9be5-683eb2e8a935", enable_metric=>true, charset=>"UTF-8">
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@workers = 1
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@manage_template = true
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@template_name = "logstash"
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@template_overwrite = false
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@parent = nil
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@idle_flush_time = 1
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@upsert = ""
15:22:34.215 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@script = ""
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@script_type = "inline"
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@script_lang = "painless"
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@script_var_name = "event"
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@scripted_upsert = false
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@retry_initial_interval = 2
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@retry_max_interval = 64
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@retry_on_conflict = 1
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@pipeline = nil
15:22:34.216 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@action = "index"
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@ssl_certificate_verification = true
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@sniffing = false
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@sniffing_delay = 5
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@timeout = 60
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@failure_type_logging_whitelist = []
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@pool_max = 1000
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@pool_max_per_route = 100
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@healthcheck_path = "/"
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@absolute_healthcheck_path = false
15:22:34.217 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@resurrect_delay = 5
15:22:34.218 [LogStash::Runner] DEBUG logstash.outputs.elasticsearch - config LogStash::Outputs::ElasticSearch/@validate_after_inactivity = 10000
15:22:34.224 [LogStash::Runner] FATAL logstash.runner - The given configuration is invalid. Reason: bad URI(is not URI?): ${ES_HOST}

Using logstash 5.1.1 it works.

conf :

elasticsearch {
      hosts => "${ES_HOST}"
      index => "logstash-%{+YYYY.MM.dd}"
    }

ES_HOST="http://elasticsearch:9200"

[aloisbarreras]

+1

[gleehokie]

+1 and I repeated this issue with logstash version 5.3.0

[ianks]

I can recreate this issue on 5.1 with the twitter plugin.

input {
  twitter {
     // this doesn't work
    consumer_key => "${TWITTER_CONSUMER_KEY}"
    consumer_secret => "${TWITTER_CONSUMER_SECRET}"
    oauth_token => "${TWITTER_OAUTH_TOKEN}"
    oauth_token_secret => "${TWITTER_OAUTH_TOKEN_SECRET}"
    keywords => ["adhawk"]
    full_tweet => true
    type => "twitter"
  }
}

output {
  if [type] == "twitter" {
    elasticsearch {
      // this works
      hosts => ["${ELASTICSEARCH_URL}"]
      template => "/usr/share/logstash/templates/twitter.json"
      template_name => "twitter"
    }
  }
}

[tedder]

I wonder if it's only the output->elasticsearch->hosts param, for whatever reason.

[ianks]

You may be correct @tedder. IME, it's confusing if the parser does not do a variable substitution before passing off the data structure to plugins.

[jordansissel]

@ianks it does do variable substitution before passing to the plugins, which is why I am so confused how this even happens. Very strange bug.

[jordansissel]

@suyograo I've marked this a blocker for 5.4.0. I'm hoping it's easy to fix.

[gcardy]

@ianks example to recreate the issue in 5.1, fixes things for me in 5.4

elasticsearch {
hosts => ["${OUTPUT_ES_HOST}"]
index => 'logstash-%{+YYYY.MM.dd.HH}'
}

bash-4.3# logstash -f /config-dir
Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
18:54:32.310 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating >new UUID {:uuid=>"e1c37e95-6b07-4cdb-80d9-fb61b312b755", :path=>"/usr/share/logstash/data/uuid"}
18:54:32.466 [LogStash::Runner] INFO logstash.codecs.cloudwatchlogs - Using version 0.1.x codec plugin 'cloudwatch_logs'. This plugin isn't well supported by the community and likely has no maintainer.
18:54:32.938 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool >URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}

[jordansissel]

Logstash 5.4.0 was released without fixing this. Moved to 5.5

[original-brownbear]

@jordansissel so just to clarify the correct behavior here:

We are interpolating all environment variables in the config and require $ to be escaped in configs for that reason, correct?

[jordansissel]

@original-brownbear We don't support escapes in configs today, but our intent is to interpret environment variables. We added this some versions ago and at some point is went weird and sometimes -- for some plugin settings -- is evaluating them correctly.

[suyograo]

We are interpolating all environment variables in the config

@original-brownbear correct ^^

We only expand env variables if it has this syntax ${ENV_VAR}

[original-brownbear]

@jordansissel @suyograo thanks guys, let me try and fix it real quick then :)

[original-brownbear]

fixed in #7411