Token timeout when using profile to authenticate

[geissbock]

I use the following Logstash config on an EC2 instance:

input {
  syslog {
    type => "syslog"
    port => 5514
  }
}

output {
  amazon_es {
      hosts => ["..."]
      region => "eu-central-1"
  }
}

There are no AWS credentials provided on the EC2 instance, i.e. the machine's profile is used to authenticate/sign the requests to the AWS Elasticsearch instance. This works fine - for some hours. After some time, the log file gets flooded with such messages:

{
:timestamp=>"2015-10-15T08:49:17.699000+0200",
:message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://...:80\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?",
:client_config=>{:hosts=>["http://...:80"], :region=>"eu-central-1", :aws_access_key_id=>nil, :aws_secret_access_key=>nil, :transport_options=>{:request=>{:open_timeout=>0, :timeout=>60}, :proxy=>nil}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::AWS, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false},
:error_message=>"[403] {\"message\":\"The security token included in the request is expired\"}",
:error_class=>"Elasticsearch::Transport::Transport::Errors::Forbidden",
:backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in `__raise_transport_error'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-0.1.0-java/lib/logstash/outputs/amazon_es/aws_transport.rb:45:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/bulk.rb:80:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-0.1.0-java/lib/logstash/outputs/amazon_es/http_client.rb:53:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-0.1.0-java/lib/logstash/outputs/amazon_es.rb:319:in `submit'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-0.1.0-java/lib/logstash/outputs/amazon_es.rb:316:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-0.1.0-java/lib/logstash/outputs/amazon_es.rb:349:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:216:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:193:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:159:in `buffer_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-0.1.0-java/lib/logstash/outputs/amazon_es.rb:309:in `receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/outputs/base.rb:88:in `handle'", "(eval):27:in `output_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:244:in `outputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:166:in `start_outputs'"],
:level=>:error
}

The error message "The security token included in the request is expired" looks a lot like the security token not getting refreshed properly.

I'd be glad to provide a pull request to fix this issue, however, I'm not really a Ruby guy. So if someone could give me hint where to start looking for the problem, any help is appreciated!

[geissbock]

After some more resarch I assume that the problem lies in the AWS SDK for Ruby: aws/aws-sdk-ruby#961

[mjallday]

👍 seeing the same thing.

[geissbock]

The guys working on the AWS SDK for Ruby added an explanation, and they also recommended a workaround: aws/aws-sdk-ruby#961 (comment)

Could anyone please comment on this and evaluate if this could be done easily in this plugin? Cheers!

[tobiasbayer]

Same here...

[malpani]

Thank you @geissbock and @tobiasbayer : Hope th ework around in 0.1.2 version fixes this for you