Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IOError error="closed stream" with fluentd-1.2.5/lib/fluent/plugin/buffer/file_chunk.rb:83 #2391

Closed
qingling128 opened this issue Apr 18, 2019 · 14 comments
Closed

IOError error="closed stream" with fluentd-1.2.5/lib/fluent/plugin/buffer/file_chunk.rb:83 #2391

qingling128 opened this issue Apr 18, 2019 · 14 comments

Comments

@qingling128
Copy link

Fluentd version: v1.2.5

Environment: Fluentd within a container running inside Kubernetes

Docker base image: debian:9.6-slim

Problem
We're seeing the following errors intermittently.

2019-04-18 22:14:18 +0000 [warn]: #18 emit transaction failed: error_class=IOError error="closed stream" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer/file_chunk.rb:83:in `pos'" tag="node-journal"
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer/file_chunk.rb:83:in `pos'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer/file_chunk.rb:83:in `rollback'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer.rb:325:in `rescue in block in write'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer.rb:318:in `block in write'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer.rb:317:in `each'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/buffer.rb:317:in `write'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/output.rb:865:in `write_guard'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/output.rb:855:in `execute_chunking'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/output.rb:778:in `emit_buffered'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/in_forward.rb:320:in `on_message'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/server.rb:599:in `on_read_without_connection'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-18 22:14:18 +0000 [warn]: #18 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-18 22:14:18 +0000 [error]: #18 unexpected error on reading data host="10.40.24.2" port=42554 error_class=IOError error="closed stream"

It happened every now and then, but did not seem to crash the agent or affect most of the logs being ingested. It did cause some logs to be dropped.

Any idea what might cause this? Note that we are using multi-worker mode. Is there some race condition among the workers?

Config

<source>
  @type forward
  # default port
  port 24224
  # only accept connections from localhost - to open this up, change to 0.0.0.0
  bind 127.0.0.1
</source>

<system>
  workers 20
  root_dir /stackdriver-log-aggregator-persistent-volume
</system>

# Do not collect fluentd's own logs to avoid infinite loops.
<match fluent.**>
  @type null
</match>

<match **>
  @type google_cloud
  @id google_cloud

  # Try to detect JSON formatted log entries.
  detect_json true
  # Allow log entries from multiple containers to be sent in the same
  # request.
  split_logs_by_tag false
  <buffer>
    # Set the buffer type to file to improve the reliability and reduce the
    # memory consumption.
    @type file
    # The max size of each chunks: events will be written into chunks until
    # the size of chunks become this size
    # Set the chunk limit conservatively to avoid exceeding the recommended
    # chunk size of 5MB per write request.
    chunk_limit_size 512k
    # Limit the number of queued chunks.
    queued_chunks_limit_size 20
    # Block processing of input plugin to emit events into that buffer.
    overflow_action block
    # Never wait more than 5 seconds before flushing logs in the non-error
    # case.
    flush_interval 5s
    # Seconds to wait before next retry to flush.
    retry_wait 5s
    # The base number of exponential backoff for retries.
    retry_exponential_backoff_base 5
    # The maximum interval seconds for exponential backoff between retries
    # while failing.
    retry_max_interval 1200
  </buffer>
  # Use multiple threads for processing.
  num_threads 2
  use_grpc true
  project_id ...
  k8s_cluster_name ...
  k8s_cluster_location ...
  adjust_invalid_timestamps false
</match>
@repeatedly
Copy link
Member

repeatedly commented Apr 22, 2019 •
edited
Loading

From the configuration, multi workers should not read same file because buffer file paths are different.
Each worker has own path based on worker id.
Fluentd changed internal flush mechanizm since v1.3 to fix weird behaviour of output/buffer so I want to know this problem happens with latest version or not.

@qingling128
Copy link
Author

Good to know! I'll upgrade to v1.3 and see if we still see this issue.

@qingling128
Copy link
Author

Closing this issue for now. If I see it again in versions >= v1.3, I'll reopen then.

@qingling128 qingling128 mentioned this issue Apr 23, 2019
Merged
@qingling128
Copy link
Author

@repeatedly - Turned out that we are still seeing this in Fluentd version v1.4.2:

2019-04-25 06:24:59 +0000 [warn]: #11 emit transaction failed: error_class=IOError error="closed stream" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:83:in `pos'" tag="node-journal"
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:83:in `pos'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:83:in `rollback'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:354:in `rescue in block in write'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:347:in `block in write'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:346:in `each'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:346:in `write'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-25 06:24:59 +0000 [warn]: #11 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-25 06:24:59 +0000 [error]: #11 unexpected error on reading data host="10.24.54.2" port=46166 error_class=IOError error="closed stream"
  2019-04-25 06:24:59 +0000 [error]: #11 suppressed same stacktrace

@qingling128 qingling128 reopened this Apr 25, 2019
@repeatedly
Copy link
Member

I see. So this problem is reproducible on your environment.
Could you apply following patch to show actual error?

diff --git a/lib/fluent/plugin/buffer.rb b/lib/fluent/plugin/buffer.rb
index 690290a1..6eec54b5 100644
--- a/lib/fluent/plugin/buffer.rb
+++ b/lib/fluent/plugin/buffer.rb
@@ -351,6 +351,7 @@ module Fluent
               end
               chunk.mon_exit
             rescue => e
+              log.warn e.inspect
               chunk.rollback
               chunk.mon_exit
               errors << e

@qingling128
Copy link
Author

Let me try that.

@qingling128
Copy link
Author

Seeing the following errors after adding "log.warn e.inspect":

2019-04-26 06:16:17 +0000 [warn]: #8 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker8/google_cloud/buffer/buffer.b58768df4495e4a892cfbe4c125823117.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="stdout"
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write_metadata'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:71:in `commit'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:334:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
2019-04-26 06:16:17 +0000 [warn]: #17 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker17/google_cloud/buffer/buffer.b58768df50c9b2fe8b88d99536e05ed2d.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="stdout"
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:17 +0000 [warn]: #8 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `concat'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/chunk.rb:75:in `append'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:601:in `block in write_once'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:589:in `write_once'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:311:in `block in write'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:17 +0000 [warn]: #17 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-26 06:16:17 +0000 [warn]: #8 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker8/google_cloud/buffer/buffer.b58768df4495e4a892cfbe4c125823117.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-bzk6g_test-components_stackdriver-k8s-tests-logging-load-test-0f643a08417e93609d310e5fa5a6d83ae70ddc5f3c83e71bb91005047a748171.log"
  2019-04-26 06:16:17 +0000 [warn]: #8 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [error]: #8 unexpected error on reading data host="10.60.2.3" port=37848 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker8/google_cloud/buffer/buffer.b58768df4495e4a892cfbe4c125823117.log.meta"
2019-04-26 06:16:17 +0000 [warn]: #17 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker17/google_cloud/buffer/buffer.b58768df50c9b2fe8b88d99536e05ed2d.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-hxlpm_test-components_stackdriver-k8s-tests-logging-load-test-9d14898fc2ab8103d8abe3fe38bf8901d4d9d0734d31a734f4a1f0368cfee15b.log"
  2019-04-26 06:16:17 +0000 [error]: #8 suppressed same stacktrace
  2019-04-26 06:16:17 +0000 [warn]: #17 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [error]: #17 unexpected error on reading data host="10.60.32.2" port=33238 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker17/google_cloud/buffer/buffer.b58768df50c9b2fe8b88d99536e05ed2d.log"
  2019-04-26 06:16:17 +0000 [error]: #17 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [warn]: #13 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker13/google_cloud/buffer/buffer.b58768df35ef69ed858f8c4b4f381a279.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="stdout"
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `concat'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/chunk.rb:75:in `append'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:601:in `block in write_once'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:589:in `write_once'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:311:in `block in write'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:17 +0000 [warn]: #13 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-26 06:16:17 +0000 [warn]: #13 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker13/google_cloud/buffer/buffer.b58768df35ef69ed858f8c4b4f381a279.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-nzplf_test-components_stackdriver-k8s-tests-logging-load-test-f244040abc72d4141da04ed00ea982698b1187821e35392a3ed0c551f9ab0850.log"
  2019-04-26 06:16:17 +0000 [warn]: #13 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [error]: #13 unexpected error on reading data host="10.60.48.2" port=47870 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker13/google_cloud/buffer/buffer.b58768df35ef69ed858f8c4b4f381a279.log"
  2019-04-26 06:16:17 +0000 [error]: #13 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [warn]: #12 [google_cloud] failed to write data into buffer by buffer overflow action=:block
2019-04-26 06:16:17 +0000 [warn]: #23 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker23/google_cloud/buffer/buffer.b58768df33d627ca63e34b6e063f99265.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="stdout"
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write_metadata'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:71:in `commit'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:334:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:17 +0000 [warn]: #23 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-26 06:16:17 +0000 [warn]: #23 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker23/google_cloud/buffer/buffer.b58768df33d627ca63e34b6e063f99265.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-b7xg6_test-components_stackdriver-k8s-tests-logging-load-test-5306746d4611ef45b5d9c29120cebf433c6fd957fa5b3a212e73da099b78563f.log"
  2019-04-26 06:16:17 +0000 [warn]: #23 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [error]: #23 unexpected error on reading data host="10.60.42.2" port=41288 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker23/google_cloud/buffer/buffer.b58768df33d627ca63e34b6e063f99265.log.meta"
  2019-04-26 06:16:17 +0000 [error]: #23 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [warn]: #27 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker27/google_cloud/buffer/buffer.b58768df3b6000a46777b669c4e478c48.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="stdout"
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `concat'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/chunk.rb:75:in `append'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:601:in `block in write_once'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:589:in `write_once'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:311:in `block in write'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `write'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:17 +0000 [warn]: #27 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-26 06:16:17 +0000 [warn]: #27 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker27/google_cloud/buffer/buffer.b58768df3b6000a46777b669c4e478c48.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-cgs8m_test-components_stackdriver-k8s-tests-logging-load-test-fe8066e5e3ecc37e0d65f6d279951335458c34389b44371ffa73816f82d2f71a.log"
  2019-04-26 06:16:17 +0000 [warn]: #27 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [error]: #27 unexpected error on reading data host="10.60.14.3" port=37504 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker27/google_cloud/buffer/buffer.b58768df3b6000a46777b669c4e478c48.log"
  2019-04-26 06:16:17 +0000 [error]: #27 suppressed same stacktrace
2019-04-26 06:16:17 +0000 [info]: Worker 1 finished unexpectedly with signal SIGKILL
2019-04-26 06:16:17 +0000 [info]: #22 [google_cloud] Successfully sent gRPC to Stackdriver Logging API.
2019-04-26 06:16:17 +0000 [info]: #16 listening port port=8989 bind="0.0.0.0"
2019-04-26 06:16:17 +0000 [warn]: #0 [google_cloud] failed to write data into buffer by buffer overflow action=:block
2019-04-26 06:16:17 +0000 [info]: #21 [google_cloud] Successfully sent gRPC to Stackdriver Logging API.
2019-04-26 06:16:17 +0000 [info]: #16 fluentd worker is now running worker=16
2019-04-26 06:16:17 +0000 [info]: #25 [google_cloud] Successfully sent gRPC to Stackdriver Logging API.
2019-04-26 06:16:18 +0000 [info]: Worker 0 finished unexpectedly with signal SIGKILL
2019-04-26 06:16:20 +0000 [info]: Worker 5 finished unexpectedly with signal SIGKILL
2019-04-26 06:16:20 +0000 [warn]: #28 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker28/google_cloud/buffer/buffer.b58768df7d2dd7da87aa1196c5cb8f921.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="stdout"
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `concat'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/chunk.rb:75:in `append'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:601:in `block in write_once'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:589:in `write_once'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:311:in `block in write'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `each'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:309:in `write'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:20 +0000 [warn]: #28 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-26 06:16:20 +0000 [warn]: #28 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker28/google_cloud/buffer/buffer.b58768df7d2dd7da87aa1196c5cb8f921.log" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:64:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-nt4m4_test-components_stackdriver-k8s-tests-logging-load-test-afe0a3b8055d701f9ee2577beb1a021b7996ac7ad2a30f4ec054e714ab25e113.log"
  2019-04-26 06:16:20 +0000 [warn]: #28 suppressed same stacktrace
2019-04-26 06:16:20 +0000 [error]: #28 unexpected error on reading data host="10.60.8.3" port=40344 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker28/google_cloud/buffer/buffer.b58768df7d2dd7da87aa1196c5cb8f921.log"
  2019-04-26 06:16:20 +0000 [error]: #28 suppressed same stacktrace
2019-04-26 06:16:20 +0000 [warn]: #22 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker22/google_cloud/buffer/buffer.b58768df7aa7590b36475907e84dcd48f.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="stdout"
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write_metadata'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:71:in `commit'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer.rb:334:in `write'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:374:in `block in handle_stream_simple'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:880:in `write_guard'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/compat/output.rb:373:in `handle_stream_simple'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:870:in `execute_chunking'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:793:in `emit_buffered'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:115:in `block in process'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `each'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/out_record_modifier.rb:114:in `process'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:782:in `emit_sync'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:327:in `on_message'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:218:in `block in handle_connection'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:255:in `block (3 levels) in read_messages'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `feed_each'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:254:in `block (2 levels) in read_messages'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/in_forward.rb:263:in `block in read_messages'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/server.rb:612:in `on_read_without_connection'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-04-26 06:16:20 +0000 [warn]: #22 /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-26 06:16:20 +0000 [warn]: #22 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker22/google_cloud/buffer/buffer.b58768df7aa7590b36475907e84dcd48f.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="k8s_container.logging-load-test-gke-patch-6-admin-default-pool-c269fa71-fkpks_test-components_stackdriver-k8s-tests-logging-load-test-2e38f020f1c9f84eb254da95c3ccef1eadcba8bec98fa81e51539c497754f87a.log"
  2019-04-26 06:16:20 +0000 [warn]: #22 suppressed same stacktrace
2019-04-26 06:16:20 +0000 [error]: #22 unexpected error on reading data host="10.60.4.3" port=50274 error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker22/google_cloud/buffer/buffer.b58768df7aa7590b36475907e84dcd48f.log.meta"
  2019-04-26 06:16:20 +0000 [error]: #22 suppressed same stacktrace
2019-04-26 06:16:21 +0000 [info]: #16 [google_cloud] Successfully sent gRPC to Stackdriver Logging API.

@repeatedly - Does this look familiar?

@qingling128
Copy link
Author

We are seeing such errors from time to time. There were a lot of them especially when the agent first started up. After a period of time, they are less likely to happen.

@onjectiondevops
Copy link

I think this issue occurs becase system goes out of memory(Cannot allocate memory ) by refer below logs:

2019-04-26 06:16:20 +0000 [warn]: #22 emit transaction failed: error_class=Errno::ENOMEM error="Cannot allocate memory @ io_write - /stackdriver-log-aggregator-persistent-volume/worker22/google_cloud/buffer/buffer.b58768df7aa7590b36475907e84dcd48f.log.meta" location="/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/file_chunk.rb:251:in `write'" tag="stdout"

@qingling128
Copy link
Author

That was my first hunch too. We allocated 4GB memory to it alreadythough, so I was wondering if there's any deadlocks. I'll try raising that threshold then see if we still get errors.

@qingling128
Copy link
Author

Seems like we need +4GB allocated memory for 20 workers (that was a bit surprising). After I reduced the worker number to 10, it seems to be fine with 4GB memory, and I'm no longer seeing this issue. Closing this ticket.

@jkohen
Copy link
Contributor

jkohen commented Jan 9, 2020

I just ran into the same problem for the same reason, and it was hard to debug. Can we make the error message more visible?

@sophieyfang
Copy link

+1 on jkohen's request

@stanhu stanhu mentioned this issue Oct 16, 2024
Open
@stanhu
Copy link

stanhu commented Oct 16, 2024

FYI, I'm seeing intermittent closed stream errors in file_chunk.rb as well with fluentd v1.16.5. It appears to lead to a potential leak at least on the metrics side: #2391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@repeatedly @stanhu @qingling128 @jkohen @sophieyfang @onjectiondevops