Security Channel log -- ErrorCode: 14

[sophieyfang]

One of our clients encountered this error. (Sorry that i can't reproduce it in my side.) The client has increased the memory from 8GB to 16 GB and now 32GB. But still it got errorCode:14 which is Out of Memory error code in Windows. Also It affects only Security channel. All other channels' logs flow in without trouble.. I looked through the code and this is the stack-trace that i think it might be happening during the crash(only security channel crashed). I wonder how could one single eventlog entry consumed that much memory considering the client has increased it from 8->16->32GB but still sees the error. Alongside there was always a memory spike when the crash happened

(Note: ordering is in invoking sequence)

• Starting from subscribe.each

• Then rb_winevt_subscribe_each is invoked by subscribe.each

• Then rb_winevt_subscribe_each_yield is invoked

• Then rb_winevt_subscribe_message is invoked

• Then get_description is invoked

• Then get_message is invoked inside get_description

• Then EvtFormatMessage is invoked but returns ERROR_OUTOFMEMORY (ErrorCode: 14)

• Finally raise an error here

Error Message:

  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:322:in `each'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:322:in `on_notify_hash'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:249:in `block (2 levels) in subscribe_channels'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluentd-1.11.2-x86-mingw32/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/cool.io-1.7.0/lib/cool.io/loop.rb:88:in `run_once'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/cool.io-1.7.0/lib/cool.io/loop.rb:88:in `run'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluentd-1.11.2-x86-mingw32/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluentd-1.11.2-x86-mingw32/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

[cosmo0920]

I'm wondering why winevt_c causes ERROR_OUTOFMEMORY in these days....
Today, I noticed that, perhaps, your client uses 32bit version of Ruby or C extensions?

2020-12-04 11:07:25 -0800 [warn]: #0 C:/Program Files (x86)/Stackdriver/LoggingAgent/Main/lib/ruby/gems/2.6.0/gems/fluentd-1.11.2-x86-mingw32/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

This means that Ruby process is working on 32bit memory spaces.

32bit process has a limitation for memory:

• 2GB as maximum per process.

ref: https://docs.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases?redirectedfrom=MSDN

I'm not sure why StackDriver Logging agent is still staying on 32bit process.
Could you ask Stackdriver technical support to provide 64bit version of its agent installer?