Installing kibana plugin fails due to node running out of heap memory

[blacktop]

Kibana version:
6.5
Elasticsearch version:
6.5
Server OS version:
NA
Browser version:
NA
Browser OS version:
docker cloud
Original install method (e.g. download page, yum, from source, etc.):

kibana-plugin install https://github.com/maliceio/malice-kibana-plugin/releases/download/v6.5.0/malice-6.5.0.zip

Describe the bug:
Installing kibana plugins in docker cloud fails I guess I can tell nodeJS in the container to limit it's heap? This just started happening with 6.5 though??

Steps to reproduce:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

---> ec24e5f549f8
Step 12/18 : RUN echo "===> Install Kibana Plugin..." && kibana-plugin install https://github.com/maliceio/malice-kibana-plugin/releases/download/v6.5.0/malice-6.5.0.zip
---> Running in 9df3e26866f5
===> Install Kibana Plugin...
Attempting to transfer from https://github.com/maliceio/malice-kibana-plugin/releases/download/v6.5.0/malice-6.5.0.zip
Transferring 89905 bytes....
...........
.....
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation was unsuccessful due to error "Command failed: /usr/local/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
(node:18) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
{"type":"log","@timestamp":"2018-11-17T23:54:21Z","tags":["info","optimize"],"pid":18,"message":"Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra, malice and kibana. This may take a few minutes"}
<--- Last few GCs --->
[18:0x563fcc154000] 2509062 ms: Mark-sweep 1310.9 (1461.8) -> 1310.9 (1462.3) MB, 4216.0 / 0.0 ms allocation failure GC in old space requested
[18:0x563fcc154000] 2513390 ms: Mark-sweep 1310.9 (1462.3) -> 1310.9 (1418.3) MB, 4327.9 / 0.0 ms last resort GC in old space requested
[18:0x563fcc154000] 2516658 ms: Mark-sweep 1310.9 (1418.3) -> 1310.9 (1416.3) MB, 3268.6 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x260396425879 <JSObject>
0: builtin exit frame: lastIndexOf(this=0x177ba6cfc091 <Very long string[6713194]>,0x25e34620ca51 <String[1]\: \n>)
1: has_nlb(aka has_nlb) [0x14f8eb6022d1 <undefined>:5970] [bytecode=0x3043773f6a11 offset=15](this=0x14f8eb6022d1 <undefined>)
2: prepend_comments [0x14f8eb6022d1 <undefined>:~5974] [pc=0x149a2086919f](this=0x16c7478f6809 <Object map = 0x1909dfcf05c9>,node=0x1e14334b06...
"

Expected behavior:
should install

[zacbayhan]

I've had similar problems while kibana was "Optimizing and caching bundles"

Attaching to docker_kibana-monitoring_1_874c3e9757b4
kibana-monitoring_1_874c3e9757b4 | {"type":"log","@timestamp":"2018-11-16T20:33:54Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, dashboardViewer, apm, canvas, infra and kibana. This may take a few minutes"}
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | <--- Last few GCs --->
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | [1:0x25c8b90]   912294 ms: Mark-sweep 1305.6 (1459.4) -> 1305.5 (1460.4) MB, 1461.0 / 0.0 ms  allocation failure GC in old space requested
kibana-monitoring_1_874c3e9757b4 | [1:0x25c8b90]   913757 ms: Mark-sweep 1305.5 (1460.4) -> 1305.5 (1420.9) MB, 1463.3 / 0.0 ms  last resort GC in old space requested
kibana-monitoring_1_874c3e9757b4 | [1:0x25c8b90]   914894 ms: Mark-sweep 1305.5 (1420.9) -> 1305.5 (1420.9) MB, 1137.1 / 0.0 ms  last resort GC in old space requested
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | <--- JS stacktrace --->
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | ==== JS stack trace =========================================
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | Security context: 0x1c5bde9a5879 <JSObject>
kibana-monitoring_1_874c3e9757b4 |     0: builtin exit frame: lastIndexOf(this=0x2fe823248a89 <Very long string[3136923]>,0x1fbf0770ca51 <String[1]\: \n>)
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 |     1: has_nlb(aka has_nlb) [0x34f0da7022d1 <undefined>:5970] [bytecode=0x20a0261a5119 offset=15](this=0x34f0da7022d1 <undefined>)
kibana-monitoring_1_874c3e9757b4 |     2: /* anonymous */(aka /* anonymous */) [0x34f0da7022d1 <undefined>:6070] [bytecode=0x985aec4d1 offset=60](this=0x34f0da7022d1 <undefined>,c=0...
kibana-monitoring_1_874c3e9757b4 | 
kibana-monitoring_1_874c3e9757b4 | FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
kibana-monitoring_1_874c3e9757b4 |  1: node::Abort() [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  2: 0x8c21ec [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  7: v8::internal::String::Flatten(v8::internal::Handle<v8::internal::String>, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 |  9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/bin/../node/bin/node]
kibana-monitoring_1_874c3e9757b4 | 10: 0x16574088697d
docker_kibana-monitoring_1_874c3e9757b4 exited with code 139

[skearns64]

This is likely to be due to the optimize/compile process taking up more memory than the default Node.js heap that kibana specifies. You can specify the max-old-space-size on the command line when you install your plugin like this:

my-host:cwd$  NODE_OPTIONS="--max-old-space-size=4096" ./bin/kibana-plugin install https://github.com/maliceio/malice-kibana-plugin/releases/download/v6.5.0/malice-6.5.0.zip

If this is the cause, I'd propose that we close this issue, as we pursue the source of the large memory usage during plugin install in a number of different ways in other issues.

[elasticmachine]

Pinging @elastic/kibana-platform

[elasticmachine]

Pinging @elastic/kibana-operations

[blacktop]

I added NODE_OPTIONS="--max-old-space-size=4096" to my dockerfile and it is NOW working!!

[blacktop]

feel free to close this issue, but note that a lot of the CI/CD services don't have very beafy VMs building these docker images in the cloud so this will continue to be an issue for people who don't know this fix

[mistic]

@blacktop have you been able to solve the problem? We are working hardly to improve this, you can found more about a thing that should land soon in order to help us improve it here: #20749

[zacbayhan]

I was able to get kibana to run using the following in my docker-compose.yml file environment: - cluster.name=${CLUSTER_NAME} - NODE_OPTIONS=--max-old-space-size=4096

…

On Mon, Nov 26, 2018 at 6:06 AM Tiago Costa ***@***.***> wrote: @blacktop <https://github.com/blacktop> have you been able to solve the problem? We are working hardly to improve this, you can found more about a thing that should land soon here: #20749 <#20749> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#25849 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AO8y7vpRhAtwXpO4qVmezMtCzeZvOE-Oks5uy8segaJpZM4YoOKg> .

[blacktop]

whoops sorry I wrote not when I meant now, so the "workaround" fixed it for me.

[davidkarlsen]

Closed w/o any related commits?
Would it make sense to have this as a default in the Docker image?