Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kibana docker hangs our test on travis and jenkins #5953

Closed
ph opened this issue Dec 23, 2017 · 7 comments
Closed

Kibana docker hangs our test on travis and jenkins #5953

ph opened this issue Dec 23, 2017 · 7 comments
Labels
:Testing

Comments

@ph
Copy link
Contributor

ph commented Dec 23, 2017

We have been noticing a few hangs in our test suite which seems to come from the merge of the Kibana docker changes in #5928

---> 9efd55ef328a
Removing intermediate container 69e5eaebda55
Step 18/20 : USER kibana
 ---> Running in 1251f3fcec91
 ---> 5f05ea710ee4
Removing intermediate container 1251f3fcec91
Step 19/20 : RUN if [ ${XPACK} = "1" ]; then bin/kibana-plugin install ${DOWNLOAD_URL}/kibana-plugins/x-pack/x-pack-${ELASTIC_VERSION}.zip?c=${CACHE_BUST}; fi
 ---> Running in 4170c8755aa5
Attempting to transfer from https://snapshots.elastic.co/downloads/kibana-plugins/x-pack/x-pack-7.0.0-alpha1-SNAPSHOT.zip?c=20171201
Transferring 282481096 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated
@ph ph added the :Testing label Dec 23, 2017
@ph ph changed the title Kibana docker hangs our test Kibana docker hangs our test on travis and jenkins Dec 23, 2017
@ph
Copy link
Contributor Author

ph commented Dec 23, 2017

@ruflin could you take a look?

@ruflin
Copy link
Contributor

ruflin commented Dec 26, 2017

@ph I think this one is out of our control. It's potentially only indirectly related to #5928 The reasons this times out (especially on travis) is because the optimize process takes longer then 10 minutes on a machine with not too much processing power. I also see this on my local laptop from time to time. The problem could be circumvented if there would be an option that during the optimize some logs are outputed but there is no option for that in Kibana.

I would expect it to work on Jenkins except if Kibana snapshot is broken (which can happen from time to time) but is a good way to figure out issues early. I tried the most recent version locally and it seemed to build Kibana as expected (but it takes a long time). I will flush all images again and retry.

The way I'm hoping to solve this problem is that we will have the docker images for snapshots also available somewhere in a registry, meaning we don't have to build it ourself each time. This is potentially going to happen soon ;-)

@ruflin ruflin mentioned this issue Dec 27, 2017
Closed
@ruflin
Copy link
Contributor

ruflin commented Dec 27, 2017

On one of the most recent jenkins builds I got an output I haven't seen so far:

04:42:33 Optimizing and caching browser bundles...
04:54:06 
04:54:06 <--- Last few GCs --->
04:54:06 
04:54:06   728057 ms: Mark-sweep 1145.1 (1434.4) -> 1144.9 (1434.4) MB, 1185.9 / 0.0 ms [allocation failure] [GC in old space requested].
04:54:06   729247 ms: Mark-sweep 1144.9 (1434.4) -> 1144.9 (1434.4) MB, 1189.8 / 0.0 ms [allocation failure] [GC in old space requested].
04:54:06   730464 ms: Mark-sweep 1144.9 (1434.4) -> 1159.5 (1403.4) MB, 1217.2 / 0.0 ms [last resort gc].
04:54:06   731680 ms: Mark-sweep 1159.5 (1403.4) -> 1174.6 (1403.4) MB, 1216.2 / 0.0 ms [last resort gc].
04:54:06 
04:54:06 
04:54:06 <--- JS stacktrace --->
04:54:06 
04:54:06 ==== JS stack trace =========================================
04:54:06 
04:54:06 Security context: 0xb9a172cf781 <JS Object>
04:54:06     2: tighten_body(aka tighten_body) [0xb9a17204381 <undefined>:5927] [pc=0x2a6081645e3b] (this=0xb9a17204381 <undefined>,statements=0x2341005fdd21 <JS Array[58]>,compressor=0x26ca8e9dc951 <a Compressor with map 0x29ea6b776df9>)
04:54:06     3: /* anonymous */(aka /* anonymous */) [0xb9a17204381 <undefined>:~7209] [pc=0x2a608081b360] (this=0xb9a17204381 <undefined>,self=0x1c75255e90f9 <an AST_Function w...
04:54:06 
04:54:06 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
04:54:06  1: node::Abort() [bin/../node/bin/node]
04:54:06  2: 0x10a03bc [bin/../node/bin/node]
04:54:06  3: v8::Utils::ReportApiFailure(char const*, char const*) [bin/../node/bin/node]
04:54:06  4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bin/../node/bin/node]
04:54:06  5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [bin/../node/bin/node]
04:54:06  6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::MacroAssembler*, v8::internal::CompilationInfo*) [bin/../node/bin/node]
04:54:06  7: v8::internal::FullCodeGenerator::MakeCode(v8::internal::CompilationInfo*) [bin/../node/bin/node]
04:54:06  8: v8::internal::Compiler::EnsureDeoptimizationSupport(v8::internal::CompilationInfo*) [bin/../node/bin/node]
04:54:06  9: 0xb56147 [bin/../node/bin/node]
04:54:06 10: 0xb575bd [bin/../node/bin/node]
04:54:06 11: v8::internal::Compiler::CompileOptimized(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ConcurrencyMode) [bin/../node/bin/node]
04:54:06 12: v8::internal::Runtime_CompileOptimized_Concurrent(int, v8::internal::Object**, v8::internal::Isolate*) [bin/../node/bin/node]
04:54:06 13: 0x2a60797092a7
04:54:06 /bin/sh: line 1:     7 Aborted                 (core dumped) bin/kibana-plugin install ${DOWNLOAD_URL}/kibana-plugins/x-pack/x-pack-${ELASTIC_VERSION}.zip?c=${CACHE_BUST}
04:54:07 Service 'kibana' failed to build: The command '/bin/sh -c if [ ${XPACK} = "1" ]; then bin/kibana-plugin install ${DOWNLOAD_URL}/kibana-plugins/x-pack/x-pack-${ELASTIC_VERSION}.zip?c=${CACHE_BUST}; fi' returned a non-zero code: 134
04:54:07 make[1]: *** [build-image] Error 1

@elastic/kibana-operations Did anything recently change in Kibana which now requieres quite a bit more memory for the optimize step? The above is based on master / snapshot.

@ruflin
Copy link
Contributor

ruflin commented Dec 28, 2017

@elasticdog I'm kind of surprised to see the above as the jenkins slaves have more the 1.5GB of memory. Is there a limitation on how much memory each container can consume on our Jenkins slaves?

@ruflin
Copy link
Contributor

ruflin commented Dec 28, 2017

I'm wondering if it could be solved by using one of the flags here on the Kibana side: facebook/react-native#12686 But that does not explain what it builds on my machine but not (anymore) on the Jenkins worker.

@jbudz
Copy link
Member

jbudz commented Dec 28, 2017

@ruflin I believe someone pinged you on slack, but for posterity Kibana's right at the file limit for production bundling and we're starting to see it fail. We're working on it now and tracking at elastic/kibana#15683.

You're spot on with the flags, depending on how Kibana's being ran you'll want to either set the NODE_OPTIONS=--max_old_space_size=4096 or pass it in as an arugment to node, node --max-old-space-size=4096 scripts/kibana. The default for v8 is ~1.4gb which is why it's cutting off there.

@ruflin
Copy link
Contributor

ruflin commented Dec 29, 2017

@jbudz Thanks for your comment. I'm working on a fix here: #5966 Any chance I could get your feedback on a question I posted there?

@ruflin ruflin mentioned this issue Dec 29, 2017
Merged
ruflin added a commit to ruflin/beats that referenced this issue Jan 4, 2018
@ruflin
Improve Kibana docker image build
7d36a7d 
* Add `NODE_OPTIONS="--max-old-space-size=4096"` to circumvent elastic/kibana#15683
* Make it more obvious in the Docker file which parts are copied over and which are from the official repo
* Remove xpack.monitoring config option as this seems to cause a rebuild.
* Increase LS timeout for health check
* Remove duplicated kafka health check
* Increase health check timeouts to 10min
* Set defaults for `ARGS` in kibana container. This makes it easy to also build the container directly.
* Adjusts docker image paths to all use `ES_BEATS` variable.
* Remove `detector_rules` for ML as not supported / required anymore in master.
* Remove unused `SHIELD=false` variable in compose file
* Fix geoip looksup that changed and improve error logging

It would be great if we could disable most of the Kibana x-pack plugins as they are not needed for our testing. But the problem is 1) we cannot install only 1 plugin AFAIK 2) disabling one of the plugins causes a rebuild which makes the build even longer.

Closes elastic#5953
andrewkroh pushed a commit that referenced this issue Jan 4, 2018
@ruflin @andrewkroh
Improve Kibana docker image build (#5966)
e7f651e 
* Add `NODE_OPTIONS="--max-old-space-size=4096"` to circumvent elastic/kibana#15683
* Make it more obvious in the Docker file which parts are copied over and which are from the official repo
* Remove xpack.monitoring config option as this seems to cause a rebuild.
* Increase LS timeout for health check
* Remove duplicated kafka health check
* Increase health check timeouts to 10min
* Set defaults for `ARGS` in kibana container. This makes it easy to also build the container directly.
* Adjusts docker image paths to all use `ES_BEATS` variable.
* Remove `detector_rules` for ML as not supported / required anymore in master.
* Remove unused `SHIELD=false` variable in compose file
* Fix geoip looksup that changed and improve error logging

It would be great if we could disable most of the Kibana x-pack plugins as they are not needed for our testing. But the problem is 1) we cannot install only 1 plugin AFAIK 2) disabling one of the plugins causes a rebuild which makes the build even longer.

Closes #5953
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ph @ruflin @jbudz