Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node Crashed #7769

Closed
KnightOfNight opened this issue Jul 19, 2016 · 8 comments
Closed

Node Crashed #7769

KnightOfNight opened this issue Jul 19, 2016 · 8 comments
Labels
bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team

Comments

@KnightOfNight
Copy link

Kibana version:
4.5.0

Server OS version:
Ubuntu 14.04.4 LTS

Browser version:
Chrome 51.0.2704.103

Browser OS version:
Mac OSX 10.11.5

Original install method (e.g. download page, yum, from source, etc.):
Debian package

Description of the problem including expected versus actual behavior:
Node crashed and our automated core dump processing reported the problem and generated the attached report. Expected behavior is that node shouldn't crash.

Steps to reproduce:
Unknown which search caused the crash. I can say that this particular Kibana instance connects to an Elasticsearch Tribe node that is configured to connect to several Elasticsearch clusters geographically spread out around the world.

Errors in browser console (if relevant):
None observed.

Provide logs and/or server output (if relevant):
{code}
"/opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli"

            "stdout": "[New LWP 3541]

[New LWP 3544]
[New LWP 3546]
[New LWP 3754]
[New LWP 3755]
[New LWP 3545]
[New LWP 3752]
[New LWP 3547]
[New LWP 3753]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000bcebae in v8::internal::ObjectVisitor::VisitCodeEntry(unsigned char*) ()
#0 0x0000000000bcebae in v8::internal::ObjectVisitor::VisitCodeEntry(unsigned char*) ()
#1 0x0000000000ad3d87 in v8::internal::SlotsBuffer::UpdateSlots(v8::internal::Heap*) ()
#2 0x0000000000ad76da in v8::internal::MarkCompactCollector::EvacuateNewSpaceAndCandidates() ()
#3 0x0000000000ad8f23 in v8::internal::MarkCompactCollector::SweepSpaces() ()
#4 0x0000000000ae2c98 in v8::internal::MarkCompactCollector::CollectGarbage() ()
#5 0x0000000000a99550 in v8::internal::Heap::MarkCompact() ()
#6 0x0000000000ab0e68 in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) ()
#7 0x0000000000ab1409 in v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const_, char const_, v8::GCCallbackFlags) ()
#8 0x0000000000ab1632 in v8::internal::Heap::CollectAllAvailableGarbage(char const*) ()
#9 0x0000000000a5abf8 in v8::internal::Factory::NewJSObject(v8::internal::Handlev8::internal::JSFunction, v8::internal::PretenureFlag) ()
#10 0x00000000008e1a5c in v8::internal::JsonParser::ParseJsonObject() ()
#11 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#12 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#13 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#14 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#15 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#16 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#17 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#18 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#19 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#20 0x00000000008e2e01 in v8::internal::JsonParser::ParseJson() ()
#21 0x0000000000c89c14 in v8::internal::Runtime_ParseJson(int, v8::internal::Object*, v8::internal::Isolate) ()
#22 0x0000137ce570963b in ?? ()
#23 0x0000000001e307d0 in ?? ()
#24 0x0000137ce5709581 in ?? ()
#25 0x00007fff8ee45930 in ?? ()
#26 0x00007fff8ee45988 in ?? ()
#27 0x0000137cf6f35780 in ?? ()
#28 0x0000255907d04101 in ?? ()
#29 0x0000137cf6e9fdeb in ?? ()
#30 0x0000000100000000 in ?? ()
#31 0x0000149dc2b9fa09 in ?? ()
#32 0x0000149dc2bae4a1 in ?? ()
#33 0x0000149dc2b9fa09 in ?? ()
#34 0x00007fff8ee459c8 in ?? ()
#35 0x0000137ce5709ff7 in ?? ()
#36 0x0000149dc2b041b9 in ?? ()
#37 0x0000255907d04101 in ?? ()
#38 0x0000149dc2bae3d9 in ?? ()
#39 0x0000000100000000 in ?? ()
#40 0x0000149dc2bae4a1 in ?? ()
#41 0x0000000a00000000 in ?? ()
#42 0x00007fff8ee45a08 in ?? ()
#43 0x0000137cf6e840c9 in ?? ()
#44 0x0000255907d04101 in ?? ()
#45 0x0000149dc2bae3d9 in ?? ()
#46 0x0000149dc2bae4a1 in ?? ()
#47 0x0000262455e5aeb1 in ?? ()
#48 0x000014152dd35591 in ?? ()
#49 0x0000262455e5aeb1 in ?? ()
#50 0x00007fff8ee45a58 in ?? ()
#51 0x0000137cfc23e1ca in ?? ()
#52 0x0000255907d04101 in ?? ()
#53 0x0000256e403fb8f1 in ?? ()
#54 0x000014152dd35591 in ?? ()
#55 0x0000149dc2b041b9 in ?? ()
#56 0xffffffff00000000 in ?? ()
#57 0x0000149dc2b041b9 in ?? ()
#58 0x0000149dc2bd1e69 in ?? ()
#59 0x0000149dc2bd1eb1 in ?? ()
#60 0x00007fff8ee45aa8 in ?? ()
#61 0x0000137cf6f12878 in ?? ()
#62 0x0000149dc2bd1f51 in ?? ()
#63 0x000000c800000000 in ?? ()
#64 0x0000255907d04101 in ?? ()
#65 0x0000149dc2b041b9 in ?? ()
#66 0x0000149dc2b041b9 in ?? ()
#67 0x0000149dc2bd1e69 in ?? ()
#68 0x0000149dc2bd1f89 in ?? ()
#69 0x0000149dc2bd1eb1 in ?? ()
#70 0x00007fff8ee45af8 in ?? ()
#71 0x0000137ce6c92888 in ?? ()
#72 0x0000149dc2bd1f51 in ?? ()
#73 0x000000c800000000 in ?? ()
#74 0x0000255907d04101 in ?? ()
#75 0x0000149dc2b041b9 in ?? ()
#76 0x0000149dc2b041b9 in ?? ()
#77 0x0000149dc2bd1f89 in ?? ()
#78 0x0000149dc2bd1fd1 in ?? ()
#79 0x0000149dc2bd2019 in ?? ()
#80 0x00007fff8ee45b30 in ?? ()
#81 0x0000137ce5709ff7 in ?? ()
#82 0x0000149dc2b041b9 in ?? ()
#83 0x0000256e403a51b1 in ?? ()
#84 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f74fecfd700 (LWP 3753)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000000000f9f429 in uv_cond_wait (cond=cond@entry=0x196dd60 , mutex=mutex@entry=0x196dd20 ) at ../deps/uv/src/unix/thread.c:380
#2 0x0000000000f902f8 in worker (arg=arg@entry=0x0) at ../deps/uv/src/threadpool.c:75
#3 0x0000000000f9ef89 in uv__thread_start (arg=) at ../deps/uv/src/unix/thread.c:49
#4 0x00007f75018d0182 in start_thread (arg=0x7f74fecfd700) at pthread_create.c:312
#5 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7f74ffcff700 (LWP 3547)):
#0 0x0000000000ad1bbe in v8::internal::MarkCompactCollector::SweepInParallel(v8::internal::Page_, v8::internal::PagedSpace_) ()
#1 0x0000000000ad211d in v8::internal::MarkCompactCollector::SweeperTask::Run() ()
#2 0x0000000000e467a9 in v8::platform::WorkerThread::Run() ()
#3 0x0000000000fa9e60 in v8::base::ThreadEntry(void*) ()
#4 0x00007f75018d0182 in start_thread (arg=0x7f74ffcff700) at pthread_create.c:312
#5 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7f74ff4fe700 (LWP 3752)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000000000f9f429 in uv_cond_wait (cond=cond@entry=0x196dd60 , mutex=mutex@entry=0x196dd20 ) at ../deps/uv/src/unix/thread.c:380
#2 0x0000000000f902f8 in worker (arg=arg@entry=0x0) at ../deps/uv/src/threadpool.c:75
#3 0x0000000000f9ef89 in uv__thread_start (arg=) at ../deps/uv/src/unix/thread.c:49
#4 0x00007f75018d0182 in start_thread (arg=0x7f74ff4fe700) at pthread_create.c:312
#5 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f7500d01700 (LWP 3545)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x0000000000fa8ea8 in v8::base::Semaphore::Wait() ()
#2 0x0000000000e46669 in v8::platform::TaskQueue::GetNext() ()
#3 0x0000000000e467bc in v8::platform::WorkerThread::Run() ()
#4 0x0000000000fa9e60 in v8::base::ThreadEntry(void*) ()
#5 0x00007f75018d0182 in start_thread (arg=0x7f7500d01700) at pthread_create.c:312
#6 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f74fdcfb700 (LWP 3755)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000000000f9f429 in uv_cond_wait (cond=cond@entry=0x196dd60 , mutex=mutex@entry=0x196dd20 ) at ../deps/uv/src/unix/thread.c:380
#2 0x0000000000f902f8 in worker (arg=arg@entry=0x0) at ../deps/uv/src/threadpool.c:75
#3 0x0000000000f9ef89 in uv__thread_start (arg=) at ../deps/uv/src/unix/thread.c:49
#4 0x00007f75018d0182 in start_thread (arg=0x7f74fdcfb700) at pthread_create.c:312
#5 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f74fe4fc700 (LWP 3754)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000000000f9f429 in uv_cond_wait (cond=cond@entry=0x196dd60 , mutex=mutex@entry=0x196dd20 ) at ../deps/uv/src/unix/thread.c:380
#2 0x0000000000f902f8 in worker (arg=arg@entry=0x0) at ../deps/uv/src/threadpool.c:75
#3 0x0000000000f9ef89 in uv__thread_start (arg=) at ../deps/uv/src/unix/thread.c:49
#4 0x00007f75018d0182 in start_thread (arg=0x7f74fe4fc700) at pthread_create.c:312
#5 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f7500500700 (LWP 3546)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x0000000000fa8ea8 in v8::base::Semaphore::Wait() ()
#2 0x0000000000e46669 in v8::platform::TaskQueue::GetNext() ()
#3 0x0000000000e467bc in v8::platform::WorkerThread::Run() ()
#4 0x0000000000fa9e60 in v8::base::ThreadEntry(void*) ()
#5 0x00007f75018d0182 in start_thread (arg=0x7f7500500700) at pthread_create.c:312
#6 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f7501502700 (LWP 3544)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x0000000000fa8ea8 in v8::base::Semaphore::Wait() ()
#2 0x0000000000e46669 in v8::platform::TaskQueue::GetNext() ()
#3 0x0000000000e467bc in v8::platform::WorkerThread::Run() ()
#4 0x0000000000fa9e60 in v8::base::ThreadEntry(void*) ()
#5 0x00007f75018d0182 in start_thread (arg=0x7f7501502700) at pthread_create.c:312
#6 0x00007f75015fd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f7502924780 (LWP 3541)):
#0 0x0000000000bcebae in v8::internal::ObjectVisitor::VisitCodeEntry(unsigned char*) ()
#1 0x0000000000ad3d87 in v8::internal::SlotsBuffer::UpdateSlots(v8::internal::Heap*) ()
#2 0x0000000000ad76da in v8::internal::MarkCompactCollector::EvacuateNewSpaceAndCandidates() ()
#3 0x0000000000ad8f23 in v8::internal::MarkCompactCollector::SweepSpaces() ()
#4 0x0000000000ae2c98 in v8::internal::MarkCompactCollector::CollectGarbage() ()
#5 0x0000000000a99550 in v8::internal::Heap::MarkCompact() ()
#6 0x0000000000ab0e68 in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) ()
#7 0x0000000000ab1409 in v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const_, char const_, v8::GCCallbackFlags) ()
#8 0x0000000000ab1632 in v8::internal::Heap::CollectAllAvailableGarbage(char const*) ()
#9 0x0000000000a5abf8 in v8::internal::Factory::NewJSObject(v8::internal::Handlev8::internal::JSFunction, v8::internal::PretenureFlag) ()
#10 0x00000000008e1a5c in v8::internal::JsonParser::ParseJsonObject() ()
#11 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#12 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#13 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#14 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#15 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#16 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#17 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#18 0x00000000008e1b91 in v8::internal::JsonParser::ParseJsonObject() ()
#19 0x00000000008e2ae8 in v8::internal::JsonParser::ParseJsonValue() ()
#20 0x00000000008e2e01 in v8::internal::JsonParser::ParseJson() ()
#21 0x0000000000c89c14 in v8::internal::Runtime_ParseJson(int, v8::internal::Object*, v8::internal::Isolate) ()
#22 0x0000137ce570963b in ?? ()
#23 0x0000000001e307d0 in ?? ()
#24 0x0000137ce5709581 in ?? ()
#25 0x00007fff8ee45930 in ?? ()
#26 0x00007fff8ee45988 in ?? ()
#27 0x0000137cf6f35780 in ?? ()
#28 0x0000255907d04101 in ?? ()
#29 0x0000137cf6e9fdeb in ?? ()
#30 0x0000000100000000 in ?? ()
#31 0x0000149dc2b9fa09 in ?? ()
#32 0x0000149dc2bae4a1 in ?? ()
#33 0x0000149dc2b9fa09 in ?? ()
#34 0x00007fff8ee459c8 in ?? ()
#35 0x0000137ce5709ff7 in ?? ()
#36 0x0000149dc2b041b9 in ?? ()
#37 0x0000255907d04101 in ?? ()
#38 0x0000149dc2bae3d9 in ?? ()
#39 0x0000000100000000 in ?? ()
#40 0x0000149dc2bae4a1 in ?? ()
#41 0x0000000a00000000 in ?? ()
#42 0x00007fff8ee45a08 in ?? ()
#43 0x0000137cf6e840c9 in ?? ()
#44 0x0000255907d04101 in ?? ()
#45 0x0000149dc2bae3d9 in ?? ()
#46 0x0000149dc2bae4a1 in ?? ()
#47 0x0000262455e5aeb1 in ?? ()
#48 0x000014152dd35591 in ?? ()
#49 0x0000262455e5aeb1 in ?? ()
#50 0x00007fff8ee45a58 in ?? ()
#51 0x0000137cfc23e1ca in ?? ()
#52 0x0000255907d04101 in ?? ()
#53 0x0000256e403fb8f1 in ?? ()
#54 0x000014152dd35591 in ?? ()
#55 0x0000149dc2b041b9 in ?? ()
#56 0xffffffff00000000 in ?? ()
#57 0x0000149dc2b041b9 in ?? ()
#58 0x0000149dc2bd1e69 in ?? ()
#59 0x0000149dc2bd1eb1 in ?? ()
#60 0x00007fff8ee45aa8 in ?? ()
#61 0x0000137cf6f12878 in ?? ()
#62 0x0000149dc2bd1f51 in ?? ()
#63 0x000000c800000000 in ?? ()
#64 0x0000255907d04101 in ?? ()
#65 0x0000149dc2b041b9 in ?? ()
#66 0x0000149dc2b041b9 in ?? ()
#67 0x0000149dc2bd1e69 in ?? ()
#68 0x0000149dc2bd1f89 in ?? ()
#69 0x0000149dc2bd1eb1 in ?? ()
#70 0x00007fff8ee45af8 in ?? ()
#71 0x0000137ce6c92888 in ?? ()
#72 0x0000149dc2bd1f51 in ?? ()
#73 0x000000c800000000 in ?? ()
#74 0x0000255907d04101 in ?? ()
#75 0x0000149dc2b041b9 in ?? ()
#76 0x0000149dc2b041b9 in ?? ()
#77 0x0000149dc2bd1f89 in ?? ()
#78 0x0000149dc2bd1fd1 in ?? ()
#79 0x0000149dc2bd2019 in ?? ()
#80 0x00007fff8ee45b30 in ?? ()
#81 0x0000137ce5709ff7 in ?? ()
#82 0x0000149dc2b041b9 in ?? ()
#83 0x0000256e403a51b1 in ?? ()
#84 0x0000000000000000 in ?? ()",

{code}
@tylersmalley
Copy link
Contributor

@KnightOfNight are you by any chance running Kibana and specifying max-old-space-size?

@KnightOfNight
Copy link
Author

No, I am not specifying a value for that configuration item. Which reminds me I should have provided the configuration that I am running. It's pretty simple, just two items...

server.port: 9292
elasticsearch.requestTimeout: 900000

@epixa
Copy link
Contributor

epixa commented Jul 22, 2016

I can't say for certain, but I suspect #7772 will fix this.

@epixa epixa added the bug Fixes for quality problems that affect the customer experience label Jul 22, 2016
@PSiAU
Copy link

PSiAU commented Jul 25, 2016

I'm seeing this problem in 4.5.3 too when using max-old-space-size. Removing max-old-space-size allows Kibana to start properly.

@tomerilp
Copy link

Any update regarding this bug?
We are facing the same issue, the node process crashes from time to time.

Kibana version: kibana-4.5.4-1.x86_64

We are not using the max-old-space-size parameter.

@tylersmalley
Copy link
Contributor

@tomerilp here is the PR to bring this change into Kibana 4.6.2 #8213. Should be merged in today.

@tylersmalley
Copy link
Contributor

#8213 was just merged and is slated for release in 4.6.2.

@jbudz jbudz added the Team:Operations Team label for Operations Team label Nov 1, 2016
@epixa
Copy link
Contributor

epixa commented Feb 14, 2017

With the release of 5.2.1, we now have no known circumstances where the node process will crash, so I'm going to close this ticket out. We're also unaware of circumstances where the node process crashes in 4.6.4.

I'm going to close this out, but if anyone encounters crashes from those versions onward, please open an issue with all the details you can provide.

@epixa epixa closed this as completed Feb 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@tylersmalley @epixa @KnightOfNight @PSiAU @jbudz @tomerilp