elasticsearch.performanceanalyzer ERROR

[aetter]

Originally reported in opendistro/for-elasticsearch-docs#52 by @sanaLiu:

Recently, I met a problem. Es has been brushing this log, please help to check it,The error report is as follows
[2019-04-12T15:00:42,433][WARN ][o.e.g.DanglingIndicesState] [node1] [[.opendistro_security/dPNNhxJUT8euAX-TUzN8Lg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-04-12T15:03:37,536][ERROR][c.a.o.e.p.m.PerformanceAnalyzerMetrics] [node1] Error in Writing to Tmp File: java.io.IOException: Bad file descriptor for keyPath:/dev/shm/performanceanalyzer/1555052610000//indices/.kibana_1/0
java.io.IOException: Bad file descriptor
at java.io.FileOutputStream.close0(Native Method) ~[?:1.8.0_144]
at java.io.FileOutputStream.access$000(FileOutputStream.java:53) ~[?:1.8.0_144]
at java.io.FileOutputStream$1.close(FileOutputStream.java:356) ~[?:1.8.0_144]
at java.io.FileDescriptor.closeAll(FileDescriptor.java:212) ~[?:1.8.0_144]
at java.io.FileOutputStream.close(FileOutputStream.java:354) ~[?:1.8.0_144]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.PerformanceAnalyzerMetrics.writeToTmp(PerformanceAnalyzerMetrics.java:158) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.PerformanceAnalyzerMetrics.emitMetric(PerformanceAnalyzerMetrics.java:121) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.MetricsProcessor.lambda$saveMetricValues$0(MetricsProcessor.java:27) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:104) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:102) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.MetricsProcessor.saveMetricValues(MetricsProcessor.java:27) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.NodeStatsMetricsCollector.collectMetrics(NodeStatsMetricsCollector.java:181) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:104) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_144]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:102) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

[gferrette]

Hello!

I'm facing the same problem:

[2019-04-15T12:01:34,421][WARN ][o.e.m.j.JvmGcMonitorService] [76eg-co] [gc][482014] overhead, spent [771ms] collecting in the last [1s]
[2019-04-15T12:01:38,491][ERROR][c.a.o.e.p.m.PerformanceAnalyzerMetrics] [76eg-co] Error in Writing to Tmp File: java.io.IOException: Bad file descriptor for keyPath:/dev/shm/performanceanalyzer/1555329690000//indices/security-auditlog-2019.04.11/4
java.io.IOException: Bad file descriptor
at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?]
at java.io.File.createNewFile(File.java:1024) ~[?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.PerformanceAnalyzerMetrics.writeToTmp(PerformanceAnalyzerMetrics.java:155) ~[opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.PerformanceAnalyzerMetrics.emitMetric(PerformanceAnalyzerMetrics.java:121) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.MetricsProcessor.lambda$saveMetricValues$0(MetricsProcessor.java:27) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:104) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at java.security.AccessController.doPrivileged(Native Method) [?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:102) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.MetricsProcessor.saveMetricValues(MetricsProcessor.java:27) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.NodeStatsMetricsCollector.collectMetrics(NodeStatsMetricsCollector.java:181) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:104) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at java.security.AccessController.doPrivileged(Native Method) [?:?]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:102) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) [opendistro_performance_analyzer-0.7.0.0.jar:0.7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]

[kdinta]

Thanks you for details. We have fixed & pull request #20 has more details.