Elasticsearch service keep crashing

[moazam1]

Elasticsearch Version

8.15.4

Installed Plugins

No response

Java Version

1.8.0_432

OS Version

Linux 5.14.0-427.28.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 2 03:44:10 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

Almost every month Elasticsearch service is crashing and not restarting automatically. I think routine upgrade might be causing crash. Any helpe will be greatly appreciated.

Steps to Reproduce

I suspect routine upgrade might be causing this issue.

Logs (if relevant)

[2024-11-13T01:30:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] riggering scheduled [ML] maintenance tasks
[2024-11-13T01:30:00,005][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] Deleting expired data
[2024-11-13T01:30:00,007][INFO ][o.e.x.m.j.r.UnusedStatsRemover] Successfully deleted [0] unused stats documents
[2024-11-13T01:30:00,007][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] Completed deletion of expired ML data
[2024-11-13T01:30:00,008][INFO ][o.e.x.m.MlDailyMaintenanceService] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2024-11-13T01:40:02,603][INFO ][o.e.c.m.MetadataMappingService] [sss_applied_late_lien_fees/F0MzKyNnS1arYCqRnbgBzw] update_mapping [_doc]
[2024-11-13T03:20:30,001][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch [masterService#updateTask][T#6560]], exiting
java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar!/META-INF/services/org.elasticsearch.index.SlowLogFieldProvider
at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1430) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:749) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:702) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:476) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV2Template(MetadataCreateIndexService.java:669) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:397) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.rolloverDataStream(MetadataRolloverService.java:406) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.rolloverClusterState(MetadataRolloverService.java:163) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$RolloverExecutor.executeTask(TransportRolloverAction.java:540) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$RolloverExecutor.execute(TransportRolloverAction.java:460) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1070) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1033) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1686) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1683) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1278) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1257) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.15.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?]
at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?]
at java.util.zip.ZipFile$CleanableResource.(ZipFile.java:724) ~[?:?]
at java.util.zip.ZipFile.(ZipFile.java:251) ~[?:?]
at java.util.zip.ZipFile.(ZipFile.java:180) ~[?:?]
at java.util.jar.JarFile.(JarFile.java:345) ~[?:?]
at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:100) ~[?:?]
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?]
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168) ~[?:?]
at sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91) ~[?:?]
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:117) ~[?:?]
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:160) ~[?:?]
at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:112) ~[elasticsearch-8.15.3.jar:?]
... 27 more

[SunnyDay6]

I confirm this issue occurs with newer version aswell.

a. Package updated at 00h00:46
b. Service failed à 03:09:09
c. Assumption: I think devs were deploying a nightly package, and elasticsearch was solicitated which crashed the service.
d. journalctl doesn't show that the service was restart with upgrade, only at failed point at 03:09:09
e. In /etc/sysconfig/elasticsearch , this is commented #RESTART_ON_UPGRADE=true, ( will test without comment and post back )
f. A manual restart of the service works as expected.
g. xpack is set to false in elasticsearch.yml

On Rocky Linux 9.4 Linux 5.14.0-427.42.1.el9_4.x86_64

[AUSER@SERVERNAME ~]# systemctl status elasticsearch
× elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Thu 2024-11-14 03:09:09 EST; 5h 38min ago
Duration: 1w 4d 9h 38min 31.988s
Docs: https://www.elastic.co
Process: 1281 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 1281 (code=exited, status=1/FAILURE)
CPU: 1h 20min 28.979s

from dnf.log

2024-11-14T00:00:03-0500 DEBUG ---> Package elasticsearch.x86_64 8.15.3-1 will be upgraded
2024-11-14T00:00:03-0500 DEBUG ---> Package elasticsearch.x86_64 8.16.0-1 will be an upgrade
2024-11-14T00:00:46-0500 DEBUG Upgraded: elasticsearch-8.16.0-1.x86_64

from elasticsearch.log

java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar!/META-INF/services/org.elasticsearch.index.SlowLogFieldProvider
at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1430) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:749) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:702) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:476) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:600) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:424) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:442) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$CreateIndexTask.execute(AutoCreateAction.java:335) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction.lambda$new$0(AutoCreateAction.java:120) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1070) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1033) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1686) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1683) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1278) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1257) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.15.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?]
at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?]
at java.util.zip.ZipFile$CleanableResource.(ZipFile.java:724) ~[?:?]
at java.util.zip.ZipFile.(ZipFile.java:251) ~[?:?]
at java.util.zip.ZipFile.(ZipFile.java:180) ~[?:?]
at java.util.jar.JarFile.(JarFile.java:345) ~[?:?]
at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:100) ~[?:?]
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?]

[EPGDigital-MW]

I think this is happening on ubuntu after an apt upgrade, it had upgraded to 8.16.0 earlier this week, a restart fixes it as x-pack-security-8.16.0.jar file exists.

[2024-11-20T10:46:16,375][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [XXXXX] fatal error in thread [elasticsearch[XXXXX][masterService#updateTask][T#1320]], exiting
java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar!/META-INF/services/org.elasticsearch.index.SlowLogFieldProvider
at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1430) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.indices.IndicesService.createIndexMapperServiceForValidation(IndicesService.java:827) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.execute(MetadataMappingService.java:110) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1070) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1033) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1686) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1683) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1278) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1257) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.15.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.15.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.15.3.jar
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?]
at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?]
at java.util.zip.ZipFile$CleanableResource.(ZipFile.java:724) ~[?:?]
at java.util.zip.ZipFile.(ZipFile.java:251) ~[?:?]
at java.util.zip.ZipFile.(ZipFile.java:180) ~[?:?]
at java.util.jar.JarFile.(JarFile.java:345) ~[?:?]
at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:100) ~[?:?]
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?]
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168) ~[?:?]
at sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91) ~[?:?]
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:117) ~[?:?]
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:160) ~[?:?]
at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:112) ~[elasticsearch-8.15.3.jar:?]
... 20 more

[moazam1]

I have locked ES update for now. Let's hope it resolve the issue.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

It sounds like you are trying to upgrade Elasticsearch in place, as it is still running. That is never a good idea (generally for any Java application).

Our upgrade guide specifically notes to shut down a node before upgrading the installation (see steps 4 and 5):
https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elasticsearch.html

Given the above, I believe Elasticsearch is working as intended, so I hope you don't mind I close this issue.

[EPGDigital-MW]

@rjernst thanks for clarifying this, I'll apt hold the package and move elasticsearch to a manually triggered upgrade vs automated when patching servers.