[bootstrap] 6.7.0 RPM installation expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip] #39495

inqueue · 2019-02-28T08:05:42Z

Elasticsearch version (bin/elasticsearch --version): 6.7.0 (Upgraded from 6.6.1)

[2019-02-28T07:44:54,138][INFO ][o.e.n.Node] [json-sut-c1] 
version[6.7.0], pid[16514], build[default/rpm/0002203/2019-02-27T11:21:09.100270Z], 
OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server 
VM/1.8.0_191/25.191-b12]

Description of the problem including expected versus actual behavior:
Elasticsearch 6.7.0 will not start using the RPM installation on CentOS Linux release 7.6.1810 (Core). The log complains GeoLite2-ASN.mmdb is missing, but it is there:

[root@json-sut-c1 elasticsearch]# pwd && tree ingest-geoip
/etc/elasticsearch
ingest-geoip
├── GeoLite2-ASN.mmdb
├── GeoLite2-City.mmdb
├── GeoLite2-City.mmdb.gz
├── GeoLite2-Country.mmdb
└── GeoLite2-Country.mmdb.gz

0 directories, 5 files

Steps to reproduce:

sudo yum install elasticserach
systemctl start elasticsearch

Provide logs:

[2019-02-28T07:45:05,349][ERROR][o.e.b.Bootstrap          ] [json-sut-c1] Exception
java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	... 14 more
[2019-02-28T07:45:05,355][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [json-sut-c1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]
	... 6 more
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]
	... 6 more

Please let me know if you need access to the repro host.

The text was updated successfully, but these errors were encountered:

danielmitterdorfer · 2019-02-28T08:50:57Z

I am not sure this is a bug. The error message says that it expects the file to NOT exist at this location:

java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]

This is caused by a leftover from an old installation and can either be resolved via uninstalling the plugin in 6.6.1 before attempting the upgrade with the --purge option, i.e.:

./bin/elasticsearch-plugin remove --purge ingest-geoip

or manually removing the files in the config directory after the upgrade.

elasticmachine · 2019-02-28T10:39:30Z

Pinging @elastic/es-core-infra

inqueue · 2019-02-28T14:53:14Z

The error message says that it expects the file to NOT exist at this location:

Sorry it was late and I missed that. That is the issue @danielmitterdorfer; the ingest-geoip plugin was removed without the purge option. I removed blocker and added the docs label. Those were files leftover from the ingest-geoip plugin that was manually installed in 6.6.1. rm -rf ingest-geoip fixed it.

I do not exactly see why we have to fail the node for files not in use. This will happen to anyone that hasn't purged those files, so it at least warrants a clear note in the documentation.

danielmitterdorfer · 2019-02-28T16:45:19Z

I do not exactly see why we have to fail the node for files not in use.

I did not implement that check but I'd reason that we want to fail fast here because those files were previously used by the plugin and we want to avoid that people think that they are still used by the module and wonder why they are not picked up.

The ingest-geoip module in 6.7.0 will check for any leftover database files from the old plugin and will prevent node startup if they are present. With this commit we add a note to the breaking changes docs for 6.7.0 that instructs users to use the `--purge` option when removing the `ingest-geoip` plugin prior to upgrading to avoid this situation. Closes #39495

danielmitterdorfer · 2019-03-21T12:33:54Z

Closed by #39553

inqueue added >bug blocker labels Feb 28, 2019

danielmitterdorfer added the feedback_needed label Feb 28, 2019

colings86 added the :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts label Feb 28, 2019

inqueue added >docs General docs changes and removed blocker labels Feb 28, 2019

danielmitterdorfer added >enhancement v6.7.0 and removed >bug feedback_needed >enhancement labels Feb 28, 2019

danielmitterdorfer mentioned this issue Mar 1, 2019

Add a note to purge the ingest-geoip plugin #39553

Merged

danielmitterdorfer added v6.7.1 v6.7.0 and removed v6.7.0 v6.7.1 labels Mar 21, 2019

danielmitterdorfer closed this as completed Mar 21, 2019

jakommo mentioned this issue Oct 27, 2020

Adding a GeoIP2 database in $ES_CONFIG/ingest-geoip results in node not starting if using default filename #64203

Closed

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bootstrap] 6.7.0 RPM installation expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip] #39495

[bootstrap] 6.7.0 RPM installation expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip] #39495

inqueue commented Feb 28, 2019

danielmitterdorfer commented Feb 28, 2019 •

edited

Loading

elasticmachine commented Feb 28, 2019

inqueue commented Feb 28, 2019

danielmitterdorfer commented Feb 28, 2019

danielmitterdorfer commented Mar 21, 2019

[bootstrap] 6.7.0 RPM installation expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip] #39495

[bootstrap] 6.7.0 RPM installation expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip] #39495

Comments

inqueue commented Feb 28, 2019

danielmitterdorfer commented Feb 28, 2019 • edited Loading

elasticmachine commented Feb 28, 2019

inqueue commented Feb 28, 2019

danielmitterdorfer commented Feb 28, 2019

danielmitterdorfer commented Mar 21, 2019

danielmitterdorfer commented Feb 28, 2019 •

edited

Loading