Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update GeoIP processor documentation #71211

Merged
merged 14 commits into from
Apr 15, 2021
2 changes: 1 addition & 1 deletion client/rest-high-level/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ testClusters.all {
testDistribution = 'DEFAULT'
systemProperty 'es.scripting.update.ctx_in_params', 'false'
systemProperty 'es.geoip_v2_feature_flag_enabled', 'true'
setting 'geoip.downloader.enabled', 'false'
setting 'ingest.geoip.downloader.enabled', 'false'
setting 'reindex.remote.whitelist', '[ "[::1]:*", "127.0.0.1:*" ]'
setting 'xpack.license.self_generated.type', 'trial'
setting 'xpack.security.enabled', 'true'
Expand Down
42 changes: 39 additions & 3 deletions docs/reference/ingest/processors/geoip.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,15 @@ IPv6 addresses.
The `ingest-geoip` module ships by default with the GeoLite2 City, GeoLite2 Country and GeoLite2 ASN GeoIP2 databases from Maxmind made available
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe change this into: The geoip processor runs by default with the GeoLite2 City, GeoLite2 Country and GeoLite2 ASN GeoIP2 databases automatically managed by Elastic from Maxmind under the CCA-ShareAlike 4.0 license.

under the CCA-ShareAlike 4.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/

The `geoip` processor can run with other city, country and ASN GeoIP2 databases
Databases will be updated automatically as long as cluster can connect to https://geoip.elastic.co/. This endpoint
can be changed using <<ingest-geoip-settings,ingest.geoip.downloader.endpoint>> setting. Updated databases will be stored in
`$ES_TMPDIR/geoip-databases/{node_id}` directory.

The `geoip` processor can run with other or manually updated city, country and ASN GeoIP2 databases
from Maxmind. The database files must be copied into the `ingest-geoip` config
directory located at `$ES_CONFIG/ingest-geoip`. Custom database files must be
stored uncompressed and the extension must be `-City.mmdb`, `-Country.mmdb`, or
`-ASN.mmdb` to indicate the type of the database. These database files can not
have the same filename as any of the built-in database names. The
`-ASN.mmdb` to indicate the type of the database. The
`database_file` processor option is used to specify the filename of the custom
database to use for the processor.

Expand Down Expand Up @@ -310,3 +313,36 @@ The `geoip` processor supports the following setting:
The maximum number of results that should be cached. Defaults to `1000`.

Note that these settings are node settings and apply to all `geoip` processors, i.e. there is one cache for all defined `geoip` processors.

===== Cluster Settings

The `geoip` processor supports the following cluster settings:

`ingest.geoip.downloader.endpoint`::

(<<static-cluster-setting,Static>>) The endpoint used to update databases used by `geoip` processor. Defaults to https://geoip.elastic.co/v1/database


`ingest.geoip.downloader.enabled`::

Set to `true` (default) to enable automic updates of GeoIP databases. If set to `false` all downloaded databases will be removed.

`ingest.geoip.downloader.poll.interval`::

How often there will be a check for new databases. Defaults to `3d`, minimum is `1d`

[[ingest-geoip-air-gapped]]
==== Updating GeoIP databases in air-gapped environments
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should both bundle file based updating of databases and using proxy/geoip tool in one section called: 'using custom databases'? Which is for use cases when custom databases are used or Elasticsearch is used in air-gapped environments. This will have 2 sections, one for updating databases via config directory (which replaces the third paragraph of geoip page) and one for setting up a http proxy.

Then the third paragraph in the geoip page can removed, which isn't that important for most folks, since they will be using the Elastic managed databases.


If cluster can't connect to default endpoint for update due to firewalls or other restrictions there are still a couple of ways
to use updated databases:

. Create secured proxy to default endpoint and use it with `ingest.geoip.downloader.endpoint` setting
. Upload databases (`*.mmdb` files) to each ingest node to `$ES_CONFIG/ingest-geoip` directory
. Create service that mimics defeult endpoint and use it with `ingest.geoip.downloader.endpoint`. There's CLI tool located at
`bin/elasticsearch-geoip` that can ease that process. To use it
.. copy databases (`*.mmdb` files) to single directory
.. run the tool with `/bin/elasticsearch-geoip -s source/directory/to/use [-t target/directory]`
.. serve static files from that directory (for example with `docker run -v source/directory/to/use:/usr/share/nginx/html:ro nginx`)

Note that for last two options databases have to be manually downloaded from MaxMind site.
2 changes: 1 addition & 1 deletion modules/ingest-geoip/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ tasks.named("internalClusterTest").configure {

testClusters.all {
systemProperty "es.geoip_v2_feature_flag_enabled", "true"
setting "geoip.downloader.enabled", "false"
setting "ingest.geoip.downloader.enabled", "false"
}

tasks.register("copyDefaultGeoIp2DatabaseFiles", Copy) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,9 @@ public class GeoIpDownloader extends AllocatedPersistentTask {

public static final boolean GEOIP_V2_FEATURE_FLAG_ENABLED = "true".equals(System.getProperty("es.geoip_v2_feature_flag_enabled"));

public static final Setting<TimeValue> POLL_INTERVAL_SETTING = Setting.timeSetting("geoip.downloader.poll.interval",
public static final Setting<TimeValue> POLL_INTERVAL_SETTING = Setting.timeSetting("ingest.geoip.downloader.poll.interval",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

TimeValue.timeValueDays(3), TimeValue.timeValueDays(1), Property.Dynamic, Property.NodeScope);
public static final Setting<String> ENDPOINT_SETTING = Setting.simpleString("geoip.downloader.endpoint",
public static final Setting<String> ENDPOINT_SETTING = Setting.simpleString("ingest.geoip.downloader.endpoint",
"https://geoip.elastic.co/v1/database", Property.NodeScope);

public static final String GEOIP_DOWNLOADER = "geoip-downloader";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@

/**
* Persistent task executor that is responsible for starting {@link GeoIpDownloader} after task is allocated by master node.
* Also bootstraps GeoIP download task on clean cluster and handles changes to the 'geoip.downloader.enabled' setting
* Also bootstraps GeoIP download task on clean cluster and handles changes to the 'ingest.geoip.downloader.enabled' setting
*/
public final class GeoIpDownloaderTaskExecutor extends PersistentTasksExecutor<GeoIpTaskParams> implements ClusterStateListener {

public static final Setting<Boolean> ENABLED_SETTING = Setting.boolSetting("geoip.downloader.enabled", GEOIP_V2_FEATURE_FLAG_ENABLED,
Setting.Property.Dynamic, Setting.Property.NodeScope);
public static final Setting<Boolean> ENABLED_SETTING = Setting.boolSetting("ingest.geoip.downloader.enabled",
GEOIP_V2_FEATURE_FLAG_ENABLED, Setting.Property.Dynamic, Setting.Property.NodeScope);

private static final Logger logger = LogManager.getLogger(GeoIpDownloader.class);

Expand Down