Skip to content

Commit

Permalink
added docker images build
Browse files Browse the repository at this point in the history
  • Loading branch information
Administrator committed Dec 22, 2018
1 parent bc9b70e commit 7f9e091
Show file tree
Hide file tree
Showing 7 changed files with 469 additions and 0 deletions.
25 changes: 25 additions & 0 deletions docker-images/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# docker-elasticsearch-kubernetes

Ready to use, lean Elasticsearch Docker image ready for using within a Kubernetes cluster.


## Current software

* Alpine Linux 3.8
* IcedTea JRE 8u171
* Elasticsearch 6.4.2

**Note:** `x-pack-ml` module is forcibly disabled as it's not supported on Alpine Linux.

## Run

See [kubernetes-elasticsearch-cluster](https://github.com/vekatkriish/elasticsearchonk8s) for instructions on how to run, scale and use Elasticsearch on Kubernetes.

## Environment variables

This image can be configured by means of environment variables, that one can set on a `Deployment`.


* [DISCOVERY_SERVICE](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#unicast) - the service to be queried for through DNS (default = `elasticsearch-discovery`).
* [MEMORY_LOCK](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#bootstrap.memory_lock) - memory locking control defaults to `false` as Kubernetes requires swap to be disabled.

66 changes: 66 additions & 0 deletions docker-images/es-docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
FROM dockerkriish/docker-jre:8u171_alpine_3.8.1
MAINTAINER [email protected]

# Export HTTP & Transport
EXPOSE 9200 9300

ENV ES_VERSION 6.4.2

ENV DOWNLOAD_URL "https://artifacts.elastic.co/downloads/elasticsearch"
ENV ES_TARBAL "${DOWNLOAD_URL}/elasticsearch-${ES_VERSION}.tar.gz"
ENV ES_TARBALL_ASC "${DOWNLOAD_URL}/elasticsearch-${ES_VERSION}.tar.gz.asc"
ENV GPG_KEY "46095ACC8548582C1A2699A9D27D666CD88E42B4"

# Install Elasticsearch.
RUN apk add --no-cache --update bash ca-certificates su-exec util-linux curl
RUN apk add --no-cache -t .build-deps gnupg openssl \
&& cd /tmp \
&& echo "===> Install Elasticsearch..." \
&& curl -o elasticsearch.tar.gz -Lskj "$ES_TARBAL"; \
if [ "$ES_TARBALL_ASC" ]; then \
curl -o elasticsearch.tar.gz.asc -Lskj "$ES_TARBALL_ASC"; \
export GNUPGHOME="$(mktemp -d)"; \
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY"; \
gpg --batch --verify elasticsearch.tar.gz.asc elasticsearch.tar.gz; \
rm -r "$GNUPGHOME" elasticsearch.tar.gz.asc; \
fi; \
tar -xf elasticsearch.tar.gz \
&& ls -lah \
&& mv elasticsearch-$ES_VERSION /elasticsearch \
&& adduser -DH -s /sbin/nologin elasticsearch \
&& mkdir -p /elasticsearch/config/scripts /elasticsearch/plugins \
&& chown -R elasticsearch:elasticsearch /elasticsearch \
&& rm -rf /tmp/* \
&& apk del --purge .build-deps

ENV PATH /elasticsearch/bin:$PATH

WORKDIR /elasticsearch

# Copy configuration
COPY config /elasticsearch/config

# Copy run script
COPY run.sh /

# Set environment variables defaults
ENV ES_JAVA_OPTS "-Xms512m -Xmx512m"
ENV CLUSTER_NAME elasticsearch-default
ENV NODE_MASTER true
ENV NODE_DATA true
ENV NODE_INGEST true
ENV HTTP_ENABLE true
ENV NETWORK_HOST _site_
ENV HTTP_CORS_ENABLE true
ENV HTTP_CORS_ALLOW_ORIGIN *
ENV NUMBER_OF_MASTERS 1
ENV MAX_LOCAL_STORAGE_NODES 1
ENV SHARD_ALLOCATION_AWARENESS ""
ENV SHARD_ALLOCATION_AWARENESS_ATTR ""
ENV MEMORY_LOCK true
ENV REPO_LOCATIONS ""

# Volume for Elasticsearch data
VOLUME ["/data"]

CMD ["/run.sh"]
148 changes: 148 additions & 0 deletions docker-images/es-docker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
## Current software

* Alpine Linux 3.8
* OpenJDK JRE 8u171
* Elasticsearch 6.4.2

**Note:** `x-pack-ml` module is forcibly disabled as it's not supported on Alpine Linux.

## Run

### Attention

* In order for `bootstrap.mlockall` to work, `ulimit` must be allowed to run in the container. Run with `--privileged` to enable this.
* [Multicast discovery is no longer built-in](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/breaking_20_removed_features.html#_multicast_discovery_is_now_a_plugin)

Ready to use node for cluster `elasticsearch-default`:
```
docker run --name elasticsearch \
--detach \
--privileged \
--volume /path/to/data_folder:/data \
dockerkriish/elasticsearchk8s:6.4.2
```

Ready to use node for cluster `myclustername`:
```
docker run --name elasticsearch \
--detach \
--privileged \
--volume /path/to/data_folder:/data \
-e CLUSTER_NAME=myclustername \
dockerkriish/elasticsearchk8s:6.4.2
```

Ready to use node for cluster `elasticsearch-default`, with 8GB heap allocated to Elasticsearch:
```
docker run --name elasticsearch \
--detach \
--privileged \
--volume /path/to/data_folder:/data \
-e ES_JAVA_OPTS="-Xms8g -Xmx8g" \
dockerkriish/elasticsearchk8s:6.4.2
```

Ready to use node with plugins (x-pack and repository-gcs) pre installed. Already installed plugins are ignored:
```
docker run --name elasticsearch \
--detach \
--privileged \
--volume /path/to/data_folder:/data \
-e ES_JAVA_OPTS="-Xms8g -Xmx8g" \
-e ES_PLUGINS_INSTALL="repository-gcs,x-pack" \
dockerkriish/elasticsearchk8s:6.4.2
```

**Master-only** node for cluster `elasticsearch-default`:
```
docker run --name elasticsearch \
--detach \
--privileged \
--volume /path/to/data_folder:/data \
-e NODE_DATA=false \
-e HTTP_ENABLE=false \
dockerkriish/elasticsearchk8s:6.4.2
```

**Data-only** node for cluster `elasticsearch-default`:
```
docker run --name elasticsearch \
--detach --volume /path/to/data_folder:/data \
--privileged \
-e NODE_MASTER=false \
-e HTTP_ENABLE=false \
dockerkriish/elasticsearchk8s:6.4.2
```

**Data-only** node for cluster `elasticsearch-default` with shard allocation awareness:
```
docker run --name elasticsearch \
--detach --volume /path/to/data_folder:/data \
--volume /etc/hostname:/dockerhost \
--privileged \
-e NODE_MASTER=false \
-e HTTP_ENABLE=false \
-e SHARD_ALLOCATION_AWARENESS=dockerhostname \
-e SHARD_ALLOCATION_AWARENESS_ATTR="/dockerhost" \
dockerkriish/elasticsearchk8s:6.4.2
```

**Client-only** node for cluster `elasticsearch-default`:
```
docker run --name elasticsearch \
--detach \
--privileged \
--volume /path/to/data_folder:/data \
-e NODE_MASTER=false \
-e NODE_DATA=false \
dockerkriish/elasticsearchk8s:6.4.2
```
I also make available special images and instructions for [AWS EC2](https://github.com/pires/docker-elasticsearch-aws) and [Kubernetes](https://github.com/pires/docker-elasticsearch-kubernetes).

### Environment variables

This image can be configured by means of environment variables, that one can set on a `Deployment`.

* [CLUSTER_NAME](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name)
* [NODE_NAME](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#node.name)
* [NODE_MASTER](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#master-node)
* [NODE_DATA](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-node)
* [NETWORK_HOST](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values)
* [HTTP_ENABLE](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#_settings_2)
* [HTTP_CORS_ENABLE](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#_settings_2)
* [HTTP_CORS_ALLOW_ORIGIN](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#_settings_2)
* [NUMBER_OF_MASTERS](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#master-election)
* [MAX_LOCAL_STORAGE_NODES](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#max-local-storage-nodes)
* [ES_JAVA_OPTS](https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html)
* [ES_PLUGINS_INSTALL](https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html) - comma separated list of Elasticsearch plugins to be installed. Example: `ES_PLUGINS_INSTALL="repository-gcs,x-pack"`
* [SHARD_ALLOCATION_AWARENESS](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html#CO287-1)
* [SHARD_ALLOCATION_AWARENESS_ATTR](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html#CO287-1)
* [MEMORY_LOCK](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#bootstrap.memory_lock) - memory locking control - enable to prevent swap (default = `true`) .
* [REPO_LOCATIONS](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_shared_file_system_repository) - list of registered repository locations. For example `"/backup"` (default = `[]`). The value of REPO_LOCATIONS is automatically wrapped within an `[]` and therefore should not be included in the variable declaration. To specify multiple repository locations simply specify a comma separated string for example `"/backup", "/backup2"`.
* [PROCESSORS](https://github.com/elastic/elasticsearch-definitive-guide/pull/679/files) - allow elasticsearch to optimize for the actual number of available cpus (must be an integer - default = 1)

### Backup
Mount a shared folder (for example via NFS) to `/backup` and make sure the `elasticsearch` user
has write access. Then, set the `REPO_LOCATIONS` environment variable to `"/backup"` and create
a backup repository:

`backup_repository.json`:
```
{
"type": "fs",
"settings": {
"location": "/backup",
"compress": true
}
}
```

```bash
curl -XPOST http://<container_ip>:9200/_snapshot/nas_repository -d @backup_repository.json`
```

Now, you can take snapshots using:
```bash
curl -f -XPUT "http://<container_ip>:9200/_snapshot/nas_repository/snapshot_`date --utc +%Y_%m_%dt%H_%M`?wait_for_completion=true"
```

35 changes: 35 additions & 0 deletions docker-images/es-docker/elasticsearch.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
cluster:
name: ${CLUSTER_NAME}

node:
master: ${NODE_MASTER}
data: ${NODE_DATA}
name: ${NODE_NAME}
ingest: ${NODE_INGEST}
max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}

processors: ${PROCESSORS:1}

network.host: ${NETWORK_HOST}

path:
data: /data/data
logs: /data/log
repo: ${REPO_LOCATIONS}

bootstrap:
memory_lock: ${MEMORY_LOCK}

http:
enabled: ${HTTP_ENABLE}
compression: true
cors:
enabled: ${HTTP_CORS_ENABLE}
allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}

discovery:
zen:
ping.unicast.hosts: ${DISCOVERY_SERVICE}
minimum_master_nodes: ${NUMBER_OF_MASTERS}

xpack.ml.enabled: false
109 changes: 109 additions & 0 deletions docker-images/es-docker/jvm.options
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

# Commented out since these are supplied as env vars in the command-line
# for the service.

#-Xms2g
#-Xmx2g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## optimizations

# disable calls to System#gc
-XX:+DisableExplicitGC

# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# force the server VM (remove on 32-bit client JVMs)
-server

# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true

# flags to keep Netty from being unsafe
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}

## GC logging

#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime

# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}

# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true
10 changes: 10 additions & 0 deletions docker-images/es-docker/log4j2.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@

status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n

rootLogger.level = info
rootLogger.appenderRef.console.ref = console
Loading

0 comments on commit 7f9e091

Please sign in to comment.