Add an official image for Apache Storm #1641
Conversation
|
Hey @31z4, sorry for the delay! A few thoughts:
|
|
@tianon many thanks for your feedback!
|
|
Great work! |
|
Nice! (31z4/storm-docker@d050a2b...d71c8ad) Regarding GPG, I did Only comment I'd have left is that you probably want to add |
|
@tianon Unfortunately, we can't get working Storm with just a single command. At minimum we have to start the Nimbus and Supervisor daemons. Of course I can start those two using |
|
Ah, naw -- that makes sense! 👍 (https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container) I'm looking over the docs now, and I'm a little worried about the Zookeeper dependency. We don't have an official |
|
@tianon There is no way to run Storm in cluster mode without Zookeeper. Though, Zookeeper is not required if you run Storm in local mode. I was also concerned about not having an official Zookeeper image. I planned to manage this after the Storm image. It seems that plan has changed 😃 |
|
Oh, I suppose before we get too far down that rabbit hole, has there been any contact with either upstream to see if they're interested in collaboration on the image? We like to make sure they're at least aware of what we're doing, especially since we've burned bridges by being too hasty without consulting upstream on whether they even wanted to be part of the official images, so we like to be a little more proactive than we've been in the past. 😅 😊 |
|
@tianon @ApacheStorm retwitted my tweet. I've also just created a JIRA issue. So I believe they're aware 😊 |
|
@HeartSaVioR Thank you for the feedback! There are several reasons why I didn't include
|
|
@31z4 ui and log viewer would be great to have in the docker image. |
|
@harshach as you run the |
|
Now that we have the zookeeper image, we can hopefully get this in soon! The only concern I have is related to the docs where all the supporting containers are run with |
|
@yosifkit thanks for your feedback! I'll look into |
|
Am I understanding correctly that Storm requires both "nimbus" and "zookeeper" to function? In the examples in the documentation, you seem to be using |
|
(See also https://github.com/docker-library/kibana/blob/b49429d8cf95f58a2ed1246b5ff24bbe0dd81911/4.6/Dockerfile#L50-L52 for an example in |
Update instruction format
|
Hello @31z4, great job, thank you. Just 2 points :
Thanks |
|
@abipit |
|
thank you @HeartSaVioR, great news, I will test with the storm container 0.10.2 ! For my first point, I didn't see this anwser with the command line to launch the UI, my bad :) |
|
For storm ui, I launch this command : docker run -d -p 8081:8080 --name=storm_ui 31z4/storm:0.10.2 storm ui -c nimbus.host=nimbus But without success, in the UI, under the "Internal Server Error" section, I have : I made several tests for the nimbus.host but i don't find the good. |
|
It's ok for storm_UI but with the docker-compose, not with a command-line. Because in command_line, the network for storm_UI was 172.18.0.0 while the 3 others containers created by the docker-compose, the network was 172.17.0.0 I put this code in the docker-compose : and it's OK now because all containers are in the same network 172.17.... |
|
I found the solution to launch the storm_UI directly in command_line, I indicate that the network is the storm network :
|
|
@HeartSaVioR @abipit thank you guys for your comments. Just added the Storm UI example into the docs. |
|
31z4/storm-docker@5fb2661...f38117c seems pretty sane to me! 👍 ❤️ |
|
Dockerization seems sane to me. I have just a few questions about logging. Does it log to |
|
@yosifkit Thanks for pointing this out. Found out that actually logs are not going to the According to logging everything to stdout by default I think it doesn't make sense. Storm has pretty complex and meaningful default logging configuration. The idea is to separate different log sources into different files to facilitate browsing and searching the logs. Logging also depends on the service you're running. For example While working Hope this makes sense. Will add a short note about logging in the image documentation. |
Logs go to /logs Data go to /data
There was a problem hiding this comment.
Build test of #1641; 0d52884 (storm):
$ bashbrew build storm:0.9.7
Building bashbrew/cache:a70c57a367c067be50d77fa33f71b4b6d94eebea2f24626dde27280d4fd0bf6c (storm:0.9.7)
Tagging storm:0.9.7
Tagging storm:0.9
$ test/run.sh storm:0.9.7
testing storm:0.9.7
'utc' [1/4]...passed
'cve-2014--shellshock' [2/4]...passed
'no-hard-coded-passwords' [3/4]...passed
'override-cmd' [4/4]...passed
$ bashbrew build storm:0.10.2
Building bashbrew/cache:b57cc8807a771f1302fc79cb14c7e33cacee6ac4584cf73e93a004b6e7458935 (storm:0.10.2)
Tagging storm:0.10.2
Tagging storm:0.10
$ test/run.sh storm:0.10.2
testing storm:0.10.2
'utc' [1/4]...passed
'cve-2014--shellshock' [2/4]...passed
'no-hard-coded-passwords' [3/4]...passed
'override-cmd' [4/4]...passed
$ bashbrew build storm:1.0.2
Building bashbrew/cache:a4c7cd706817216c8172ba54a9bcd6a4b39c4f948aa34eef78aef58f966eb906 (storm:1.0.2)
Tagging storm:1.0.2
Tagging storm:1.0
Tagging storm:latest
$ test/run.sh storm:1.0.2
testing storm:1.0.2
'utc' [1/4]...passed
'cve-2014--shellshock' [2/4]...passed
'no-hard-coded-passwords' [3/4]...passed
'override-cmd' [4/4]...passed
diff --git a/storm_0.10/Dockerfile b/storm_0.10/Dockerfile
new file mode 100644
index 0000000..c851d57
--- /dev/null
+++ b/storm_0.10/Dockerfile
@@ -0,0 +1,43 @@
+FROM openjdk:8-jre-alpine
+MAINTAINER Elisey Zanko <[email protected]>
+
+# Install required packages
+RUN apk add --no-cache \
+ bash \
+ python \
+ su-exec
+
+ENV STORM_USER=storm
+ENV STORM_CONF_DIR=/conf
+ENV STORM_DATA_DIR=/data
+ENV STORM_LOG_DIR=/logs
+
+# Add a user and make dirs
+RUN set -x \
+ && adduser -D "$STORM_USER" \
+ && mkdir -p "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR" \
+ && chown -R "$STORM_USER:$STORM_USER" "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR"
+
+ARG GPG_KEY=ACEFE18DD2322E1E84587A148DE03962E80B8FFD
+ARG DISTRO_NAME=apache-storm-0.10.2
+
+# Download Apache Storm, verify its PGP signature, untar and clean up
+RUN set -x \
+ && apk add --no-cache --virtual .build-deps \
+ gnupg \
+ && wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz" \
+ && wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" \
+ && export GNUPGHOME="$(mktemp -d)" \
+ && gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \
+ && gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \
+ && tar -xzf "$DISTRO_NAME.tar.gz" \
+ && chown -R "$STORM_USER:$STORM_USER" "$DISTRO_NAME" \
+ && rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \
+ && apk del .build-deps
+
+WORKDIR $DISTRO_NAME
+
+ENV PATH $PATH:/$DISTRO_NAME/bin
+
+COPY docker-entrypoint.sh /
+ENTRYPOINT ["/docker-entrypoint.sh"]
diff --git a/storm_0.10/docker-entrypoint.sh b/storm_0.10/docker-entrypoint.sh
new file mode 100755
index 0000000..a8aea91
--- /dev/null
+++ b/storm_0.10/docker-entrypoint.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+# Allow the container to be started with `--user`
+if [ "$1" = 'storm' -a "$(id -u)" = '0' ]; then
+ chown -R "$STORM_USER" "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR"
+ exec su-exec "$STORM_USER" "$0" "$@"
+fi
+
+# Generate the config only if it doesn't exist
+CONFIG="$STORM_CONF_DIR/storm.yaml"
+if [ ! -f "$CONFIG" ]; then
+ cat << EOF > "$CONFIG"
+storm.zookeeper.servers: [zookeeper]
+nimbus.seeds: [nimbus]
+storm.log.dir: "$STORM_LOG_DIR"
+storm.local.dir: "$STORM_DATA_DIR"
+EOF
+fi
+
+exec "$@"
diff --git a/storm_0.9/Dockerfile b/storm_0.9/Dockerfile
new file mode 100644
index 0000000..5830aee
--- /dev/null
+++ b/storm_0.9/Dockerfile
@@ -0,0 +1,43 @@
+FROM openjdk:8-jre-alpine
+MAINTAINER Elisey Zanko <[email protected]>
+
+# Install required packages
+RUN apk add --no-cache \
+ bash \
+ python \
+ su-exec
+
+ENV STORM_USER=storm
+ENV STORM_CONF_DIR=/conf
+ENV STORM_DATA_DIR=/data
+ENV STORM_LOG_DIR=/logs
+
+# Add a user and make dirs
+RUN set -x \
+ && adduser -D "$STORM_USER" \
+ && mkdir -p "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR" \
+ && chown -R "$STORM_USER:$STORM_USER" "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR"
+
+ARG GPG_KEY=ACEFE18DD2322E1E84587A148DE03962E80B8FFD
+ARG DISTRO_NAME=apache-storm-0.9.7
+
+# Download Apache Storm, verify its PGP signature, untar and clean up
+RUN set -x \
+ && apk add --no-cache --virtual .build-deps \
+ gnupg \
+ && wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz" \
+ && wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" \
+ && export GNUPGHOME="$(mktemp -d)" \
+ && gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \
+ && gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \
+ && tar -xzf "$DISTRO_NAME.tar.gz" \
+ && chown -R "$STORM_USER:$STORM_USER" "$DISTRO_NAME" \
+ && rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \
+ && apk del .build-deps
+
+WORKDIR $DISTRO_NAME
+
+ENV PATH $PATH:/$DISTRO_NAME/bin
+
+COPY docker-entrypoint.sh /
+ENTRYPOINT ["/docker-entrypoint.sh"]
diff --git a/storm_0.9/docker-entrypoint.sh b/storm_0.9/docker-entrypoint.sh
new file mode 100755
index 0000000..a8aea91
--- /dev/null
+++ b/storm_0.9/docker-entrypoint.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+# Allow the container to be started with `--user`
+if [ "$1" = 'storm' -a "$(id -u)" = '0' ]; then
+ chown -R "$STORM_USER" "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR"
+ exec su-exec "$STORM_USER" "$0" "$@"
+fi
+
+# Generate the config only if it doesn't exist
+CONFIG="$STORM_CONF_DIR/storm.yaml"
+if [ ! -f "$CONFIG" ]; then
+ cat << EOF > "$CONFIG"
+storm.zookeeper.servers: [zookeeper]
+nimbus.seeds: [nimbus]
+storm.log.dir: "$STORM_LOG_DIR"
+storm.local.dir: "$STORM_DATA_DIR"
+EOF
+fi
+
+exec "$@"
diff --git a/storm_latest/Dockerfile b/storm_latest/Dockerfile
new file mode 100644
index 0000000..972e3b3
--- /dev/null
+++ b/storm_latest/Dockerfile
@@ -0,0 +1,43 @@
+FROM openjdk:8-jre-alpine
+MAINTAINER Elisey Zanko <[email protected]>
+
+# Install required packages
+RUN apk add --no-cache \
+ bash \
+ python \
+ su-exec
+
+ENV STORM_USER=storm
+ENV STORM_CONF_DIR=/conf
+ENV STORM_DATA_DIR=/data
+ENV STORM_LOG_DIR=/logs
+
+# Add a user and make dirs
+RUN set -x \
+ && adduser -D "$STORM_USER" \
+ && mkdir -p "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR" \
+ && chown -R "$STORM_USER:$STORM_USER" "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR"
+
+ARG GPG_KEY=ACEFE18DD2322E1E84587A148DE03962E80B8FFD
+ARG DISTRO_NAME=apache-storm-1.0.2
+
+# Download Apache Storm, verify its PGP signature, untar and clean up
+RUN set -x \
+ && apk add --no-cache --virtual .build-deps \
+ gnupg \
+ && wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz" \
+ && wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" \
+ && export GNUPGHOME="$(mktemp -d)" \
+ && gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \
+ && gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \
+ && tar -xzf "$DISTRO_NAME.tar.gz" \
+ && chown -R "$STORM_USER:$STORM_USER" "$DISTRO_NAME" \
+ && rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \
+ && apk del .build-deps
+
+WORKDIR $DISTRO_NAME
+
+ENV PATH $PATH:/$DISTRO_NAME/bin
+
+COPY docker-entrypoint.sh /
+ENTRYPOINT ["/docker-entrypoint.sh"]
diff --git a/storm_latest/docker-entrypoint.sh b/storm_latest/docker-entrypoint.sh
new file mode 100755
index 0000000..a8aea91
--- /dev/null
+++ b/storm_latest/docker-entrypoint.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+# Allow the container to be started with `--user`
+if [ "$1" = 'storm' -a "$(id -u)" = '0' ]; then
+ chown -R "$STORM_USER" "$STORM_CONF_DIR" "$STORM_DATA_DIR" "$STORM_LOG_DIR"
+ exec su-exec "$STORM_USER" "$0" "$@"
+fi
+
+# Generate the config only if it doesn't exist
+CONFIG="$STORM_CONF_DIR/storm.yaml"
+if [ ! -f "$CONFIG" ]; then
+ cat << EOF > "$CONFIG"
+storm.zookeeper.servers: [zookeeper]
+nimbus.seeds: [nimbus]
+storm.log.dir: "$STORM_LOG_DIR"
+storm.local.dir: "$STORM_DATA_DIR"
+EOF
+fi
+
+exec "$@" |
|
LGTM |
|
@31z4 Great job, your image is now "Official" on the docker hub ;-) 👍 About logs, is it possible to have the logviewer of supervisor on port 8000 ? 8000 is normally the default port, I tested to export it but without success, no page. It's for my developper, he don't have access on the server but he needs to consult the topology's log (in /logs/workers-artifacts/...). Do you have an idea ? Thank you |
|
@abipit There are some known issues with the I see two major approaches here. The first is starting the I'm thinking of creating an image variant with the |
As version 1.0.0 is a huge milestone for Apache Storm it would be great to finally have an official image for it. See corresponding docs PR.
Checklist for Review
foobarneeds Node.js, hasFROM node:...instead of grabbingnodevia other means been considered?)ifFROM scratch, tarballs only exist in a single commit within the associated history?