build(docker): refactor docker build scripts #1687
Conversation
- add "build" option to docker-compose files to simplify rebuilding of images - create "start.sh" script so it's easier to override "command" in the quickstart's docker-compose file - use dockerize to wait for requisite services to start up - add a dedicated Dockerfile for kafka-setup This fixes datahub-project#1549 & datahub-project#1550
| ``` | ||
| This command will start the container. If you have the image available in your local store, this image will be used | ||
| for the container otherwise it will download the `latest` image from Docker Hub and then start that. | ||
| To start a container using an existing image, run the same command without the `--build` flag. |
There was a problem hiding this comment.
I guess it would be good to note that: Once you build an image in your local, docker-compose up will simply use already built image, it will not pull the latest image from Docker Hub. You need to force it to pull the latest image by running docker-compose pull before running up.
There was a problem hiding this comment.
I believe the intention of these Dockerfiles is for local development, instead of quickstart. As such pulling the latest image from docker hub is probably not a common operation in this case?
| for the container otherwise it will download the `latest` image from Docker Hub and then start that. | ||
| This command will rebuild the local docker image and start a container based on the image. | ||
|
|
||
| To start a container using an existing image, run the same command without the `--build` flag. |
| ``` | ||
| This command will start the container. If you have the image available in your local store, this image will be used | ||
| for the container otherwise it will download the `latest` image from Docker Hub and then start that. | ||
| To start a container using a previously built image, run the same command without the `--build` flag. |
| ``` | ||
| This command will start the container. If you have the image available in your local store, this image will be used | ||
| for the container otherwise it will download the `latest` image from Docker Hub and then start that. | ||
| To start a container using a previously built image, run the same command without the `--build` flag. |
| KAFKA_BROKER_ID: ignored | ||
| KAFKA_ZOOKEEPER_CONNECT: ignored |
There was a problem hiding this comment.
We need to add this env vars here now that Dockerfile use those.
docker/quickstart/docker-compose.yml
Outdated
| command: "sh -c 'while ping -c1 kafka-setup &>/dev/null; do echo waiting for kafka-setup... && sleep 1; done; \ | ||
| echo kafka-setup done! && /start.sh'" |
There was a problem hiding this comment.
Actually, GMS doesn't strictly need to wait for kafka-setup unlike mxe consumers. So, this can simply be a './start.sh'. But, anyways, doesn't hurt to wait for that either.
There was a problem hiding this comment.
GMS also sends MAE right? Wouldn't it need to wait for the topic creation as well?
There was a problem hiding this comment.
Actually, it doesn't because it's the producer. Even if topic is not created, when GMS starts to produce, topic will be automatically created because auto.create.topics.enable is true by default. http://kafka.apache.org/documentation.html#auto.create.topics.enable
However, topics are not created on Kafka, mxe consumers fails during initialization.
There was a problem hiding this comment.
Got it. Yeah auto topic creation doesn't work for Kafka stream apparently: https://docs.confluent.io/current/streams/developer-guide/manage-topics.html
* build(docker): refactor docker build scripts - add "build" option to docker-compose files to simplify rebuilding of images - create "start.sh" script so it's easier to override "command" in the quickstart's docker-compose file - use dockerize to wait for requisite services to start up - add a dedicated Dockerfile for kafka-setup This fixes datahub-project#1549 & datahub-project#1550
This fixes #1549 & #1550
Checklist