Environment Variables not replaced in application.properties when running as Docker container

[ivangfr]

Describe the bug
In my application.properties, I have the following property

mp.messaging.outgoing.news.bootstrap.servers=${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}

When I run the application in development mode ./mvnw compile quarkus:dev the url to local kafka cluster is written correctly, i.e, localhost:9092

Then, I build the Docker image

./mvnw clean package -DskipTests
docker build -f src/main/docker/Dockerfile.jvm -t docker.mycompany.com/quarkus-producer-api-jvm:1.0.0 .

Finally, when I run the container

docker run -d --rm --name quarkus-producer-api-jvm \
  --network producer-consumer_default \
  -e KAFKA_HOST=kafka \
  -p 9100:8080 \
  docker.mycompany.com/quarkus-producer-api-jvm:1.0.0

I get the following in the logs

INFO  [org.apa.kaf.cli.pro.ProducerConfig] (main) ProducerConfig values: 
	...
	bootstrap.servers = [${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}]
	...
Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: ${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}

Expected behavior
The correct url to Kafka cluster should be

INFO  [org.apa.kaf.cli.pro.ProducerConfig] (main) ProducerConfig values: 
	...
	bootstrap.servers = [kafka:9092]
	...

Actual behavior

INFO  [org.apa.kaf.cli.pro.ProducerConfig] (main) ProducerConfig values: 
	...
	bootstrap.servers = [${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}]
	...

Configuration

mp.messaging.outgoing.news.connector=smallrye-kafka
mp.messaging.outgoing.news.topic=quarkus.news.json
mp.messaging.outgoing.news.bootstrap.servers=${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}
mp.messaging.outgoing.news.value.serializer=io.quarkus.kafka.client.serialization.JsonbSerializer

Environment (please complete the following information):

• Output of uname -a or ver:
• Output of java -version:
  openjdk version "1.8.0_222"
  OpenJDK Runtime Environment (build 1.8.0_222-20190711112007.graal.jdk8u-src-tar-gz-b08)
  OpenJDK 64-Bit GraalVM CE 19.2.0 (build 25.222-b08-jvmci-19.2-b02, mixed mode)
• GraalVM version (if different from Java):
• Quarkus version or git rev: 0.24.0

[xfh]

Did this ever work? I don't know how far the Quarkus support for configuration properties at runtime is supposed to go.

@ivangfr you can add the environment variable KAFKA_BOOTSTRAP_SERVERS to your Quarkus container, to specify the bootstrap server globally, and remove mp.messaging.outgoing.news.bootstrap.servers. Maybe that works for you? You can also try if you get it working with MP_MESSAGING_OUTGOING_NEWS_BOOTSTRAP_SERVERS=HOST:PORT.
See also https://quarkus.io/guides/application-configuration-guide#overriding-properties-at-runtime

I think the Kafka Guide is not very specific about how properties are handled and which properties are fixed at build time. The Kafka Streams Guide is more explicit about properties.

[ivangfr]

Hey @xfh,

Using MP_MESSAGING_OUTGOING_NEWS_BOOTSTRAP_SERVERS=HOST:PORT works.

So now, my docker command looks like

docker run -d --rm --name quarkus-producer-api-jvm \
  -p 9100:8080 \
  -e MP_MESSAGING_OUTGOING_NEWS_BOOTSTRAP_SERVERS=kafka:9092 \
  --network producer-consumer_default \
  docker.mycompany.com/quarkus-producer-api-jvm:1.0.0

What I also did was to set two properties in application.properties, one for development and another for "production"

%dev.mp.messaging.outgoing.news.bootstrap.servers=localhost:9092
mp.messaging.outgoing.news.bootstrap.servers=kafka:9092

So, for "production" the default is kafka:9092. If for some reason the kafka host or port is different (for example, kafka-service:9093), I can overwrite it using the environment variable MP_MESSAGING_OUTGOING_NEWS_BOOTSTRAP_SERVERS=kafka-service:9093

I have just asked because other frameworks like Spring Boot and Micronaut work when the property in application.properties has the format as shown below

mp.messaging.outgoing.news.bootstrap.servers=${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}

[djajcevic]

This works for me: quarkus.datasource.username=${DB_USERNAME:na}.
Configuration for REST clients do no work; mp-rest/url=${SOME_URL:http://...}, but this is OK since it's evaluated at build time. Would be great if this can be dynamic.

[xfh]

I think more of the properties with the mp.messaging.* should be evaluated at runtime. I've found another issue in combination with confluentinc's Schema Registry: the URL of the schema registry should be configurable for different environments as well.

A non-breaking strategy would be to make all these properties runtime configurable, unless they are known to be static, like *.connector, *.value.serializer, *.value.deserializer, *.key.serializer, *.key.deserializer

[xfh]

Hi @dmlloyd does #5387 also fix this issue?

[dmlloyd]

Yes it should but we'll need to explicitly verify it to be sure.