Environment Variable KAFKA_BOOTSTRAP_SERVERS not being read by smallrye client #7791

ricardozanini · 2020-03-11T19:21:48Z

Hi!

Describe the bug
As stated by the docs, now we have the property kafka.boostrap.servers that can be set and it should be applied as the connection property to every Kafka connector within the application.

That said, we injected the KAFKA_BOOTSTRAP_SERVERS environment variable into our containers, without explicit setting it in the application.properties file. The application fails with:

2020-03-11 10:15:33,258 ERROR [io.sma.rea.mes.imp.ConfiguredChannelFactory] (main) Unable to create the publisher or subscriber during initialization: java.util.NoSuchElementException: Cannot find attribute bootstrap.serversfor channelkogito-processinstances-events. Has been tried: mp.messaging.outgoing.kogito-processinstances-events.bootstrap.servers and mp.messaging.connector.smallrye-kafka.bootstrap.servers

Expected behavior
That the environment variable KAFKA_BOOTSTRAP_SERVERS is being read by the application even if it's not defined in the application.properties file.

Actual behavior
The variable is ignored.

To Reproduce
Steps to reproduce the behavior:

Create an example application that consumes a Kafka topic
Clear any the properties with the suffix *.bootstrap.servers in the application.properties file
Create an env var named KAFKA_BOOTSTRAP_SERVERS and set it with the actual location of the Kafka server (you can use Spotify's docker image in a different forward port for this test)
The application will fail to connect

Configuration

# Add your application.properties here, if applicable.
quarkus.infinispan-client.server-list=localhost:11222
quarkus.http.cors=true

mp.messaging.outgoing.kogito-processinstances-events.bootstrap.servers=
mp.messaging.outgoing.kogito-processinstances-events.connector=smallrye-kafka
mp.messaging.outgoing.kogito-processinstances-events.topic=kogito-processinstances-events
mp.messaging.outgoing.kogito-processinstances-events.value.serializer=org.apache.kafka.common.serialization.StringSerializer

mp.messaging.outgoing.kogito-usertaskinstances-events.bootstrap.servers=
mp.messaging.outgoing.kogito-usertaskinstances-events.connector=smallrye-kafka
mp.messaging.outgoing.kogito-usertaskinstances-events.topic=kogito-usertaskinstances-events
mp.messaging.outgoing.kogito-usertaskinstances-events.value.serializer=org.apache.kafka.common.serialization.StringSerializer

Environment (please complete the following information):

Output of uname -a or ver:
Linux skywalker 5.5.8-100.fc30.x86_64 Switch to the Maven distributed copy of the SubstrateVM annotations #1 SMP Thu Mar 5 21:55:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Output of java -version:
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-20191008104205.buildslave.jdk8u-src-tar--b07)
OpenJDK 64-Bit GraalVM CE 19.2.1 (build 25.232-b07-jvmci-19.2-b03, mixed mode)
GraalVM version (if different from Java):
Quarkus version or git rev: 1.3.0.Alpha2
Build tool (ie. output of mvnw --version or gradlew --version):
Apache Maven 3.5.4 (Red Hat 3.5.4-5)

Additional context
We already had a similar problem before: #5708
This one might be related: #4571

Many thanks!

The text was updated successfully, but these errors were encountered:

geoand · 2020-03-12T08:31:21Z

@ricardozanini Hi,

Would it be possible to provide a minimum reproducer (no Infinispan for example :))?
I could of course do it myself, but I prefer to have it ready made to avoid me making any mistakes and loosing time in back and forths just to make sure I have the proper reproducer :)

geoand · 2020-03-12T09:17:39Z

FWIW, I was unable to reproduce this issue in our Kafka integration test (https://github.com/quarkusio/quarkus/blob/master/integration-tests/kafka/pom.xml)

I basically clear out kafka.bootstrap.servers from application.properties and ran the test like so:

KAFKA_BOOTSTRAP_SERVERS=localhost:19092 mvn clean test

Everything worked as expected.

Note that this was with Quarkus master.

@ricardozanini any chance you can check your project with 1.3.0.CR2?

ricardozanini · 2020-03-12T14:06:38Z

@geoand sure thing, will work on a simpler reproducer and will get back to you soon.

ricardozanini · 2020-03-12T20:25:52Z

@geoand I can confirm that this is not working on 1.3.0.CR2 either and I think I've found the problem. I've disabled the Infinispan integration. Steps to reproduce:

Start kafka instance with: docker run -p 2181:2181 -p 9093:9092 --env ADVERTISED_PORT=9092 spotify/kafka
Git clone: https://github.com/kiegroup/kogito-examples
Install kogito-examples BOM: mvn clean install -N
Compile jbpm-quarkus-example from the same repo enabling Kafka integration: mvn clean install -DskipTests -Pevents
Run: KAFKA_BOOTSTRAP_SERVERS=localhost:9093 java -jar target/jbpm-quarkus-example-8.0.0-SNAPSHOT-runner.jar

The application.properties file contains:

mp.messaging.outgoing.kogito-processinstances-events.bootstrap.servers=
(...)
mp.messaging.outgoing.kogito-usertaskinstances-events.bootstrap.servers=

If we delete them it works. I wonder if is this the expected behavior? I mean, since those are empty, the client could take into account the env property instead as a fallback.

geoand · 2020-03-12T21:03:58Z

Well, I don't think it should fallback to KAFKA_BOOTSTRAP_SERVERS but I am not sure what the expected behavior is. @cescoffier your thoughts?

radtriste · 2020-03-24T10:31:35Z

I will add that in native mode, QUARKUS_KAFKA_BOOTSTRAP_SERVERS is not handled whereas it is in non-native...

cescoffier · 2020-03-25T18:09:17Z

You can setup the broker location using: kafka.bootstrap.servers.

ricardozanini · 2020-03-25T18:36:57Z

@cescoffier that's the point. We are setting the broker location with KAFKA_BOOTSTRAP_SERVERS environment variable, which is not getting read in runtime if the properties file has mp.messaging.*.bootstrap.servers properties set.

We should get rid of them and use only kafka.bootstrap.servers? What's the priority?

cescoffier · 2020-07-09T13:14:04Z

Sorry, just coming back to this issue.

The priority is channel first, then the global value (because you can have multiple Kafka brokers).
So, unfortunately, in your case, you cannot have both.

What we can do is to check if the value is empty and in that case use the global one. That would fix your issue.

I've created smallrye/smallrye-reactive-messaging#649 to track this. It's an easy fix.

ricardozanini · 2020-07-09T14:18:10Z

Thank you @cescoffier!

cescoffier · 2020-07-10T06:18:01Z

Actually, I believe it's already fixed.

I've the following properties:

# Configure the Kafka sink (we write to it)
mp.messaging.outgoing.generated-price.connector=smallrye-kafka
mp.messaging.outgoing.generated-price.bootstrap.servers=
mp.messaging.outgoing.generated-price.topic=prices
mp.messaging.outgoing.generated-price.value.serializer=org.apache.kafka.common.serialization.IntegerSerializer

# Configure the Kafka source (we read from it)
mp.messaging.incoming.prices.connector=smallrye-kafka
mp.messaging.incoming.prices.topic=prices
mp.messaging.incoming.prices.value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer

The first channel uses a "blank" bootstrap.servers, the second channel omits it completely.

My broker runs on 9093 (while the default is 9092).

I tried:

export KAFKA_BOOTSTRAP_SERVERS=localhost:9093
mvn compile quarkus:dev

mvn compile quarkus:dev -Dkafka.boostrap.servers=localhost:9093

export KAFKA_BOOTSTRAP_SERVERS=localhost:9093
java -jar ...

All these configurations work.

Can you recheck with the latest release? Maybe I miss something obvious.

r00ta · 2020-07-13T16:18:02Z

Actually, I believe it's already fixed.

I've the following properties:

# Configure the Kafka sink (we write to it)
mp.messaging.outgoing.generated-price.connector=smallrye-kafka
mp.messaging.outgoing.generated-price.bootstrap.servers=
mp.messaging.outgoing.generated-price.topic=prices
mp.messaging.outgoing.generated-price.value.serializer=org.apache.kafka.common.serialization.IntegerSerializer

# Configure the Kafka source (we read from it)
mp.messaging.incoming.prices.connector=smallrye-kafka
mp.messaging.incoming.prices.topic=prices
mp.messaging.incoming.prices.value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer

The first channel uses a "blank" bootstrap.servers, the second channel omits it completely.

My broker runs on 9093 (while the default is 9092).

I tried:

export KAFKA_BOOTSTRAP_SERVERS=localhost:9093
mvn compile quarkus:dev

mvn compile quarkus:dev -Dkafka.boostrap.servers=localhost:9093

export KAFKA_BOOTSTRAP_SERVERS=localhost:9093
java -jar ...

All these configurations work.

Can you recheck with the latest release? Maybe I miss something obvious.

Hi @cescoffier , we've just upgraded our services to Quarkus 1.6 and it works fine. Thank you very much!

ricardozanini · 2020-07-13T16:32:14Z

Thanks @r00ta and @cescoffier, I'm closing this one for now. :)

ricardozanini added the kind/bug Something isn't working label Mar 11, 2020

ricardozanini mentioned this issue Mar 11, 2020

[KOGITO-1294] Adding support for binary builds apache/incubator-kie-kogito-operator#201

Merged

10 tasks

ricardozanini mentioned this issue Mar 12, 2020

KOGITO-1216 Bump services to Quarkus 1.3.0.CR2 apache/incubator-kie-kogito-runtimes#354

Merged

kenfinnigan added the area/reactive-messaging label Mar 17, 2020

sutaakar mentioned this issue Mar 23, 2020

KOGITO-1405 - Use manual acknowledgment for processing messages apache/incubator-kie-kogito-apps#113

Closed

ricardozanini mentioned this issue Apr 9, 2020

KOGITO-1712: use kafka.bootstrap.servers. apache/incubator-kie-kogito-examples#184

Merged

cescoffier added this to the 1.7.0 - master milestone Jul 9, 2020

radtriste mentioned this issue Jul 10, 2020

KOGITO 2707 - add trusty image apache/incubator-kie-kogito-images#188

Merged

7 tasks

ricardozanini closed this as completed Jul 13, 2020

ricardozanini mentioned this issue Jul 17, 2020

[KOGITO-2046] - Remove "infinispan properties module" from Data Index… apache/incubator-kie-kogito-images#189

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Environment Variable KAFKA_BOOTSTRAP_SERVERS not being read by smallrye client #7791

Environment Variable KAFKA_BOOTSTRAP_SERVERS not being read by smallrye client #7791

ricardozanini commented Mar 11, 2020

geoand commented Mar 12, 2020 •

edited

Loading

geoand commented Mar 12, 2020 •

edited

Loading

ricardozanini commented Mar 12, 2020

ricardozanini commented Mar 12, 2020 •

edited

Loading

geoand commented Mar 12, 2020

radtriste commented Mar 24, 2020

cescoffier commented Mar 25, 2020

ricardozanini commented Mar 25, 2020

cescoffier commented Jul 9, 2020

ricardozanini commented Jul 9, 2020

cescoffier commented Jul 10, 2020

r00ta commented Jul 13, 2020

ricardozanini commented Jul 13, 2020

Environment Variable KAFKA_BOOTSTRAP_SERVERS not being read by smallrye client #7791

Environment Variable KAFKA_BOOTSTRAP_SERVERS not being read by smallrye client #7791

Comments

ricardozanini commented Mar 11, 2020

geoand commented Mar 12, 2020 • edited Loading

geoand commented Mar 12, 2020 • edited Loading

ricardozanini commented Mar 12, 2020

ricardozanini commented Mar 12, 2020 • edited Loading

geoand commented Mar 12, 2020

radtriste commented Mar 24, 2020

cescoffier commented Mar 25, 2020

ricardozanini commented Mar 25, 2020

cescoffier commented Jul 9, 2020

ricardozanini commented Jul 9, 2020

cescoffier commented Jul 10, 2020

r00ta commented Jul 13, 2020

ricardozanini commented Jul 13, 2020

geoand commented Mar 12, 2020 •

edited

Loading

geoand commented Mar 12, 2020 •

edited

Loading

ricardozanini commented Mar 12, 2020 •

edited

Loading