Access Kafka outside Demo setup

[ghvlg]

How to directly access Kafka made in Demo Setup, from outside, e.g. from separate Docker container running outside of one of Kubernetes clusters made by Demo setup?
Thanks in advance

[albrechtflo-hg]

Hi @ghvlg, if you are using Minikube for the Demo setup, the Kafka clusters are exposed using a NodePort service. You can run this command to see which ports are used for the Kafkas:

kubectl get svc --namespace kafka-pseudodev
kubectl get svc --namespace kafka-pseudoprod

Example output:

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
kafka-service       NodePort    10.101.212.124   <none>        9092:30635/TCP,29092:32003/TCP   75s
zookeeper-service   ClusterIP   10.110.130.38    <none>        2181/TCP                         105s

This means that the Kafka Cluster is reachable using your Minikube's host IP (e.g. 192.168.56.101 when running in a VM, or 127.0.0.1 when running on localhost) and port 30635 for SSL, or 32003 for PLAINTEXT connection (note that PLAINTEXT may not work, as the Kafka is configured to get client authentication from a client SSL certificate).

So you could use e.g. 192.168.56.101:30635 as your "bootstrap servers" property.

An example Spring Boot config file can be downloaded from your running Galapagos. You will need to do the following actions and adjustments:

• Download the truststore from the running Galapagos demo (on Dashboard page)
• Create and download a certificate and private key for your application (choose any of the Demo applications) in Galapagos
• Edit the Spring Boot config to use your downloaded certificate file

As the server certificate of the Kafka broker will most likely not match the correct (used) IP address, you should disable hostname verification, which can be added to the Spring Boot config. The final config may look like this:

spring:
  kafka:
    bootstrap-servers: "192.168.56.101:30635"
    ssl:
      trust-store-location: file:./kafka-truststore.jks
      trust-store-password: changeit
      trust-store-type: PKCS12
      key-store-location: file:./aramis_devtest.p12
      key-store-password: changeit
      key-store-type: PKCS12
      protocol: SSL
    properties:
      security.protocol: SSL
      # required for local Kafka server
      ssl.endpoint.identification.algorithm: ""

Two example applications (one for producer, and one for consumer) will be added to the HermesGermany workspace soon. There, you will only have to add the mentioned files (truststore and keystore) and set correct bootstrap servers.

Hope that helps!
Florian

[draskov]

Hi @albrechtflo-hg , one more question related to this topic.

If we put Galapagos DEMO on to VPS which has PUBLIC IP, and want to access it by public DNS/IP (from our local machine for example)... We are able to establish a connection, do SSL handshake and so on. But when we are about to send the message to some topic, Kafka returns us private IP of the node which is not reachable.

Do you have configuration/solution for that situation?

Best Regards,

[albrechtflo-hg]

Hi @draskov, this can easily be configured via the ConfigMap kafka-config in each Kafka Namespace.

The Demo Kafkas are configured to use the advertised.listeners property in the kafka-config config map as their KAFKA_ADVERTISED_LISTENERS environment variable (see kafka.yml where this is linked together).

By default, this property is populated with your K8S IP (when using Minikube) and the NodePort assigned to the Kafka Service (see setup-demo.sh where this is done - search for kafka-config) .

If this does not work for you (e.g. not using Minikube), you can adjust the setting via kubectl:

kubectl config set-context --current --namespace=kafka-pseudodev   # or -pseudoprod
kubectl delete configmap kafka-config
kubectl create configmap kafka-config "--from-literal=advertised.listeners=SSL://your.public.host:12345,SASL_PLAINTEXT://127.0.0.1:29092"

The port 12345 has to be replaced by the public port for the Kafka Service - see answer above how to retrieve it (obviously, you were already able to retrieve it).

The SASL_PLAINTEXT part must also be present for Kafka internal connections.

Please note that after changing the ConfigMap, you will have to restart the Kafka Pods. Easiest way is to just delete the existing Kafka Pod in each namespace, as the Kafka Deployment will immediately start a new one.

HTH,
Florian

[draskov]

Hi @albrechtflo-hg,

I think your proposal/solution won't work as Galapagos is connecting to 9092/LISTENER.

We managed to make another LISTENER/SSL which has ADVERTISED.LISTENER pointing to our public DNS. Additionally made one more port exposed onto kafka-service.

Introducing new LISTENER forced us to have kafka.yml configuration a bit different regarding the syntax.

(Writing this in order someone would like to have this solution for their local development setup).

However, this solution is temporary until we get our own infrastructure up and running with Kubernetes, where Galapagos will take place on its own.

Regards,
Drasko