Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify kafka client bootstrap server names and ports config #619

Closed
5 tasks
jfallows opened this issue Dec 5, 2023 · 0 comments · Fixed by #710
Closed
5 tasks

Simplify kafka client bootstrap server names and ports config #619

jfallows opened this issue Dec 5, 2023 · 0 comments · Fixed by #710
Assignees
Labels
story Feature description from user's perspective

Comments

@jfallows
Copy link
Contributor

jfallows commented Dec 5, 2023

Describe the desired outcome from the user's perspective
As a developer, I want to be able to copy and paste the bootstrap server host and port into zilla.yaml and not need to understand the details of tls server name or tcp host and port.

Acceptance criteria

  • Support configuration of server (array) option in kafka client binding for bootstrap servers
  • Eliminate need for any tls client binding options or routes to support kafka bootstrap servers
  • Eliminate need for any tcp client binding options or routes to support kafka bootstrap servers

Additional context
Today, we need to separate the components of the Kafka bootstrap server hostname and port across different client bindings to cause the correct behavior to occur at each layer in the protocol stack.

With plaintext tcp connectivity between Zilla and Kafka:

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    options:
      host: ${{env.KAFKA_HOST}}
      port: ${{env.KAFKA_PORT}}
    routes:
      - when:
          - cidr: 0.0.0.0/0

With tls encryption between Zilla and Kafka:

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    exit: south_tls_client
  south_tls_client:
    type: tls
    kind: client
    options:
      trustcacerts: true
      sni: ${{env.KAFKA_HOST}}
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    options:
      host: ${{env.KAFKA_HOST}}
      port: ${{env.KAFKA_PORT}}
    routes:
      - when:
          - cidr: 0.0.0.0/0

This can be error prone to configure, as typical Kafka client bootstrap server(s) are provided as a list of hostname:port.

Consider simplifying the configuration by adding an array of server properties to kafka binding client options.

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_HOST}}:${{env.KAFKA_PORT}}
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    routes:
      - when:
          - cidr: 0.0.0.0/0

Simplifies migration from tcp to tls.

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_HOST}}:${{env.KAFKA_PORT}}
    exit: south_tls_client
  south_tls_client:
    type: tls
    kind: client
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    routes:
      - when:
          - cidr: 0.0.0.0/0

Note: this simplest possible approach would also require defaulting trustcacerts to true when no explicit trust option specified in tls binding, and allowing cidr of 0.0.0.0/0 by default in tcp binding.

Note: the servers list of kafka binding client options would be used to randomly select a specific server to bootstrap discovery of the brokers in the cluster.

Tasks

  • Support kafka client option for server array of bootstrap servers in the syntax host:port
  • Send (random) bootstrap server as initial proxy begin extension, same as for discovered brokers
  • Change default behavior of tcp client binding to not require host or port options, nor routes
  • Update kafka binding reference docs
  • Update tcp binding reference docs
@jfallows jfallows added the enhancement New feature or request label Dec 5, 2023
@jfallows jfallows added story Feature description from user's perspective and removed enhancement New feature or request labels Jan 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
story Feature description from user's perspective
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants