Ingester not consuming messages with kafka server 2.x

[marqc]

Requirement - what kind of business use case are you trying to solve?

Use recent kafka server with jaeger-collector and jaeger-ingester.
Specifically Kafka 2.0.1 with log.message.format.version=2.0-IV1

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger depends on Shopify/Sarama:1.16 which does not support recent versions of kafka servers and message formats. With above kafka server config it does not consume messages. No errors/warnings is printed to log.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Update Shopify/Sarama dependency to latest stable version (1.20). https://github.com/Shopify/sarama/releases

Any open questions to address

[Mikedu1988]

I owe you big time! downgrade kafka version, messages finally got processed.

[ledor473]

@jaegertracing/jaeger-maintainers sounds like this is closed by #1248

[ivan-klass]

Still reproducible in 1.13.1.
With debug level enabled, I never reached this line:
https://github.com/jaegertracing/jaeger/blob/master/cmd/ingester/app/consumer/consumer.go#L141, there are no logs after {"level":"info","ts":1563794224.2705138,"caller":"consumer/consumer.go:78","msg":"Starting main loop"}
The app is then killed after "deadlockInterval" (set to 5min), but I'm 100% sure there are messages in topic during this interval (these messages are visible in other consuming tools, also I've binary to generate such messages).

I'm using docker-compose & fast-data-dev

fast-data-dev:
  image: landoop/fast-data-dev:2.2
  ports:
    - 2181:2181
    - 3030:3030
    - 8081-8083:8081-8083
    - 9581-9585:9581-9585
    - 9092:9092
  environment:
    - ADV_HOST=127.0.0.1
    - SAMPLEDATA=0
    - RUNTESTS=0
    - FORWARDLOGS=0
    - DISABLE_JMX=1

jaeger-ingester:
  image: jaegertracing/jaeger-ingester:1.13.1
  command: [
    "--log-level=debug",
    "--cassandra.keyspace=jaeger_v1_dc1",
    "--cassandra.servers=cassandra",
    "--kafka.consumer.brokers=fast-data-dev:9092",
    "--kafka.consumer.encoding=json",
    "--kafka.consumer.topic=my.tracing.zipkin.data"] # plain json payload
  environment:
    - SPAN_STORAGE_TYPE=cassandra
  ports:
    - 14270:14270
    - 14271:14271
  restart: on-failure
  depends_on:
    - fast-data-dev
    - cassandra
    - cassandra-schema

[marqc]

@klass-ivan You might be interested in #1640

ingester uses depracated sarama-cluster package, that sets kafka version to 0.9, this patch allows to override this setting with app settings. It is merged to master, but not released yet

[ivan-klass]

@marqc nevermind, the problem was in kafka advertised host incorrect for in-docker network. Thanks