Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Messages produced with repetitive offset #993

Closed
tchap opened this issue Nov 30, 2017 · 5 comments
Closed

Messages produced with repetitive offset #993

tchap opened this issue Nov 30, 2017 · 5 comments
Labels
producer

Comments

@tchap
Copy link

tchap commented Nov 30, 2017

Versions

Sarama Version: 1.14.0
Kafka Version: 0.11.0.0
Go Version: 1.9.2

Configuration

What configuration values are you using for Sarama and Kafka?

       config := sarama.NewConfig()
       config.Net.DialTimeout = Timeout
       config.Net.ReadTimeout = Timeout
       config.Net.WriteTimeout = Timeout
       config.Metadata.Retry.Max = 15
       config.Metadata.Retry.Backoff = 2 * time.Second
       config.Producer.Return.Successes = true
       config.Producer.Return.Errors = true
       config.ChannelBufferSize = channelBufferSize
       config.Version = sarama.V0_11_0_0

Kafka is using the default config.

Logs

Sarama:

sarama]2017/11/30 15:30:42 Initializing new client
[sarama]2017/11/30 15:30:42 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/11/30 15:30:42 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/11/30 15:30:42 client/metadata fetching metadata for all topics from broker kafka:9092
[sarama]2017/11/30 15:30:42 Connected to broker at kafka:9092 (unregistered)
[sarama]2017/11/30 15:30:42 client/brokers registered new broker #1001 at kafka:9092
[sarama]2017/11/30 15:30:42 Successfully initialized new client
...
[sarama]2017/11/30 15:32:54 client/metadata found some partitions to be leaderless
[sarama]2017/11/30 15:32:54 client/metadata retrying after 2000ms... (15 attempts remaining)
[sarama]2017/11/30 15:32:56 client/metadata fetching metadata for [data.streams] from broker kafka:9092
[sarama]2017/11/30 15:32:56 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/11/30 15:32:56 producer/broker/1001 starting up
[sarama]2017/11/30 15:32:56 producer/broker/1001 state change to [open] on data.streams/0
[sarama]2017/11/30 15:32:56 Connected to broker at kafka:9092 (registered as #1001)
Problem Description

Produced 60 messages into data.streams and it seems that the first message was produced with offset 0, all other with offset 59. My program is misbehaving because of that and here is the output of kafkacat:

$ kafkacat -C -b kafka:9092 -t data.streams -o beginning -f '%t[%p]@%o: %s\n'
data.streams[0]@0: {"schema_version":1,"payload":{"timestamp":1512053152,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emb"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053154,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emb"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053154,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emb"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053153,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emc"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053155,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emc"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053155,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emc"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053154,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emd"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053156,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emd"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053156,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emd"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053155,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["eme"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053157,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["eme"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053157,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["eme"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053156,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emf"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053158,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emf"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053158,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emf"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053157,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emg"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053159,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emg"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053159,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emg"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053158,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emh"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053160,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emh"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053160,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emh"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053159,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emi"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053161,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emi"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053161,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emi"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053160,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emj"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053162,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emj"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053162,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emj"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053161,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emk"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053163,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emk"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053163,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emk"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053162,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emm"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053164,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emm"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053164,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emm"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053163,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emn"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053165,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emn"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053165,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emn"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053164,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emo"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053166,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emo"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053166,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emo"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053165,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emp"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053167,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emp"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053167,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emp"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053166,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emq"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053168,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emq"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053168,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emq"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053167,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emr"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053169,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emr"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053169,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emr"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053168,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["ems"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053170,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["ems"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053170,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["ems"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053169,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emt"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053171,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emt"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053171,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emt"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053170,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emu"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053172,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emu"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053172,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emu"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053171,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emv"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053173,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emv"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512053173,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emv"]}]}}
% Reached end of topic data.streams [0] at offset 60

I was actually expecting to see the following output. This is what is produced when I use confluent-kafka-go instead of Sarama in the same producer logic:

$ kafkacat -C -b kafka:9092 -t data.streams -o beginning -f '%t[%p]@%o: %s\n'
data.streams[0]@0: {"schema_version":1,"payload":{"timestamp":1512055014,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emb"]}]}}
data.streams[0]@1: {"schema_version":1,"payload":{"timestamp":1512055016,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emb"]}]}}
data.streams[0]@2: {"schema_version":1,"payload":{"timestamp":1512055016,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emb"]}]}}
data.streams[0]@3: {"schema_version":1,"payload":{"timestamp":1512055015,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emc"]}]}}
data.streams[0]@4: {"schema_version":1,"payload":{"timestamp":1512055017,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emc"]}]}}
data.streams[0]@5: {"schema_version":1,"payload":{"timestamp":1512055017,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emc"]}]}}
data.streams[0]@6: {"schema_version":1,"payload":{"timestamp":1512055016,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emd"]}]}}
data.streams[0]@7: {"schema_version":1,"payload":{"timestamp":1512055018,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emd"]}]}}
data.streams[0]@8: {"schema_version":1,"payload":{"timestamp":1512055018,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emd"]}]}}
data.streams[0]@9: {"schema_version":1,"payload":{"timestamp":1512055017,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["eme"]}]}}
data.streams[0]@10: {"schema_version":1,"payload":{"timestamp":1512055019,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["eme"]}]}}
data.streams[0]@11: {"schema_version":1,"payload":{"timestamp":1512055019,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["eme"]}]}}
data.streams[0]@12: {"schema_version":1,"payload":{"timestamp":1512055018,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emf"]}]}}
data.streams[0]@13: {"schema_version":1,"payload":{"timestamp":1512055020,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emf"]}]}}
data.streams[0]@14: {"schema_version":1,"payload":{"timestamp":1512055020,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emf"]}]}}
data.streams[0]@15: {"schema_version":1,"payload":{"timestamp":1512055019,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emg"]}]}}
data.streams[0]@16: {"schema_version":1,"payload":{"timestamp":1512055021,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emg"]}]}}
data.streams[0]@17: {"schema_version":1,"payload":{"timestamp":1512055021,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emg"]}]}}
data.streams[0]@18: {"schema_version":1,"payload":{"timestamp":1512055020,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emh"]}]}}
data.streams[0]@19: {"schema_version":1,"payload":{"timestamp":1512055022,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emh"]}]}}
data.streams[0]@20: {"schema_version":1,"payload":{"timestamp":1512055022,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emh"]}]}}
data.streams[0]@21: {"schema_version":1,"payload":{"timestamp":1512055021,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emi"]}]}}
data.streams[0]@22: {"schema_version":1,"payload":{"timestamp":1512055023,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emi"]}]}}
data.streams[0]@23: {"schema_version":1,"payload":{"timestamp":1512055023,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emi"]}]}}
data.streams[0]@24: {"schema_version":1,"payload":{"timestamp":1512055022,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emj"]}]}}
data.streams[0]@25: {"schema_version":1,"payload":{"timestamp":1512055024,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emj"]}]}}
data.streams[0]@26: {"schema_version":1,"payload":{"timestamp":1512055024,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emj"]}]}}
data.streams[0]@27: {"schema_version":1,"payload":{"timestamp":1512055023,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emk"]}]}}
data.streams[0]@28: {"schema_version":1,"payload":{"timestamp":1512055025,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emk"]}]}}
data.streams[0]@29: {"schema_version":1,"payload":{"timestamp":1512055025,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emk"]}]}}
data.streams[0]@30: {"schema_version":1,"payload":{"timestamp":1512055024,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emm"]}]}}
data.streams[0]@31: {"schema_version":1,"payload":{"timestamp":1512055026,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emm"]}]}}
data.streams[0]@32: {"schema_version":1,"payload":{"timestamp":1512055026,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emm"]}]}}
data.streams[0]@33: {"schema_version":1,"payload":{"timestamp":1512055025,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emn"]}]}}
data.streams[0]@34: {"schema_version":1,"payload":{"timestamp":1512055027,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emn"]}]}}
data.streams[0]@35: {"schema_version":1,"payload":{"timestamp":1512055027,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emn"]}]}}
data.streams[0]@36: {"schema_version":1,"payload":{"timestamp":1512055026,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emo"]}]}}
data.streams[0]@37: {"schema_version":1,"payload":{"timestamp":1512055028,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emo"]}]}}
data.streams[0]@38: {"schema_version":1,"payload":{"timestamp":1512055028,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emo"]}]}}
data.streams[0]@39: {"schema_version":1,"payload":{"timestamp":1512055027,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emp"]}]}}
data.streams[0]@40: {"schema_version":1,"payload":{"timestamp":1512055029,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emp"]}]}}
data.streams[0]@41: {"schema_version":1,"payload":{"timestamp":1512055029,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emp"]}]}}
data.streams[0]@42: {"schema_version":1,"payload":{"timestamp":1512055028,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emq"]}]}}
data.streams[0]@43: {"schema_version":1,"payload":{"timestamp":1512055030,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emq"]}]}}
data.streams[0]@44: {"schema_version":1,"payload":{"timestamp":1512055030,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emq"]}]}}
data.streams[0]@45: {"schema_version":1,"payload":{"timestamp":1512055029,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emr"]}]}}
data.streams[0]@46: {"schema_version":1,"payload":{"timestamp":1512055031,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emr"]}]}}
data.streams[0]@47: {"schema_version":1,"payload":{"timestamp":1512055031,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emr"]}]}}
data.streams[0]@48: {"schema_version":1,"payload":{"timestamp":1512055030,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["ems"]}]}}
data.streams[0]@49: {"schema_version":1,"payload":{"timestamp":1512055032,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["ems"]}]}}
data.streams[0]@50: {"schema_version":1,"payload":{"timestamp":1512055032,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["ems"]}]}}
data.streams[0]@51: {"schema_version":1,"payload":{"timestamp":1512055031,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emt"]}]}}
data.streams[0]@52: {"schema_version":1,"payload":{"timestamp":1512055033,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emt"]}]}}
data.streams[0]@53: {"schema_version":1,"payload":{"timestamp":1512055033,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emt"]}]}}
data.streams[0]@54: {"schema_version":1,"payload":{"timestamp":1512055032,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emu"]}]}}
data.streams[0]@55: {"schema_version":1,"payload":{"timestamp":1512055034,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emu"]}]}}
data.streams[0]@56: {"schema_version":1,"payload":{"timestamp":1512055034,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emu"]}]}}
data.streams[0]@57: {"schema_version":1,"payload":{"timestamp":1512055033,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emv"]}]}}
data.streams[0]@58: {"schema_version":1,"payload":{"timestamp":1512055035,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emv"]}]}}
data.streams[0]@59: {"schema_version":1,"payload":{"timestamp":1512055035,"channel":10,"device":7,"sensor_data":[{"medium":12,"values":[0.5],"stream_ids":["emv"]}]}}
@tchap tchap changed the title Messages produces with repetitive offset Messages produced with repetitive offset Nov 30, 2017
@eapache
Copy link
Contributor

eapache commented Nov 30, 2017

Offsets are set by the broker when the messages are received, not by the producer. I don't understand what the difference could be here.

@tchap
Copy link
Author

tchap commented Nov 30, 2017

I tried many times and the result was always the same. The weird thing is that as far as I remember the offsets reported back by Sarama on the success channel were correct :-|

@eapache
Copy link
Contributor

eapache commented Nov 30, 2017

@edenhill could this be a bug in kafkacat somehow?

@tchap
Copy link
Author

tchap commented Jan 17, 2018 •
edited
Loading

I logged all messages on the consumer part as well and offsets are indeed messed up. Doesn't look like a kafkacat issue. On the other hand, the consumer uses librdkafka.

BTW tried with Sarama 1.15.0, no change.

@eapache
Copy link
Contributor

eapache commented Feb 9, 2018

In hindsight #1032 was almost certainly a duplicate of this, but it should be fixed in master now.

@eapache eapache closed this as completed Feb 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
producer
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tchap @eapache