spark-sql-streaming-KafkaSourceOffset.adoc

KafkaSourceOffset

Caution

FIXME

KafkaSourceOffset is created for partitionToOffsets collection of TopicPartitions and their offsets.

Creating KafkaSourceOffset Instance

Caution

FIXME

Getting Partition Offsets — getPartitionOffsets Method

getPartitionOffsets(offset: Offset): Map[TopicPartition, Long]

getPartitionOffsets takes KafkaSourceOffset.partitionToOffsets from offset.

If offset is KafkaSourceOffset, getPartitionOffsets takes the partitions and offsets straight from it.

If however offset is SerializedOffset, getPartitionOffsets deserializes the offsets from JSON.

getPartitionOffsets reports an IllegalArgumentException when offset is neither KafkaSourceOffset or SerializedOffset.

Invalid conversion from offset of [class] to KafkaSourceOffset

Note	getPartitionOffsets is used exclusively when KafkaSource generates a DataFrame with records from Kafka for a batch.