Skip to content

Latest commit

 

History

History
77 lines (63 loc) · 3.93 KB

kafka-streams-internals-GroupedTableOperationRepartitionNode.adoc

File metadata and controls

77 lines (63 loc) · 3.93 KB

GroupedTableOperationRepartitionNode

GroupedTableOperationRepartitionNode is a StreamsGraphNode (indirectly as a BaseRepartitionNode) that is used by KGroupedTableImpl for all supported streaming operators, i.e. reduce, count and aggregate.

// Building a topology using Streams DSL
// Scala API for Kafka Streams
import org.apache.kafka.streams.scala._
import ImplicitConversions._
import Serdes._

val builder = new StreamsBuilder
builder
  .table[String, String]("input")
  .groupBy[String, String]((k,v) => (k,v)) // creates a KGroupedTableImpl
  .count // creates a GroupedTableOperationRepartitionNode
val topology = builder.build
// Note the "*-repartition" topic
scala> println(topology.describe)
Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000001 (topics: [input])
      --> KTABLE-SOURCE-0000000002
    Processor: KTABLE-SOURCE-0000000002 (stores: [input-STATE-STORE-0000000000])
      --> KTABLE-SELECT-0000000003
      <-- KSTREAM-SOURCE-0000000001
    Processor: KTABLE-SELECT-0000000003 (stores: [])
      --> KSTREAM-SINK-0000000005
      <-- KTABLE-SOURCE-0000000002
    Sink: KSTREAM-SINK-0000000005 (topic: KTABLE-AGGREGATE-STATE-STORE-0000000004-repartition)
      <-- KTABLE-SELECT-0000000003

  Sub-topology: 1
    Source: KSTREAM-SOURCE-0000000006 (topics: [KTABLE-AGGREGATE-STATE-STORE-0000000004-repartition])
      --> KTABLE-AGGREGATE-0000000007
    Processor: KTABLE-AGGREGATE-0000000007 (stores: [KTABLE-AGGREGATE-STATE-STORE-0000000004])
      --> KTABLE-MAPVALUES-0000000008
      <-- KSTREAM-SOURCE-0000000006
    Processor: KTABLE-MAPVALUES-0000000008 (stores: [])
      --> none
      <-- KTABLE-AGGREGATE-0000000007

GroupedTableOperationRepartitionNode is created exclusively when GroupedTableOperationRepartitionNodeBuilder is requested to build one.

GroupedTableOperationRepartitionNode takes the following when created:

  • Name of the node

  • Key Serde (Apache Kafka’s Serde for record keys)

  • Value Serde (Apache Kafka’s Serde for record values)

  • Sink Name

  • Source Name

  • Name of the repartition topic

  • ProcessorParameters

GroupedTableOperationRepartitionNode allows creating GroupedTableOperationRepartitionNodeBuilders using groupedTableOperationNodeBuilder factory method.

GroupedTableOperationRepartitionNodeBuilder<K1, V1> groupedTableOperationNodeBuilder()

groupedTableOperationNodeBuilder is used exclusively when KGroupedTableImpl is requested to createRepartitionNode (for all supported streaming operators, i.e. reduce, count and aggregate).

writeToTopology Method

void writeToTopology(final InternalTopologyBuilder topologyBuilder)
Note
 writeToTopology is part of the StreamsGraphNode Contract to…​FIXME.

writeToTopology…​FIXME