SUMMARY.adoc

Summary

1. Introduction

2. Spark Structured Streaming — Streaming Datasets

Developing Continuous Applications

1. DataStreamReader — Loading Data from Streaming Data Source

2. Streaming Source

   1. FileStreamSource

   2. KafkaSource

      1. KafkaRelation

      2. KafkaSourceRDD

      3. CachedKafkaConsumer

      4. KafkaOffsetReader

      5. ConsumerStrategy Contract for KafkaConsumer Providers

      6. KafkaSourceOffset

   3. MemoryStream

   4. RateStreamSource

   5. TextSocketSource

3. Streaming Operators / Streaming Dataset API

   1. dropDuplicates Operator — Streaming Deduplication

   2. explain Operator — Explaining Query Plan

   3. groupBy Operator — Untyped Streaming Aggregation (with Implicit State Logic)

   4. groupByKey Operator — Streaming Aggregation (with Explicit State Logic)

   5. withWatermark Operator — Event Time Watermark

4. window Function — Stream Time Windows

5. KeyValueGroupedDataset — Streaming Aggregation

   1. mapGroupsWithState Operator — Stateful Streaming Aggregation (with Explicit State Logic)

   2. flatMapGroupsWithState Operator — Arbitrary Stateful Streaming Aggregation (with Explicit State Logic)

6. GroupState — State Per Group in Stateful Streaming Aggregation

   1. GroupStateImpl

   2. GroupStateTimeout

7. DataStreamWriter — Writing Datasets To Streaming Data Sinks

   1. ForeachWriter

   2. OutputMode

   3. Trigger — How Frequently to Check Sources For New Data

8. Streaming Sink — Adding Batches of Data to Storage

   1. ConsoleSink for Showing DataFrames to Console

   2. FileStreamSink

   3. ForeachSink

   4. KafkaSink

   5. MemorySink

9. StreamingQuery

10. StreamingQueryManager — Streaming Query Management

    1. UnsupportedOperationChecker

11. Configuration Properties

Structured Streaming V2

1. StreamWriteSupport

Demos

1. groupBy Streaming Aggregation with Append Output Mode

2. Developing Custom Streaming Sink (and Monitoring SQL Queries in web UI)

3. current_timestamp Function For Processing Time in Streaming Queries

4. Using StreamingQueryManager for Query Termination Management

Monitoring

1. StreamingQueryListener — Intercepting Streaming Events

   1. StreamingQueryProgress

2. Web UI

3. Logging

Extending Structured Streaming

1. DataSource — Pluggable Data Source

2. StreamSourceProvider — Streaming Data Source Provider

   1. KafkaSourceProvider — Data Source Provider for Apache Kafka

   2. RateSourceProvider

   3. TextSocketSourceProvider

3. StreamSinkProvider

   1. ConsoleSinkProvider

Query Planning and Execution

1. StreamExecution — Execution Environment of Streaming Dataset

   1. StreamingQueryWrapper

   2. ProgressReporter

2. TriggerExecutor

3. IncrementalExecution — QueryExecution of Streaming Datasets

4. StatefulOperatorStateInfo

5. StreamingQueryListenerBus — Notification Bus for Streaming Events

Logical Operators

1. EventTimeWatermark Unary Logical Operator

2. FlatMapGroupsWithState Unary Logical Operator

3. Deduplicate Unary Logical Operator

4. MemoryPlan Logical Query Plan

5. StreamingRelation Leaf Logical Operator for Streaming Source

6. StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution

Physical Operators

1. EventTimeWatermarkExec Unary Physical Operator for Accumulating Event Time Watermark

   1. EventTimeStatsAccum Accumulator

2. FlatMapGroupsWithStateExec Unary Physical Operator

3. StateStoreRestoreExec Unary Physical Operator — Restoring State of Streaming Aggregates

4. StateStoreSaveExec Unary Physical Operator — Saving State of Streaming Aggregates

   1. Demo: StateStoreSaveExec with Complete Output Mode

   2. Demo: StateStoreSaveExec with Update Output Mode

5. StreamingDeduplicateExec Unary Physical Operator for Streaming Deduplication

6. StreamingRelationExec Leaf Physical Operator

7. StreamingSymmetricHashJoinExec

8. WatermarkSupport Contract for Streaming Watermark in Unary Physical Operators

Execution Planning Strategies

1. FlatMapGroupsWithStateStrategy Execution Planning Strategy for FlatMapGroupsWithState Logical Operator

2. StatefulAggregationStrategy Execution Planning Strategy for EventTimeWatermark and Aggregate Logical Operators

3. StreamingDeduplicationStrategy Execution Planning Strategy for Deduplicate Logical Operator

4. StreamingRelationStrategy Execution Planning Strategy for StreamingRelation and StreamingExecutionRelation Logical Operators

Offsets and Checkpointing

1. Offset

2. MetadataLog — Contract for Metadata Storage

3. HDFSMetadataLog — MetadataLog with Hadoop HDFS for Storage

   1. BatchCommitLog — HDFSMetadataLog for Batch Completion Log

   2. CompactibleFileStreamLog

   3. OffsetSeqLog — HDFSMetadataLog with OffsetSeq Metadata

4. OffsetSeqMetadata

Managing State in Stateful Streaming Aggregations

1. StateStore — Streaming Aggregation State Management

   1. StateStoreOps — Implicits Methods for Creating StateStoreRDD

   2. StateStoreProvider

   3. StateStoreUpdater

   4. StateStoreWriter — Recording Metrics For Writing to StateStore

2. HDFSBackedStateStore

   1. HDFSBackedStateStoreProvider

3. StateStoreRDD — RDD for Updating State (in StateStores Across Spark Cluster)

4. StateStoreCoordinator — Tracking Locations of StateStores for StateStoreRDD

   1. StateStoreCoordinatorRef Interface for Communication with StateStoreCoordinator

Varia

1. StreamProgress Custom Scala Map