add watermark metrics for streaming jobs #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
lgtm |
steveniemitz
force-pushed
the
add-watermark-tracking
branch
from
a280c8d to
58a0f89
Compare
tushar19
pushed a commit
that referenced
this pull request
Mar 18, 2022
Adds ReadChangeStreamPartitionDoFn, which is an SDF to read partitions from change streams and process them accordingly. This component receives a change stream name, a partition, a start time and an end time to query. It then initiates a change stream query with the received parameters. Within a change stream, 3 types of records can be received: 1. A Data record 2. A Heartbeat record 3. A Child partitions record Upon receiving #1, the function updates the watermark with the record's commit timestamp and emits the record into the output PCollection. Upon receiving #2, the function updates the watermark with the record's timestamp, but it does not emit any record into the PCollection. Upon receiving #3, the function updates the watermark with the record's timestamp and writes the new child partitions into the metadata table. These partitions will be later scheduled by the DetectNewPartitions component. Once the change stream query for the element partition finishes, it marks the partition as finished in the metadata table and terminates.
steveniemitz
pushed a commit
that referenced
this pull request
Sep 13, 2022
* Tour of Beam frontend blank project * TOBF (ToB frontend): welcome screen (#226) * theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Co-authored-by: darkhan.nausharipov <[email protected]> * removed whitespace from readme (issue-22583) * renamed "content" to "child" to mimic widgets * README in tour-of-beam * translation path rename, grey dot with opacity, footer link text style * report issue in github, grey dot color * table instead of row to clip the laptop image * horizontalHalves & verticalHalves * cropped laptop image * row in an expensive intrinsic height * laptop image in the bottom * ScreenBreakpoints * intrinsic height is not needed! * _WideWelcome and _NarrowWelcome * draft readme * blank line in readme * removed irrelevant info from readme * removed whitespace Co-authored-by: Alexey Inkin <[email protected]> Co-authored-by: darkhan.nausharipov <[email protected]> Co-authored-by: alexeyinkin <[email protected]>
kkdoon
pushed a commit
that referenced
this pull request
Sep 29, 2022
* [Tour of Beam][Frontend][apache#22600] TourScreen layout * theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Tour of Beam frontend blank project [Tour of Beam][Frontend][apache#22600] TourScreen layout TourScreen layout (apache#22600) common theme, constants, split view missing license flutter_gen, summary layout details content layout details no functional widgets in split view main screen todos & translation main screen todos & translation comment fixes #1 ExpansionTileWrapper SplitViewController lists in tour screen widgets comment fixes #1 (31.08) split view package in PGC fixed button overflow splitter theme color comment fixes #2 (31.08) gradlew check welcome screen overflow test (apache#22600) SDK dropdown (apache#22600) flexible complete unit OutlinedButton (apache#22600) renamed PageContainer to TobScaffold dropdown style refinement DropdownButton implicit type sdk instead of e licenses apache#22600 renamed _ShrinkedTour to _NarrowTour apache#22600 tour screen style refinement apache#22600 BeamDivider in PGC apache#22600 removed todo, added license apache#22600 built with text apache#22600 _WideWelcome with IntrinsicHeight (apache#22600) Co-Authored-By: darkhan.nausharipov <[email protected]> * addressing review comments apache#22600 replaced magic numbers apache#22600 comments (apache#22600) comments apache#22600 comments apache#22600 comments apache#22600 comments apache#22600 comments, flutter 3.3.0 upgrade apache#22600 renamed ActionPadding to ActionVerticalPadding apache#22600 actions formatting apache#22600 * branded sign in buttons apache#22600 * _BrandedSignInButtons apache#22600 * _Divider color apache#22600 * profile apache#22600 * moved split_view from PGC into ToB apache#22600 * indentation fix apache#22600 * split ProfileContent into widgets apache#22600 * Extract playground components to a separate package (apache#22600) * Minor fixes (apache#22600) * Address review issues (apache#22600) * Upgrade Flutter to v3.3.2 (apache#22600) * Add precommit Gradle task for playground_components, add code generation to frontend Gradle task, remove generated mocks, fix linter issues (apache#22600) * startTour button (apache#22600) * lint fixes (apache#22600) * Fix highlighting for Python and SCIO (apache#22600) Co-authored-by: darkhan.nausharipov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.