Worker and external worker service clean up #199
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Move snapshot build into cut release branch step * Remove snapshot script
Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.7 to 1.10.9. - [Release notes](https://github.com/lib/pq/releases) - [Commits](lib/pq@v1.10.7...v1.10.9) --- updated-dependencies: - dependency-name: github.com/lib/pq dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Eliminate nullness and rawtype errors from java SDK Read class
Eliminate nullness errors from sdks/java/extensions/sql/hcatalog
Lower the lower bound of protobuf to 3.20.3.
Co-authored-by: tvalentyn <[email protected]>
* Add artifact validation to run_rc_validation * config file for artifacts * use java bom contents for validation
Run unit tests with pre-released dependencies.
* content tree rebuilds on sdk change * expandable parent node widget * comment fix * comment fix --------- Co-authored-by: darkhan.nausharipov <[email protected]>
…hon. The vast majority of pipelines are already running on Runner v2 (possibly opted in by the service) and the majority of those that are not are on old SDKs. This paves the way for removing obsolete Runner v1 support from newer SDKs, starting with Beam 2.50.
* added timer package * add timer changes and merged with rebo's pr * timer fired in stateful * error setting new timer in ontimer * looping timers work * send fv instead of bytes * changes to coder/pardo * works for all cases, only cleanup left * remove comments and validate onTimer * generic coder for user key * fixes coder end to end * remove logs * add unit test and refactor * add docs * new example * fix static lint * support emitters * allow input col of CoGBK as well * unit tests, periodic impulse, minor refactor * update PipelineTimer interface, minor refactor, doc comment for example * add warn message * single edge timer coder, rm kv coder check, cache encoder,decoder
…n/chromedriver-binary-gte-100-and-lt-114 Update chromedriver-binary requirement from <113,>=100 to >=100,<114 in /sdks/python
* Added Terraform scripts for TOB infra * ToB Frontend related updates * Update settings.gradle.kts * Deleted redundant file and minor README change * Addressing comments in the PR * Added newline at the end of variables.tf file * Update README.md * Updates related to Tour of Beam infrastructure * Update locals.tf * Output.tf updates * Update output.tf * Updates * Update main.tf * Updates to cloudfunctions_bucket variable * service_account_id changes * Update main.tf * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Bulk update of terraform scripts * Update README.md * Update README.md * Datastore_namespace updates * Update README.md * Update README.md * Update README.md * Update README.md * Update main.tf * Update README.md * Update README.md * Update README.md * Some minor TF updates * Update README.md
* Modify batch IT to use count instead of hash * remove unused varaiable * run spotless, update pipeline state checking * Update timeout to 45 minutes * revert timeout, add additional counter to try and pinpoint missing records * add a log to notify ranges used when workers restart * change counts from metrics to combiners * add a window to streaming test * move the passert to the correct place * Remove extra counter, apply spotless * add additional metric to KafkaWriter * Remove debugging metrics * verify pipeline is not failed * remove extra newline
…pache#26466) This reverts commit 9903b2f.
* keep retrying mass_comment until it has started all jobs * fix lookups
Co-authored-by: tvalentyn <[email protected]>
feat: set override coder via `PValue::apply_with_coder()`
feat: Coder made into object safe
refactor: CoderUrnTree and dyn Coder
chore: complete URN -> dyn Coder codes
feat: add multiplexing data channel
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
http::Uri, instead usingtonic::transport::Uri(which is effectively just ahttp::Urire-export, but could obviously change).std::sync::Mutexwithtokio::sync::Mutex, made sure to avoid creating, starting and re-inserting a worker if their worker id already exists and made sure to start the worker. I'm not sure if external Rust workers will be necessary ever, but we can look into that somewhere in the future.Worker::stopis async now and closes the control channelWorker::newnow just returnsSelf, I checked that all synchronization is handled inimpl WorkerRegisterinstructions has been added (1 down and 6 to go, although register is deprecated)