Flush aggregator on shutdown #3971
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
|
Change the publisher to wait for enqueued events to be published when Stop is called, up to the configured ShutdownTimeout.
Remove the context param from Aggregator.Run, and instead add a Stop method which accepts a context. When Stop is called it directs Run to exit after performing one final publication. Stop waits for Run to exit, or for the context to be cancelled, whichever comes first. We call the Stop method with a timeout based on the `apm-server.shutdown_timeout` configuration.
publisher.Stop now accepts a context, which is used to interrupt waiting for published events to be acknowledged. This is implemented by introducing a custom beat.ACKer.
simitt
approved these changes
Aug 3, 2020
axw
added a commit
to axw/apm-server
that referenced
this pull request
Aug 20, 2020
* publish: publish enqueued events on shutdown Change the publisher to wait for enqueued events to be published when Stop is called, up to the configured ShutdownTimeout. * aggregation/txmetrics: graceful shutdown Remove the context param from Aggregator.Run, and instead add a Stop method which accepts a context. When Stop is called it directs Run to exit after performing one final publication. Stop waits for Run to exit, or for the context to be cancelled, whichever comes first. We call the Stop method with a timeout based on the `apm-server.shutdown_timeout` configuration. * tests/system: add test for flushing aggregations * publish: add context to publisher.Stop publisher.Stop now accepts a context, which is used to interrupt waiting for published events to be acknowledged. This is implemented by introducing a custom beat.ACKer. * Move publisher ShutdownTimeout handling to beater * Update changelog
axw
added a commit
that referenced
this pull request
Aug 20, 2020
* publish: publish enqueued events on shutdown Change the publisher to wait for enqueued events to be published when Stop is called, up to the configured ShutdownTimeout. * aggregation/txmetrics: graceful shutdown Remove the context param from Aggregator.Run, and instead add a Stop method which accepts a context. When Stop is called it directs Run to exit after performing one final publication. Stop waits for Run to exit, or for the context to be cancelled, whichever comes first. We call the Stop method with a timeout based on the `apm-server.shutdown_timeout` configuration. * tests/system: add test for flushing aggregations * publish: add context to publisher.Stop publisher.Stop now accepts a context, which is used to interrupt waiting for published events to be acknowledged. This is implemented by introducing a custom beat.ACKer. * Move publisher ShutdownTimeout handling to beater * Update changelog
|
successfully tested;
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation/summary
Flush aggregated transaction metrics on shutdown, to minimise data loss. This could be important in a rolling upgrade.
There are some significant, and necessary, changes to the publisher to support an interruptible graceful shutdown-with-timeout. The publisher's Stop method now takes a context, and waits until the context is cancelled, or the published events are acknowledged, before returning. This is all necessary because the shutdown timeout, from the user's point of view, should encompass the entire shutdown procedure -- not just closing the pipeline client.
Checklist
I have considered changes for:
- [ ] documentation- [ ] logging (add log lines, choose appropriate log selector, etc.)- [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)- [ ] telemetry- [ ] Elasticsearch Service (https://cloud.elastic.co)- [ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)- [ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)How to test these changes
Related issues
Closes #3789