Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc: Enhance and expand DLQ docs #12959

Merged
merged 2 commits into from
Jul 22, 2021
Merged

Conversation

karenzone
Copy link
Contributor

@karenzone karenzone commented Jun 2, 2021

PREVIEW: https://logstash_12959.docs-preview.app.elstc.co/guide/en/logstash/master/resiliency.html

Release notes

Expand documentation for the dead letter queue (DLQ) to help users understand its purpose and management

What does this PR do?

We have identified improvements for the DLQ (#10493), and those improvements are pending design, scoping, and prioritization. We can help users in the interim by better explaining the design intent and offering maintenance techniques to help them use DLQs more effectively.

@karenzone
Copy link
Contributor Author

@yaauie @acchen97 Thanks for the productive conversation on Wednesday. Here's a PR that captures what we discussed.
I noticed that the original docs issue mentioned tracking DLQ size on disk, and so I created a place in the doc for that info. We can fill the gaps with another quick conversation, comments/suggestions in the PR, a thread in slack... whatever works best for you.

@karenzone
Copy link
Contributor Author

@yaauie Let's circle back around on this one when you have time, please. <3

Copy link
Member

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry for missing this -- I've added some specifics to hopefully help you fill the gaps.

In general, I think this improves the docs by better setting the tone and expectations.

docs/static/dead-letter-queues.asciidoc Show resolved Hide resolved
[[dlq-size]]
==== Track dead letter queue size

ToDo: Add info on how to track size before it becomes a problem
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The size of the DLQ for each pipeline that has it enabled is available in the node stats API at the address:
pipelines.${pipeline_id}.dead_letter_queue.queue_size_in_bytes.

@karenzone karenzone marked this pull request as ready for review July 10, 2021 00:06
@karenzone
Copy link
Contributor Author

@yaauie Thanks for the comments! I've added them and would appreciate another review.

Copy link
Member

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ✨

@karenzone karenzone merged commit f050458 into elastic:master Jul 22, 2021
@karenzone karenzone deleted the dlq-notes branch July 22, 2021 20:13
karenzone added a commit to karenzone/logstash that referenced this pull request Jul 22, 2021
karenzone added a commit to karenzone/logstash that referenced this pull request Jul 22, 2021
karenzone added a commit to karenzone/logstash that referenced this pull request Jul 22, 2021
karenzone added a commit that referenced this pull request Jul 22, 2021
karenzone added a commit that referenced this pull request Jul 22, 2021
kares added a commit to kares/logstash that referenced this pull request Aug 23, 2021
* master:
  Update releases list (elastic#13149)
  Bundler: freeze lockfile on run, and "normalize" platform on plugin changes (elastic#13015)
  Doc: Move shared region tags for breaking changes to 8.0.0 content (elastic#13131)
  Update Snakeyaml version to 1.29 (elastic#13129)
  Doc: Add breaking changes to breaking changes page (elastic#13130)
  Added faraday-* and ruby2_keywords notices to licences list (elastic#13126)
  Release notes draft for 8.0.0-alpha1 (elastic#13098)
  valid variable expansion for the batch files (elastic#12912)
  Doc: Enhance and expand DLQ docs (elastic#12959)
  Update releases list with 7.13.4 (elastic#13088)
  Doc:Fix typo and adjust keystore text (elastic#12779)
  doc: add pipeline.ecs_compatibility docs (elastic#12421)
  ecs_compatibility: revert breaking change; keep `disabled` as default for 8.0.0 (elastic#13080)
  Doc: Add kafka schema registry setting to troubleshooting (elastic#13073)
  [Test] Fix Unix acceptance tests (elastic#13071)
  Move retrieval of Stack version from Gradle's configuration to execution (elastic#13042)
  Update releases list with 6.8.17 and 7.13.3 (elastic#13041)
  Docs: keep elastic_app_search output meta-data (elastic#13048)
  Fix LS benchmarking tool to work with releases >= 7.10.0 (elastic#13052)
  Revert "Change Gradle's :logstash-integration-tests:integrationTests task to depends on copyES (elastic#12847)" (elastic#13045)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants