Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kafka extension to agent #1338

Closed
kford-newrelic opened this issue Jun 22, 2023 · 1 comment
Closed

Add kafka extension to agent #1338

kford-newrelic opened this issue Jun 22, 2023 · 1 comment
Assignees
Labels
3 Story Point Estimate enhancement New feature or request

Comments

@kford-newrelic
Copy link
Contributor

Description

Add existing kafka  extension to the agent such that additional metrics are reported (see also this PoC). 

Acceptance Criteria

  • clientId is reported for all metrics, allowing us to query metrics for specific clients
  • Per-topic and per-node metrics are reported by default. (As of release 2.1.1, you can disable per-topic metrics by setting kafka.metrics.topic.metrics.disabled: true and disable per-node metrics by setting kafka.metrics.node.metrics.disabled: true in the agent config. The environment variable equivalents are NEW_RELIC_KAFKA_METRICS_TOPIC_METRICS_DISABLED and NEW_RELIC_KAFKA_METRICS_NODE_METRICS_DISABLED)
  • The client version is reported as a gauge by default
  • counter metrics are reported as deltas of total values which allows "natural" bucketing of rates in NRQL. The more popular rate gauges we typically use are bucketed on the Kafka/client side.
  • Requests to seed brokers and group coordinators are called out clearly and (in the case of group coordinators) node IDs properly identified.
  • The structure of the metric namespace has been redesigned to play nice with NRQL semantics for timeslice queries. We currently use custom metrics.
  • The Kafka APM Nerdlet will automatically display Kafka metrics generated by this agent extension, giving service owners Kafka metrics "for free".
  • Events are produced describing per-clientId configuration to unambiguously identify the effective configuration
  • Report (and allow faceting by) partitionId (NEWRELIC-343)
  • Report (and allow faceting by) topicId (NEWRELIC-3246)

Design Consideration/Limitations

Note that this extension could result in high overhead, so users should be able to configure whether this functionality is enabled (it should be disabled by default)

Dependencies

N/A

Additional context

N/A

@kford-newrelic kford-newrelic added the enhancement New feature or request label Jun 22, 2023
@workato-integration
Copy link

@kford-newrelic kford-newrelic added the 3 Story Point Estimate label Jun 22, 2023
@meiao meiao self-assigned this Jul 13, 2023
@kford-newrelic kford-newrelic moved this to In progress in Java Engineering Board Jul 14, 2023
@meiao meiao moved this from In progress to Needs Review in Java Engineering Board Jul 31, 2023
@meiao meiao closed this as completed Aug 28, 2023
@github-project-automation github-project-automation bot moved this from Needs Review to Code Complete/Done in Java Engineering Board Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 Story Point Estimate enhancement New feature or request
Projects
Archived in project
Development

No branches or pull requests

2 participants