Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/Flinkmetricsreceiver] Apache Flink Metric Receiver #10121

Merged
merged 36 commits into from
Jun 1, 2022

Conversation

JonathanWamsley
Copy link
Contributor

Description: Add New Component: Apache Flink Metric Receiver

Link to tracking Issue: New Component: Apache Flink Metric Receiver

Testing: A series of testing was added including:

  • config_test.go
  • client_test.go
  • scraper_test.go
  • factory_test.go
  • integration_test.go

Documentation: README.md documentation.md

@JonathanWamsley JonathanWamsley force-pushed the flinkmetricsreceiver branch 2 times, most recently from d086817 to 5744c6e Compare May 18, 2022 09:57
internal/components/components.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/client.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/client.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/client.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/client.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/config.go Show resolved Hide resolved
receiver/flinkmetricsreceiver/factory.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/integration_test.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/README.md Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/README.md Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/Makefile Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/config.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/scraper.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/client.go Outdated Show resolved Hide resolved
@JonathanWamsley JonathanWamsley force-pushed the flinkmetricsreceiver branch 2 times, most recently from 32ced72 to f8eed3a Compare May 18, 2022 17:54
@JonathanWamsley JonathanWamsley marked this pull request as ready for review May 18, 2022 19:40
@JonathanWamsley JonathanWamsley requested review from a team and djaglowski May 18, 2022 19:40
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/scraper.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/scraper.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Show resolved Hide resolved
@JonathanWamsley JonathanWamsley force-pushed the flinkmetricsreceiver branch 5 times, most recently from 553ec66 to 1743f8f Compare May 23, 2022 13:28
@JonathanWamsley
Copy link
Contributor Author

I updated pr feedback @djaglowski and it is ready to be looked at again please.

receiver/flinkmetricsreceiver/README.md Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
@JonathanWamsley JonathanWamsley force-pushed the flinkmetricsreceiver branch 2 times, most recently from 7f104f5 to 7465bfc Compare May 25, 2022 13:11
@JonathanWamsley
Copy link
Contributor Author

PR is ready for review again please @djaglowski

receiver/flinkmetricsreceiver/metadata.yaml Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/metadata.yaml Outdated Show resolved Hide resolved
@JonathanWamsley JonathanWamsley force-pushed the flinkmetricsreceiver branch 3 times, most recently from 9e336d0 to 07cce9f Compare May 26, 2022 14:56
@JonathanWamsley
Copy link
Contributor Author

JonathanWamsley commented May 26, 2022

Ready to be looked at again @djaglowski please

flink.jvm.cpu.load:
enabled: true
description: The CPU usage (%) of the JVM for a jobmanager or taskmanager.
unit: 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

%?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. I thought you said this metric is measured in %. That's my interpretation from the article you linked.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I could not find a unit though and searching through the metadata.yaml mongo atlas had that convention. so I put it in. I thought you were saying with "%?" to remove that convention or something.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, should the units be % or not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes they should be. The unit can be 1 but not % because that will break in the metadata generation. Maybe "{percentage}" instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"%" should work

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated unit

CHANGELOG.md Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/process.go Outdated Show resolved Hide resolved
receiver/flinkmetricsreceiver/scraper.go Outdated Show resolved Hide resolved
enum: [ jobmanager, taskmanager ]

attributes:
operator_name:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be a followup issue, but I think this should probably be user-visible as name.

Suggested change
operator_name:
operator_name:
value: name

operator_name:
description: The operator name.
type: string
garbage_collector_name:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
garbage_collector_name:
garbage_collector_name:
value: name

@djaglowski djaglowski merged commit c135dab into open-telemetry:main Jun 1, 2022
kentquirk pushed a commit to McSick/opentelemetry-collector-contrib that referenced this pull request Jun 14, 2022
…lemetry#10121)

* wip before integration and concurrency

* tidy up client names

* wip integration test

* wip integration test

* update to flinkmetrics and integration test

* move attributes to resource attributes

* add flink integration files

* update model import name

* add changelog and tidy up

* update mod and lint check

* update with pr feedback

* update pr feedback with status header

* unexport process methods

* go tidy up

* add partial metrics collecting on taskmanager, jobs, and subtasks

* go tidy update

* fix shadow declaration of err

* update pr feedback

* tidy up

* apply scope version update

* make expected responses easier to test when metadata/generated files update

* update to host.name and move status table

* combine jobmanager and taskmanager metrics

* tidy up

* update generated files

* fix mod version

* update pr feedback

* tidy up

* seperated checkpoint in_progress as non-monotonic sum metric

* tidy up local receiver

* update version

* update cpu descriptions

* fix taskmanager/jobs bug and added tests with multiple taskmanagers/jobs

* space

* change non_heap to nonheap

* update unit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants