Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to block checking if metric descriptors exist to prevent rate limiting by Google #621

Closed
sethrwebster opened this issue Sep 19, 2023 · 0 comments · Fixed by #623
Closed

Comments

@sethrwebster
Copy link
Contributor

Is your feature request related to a problem? Please describe.
When a large number of clients are started at the same time (e.g. 1000), the calls to the Google Cloud Monitoring API to check if metric descriptors exist will result in the clients being rate limited. Until a client successfully completes the metric descriptor check, it will be blocked from sending metrics. This causes some clients to fail to send metrics for some number of minutes. Relatively short-lived clients, such as those that are only running for 5-10 minutes, may never be able to send metrics. This rate limit is also a project-level rate limit so any other client in the project that tries to get or create metric descriptors will also be rate limited.

Describe the solution you'd like
Other projects that interact with the Google Cloud Metrics API have an option to assume metric descriptors already exist. It would be useful to have that option in this library such that short lived clients could come and go without contributing to the rate limit.

Additional context
I will submit a PR shortly (it is a pretty small change).

sethrwebster added a commit to mablhq/opentelemetry-operations-js that referenced this issue Sep 19, 2023
This change creates an option to skip checking if metric descriptors exist before
sending metrics. Skipping the metric descriptor check can prevent the clients from
getting rate limited by Google when a large number are all started at the same time.
aabmass pushed a commit that referenced this issue Sep 22, 2023
…to skip checking for metric descriptor existence (#623)

* Issue #621

This change creates an option to skip checking if metric descriptors exist before
sending metrics. Skipping the metric descriptor check can prevent the clients from
getting rate limited by Google when a large number are all started at the same time.

* Updated option name based on PR feedback.

* Updated comment based on PR feedback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant