Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix some flaky tests that were checking if metrics were being emitted. #517

Merged
merged 2 commits into from
Mar 1, 2024

Conversation

gregschohn
Copy link
Collaborator

Fix some flaky tests that were checking if metrics were being emitted.

They had a resource lock around them because they were using some static constructs provided by the Otel SDK. Now the resource locks on tests are gone for the ConditionallyReliableLoggingHttpHandlerTest and each test constructs its own non-static otel sdk. I've also tried to make tests that confirm otel instrumentation a bit safer. They properly shut down metric exporter threads now. They also wait a couple cycles before returning the in-memory exported metrics. I really wish that there were an exporter that didn't have to wait, but at least for now, this seems like the only option. At some point, it might be worth implementing a different MeterProvider which would in turn build out a different implementation hierarchy to return instruments that have getters. Using mock classes to spy on calls could work too.

Description

  • Category: Test fixes
  • Why these changes are required? Make builds more reliable
  • What is the old behavior before changes and new behavior after changes? Tests should be more stable and use less resources (threads)

Issues Resolved

No Jira

Testing

[Please provide details of testing done: unit testing, integration testing and manual testing]

Check List

  • New functionality includes testing
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@AndreKurait
Copy link
Member

I really wish that there were an exporter that didn't have to wait.

Why can't we use InMemoryMetricReader/InMemoryMetricExporter?

Copy link

codecov bot commented Feb 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.88%. Comparing base (9fedef8) to head (7eaf632).
Report is 9 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #517      +/-   ##
============================================
+ Coverage     76.80%   76.88%   +0.08%     
+ Complexity     1388     1386       -2     
============================================
  Files           158      158              
  Lines          6105     6105              
  Branches        532      532              
============================================
+ Hits           4689     4694       +5     
+ Misses         1065     1059       -6     
- Partials        351      352       +1     
Flag Coverage Δ
unittests 76.88% <ø> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

They had a resource lock around them because they were using some static constructs provided by the Otel SDK.
Now the resource locks on tests are gone for the ConditionallyReliableLoggingHttpHandlerTest and each test constructs its own non-static otel sdk.
I've also tried to make tests that confirm otel instrumentation a bit safer.  They properly shut down metric exporter threads now.  They also wait a couple cycles before returning the in-memory exported metrics.
I really wish that there were an exporter that didn't have to wait, but at least for now, this seems like the only option.
At some point, it might be worth implementing a different MeterProvider which would in turn build out a different implementation hierarchy to return instruments that have getters.  Using mock classes to spy on calls could work too.

Signed-off-by: Greg Schohn <[email protected]>
@gregschohn gregschohn merged commit f35c54a into opensearch-project:main Mar 1, 2024
7 checks passed
@gregschohn gregschohn deleted the FixFlakyOtelTests branch April 3, 2024 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants