Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase HTTP client timeout in kubernetes state_deployment tests #17001

Merged
merged 1 commit into from
Mar 13, 2020

Conversation

jsoriano
Copy link
Member

Test data for state_deployment metricset is quite big, and the metricset
may need more than the 10 seconds default timeout of HTTP clients,
provoking sporadic failures in CI.
Increase this timeout for this case.

This should fix flaky tests like:

10:55:29 --- FAIL: TestData (12.25s)
10:55:29     --- FAIL: TestData/ksm-v1.8.0.plain (11.22s)
10:55:29         testdata.go:272: Event was not expected: {"error":{"message":"decoding of metric family failed: net/http: request canceled (Client.Timeout exceeded while reading body)"},"event":{"dataset":"kubernetes.state_deployment","duration":115000,"module":"kubernetes"},"metricset":{"name":"state_deployment","period":10000},"service":{"address":"127.0.0.1:55555","type":"kubernetes"}}
10:55:29         testdata.go:277: Some events were missing:
...
10:55:29         testdata.go:281: 
10:55:29 FAIL
10:55:29 coverage: 74.1% of statements
10:55:29 FAIL	github.com/elastic/beats/v7/metricbeat/module/kubernetes/state_deployment	12.588s

Test data for state_deployment metricset is quite big, and the metricset
may need more than the 10 seconds default timeout of HTTP clients,
provoking sporadic failures in CI.
Increase this timeout for this case.
@jsoriano jsoriano added review flaky-test Unstable or unreliable test cases. [zube]: In Review Team:Platforms Label for the Integrations - Platforms team labels Mar 13, 2020
@jsoriano jsoriano self-assigned this Mar 13, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@blakerouse
Copy link
Contributor

Why is 10 secs not enough time based on the size of the test data? Seems that 10 sec is a large amount of time for an HTTP request.

@jsoriano
Copy link
Member Author

Why is 10 secs not enough time based on the size of the test data? Seems that 10 sec is a large amount of time for an HTTP request.

Yeah, it should be definitely enough, we could investigate it more later. I would only like to address flakiness of this test by now.

It could be about memory. Big Prometheus responses take a lot of memory in Metricbeat, recently I did some profiling with a 27MB response and it required more than 600MB in Metricbeat. The one in this test case is 12MB big, maybe in an overloaded Jenkins worker this is enough to make things slow.

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to fix the test for now, but yeah we should look into why it needs that much time.

@jsoriano jsoriano merged commit 610f996 into elastic:master Mar 13, 2020
@jsoriano jsoriano deleted the flaky-state-deployment branch March 13, 2020 19:06
jsoriano added a commit to jsoriano/beats that referenced this pull request Mar 13, 2020
…astic#17001)

Test data for state_deployment metricset is quite big, and the metricset
may need more than the 10 seconds default timeout of HTTP clients,
provoking sporadic failures in CI. Increase this timeout for this case.

(cherry picked from commit 610f996)
jsoriano added a commit that referenced this pull request Mar 20, 2020
…7001) (#17006)

Test data for state_deployment metricset is quite big, and the metricset
may need more than the 10 seconds default timeout of HTTP clients,
provoking sporadic failures in CI. Increase this timeout for this case.

(cherry picked from commit 610f996)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Unstable or unreliable test cases. review Team:Platforms Label for the Integrations - Platforms team v7.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants