Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: Reduce flakiness of test_osqueryi #6688

Merged
merged 3 commits into from
Oct 5, 2020

Conversation

theopolis
Copy link
Member

This PR aims to reduce the flakiness of test_osqueryi by reducing the complexity of flag usage. It also adds the 'default flags' to TimeoutRunner, preventing local state from impacting the tests.

@theopolis theopolis added the test label Oct 2, 2020
@theopolis
Copy link
Member Author

This attempts to address #6667 and #6668.

@Smjert
Copy link
Member

Smjert commented Oct 2, 2020

Thanks for looking into this!

I had a quick look but the part that doesn't convince me is adding more @test_base.flaky decorators.
I would just remove those decorators; those tests should be deterministic, and while originally attempting to restore these python tests, my experience is that the flaky decorator has hidden issues several times.

@theopolis
Copy link
Member Author

I agree, can do. I am pretty optimistic about the other changes in this PR. I will update and remove the decorators momentarily.

@theopolis theopolis closed this Oct 4, 2020
@theopolis theopolis reopened this Oct 4, 2020
@theopolis
Copy link
Member Author

(just running the tests again)

Copy link
Member

@directionless directionless left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable

@theopolis theopolis merged commit c9f89cd into osquery:master Oct 5, 2020
@mike-myers-tob mike-myers-tob added the flaky test A unit test that sometimes or on some systems does not return success, when it should label Oct 5, 2020
aikuchin pushed a commit to aikuchin/osquery that referenced this pull request Jul 11, 2023
…1 to master

* commit 'dc2c7d0ef21f7913448c664ae0ebddd034afc977': (43 commits)
  Changelog 4.5.1 (osquery#6692)
  Fix incorrect stat return checking within process_events (osquery#6694)
  tests: Reduce flakiness of test_osqueryi (osquery#6688)
  Flush stdout with --help (osquery#6693)
  Enable cppcheck target in macOS builds (osquery#6685)
  Fix dirPathsAreEqual the documented way (osquery#6690)
  Add broad exception catching for table execution (osquery#6689)
  Authenticode table with catalog file info (osquery#6677)
  Document max interval for scheduled queries (osquery#6683)
  Fix container overflow in curl_certificate (osquery#6664)
  Update documentation around build steps (osquery#6681)
  Incorporate suggested changes on PR 5789 from Directionless
  Copy-edit and Markdown lint the remaining deployment docs
  Copy-edit and Markdown lint, clarify section headers for HTTP API doc
  Copy-edit and Markdown lint, remove old comment about CI
  Copy-edit and Markdown lint many deployment docs
  Fixed handling of invalid array bound error with EvtNext function (osquery#6660)
  Copy-edit and Markdown lint (just nits) remaining developer docs
  Copy-edit and Markdown lint (just nits) several developer docs
  Copy-edit and Markdown lint the configuration plugin guidance; minor clarification
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky test A unit test that sometimes or on some systems does not return success, when it should test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants