-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI] Failure in ml.integration.RegressionIT.testStopAndRestart #47612
Comments
Pinging @elastic/ml-core (:ml) |
@talevy Could you please add links to CI and the build scans? |
Here’s one example: Jenkins: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/362/console |
Looks like it’s due to:
|
This is really unexpected. I'll look into how this could happen. |
It just failed again, but on a different assertion and no message about the C++ process exiting early (unless I missed it - possible as I looked on a phone). Details are: |
I'll mute it for now. |
I have understood why these failures may happen. If the stopping occurs right after reindexing is finished but before we refresh the destination index, we don't refresh at all. The job is started again right after and jumps into the analyzing state. However, the data is still not searchable. This is why we see that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. The fix is quite simple actually. There is no reason to perform the refresh at that part of the code. We can instead do it in |
If a job stops right after reindexing is finished but before we refresh the destination index, we don't refresh at all. If the job is started again right after, it jumps into the analyzing state. However, the data is still not searchable. This is why we were seeing test failures that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. We fix this by moving the refresh of the dest index right before we start the process so it always ensures the data is searchable. Closes elastic#47612
) If a job stops right after reindexing is finished but before we refresh the destination index, we don't refresh at all. If the job is started again right after, it jumps into the analyzing state. However, the data is still not searchable. This is why we were seeing test failures that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. We fix this by moving the refresh of the dest index right before we start the process so it always ensures the data is searchable. Closes #47612
…elastic#48090) If a job stops right after reindexing is finished but before we refresh the destination index, we don't refresh at all. If the job is started again right after, it jumps into the analyzing state. However, the data is still not searchable. This is why we were seeing test failures that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. We fix this by moving the refresh of the dest index right before we start the process so it always ensures the data is searchable. Closes elastic#47612 Backport of elastic#48090
…elastic#48090) If a job stops right after reindexing is finished but before we refresh the destination index, we don't refresh at all. If the job is started again right after, it jumps into the analyzing state. However, the data is still not searchable. This is why we were seeing test failures that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. We fix this by moving the refresh of the dest index right before we start the process so it always ensures the data is searchable. Closes elastic#47612 Backport of elastic#48090
…#48090) (#48196) If a job stops right after reindexing is finished but before we refresh the destination index, we don't refresh at all. If the job is started again right after, it jumps into the analyzing state. However, the data is still not searchable. This is why we were seeing test failures that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. We fix this by moving the refresh of the dest index right before we start the process so it always ensures the data is searchable. Closes #47612 Backport of #48090
…#48090) (#48197) If a job stops right after reindexing is finished but before we refresh the destination index, we don't refresh at all. If the job is started again right after, it jumps into the analyzing state. However, the data is still not searchable. This is why we were seeing test failures that we start the process expecting X rows (where X is lower than the expected number of docs) and we end up getting X+. We fix this by moving the refresh of the dest index right before we start the process so it always ensures the data is searchable. Closes #47612 Backport of #48090
This still seems to be failing, in |
This seems to be a different failure:
@przemekwitek Could you take a look at this? |
Sure |
This failure is likely unrelated as it affects outlier detection analysis, not regression analysis. |
Something has triggered this test to fail 14 times in CI, starting on October 2.
reproduction step:
Failed with stacktrace:
The text was updated successfully, but these errors were encountered: