Remove happy tests #11373
Merged
Remove happy tests #11373
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
PR #11373: Size comparison from 4947759 to 01ff8df Full report (38 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
|
woody-apple
approved these changes
Nov 3, 2021
|
/rebase |
|
Rebasing to see if this works, failing a lot this weekend... |
woody-apple
force-pushed
the
remove_happy_tests
branch
from
01ff8df to
70df612
Compare
|
PR #11373: Size comparison from 0c35285 to 70df612 Full report (38 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
|
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
/rebase |
woody-apple
force-pushed
the
remove_happy_tests
branch
from
70df612 to
ecf4740
Compare
pullapprove
bot
requested review from
cecille,
chrisdecenzo,
chulspro,
Damian-Nordic,
electrocucaracha,
emargolis,
harimau-qirex and
hawk248
|
Fast track: this PR has been up while trying to figure out the reason for happy failures and it could not be resolved. PR does a pure deletion. |
andy31415
force-pushed
the
remove_happy_tests
branch
from
ecf4740 to
fe0a74a
Compare
|
PR #11373: Size comparison from 4efa5de to fe0a74a Full report (21 builds for efr32, k32w, linux, p6, qpg, telink)
|
|
Is the functionality tested by happy already covered by cirque? |
mspang
reviewed
Nov 17, 2021
Not as far as I am aware. It is a single test that validates 'multicast works with 5 nodes' from what I can tell. The problem I have with this test is that it seems to be quite flaky and the flake never uncovered any bugs. I am restarting builds several times a day due to this test and I see it causing more pain than value. |
|
PR #11373: Size comparison from e568d70 to 4ae5f11 Full report (26 builds for efr32, k32w, linux, mbed, p6, qpg, telink)
|
|
Removed fast track: had to iterate over it quite a bit and the dns tests may be a material change. For DNS tests, the state right now is that the tests will run on darwin and not on linux Feedback would be appreciated: @mspang @cecille @yunhanw-google @erjiaqing or other folks with experience with happy. |
|
PR #11373: Size comparison from e568d70 to c22f0e8 Full report (38 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
|
|
PR #11373: Size comparison from 02d8a1a to 3793a07 Full report (38 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
|
|
PR #11373: Size comparison from 4d341d4 to 621aac0 Full report (38 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
|
yunhanw-google
approved these changes
Nov 18, 2021
turon
approved these changes
Nov 18, 2021
There was a problem hiding this comment.
I support removing the happy test as well.
Cirque is essentially a superset of happy, and we should just invest in that framework for such functional tests.
It's best to remove the happy dependency now when we only have 1 test using the framework than later when the cost is higher.
|
fast track: PR up for a while and had chats with people familiar with happy and now the PR has more checkmarks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Happy contains a single test (multicast) that is currently very flaky in CI, however I could not reproduce the error locally.
Since the happy platform seems to not be that used and we have alternatives (cirque for full end-to-end with multiple hosts), it seems easier to just remove the happy version instead of trying to debug the flakyness (I was unable to - we don't even seem to preserve debug log artifacts)
Happy seems to come with a lot of complexity (see that almost 3K lines deleted in this PR) for running a single test.
Change overview
Remove the happy test
Testing
CI validates this