test(e2e): settings tests #167
Conversation
QZera
force-pushed
the
feat/settings-e2e-tests
branch
11 times, most recently
from
65d319d to
1046eb8
Compare
QZera
force-pushed
the
feat/settings-e2e-tests
branch
from
1046eb8 to
800e371
Compare
QZera
force-pushed
the
feat/settings-e2e-tests
branch
17 times, most recently
from
46b1f43 to
5a28f5e
Compare
QZera
force-pushed
the
feat/settings-e2e-tests
branch
10 times, most recently
from
fb82cd9 to
a62b0b6
Compare
QZera
force-pushed
the
feat/settings-e2e-tests
branch
from
c29da1e to
c1b8f16
Compare
- adds tests for the "path to rad binary" setting - adds cleanup logic to the onboarding suite - updates config to run onboarding and settings suites sequentially - updates e2e ci workflow to ensure both platforms run regardless of each other's outcome Signed-off-by: Zacharias Fragkiadakis <[email protected]>
- Adds test for the "path to node home" setting
- Allows skipping tests based on platform by including `@skip{platform}CI` in the test name
Signed-off-by: Zacharias Fragkiadakis <[email protected]>
- Sets "waitFor" timeout globally - Extracts some functions to helper files - Generally cleans up the test suite Signed-off-by: Zacharias Fragkiadakis <[email protected]>
QZera
force-pushed
the
feat/settings-e2e-tests
branch
from
747f07b to
b79dae5
Compare
There was a problem hiding this comment.
I still need to have a second look, particularly for the details of the logic in settings.spec.ts, but for now some early comments.
In addition to the inline ones, I want to address a couple more general ones:
- regarding changing setting values, I wonder if we take into account (and if we should, if not) the debounce that seems to be between setting changing the input field's value and it "registering". Perhaps we could "blur" the focus instead of waiting which seems to trigger registering of the new value instantly.
- if running (a subset, as previously discussed) of the e2e tests locally is now safe, i.e. won't impact local setup, we should revisit restoring the npm script
test:e2einside the top-level script "test".
| /** | ||
| * In Linux CI: | ||
| * - The extension is having issues resolving the correct node home directory (RAD_HOME). | ||
| * - Even when node home is set explicitly in the settings, the extension incorrectly reports it | ||
| * as non-authenticated. | ||
| */ | ||
| // eslint-disable-next-line max-len | ||
| describe('VS Code, when updating the "Path to Radicle to Node Home" setting, @skipLinuxCI', () => { |
There was a problem hiding this comment.
Skipping tests, especially on Linux CI, along with the whole setup that comes to support this on the runner level is both hacky and and risky.
These various odd issues we've been having on the environment level smell to me that we need to look upwards to sniff out the root cause (e.g. in e2e-testing.yaml, or wdio config, or investigate CLI bug with the maintainers, ...?) or it will keep forcing more hacks down our throats.
At the very least, here could we at least source the value of RAD_HOME alternatively e.g. with rad self --home and remove all those @skipLinuxCI markers and related supporting logic?
There was a problem hiding this comment.
I believe this might actually be a bug in the extension on Linux specifically.
I've created a branch and a draft PR with a minimal test to illustrate this. (Check the result of the ubuntu job)
However since I'm not on Linux, I can't reproduce it locally, and I'd like to confirm with you before creating a bug. Could you help reproduce this?
As far a I can tell, based on e2e-testing.yaml, the steps to reproduce this are:
- before installing Radicle, set
RAD_HOMEto non-default path - install radicle
- create a new radicle identity (
rad auth) - start VSCode
- open the Radicle channel in the Output Panel
The radicle identity should be picked up correctly, but according to my testing it is not.
There was a problem hiding this comment.
It looks like for the tests we're creating a non-password protected Radicle identity.
Yet in the test you shared we're (falsely!) expecting to read "Using already unsealed Radicle identity" in the Radicle Output channel. In this case the expected output is "Using non-password-protected Radicle identity".
In fact it would be better that all our tests use a password-protected identity by default to better simulate real-world scenarios. There's also good argument to cover with tests some basic functionality (similar to what you've done in your testing PR above, actually) to ensure we haven't broken support for non-password-protected identities (has happened to me in the past..).
Signed-off-by: Zacharias Fragkiadakis <[email protected]>
QZera
force-pushed
the
feat/settings-e2e-tests
branch
2 times, most recently
from
a2e4e4f to
1e20b85
Compare
Signed-off-by: Zacharias Fragkiadakis <[email protected]>
Signed-off-by: Zacharias Fragkiadakis <[email protected]>
Signed-off-by: Zacharias Fragkiadakis <[email protected]>
Prerequisite #162
Notes:
RAD_HOMEenv var)