bunit.web.e2e: Make it easy to create E2E tests that execute JavaScript

[egil]

Some test scenarios that rely on JSInterop and JavaScript to execute to work, cannot be tested with bUnit currently.

This could be solved by:

• Providing a new test context, E2ETestContext, in its sub library e.g. bunit.web.e2e.
• Use Selenium in E2ETestContext to drive tests, and abstract away all the hardship of setting that up.
• E2ETestContext and other APIs should mimic those in the bunit.web library as much as possible, to make the transition between the normal component tests and e2e tests easier.

Questions:

1. Would this work in Razor based tests?
2. Are there other alternatives to Selenium worth considering?

This could be based on the code that the Blazor team has in its E2E tests for Blazor.

[mrpmorris]

Selenium will be slow.

Could you use the Blazor HTML render to create your HTML string, set the HTML in a browser component, and have some kind of interceptor for IJSRuntime so you can assert exchanges between the two?

I'm hoping that pumping HTML directly into the component will be quicker than automating a browser.

Having a separate context for E2E could be good. It could be used to build up the rest of the page by specifying the <App> component to use as the starting point for the whole HTML rendering.

[egil]

My primary reason for thinking about Selenium is because that is what the Blazor team uses in their E2E tests, e..g. https://github.com/dotnet/aspnetcore/blob/master/src/Components/test/E2ETest/Tests/ComponentRenderingTest.cs

An initial proof-of-concept need not be fast, it for the few cases that are hard to test in regular bUnit, and it is more important that it is easy to write stable and consistent tests.

[mrpmorris]

Selenium is what the Blazor team uses, but I expect they aren't happy that a single test takes a few seconds - which is one of the reasons bUnit is so good.

I think my suggested approach would be stable and consistent, if it is possible at all.

[egil]

I think my suggested approach would be stable and consistent, if it is possible at all.

That would be great. Can you go into a bit more detail on what you mean by "a browser component" and "kind of interceptor for IJSRuntime".

Remember, the need here is to actually run the JavaScript in a browser context, such that it can update the DOM if needed. Intercepting IJSRuntime invocations is already possible with bUnit, the problem is when JavaScript needs to execute to make a test work. The inspiration for this is #141.

[mrpmorris]

A browser component

Something that will work on Windows, Mac, Linux: Like the Steve Sanderson used here - https://github.com/SteveSandersonMS/WebWindow

the problem is when JavaScript needs to execute to make a test work

Your class implementation could act like a Mock but that actually passes the invokes through to the real IJSRuntime. In the Browser perhaps you could swap out the JS reference that calls through to C# and have it call your object instead (which then passes it through). A full man-in-the-middle scenario.

[egil]

I would love to see a proof of concept of this... admittedly I am having a hard time following your thinking of this.

[PandaMagnus]

Would something like Playwright work? It seems to avoid some of the heartburn that Selenium has but still seems to drive a "full" (albeit, defaulted to headless which should help with the reliability,) browser.

https://playwright.dev/

I have zero experience with playwright other than reading about it, so I could be way off base.

Otherwise, I'd be happy to help work on a Selenium-based example.

[egil]

@PandaMagnus thanks.

I think, in the spirit of Blazor, we need to stay as much as possible in C# land, which rules out playwright, as far as I can see, at least according to their page: "Playwright is a Node library to automate Chromium, Firefox and WebKit with a single API. Playwright is built to enable cross-browser web automation that is ever-green, capable, reliable and fast."

Otherwise, I'd be happy to help work on a Selenium-based example.

Please do. Create a bunit.web.e2e and bunit.web.e2e.tests projects and put your code there and send a PR in. Doesn't have to be ready for production, an experimental/alpha thing is a good start.

Remember, to me, the point of this is not just to expose Selenium and its API to the user, but to wrap it in a bUnit like API. Otherwise, I would rather just post a guide in the docs on how to do e2e testing using Selenium, and not have to maintain an additional package.

Ideally, we abstract away Selenium enough that it can be replaced with an alternative down the line. If too much of Selenium shines through, I think we should consider naming the package and project bunit.web.selenium instead of bunit.web.e2e, to make it obvious to users that they are getting Selenium with bUnit helpers.

Before you commit a bunch of time to this, please post your design thoughts first so we can align and not waste time. Thanks!

[PandaMagnus]

@egil Just following up on this. Apologies I got busy and this fell off my radar. It's back on it now, though, and I intend to sit down this weekend or next to put together a working proof of concept.

[egil]

@egil Just following up on this. Apologies I got busy and this fell off my radar. It's back on it now, though, and I intend to sit down this weekend or next to put together a working proof of concept.

No apologies needed. I look forward to see what you come up with.

[PandaMagnus]

I gotta fix some issues with .net5 on my machine, so I'm not ready to PR this yet (should have that sorted out in a few days,) but if you wanted to look at the stubbed out version, it's here:

main...PandaMagnus:main

I'm using a package I maintain for lightly wrapping Selenium so I don't have to re-implement a bunch of webdriver creation/webdriverwait code. If you don't want that dependency, it should be pretty straightforward to strip out and replace. I haven't added samples yet, but for comparison to the ClickMeTest, the E2E syntax will look something like:

// Arrange
using var context = new E2ETestContext();
var page = context.PageComponent<SomePageUnderTest>();
var buttonElement = page.Find("input[name='someBtn']")
   .SetPollingIntervalMilliseconds(100) 
   .SetTimeoutSeconds(1); // Avoid throwing for up to 1s if we try to find the button before it's available in the DOM

// Act
buttonElement.ClickWhenReady(); // Specific syntax from my package to automagically re-try the wait if the button isn't quite ready yet.
// Not yet a good way to do consecutive clicks or ctrl+click without just recalling the same code or building an Actions object. Extension or wrapper method probably makes sense to handle that with bUnit style syntax.

// Assert

[mrpmorris]

I like it, but it's a bit verbose. What about having polling and timeout as defaulted parameters on Page.Find? Also consider making that async to improve performance on parallel tests (if that's even possible).

[PandaMagnus]

Makes sense to default the timeout and polling. I'll make that change before I sort out my local copy and put the actual PR in.

RE: async, Selenium doesn't have any async calls that I know of (forewarning, I haven't looked into the new version they've got in beta right now, so that could change,) but you can run tests in parallel as long as each test gets its own WebDriver instance. So as long as the test context is instantiated per test, parallelization shouldn't be a problem as long as so many don't run in parallel that the machine runs out of memory (each instance will get it's own chrome process.)

[mrpmorris]

I was thinking of the polling. If the 100ms loop were a Task.Delay

Maybe a future enhancement after testing if it makes enough of a difference :)

[egil]

@PandaMagnus I am exited about this. A few thoughts based on what you have shared:

1. For this to really make sense under the bUnit brand, the API should follow bUnit's as much as possible. Ideally (but not realistic) would be that users could simply replace TestContext with E2ETestContext in their tests. So instead of a PageComponent method we should have a RenderComponent method, etc.

2. An important philosophy of my design decisions with bUnit is that tests should be as easy and intuitive to write as regular POCO unit tests. That means providing good defaults as much as possible, getting folks 90% of the way (put differently, having defaults for polling/timeouts like Peter suggests).

3. Could we render a component with both bUnit and selenium, and maybe use the output from bUnit to detect when selenium is done renderinger and is ready to be queried. Or maybe always render components inside a "bunit root component" that can detect renders, child component state, and signal to selenium when the page is ready?

4. I prefer to have as few dependencies as possible to 3rd party libraries. It just makes things simpler in the long run, also with the plans for joining the .net foundation when bUnit hits v1.

I have no experience with Selenium myself, so I do not know what is possible, so I am very happy to see lots of community engagement on this issue.

Thanks again for your efforts with this!

[PandaMagnus]

@mrpmorris Understood. I'll go back through Selenium's polling code, I seem to recall it's invoking Thread.Sleep() under the covers, but could be wrong. Thanks for catching that!

@egil I'll do the PageComponent rename and defaulting today or tomorrow, along with moving implementation directly to Selenium. At that point, I'll put in the PR (I got my VS sorted out so I can actually verify the code builds now. :)) Do you want samples included with that?

Point 3 is interesting; I'll tinker around with that as my next item to work with.

[egil]

With regards to testing when a component is rendered, we can also utilize the new IComponentActivator hook in Blazor to at least know when a component is instantiated, since that is controlled through it.

In theory, we could create an adapter around components that are rendered in tests, so we know exactly when their life cycle method are called.

At the very least, we will know when to poll the DOM, so it doesn't have to be after X ms, but can happen as soon as a component is rendered. That should help with the speed of the tests.

[PandaMagnus]

@egil Okay, updated the syntax to mirror bunit closer. I also stubbed out an example of mocking Selenium to unit test this. Also pulled the project in to stub out a sample (although the sample will need a running website to truly work.)

main...PandaMagnus:main

I'd like to take the time to add some real unit tests before putting the PR in. But let me know any other thoughts on this. I'm deliberately holding off on hooking into the IComponentActivator and/or using bunit output purely because I'm not as comfortable with that yet.

#mrpmorris I did confirm that under the covers Selenium is using a Thread.Sleep in that particular polling statement. Although now that I type that, I suppose we could implement our own awaitable polling statement. I started to go down that route a year or two ago before abandoning it in favor of their current WebDriverWait (that uses Thread.Sleep.) The logic ended up looking a heck of a lot like Polly, which is why I ultimately abandoned it.

[PandaMagnus]

Maybe I'm slow on the uptick, but what if a single razor component was dynamically wrapped into a single page for a test? For some reason I had it in my head that these tests would be used on a fully deployed website, but if we had a... I dunno, "page scaffolder" or something to consume a single razor component, deploy just that, and then run these tests, it would help maintain syntactic parity between the unit and E2E tests.

[egil]

... I dunno, "page scaffolder" or something to consume a single razor component, deploy just that, and then run these tests, it would help maintain syntactic parity between the unit and E2E tests.

You are heading in the right direction here. Bunit actually also uses a wrapper component that it renders the component under test inside.

Have you looked at how the Blazor teams e2e tests works? I'm am pretty sure they are using Selenium, but have something like you are talking about here.

[PandaMagnus]

@egil I had glanced through them a bit, but didn't spend much a whole lot of time going through them. I'll take a deeper dive into their e2e tests today.

[mrpmorris]

Yes, and they are slow