Test Blazor topic

[guardrex]

From @guardrex at Test and debug Blazor apps topic ...

• Blazor development-time testing (as of 0.2.0, live reloading is still in design)
• Blazor unit testing

A debug topic exists. Because testing isn't the same thing as debugging, the test aspects probably go into a new topic. The dev-time testing/live reloading perhaps go into the existing debugging topic.

[danroth27]

It's going to be a bit before we have our testing story for Razor Components worked out.

[guardrex]

I think this is on hold for dotnet/aspnetcore#5458 and on the roadmap backlog (without an issue) on dotnet/aspnetcore#8177.

I'm going to move this to our HOLD column on the Blazor project for the time being.

[danroth27]

@guardrex Instead of building our own Blazor unit test framework, we've decided to point folks to community based frameworks instead, like bUnit.

@egil do you want to propose an outline of of what this topic should look like?

[egil]

I will Dan. I am currently reworking the bUnit documentation so I am in a writing mode/mood right now :)

@guardrex / @danroth27, a few questions before I dig in:

1. Can you give an indication of the context the testing content will exist in, e.g. on an existing page as a sub section or as a new page? If it is a new page/standalone page, where would it fit into the page hierarchy of the docs?
2. How many words/characters/pages do you expect this topic should cover?

[guardrex]

Hi @egil ... I'm OOF at the moment. I'll get back to you this evening with that info.

[egil]

Hi @guardrex, no problem, it is the weekend after all, there is no rush.

[guardrex]

@egil

1. It will be a new page in the Test, debug, and troubleshoot node of the TOC, probably called Test Razor components ... whatever @danroth27 prefers. You can look at some of the other topics there to see what other authors have done, but don't feel bound by you see there. If you have any questions, feel free to ping us here.
2. One page, and there's no word/character limit.

The best approach is for you to place the content here in an issue comment. I'll set the topic up. We don't have full contribution guidelines for all of the little rules that we follow, and it would take longer for me to explain the whole setup than to just do it. When I load the content, I'll perform a grammar+style pass on it and ping you for review. Then, engineering will review. After sign-off, that's it ... I'll merge it and it will go live.

[egil]

Ok @guardrex, I'll post an outline as soon as possible and get yours, @danroth27 and others feedback, before I start writing.

Ps. I'm using docfx for bUnits docs as well. Can you share how you get syntax highlighting for razor code in your docs? The best I've found is to highlight it as html :-)

[guardrex]

We use razor for the Razor component syntax highlighting. It's an alias for cshtml.

[egil]

I've tried with cshtml as well, it is not recognized by the built-in highlighter in docfx, which I believe is highlight.js. Do you have an extension that adds support for cshtml/razor?

[guardrex]

Yes, I believe that's correct. I saw your issue on their repo, and I hope that they will agree.

I've also asked them about getting the published doc stylesheets used locally so that a local build produces docs that exactly match the styles used by the published docs. They told me that they were working on that for a future release.

[egil]

Just a quick update. This is not forgotten, I just have a lot on my plate the next week's, moving etc., but will try to get something here if time allows.

[danroth27]

It will be a new page in the Test, debug, and troubleshoot node of the TOC, probably called Test Razor components

Yup, sounds right. It would be nice if the same topic also showed up under the Blazor node so that folks can find it there too.

[egil]

It will be a new page in the Test, debug, and troubleshoot node of the TOC, probably called Test Razor components

Yup, sounds right. It would be nice if the same topic also showed up under the Blazor node so that folks can find it there too.

Yeah, I am not sure how obvious it is to most folks that a Razor component = Blazor component, so it should probably located under Blazors TOC too.

[tncbbthositg]

Instead of building our own Blazor unit test framework, we've decided to point folks to community based frameworks instead, like bUnit.

I'm fully in support of having the community develop testing frameworks and bunit is really great. The only concern I have is that at some point, it could be valuable for the Blazor team to dogfood the testing to see where the testing ergonomics could be improved.

One example is that it's difficult it is (well, at least it was difficult for me . . . for what that's worth) to mock components in the component hierarchy or to use shallow rendering to isolate components in unit tests. A lot of the javascript based web frameworks use JS's dynamic nature to support shallow rendering (like Enzyme for React). Others use IoC (like Angular).

I think that it would be a great addition to the value of Blazor if the framework had a healthy set of unit tests. Those tests would not only drive testability, but would also provide some self-documenting of the features too.

IMHO

[SteveSandersonMS]

I think that it would be a great addition to the value of Blazor if the framework had a healthy set of unit tests. Those tests would not only drive testability, but would also provide some self-documenting of the features too.

We do have comprehensive set of end-to-end (E2E) tests of all our components at https://github.com/dotnet/aspnetcore/tree/master/src/Components/test/E2ETest/Tests. Generally we focus more on E2E testing for our components than just unit testing them, because it's important that we validate things like the JS part of the renderer all work correctly with every combination of components.

I understand the testing goals for an application will be a bit different from the testing goals for the framework itself, since you're allowed to assume the framework works correctly :)

We don't currently have an easy way to "mock" a component or do shallow rendering. However if we merge dotnet/aspnetcore#19642 then I think there'd be an easy way to do it.

[tncbbthositg]

Generally we focus more on E2E testing for our components than just unit testing them, because it's important that we validate things like the JS part of the renderer all work correctly with every combination of components.

Thanks for responding @SteveSandersonMS! I just wanted to clarify one thing because I may have caused some confusion. I didn't mean that there should be unit tests in lieu of E2E tests. Personally, I think E2E tests are an important part of a healthy automated test suite. I only mentioned unit tests specifically because they're the ones that seem to be missing.

I think dotnet/aspnetcore#19642 will address the mocking issue, but I don't think this card is about mocking per se; I think it's about testing more generally.

My concern is that if you build a framework where unit testing is not a primary concern, you're more likely to build a framework where unit testing is difficult. I think that eating your own dog food, from a testing perspective, is likely to give you a more testable framework.

I'd like to see Blazor build a large community like React and Angular have, and my fear is that it may be hard to do if testability doesn't improve. I think that it's great to point developers to community based testing frameworks. I just don't think that should be mutually exclusive with unit testing the framework code at development time.

[SteveSandersonMS]

Thanks for your feedback! I appreciate your point.

Like all software teams, we optimize for delivering the best result given the available development resources. This is what's led us to the specific blend of unit and E2E tests we have in the repo today. So far I'm pretty happy that this form of testing has given good results in terms of the correctness and agility of the codebase. However, when opportunities appear to add different forms of tests, I'm sure we'll be happy to take them!

[egil]

Hey @guardrex, sorry about the radio silence on this, I apparently have too many balls in the air right now.

Here is a suggested outline:

1. Introduction to Blazor component testing
   This would cover what it means to test a Blazor component, i.e. types of input you want to provide and the type of things you would normally want to verify during a test. This could also include a short discussion things to be aware of in UI testing, such as verifying html markup.
2. Testing approaches
   This would cover component testing (unit testing) and E2E testing in general, the pros and cons of both. Note, I do not have any experience personally with E2E testing in Blazor, but can write something in general like @SteveSandersonMS does in his blog post on the topic.
3. Example of component testing
   This would show a basic example based on bUnit, perhaps something similar to test example on bUnit’s front page, with a short description and a link to bUnit’s page to learn more.
   I think it is best to not have too many specific bUnit examples, as I would then need to keep these docs updated if the API changes. Maybe there should not be any examples, just a description and a link?

What do you, @SteveSandersonMS, and @danroth27, think?

[egil]

Hi @guardrex

I'm back from vacation and have a first draft for you. Let me know if it matches your expectations, and feel free to adjust anything to your liking:

---

Testing Blazor Components

Having good tests is a big part of building stable and maintainable software, and Blazor components is no exception.

To test a Blazor component (or page), it must first be rendered with relevant input for the test, and then the component under test and its rendered markup can be inspected. It is also common to need to interact with the component under test, for example by triggering event handlers like those attached to a button’s onclick event or to inspect the components output during different parts of its life cycle.

Testing approaches

The two most common approaches to testing Blazor components are end-2-end testing and component testing:

• With end-2-end testing, a test runner runs a Blazor application containing the component under test and automates a browser instance that enables the testing tool to inspect and interact with the component under test through the browser. [Selenium](link to selenium) is an example of a framework that enables end-2-end testing.
• With component testing, tests are written like regular unit tests, using a library that provides a way to render a component, inspect its output and state, and trigger event handlers and life cycle methods. bUnit is an example of a library that enables component testing.

The following table shows the difference between the two test approaches:

	End-2-end testing	Component testing
Test scope	Blazor component including related JavaScript and CSS	Blazor component only (Razor and C# code)
Test runtime	Seconds	Milliseconds
Access to component instance	No	Yes
Sensitive to environment	Yes	No

Test scope refers to what is involved in the test. In component testing, only the Blazor component itself is involved, i.e. the C# and Razor parts, and external dependencies such as services and JavaScript-interop must be mocked.

In end-2-end testing, the Blazor component and all auxiliary parts of it is part of the test. This includes any CSS, which can affect the appearance of the component’s markup, JavaScript written for the component, and indeed the entire DOM- and browser-APIs, which might be used by the component.

The test scope typically has an influence on the speed with which the test runs. Component tests run much like regular unit tests, which means they run in milliseconds, whereas end-2-end tests can take seconds to complete.

Component testing also provides access to the instance of the component under test, allowing for inspection and verification of the component’s state. This is normally not something that is available in end-2-end testing.

The last point is the test type’s sensitivity to its environment. With end-2-end tests extra care must be taken to make sure the expected state has been reached before verification starts, otherwise the result is flaky tests. With component testing the rendering of the component under test and the life cycle of the test is much more integrated, which means more stable and robust tests out of the box.

Picking the right testing approach

There is unfortunately no silver bullet, so picking a testing approach is all about choosing one that works for the scenario at hand. Let us consider a few scenarios:

Scenario	Suggested approach	Remark
Component without JavaScript interop logic	Component testing	When there is no dependency on JavaScript interop in a Blazor component, it means it can be tested without any access to JavaScript or the DOM API. In this case, there are no disadvantages to choosing component testing.
Component with simple JavaScript interop logic	Component testing	It is common for components to query the DOM or trigger animations through JavaScript interop, but component testing is in most cases still preferred, since it is simple to mock the JavaScript interaction available through the IJSRuntime interface in Blazor.
Component that depends on complex JavaScript code	Component testing and separate JavaScript testing	If a component uses JavaScript interop to call a large or complex JavaScript library, but the interaction between Blazor component and JavaScript library is simple, then the best approach is likely to treat the component and JavaScript library or code as two separate parts, and test each individually, i.e. the Blazor component with a component testing library and the JavaScript with a regular JavaScript unit testing library.
Component with logic that depends on JavaScript manipulation of the browser DOM	End-2-End testing	When a component’s functionality is very dependent on JavaScript and its manipulation of the DOM, it will be easiest to verify both the JavaScript and Blazor code together in an end-2-end test.   This is the approach the Blazor team has taken with Blazor’s browser rendering logic, which has tightly coupled C# and JavaScript code, which must work together to correctly render Blazor components in a browser.

Component testing with bUnit

There is no official testing framework from Microsoft for Blazor, instead we recommend the community driven project bUnit, which provides a convenient way to test Blazor components using the component testing approach.

bUnit works together with general purpose testing frameworks like xUnit, NUnit, or MSTest, which make bUnit tests looks a feel like regular unit tests, and they are also executed in the same way, using e.g. Visual Studio’s test runner or dotnet test.

Let us look at a quick example of a test of the <Counter /> component, which is part of the default Blazor project.

As a reminder, the <Counter /> component looks like this:

<h1>Counter</h1>

<p>
    Current count: @currentCount
</p>

<button class="btn btn-primary" @onclick="IncrementCount">Click me</button>

@code {
  int currentCount = 0;

  void IncrementCount()
  {
    currentCount++;
  }
}

To verify that the counter is incremented correctly when the button is clicked, do the following:

[Fact]
public void CounterShouldIncrementWhenClicked()
{
  // Arrange: render the <Counter/> component using bUnit’s TestContext
  using var ctx = new TestContext();
  var cut = ctx.RenderComponent<Counter>();

  // Act: find and click the <button> element to increment
  // the counter in the <p> element
  cut.Find("button").Click();

  // Assert: first find the <p> element, then verify its content
  //         matches the expected value.
  cut.Find("p").MarkupMatches("<p>Current count: 1</p>");
}

Head over to bUnit’s docs to learn more.

[guardrex]

Thanks @egil ... I should be free on Monday morning. I'll move this issue to the top of the list.

[guardrex]

@egil ... I'm in a good spot now to get the content into a PR for review.

Since we last discussed where the topic would be placed, we've made some changes: Blazor docs are all in the blazor folder now and (mostly) only appear in the Blazor ToC node. It's better for readers and faster for me to maintain the docs. Therefore, I'll place this new topic in the blazor folder and only have a ToC entry under "Blazor" in the sidebar ToC.

I don't anticipate this taking more than a day. I'll set it up today on a local branch, sleep on it 🛌💤 , and make an edit pass on Tuesday morning. I'll try to have the PR submitted on Tuesday. I'll ping u on it for review and further updates. After we have it ready between us, I'll request a review from Dan. He may call upon others or just sign-off on it.

[egil]

Sounds good @guardrex. Look forward to getting your input.

[guardrex]

@danroth27 @mkArtakMSFT @egil A couple of questions have arisen WRT the Blazor testing content here.

Terminology

@egil, what you're calling "component testing" is AFAICT "unit testing of components." The term "component testing" seems overloaded to me ... it could be either unit or E2E taking place, so that term doesn't seem to distinguish anything in particular. It seems to me 🤔 that we'd stick with normal usage of "unit testing" in this context ... of components, of course.

ToC for the future of testing coverage

Dan, Artak ... If you envision that there will be integration/E2E testing content added later in addition to the unit testing coverage, then there are two ways we can go.

Option1

Everything will always be in one topic. We would just add the integration testing/E2E section(s) at the end of this topic. This choice makes the most sense if, as we're doing with bUnit, we're just going to link off for almost everything and only show a bare bones example of an integration/E2E test (Selenium, perhaps).

Option 2

More 🐄🔔 in the future for testing, either for unit testing and/or integration/E2E testing. If you think we'd carry more content than quick examples with external links, we'd need ...

• A node called "Test" in the Blazor ToC.
• An overview topic that addresses the testing approaches (unit testing and E2E testing). The overview would have the first half of what @egil is providing here.
• A unit testing topic under the overview with the 2nd half of what @egil has provided that just focuses on that aspect, bUnit.
• At some point in the future, we'd add an integration testing/E2E topic with a quick example ... Selenium perhaps ... and link off to that framework's docs for the details.

❓ How do u want to play that? 🎲🎲

Update

I have the draft now on a local branch that I'll update based on your responses ☝️. I'll edit in the morning and ping u, @egil, for further work on it. So far, so good! 👍

[egil]

@guardrex wrt. component testing vs unit testing, it was a term I picked up from Dan and thought it made sense. I am completely fine with using unit testing, since component testing and unit testing means the same in this context, at least in my view/wrt. bUnit.

[egil]

With regards to e2e testing: I can do an update to the docs on that when I have this issue closed: bUnit-dev/bUnit#144