Add the capability to disable page reloading between tests

[AlexanderMoskovkin]

Based on the #1509 question. It's actual for the SPA apps.

[inikulin]

It would be nice to see exact scenarios for this first. For now it sounds like we are going to introduce yet another footgun, there each test results depend on previous one.

[thecodejack]

I partially agree. But I am sure there are many scenarios where I don't want to refresh whole page between tests. This will optimize test time as well.

Just to avoid refresh, I put all the scenarios in one test and use console.log to makesure it passed all scenarios.

[inikulin]

This will optimize test time as well

IMHO it's a slippery slope. It's not a good idea to sacrifice consistency for performance.

[oneillci]

I think this would be hugely beneficial for single page applications, but I also think it should be an explicit opt-in so that a user is aware that they are not starting with a blank slate for each test and that people testing regular websites are not affected.

One of the things that is more difficult in a SPA is managing the client side state. Every time you refresh the page, the app bootstraps so you're in a pristine state, but some of the more difficult to track bugs in a SPA arise after editing an item, navigating and expecting an unrelated component to be updated.

A typical scenario for me would be:

1. Navigate to an edit page, test the inputs - cascading dropdowns, field enabled/disabled or hidden based on user selection etc.
2. Click the in-app save button - this saves the item
3. Navigate back to a grid page - the updated (or newly added) item should appear in the grid.
4. Depending on the item saved it could update other parts of the UI - saving a user for example could update a status bar containing the current user name.

A test like this could be achieved by putting all logic within a test(), but it would be nice to separate these different test responsibilities into different tests(), but at the moment if I put them in different tests(), the page reloads and I'm not testing that the client side behaviour is correct.

Currently a test like this would be written

fixture("test save item saves a updates grid');

test("check inputs, save, navigate, check grid" async t => {
  //test cascading dropdowns
  //check boxes
  // assert items are enabled/disabled etc
  // do a save
  // assert no validation errors
  // navigate to grid page
  // assert updated item exists in grid
});

IMO, a nicer way to write these would be each test has a single responsibility

fixture("test save item saves a updates grid', { noReloadBetweenTests: true });

test("check inputs" async t => {
  //test cascading dropdowns
  //check boxes
  // assert items are enabled/disabled etc
});
// No page refresh please!!

test("save without validation errors" async t => {
  // do a save
  // assert no validation errors
});
// No page refresh please!!

test("item exists in grid" async t => {
  // navigate to grid page
  // assert updated item exists in grid
});

The fact that it would improve the performance of running the tests in the fixture would be a bonus (for me) rather than the goal.

As mentioned by @iexplore in #1509, enabling this via a 'noReloadBetweenTests' option or similar on the fixture would be a great solution to this.

[classicalConditioning]

+1 for adding noReloadBeetweenTests option.
Same as in a couple of comments above - I would rather utilize test cafe's ability to define more granular tests, but due to my system under test being a SPA - I can't achieve this. I end up with much larger tests that test multiple things at once.

[brettveenstra]

@inikulin - what about supporting shared step(s) per Fixture (e.g. .useRole(), or other async func) and not just at the Test level? Most testing libraries have this concept already where an expensive context/scenario can be initialized for a group of Tests... would be very handy in SPA and other cases where setup is consistent (and immutable)

[AlexanderMoskovkin]

@brettveenstra, If I get you idea right you can try to use Role not for logging actions only. For example you can try to create a Role and perform in the initialization function some actions with your SPA that will change the state of the application. And try to use this role in different tests. In fact User Role saves and restores cookies and local/session storages in the browser. Is it enough to restore the state of the SPA?

[AlexanderMoskovkin]

@oneillci, @classicalConditioning I'm not sure that splitting the test case to several tests helps. For example if you have three tests the second depends on the first one and the third depends on the second:

test('test1', () => { // check create user form and create a user });
// no reload
test('test2', () => { // check user is created in some other view });
// no reload
test('test3', () => { // do something else with the user });

For example if the test1 is failed there is no reason to run test2 and test3 (because user is not created).
If the test3 is failed you have to run all tests to check the issue.

If you put all these actions into one test you have the same. If the test fails TestCafe shows the line where it happens, so you can easily determine what the part of the test is broken.

[classicalConditioning]

@AlexanderMoskovkin
here is an example of what I am talking about.
Given a grid that has a filtering capability. There are three filters: First Name, Last Name, City.

Here are my tests:

1. this is the way I need to structure them if I don't want current testcafe to reset browser's localStorage and cookies:

Test 1:

• login
• filter by First Name
• assert that result is correct
• reset filter
• filter by Last Name
• assert that result is correct
• reset filter

...

2. This is the way I would like to structure my tests to keep them more readable, but I can't since the browser's localStorage and Cookies are reset after each test, therefore I have to login before each test

Test 1:

• login
• filter by First Name
• assert that result is correct
• reset filter

Test 2:

• login
• filter by Last Name
• assert that result is correct
• reset filter

...

3. This is the way I would like to structure my tests if there was an option to not reset browser's localStorage and cookies. (login only in the first test)

Test 1:

• login
• filter by First Name
• assert that result is correct
• reset filter

Test 2:

• filter by Last Name
• assert that result is correct
• reset filter

...

I hope it is clear that in this scenario tests do not depend on each other

[classicalConditioning]

@AlexanderMoskovkin even better is what @brettveenstra is proposing.
If there was be a way to specify with which role an entire fixture is executed - that would be very handy, IMHO.

[AlexanderMoskovkin]

If there was be a way to specify with which role an entire fixture is executed - that would be very handy, IMHO.

It seems you can try this approach with the current API:

const someRole = Role('http://start-page', async t => {
    // do some actions
}, { preserveUrl: true });

fixture `fixture1`
    .beforeEach(async t => {
        await t.useRole(someRole);
    });

//...

[classicalConditioning]

@AlexanderMoskovkin this seems to work, thanks. I saw preserveUrl in documentation, however it was not clear that it would solve my problem. If I get it right, when you use a t.useRole(userRole) all steps defined in the userRole will be executed. So it feels like these same steps will be executed before each test, If you put it in .beforeEach. It would be nice if there was a cleaner way to say that I want to use a particular user role for entire fixture.

[brettveenstra]

@AlexanderMoskovkin thanks for the quick reply ... if that works as you say, is there a reason why the fixture api doesn't support useRole?

example:

fixture `fixture1`
   .useRole(someRole);

per the documentation, the beforeEach suggests that your workaround will fire for EACH test belonging to the fixture (which is inherently what I don't want) so cognitive hurdle there with the API ...

putting useRole at fixture level could then automatically set the preserveUrl parameter and I could re-use the same someRole across other tests

[AlexanderMoskovkin]

I'm sorry, it seems I missed your questions.

So it feels like these same steps will be executed before each test, If you put it in .beforeEach

@classicalConditioning, A steps for the role initialization function will be executed once for the whole test run.

For example, if you have Role(async () => { /* await someActions() */ }) and call useRole in the beforeEach hook it will work in the following way:

1. When the first test is started TestCafe calls useRole from the beforeEach hook. Since the role is not initialized TestCafe executes its initialization function. Then TestCafe saves the page state (cookies, storages..).
2. When each next test is started TestCafe calls useRole again. But since the role was initialized before and we have cookies/storages for it, TestCafe just restore cookies/storages and reloads the page.

is there a reason why the fixture api doesn't support useRole

@brettveenstra Adding useRole to the fixture sounds reasonable, we'll consider it. Meanwhile the approach with beforeEach should work properly as I described above. It adds only one additional page reloading for each test.

[classicalConditioning]

@AlexanderMoskovkin I get that, however it is not clear from naming conventions being used that useRole is executed only once even though it is inside of beforeEach hook.

[iexplore]

@AlexanderMoskovkin The restore of cookies/storages solves a lot.
But can there be option to write an involved test with multiple steps and see messages of those steps like you see between tests? It's like another sublevel within tests.

[AlexanderMoskovkin]

it is not clear from naming conventions being used that useRole is executed only once even though it is inside of beforeEach hook.

To be accurate the useRole command is executed in each test in this case. But it works in different ways for the first time and for the next times.
A goal of the useRole command is to set necessary cookies/storages values. If it has already saved values then it just restores them, if not - it runs the initialization function.

[AlexanderMoskovkin]

@iexplore, It seems we have a proposal for this (Implement test sections) but now we can't get any estimates about it.

[raDiesle]

How speed can be improved, what is not supported at the moment:

1. not to reload page when Role code is performed, before test ist executed with useRole.
   To be more precise: Role(url?, ...) should be optional

[AlexanderMoskovkin]

Hi @raDiesle,

It seems the preserveUrl option should meet your needs.

Take a look at the example:

fixture `My fixture`;  // don't set the url here

const role = Role('https://login-page', async t => {
    // some login actions that redirects the page to the `https://start-page`
});

test('test-1', async t => {
    await t.useRole(role);
    // you are on the `https://start-page` here
});

test('test-2', async t => {
    await t.useRole(role);
    // you are on the `https://start-page` here
});

Note that cookies restoring works properly in this case but it seems it can be problems with local storage values restoring when the preserveUrl option is enabled (#2015). We're going to fix it soon.

[raDiesle]

I am using preserveUrl, so it helps "after you switch to the role"
but it does not help, before role code is executed, because it opens fixture url, afterwards Role url

[edimistra]

Hi @AlexanderMoskovkin is there an intention to implement the 'noReloadBetweenTests' option or another way to control page reloading between tests? I agree with @oneillci point of view towards the organization of tests.

[l-i-x-u]

Sounds like a good idea to provide an option to disable page reload after each test. It would be very handy for testing web forms that only allow you to go to the next section after completing the current section.

Would you consider to implement it @AndreyBelym ?

[AndreyBelym]

Hi @lixualinta, our team is discussing this feature right now. I've made a prototype build without test page reloading. It's really dirty and can be extremely unstable, USE IT AT YOUR OWN RISK.

All tests in all fixtures must have the same test page URL, otherwise it won't work.

You can download it from GitHub: https://github.com/AndreyBelym/testcafe/releases/download/no-reload-demo-1/testcafe-0.20.0-alpha.3.tgz and then install the TGZ archive with npm, or just install it with npm i https://git.io/vp6Pf.

[ghost]

Before:

After using @AndreyBelym no-reload-demo-1

After:

~400% faster

[honsq90]

how's this going? :)

[dudaming]

@AndreyBelym,

As i have read the disablePageReloads() has been release on version 0.21. However, I am using the version 0.22, when i try to call test.disablePageReloads(), it gives error that Property 'disablePageReloads' does not exist on type 'TestFn'. The method is not in the Interface TestFn, it is in src/runner/index.js.

Do you have any idea about this?

Thanks.

[dkaplan-pivotal]

@AndreyBelym You said This mode can also memory leaks if you don't write proper clean-up code, can you give some examples of how this can happen and how to properly clean it up?

[darrylgrant]

Hey @VasilyStrelyaev - I noticed that you removed the "Documentation: required" label. Does this mean that disablePageReloads is now documented somewhere?

I'd love to take a look at those docs if that is the case as disablePageReloads is a feature that would really be useful to me. Is there a link for the docs?

Test Cafe is great by the way :)

[VasilyStrelyaev]

Hi @darrylgrant ,

We've decided to keep this feature 'for internal use' for now. That's why we didn't announce it in our release notes or mention in the documentation.
 
There are several reasons behind this decision:
 

• We assume that it's hard to write tests so that they don't require page reloads, which makes this feature difficult to use.
• We think that writing tests in such a way affects their stability.
• There are some aspects of this feature that aren't finished yet. For example, using this feature when tests are run concurrently.

[darrylgrant]

Thanks @VasilyStrelyaev - I appreciate the info. 👍

[Foxandxss]

I would ask to please not forget about this feature @VasilyStrelyaev.

I implemented this in different projects and the feedback was always the same. Please make them sequential (the login process with IdentityServer is veeeeeeery slow). I can hack my way, but that is not good.

I also give talks about testcafé

And the feedback I always get is the same, sequential tests.

[ilyasbasharov]

Testcafe team,

Do you have any plans to release this feature in the near future?

[subbiah2806]

Hello team,

I tested few scenarios:

fixture `google`
  .disablePageReloads
  .page`www.google.com`

test('type Hello', async t => {
  await t
            .typeText('#tsf > div:nth-child(2) > div.A8SBwf > div.RNNXgb > div > div.a4bIc > input', 'Hello ');
})

test('type World', async t => {
  await t
            .typeText('#tsf > div:nth-child(2) > div.A8SBwf > div.RNNXgb > div > div.a4bIc > input', 'World');
})

Now on above code both test's url are same so page is not reloaded. It types Hello World.

fixture `google`
  .disablePageReloads
  .page`www.google.com`

test('type Hello', async t => {
  await t
            .typeText('#tsf > div:nth-child(2) > div.A8SBwf > div.RNNXgb > div > div.a4bIc > input', 'Hello ');
})

test('click search button', async t => {
  await t
            .click(Selector('#tsf > div:nth-child(2) > div.A8SBwf > div.FPdoLc.VlcLAe > center > input[type="submit"]:nth-child(1)'));
})

test('click map's button' async t => {
  await t 
                .click(Selector('#hdtb-msb-vis > div:nth-child(2) > a'));
})

Now in above code after typing Hello it clicks search button, Then it should click on map's button in next screen but page reload's and again comes to www.google.com where there is no map's button.

Is there any way to preserve url between tests or stop reloading???

[BuruY]

Will we see an implementation in the near future? I still don't get why it is not implemented yet

[peteygao]

I understand the "current" implementation is for internal use only, but I just want to report that even the current undocumented implementation does not work when used together with the Roles feature. Using useRole() will always trigger a refresh, even if the start/end page are the same URL.

[valgussev]

Any plans on implementing the feature in order to use with useRole()? It seems currently in order to use useRole without reloads you need to specify it only for the first test in the fixture.

[valgussev]

Well, my previous suggestion will only work across fixtures using the same .page() url, which is unsuitable.

@VasilyStrelyaev

• We think that writing tests in such a way affects their stability.

I would love this to be an opt-in so developers may decide about the stability and possible outcomes. As was mentioned above, SPAs are a bit different from a regular websites, which really needed reload prevention. This is the only suggestion.

Other than that Testcafe is an awesome tool, please keep rolling!

[akdor1154]

For my usecase, it would make sense if failed tests triggered a reload for the subsequent test. This would significantly simplify writing cleanup code.
My usecase is an SPA where the tests are independent. Each test includes "navigating" to the desired part of the application as part of the test, and resetting the state of the app when done.

[Waterstraal]

I see that this issue is closed, but there is no official implementation yet as far as I can tell. Am I missing something? This is an important feature for SPA's imho.

Could someone give an update on this issue please?

[AndreyBelym]

There are no updates about this feature, this comment completely describes the situation.

[Waterstraal]

Thanks for the update @AndreyBelym. Here is my 2 cents on the matter.

• We assume that it's hard to write tests so that they don't require page reloads, which makes this feature difficult to use.
• We think that writing tests in such a way affects their stability.
• There are some aspects of this feature that aren't finished yet. For example, using this feature when tests are run concurrently.

I just did a quick test with Cypress and they don't reload between tests either, so it should be possible to roll this out if they can do it. Cypress can also run in parallel, although I have not used or tested that feature with the code below.

If anything, this flag should be opt-in like @valerii-cognite mentioned in the comment above.

I used the following Cypress to test my claim:

context('Testcafé homepage', () => {
    before(() => {
      cy.visit('https://devexpress.github.io/testcafe')
    })
  
    it('should have a get started button', () => {
        cy.get('.get-started-button').contains('Get Started');
    })

    it('should have a how it works section', () => {
        cy.get('h1').should('have.text', 'How it works!');
    })
  })

[oneillci]

I like so much about test cafe but I think this will be a deal breaker for a lot of people testing SPA's. I've followed this thread for about a year now and IMO it's much more of a philosophical difference rather than any technical limitation. It's up the owners of this project to decide the direction they want to pursue and I respect that. But closing this issue flies in the face of how developers of SPA's work and also how they behave at runtime. It strikes me as purism over practicality and I say this with a lot of respect for the testcafe development team as, aside from this, it truly is an excellent tool. IMO, testing SPA's will not be a 1st class citizen in testcafe unless this scenario is addressed

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs or feature requests. For TestCafe API, usage and configuration inquiries, we recommend asking them on StackOverflow.