-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cypress slows down drastically over long runs - restart renderer between spec files? #431
Comments
Any response to this? |
We've had memory leaks before, but we wrote regression tests for this and it seems quite unlucky there are memory leaks in Cypress itself because nobody else is reporting this behavior. Back when we did have memory leaks a lot of people were complain simultaneously. Not saying it's impossible, but its unlikely. We have our own very long internal runs at Cypress, projects that run for 20 minutes are not unusual. We currently don't see any increase in memory on those projects. The So it's unlikely to be the Cypress app running in the browser, and its not the node process. That basically means that either your app itself is leaking memory - and this is somehow not being GC'd when running in the Cypress environment, or there is a leak in Electron which is also a possibility. Unfortunately debugging this is incredibly difficult. Unless we see a consistent pattern across multiple apps, there isn't a lot for us to do at this moment. Can you try to reduce the amount of variables in your test? For instance, if you could loop through the same test and run it over and over again and see that a single test causes a memory leak - that would be extremely helpful. Once you do that, start removing all cypress commands until you can no longer repeat the problem. It's possible just having a single From there, if we could then point it to a public facing server which replicates the problem. That's the way we'll be able to figure out the root cause and fix it. Adding something like More information would help such as:
If you were to independently run each spec file would that work? How far in the test run does it take before you're seeing an increase in memory? Can you export a graph or chart to show the usage in memory increasing over a period of time? If this is a lot of work don't worry about it, but it would be helpful. I'll note that our internal team probably needs to:
|
Thank you again for the timely response!!! Our whole team has found that after the suite has been running for a while, maybe 10-20 reruns of the same test (any test) it just really starts to slow down, much to the point where the default timeouts end up throwing because it is just lagging behind so much. None of us are able to run the entire suite we have written from start to finish, but I understand that this is of no help to you. If I get some time I will try and gather the information you have requested! So folder structure for the cypress application is relatively flat, not going more than 2 folders deep for module / submodule testing. We have a lot of tests already, it's only been one week and we're already at 10 minutes of tests. After about 300 seconds (5 minutes) the app comes to a crawl, where things as simple as That was around the 600 second mark. We use context for each suite, so we don't rely on any form of cacheing or data retaining between tests. Running tests individually is our only option at the moment. Albeit not a terrible one! I'm not actually seeing a Cypress Helper that is getting lots of memory though, what I seem to be experiencing is what seems like a DOM size issue, where the DOM is so big that it just lags around and feels sludgey. I haven't confirmed, but I don't think we're experiencing this issue via the CLI. Do your large suites run with the GUI? And do they output a large amount of HTML on the left section that scrolls? Should we be Thank you for your time! |
No. We definitely do not run anything from the GUI. The GUI isn't meant to run entire test suites. It's meant to be used in your day to day development typically on a single test file or maybe just a few. The GUI is not optimized for long runs, only headless is. For instance This configuration option is automatically set to So..... all of what I said above was about headless runs. If you're trying to use the GUI for this, it definitely won't work and likely won't ever work for all of these reasons. I thought you were having problems running headlessly (?) |
We're not from the same company. I've been running the tests headlessly. Today I tried a simple bash script that just did As a sidenote, I have noticed the GUI slowing down significantly when I repeat the same test a number of time (e.g. while writing it) - this may be relevant to Andrew's problem. |
Yes it's possible. If there were slowdowns during the same test re-run over a period of time that would be indicative of a memory leak. It would actually be very surprising to me as we have memory test regressions and we've done a lot of analysis to make sure there isn't any - but its possible. Because every app is different and everyone uses different Cypress commands, there could be a weird edge case / combination that we can't otherwise reproduce. If it's happening in the GUI you can actually do the profiling yourself / export it to show it happening. I'm not going to lie, it's a lot of work and can take hours investigating just the area you think it may be happening before even understanding why, and then implementing on a fix. When we open source Cypress this can be made much easier. Regardless even if there is or isn't a memory leak, some of my above suggestions will likely make their way in which will help with avoiding these problems as well. |
I am having an issue with a test getting slower as it runs as well. Current behavior:I am running a single test in the GUI that runs over 1500 commands. The commands start off executing very quickly. As the test keeps running, the commands execute slower and slower. Expected behavior:I expect the tests to run the same speed at the end of the test as the beginning. How to reproduce the current behavior:The test I am running visits every page within the sidebar of our documentation. It then checks that every link in it's table of contents matches a corresponding header in that page.
Test code: context "Table of Contents", ->
before ->
cy.visit(API_PATH + ".html")
beforeEach ->
cy.get('.sidebar-link').as('sidebarLink')
it "displays toc headers", ->
cy.get("@sidebarLink").each (linkElement) ->
cy.visit(linkElement[0].href)
cy.get('.article h1').not('.article-title').each ($h1, i) =>
cy.get('.toc-level-1>.toc-link').eq(i).then ($link) =>
expect($link.text()).to.eq($h1.text())
expect($link.attr('href')).to.eq('#' + $h1.attr('id'))
cy.get('.article h2').not('.article-title').each ($h2, i) =>
cy.get('.toc-level-2>.toc-link').eq(i).then ($link) =>
expect($link.text()).to.eq($h2.text())
expect($link.attr('href')).to.eq('#' + $h2.attr('id')) |
I am also experiencing this. When I run By cypress I mean process with name When I do exactly same steps without cypress - it consume only cca 330MB and with cypress using electron it goes up to 2g and the it crashes... |
@ddeath the GUI mode is not really 'built' to run all the tests - even though we give you that option. You should be running all your tests in headless mode with |
@brian-mann we do not run all tests in GUI just selected portion of them, just one file. Yes in headless mode it takes only 900MB which is still about 600 more then in chrome. |
I'm experiencing the same problem that others have mentioned. After a few runs in GUI mode Chrome is consuming a huge amount of memory and the Cypress steps slow down causing non-deterministic test runs. I did a little profiling and tracked some of the problem down -- at least for my setup -- to sourcemaps. According to the Chrome task manager, here is the tab's memory utilization after 5 runs of my single test:
It's worth noting that when the app initially loads in development, it uses ~450 MB and stays pretty close to the number throughout the day. Thereore, there still exists a problem which can't be blamed on sourcemaps. |
We are looking into this and have a pretty good idea why it's happening. It's a priority right behind the |
Actually @derekharmel can you be more specific when you say "After a few runs in GUI mode". What does a few runs mean? A few tests in a SINGLE run? Manually reloading / refreshing the browser? Clicking the reload button? Modifying a spec file and Cypress automatically re-running? The GUI mode is specifically not built to run all or even that many tests. Because of the way snapshots work you may actually be seeing the intended behavior of Cypress. Snapshotting is basically a controlled memory leak because we specifically hold references to previous pages / XHR's, and DOM objects in memory, which enables you to travel back in time to them. For running a collection of tests thats why headless mode exists - it internally makes lots of optimizations for long runs and doesn't hold references to objects (which removes the ability to debug it). Whereas the GUI mode is specifically optimized for running a single test and inspecting objects and debugging your app. We've internally discussed doing a better job explaining this such as potentially throwing up a warning in the GUI mode that detects when you're trying to run too many tests and warns you. |
One test, no changes to the test or app code, rerun 5 times by clicking the reload button inserted into the page by Cypress (not the browser's refresh button). Another observation was that on each run the increase in memory utilization rate decreased with each run. In other words, the first run was always the largest increase. By the 5th or 6th run, there was little increase. Also, I did the above process 3 times and took a heap snapshot after each run. I'm not very skilled at reading those, and this is probably stating the obvious, but it looks like the GC is not doing much on successive runs. In fact, you could see that the amount of allocations for "code" grows on each run as well. That surprised me a bit. And, I think that explains why the "eval" type sourcemaps resulted in more memory utilization, because of the large strings of code retained between each run and all of the code produced by eval'ing. |
I have the same issue, there is a huge memory leak, it seems cypress doesn't clean everything before a reload. $ npx cypress --version
Cypress package version: 1.0.2
Cypress binary version: 1.0.2 I have 11 tests. 1 First load of the spec, click on stop (before it begins)2 Post first run3 reload via CTRL S inside my editor4 reload via the button reloadThe first load all tests are green, it's fast. Without the process manager it slow after a few reloads. Even cy.type is broken because it. |
@dhoko Thank you for a showing a step-by-step process. However, this is not necessarily indicative of any kind of memory leak. The system task manager will not provide you a reliable means of calculating memory because of the way garbage collection works in the browser. Browsers will prime and hold onto object references way beyond whether or not they should be garbage collected. There is no way to force GC (without running Chrome in a specific mode). It's up to the browser to choose when and how it runs GC. To figure out a real memory usage you either need to use the Timeline Profiler which forces GC every 100ms or so. This mode will make Cypress tests run unimaginably slow which makes it somewhat difficult to use. The other option is to monitor the Chrome Task Manager. It looks like this... You have to look at Javascript Memory - and more specifically the numbers in parenthesis. What this is distinguishing is primed memory as well as memory that's scheduled to be GC. You will see growth for each test that runs but then when you "rerun" the entire suite, it will drop back down temporarily before going back up again. The real issue here is what I've described many times before. There is an entire proposal that goes into a lot of detail: #681 (comment) It is simply this... You cannot use the GUI to run all of your tests.That's the problem and the solution. You cannot run all of your tests. The GUI is a mode built specifically to run one or a small number of tests at a time. Cypress is capturing snapshots of the DOM for every single command which uses up a massive amount of memory. Why does it do this? So you can debug your tests and application as you iterate on a single or small number of tests at a time. The GUI is not built to handle entire runs. Currently you can tell Cypress to cleanup snapshots by setting the If you want to run all your tests you need to do it from the command line using |
OS:
I'm writing tests, I don't run every tests. Only one spec file for DEV. I added my comment only about dev mode. I think this is pretty obvious that the GUI is only for the debug mode. Another ex, with only 6 tests ~ 18s. 1 Load2: 1st run3: 2sd run (button reload)4: 3rd run (button reload)5: 4th run6: 5th run7: 6th runVoilà.
Does cypress purge them on reload ? The GUI is awesome for debug, you did a great job 👍 but something is wrong. |
If you were to say hard refresh instead of reloading using the GUI's reload
button then the memory usage *has* to go down. There's no way objects can
persist beyond a refresh. Can you instead do 5 reloads as you just
performed but with hard refreshes (CMD+R) and post the results?
…On Fri, Oct 27, 2017 at 5:26 AM, dhoko ***@***.***> wrote:
You cannot use the GUI to run all of your tests.
I'm writing tests, I don't run every tests. Only one spec file for DEV. I
added my comment only about dev mode. I think this is pretty obvious that
the GUI is only for the debug mode.
Another ex, with only 6 tests ~ 18s.
From 11:10 35s to 11:13 58s => ~ 210s
1 Load
[image: screenshot from 2017-10-27 11-10-35]
<https://user-images.githubusercontent.com/713283/32097038-8e42b158-bb08-11e7-871f-720d8b63923b.png>
2: 1st run
[image: screenshot from 2017-10-27 11-11-26]
<https://user-images.githubusercontent.com/713283/32097057-9a409416-bb08-11e7-8949-40e4e203e70b.png>
3: 2sd run (button reload)
[image: screenshot from 2017-10-27 11-11-58]
<https://user-images.githubusercontent.com/713283/32097087-a9aa3df8-bb08-11e7-8de0-0caadedf3329.png>
4: 3rd run (button reload)
[image: screenshot from 2017-10-27 11-12-30]
<https://user-images.githubusercontent.com/713283/32097101-b305cd0e-bb08-11e7-8dc4-9f52fa766ff6.png>
5: 4th run
[image: screenshot from 2017-10-27 11-12-55]
<https://user-images.githubusercontent.com/713283/32097116-bba61f7c-bb08-11e7-9973-4440636e22df.png>
6: 5th run
[image: screenshot from 2017-10-27 11-13-25]
<https://user-images.githubusercontent.com/713283/32097123-c0c89d40-bb08-11e7-9d5d-42638b65eeca.png>
7: 6th run
[image: screenshot from 2017-10-27 11-13-58]
<https://user-images.githubusercontent.com/713283/32097142-cecfaad2-bb08-11e7-9b1f-010d98645aad.png>
Voilà.
Currently you can tell Cypress to cleanup snapshots by setting the {
numTestsKeptInMemory: 0 } in cypress.json. This will purge snapshots
aggressively but then you will lose the ability to debug any of your tests.
Does cypress purge them on reload ?
The GUI is awesome for debug, you did a great job 👍 but something is
wrong.
I need to close chrome often and re-open my spec file :/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#431 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABNc8Buo33e5GQhbqgHuykAonQll6vMuks5swaHPgaJpZM4MASHO>
.
|
@dhoko do you mind pasting me a small amount of your test code and providing the site / app you're using to visit (if it's not private). Just provide whatever is the smallest amount of code that it takes to reproduce the memory leak. It might be as small as a single The reason I ask to see your test code - is that its possible to create memory leaks using your test code if you're doing some funky things. I assume you're not, but just want to double check. Also as I've spent many many days in debug land tracking down memory leaks - I've found that often it takes a perfect storm of conditions that don't always manifest themselves if you're looking at a single thing. In other words, if you can point me towards the app you're using / testing which yields the memory leak, that would be helpful. |
Voilà -> https://gist.github.com/dhoko/2f6a51e78e39a09f51cc5d2738654fd3 I'm testing one feature from |
I too am getting this issue, using v1.4.1. Running all of my 30+ tests in the GUI causes a total absorption of my 16gb of memory and ultimately some random test will fail with a timeout error rather than the GUI crashing or failing. I accept that I shouldn't be running all the tests all at once and so I'm typically developing each new test using |
As a workaround, I created a folder in Cypress to house all of my test suites. I simply move the folder(s) in and out of the integration folder to run individual suites in the GUI instead of all of my test cases. It's not ideal, but it's better than running all of my tests and dealing with the slowness/crashing of the browser. |
It would be great to control not only numTestsKeptInMemory but maxDOMSnapshotsPerTest (which defauts to all of them) and frees up DOM snapshots, oldest first, as it exceeds this value. This way for big tests that we are debugging in a browser we can adjust this value to only have the DOM snapshots close to where it fails, eg. last 5 or 10. When taking a heap snapshot when Canary grinds to a halt on a complex test a majority of the memory seems to be taken up by the storage of the DOM snapshots. |
I'm trying to figure out why my GUI is crashing all the time how do you guys monitor the issues. Just in the task manager? My memory usage it arround the 50% and crashes between 2.500 and 3.000mb... my system has also 16GB of ram. This is with the default configuration. If i lower the snapshots for example from 50 to only 10. Then the GUI doesn't crash with the same amount of test and the momory takes around the 2.000mb of ram usage. Hope in the near future we can run and do more in the GUI mode without those crashes/whitescreens. The headless mode i don't have the problems and it's indeed working without any troubles. |
Fixed by PR #1583 |
Released in |
I am still having this issue in My tests don't actually slow down until my computer runs out of memory. The first time I noticed this issue, Cypress was taking over 10 GB of memory. That may have been an anomaly, because I haven't been able to reproduce anything like that since. I'm not convinced this is a separate issue. According to the release notes for Here are the memory results I'm getting. Each screenshot is taken directly after running my test file.
|
@SecondFlight appreciate you putting this together. However you don't mention whether or not you're running this in We've noted in other issues regarding the use of When in I would recommend reading this issue (now closed) but we opened a new one to talk about the changes for interactive mode |
@brian-mann, thank you for the quick response. I am using I want to clarify that the current behavior is not unexpected to me per se. I understand the current tradeoffs of the debug tools. I compiled that together because I had not seen anyone else reporting issues with the memory usage during normal development. The test file I'm running is a single test, and I will be writing many more like it. Its current memory consumption is manageable as I have plenty of ram, but I can't add much more to my test without reducing my data set, which I certainly can do. I realize that the DOM complexity of the application I'm working with is something of a special case. Really, what I'm experiencing is an extreme case of the currently expected behavior. I just wanted to share my use case as I haven't seen it anywhere else. |
I am facing the exact same issue with 3.0.1. This is when I do When I used
|
The initial performance issues for these tests has been resolved in Cypress - cypress-io/cypress#431
The initial performance issues for these tests has been resolved in Cypress - cypress-io/cypress#431
Hi @brian-mann , I'm just scratching the surface of Cypress but i would appreciate if any of you could help me with the following situation: describe('Hitting Complete email button', function() {
it("Completes your email", function() {
cy.contains('Complete')
.click()
cy.get('.modal-wrapper').contains('!DOCTYPE html PUBLIC');
});
}); But after clicking that button i get CypressError:
But my test never gets to assert that the text I'm looking for does exist, cypress never gets to that point in my app (and it's working ok) Any help would be very much appreciated, apologies in advance this is my very first post and I'm not a natural speaker. Thanks a lot |
@L4mPi2 Issues in our GitHub repo are reserved for potential bugs or feature requests. We recommend questions relating to how to use Cypress be asked in our community chat. Also try searching our existing GitHub issues, reading through our documentation, or searching Stack Overflow for relevant answers. |
Running on a Intel i7-10700 @ 2.90, 8 cores, 16 procs, 16 GB RAM. Started a 1000 json object web test, like 40 mins back. It takes about 10 seconds to finish 1 test case. It slowed down so drastically after initial 10-20 inserts. About 120 objects inserted since start of the test. Starting test (1 object) takes about 10 seconds, by that calculation i.e. 10 seconds i.e. 6/minute x 40 minutes, should have been 240? Running headless on electron... Regards, SG |
4 Years later and i still experience this problem... |
@srotak5 have you tried using the experimentMemoryManagement flag? There are also other things you can do to improve performance such as disabling video compression. If you have a reproducible example you can share with us please open a new issue and we will be happy to investigate. Also, check out our community chat, it can be helpful for debugging or answering questions on how to use Cypress. |
Description
When I try to run all tests at once with
cypress run
, the tests slow down part-way through. I think this is related to the memory usage of the renderer, since a process called 'Cypress helper' has a continuously increasing memory usage, and the slowdown hits when it reaches 3 to 3.5GB of RAM (I only have 4GB on my system). Eventually, a test that usually takes 10-15 seconds took over 3 minutes to run.I'm running the tests locally on a MacBook, using Cypress CLI 0.13.1 and App 0.19.0.
Additional Info
As suggested in issue #349, I think an option like
restartBrowserBetweenSpecFiles: true
in cypress.json would most likely fix the issue for me.Not many options seem to help the memory usage much, turning off video recording and setting number of tests kept in memory to 0 probably made it last a little bit longer, but it's not the Cypress process itself that is using all the memory.
CYPRESS_MEMORY=1 /Applications/Cypress.app/Contents/MacOS/Cypress --project <path_to_your_project> --show-headless-gui
Brian suggested running the above command to view the memory usage for the node process, but the value reported never went over 100MB so I doubt there's any problem here. For reference, the output in the terminal looked like this:
info { rss: 132407296, heapTotal: 102604800, heapUsed: 89873952 }
The text was updated successfully, but these errors were encountered: