-
Notifications
You must be signed in to change notification settings - Fork 824
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ASSETS] MigrateFileTask runs out of memory #8664
Comments
We actually have an issue to determine rough maximums for this task: https://github.com/silverstripeltd/open-sourcerers/issues/45 Out of interest, do you have very large images (>4000px dimensions) in your site? How many files do you have in total, and how many are processed before it runs out of memory? Just trying to narrow down the use case. 4GB should definitely be enough to run this, I think 512MB is a reasonable max that should allow most environments to run this task. |
There's 23.0GB of files, and 197,667 files in total. I'm imagining there's a few large images over 4000px in there. It seems to fault on the second part of the task, generating thumbnails. it gets to about 68k in the list of files. |
I'm overwrote the memory limit in the Envornment file, to see if it would work with that. It looks like it was setting the limit to the default PHP amount. after running that I'm getting a new error:
which looks like "you're running out of memory" |
Looks like GD? Had the same issue. I've worked around using "imagick" as ImageManager driver. If that works for you as well, it points to something smelly with GD. |
@lerni I tried swapping to GD, it seems to have very similar memory issues |
@torleif Doesn't the error above already look like GD? It worked for me with "imagick" but not with "GD"! |
apologies, I meant to write I swapped to imagick from GD, which resulted in similar errors. |
Can any one that got errors tell me what version of PHP they were using? I'm having problem replicating this locally with these parameters:
|
Those ACs are taken from our team's DoD. This is mainly about establishing limits on different configurations, and then extrapolating from that. I don't want you to spend a long time generating that amount of data in order to establish a higher maximum. As discussed, please start by talking to SilverStripe Operations about their past experiences. |
Had another go at it. The actual memory usage was going over the limit I had set on the CLI. So something must be calling |
I found that when you try to get the actual memory usage with silverstripe/silverstripe-asset-admin#904 addresses the memory issues by reverting to using ImageBackendFactory to create the ImageBackend object. This appears to improve garbage collection. OS reported memory usage never goes above 250MB on my test case. |
@maxime-rainville Yeah I’d definitely recommend a profiler rather than |
I’m not sure how the Ubuntu resource monitor works, but I’d guess that the 1.4gb figure is total memory used (allocated, de-allocated, re-allocated, de-allocated etc) rather than the max used at any single point in time, so I don’t think that would cause out-of-memory errors. |
@bergice Here's some pointers on reviewing things. I split this one up into many PRs. They don't have any cross dependencies and can all be tested individually.
|
Affected Version
Framework / assets 4.3.0-RC1
Description
I have a site with gigabytes of files, and running
vendor/silverstripe/framework/sake dev/tasks/MigrateFileTask
causes out of memory errors. Running it returns errors such asFatal Error (E_ERROR): Allowed memory size of 4487905280 bytes exhausted (tried to allocate 28800001 bytes)
I've upped the memory limit to 4 gigs to try and get through the task. It seems it is progressing each time it's run, as the byte allocation is different. Ideally there wouldn't be a memory leak so the entire task can run without running out of memory.
Acceptance Criteria
Notes
vendor/silverstripe/framework/sake dev/tasks/MigrateFileTask
with a large asset baseRelated PRs
The text was updated successfully, but these errors were encountered: