Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ASSETS] MigrateFileTask runs out of memory #8664

Closed
5 tasks done
torleif opened this issue Dec 10, 2018 · 17 comments
Closed
5 tasks done

[ASSETS] MigrateFileTask runs out of memory #8664

torleif opened this issue Dec 10, 2018 · 17 comments

Comments

@torleif
Copy link
Contributor

torleif commented Dec 10, 2018

Affected Version

Framework / assets 4.3.0-RC1

Description

I have a site with gigabytes of files, and running vendor/silverstripe/framework/sake dev/tasks/MigrateFileTask causes out of memory errors. Running it returns errors such as Fatal Error (E_ERROR): Allowed memory size of 4487905280 bytes exhausted (tried to allocate 28800001 bytes)

I've upped the memory limit to 4 gigs to try and get through the task. It seems it is progressing each time it's run, as the byte allocation is different. Ideally there wouldn't be a memory leak so the entire task can run without running out of memory.

Acceptance Criteria

  • Task can be performed on 10,000 files, and at least one folder with >1000 files
  • Task can be performed on files containing a variety of types, e.g. JPEGs with 2MB file size
  • Regenerates thumbnails where feasible, and gracefully skips if not feasible (doesn't halt migration)
  • Can determine already migrated files and skip them on subsequent runs
  • Clearly communicates the status of the task on subsequent runs
  • Works with <=512MB of PHP memory (and states that expectation in the docs)
  • Works on supported image drivers (GD and ImageMagick)

Notes

Related PRs

@chillu
Copy link
Member

chillu commented Dec 11, 2018

We actually have an issue to determine rough maximums for this task: https://github.com/silverstripeltd/open-sourcerers/issues/45

Out of interest, do you have very large images (>4000px dimensions) in your site? How many files do you have in total, and how many are processed before it runs out of memory? Just trying to narrow down the use case. 4GB should definitely be enough to run this, I think 512MB is a reasonable max that should allow most environments to run this task.

@torleif
Copy link
Contributor Author

torleif commented Dec 11, 2018

There's 23.0GB of files, and 197,667 files in total. I'm imagining there's a few large images over 4000px in there.

It seems to fault on the second part of the task, generating thumbnails. it gets to about 68k in the list of files.

@wernerkrauss
Copy link
Contributor

@torleif could this be related to #8570 ? Dunno if that patch is available in 4.3RC-1 or was merged later.

What if you increase your memory limit manually in FileMigrationHelper::run()?

ini_set('memory_limit', -1); 

helped for me in SS4.2.

@torleif
Copy link
Contributor Author

torleif commented Dec 11, 2018

I'm overwrote the memory limit in the Envornment file, to see if it would work with that. It looks like it was setting the limit to the default PHP amount.

after running that I'm getting a new error:

VirtualAlloc() failed: [0x000005af] The paging file is too small for this operation to complete.


VirtualAlloc() failed: [0x000005af] The paging file is too small for this operation to complete.


Fatal error: Out of memory (allocated 16114515968) (tried to allocate 24576 bytes) in C:\Users\torleifw\op\vendor\intervention\image\src\Intervention\Image\Gd\Decoder.php on line 136

VirtualAlloc() failed: [0x000005af] The paging file is too small for this operation to complete.


Fatal error: Out of memory (allocated 16114515968) (tried to allocate 32768 bytes) in C:\Users\torleifw\op\vendor\monolog\monolog\src\Monolog\Formatter\NormalizerFormatter.php on line 1

which looks like "you're running out of memory"

@lerni
Copy link
Contributor

lerni commented Dec 11, 2018

Looks like GD? Had the same issue. I've worked around using "imagick" as ImageManager driver. If that works for you as well, it points to something smelly with GD.

@torleif
Copy link
Contributor Author

torleif commented Dec 12, 2018

@lerni I tried swapping to GD, it seems to have very similar memory issues

@lerni
Copy link
Contributor

lerni commented Dec 12, 2018

@torleif Doesn't the error above already look like GD? It worked for me with "imagick" but not with "GD"!

@torleif
Copy link
Contributor Author

torleif commented Dec 13, 2018

apologies, I meant to write I swapped to imagick from GD, which resulted in similar errors.

@maxime-rainville
Copy link
Contributor

maxime-rainville commented Jan 10, 2019

Can any one that got errors tell me what version of PHP they were using? I'm having problem replicating this locally with these parameters:

  • PHP7.1
  • 700 images with a total of 2100MB.
  • My image sizes vary between 1MB and 6MB.
  • I capped my memory usage at 100MB
  • I'm using the CLI
  • With GD set in my ImageManager

@maxime-rainville
Copy link
Contributor

@ingo Just looking back through the comments and @torleif set up is way beyond our acceptance criteria of 10K files with 2MB JPEG.

There's 23.0GB of files, and 197,667 files in total. I'm imagining there's a few large images over 4000px in there.

Do we want to revise our acceptance criteria up?

@chillu
Copy link
Member

chillu commented Jan 10, 2019

Those ACs are taken from our team's DoD. This is mainly about establishing limits on different configurations, and then extrapolating from that. I don't want you to spend a long time generating that amount of data in order to establish a higher maximum. As discussed, please start by talking to SilverStripe Operations about their past experiences.

@maxime-rainville
Copy link
Contributor

Had another go at it. The actual memory usage was going over the limit I had set on the CLI. So something must be calling ini_set('memory_limit') to increase the minimum amount of memory available.

@maxime-rainville
Copy link
Contributor

I found that when you try to get the actual memory usage with memory_get_peak_usage you always get something very small. While if you look at the OS resource monitor you get a much higher number.
In the example below, PHP report's a peak memory usage of 18MB while the UBuntu resource monitor reports 1.4GB for my PHP process.

https://youtu.be/fsJg1RReYmU

silverstripe/silverstripe-asset-admin#904 addresses the memory issues by reverting to using ImageBackendFactory to create the ImageBackend object. This appears to improve garbage collection. OS reported memory usage never goes above 250MB on my test case.

https://youtu.be/vCMvbySoXj0

@kinglozzer
Copy link
Member

@maxime-rainville Yeah I’d definitely recommend a profiler rather than memory_get_usage() / memory_get_peak_usage(). I’ve found https://blackfire.io to be good if you don’t want the pain of setting up something like tideways

@maxime-rainville
Copy link
Contributor

I tried blackfire and it came back with the same memory usage as memory_get_peak_usage
image

@kinglozzer
Copy link
Member

kinglozzer commented Jan 23, 2019

I’m not sure how the Ubuntu resource monitor works, but I’d guess that the 1.4gb figure is total memory used (allocated, de-allocated, re-allocated, de-allocated etc) rather than the max used at any single point in time, so I don’t think that would cause out-of-memory errors.

@maxime-rainville
Copy link
Contributor

@bergice Here's some pointers on reviewing things. I split this one up into many PRs. They don't have any cross dependencies and can all be tested individually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants