-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Large temp folder due to failing thumbnail generation (in the admin) #3335
Comments
I'm unable to reproduce this locally with a 22MB image. To be fair, my thumbnails gets generated. Can you clear out your log folder, reproduce this and see if there's anything in the logs? The reason I'm asking is that the file should be removed from that folder as soon as it's downloaded and moved to |
It's the memory limit of the server that I'm running into:
The |
Ah, I think I see what's going on here. There's a Craft setting for caching cloud images for generating transforms and thumbnails. (https://docs.craftcms.com/v3/config/config-settings.html#maxcachedcloudimagesize) So, as you request a thumbnail for an image, the image is pulled down to the server from the remote source and cached locally according to the setting. It defaults to a source image of 2000x2000, so, if you have that setting left at default, Craft tries to resize the source image to that size. At this point, since we're talking about an 11MB image, Craft is unable to process that using GD library with only 128MB of RAM. There are multiple ways to approach this:
I would recommend a combination of the last two options. |
Thanks for further investigating this, @andris-sevcenko. I was wondering if you could re-open this issue. Not to try to work around the currently set limitations (which won't be possible), but to have a failsafe in place which prevents the A deletion of files that have the same filename(prefix) before the source image is being pulled from the remove source would be nice addition: That way only one copy of a file can be left in |
There we go. |
I'm running into this issue currently 🤔 I'm running latest CraftCMS 3.7.18.2 (latest as of this post) on PHP 7.4.25. I tried to set my Temp folder to be in Google Cloud Storage but that doesn't seem to be working. Instead, my main server is running out of space due to multiple copies of the same images getting stored in the Temp folder. |
@cballenar can you list a few of the filenames that are multiple copies? |
@andris-sevcenko thanks for follow up! Below is the format these files use where the first portion seems to be the actual filename and is then followed by an automatically generated string. I found that one of these files was created 465 times :|
I've looked in phperrors.log but didn't find anything associated to this. I had a harder time navigating web.log but I think this one is it:
Does this indicate an issue with permissions? I've seen something funky happen in recent updates where I used to keep all files and directories under Thanks! |
@cballenar it's not directly related to the original issue, although it is quite similar. I'll see if we can maybe have garbage collector also prune the temp folder and get rid of any files that are older than, for example, a week or so. |
Thank you! Heads up, these are generating quite rapidly. During my troubleshooting I cleared these files and just a couple days later the server storage was maxed out, which meant ~3Gb were generated in that time. Should I consider bumping up this storage space nonetheless? |
Do you have a frontend upload form or something like that? Perhaps you can send over your entire log folder to [email protected] and reference this issue? |
Nothing like that :\ In reality only one admin adds content. I will do that! Thank you very much! |
Quick follow up in case anyone stumbles on this in the future. In my case this issue was caused due to a misconfiguration in the permissions of the folders. Some were owned by a different user which led to some things working correctly and others not so much. Thanks! |
Description
For a Craft project of mine I have defined the Asset Volumes to be S3 Volumes. This way there are no huge disk space requirements on the server hosting the Craft project, as the files are stored on S3. (For reference: there are already 5GB worth of photos uploaded into the bucket)
Combined with imgix for resizing the original images (which are huge) I've moved away some more load from the server as the CPU-expensive image transforms + caching needs are now imgix's thing to tackle.
To my surprise however I was getting disk usage warnings from my server: the project was using a whopping 7.6GB of disk space.
Looking onto the server I noticed that the culprit is the
storage/runtime/temp/
folder, which takes up 5.7GB:Digging deeper into that big
storage/runtime/temp/
folder, I notice that it's filled with some – but not all – of the images the client has uploaded. Above that those files appear in there a lot of times. Here's a list of just one of the files:Hmmz … debug modus on:
Checking the S3 bucket, I notice that there indeed is a file named
ZWIJNDRECHT-06b.jpg
on there, which also is about 11MB.Checking the contents of those files in the
storage/runtime/temp/
, they indeed are copies of the one stored in the S3 bucket.Moving over the the CraftCMS admin I indeed see no thumbnail for the asset
ZWIJNDRECHT-06b.jpg
appear.A stop in the DB later I know that the ID of the asset is
881
.Swinging back to the disk contents and it's confirmed, there's no thumbnail there for said asset:
In short:
The admin needs to generate a thumbnail for the image. Since it it has no knowledge of imgix being available (on the frontend), it performs the resize action itself.
Therefore it downloads the original asset from S3 into
storage/runtime/temp/
and starts resizing it. That action however fails (file too big, action taking too much time, …) and the temp downloaded file is left instorage/runtime/temp/
.Next time a thumbnail needs to be shown this thing starts all over again.
Rinse. Repeat.
Steps to reproduce
storage/runtime/temp/
after generation has failed, you'll see a temp file left in there.Additional info
The text was updated successfully, but these errors were encountered: