Imaginary preview generation is not working as expected for folders containing HEIC images. #35643

Gunzinger · 2022-12-07T00:22:03Z

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Steps to reproduce

Install imaginary plugin for preview generation
Create a folder with several (tested with ~110) HEIC images (Images shot on a Samsung A52s phone with the stock camera app)
Try to open/preview that folder in the Web-UI

Expected behavior

Images with accompanying previews should appear.

Actual behavior

The imaginary binary from the imaginary docker container commits all available memory and swap space, finally getting killed a few minutes in after running out of memory.
Preview generations succeed very slowly but it seems that already generated previews do not get used successfully, at least I could never get them to show up.

It is notable that the previews for other files in the folder (.mp4 videos, 4 .jpg images) work and appear.

Host OS

Ubuntu 20.04.5 LTS (VPS with 4vCPUs, 8GB of RAM and 2GB configured swap)

Nextcloud AIO version

Nextcloud AIO v3.1.0, Nextcloud 24.0.7

Current channel

latest

Other valuable info

Relevant link with a bug report of another user: https://help.nextcloud.com/t/imaginary-not-working-on-nextcloud-aio/146725

Logfile of the imaginary container: aio_imaginary_log.txt

Screenshots of htop on the VPS

szaimen · 2022-12-07T08:14:22Z

I fear this is nothing I can fix here. It is rather something that needs to be fixed in server.

loeffelpan · 2023-01-05T08:44:43Z

Facing the same issue.
My Ubuntu Server and all other services (DHCP, DNS, NFS) are unusable during preview generation via Imaginary.

Are we sure, that this has to be fixed in the server, rather that in Imaginary itself?
If this is server related, it is not fixed with 25.x.x.

Jip-Hop · 2023-02-16T20:36:30Z

The errors that appear in the log are:

set_mempolicy: Operation not permitted

Just posting it so people can find this issue when searing for set_mempolicy.

ikamaletdinov · 2023-02-23T18:21:38Z

Issue still exist. Default preview-generator is faster than imaginary on HEIC files.

Any plans to fix this?

deetuned · 2023-02-24T03:11:07Z

I have upgraded to Nextcloud 25.0.4 and HEIC images still can't have their previews generated. Receiving error set_mempolicy: Operation not permitted.

ikamaletdinov · 2023-03-05T16:24:17Z

Hey, I have made small investigation.

set_mempolicy: Operation not permitted could be easily fixed adding additional caps in docker compose file^

   cap_add:
    - CAP_SYS_NICE

Updating libheif1 to the latest one SIGNIFICANTLY increases performance:

FROM nextcloud/aio-imaginary:latest


USER root

RUN echo  "deb http://deb.debian.org/debian sid main\n" > /etc/apt/sources.list.d/sid.list

RUN \
   apt update && \
   apt install -t sid -y libheif1 libde265-0 libx265-199 libvips
USER nobody

szaimen · 2023-03-06T10:49:36Z

Hi, thanks for your investigation! Will adjust this in AIO for the next release.

loeffelpan · 2023-03-06T16:13:42Z

Would love to hear and test when this is available in beta channel.
I have dozens of Heic-images in another .nomedia-Folder. I'll move them in my photos folder for testing.

szaimen · 2023-03-07T11:39:31Z

This is now released with v4.6.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel

loeffelpan · 2023-03-07T21:35:32Z

Just tested with one or two dozens of heic images. Kind of better, but not good yet.

Previews of HEIC are not created and anytime I access that folder with photos oder files app imaginary has high CPU and memory for about 10 or 15 minutes and other services on that server are affected.

szaimen · 2023-03-07T21:53:18Z

I suppose this might happen beccause it is creating too many previews at once which will be improved with Nc26 by limiting the amount of previews that get generated in parallel based on the amount of cpu cores

szaimen · 2023-03-07T22:04:55Z

BTW, I just added some instructions to nextcloud/all-in-one#1846 how you can try out nc26 already if you feel brave.

loeffelpan · 2023-03-07T22:18:33Z

I think I will try out and revert back from backup to wait until you release aio with nc26.
I'll give feedback tomorrow.

szaimen · 2023-03-07T22:35:38Z

Now that I am thinking about it, the limit will likely not work yet because the container is currently missing a php extension that is required for it to work. You'll see a warning for that in the admin overview after the update if that should be the case. However you could use https://github.com/nextcloud/all-in-one#how-to-add-php-extensions-permanently-to-the-nextcloud-container to add the extension temporarily until it gets added officially with the official update to nc26.

Additionally, please make sure that OC\Preview\HEIC or OC\\Preview\\HEIC is removed from your config.php if it should be present as it might conflict with imaginary.

loeffelpan · 2023-03-08T07:20:55Z

Thanks for your instructions.
Upgrade to NC 26.0.0 RC2 was successful including PHP extension sysvsem.
Config.php has imaginary at position 0 and no HEIC entry.

Unfortunately no change with preview generation.
Imaginary runs even longer as before but grabs all the memory and makes my system unusable anyway. HEIC previews are not generated. I also noticed, that imaginary container is restarting in between NC is trying to generate previews.

Off-topic:
Maybe you broke something concerning the backups? Backups failing since 4.6.0 beta.

Performing backup...
Starting the backup...
usage: borg create [-h] [--critical] [--error] [--warning] [--info] [--debug]
                   [--debug-topic TOPIC] [-p] [--iec] [--log-json]
                   [--lock-wait SECONDS] [--bypass-lock] [--show-version]
                   [--show-rc] [--umask M] [--remote-path PATH]
                   [--remote-ratelimit RATE] [--upload-ratelimit RATE]
                   [--remote-buffer UPLOAD_BUFFER]
                   [--upload-buffer UPLOAD_BUFFER] [--consider-part-files]
                   [--debug-profile FILE] [--rsh RSH] [-n] [-s] [--list]
                   [--filter STATUSCHARS] [--json] [--no-cache-sync]
                   [--stdin-name NAME] [--stdin-user USER]
                   [--stdin-group GROUP] [--stdin-mode M]
                   [--content-from-command] [--paths-from-stdin]
                   [--paths-from-command] [--paths-delimiter DELIM]
                   [-e PATTERN] [--exclude-from EXCLUDEFILE]
                   [--pattern PATTERN] [--patterns-from PATTERNFILE]
                   [--exclude-caches] [--exclude-if-present NAME]
                   [--keep-exclude-tags] [--exclude-nodump] [-x]
                   [--numeric-owner] [--numeric-ids] [--noatime] [--atime]
                   [--noctime] [--nobirthtime] [--nobsdflags] [--noflags]
                   [--noacls] [--noxattrs] [--sparse] [--files-cache MODE]
                   [--read-special] [--comment COMMENT]
                   [--timestamp TIMESTAMP] [-c SECONDS]
                   [--chunker-params PARAMS] [-C COMPRESSION]
                   ARCHIVE [PATH ...]
borg create: error: argument ARCHIVE: "/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/nextcloud.log.1": No archive specified
Deleting the failed backup archive...
Archive 20230308_053631-nextcloud-aio not found (1/1).

szaimen · 2023-03-08T09:10:04Z

Additionally, please make sure that OC\Preview\HEIC or OC\\Preview\\HEIC is removed from your config.php if it should be present as it might conflict with imaginary.

Did you check this?

I also noticed, that imaginary container is restarting in between NC is trying to generate previews

Is something in the imaginary or nextcloud logs maybe?

Maybe you broke something concerning the backups? Backups failing since 4.6.0 beta.

~~Yes, will be fixed in the next beta~~ Edit: This is now fixed with v4.6.1 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel

szaimen · 2023-03-08T16:12:16Z

I could finally reproduce the problem with this sample image:

https://github.com/dsoprea/heic-exif-samples/blob/master/image1.heic

It seems like the problem is that generating a preview takes forever:
172.18.0.8 - - [08/Mar/2023 16:10:28] "POST /pipeline?operations=%5B%7B%22operation%22%3A%22autorotate%22%7D%2C%7B%22operation%22%3A%22fit%22%2C%22params%22%3A%7B%22width%22%3A2048%2C%22height%22%3A2048%2C%22stripmeta%22%3A%22true%22%2C%22type%22%3A%22jpeg%22%2C%22norotation%22%3A%22true%22%2C%22quality%22%3A%2260%22%7D%7D%5D HTTP/1.1" 200 3915 54.6146

szaimen · 2023-03-08T16:12:51Z

However this is likely an imaginary issue so not sure how to fix this here...

szaimen · 2023-03-08T16:43:16Z

Okay, it looks like i found something. It looks like autorotate is broken for heic.

loeffelpan · 2023-03-08T17:59:15Z

I confirm that messages in imaginary logs.
Hopefully you got it with that PR.
But NC27 is quite a late milestone for fix.

szaimen · 2023-03-08T18:39:38Z

Dont worry. Since this is a bugfix it will be backported after the merge in master.

szaimen · 2023-03-08T23:21:35Z

Hi, can you please check if this patch makes things work for you? (works in my testing at least)
https://patch-diff.githubusercontent.com/raw/nextcloud/server/pull/37140.patch
Thanks a lot!

a3linux · 2023-03-09T04:07:36Z

h2non/imaginary#187 does said Imaginary supports HEIC already?

szaimen · 2023-03-09T07:51:32Z

Yeah but autorotate seems tobe broken for heic which will be fixed with the patch by disabling it for heic.

loeffelpan · 2023-03-09T09:49:47Z

Hi, can you please check if this patch makes things work for you? (works in my testing at least)
https://patch-diff.githubusercontent.com/raw/nextcloud/server/pull/37140.patch
Thanks a lot!

How to apply this?
I have to change the file inside the nextcloud-container, correct?

szaimen · 2023-03-09T09:51:01Z

# Go into the container
sudo docker exec --user www-data -it nextcloud-aio-nextcloud bash

# Now inside the container
wget https://patch-diff.githubusercontent.com/raw/nextcloud/server/pull/37140.patch
git apply --check 37140.patch && git apply 37140.patch
exit

loeffelpan · 2023-03-09T10:02:37Z

Thanks. Didn't know that git is available inside your container. Will check that today in the evening.

loeffelpan · 2023-03-09T19:23:21Z

Line 41 trailing whitespaces error
/lib/Private/preview/Imaginary.php:82 patch does not apply

szaimen · 2023-03-09T19:25:32Z

Line 41 trailing whitespaces error /lib/Private/preview/Imaginary.php:82 patch does not apply

Can you try without the --check? this should then actually apply it. you check if it applied if a second apply fails with a different error...

loeffelpan · 2023-03-09T19:27:59Z

Same error as before even without --check.
Check found the trailing whitespaces. The other error occurs when I fix that or omit the check.

ikamaletdinov · 2023-03-09T19:28:06Z

I could finally reproduce the problem with this sample image:

https://github.com/dsoprea/heic-exif-samples/blob/master/image1.heic

It seems like the problem is that generating a preview takes forever: 172.18.0.8 - - [08/Mar/2023 16:10:28] "POST /pipeline?operations=%5B%7B%22operation%22%3A%22autorotate%22%7D%2C%7B%22operation%22%3A%22fit%22%2C%22params%22%3A%7B%22width%22%3A2048%2C%22height%22%3A2048%2C%22stripmeta%22%3A%22true%22%2C%22type%22%3A%22jpeg%22%2C%22norotation%22%3A%22true%22%2C%22quality%22%3A%2260%22%7D%7D%5D HTTP/1.1" 200 3915 54.6146

Wow, that is really long. In my case it was about 6 sec before and 3 sec after updating libheif. It also dropped CPU usage from 100% to about 40%. Imagemagick could do that less than a second with 100% load.

szaimen · 2023-03-09T19:49:19Z

Check found the trailing whitespaces. The other error occurs when I fix that or omit the check.

It should have still applied the patch at least this was the case in my testing.

szaimen · 2023-03-09T19:50:17Z

Wow, that is really long. In my case it was about 6 sec before and 3 sec after updating libheif. It also dropped CPU usage from 100% to about 40%. Imagemagick could do that less than a second with 100% load.

Did you test with the same file? In my testing the only way to make it work was to skip the autorotation for heic which was done in the mentioned PR and patch.

loeffelpan · 2023-03-09T21:04:37Z

Check found the trailing whitespaces. The other error occurs when I fix that or omit the check.

It should have still applied the patch at least this was the case in my testing.

Can't get that patch working.
Now I donwloaded the file from github as raw. Seems to work. Thanks a lot.
It's not that fast, but OK on my dual core celeron.

172.20.1.7 - - [09/Mar/2023 20:59:45] "POST /pipeline?operations=%5B%7B%22operation%22%3A%22fit%22%2C%22params%22%3A%7B%22width%22%3A256%2C%22height%22%3A256%2C%22stripmeta%22%3A%22true%22%2C%22type%22%3A%22jpeg%22%2C%22norotation%22%3A%22true%22%2C%22quality%22%3A%2260%22%7D%7D%5D HTTP/1.1" 200 6637 3.3129

ikamaletdinov · 2023-03-09T21:32:00Z

Wow, that is really long. In my case it was about 6 sec before and 3 sec after updating libheif. It also dropped CPU usage from 100% to about 40%. Imagemagick could do that less than a second with 100% load.

Did you test with the same file? In my testing the only way to make it work was to skip the autorotation for heic which was done in the mentioned PR and patch.

Yeap, I have checked on same small HEIC database (~1000 files) got from IPhone without any conversion. I guess in my case all files were rotated correctly. Disabling crop/rotate and other functions didn't speed up the whole set.

Is that possible to add some kind of checkbox to enable/disable HEIC support in imaginary? I guess for single/dual core usage it would be better to use Imaginary for everything except HEIC and regular previews for HEIC (single threaded but fast).
Currently (with updated version of libheif1 that was already merged as I can see) we can get about ~3 sec per image. It is still three times slower than Imagemagick's conversion but could be suitable for multithreaded load.

loeffelpan · 2023-03-09T21:40:09Z

Currently (with updated version of libheif1 that was already merged as I can see) we can get about ~3 sec per image. It is still three times slower than Imagemagick's conversion but could be suitable for multithreaded load.

So you're saying Imaginary on my (dual-core) system is slower as default preview generation would be?
Never tested.

a3linux · 2023-03-10T22:04:45Z

Manually patch my server both test and production environments, got the similar result, not much faster, but it works.
Bring high CPU and system load with large amount of HEIC folder, but preview works.

MrRinkana · 2024-09-27T15:24:46Z

Are the patches applied to the nextcloud/aio-imaginary container? Or do we still have to apply the patches manually?

The latest documentation states imaginary has issues with HEIC and links to this issue, which is closed. Maybe there are other issues that should instead be linked to?

joshtrichards · 2024-09-27T19:46:58Z

@MrRinkana The patching was only a discussion relevant before the linked PR was merged. It's been merged for well over a year now.

Gunzinger added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Dec 7, 2022

szaimen transferred this issue from nextcloud/all-in-one Dec 7, 2022

szaimen added the 24-feedback label Dec 7, 2022

ikamaletdinov mentioned this issue Feb 23, 2023

Enable Imaginary by default with the upgrade to Nc26 nextcloud/all-in-one#1833

Closed

szaimen mentioned this issue Mar 6, 2023

fix set_memory for imaginary and move cap_add to containers.json nextcloud/all-in-one#2113

Merged

szaimen added 2. developing Work in progress and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Mar 6, 2023

This was referenced Mar 6, 2023

update some imaginary dependencies to improve performance nextcloud/all-in-one#2114

Merged

update imaginary docs to reflect cap_add sys_nice nextcloud/documentation#9724

Merged

szaimen closed this as completed Mar 7, 2023

szaimen reopened this Mar 8, 2023

szaimen mentioned this issue Mar 8, 2023

imaginary - fix autorotate for heic and improve the logic #37140

Merged

szaimen closed this as completed in #37140 Mar 9, 2023

matiasdelellis mentioned this issue May 21, 2023

Imaginary autorotate post call times out too quickly for large HEIC panorama image matiasdelellis/facerecognition#662

Closed

traklo mentioned this issue Jun 18, 2023

Imaginary cURL error 28 #37654

Closed

9 tasks

joshtrichards added the feature: previews and thumbnails label Apr 5, 2024

Imaginary preview generation is not working as expected for folders containing HEIC images. #35643

Imaginary preview generation is not working as expected for folders containing HEIC images. #35643

Comments

Gunzinger commented Dec 7, 2022 • edited Loading

How to use GitHub

Steps to reproduce

Expected behavior

Actual behavior

Host OS

Nextcloud AIO version

Current channel

Other valuable info

szaimen commented Dec 7, 2022

loeffelpan commented Jan 5, 2023 • edited Loading

Jip-Hop commented Feb 16, 2023

ikamaletdinov commented Feb 23, 2023

deetuned commented Feb 24, 2023

ikamaletdinov commented Mar 5, 2023 • edited Loading

szaimen commented Mar 6, 2023

loeffelpan commented Mar 6, 2023 • edited Loading

szaimen commented Mar 7, 2023

loeffelpan commented Mar 7, 2023

szaimen commented Mar 7, 2023 • edited Loading

szaimen commented Mar 7, 2023

loeffelpan commented Mar 7, 2023 • edited Loading

szaimen commented Mar 7, 2023 • edited Loading

loeffelpan commented Mar 8, 2023

szaimen commented Mar 8, 2023 • edited Loading

szaimen commented Mar 8, 2023

szaimen commented Mar 8, 2023

szaimen commented Mar 8, 2023 • edited Loading

loeffelpan commented Mar 8, 2023 • edited Loading

szaimen commented Mar 8, 2023

szaimen commented Mar 8, 2023 • edited Loading

a3linux commented Mar 9, 2023

szaimen commented Mar 9, 2023

loeffelpan commented Mar 9, 2023

szaimen commented Mar 9, 2023

loeffelpan commented Mar 9, 2023 • edited Loading

loeffelpan commented Mar 9, 2023

szaimen commented Mar 9, 2023 • edited Loading

loeffelpan commented Mar 9, 2023 • edited Loading

ikamaletdinov commented Mar 9, 2023 • edited Loading

szaimen commented Mar 9, 2023

szaimen commented Mar 9, 2023 • edited Loading

loeffelpan commented Mar 9, 2023 • edited Loading

ikamaletdinov commented Mar 9, 2023 • edited Loading

loeffelpan commented Mar 9, 2023

a3linux commented Mar 10, 2023

MrRinkana commented Sep 27, 2024

joshtrichards commented Sep 27, 2024

Gunzinger commented Dec 7, 2022 •

edited

Loading

loeffelpan commented Jan 5, 2023 •

edited

Loading

ikamaletdinov commented Mar 5, 2023 •

edited

Loading

loeffelpan commented Mar 6, 2023 •

edited

Loading

szaimen commented Mar 7, 2023 •

edited

Loading

loeffelpan commented Mar 7, 2023 •

edited

Loading

szaimen commented Mar 7, 2023 •

edited

Loading

szaimen commented Mar 8, 2023 •

edited

Loading

szaimen commented Mar 8, 2023 •

edited

Loading

loeffelpan commented Mar 8, 2023 •

edited

Loading

szaimen commented Mar 8, 2023 •

edited

Loading

loeffelpan commented Mar 9, 2023 •

edited

Loading

szaimen commented Mar 9, 2023 •

edited

Loading

loeffelpan commented Mar 9, 2023 •

edited

Loading

ikamaletdinov commented Mar 9, 2023 •

edited

Loading

szaimen commented Mar 9, 2023 •

edited

Loading

loeffelpan commented Mar 9, 2023 •

edited

Loading

ikamaletdinov commented Mar 9, 2023 •

edited

Loading