[NC 20] Large file uploads fail - File locking issue

[SimplyCorbett]

I'm having a very interesting problem with nextcloud 20.

When uploading 12GB files, it states there is a chunk error and doesn't finish in the GUI. However, it continues in the background PHP process until it's fully assembled. File is then successfully put into the correct folder in the WebGUI and checksums match.

But when I upload large 50GB+ .tar.gz files one of two things happen:

1. Like above, chunk errors. PHP process continues to burst up to 100% CPU with 100% I/O wait for a good half hour, gives up and fails.

2. File locking errors.

In both cases the uploaded file remains in chunks in the user/uploads folder, taking up 50GB+ space and does not automatically clear.

I have attempted to modify all set limits to resolve this. I have had no success.

There are no errors in any logs.

Server Specifications:

2 core 3900x
6GB RAM
50GB SSD root file system with 10TB drives attached to user/files, user/files_trashbin, user/uploads. Root file system does not grow and maintains it's 25GB usage during this entire process.

These are bind mounts so nextcloud shouldn't be able to tell the difference or even know there is a mount there.

Server software:
Nginx Mainline 1.19.3
Mariadb 10.5.6 in a remote server (LAN)
PHP-FPM 7.4.11
Nextcloud 20.0.1
Linux

Settings:

PHP (cli & fpm)
Memory limit set to 2048M (tested up to 3GB)
Execution limit set to 3600
Max upload size/post max size 200G
Nginx Body size set to 100G

Web browser:

Firefox 81
Linux

**Server config.php**

pastebin: https://pastebin.com/raw/eHY5RN6d

[TheBluestBird]

Have really embarrassed myself with "my geecky tech whatever". Have the same problem but with apache

[SimplyCorbett]

Any ideas? Nextcloud cannot be my primary storage platform with issues like this.

I'm also running into various issues with uploading multiple files in one go. Throw a folder in there with 500+ files (via web gui) and it will typically not upload every file or fail with errors. I've also had some corrupt files before after upload.

Disabling file locking helps on the second problem but does not solve the issue.

[SimplyCorbett]

Going to reopen this. Disabling the bind mounts and just straight mounting the data folder on the 10TB drive solved this issue but did not fix this one.

I can install nextcloud on a baremetal install with no mounts to test but it would defeat my use purpose. I want to keep the installation on a SSD and if I use a external local mount images will no longer load previews because of how the app is designed... issue.

[SimplyCorbett]

The solution to this was disabling http/2 (upload speed jumped from 1.5MB/s to 50MB/s) and disabling file locking. The backend can now successfully assemble the 50GB file but does still throw chunk errors. Final checksums match though so I'm happy.

@kesselb If files aren't assembling because of file locking that seems like a nextcloud bug to me. I don't have redis installed but don't see why that would matter for file uploads.

Going to go ahead and close this anyway since the problem is solved for me. I don't need file locking for my use case but that's pretty weird.

[brainchild0]

I remain affected by the same issue, on Nextcloud 27.x (most recently confirmed in 27.0.1).

Despite the suggested workaround, the problem persists in the application.

As such, I would suggest that the issue remain open, until it may be addressed properly in the form of a bug fix contributed to the application.

I agree with the reference to #1928, which appears to be essentially the same issue, only occurring with perhaps greater frequency in the particular context of containers.