-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nginx + NC12 having problems with small file uploads via web interface #6734
Comments
Hi, we have similar problems during upload of some files. We use CentOS 7.4 with Apache, PHP-FPM and PHP 7.0. Our problem seems to be an locking issue. Files get uploaded at normal speed and the process get stuck for about 30 seconds. One php worker generates a high load and the redis server is used in an "abnormal" manner.
Do you use redis too? We see this high redis usage during opening the trashbin and while loading the Apps page too. |
I implemented Redis to see if it would solve my issue, however it made no difference. I have this problem both with and without Redis. However, I wonder if this is a PHP-FPM thing since I've been using Apache and PHP7 without PHP-FPM and I'm not experiencing that issue (or didn't before switching to Nginx). So I wonder, if Apache + PHP-FPM is having the issue, maybe there's a config there missing? Apache is using mpm-prefork and I'm assuming you're using mpm-worker or event yes? |
I'm continuing my suspicion of it being PHP-FPM or nginx. I set up a stock setup (no perf optimizations) of nextcloud on the exact same machine (both of these are operating within jails. Apache took less than half the time to upload the same set of files. Again bear in mind, desktop clients are not affected by this slow upload issue |
Just eliminated PHP-FPM and nginx as factors I believe. On precisely the same server I switched back to apache and the problem still occurred, I'm still not convinced it's a database issue though since the desktop client is so fast. I'm completely at a loss. I feel I have tried everything and this is just a dead instance of Nextcloud that needs an utter redo |
Hi,
yesterday I fixed our problem, we had about 150k entries in the oc_file_locks
table and after each upload the system locked, unlocked each of them in the
redis server which caused the delay.
I have no idea why the files where locked in the MySQL. Some weeks ago
we had trouble with locked files and we changed the locking settings
in the config.php serveral times. Maybe this lead to the entries in the
table.
If you enable debugging, maybe you find your problem too.
|
As far as I know, I'm not seeing that issue (or maybe I'm missing it), I did see this now:
and {"reqId":"zItXgO5ojMfWsFJpZr93","level":4,"time":"2017-10-06T01:54:43+00:00","remoteAddr":"192.168.10.47","user":"courtney","app":"webdav","method":"PUT","url":"/remote.php/webdav/Programs/webroot/bin/System.Web.WebPages.dll","message":"Exception: {"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"expected filesize 211656 got 9222","Code":0,"Trace":"#0 \/usr\/local\/www\/nginx-dist\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php(151): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #9)\n#1 \/usr\/local\/www\/nginx-dist\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1096): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('System.Web.WebP...', Resource id #9)\n#2 \/usr\/local\/www\/nginx-dist\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(525): Sabre\\DAV\\Server->createFile('Programs\/webroo...', Resource id #9, NULL)\n#3 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#4 \/usr\/local\/www\/nginx-dist\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#5 \/usr\/local\/www\/nginx-dist\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#6 \/usr\/local\/www\/nginx-dist\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#7 \/usr\/local\/www\/nginx-dist\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php(76): Sabre\\DAV\\Server->exec()\n#8 \/usr\/local\/www\/nginx-dist\/nextcloud\/remote.php(162): require_once('\/usr\/local\/www\/...')\n#9 {main}","File":"\/usr\/local\/www\/nginx-dist\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","Line":151}","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0","version":"12.0.3.3"} I will see if there's anything in the docs or other issues. Found a typo setting up HSTS but that unfortunately wasn't the issue |
I have 7K file locks in my database. Could you lead me to what you did to solve that to see if that fixes my issue? |
Just try to drop all content from the lock table. |
Dropped all the contents from the lock table with no change. Still watching nextcloud.log and I am still only getting the errors as stated in my 2nd recent post. Still no change in performance, nginx is giving me: however, this error seems to occur even when not doing an upload. My guess now though is it is related to that webdav error, since it is spitting that error out when doing PUT |
In more recent versions (13.0.4 and 12.0.9) we reduced drastically the amount of lock statements which should avoid most of the problems in here. Thus I will close it. If there is still stuff that breaks have a look at #9305 and it's linked issues. Another workaround: use Redis as locking backend improves the situation as well. Hopefully this solves the issue you have with the upload of the small files. If this still happens and is unrelated to file locking (maybe test it by disable it completely). We can reopen this ticket. |
Steps to reproduce
Expected behaviour
Files should upload at an expected speed.
Actual behaviour
It takes about 20 minutes to upload about 20MB of small files
Server configuration
Operating system: FreeBSD 11.1
Web server: 1.12.1
Database: 10.2.6
PHP version: 7.1.9
Nextcloud version: (see Nextcloud admin page) 12.0.3
Updated from an older Nextcloud/ownCloud or fresh install: From 12.0.1
Where did you install Nextcloud from: Source packages on nextcloud.com
Signing status:
Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: none
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
Client configuration
Browser: Firefox 57, 58, Chromium 60
Operating system: Windows 10, Fedora 26
Logs
Web server error log
Web server error log
Here is my nginx config file, this problem doesn't appear to happen on Apache
I also have OPCache being used. It is strange that the issue only occurs via the web interface, and not from the desktop clients
The text was updated successfully, but these errors were encountered: