Deleting a folder with many files via client or web interface take longer

[iliyasbasir]

[3159]

Expected behavior

Folder should be deleted immaterially

Actual behavior & Steps to reproduce

I've been trying to track down a problem that I thought was in our db configuration and I think it actually is a problem with ownCloud. When you delete a folder with many files in it, either via the desktop sync client or in the web interface, it takes AGES (if it succeeds at all) and UPDATE queries to our database go from less than 1/s to 30/s.

The worst part is that the desktop client will timeout after 5 minutes (1.7.1) and then a short time later start again. But the original httpd process is still running. So now you've got two httpd processes (DELETE verb in the request) and 60 UPDATE queries per second. And then again, and again until the database is starting to have problems. The only solution is to kill the httpd processes. But that only fixes things until the client starts trying to sync again.

I added a timeout value to my desktop config file of 600 to set it to 10 minutes. It takes 10 minutes to timeout but at the 5 minute mark it starts another sync run anyway and I see two httpd requests for this folder 300 seconds apart. When it times out, it starts again after a few seconds and then you've got 3.

I can't find any entries at all in the owncloud log (even with debug turned on) other than an audit entry stating that I was trying to delete that folder.

The folder I've been testing with is the src folder from the client repository so it has roughly 1000 files.

I set mysql to log all queries. Here is a sample of the queries that are happening from the moment a delete starts. It repeats with essentially the same stuff.

UPDATE `oc_filecache` SET `size`='0' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431624421', `storage_mtime`='1431624421' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `unencrypted_size`='0' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd0454d3' WHERE `fileid` = '8';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd048c81' WHERE `fileid` = '1535577';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd04dbcf' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `mtime` = '1431620325', `size` = '-1', `etag` = '5554dbd05acd7', `storage_mtime`='1431620325' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `size`='0' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431624421', `storage_mtime`='1431624421' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `unencrypted_size`='0' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431620325', `etag`='5554dbd0889cf' WHERE `fileid` = '8';
UPDATE `oc_filecache` SET `mtime` = '1431620325', `etag`='5554dbd08cb36' WHERE `fileid` = '1535577';
UPDATE `oc_filecache` SET `mtime` = '1431620325', `etag`='5554dbd090b30' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `size`='0' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `mtime` = '1431620325', `storage_mtime`='1431620325' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `unencrypted_size`='0' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd0eca58' WHERE `fileid` = '8';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd0f16f2' WHERE `fileid` = '1535577';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd113901' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `etag`='5554dbd117c1d' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431620079', `size` = '-1', `etag` = '5554dbd124e16', `storage_mtime`='1431620079' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `size`='0' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `mtime` = '1431624656', `storage_mtime`='1431624656' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `unencrypted_size`='0' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `mtime` = '1431620079', `etag`='5554dbd1820cf' WHERE `fileid` = '8';
UPDATE `oc_filecache` SET `mtime` = '1431620079', `etag`='5554dbd185d0a' WHERE `fileid` = '1535577';
UPDATE `oc_filecache` SET `mtime` = '1431620079', `etag`='5554dbd18817d' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `mtime` = '1431620079', `etag`='5554dbd18bd0e' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `storage_mtime`='1431624657' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `size`='1140' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `size`='1140' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `size`='3673118' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `size`='3673118' WHERE `fileid` = '1535577';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `etag`='5554dbd1d1376' WHERE `fileid` = '8';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `etag`='5554dbd1dedd6' WHERE `fileid` = '1535577';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `etag`='5554dbd1ed3c7' WHERE `fileid` = '1535581';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `etag`='5554dbd205849' WHERE `fileid` = '1536945';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `etag`='5554dbd20bc7c' WHERE `fileid` = '1536949';
UPDATE `oc_filecache` SET `mtime` = '1411100797', `etag` = '5554dbd210eb2', `storage_mtime`='1411100797' WHERE `fileid` = '1536953';
UPDATE `oc_filecache` SET `mtime` = '1431624657', `storage_mtime`='1431624657' WHERE `fileid` = '1536949';

In my trash I'll the src folder with a subset of content. The src folder in my account still remains untouched in the web interface. Each time the client tries a delete a new src folder appears in the trash. It seems like it's trying to copy it to trash before removing it from my account (a cp then rm as opposed to mv).

I've tried emptying the trash. I've seen the same behavior with deletes from the web interface with the ajax/delete.php script in the http request. Same fix, kill that httpd process.

I'm not sure what is causing all these masses of update queries but at 30 queries/sec for ~1000 files I can't see why this goes on for such a long time without success.

I saw one github issue on #15315 that referred to a bug in PHP when moving content across file systems. Our data directory is on an SMB mount (/mnt/oc_data). I'm not sure if deleting files and getting them into the trash involves moving them to the local hard drive (or to /dev/shm or to php tmp which is on a different drive again) and then back to the data dir via smb but if so, then this php bug could be causing the problem, in which case we need to get that fix backported to 7.

Server configuration

Operating system: centos 6.6
Web server: Apache 2.2.15
Database:
PHP version: 5.5.20
ownCloud version:7.0.4

[MorrisJobke]

should be a lot faster with 8.0 and even more with 8.1

ref #13948 #15360, #13360

@PVince81 and @icewind1991 Maybe know more about this.

[MorrisJobke]

#15549 should reduce the amount of UPDATE statements to the database and will be in 8.0.4 and 8.1.0

[PVince81]

8.0 moves files to trash instead of copy, improves speed.
8.1 defers the expiration logic that used to happen at delete time. Now the expiration logic schedules an async job for that. So deletion should be even faster.

[ghost]

@jnfrmarks Please have QA test this. Merged already, we need to know if it's closed

[jnfrmarks]

@cmonteroluque @gig13

I"m not sure what the test case is here. Can someone please put the steps in here? Thanks!

[SergioBertolinSG]

I've tried this using 8.0.4 oC with a SMB mount and deleting a complex folder structure inside, no problems found. It was deleted quickly.

[jnfrmarks]

@SergioBertolinSG

Thanks! Sounds like we can close this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.