files locked for no obvious reason

[Floflobel]

Steps to reproduce

1. Make files into synchronisation directory
2. Synchronisation has launch
3. Failed

Expected behaviour

Files synchronise

Actual behaviour

Files not synchronise

Server configuration

Operating system: Ubuntu 16.04.4 LTS

Web server: apache2 2.4.18

Database: postgres 9.6

PHP version: 7.0.25

Nextcloud version: (see Nextcloud admin page) 13.0.0.14

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from:

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.6.1
  - bruteforcesettings: 1.0.3
  - calendar: 1.6.0
  - comments: 1.3.0
  - contacts: 2.1.2
  - dav: 1.4.6
  - federatedfilesharing: 1.3.1
  - federation: 1.3.0
  - files: 1.8.0
  - files_external: 1.4.1
  - files_markdown: 2.0.4
  - files_pdfviewer: 1.2.0
  - files_sharing: 1.5.0
  - files_texteditor: 2.5.1
  - files_trashbin: 1.3.0
  - files_versions: 1.6.0
  - files_videoplayer: 1.2.0
  - firstrunwizard: 2.2.1
  - gallery: 18.0.0
  - impersonate: 1.0.2
  - logreader: 2.0.0
  - lookup_server_connector: 1.1.0
  - metadata: 0.6.0
  - nextcloud_announcements: 1.2.0
  - notifications: 2.1.2
  - oauth2: 1.1.0
  - onlyoffice: 1.2.0
  - password_policy: 1.3.0
  - provisioning_api: 1.3.0
  - serverinfo: 1.3.0
  - sharebymail: 1.3.0
  - survey_client: 1.1.0
  - systemtags: 1.3.0
  - tasks: 0.9.6
  - theming: 1.4.1
  - twofactor_backupcodes: 1.2.3
  - twofactor_totp: 1.4.1
  - updatenotification: 1.3.0
  - user_usage_report: 1.0.1
  - workflowengine: 1.3.0
Disabled:
  - admin_audit
  - encryption
  - files_clipboard
  - files_reader
  - keeweb
  - mail
  - ojsxc
  - spreed
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "box.home.floheberg.xyz"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/box.home.floheberg.xyz",
        "dbtype": "pgsql",
        "version": "13.0.0.14",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "5432",
        "dbtableprefix": "",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": "true",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 1.5,
            "dbindex": 0
        },
        "logtimezone": "Europe\/Paris",
        "loglevel": 2,
        "log_type": "owncloud",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "logdateformat": "Y-m-d H:i:s T",
        "log_query": false,
        "log_authfailip": false,
        "log_rotate_size": 104857600,
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "maintenance": false,
        "mail_smtpsecure": "ssl",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "theme": ""
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
Yes local storage

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
Just nextcloud client

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

-rw-r--r--   1 www-data www-data    0 Oct 28 00:36 nextcloud.log

[Floflobel]

When I try to upload files with web browser it's return:
Error when assembling chunks, status code 403 Forbidden

[enoch85]

Releated: #6655

Still get file locks from time to time without any reason. Just by downloading the file (I think) it get's locked which is kind of stupid.

[Floflobel]

I can download any files but I can not upload files on my nextcloud.
The table "files_lock" are empty and I tried to put "filelocking.enabled' => 'false'" into the config.php and it's does not work.

[enoch85]

@Floflobel Yes I understand that, but what I was meaning is that it gets locked when someone downloads and therefore you can't upload or download, that's what I think at least.

Strange thing is that it happens randomly, and I have no clue how to debug it.

My log is filled with "file is locked" with this file: https://cloud.techandme.se/s/whxC00V1I0l4CY8 and it's been like that forever. Just giving my input on this matter. :)

[Floflobel]

@enoch85 Ok, I understand better. I do not think it's the fact that it's downloading it locks it because I'm the only user ..
Unlike you I do not have a single file to lock but thousands.

Hoping that we will find something to debug the problem. ;)

[enoch85]

@Floflobel Feel free to debug it, if you find the cause then we can fix it. :)

[Wenzel]

Hi,
i just have the same situation here.

i cannot synchronize my files, because a file has been "locked" on the server.
I cannot even remove this file anymore on the server, my sync is broken !

What can i do to remove it ?

[enoch85]

@Wenzel Do you have any shared folders which syncs with the client? Like federated shares or similar.

[Wenzel]

@enoch85 not at all.

I tried to remove the file on the filesystem, and i'm now rescanning the filesystem with ./occ files:scan

[enoch85]

@Wenzel OK, please post the output.

[Wenzel]

Now i have new files being locked for no reasons:

Please help, we have a bug here.

[Wenzel]

And i can't delete the file:

translated: "error while trying to delete 'hypervisor_debugger.md'"

[enoch85]

@Wenzel You should always use ./occ files:scan with caution. In your case I would turn of the webserver (Apache or whatever) and then wait for 20 minutes and then remove all files locally, . When you've done that, turn on the webserver again and resync all the files.

Do it with caution though, and I'm no expert just so you know.

[Wenzel]

hello,

It happened again today

This is definitely a bug in Nextcloud server, we need to investigate.

[enoch85]

Ping @MorrisJobke and others (don't know who this belongs to).

[MorrisJobke]

cc @icewind1991 and @rullzer because we discussed about this lately

[vbspam]

I just want to confirm it is happening also to me. I run two NC servers, one 12.0.0 the other 13.0.2. Both suffers of this ".. server replied: Locked (''/path/to/filename/" is locked) issue.

My NC desktop client is "Version 2.3.3", it however from the response it looks it is probably server issue.

My observation is that this is happening on files which change frequently. Could it be something with the timestamp resoluttion?

Just to note, I do not believe it may cause the issue, my server runs UTC time, my client runs CEST time, both auto synced through NTP.

Server runs on Ubuntu 16.04, nginx version: nginx/1.13.6, PHP 7.0.28-0ubuntu0.16.04.1 (fpm-fcgi).
Client runs Ubuntu 16.04 Neon flavour.

Also the suggested fix to disable locking and './occ files:scan' did not solved the issue.

[warnerbryce]

I have the same kind of problem... but the files syncs well after waiting a while. It occures when uploading a folder with a lot of files.

I have write a topic on help nextcloud https://help.nextcloud.com/t/sync-some-files-are-blocked-error-during-sync-and-then-synced-after-a-second-third-try-no-app-in-context/30986

[ConorIA]

Same issue here. Typically it happens when a file is edited both on my server and on my laptop. (I think). The lock then persists for days or longer, unless manually removed.

The errors seem to come in pairs. One no-app-in-context error, followed by the file lock error.

Fatal | webdav            | OCA\DAV\Connector\Sabre\Exception\FileLocked: "Documents/Extra-Curricular/TF 2018/book" is locked                     | a minute ago
Error | no app in context | OCP\Lock\LockedException: "files/485fa3ae591d79ba0220864f977e1b2e" is locked, existing lock on file: -1 shared locks  | a minute ago
Fatal | webdav            | OCA\DAV\Connector\Sabre\Exception\FileLocked: "Documents/Extra-Curricular/TF 2018/mdbook" is locked                   | a minute ago
Error | no app in context | OCP\Lock\LockedException: "files/42e92d9acbfd2f1d7edfa2a953a5bc68" is locked, existing lock on file: -10 shared locks | a minute ago

Negative numbers of shared locks seems suspect to me, especially considering that this post seems to assume that they should all be 1, but I really don't know much about this. Redis is active, but this issue was much more prevalent before turning on Redis.

EDIT: This doesn't seem right:

MariaDB [nextcloud]> SELECT * FROM oc_file_locks;                         
Empty set (0.000 sec)

EDIT2: Solution was docker exec -it nextcloud_redis_1 redis-cli flushall

[JamesBream]

Have also experienced the same issue as @ConorIA with the logs showing negative numbers of file locks when using redis as a result of syncing via the Mac desktop client. There were no locks in the MySQL database as expected.

Didn’t know about the redis flush command but the issue resolved itself after a few hours, through did persist through restarts.

[enoch85]

This has been an issue for quite some time I'm afraid. Maybe NC 14 will be more stable, let's hope for that anyway.

[vbspam]

I understand it may be here for some time, it however does not solve the issue and leave the Nextcloud unusable (due to the lack of trust) for everyday syncing of multiple devices.

Is there test case which always fails and can be reproduced?

[icewind1991]

OCP\Lock\LockedException: "files/42e92d9acbfd2f1d7edfa2a953a5bc68" is locked, existing lock on file: -10 shared locks

Lock values getting negative should-not-happen™ and is a likely cause of files being persistently locked.

#10116 should prevent this from happening.

[nhadsall]

I stopped apache, applied the patch, cleared the locks table, then started apache. Still getting dozens of "OCP\Lock\LockedException"

[kesselb]

There a different options for the file locking. Which one are you currently using (Redis or DB?). You could try to disable it (and enabled it again after you restored the files) like suggested here #8723 (comment).

It's easier for us to help you if you provide the information from the issue template: https://github.com/nextcloud/server/issues/new?labels=bug%2C+0.+Needs+triage&template=Bug_report.md

[MorrisJobke]

I stopped apache, applied the patch, cleared the locks table, then started apache. Still getting dozens of "OCP\Lock\LockedException"

Then it's something else. Could you enable the debug log of the file locking?

See

server/config/config.sample.php

Lines 1621 to 1630 in e6333c8

	/**
	* Enable locking debug logging
	*
	* Note that this can lead to a very large volume of log items being written which can lead
	* to performance degradation and large log files on busy instance.
	*
	* Thus enabling this in production for longer periods of time is not recommended
	* or should be used together with the ``log.condition`` setting.
	*/
	'filelocking.debug' => false,

[nhadsall]

Then it's something else. Could you enable the debug log of the file locking?

Set 'filelocking.debug' to false to enable it or set it to true? Should I open new bug report?

[MorrisJobke]

Set 'filelocking.debug' to false to enable it or set it to true? Should I open new bug report?

set it to true. And yes a new ticket that has a link to this one here is maybe best.

[optimistic5]

Looks like on 15.0.2 and now on 15.0.6 everything is fine. No locks. Thanks.

[vbspam]

Looks like desktop client version 2.5.2git with server from 13.x to 15.0.7 still has the issue.

[enoch85]

Looks like desktop client version 2.5.2git with server from 13.x to 15.0.7 still has the issue.

Just as a reference, since Nextcloud 15.0.3 + desktop client 2.5.1 I didn't have any issues at all. Before this got fixed I had issues every time I uploaded the Nextcloud VM using the client.

So for me this is totally fixed, and I'm a pretty heavy user / tester.

[kesselb]

Please open a new issue (https://github.com/nextcloud/server/issues/new?labels=bug%2C+0.+Needs+triage&template=Bug_report.md) if you still observe locking problems. Let us know if you use redis or db for locking.

[vbspam]

Opened here: #15235; I do not know if my instance is using Redis; when checked for 'redis' process through ps auwx |grep redis did not find any process.

[Wenzel]

2 years later, and the same bug is still lurking around in Nextcloud code.

Some of my files have been locked again, and I can't remove them anymore.
I'm unable to run the occ command because I'm on a hosted service, without SSH access.

Why this bug is still here ? :(

[bpcurse]

@Wenzel Can you use occ web app for this task?
https://apps.nextcloud.com/apps/occweb

[Wenzel]

hey @bpcurse, thank you so much for the quick reply !
I totally missed occweb in the app store.

I just installed it.
What command should I run to remove the file locks ?

[warnerbryce]

@Wenzel you can try a
occ files:scan yourusername

[Wenzel]

@warnerbryce I tried that, and there is a locked folder that I still can't remove.
It didn't solve my problem

[warnerbryce]

Okay so you have to interact with the database of your nextcloud. Do you have access to your mysql database with phpmyadmin or something like that ?

[bpcurse]

@Wenzel I suppose you have database access?

What you could try is occ files:cleanup and if that doesn't help delete all entries in the table file_locks.

WARNING: Make sure to have a backup files/database and no user/no sync client connected to your cloud! Manipulating the database is always a risk and I am not a developer.

If you need further support, please use https://help.nextcloud.com.

[Wenzel]

@warnerbryce @bpcurse I will try occ files:cleanup.

however, I'm using a hosted solution, and I don't have SSH access, so no database either.

[warnerbryce]

@Wenzel you must have access to a phpmyadmin or other page to manipulate your database. It can’t be other way.
Go on the help.nextcloud.com discourse. My name is Nemskiller PM me and i will help you.

[stagg54]

This issue is still here. None of the fixes I have found anywhere work. I am using redis. tried flushing it. did not work.

[krusta4711]

I can reproduce this issue every time I use the following workflow (NC 18.04., encrypted enabled, shared hosting with 128MB RAM):

1.) On Win10 PC: Create a new folder 'test' and copy at least 6 JPEGs to it (each 5MB)
2.) On IOS or Android App: Go to the new folder and wait until it is synchronized.
3.) Move up to the parent folder and delete the 'test' folder

The deletion ALWAYS fails for me in this setup. I get the message, that the data is locked. Trying to delete the JPEGs file by file after that does work for some images, for others not (with locked message).

I do NOT get errors, when I delete the folder from Android/IOS App without going into it before.

[warnerbryce]

@krusta4711 encryption with a 128MB ram on shared hosting is really low specs for having a good working nextcloud server.

[krusta4711]

Yes you are right @warnerbryce
But in general it works well. We are just two users and there is no heavy usage. Just a few files that are shared. The locking issue is the only one I observed until now. I think I will deactivate the locking in future ('filelocking.enabled' => false,). I do not expect a lot issues with two users.

So I wrote down my workflow to help the other users in this thread with this issue. Nobody could state a workflow to reproduce the issue until now. Maybe someone with a bigger server can try to reproduce my workflow. If it fails too that might help to solve the issue.

[beatrfrm]

Hello the same here.
I uploaded a 260 MB .mp4 video. I can delete it.
If I open this video and then want to delete it, I get the Error deleting File massage.
If I restart Apache then I can delete the video again.
I uploaded a 200MB file without extension, wich I can't open therefore I can delete it.
APCu, Redis makes the same result.
Thank you.

[warnerbryce]

@beatrfrm and you have encryption enabled or not ?

[beatrfrm]

@beatrfrm and you have encryption enabled or not ?

@warnerbryce as I know I didn't enabled any encryption. Actually is a new installation, with some apps. But if I disable all (the basic apps to) apps the problem is still here. It is a LAMP Server. filelocking is disabled.
I opened a issue for my problem: #21170

[p6002]

I see that the problem has been around for 3 years and has not been resolved in any way.
This is the opensource world.
If you have any ideas on how I can remove MY files from MY cloud please write.

[stefan123t]

@p6002 in my case it was caused by a mis-configuration of the basic setting for Cron instead of AJAX Cron.

So if you go to Settings > Administration > Basic Settings it will tell you when it last ran the Background Jobs
That wrong setting prevented to run the BackgroundJob/CleanupFileLocks.php for more than five months in my case.

I found something along the following SQL quite useful for debugging that cause:
SELECT FROM_UNIXTIME( file_locks.ttl, '%Y-%m-%d %h' ) AS DATE, file_locks.lock, COUNT(*) AS LOCKS FROM file_locks GROUP BY DATE, file_locks.lock ORDER BY LOCKS DESC

See #15235 for more details

Kind regards,
Stefan