Already synced files are synced again

[meixi]

Expected behaviour

Client should do nothing

Actual behaviour

Client syncs already in-sync files again and again

Steps to reproduce

1. Have an already synced filebase on the local PC nad the Server
2. Start PC and ownCloud client

Server configuration

Operating system:
Ubuntu 12.04.2

Web server:
Apache 2.2.22

Database:
mysql Ver 14.14 Distrib 5.5.31

PHP version:
5.3.10-1ubuntu3.6 with Suhosin-Patch

ownCloud version:
5.0.8

Storage backend:
?

Client configuration

Client version:
1.3.0

Operating system:
Windows 7 64bit

OS language:

Installation path of client:
c:\Program Files (x86)\ownCloud\

Logs

I added the parts of the client log i found regarding one of the files as an example.
This happens for all other files as well.
Why are MD5 different ? Files did not change for sure.

07-18 21:34:15:709 csync_vio_stat: Win32: STAT-inode for D:\Eigene Dateien\OwnCloud/Bilder_Share/Paul/0292_DSCN0079.jpg: 549961
07-18 21:34:15:709 csync_ftw: Uniq ID from Database: Bilder_Share/Paul/0292_DSCN0079.jpg -> 51c4b71ec800c
07-18 21:34:15:709 csync_walker: file: D:\Eigene Dateien\OwnCloud/Bilder_Share/Paul/0292_DSCN0079.jpg
07-18 21:34:15:709 _csync_detect_update: ==> file: Bilder_Share/Paul/0292_DSCN0079.jpg - hash 3761590186975079667, mtime: 1342910702
07-18 21:34:15:709 _csync_detect_update: Database entry found, compare: 1342910702 <-> 1342910702, md5: 51c4b71ec800c <-> 51c4b71ec800c
07-18 21:34:15:709 _csync_detect_update: file: Bilder_Share/Paul/0292_DSCN0079.jpg, instruction: INSTRUCTION_NONE <<=

07-18 21:34:45:159 _csync_merge_algorithm_visitor: INSTRUCTION_NONE file: Bilder_Share/Paul/0292_DSCN0079.jpg

07-18 21:54:25:580 oc_module: => open called for ownclouds://owncloud.xxxxxx.yy/remote.php/webdav/clientsync/Bilder_Share/Paul/0292_DSCN0079.jpg
07-18 21:54:25:580 oc_module: GET request on /remote.php/webdav/clientsync/Bilder_Share/Paul/0292_DSCN0079.jpg
07-18 21:54:25:610 oc_module: Sendfile handling request type GET.
07-18 21:54:25:610 oc_module: -- GET on ownclouds://owncloud.xxxxxx.yy/remote.php/webdav/clientsync/Bilder_Share/Paul/0292_DSCN0079.jpg
07-18 21:54:25:750 oc_module: Content encoding ist with status 200
07-18 21:54:34:610 oc_module: GET http result 200 (OK)
07-18 21:54:34:610 oc_module: http request all cool, result code 200
07-18 21:54:34:640 oc_module: Get file ID for ownclouds://owncloud.xxxxxx.yy/remote.php/webdav/clientsync/Bilder_Share/Paul/0292_DSCN0079.jpg: 51e2f8be22c75
07-18 21:54:34:640 _get_md5: MD5 for ownclouds://owncloud.xxxxxx.yy/remote.php/webdav/clientsync/Bilder_Share/Paul/0292_DSCN0079.jpg: 51e2f8be22c75
07-18 21:54:34:640 _csync_push_file: FINAL MD5: 51e2f8be22c75
07-18 21:54:34:640 _csync_push_file: PUSHED file: D:\Eigene Dateien\OwnCloud/Bilder_Share/Paul/0292_DSCN0079.jpg

[notDavid]

I have the same problem on OS X 10.8.4, using ownCloud-1.3.0-nightly20130716.dmg.

If you want log files, tell me which and where i can find them...

[Hedders]

I have the same issue with 1.4beta1 on Windows 7 64 bit. Happy to provide logs or any other information as requested.

[mhofer117]

Same here with 1.4beta1 on Win8 64bit. Server is running 5.0.9.
Can also provide any logs needed

[ogoffart]

What is called MD5 in the log is not really an MD5, it is an etag.

The etag may have changed on the server. Or was not properly saved on the local db. The full log would be required to see what is hapenning.

[artem-sidorenko]

Same with 1.4.0 release on Win7 64bit, server 5.0.10

[dragotin]

Please somebody could provide a logfile? See http://doc.owncloud.org/desktop/1.4/troubleshooting.html on how to do that.

[artem-sidorenko]

I reproduced the issue with 3 files and tested it with client 1.3.0 on Win7 64bit, server 5.0.10. Works fine without to upload everything all the time.
Here is the log for 1.4.0 client on win7 64 bit, client is uploading all the time with shorts breaks in between: https://gist.github.com/artem-sidorenko/2d356a9ae5143d380d57/raw/b587cf70fa6e72068fbdbd6af82d4db53619204e/gistfile1.txt

[etiess]

I have a slightly different issue here. Step to reproduce:

• setup a sync with a local folder
• upload files, sync is OK
• delete this setup
• setup the same sync again
• it will download ALL the files!

Is it a normal behavior? With other sync client (Dropbox, SugarSync, ...), it just verifies that the files are the same, and it does not download all files again.

It's a problem for me: for a large amount of data, how can I copy all my files to a new computer, and then setup the sync client, and then launch the sync with the files already present locally?

Should I start a new subject?

Thank you and congratulation to the team!
(config: 5.0.11 on Ubuntu Server 13.04, client 1.4.0 on Windows 8)

Etienne

[danimo]

@etiess ownCloud does not have hash sums (architectural decision, because there are backends not supporting it), so the only way to verify equality is to download and compare. We can optimize this later on, but for now it is what it is.

[etiess]

@danimo Thank you for your quick answer!

I live and work in Africa: Internet access are slow, and most of the time data plans are limited in download. So it's very expensive to resync a folder on a new computer with some GB of data. With Dropbox, we just copy the files on an USB stick, and copy them to the new computer before syncronizing it.

With your message, I understand that there is no way to have a similar result with Owncloud. Is it correct?

It would be a bad news for us: owncloud was perfect for small african organization, but unfortunately we cannot download everything each time :-(

Thank you!

Etienne

[zatricky]

Thank you for the feedback, @danimo.

Hash-based checksums are probably supported on your toaster, thus I'm having a hard time understanding that decision - of course things change and that could simply have been a sensible decision at the time.

Personally I would have designed the architecture around hashes from the beginning. Unfortunately I'm an Engineer, not a Dev. ;)

[artem-sidorenko]

I think there are different issues here we are talking about.
One topic is about downloading/uploading of entire folder after client update/installation, so its about the missing checksums on the server side as well.
Another topic is about the permanent upload of all files with a new client on the windows platform (and only on windows)

[artem-sidorenko]

This issue is still with 1.4.1rc2 on win7 64bit there
Client log: https://gist.github.com/artem-sidorenko/8c97b84104b2a748a4be

[dragotin]

ok, @artem-sidorenko your logfile is pointing to something. It shows that the file is downloaded again because csync detected that the inode changed:

09-24 16:15:45:022 _csync_detect_update: Database entry found, compare: 1377027030 <-> 1377027030, md5: 52419e52ec037 <-> 52419e52ec037, inode: 17840455840 <-> 660586656
09-24 16:15:45:022 _csync_detect_update: file: test1/Linux-Ubuntu-DFN-Anleitung-WPA2.pdf, instruction: INSTRUCTION_EVAL <<=

while etag and mtime are the same. We additionally added the inode to the comparison for 1.4 because of this reason explained by @ogoffart:

[16:29] <ogoffart> echo a > a.txt;  echo b > b.txt
[16:29] <ogoffart> let it sync
[16:29] <ogoffart> a and b have the same modtime
[16:30] <ogoffart> then   rm a.txt;  mv  b.txt a.txt
[16:30] <dragotin> oh yeah!

But in this case, it looks like the inode changed unexpectedly. Is this a network mounted file system?

[artem-sidorenko]

@dragotin its not downloaded, it gets uploaded all the time

no, its a local NTFS on Windows 7

I don't know how csync gets the inode information on Windows, probably here is something wrong..

[dragotin]

@artem-sidorenko you have a local NTFS as drive R: ? Wow... Well, the "inode" for win32 is queried by win32 function GetFileInformationByHandle, which returns a FileIndex number. But that does not seem to be reliable enough...

[artem-sidorenko]

@dragotin yes I do, I have a lot of drives;)
Is there something how I can assist/help with troubleshooting here?
Imho its not a nice bug and it blocks me to use 1.4 client on windows completely..

[notDavid]

Ok so suddenly out of the blue the owncloud client starts syncing unchanged files, i'm running client 1.4.0 final on OS X 10.8.5. The server is running 5.0.11. In case it's relevant, i should note that the server owncloud data directory is encrypted(mounted) with encfs.
I dug through the log file, these are the relevant parts (i cut out all the parts referring to 1 specific file): https://gist.github.com/DavidStaron/6691614

[etiess]

Hello,
I'm experiencing the same: I started the sync of a large amount of pictures (9GB, 1400 files). Once everything was uploaded, the client decided to download 3.6 GB of these pictures back to my computer.
Log file of the client is here: https://www.sugarsync.com/pf/D6476655_61894308_905103

Config: Server 5.0.11 on Ubuntu Server 12.04 (up-to-date), Mirall 1.4.0 on Windows 8 64

Extract of this log for a picture (SAM_7057.JPG) which had been uploaded from my computer, and which is -after that - downloaded from the server:

09-25 17:48:24:726 oc_module: ** Finished download ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_6789.SRW
09-25 17:48:24:730 oc_module: Get file ID for ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_6789.SRW: 5240025369b76
09-25 17:48:24:730 _get_md5: MD5 for ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_6789.SRW: 5240025369b76
09-25 17:48:24:730 _csync_push_file: FINAL MD5: 5240025369b76
09-25 17:48:24:730 _csync_push_file: PUSHED file: D:\Photos/2013.04.28 Iringa/SAM_6789.SRW
09-25 17:48:24:730 oc_module: => open called for ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_7057.JPG
09-25 17:48:24:730 oc_module: GET request on /remote.php/webdav/Photos/2013.04.28%20Iringa/SAM_7057.JPG
09-25 17:48:24:731 oc_module: Sendfile handling request type GET. fd 3
09-25 17:48:24:731 oc_module: -- GET on ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_7057.JPG
09-25 17:48:24:781 * Compare etag with previous etag: false
09-25 17:48:24:905 * Compare etag with previous etag: true
09-25 17:48:24:906 Schedule folder "Bureau" to sync!
09-25 17:48:24:907 II> Sync for folder "Bureau" already scheduled, do not enqueue!
09-25 17:48:24:907 Currently folder "Photos" is running, wait for finish!
09-25 17:48:25:036 * Compare etag with previous etag: false
09-25 17:48:25:174 * Compare etag with previous etag: false
09-25 17:48:25:342 oc_module: Content encoding ist with status 200
09-25 17:48:38:934 * Polling "Bureau" for changes. (time since next sync: 360 s)
09-25 17:48:38:936 * Force Sync now
09-25 17:48:38:938 Schedule folder "Bureau" to sync!
09-25 17:48:38:940 II> Sync for folder "Bureau" already scheduled, do not enqueue!
09-25 17:48:38:942 Currently folder "Photos" is running, wait for finish!
09-25 17:48:42:242 * Polling "Atestoc2" for changes. (time since next sync: 390 s)
09-25 17:48:42:244 * Force Sync now
09-25 17:48:42:246 Schedule folder "Atestoc2" to sync!
09-25 17:48:42:247 II> Sync for folder "Atestoc2" already scheduled, do not enqueue!
09-25 17:48:42:248 Currently folder "Photos" is running, wait for finish!
09-25 17:48:48:522 * Compare etag with previous etag: false
09-25 17:48:48:598 oc_module: GET http result 200 (OK)
09-25 17:48:48:598 oc_module: http request all cool, result code 200
09-25 17:48:48:598 oc_module: ** Finished download ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_7057.JPG
09-25 17:48:48:602 oc_module: Get file ID for ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_7057.JPG: 524002536f2f1
09-25 17:48:48:602 _get_md5: MD5 for ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_7057.JPG: 524002536f2f1
09-25 17:48:48:602 _csync_push_file: FINAL MD5: 524002536f2f1
09-25 17:48:48:602 _csync_push_file: PUSHED file: D:\Photos/2013.04.28 Iringa/SAM_7057.JPG
09-25 17:48:48:603 oc_module: => open called for ownclouds://XXXX/remote.php/webdav/Photos/2013.04.28 Iringa/SAM_6838.JPG

[dragotin]

ok, I disabled the inode check with commit f1b4a7a in csync dav branch. With that the cases @artem-sidorenko and also @etiess experienced should not longer happen. The fix will be released with version 1.4.1

I will close this bug. If this issue happens again, please open a new one.

[zatricky]

@dragotin: Is the inode check not necessary? Given your relayed explanation from @ogoffart, I'd think it is actually very important to keep. I'd suggest one of two processes:
a) leave this closed but ensure there will be a future fix to implement it properly (given the discussion and number of bug reports regarding hashing, this might already be satisfied?)
b) reopen and find an alternative fix other than removing the inode check

[dragotin]

On 26.09.2013 09:50, zatricky wrote:

@dragotin: Is the inode check not necessary? Given your relayed explanation from @ogoffart, I'd think it is actually very important to keep. I'd suggest one of two processes:
a) leave this closed but ensure there will be a future fix to implement it properly (given the discussion and number of bug reports regarding hashing, this might already be satisfied?)
b) reopen and find an alternative fix other than removing the inode check

I agree we have to find a solution to the problem described by @ogoffart.

[etiess]

Hello,

Thank you for your release of 1.4.1

As mentioned before, I had the problem with 1.4.0. It wanted to re-download 3.6 GB of data. So I killed Mirall before it downloads the 3.6 GB.

Today I installed Mirall 1.4.1, by uninstalling the previous version (1.4.0), but I did not delete the data folder as proposed in the installation process.

The issue is still there: it wants to redownlad the 3.6 GB. The log is below, as an example for the picture SAM_0376.JPG

Do I have to initialize something? Thanks!

09-28 01:52:15:865 Folder in overallStatus Message: Mirall::Folder(0x50d47f0) with name "Photos"
09-28 01:52:15:868 Sync state changed for folder "Photos" : "Sync Running"
09-28 01:52:11:735 csync_propagate: Propagation for local replica took 0.00 seconds visiting 1331 files.
09-28 01:52:11:736 oc_module: => open called for ownclouds://XXXX/remote.php/webdav/Photos/EGG Party/SAM_0376.JPG
09-28 01:52:11:736 oc_module: GET request on /remote.php/webdav/Photos/EGG%20Party/SAM_0376.JPG
09-28 01:52:11:738 oc_module: Sendfile handling request type GET. fd 3
09-28 01:52:11:738 oc_module: -- GET on ownclouds://XXXX/remote.php/webdav/Photos/EGG Party/SAM_0376.JPG
09-28 01:52:11:986 oc_module: Content encoding ist with status 200

[zatricky]

Hey, @etiess.

This issue is specifically referring to changed inode references causing a re-sync, the antithesis of #201.

I'll be following #994 and #110 (along with the corresponding issue at /owncloud/core/issues/523) closely myself since it is also preventing me from utilising ownCloud fully.

[etiess]

Hello,

Yes, I'm following all these issues too.

Something I don't understand is the commercial version of owncloud. How is it possible to have a viable commercial offer with such important issues? For me, owncloud is far from a product I could put in production in a company. Am I missing something? Are these bugs related only to the last versions of OC?

And concerning all the issues you referenced, I have the feeling that the core component of owncloud (the "sync" !) is really not working properly (as far as I understand, mostly because of the absence of hash sum). While a lot of additional functionalities are worked out (music player, calendar, sharing...).

I quote @mkhpalm in #994

"Updated to 1.4.1 against 5.0.11 and its still re-downloading random blocks of files (6.1 gigs worth right now) on each of my clients. [...] We can't afford the traffic that results from something like this and its going to get people in trouble with their ISPs if they're downloading hundreds of gigs of data every week or so."

The idea of owncloud is amazing, and congratulation to the development team. But for the moment, I'm quite disoriented, to know if I can trust the program with my personal data or not. What is sure, is that the software is not usable for the moment.