File Upload fails sometimes for some files, App 2.0.1, OC 9.0.3 #1737
Comments
|
Mmmm, could be a problem with the redirection of uploads... |
|
In the fails, I see a single 408 response. The rest of the redirected PUTs are not followed by any other successful PUT. Could be stalled connections, or similar. Are you using other clients (desktop, iOS)? No problem with them? What about the web interface, no problem uploading? |
|
I use Linux desktop sync. This works perfectly. It's strange, it may fail even if I am at home in the sam local area network, where the server is. |
|
The problem is mostly with movies, less with images. So it seems to be a matter of file size. What I see now: In previous versions, the upload seemed to be very stable, means, that if the connection breaks, it was automatically restarted later. All uploads were successful, as fas as I have seen. This was (a) in versions of the client prior to march 2016 and in owncloud 8.x. In march, I had a server crash and since I have set it up newly, I am using owncloud 9.x and since then I have these troubles. But the setup is now different too: It used to be apache for the reverese proxy, now it is nginx and all within docker containers. But it is not a completely false configuration, since after many many retries, the videos are uploaded too, so let's say, after 10-20 fails, it is successful once. Something else I noticed: Very often I see that at the end of the upload, when everything seems to have been successful, the upload-bar reaches 100% in the client, then still the message is «failed». So it seems that the final «success» is not received in the client. Probably correlated to this: When it should upload let's say «MOV_0149.mp4», I get a failure status, then later I have a failure status of «MOV_0149 (2).mp4», then «MOV_0149 (2) (2).mp4», and so on, actually in this example, it is trying to upload «MOV_0149 (2) (2) (2) (2) (2).mp4» (this is a real current example). And when all this was successful, I have the same movie on the server five times, diff tells me, that the files are identical! So the upload often works, but the client does not know this fact. Any idea? |
|
Here a screenshot how it looks like:
|
That could mean that something is failing in the server while putting all the chunks together. But it doesn't fit the first logs, where the 408 error appears in the first chunk of five. Do you have more recent logs? The change to Nginx is a suspect also; it has often brought extra problems with its configuration. When you upload files from the desktop client or the web interface, does the Android app refresh correctly? Can you see them appear in the app? The retries with different suffix number mean that the app is detecting the existence of the file in the server. |
|
Mmm, probably related to #1768 . |
My problem is, that it is difficult to map an exact behaviour to a specific log, because sometime it works, sometimes not. Upload of long files lasts long, so it is difficult to stay there all time an watch what happens when. Of course I can create new logs if you can name a scenario that I should try.
Well, setting up a generic nginx reverse proxy for docker wasn't an easy task. ;) Do you have some recommended settings for a nginx reverse proxy? But then: After a lot of retries, it works, and it generates all this «… (2) (2) …» stuff, so probably the root cause of my problems is that the client does not get a successful result, when the upload was in fact successful?
Yes. The OwnCloud app seems to work, except for the uploads.
Yes, but it is 100% the same file from exactly the same app, that was previously successfully uploaded, but is still in the list of failed uploads, so in a retry, it creates a copy. The problem here seems to be that the app does not recognize all of its own successful uploads.
This seems possible to me. in that case, the upload was successful, assembly is successful, but the app does not get notice of it thinks that the assembly failed. That would explain the behaviour. |
|
Another point here is, that failed uploads are not retried automatically. Did the behaviour change regarding this point? On a mobile phone, the connection is not always good enough and some failures during upload are absolutely normal. For me it is important, that I can fully trust the upload function, that means, it should make sure, all media arrives at the server as soon as possible, without any manual intervention. I also run this app on the phones of my family members, and I can't tell them, that they should go to the app and retry broken uploads manually. The danger is too high, that they accidentally remove the current failure list, or that they simply forget it. Previously, my impression was, that the upload was stable. So has there been a change in this regard? So, do you know anything about this? Eventually I could open another ticket for this aspect of my problem. |
I'm sorry, but I don't. The only specific documentation I know about Nginx is https://doc.owncloud.org/server/9.1/admin_manual/installation/nginx_owncloud_9x.html, but there's nothing about using it as a reverse proxy AFAIK. I think we should try first to increase the timeout for the last chunk before digging other options.
There were some changes in how the detection of WiFi recovery is done but there is plenty of work to do with automatic retries. This feature was never developed in depth; automatic retries are only done with instant uploads of pictures, nothing else, and we need to look further and apply automatic "resyncs" for any kind of failed transfer that could be recovered without user intervention. Please, don't open a new issue about that topic. There are several already in the repo. For instance, #349. |
|
Probably it's nothing with nginx anyway, but with the chunk-assembly. I hope you agree, that changing the timeout for the last put cannot be the final solution? I suggest the following:
That means, the client should keep track of the status of its own uploads on the server. |
|
That would be ideal, and its in our radar, but would require more time that what we'll have for release 2.2.0. As an extra problem: there is no way to ask to the server about the state of the assembly. |
|
Ok, please keep it on the radar. Regarding the increase of timeout: This is client side and will be implemented in the next release, correct? It is not an nginx / apache configuration on the server? |
|
Yes, yes. And it's an easy sneak. I just added the milestone 2.2.0 to #1768 because your original logs don't match what I'd expect of this error, just in case there's something else to fix for your case. But I hope same solution works for both issues. |
|
@mwaeckerlin , could you try the test APK in the link below to see if the fix works for you? It will appear in your device as a separate ownCloudTest app, not conflicting with current ownCloud app. https://github.com/owncloud/android/raw/upload_bigfiles/ownCloudTest.apk |
Actual behaviour
Configuration
OwnCloud is configured with a base path /owncloud using docker image mwaeckerlin/owncloud:
In front of OwnCloud, there is a Nginx reverse proxy, that does SSL (with a valid certificate), and redirects http to https, using docker image mwaeckerlin/reverse-proxy, the relevant part here is:
I configures instant upload, also from mobile.
What Happens
Some files succeed, some files fail. I don't see any major difference between those files. Sometimes files do upload after some numbers of retrials, some did not (yet), even so I retried many many times.
Upload bar goes to 100%, then fails.
Uploads may fail regardless whether they are triggered by the instant upload or manually by using «share» with the OwnCloud app.
Expected behaviour
All files always succeed.
Steps to reproduce
Can this problem be reproduced with the official owncloud server?
(url: https://demo.owncloud.org, user: test, password: test)
Since it is an occasional behaviour, I can't really test that.
Environment data
Android version: 5.1, 6.0.1,.5.1.1
Device model: Sony XPeria Z3 Compact, Xperia M5, Xperia Z2 tablet
Stock or customized system: -
ownCloud app version: 2.0.1
ownCloud server version: 9.0.3
Logs
Web server error log
Funny, upload just succeeded, when I wanted to get the logs for the failure. This after about 10-20 fails for the same file.
So here you see all the logs for the file where I replaced the original file name by filename and the external IP by 66.66.66.66. The first attempts were fails, the last attempt was successful:
On Nginx:
OwnCloud + Apache:
ownCloud log (data/owncloud.log)
Nothing special, at least the file name does not appear. Unfortunately, there is a lot of traffic, several desktop clients and Android DAV clients do their synchronisation tasks at the same time. So I need to filter the relevant parts and cannot just paste everything here.
The text was updated successfully, but these errors were encountered: