Files over a certain size fail to upload to TLS enabled servers. #59
Comments
willgraf
changed the title
|
I tried recreating this bug with the
Everything worked fine; the file uploaded and processed in a little under 10 minutes. |
willgraf
changed the title
|
I'm having trouble getting this to consistently happen. It definitely seems to be related to TLS, as I cannot get this to occur over HTTP. However, occasionally (maybe 10% of the time) the request just works instead of getting the above error. |
|
This open treq issue sounds related: POST with TLS causes double-close on body producer |
|
We're encountering the same issue in angelolab/ark-analysis/issues/297. Ideally we'd like to be able to have our users create a single zip file with all of their image data and upload it one go, as this would be much simpler than requiring multiple distinct uploads and downloads |
|
It looks like this issue might be solved in twisted's next release. Well, after trying to install the above version manually and testing with a large zip file, the issue is NOT resolved. |
|
We're now running into this issue for a 23.3 MB zip file. The verdict is that they didn't actually fix the problem in the newest release? |
|
Right, the issue linked above is said to be closed by the new release but we are still seeing uploads get rejected. The upload DOES seem to work eventually, so maybe we can just use some retries to fix this? |
|
I upgraded to Uploading a small file, then a large file. The large file fails but the small file passes and prints out the correct response. import logging, os, sys, tempfile
import numpy as np
import tifffile
import treq
from twisted.internet import reactor, defer
def save_image(path, size):
x = np.random.random(size).astype('float32')
tifffile.imsave(path, x)
@defer.inlineCallbacks
def upload_file(host, filepath):
with open(filepath, 'rb') as f:
request = treq.post(HOST, files={'file': (filepath, f)})
response = yield request
json_content = yield response.json()
print(json_content)
if __name__ == '__main__':
logger = logging.getLogger()
console = logging.StreamHandler(stream=sys.stdout)
console.setLevel(logging.DEBUG)
logger.addHandler(console)
with tempfile.TemporaryDirectory() as temp_dir:
path = os.path.join(temp_dir, 'bigTestImage.tiff')
save_image(path, (2, 10000, 10000))
upload_file('https://httpbin.org/post', path)
reactor.run()$ python test.py
Unhandled Error
Traceback (most recent call last):
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/python/log.py", line 103, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/python/log.py", line 86, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite
why = getattr(selectable, method)()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/tcp.py", line 243, in doRead
return self._dataReceived(data)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/endpoints.py", line 132, in dataReceived
return self._wrappedProtocol.dataReceived(data)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/protocols/tls.py", line 325, in dataReceived
self._unbufferPendingWrites()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/protocols/tls.py", line 498, in _unbufferPendingWrites
self._producer.resumeProducing()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/protocols/tls.py", line 101, in resumeProducing
self._producer.resumeProducing()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/treq/multipart.py", line 117, in resumeProducing
self._currentProducer.resumeProducing()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/web/client.py", line 1217, in resumeProducing
self._task.resume()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/task.py", line 460, in resume
raise NotPaused()
twisted.internet.task.NotPaused:
Unexpected exception from treq.multipart.MultiPartProducer.stopProducing
Traceback (most recent call last):
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/protocols/policies.py", line 125, in connectionLost
self.wrappedProtocol.connectionLost(reason)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1050, in dispatcher
return func(*args, **kwargs)
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1734, in _connectionLost_TRANSMITTING
self._currentRequest.stopWriting()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/web/_newclient.py", line 951, in stopWriting
_callAppFunction(self.bodyProducer.stopProducing)
--- <exception caught here> ---
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/web/_newclient.py", line 197, in _callAppFunction
function()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/treq/multipart.py", line 91, in stopProducing
self._currentProducer.stopProducing()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/web/client.py", line 1161, in stopProducing
self._task.stop()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/task.py", line 497, in stop
self._checkFinish()
File "/vanvalenlab/kiosk-client/env/lib/python3.7/site-packages/twisted/internet/task.py", line 507, in _checkFinish
raise self._completionState
twisted.internet.task.TaskStopped: |
Uploading a ~10MB zip file works, while a ~150MB fails with the following error:
The
kiosk-frontendis configured to allow file uploads of over 1GB, so I believe the error is on the client side. This may be ALL files or only.zipfiles, I am not sure (I don't have any image files that are 150MB+).The text was updated successfully, but these errors were encountered: