Downloads from ODP fail due to time-outs

[forman]

Expected behavior

Dataset downloads from ODP should be complete and finish in time (given a maximum observed download speed of 3 MB/sec).

Actual behavior

Now after using newly deployed ODP services we often encounter download failures due to time outs.

Steps to reproduce the problem

1. Open Cate Desktop
2. Select fiirst AOD data source from list esacci.AEROSOL.satellite-orbit-frequency.L2P.AOD.MERIS.Envisat.MERIS_ENVISAT.2-2.r1
3. Press Download, use time range 2008-01-01, 2008-01-05
4. Observe download task for 50 files to be downloaded

After ca 10 secs the download rate drops from 2 MB/sec to 0.4 MB/sec;

Finally the it drops to zero ending in a time-out.

Specifications

Cate 2.0.0.dev25

Traceback

Traceback (most recent call last):
  File "D:\Projects\PycharmProjects\cate\cate\ds\esa_cci_odp.py", line 987, in _make_local
    urllib.request.urlretrieve(url[protocol], filename=dataset_file, reporthook=reporthook)
  File "D:\Miniconda3\envs\cate-env\lib\urllib\request.py", line 277, in urlretrieve
    block = fp.read(bs)
  File "D:\Miniconda3\envs\cate-env\lib\http\client.py", line 449, in read
    n = self.readinto(b)
  File "D:\Miniconda3\envs\cate-env\lib\http\client.py", line 493, in readinto
    n = self.fp.readinto(b)
  File "D:\Miniconda3\envs\cate-env\lib\socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "D:\Miniconda3\envs\cate-env\lib\ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Miniconda3\envs\cate-env\lib\ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "D:\Miniconda3\envs\cate-env\lib\ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Projects\PycharmProjects\cate\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result
    result = future.result()
  File "D:\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "D:\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "D:\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "D:\Projects\PycharmProjects\cate\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\webapi\websocket.py", line 295, in set_workspace_resource
    monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\core\wsmanag.py", line 320, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\core\workspace.py", line 662, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\core\workflow.py", line 627, in invoke_steps
    steps[0].invoke(context=context, monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\core\workflow.py", line 318, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\core\workflow.py", line 980, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "D:\Projects\PycharmProjects\cate\cate\core\op.py", line 216, in __call__
    return_value = self._wrapped_op(**input_values)
  File "D:\Projects\PycharmProjects\cate\cate\ops\io.py", line 83, in open_dataset
    monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\core\ds.py", line 670, in open_dataset
    monitor=monitor.child(80))
  File "D:\Projects\PycharmProjects\cate\cate\ds\esa_cci_odp.py", line 1065, in make_local
    raise e
  File "D:\Projects\PycharmProjects\cate\cate\ds\esa_cci_odp.py", line 1058, in make_local
    self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
  File "D:\Projects\PycharmProjects\cate\cate\ds\esa_cci_odp.py", line 1001, in _make_local
    error_cls=NetworkError) from e
cate.core.ds.NetworkError: Failed to synchronize data source "esacci.AEROSOL.satellite-orbit-frequency.L2P.AOD.MERIS.Envisat.MERIS_ENVISAT.2-2.r1" for given time range: The read operation timed out

[forman]

This happened after we configured Cate to use the new ODP services (regression).