In test_config.py, moto doesn't play nicely with s3fs

[JackKelly]

Describe the bug
test_config.test_save_to_aws() currently fails in PR #152 (so I've marked it as a test to skip for now).

As far as I can tell, this is a known incompatibility between moto and aiobotocore. See these issues:

• aiobotocore expects raw_headers - missing from batch job description response getmoto/moto#2706
• support moto client wrappers aio-libs/aiobotocore#755 (this has some suggestions for possible work-arounds)

Here's the raw error log:

(nowcasting_dataset) jack@leonardo:~/dev/ocf/nowcasting_dataset$ py.test -s tests/config/test_config.py 
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/jack/dev/ocf/nowcasting_dataset
plugins: cov-2.12.1
collected 7 items                                                                                                                                                          

tests/config/test_config.py ...sFs.

================================================================================= FAILURES =================================================================================
_____________________________________________________________________________ test_save_to_aws _____________________________________________________________________________

    @moto.mock_s3()
    def test_save_to_aws():
        """
        Check that configuration can be saved to gcs
        """
    
        bucket_name = "test_bucket"
    
        s3_client = boto3.client("s3")
        s3_client.create_bucket(Bucket=bucket_name)
    
>       save_yaml_configuration(
            configuration=Configuration(),
            filename=f's3://{bucket_name}/test_config.yaml')

tests/config/test_config.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nowcasting_dataset/config/save.py:29: in save_yaml_configuration
    with fsspec.open(filename, "w") as yaml_file:
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/fsspec/core.py:452: in open
    return open_files(
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/fsspec/core.py:297: in open_files
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/fsspec/core.py:297: in <listcomp>
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/fsspec/asyn.py:91: in wrapper
    return sync(self.loop, func, *args, **kwargs)
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/fsspec/asyn.py:71: in sync
    raise return_result
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/fsspec/asyn.py:25: in _runner
    result[0] = await coro
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/s3fs/core.py:731: in _makedirs
    await self._mkdir(path, create_parents=True)
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/s3fs/core.py:716: in _mkdir
    await self._call_s3("create_bucket", **params)
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/s3fs/core.py:268: in _call_s3
    raise err
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/s3fs/core.py:248: in _call_s3
    out = await method(**additional_kwargs)
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/aiobotocore/client.py:141: in _make_api_call
    http, parsed_response = await self._make_request(
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/aiobotocore/client.py:161: in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/aiobotocore/endpoint.py:79: in _send_request
    success_response, exception = await self._get_response(
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/aiobotocore/endpoint.py:112: in _get_response
    success_response, exception = await self._do_get_response(
../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/aiobotocore/endpoint.py:162: in _do_get_response
    response_dict = await convert_to_response_dict(http_response,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

http_response = <botocore.awsrequest.AWSResponse object at 0x7f5e851080a0>, operation_model = OperationModel(name=CreateBucket)

    async def convert_to_response_dict(http_response, operation_model):
        """Convert an HTTP response object to a request dict.
    
        This converts the requests library's HTTP response object to
        a dictionary.
    
        :type http_response: botocore.vendored.requests.model.Response
        :param http_response: The HTTP response from an AWS service request.
    
        :rtype: dict
        :return: A response dictionary which will contain the following keys:
            * headers (dict)
            * status_code (int)
            * body (string or file-like object)
    
        """
        response_dict = {
            # botocore converts keys to str, so make sure that they are in
            # the expected case. See detailed discussion here:
            # https://github.com/aio-libs/aiobotocore/pull/116
            # aiohttp's CIMultiDict camel cases the headers :(
            'headers': HTTPHeaderDict(
                {k.decode('utf-8').lower(): v.decode('utf-8')
>                for k, v in http_response.raw_headers}),
            'status_code': http_response.status_code,
            'context': {
                'operation_name': operation_model.name,
            }
        }
E       AttributeError: 'AWSResponse' object has no attribute 'raw_headers'

../../../miniconda3/envs/nowcasting_dataset/lib/python3.8/site-packages/aiobotocore/endpoint.py:40: AttributeError

[peterdudfield]

Can you post what 'save_yaml_configuration' has been changed too, and then I can always have a play?

[JackKelly]

Sure! Thanks! Sorry, I should've linked to the PR in the text for this issue! The code is in PR #152

[peterdudfield]

Now we have moved to fsspec I dont think this is needed

[JackKelly]

Cool beans, thanks, I'll close this issue for now. Please re-open if I misunderstood!