Moto integration

[akshara08]

Description of Change

PR adds a patch to endpoint.convert_to_response_dict such that it can work with moto while not changing the expected behavior of the method. I also added tests to replicate the problem and fix the issue.

Issue (#979)

The primary problem with moto integration is this method. There are 2 main issues with that method

1. It expects raw to have raw_headers just to convert the headers to lowercase.
2. Moto passes MockRawResponse which is not awaitable causing the breaks with lines containing await.

How to reproduce the error

# test_response.py

@pytest.fixture
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ["AWS_ACCESS_KEY_ID"] = "testing"
    os.environ["AWS_SECRET_ACCESS_KEY"] = "testing"
    os.environ["AWS_SECURITY_TOKEN"] = "testing"
    os.environ["AWS_SESSION_TOKEN"] = "testing"


@pytest.fixture
def s3(aws_credentials):
    with mock_s3():
        conn = boto3.resource("s3")
        conn.create_bucket(Bucket='testbucket')
        yield conn


def test_moto(s3):
    for i in range(3):
        s3.Bucket('testbucket').put_object(
            Key=f'glob_{i}.txt', Body=f"test glob file {i}"
        )
    path = 's3://testbucket/glob_*.txt'
    fs = s3fs.S3FileSystem()
    files = fs.glob(path) # Fails 
    assert len(files) == 3

Resolution

It's a two part solution
1. Remove the need for raw_headers and use the headers directly using botocore.utils.lowercase_dict

1. Since raw_headers are necessary, I've created a check to use headers if the httpresponse came from moto
2. Only await for awaitable objects. This check can be done using inspect.isawaitable

Assumptions

1. This patch was added only to convert to lowercase and this botocore util does the same
^Not valid anymore since raw_headers remain

1. Assigning non-awaitable objects to response_dict['body'] without await is not expected to break any current behavior

Alternative methods considered

I considered added a raw_headers to MockRawResponse but it seemed unnecessary especially if the point to convert to lowercase based on the comment

Checklist for All Submissions

• I have added change info to CHANGES.rst
• If this is resolving an issue (needed so future developers can determine if change is still necessary and under what conditions) (can be provided via link to issue with these details):
  • Detailed description of issue
  • Alternative methods considered (if any)
  • How issue is being resolved
  • How issue can be reproduced
• (NA) If this is providing a new feature (can be provided via link to issue with these details):
  • Detailed description of new feature
  • Why needed
  • Alternatives methods considered (if any)

Checklist when updating botocore and/or aiohttp versions

• I have read and followed CONTRIBUTING.rst
• I have updated test_patches.py where/if appropriate (also check if no changes necessary)
• I have ensured that the awscli/boto3 versions match the updated botocore version

‼️ Note ‼️

I've been getting botocore.exceptions.NoCredentialsError: Unable to locate credentials for some of the tests (5 to be exact). Ive got them even before I made changes, so I'm unsure if I'm breaking anything with this PR

[thehesiod]

this introduces a circular dependency, can't do it as it may later be incompatible with the version of aiobotocore this becomes

[akshara08]

Makes sense. Removed the dependency, used paginator for testing instead

[thehesiod]

it's important we use raw_headers as the non-raw doesn't preserve casing of the value

[akshara08]

Alright, the alternative is kind of an ugly patch. Let me know what ya think

[akshara08]

@thehesiod any update on this?

[thehesiod]

re NOTE in your description you need to run with -m moto to limit to moto marked tests, all other tests are only to be run with private key as moto didn't have support for them when tests were originally written (we should revisit these tests and mark what exactly is missing for the test).

[thehesiod]

this is more of add support for moto in-place mock methods

[thehesiod]

moto is not a dependency of this library, so you can't do this. I think this is really a bug in moto, it needs support for aiobotocore. If you were to do this it would need to be a try/except ImportError. Another option would be for aiobotocore to expose some moto helpers that correctly patch the response. I'd do that instead as it will be a lot cleaner.

[akshara08]

moto is not a dependency of this library, so you can't do this.

Ack! I missed that!
I'm a little busy this week but I'll try to sneak in some time to dig into moto repo and see if there's a cleaner solution to this. I tried sending encoded raw_headers here and it broke a bunch of tests. Let me try that again and see

[thehesiod]

no worries, thanks for working on this

[thehesiod]

sorry for delay, I think either moto needs to add async patches, or we ship like a moto_helpers.py which does the right thing.

[akshara08]

Closing this to avoid piling on irrelevant commits. See #1007