Reusable Initialization of aiobotocore S3 Client

[all4one-max]

I'm in the process of migrating our Python package, used by various applications within our organization, from boto3 to aiobotocore. In our existing implementation with boto3, we ensure singleton initialization of the S3 client to optimize resource usage and avoid redundant client creation.

Existing Implementation with boto3:

  class AWSMeta(type):
	  _instances: Dict[Any, Any] = {}
  
	  def __call__(cls) -> Any:
		  if cls not in cls._instances:
			  instance = super().__call__()
			  cls._instances[cls] = instance
		  return cls._instances[cls]


  class S3ClientSingleton(metaclass=AWSMeta):
	  _s3_client = None
  
	  @classmethod
	  def get_s3_client(cls) -> Any:
		  if not cls._s3_client:
			  # Lazy initialization of the S3 client
			  cls._s3_client = boto3.client(
				  "s3",
				  aws_access_key_id=AWS_ACCESS_KEY,
				  aws_secret_access_key=AWS_ACCESS_KEY_SECRET,
				  region_name="us-east-1",
			  )
		  return cls._s3_client

However, transitioning to aiobotocore poses a challenge as it requires using context managers for client creation, and the client doesn't exist outside the context. I aim to encapsulate this singleton logic within our package itself, rather than relying on configuration in the application lifecycle.

I've explored several sources, including the aiobotocore documentation and relevant discussions on GitHub and Stack Overflow, but haven't found a satisfactory solution yet.

Sources referred:

1. https://github.com/aio-libs/aiobotocore
2. Feature: Get a long-lived client object #928
3. https://stackoverflow.com/questions/77095898/reuse-create-client-in-aiobotocore-for-better-performences (no one answered here but more or less the use case but I have to do this inside my package logic itself)

[thehesiod]

so if you're using boto3 you should look into aioboto3. Given it's a singleton you don't need to worry about teardown so you can just do .create_client(...).__aenter__() or use AsyncExitStack if you want it slightly cleaner

[all4one-max]

I was thinking of using aiobotocore and have went through their documentation, also the links shared in reference. It mentions to use context manager but that would mean it won't be long lived. I did read something about setting things up in startup and close event on fastAPI server, but not sure what would that mean. Also why do you say no need to worry about teardown? Further if you could share a possible snippet where singleton instance of aiobotocore/aioboto3 client is created it would be great.

[all4one-max]

Hey!
Would be great if you could help here? Not sure how to proceed?

[all4one-max]

@thehesiod is this correct way to do, I didn't do any teardown or close as you mentioned.

class AsyncS3ClientSingleton(metaclass=AWSMeta):
    _async_s3_client = None

    @classmethod
    async def get_s3_client(cls) -> Any:
        if not cls._async_s3_client:
            # Lazy initialization of the S3 client
            session = aioboto3.Session()
            ctx = session.client(
                "s3",
                aws_access_key_id=AWS_ACCESS_KEY,
                aws_secret_access_key=AWS_ACCESS_KEY_SECRET,
                region_name="us-east-1",
            )
            cls._async_s3_client = await ctx.__aenter__()
        return cls._async_s3_client

[all4one-max]

@thehesiod the above is not working for me. I am using it in my celery tasks, the first time operations work fine, but after that I am getting Runtime Error, Event loop is closed.

But when I create the session and client for every operation as shown below it works fine, though it defeats the purpose of reusing the client.

        session = aioboto3.Session()
        async with session.client(
            "s3",
            aws_access_key_id=AWS_ACCESS_KEY,
            aws_secret_access_key=AWS_ACCESS_KEY_SECRET,
            region_name="us-east-1",
        ) as async_s3_client:
            await async_s3_client.upload_file(file_name, bucket, key)

Can you please help me in fixing the event loop is closed error

[thehesiod]

could you give a full example? it's hard to tell what's going on with this snippet