web3 async discussion continued

[dylanjw]

This issue supersedes #1055.

Ive summarized our discussion re the introduction of an async web3 api below, and expanded it a bit. After learning from experiments in #1088 we discussed pursuing the following design:

(1) Replace api methods with instances of a Method class, with a middleware like layer for formatting/normalizing/validating method input/output.
This will allow the procedural formatting code to be reused between async and sync api methods.

(2) Reduce middlewares to only those that make calls to the web3 api, moving the formatting middlewares to the Method input/output formatters.
Having fewer middlewares reduces maintenance overhead of duplicate async variants.

(3) Separate async and sync execution pathway through RequestManagers -> middlewares -> providers.
Async code needs to be purely async from the top api entry-point, all the way to the request. Sync code similarly needs to be purely sync. Converting async to sync code by running in a event loop incurred to much overhead.

First milestone: Minimum async call stack that gets the node version. No middlewares or other special features.

Any comments on this api?

import asyncio
from web3.providers import coro

aw3 = web3.Web3(coro.IPCProvider(), async=False) <-- Raises exception 
#  "TypeError, cant use async provider in a synchronous Web3 instance.")

aw3 = web3.Web3(coro.IPCProvider(), async=True)
asyncio.run(aw3.version.node())

Im leaning toward sharing the Web3 and RequestManager classes and having an async setting that will set the manager request method to the async execution path, and load the default async web3 modules. The providers (and eventually middlewares) will be set by the user, and will be checked to be async compatible.

TODO:

• Add a single async provider (coro.EthereumTesterProvider?).

• add async RequestManager execution pathway: request function -> middleware stack -> provider.

• Design and implement Method classes.

  I propose extracting the method configuration and method selection into a "MethodManager" object that will return either the async or sync callable. A modified __getattr__ will hand over control to the method_manager. Modules will simply compose the right pieces:

class AsyncVersion(Module, Meta=MetaModule):
    method_caller = AsyncMethod
    method_manager = MethodManager(VERSION_METHOD_CONFIG)

I may have to use a meta class to change __getattr__ for the class.

class MetaModule(type):
    @property
    def method_caller():
        raise NotImplementedError
    @property
    def method_manager():
        raise NotImplementedError
    def __getattr__(cls, attr):
        if attr not in cls.method_config:
            raise AttributeError
        return method_manager.get_method(attr)

• Method middleware manager. I think this needs to be more robust than a list. There is complexity with methods that select the json_rpc method string based on the input parameters. Some of these may need a different set of formatters for each selected method. Maybe this can be put off until a more featureful module is worked on. EDIT: This has ended up being the Method object, which is mainly concerned with how the method inputs are processed according to the method configuration.

• Implement async Version module.

• Implement async setting for Web3, to set async request function and default async modules.

• Check provider is async compatible.

• ?Implement sync Version module using Method class.? Implementing the above action items can be done while affecting the existing api as little as possible. However, it might be a good idea to demo how the existing sync modules will change.

[pacrob]

Closing, tracking async in Issue #1413