✨ Add AsyncIPCProvider #2984

DavidRomanovizc · 2023-06-07T10:20:33Z

What was wrong?

IPCProvider needs async version

Related to Issue #2973
Closes #1413

How was it fixed?

Add AsyncIPCProvider

Todo:

Add entry to the release notes

Cute Animal Picture

web3/providers/async_ipc.py

web3/_utils/threads.py

web3/providers/async_ipc.py

JustAnotherC0der · 2023-07-31T16:32:19Z

Pardon my ignorance to github work flow but is this implemented yet? Can't seem to import AsyncIPCProvider on newest web3

kclowes · 2023-07-31T16:37:34Z

@JustAnotherC0der It's not yet. We're still working on it.

kclowes · 2023-12-20T22:34:08Z

This is ready for review (finally)! I will add docs tomorrow, but wanted to get the code out there for feedback!

SheldonHolmgren · 2024-01-01T08:15:52Z

Thanks for your hard work. I will really appreciate this feature.

Any plans to make AsyncIPCProvider a PersistentConnectionProvider? I think without this it will be impossible to w3.eth.subscribe():


Web3ValidationError                       Traceback (most recent call last)
<ipython-input-12-2fc2efbc61b6> in <module>
----> 1 async with AsyncWeb3.persistent_websocket(AsyncIPCProvider("/mnt/ssd/optimism_data/geth.ipc")) as w3:
      2     await w3.eth.subscribe("newHeads")

~/dev/web3.py/web3/main.py in persistent_websocket(provider, middlewares, modules, external_modules, ens)
    516         a ``PersistentConnectionProvider`` instance.
    517         """
--> 518         return _PersistentConnectionWeb3(
    519             provider,
    520             middlewares,

~/dev/web3.py/web3/main.py in __init__(self, provider, middlewares, modules, external_modules, ens)
    541     ) -> None:
    542         if not isinstance(provider, PersistentConnectionProvider):
--> 543             raise Web3ValidationError(
    544                 "Provider must inherit from PersistentConnectionProvider class."

This is not a big deal since I can still use filters to get similar functionality.

pacrob

lgtm!

fselmo

Thanks for everyone's contribution here. I think this will be a great option to have. I left some initial comments. Mostly curious about the PersistentConnectionProvider question.

web3/tools/benchmark/main.py

web3/providers/async_ipc.py

SheldonHolmgren · 2024-02-13T07:23:01Z

This code won't work if the client receives multiple responses before it reads the first one. I have coded up a dirty fix for my own use. Perhaps it might be useful as a hint: DavidRomanovizc/web3.py@main...SheldonHolmgren:async_ipc_multiple_resps

kclowes · 2024-02-15T16:32:33Z

Thank you @SheldonHolmgren! I'll see what I can do.

kclowes · 2024-02-21T22:17:07Z

All right, I think this is ready again for another look. Thanks @fselmo for the boost!

fselmo

I caught one minor doc change we missed but that was all I saw (and I went ahead and pushed it). Thanks for all the work here @DavidRomanovizc, @kclowes, and @SheldonHolmgren! lgtm 👍🏼

@kclowes I'll let you merge this in when you are good with it. Maybe some more squashing if it makes sense too?

- Iron out some issues with rebasing; update async_ipc

- Time subscription tests correctly - Fix lint - Fix doc underline - Remove sleeps in test to test multiple responses coming in quick succession - Handle when multiple responses come in at once - Add test for subscription - timeout -> request_timeout Updates to naming: - Remove ws specific naming from manager methods - _ws_message_listener -> _message_listener - WebsocketConnection -> PersistentConnection - persistent_websocket -> persistent_connection

- Clean up ``AsyncIPCProvider``. Some logging c/p was left in the code. - Remove the need for the ``PersistentSocket`` class. - Move the ``silence_listener_task_exceptions`` setting to the base class, ``PersistentConnectionProvider``. - Use the suggestion put forth in the PR #2984 for parsing messages from the IPC socket more efficiently in case more than one message is received at once.

…file

- Update docs for persistent connection provider refactor Remove sleeps to emulate responses in rapid succession

pacrob · 2024-02-22T20:44:18Z

web3/manager.py


-    async def _ws_message_stream(self) -> AsyncGenerator[RPCResponse, None]:
+    async def _message_stream(self) -> AsyncGenerator[RPCResponse, None]:
        if not isinstance(self._provider, PersistentConnectionProvider):
            raise TypeError(
                "Only websocket providers that maintain an open, persistent connection "


It looks like we're removing references to websockets, and either just using "socket" or "persistent connection". Should that happen here too?

oop, yep. Nice catch!

pacrob · 2024-02-22T22:54:19Z

docs/providers.rst

@@ -261,7 +376,8 @@ Persistent Connection Providers

 .. py:class:: web3.providers.persistent.PersistentConnectionProvider(endpoint_uri: str, request_timeout: float = 50.0, subscription_response_queue_size: int = 500)

-    This is a base provider class, currently inherited by the ``WebsocketProviderV2``.
+    This is a base provider class, currently inherited by the ``WebsocketProviderV2``,
+    and the ``AsyncIPCProvider``.
    It handles interactions with a persistent connection to a JSON-RPC server. Among
    its configuration, it houses all of the
    :class:`~web3.providers.websocket.request_processor.RequestProcessor` logic for


This moved to web3.providers.persistent.request_processor I believe.

DavidRomanovizc commented Jun 9, 2023

View reviewed changes

web3/providers/async_ipc.py Outdated Show resolved Hide resolved

DavidRomanovizc commented Jun 9, 2023

View reviewed changes

web3/_utils/threads.py Outdated Show resolved Hide resolved

DavidRomanovizc commented Jul 26, 2023

View reviewed changes

web3/providers/async_ipc.py Outdated Show resolved Hide resolved

DavidRomanovizc commented Jul 26, 2023

View reviewed changes

web3/providers/async_ipc.py Outdated Show resolved Hide resolved

web3/providers/async_ipc.py Outdated Show resolved Hide resolved

kclowes requested review from fselmo, pacrob and reedsa December 20, 2023 22:32

pacrob approved these changes Jan 2, 2024

View reviewed changes

fselmo reviewed Jan 2, 2024

View reviewed changes

DavidRomanovizc added 2 commits February 14, 2024 10:41

✨ Add AsyncIPCProvider

b4946b4

⚡️Add support for asynchronous context managers to Timeout

efc3338

kclowes requested review from fselmo and pacrob February 21, 2024 22:16

fselmo approved these changes Feb 21, 2024

View reviewed changes

DavidRomanovizc and others added 9 commits February 22, 2024 12:19

✅ Update tests

74a9d9a

Add newsfragment

f0d408e

Add docs for AsyncIPCProvider, fix docs for IPCProvider

5a7dbbd

Inherit from PersistentConnectionProvider for AsyncIPCProvider

0af9bb9

- Iron out some issues with rebasing; update async_ipc

Clean up + fix AsyncIPC tests; move fixtures and constants to top of …

d9812c6

…file

Ensure the listen event is cleared before waiting until set

380265a

Refactor wsv2 and async_ipc into providers/persistent

cf73ebb

- Update docs for persistent connection provider refactor Remove sleeps to emulate responses in rapid succession

pacrob reviewed Feb 22, 2024

View reviewed changes

clean up websocket verbiage

518d23a

pacrob reviewed Feb 22, 2024

View reviewed changes

Update docs, minor fixes

2dfc55a

kclowes merged commit a02516c into ethereum:main Feb 23, 2024
85 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add AsyncIPCProvider #2984

✨ Add AsyncIPCProvider #2984

DavidRomanovizc commented Jun 7, 2023 •

edited by pacrob

Loading

JustAnotherC0der commented Jul 31, 2023

kclowes commented Jul 31, 2023

kclowes commented Dec 20, 2023

SheldonHolmgren commented Jan 1, 2024

pacrob left a comment

fselmo left a comment

SheldonHolmgren commented Feb 13, 2024 •

edited

Loading

kclowes commented Feb 15, 2024

kclowes commented Feb 21, 2024

fselmo left a comment •

edited

Loading

pacrob Feb 22, 2024

kclowes Feb 22, 2024

pacrob Feb 22, 2024

✨ Add AsyncIPCProvider #2984

✨ Add AsyncIPCProvider #2984

Conversation

DavidRomanovizc commented Jun 7, 2023 • edited by pacrob Loading

What was wrong?

How was it fixed?

Todo:

Cute Animal Picture

JustAnotherC0der commented Jul 31, 2023

kclowes commented Jul 31, 2023

kclowes commented Dec 20, 2023

SheldonHolmgren commented Jan 1, 2024

pacrob left a comment

Choose a reason for hiding this comment

fselmo left a comment

Choose a reason for hiding this comment

SheldonHolmgren commented Feb 13, 2024 • edited Loading

kclowes commented Feb 15, 2024

kclowes commented Feb 21, 2024

fselmo left a comment • edited Loading

Choose a reason for hiding this comment

pacrob Feb 22, 2024

Choose a reason for hiding this comment

kclowes Feb 22, 2024

Choose a reason for hiding this comment

pacrob Feb 22, 2024

Choose a reason for hiding this comment

DavidRomanovizc commented Jun 7, 2023 •

edited by pacrob

Loading

SheldonHolmgren commented Feb 13, 2024 •

edited

Loading

fselmo left a comment •

edited

Loading