Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DNM: Explore performance of adding a Cython WebSocket frame parser #9541

Closed
wants to merge 12 commits into from
Closed

DNM: Explore performance of adding a Cython WebSocket frame parser #9541

wants to merge 12 commits into from

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Oct 25, 2024 •
edited
Loading

DNM: this is a rough version to check performance

It works out to ~30% faster.... I'm not sure if its worth it or not to have to maintain the parse_frame function twice

@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Oct 25, 2024 •
edited
Loading

CodSpeed Performance Report

Merging #9541 will improve performances by 30.94%

Comparing websocket_reader_cython (2b94609) with master (7982b59)

Summary

⚡ 1 improvements
✅ 3 untouched benchmarks

Benchmarks breakdown

Benchmark master websocket_reader_cython Change
⚡ test_read_one_hundred_websocket_text_messages[pyloop] 1,292.7 µs 987.3 µs +30.94%

@codecov Codecov
Copy link

codecov bot commented Oct 26, 2024 •
edited
Loading

❌ 10 Tests Failed:

Tests completed Failed Passed Skipped
3347 10 3337 24
View the top 3 failed tests by shortest run time 
tests.test_websocket_writer test_concurrent_messages[4096-<lambda>]
Stack Traces | 0.004s run time 
protocol = <Mock id='4495394064'>, transport = <Mock id='4495404560'>
max_sync_chunk_size = 4096
payload_point_generator = <function <lambda> at 0x109c8a980>

    #x1B[0m#x1B[37m@pytest#x1B[39;49;00m.mark.parametrize(#x1B[90m#x1B[39;49;00m
        (#x1B[33m"#x1B[39;49;00m#x1B[33mmax_sync_chunk_size#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mpayload_point_generator#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
        (#x1B[90m#x1B[39;49;00m
            (#x1B[94m16#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: count),#x1B[90m#x1B[39;49;00m
            (#x1B[94m4096#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: count),#x1B[90m#x1B[39;49;00m
            (#x1B[94m32#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: #x1B[94m64#x1B[39;49;00m + count #x1B[94mif#x1B[39;49;00m count % #x1B[94m2#x1B[39;49;00m #x1B[94melse#x1B[39;49;00m count),#x1B[90m#x1B[39;49;00m
        ),#x1B[90m#x1B[39;49;00m
    )#x1B[90m#x1B[39;49;00m
    #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92mtest_concurrent_messages#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        protocol: BaseProtocol,#x1B[90m#x1B[39;49;00m
        transport: asyncio.Transport,#x1B[90m#x1B[39;49;00m
        max_sync_chunk_size: #x1B[96mint#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        payload_point_generator: Callable[[#x1B[96mint#x1B[39;49;00m], #x1B[96mint#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m
    ) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Ensure messages are compressed correctly when there are multiple concurrent writers.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    This test generates is parametrized to#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Generate messages that are larger than patch#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 16#x1B[39;49;00m
    #x1B[33m      where compression will run in the executor#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Generate messages that are smaller than patch#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 4096#x1B[39;49;00m
    #x1B[33m      where compression will run in the event loop#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Interleave generated messages with a#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 32#x1B[39;49;00m
    #x1B[33m      where compression will run in the event loop#x1B[39;49;00m
    #x1B[33m      and in the executor#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94mwith#x1B[39;49;00m mock.patch(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33maiohttp.http_websocket.WEBSOCKET_MAX_SYNC_CHUNK_SIZE#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, max_sync_chunk_size#x1B[90m#x1B[39;49;00m
        ):#x1B[90m#x1B[39;49;00m
            writer = WebSocketWriter(protocol, transport, compress=#x1B[94m15#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            queue: DataQueue[WSMessage] = DataQueue(asyncio.get_running_loop())#x1B[90m#x1B[39;49;00m
            reader = WebSocketReader(queue, #x1B[94m50000#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            writers = []#x1B[90m#x1B[39;49;00m
            payloads = []#x1B[90m#x1B[39;49;00m
            #x1B[94mfor#x1B[39;49;00m count #x1B[95min#x1B[39;49;00m #x1B[96mrange#x1B[39;49;00m(#x1B[94m1#x1B[39;49;00m, #x1B[94m64#x1B[39;49;00m + #x1B[94m1#x1B[39;49;00m):#x1B[90m#x1B[39;49;00m
                point = payload_point_generator(count)#x1B[90m#x1B[39;49;00m
                payload = #x1B[96mbytes#x1B[39;49;00m((point,)) * point#x1B[90m#x1B[39;49;00m
                payloads.append(payload)#x1B[90m#x1B[39;49;00m
                writers.append(writer.send_frame(payload, WSMsgType.BINARY))#x1B[90m#x1B[39;49;00m
            #x1B[94mawait#x1B[39;49;00m asyncio.gather(*writers)#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[94mfor#x1B[39;49;00m call #x1B[95min#x1B[39;49;00m writer.transport.write.call_args_list:  #x1B[90m# type: ignore[attr-defined]#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            call_bytes = call[#x1B[94m0#x1B[39;49;00m][#x1B[94m0#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m
            result, _ = reader.feed_data(call_bytes)#x1B[90m#x1B[39;49;00m
>           #x1B[94massert#x1B[39;49;00m result #x1B[95mis#x1B[39;49;00m #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           assert True is False#x1B[0m

_          = b''
call       = call(b'\xc2\x03b\x04\x00')
call_bytes = b'\xc2\x03b\x04\x00'
count      = 64
max_sync_chunk_size = 4096
payload    = b'@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@'
payload_point_generator = <function <lambda> at 0x109c8a980>
payloads   = [b'\x01', b'\x02\x02', b'\x03\x03\x03', b'\x04\x04\x04\x04', b'\x05\x05\x05\x05\x05', b'\x06\x06\x06\x06\x06\x06', ...]
point      = 64
protocol   = <Mock id='4495394064'>
queue      = <aiohttp.streams.DataQueue object at 0x10bf27f50>
reader     = <aiohttp.http_websocket.WebSocketReader object at 0x10becd850>
result     = True
transport  = <Mock id='4495404560'>
writer     = <aiohttp.http_websocket.WebSocketWriter object at 0x10bf27150>
writers    = [<coroutine object WebSocketWriter.send_frame at 0x10be91e40>, <coroutine object WebSocketWriter.send_frame at 0x10be9...e object WebSocketWriter.send_frame at 0x10be920c0>, <coroutine object WebSocketWriter.send_frame at 0x10be92200>, ...]

#x1B[1m#x1B[31mtests/test_websocket_writer.py#x1B[0m:170: AssertionError

tests.test_websocket_writer test_concurrent_messages[16-<lambda>]
Stack Traces | 0.009s run time 
protocol = <Mock id='4492260688'>, transport = <Mock id='4492257552'>
max_sync_chunk_size = 16
payload_point_generator = <function <lambda> at 0x109c8a8e0>

    #x1B[0m#x1B[37m@pytest#x1B[39;49;00m.mark.parametrize(#x1B[90m#x1B[39;49;00m
        (#x1B[33m"#x1B[39;49;00m#x1B[33mmax_sync_chunk_size#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mpayload_point_generator#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
        (#x1B[90m#x1B[39;49;00m
            (#x1B[94m16#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: count),#x1B[90m#x1B[39;49;00m
            (#x1B[94m4096#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: count),#x1B[90m#x1B[39;49;00m
            (#x1B[94m32#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: #x1B[94m64#x1B[39;49;00m + count #x1B[94mif#x1B[39;49;00m count % #x1B[94m2#x1B[39;49;00m #x1B[94melse#x1B[39;49;00m count),#x1B[90m#x1B[39;49;00m
        ),#x1B[90m#x1B[39;49;00m
    )#x1B[90m#x1B[39;49;00m
    #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92mtest_concurrent_messages#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        protocol: BaseProtocol,#x1B[90m#x1B[39;49;00m
        transport: asyncio.Transport,#x1B[90m#x1B[39;49;00m
        max_sync_chunk_size: #x1B[96mint#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        payload_point_generator: Callable[[#x1B[96mint#x1B[39;49;00m], #x1B[96mint#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m
    ) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Ensure messages are compressed correctly when there are multiple concurrent writers.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    This test generates is parametrized to#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Generate messages that are larger than patch#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 16#x1B[39;49;00m
    #x1B[33m      where compression will run in the executor#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Generate messages that are smaller than patch#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 4096#x1B[39;49;00m
    #x1B[33m      where compression will run in the event loop#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Interleave generated messages with a#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 32#x1B[39;49;00m
    #x1B[33m      where compression will run in the event loop#x1B[39;49;00m
    #x1B[33m      and in the executor#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94mwith#x1B[39;49;00m mock.patch(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33maiohttp.http_websocket.WEBSOCKET_MAX_SYNC_CHUNK_SIZE#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, max_sync_chunk_size#x1B[90m#x1B[39;49;00m
        ):#x1B[90m#x1B[39;49;00m
            writer = WebSocketWriter(protocol, transport, compress=#x1B[94m15#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            queue: DataQueue[WSMessage] = DataQueue(asyncio.get_running_loop())#x1B[90m#x1B[39;49;00m
            reader = WebSocketReader(queue, #x1B[94m50000#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            writers = []#x1B[90m#x1B[39;49;00m
            payloads = []#x1B[90m#x1B[39;49;00m
            #x1B[94mfor#x1B[39;49;00m count #x1B[95min#x1B[39;49;00m #x1B[96mrange#x1B[39;49;00m(#x1B[94m1#x1B[39;49;00m, #x1B[94m64#x1B[39;49;00m + #x1B[94m1#x1B[39;49;00m):#x1B[90m#x1B[39;49;00m
                point = payload_point_generator(count)#x1B[90m#x1B[39;49;00m
                payload = #x1B[96mbytes#x1B[39;49;00m((point,)) * point#x1B[90m#x1B[39;49;00m
                payloads.append(payload)#x1B[90m#x1B[39;49;00m
                writers.append(writer.send_frame(payload, WSMsgType.BINARY))#x1B[90m#x1B[39;49;00m
            #x1B[94mawait#x1B[39;49;00m asyncio.gather(*writers)#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[94mfor#x1B[39;49;00m call #x1B[95min#x1B[39;49;00m writer.transport.write.call_args_list:  #x1B[90m# type: ignore[attr-defined]#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            call_bytes = call[#x1B[94m0#x1B[39;49;00m][#x1B[94m0#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m
            result, _ = reader.feed_data(call_bytes)#x1B[90m#x1B[39;49;00m
>           #x1B[94massert#x1B[39;49;00m result #x1B[95mis#x1B[39;49;00m #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           assert True is False#x1B[0m

_          = b''
call       = call(b'\xc2\x03b\x04\x00')
call_bytes = b'\xc2\x03b\x04\x00'
count      = 64
max_sync_chunk_size = 16
payload    = b'@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@'
payload_point_generator = <function <lambda> at 0x109c8a8e0>
payloads   = [b'\x01', b'\x02\x02', b'\x03\x03\x03', b'\x04\x04\x04\x04', b'\x05\x05\x05\x05\x05', b'\x06\x06\x06\x06\x06\x06', ...]
point      = 64
protocol   = <Mock id='4492260688'>
queue      = <aiohttp.streams.DataQueue object at 0x10bf10fd0>
reader     = <aiohttp.http_websocket.WebSocketReader object at 0x10bbb51d0>
result     = True
transport  = <Mock id='4492257552'>
writer     = <aiohttp.http_websocket.WebSocketWriter object at 0x10bf102d0>
writers    = [<coroutine object WebSocketWriter.send_frame at 0x10bad4400>, <coroutine object WebSocketWriter.send_frame at 0x10bad...e object WebSocketWriter.send_frame at 0x10bad5080>, <coroutine object WebSocketWriter.send_frame at 0x10bad51c0>, ...]

#x1B[1m#x1B[31mtests/test_websocket_writer.py#x1B[0m:170: AssertionError

tests.test_websocket_writer test_concurrent_messages[32-<lambda>]
Stack Traces | 0.011s run time 
protocol = <Mock id='4494752144'>, transport = <Mock id='4494750608'>
max_sync_chunk_size = 32
payload_point_generator = <function <lambda> at 0x109c8aa20>

    #x1B[0m#x1B[37m@pytest#x1B[39;49;00m.mark.parametrize(#x1B[90m#x1B[39;49;00m
        (#x1B[33m"#x1B[39;49;00m#x1B[33mmax_sync_chunk_size#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mpayload_point_generator#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
        (#x1B[90m#x1B[39;49;00m
            (#x1B[94m16#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: count),#x1B[90m#x1B[39;49;00m
            (#x1B[94m4096#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: count),#x1B[90m#x1B[39;49;00m
            (#x1B[94m32#x1B[39;49;00m, #x1B[94mlambda#x1B[39;49;00m count: #x1B[94m64#x1B[39;49;00m + count #x1B[94mif#x1B[39;49;00m count % #x1B[94m2#x1B[39;49;00m #x1B[94melse#x1B[39;49;00m count),#x1B[90m#x1B[39;49;00m
        ),#x1B[90m#x1B[39;49;00m
    )#x1B[90m#x1B[39;49;00m
    #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92mtest_concurrent_messages#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        protocol: BaseProtocol,#x1B[90m#x1B[39;49;00m
        transport: asyncio.Transport,#x1B[90m#x1B[39;49;00m
        max_sync_chunk_size: #x1B[96mint#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        payload_point_generator: Callable[[#x1B[96mint#x1B[39;49;00m], #x1B[96mint#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m
    ) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Ensure messages are compressed correctly when there are multiple concurrent writers.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    This test generates is parametrized to#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Generate messages that are larger than patch#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 16#x1B[39;49;00m
    #x1B[33m      where compression will run in the executor#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Generate messages that are smaller than patch#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 4096#x1B[39;49;00m
    #x1B[33m      where compression will run in the event loop#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    - Interleave generated messages with a#x1B[39;49;00m
    #x1B[33m      WEBSOCKET_MAX_SYNC_CHUNK_SIZE of 32#x1B[39;49;00m
    #x1B[33m      where compression will run in the event loop#x1B[39;49;00m
    #x1B[33m      and in the executor#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94mwith#x1B[39;49;00m mock.patch(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33maiohttp.http_websocket.WEBSOCKET_MAX_SYNC_CHUNK_SIZE#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, max_sync_chunk_size#x1B[90m#x1B[39;49;00m
        ):#x1B[90m#x1B[39;49;00m
            writer = WebSocketWriter(protocol, transport, compress=#x1B[94m15#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            queue: DataQueue[WSMessage] = DataQueue(asyncio.get_running_loop())#x1B[90m#x1B[39;49;00m
            reader = WebSocketReader(queue, #x1B[94m50000#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            writers = []#x1B[90m#x1B[39;49;00m
            payloads = []#x1B[90m#x1B[39;49;00m
            #x1B[94mfor#x1B[39;49;00m count #x1B[95min#x1B[39;49;00m #x1B[96mrange#x1B[39;49;00m(#x1B[94m1#x1B[39;49;00m, #x1B[94m64#x1B[39;49;00m + #x1B[94m1#x1B[39;49;00m):#x1B[90m#x1B[39;49;00m
                point = payload_point_generator(count)#x1B[90m#x1B[39;49;00m
                payload = #x1B[96mbytes#x1B[39;49;00m((point,)) * point#x1B[90m#x1B[39;49;00m
                payloads.append(payload)#x1B[90m#x1B[39;49;00m
                writers.append(writer.send_frame(payload, WSMsgType.BINARY))#x1B[90m#x1B[39;49;00m
            #x1B[94mawait#x1B[39;49;00m asyncio.gather(*writers)#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[94mfor#x1B[39;49;00m call #x1B[95min#x1B[39;49;00m writer.transport.write.call_args_list:  #x1B[90m# type: ignore[attr-defined]#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            call_bytes = call[#x1B[94m0#x1B[39;49;00m][#x1B[94m0#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m
            result, _ = reader.feed_data(call_bytes)#x1B[90m#x1B[39;49;00m
>           #x1B[94massert#x1B[39;49;00m result #x1B[95mis#x1B[39;49;00m #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           assert True is False#x1B[0m

_          = b''
call       = call(b'\xc2\x06rt\xa4\x10\x00\x00')
call_bytes = b'\xc2\x06rt\xa4\x10\x00\x00'
count      = 64
max_sync_chunk_size = 32
payload    = b'@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@'
payload_point_generator = <function <lambda> at 0x109c8aa20>
payloads   = [b'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA', b'\x02\x02', b'CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC...4\x04\x04', b'EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE', b'\x06\x06\x06\x06\x06\x06', ...]
point      = 64
protocol   = <Mock id='4494752144'>
queue      = <aiohttp.streams.DataQueue object at 0x10bec3790>
reader     = <aiohttp.http_websocket.WebSocketReader object at 0x10beee5d0>
result     = True
transport  = <Mock id='4494750608'>
writer     = <aiohttp.http_websocket.WebSocketWriter object at 0x10bec3450>
writers    = [<coroutine object WebSocketWriter.send_frame at 0x10beaae80>, <coroutine object WebSocketWriter.send_frame at 0x10bea...e object WebSocketWriter.send_frame at 0x10beab100>, <coroutine object WebSocketWriter.send_frame at 0x10beab240>, ...]

#x1B[1m#x1B[31mtests/test_websocket_writer.py#x1B[0m:170: AssertionError

#x1B[33mDuring handling of the above exception, another exception occurred:#x1B[0m

cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x10bebc720>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    #x1B[0m#x1B[37m@classmethod#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m #x1B[92mfrom_call#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        #x1B[96mcls#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        func: Callable[[], TResult],#x1B[90m#x1B[39;49;00m
        when: Literal[#x1B[33m"#x1B[39;49;00m#x1B[33mcollect#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33msetup#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mcall#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mteardown#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m
        reraise: Optional[#x1B[90m#x1B[39;49;00m
            Union[Type[#x1B[96mBaseException#x1B[39;49;00m], Tuple[Type[#x1B[96mBaseException#x1B[39;49;00m], ...]]#x1B[90m#x1B[39;49;00m
        ] = #x1B[94mNone#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
    ) -> #x1B[33m"#x1B[39;49;00m#x1B[33mCallInfo[TResult]#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Call func, wrapping the result in a CallInfo.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    :param func:#x1B[39;49;00m
    #x1B[33m        The function to call. Called without arguments.#x1B[39;49;00m
    #x1B[33m    :param when:#x1B[39;49;00m
    #x1B[33m        The phase in which the function is called.#x1B[39;49;00m
    #x1B[33m    :param reraise:#x1B[39;49;00m
    #x1B[33m        Exception or exceptions that shall propagate if raised by the#x1B[39;49;00m
    #x1B[33m        function, instead of being wrapped in the CallInfo.#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        excinfo = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        start = timing.time()#x1B[90m#x1B[39;49;00m
        precise_start = timing.perf_counter()#x1B[90m#x1B[39;49;00m
        #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
>           result: Optional[TResult] = func()#x1B[90m#x1B[39;49;00m

cls        = <class '_pytest.runner.CallInfo'>
duration   = 0.010562041999946814
excinfo    = <ExceptionInfo PytestUnraisableExceptionWarning('Exception ignored in: <socket.socket fd=-1, family=1, type=1, proto=0...   def __enter__(self):\n    \nResourceWarning: unclosed <socket.socket fd=26, family=1, type=1, proto=0>\n') tblen=12>
func       = <function call_and_report.<locals>.<lambda> at 0x10bebc720>
precise_start = 1240.065390916
precise_stop = 1240.075952958
reraise    = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
result     = None
start      = 1729904411.6852539
stop       = 1729904411.695818
when       = 'call'

#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11....../site-packages/_pytest/runner.py#x1B[0m:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11....../site-packages/_pytest/runner.py#x1B[0m:240: in <lambda>
    #x1B[0m#x1B[94mlambda#x1B[39;49;00m: runtest_hook(item=item, **kwds), when=when, reraise=reraise#x1B[90m#x1B[39;49;00m
        item       = <Function test_concurrent_messages[32-<lambda>]>
        kwds       = {}
        runtest_hook = <HookCaller 'pytest_runtest_call'>
#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11.../site-packages/pluggy/_hooks.py#x1B[0m:513: in __call__
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._hookexec(#x1B[96mself#x1B[39;49;00m.name, #x1B[96mself#x1B[39;49;00m._hookimpls.copy(), kwargs, firstresult)#x1B[90m#x1B[39;49;00m
        firstresult = False
        kwargs     = {'item': <Function test_concurrent_messages[32-<lambda>]>}
        self       = <HookCaller 'pytest_runtest_call'>
#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11.../site-packages/pluggy/_manager.py#x1B[0m:120: in _hookexec
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._inner_hookexec(hook_name, methods, kwargs, firstresult)#x1B[90m#x1B[39;49;00m
        firstresult = False
        hook_name  = 'pytest_runtest_call'
        kwargs     = {'item': <Function test_concurrent_messages[32-<lambda>]>}
        methods    = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '........./Library/Frameworks/Python.framework/Versions/3............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11........./site-packages/_pytest/unraisableexception.py'>>, ...]
        self       = <_pytest.config.PytestPluginManager object at 0x1057fda50>
#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11....../site-packages/_pytest/threadexception.py#x1B[0m:87: in pytest_runtest_call
    #x1B[0m#x1B[94myield from#x1B[39;49;00m thread_exception_runtest_hook()#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11....../site-packages/_pytest/threadexception.py#x1B[0m:63: in thread_exception_runtest_hook
    #x1B[0m#x1B[94myield#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        cm         = <_pytest.threadexception.catch_threading_exception object at 0x10bd37690>
#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11........./site-packages/_pytest/unraisableexception.py#x1B[0m:90: in pytest_runtest_call
    #x1B[0m#x1B[94myield from#x1B[39;49;00m unraisable_exception_runtest_hook()#x1B[90m#x1B[39;49;00m
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    #x1B[0m#x1B[94mdef#x1B[39;49;00m #x1B[92munraisable_exception_runtest_hook#x1B[39;49;00m() -> Generator[#x1B[94mNone#x1B[39;49;00m, #x1B[94mNone#x1B[39;49;00m, #x1B[94mNone#x1B[39;49;00m]:#x1B[90m#x1B[39;49;00m
        #x1B[94mwith#x1B[39;49;00m catch_unraisable_exception() #x1B[94mas#x1B[39;49;00m cm:#x1B[90m#x1B[39;49;00m
            #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                #x1B[94myield#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[94mfinally#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                #x1B[94mif#x1B[39;49;00m cm.unraisable:#x1B[90m#x1B[39;49;00m
                    #x1B[94mif#x1B[39;49;00m cm.unraisable.err_msg #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                        err_msg = cm.unraisable.err_msg#x1B[90m#x1B[39;49;00m
                    #x1B[94melse#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                        err_msg = #x1B[33m"#x1B[39;49;00m#x1B[33mException ignored in#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
                    msg = #x1B[33mf#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[33m{#x1B[39;49;00merr_msg#x1B[33m}#x1B[39;49;00m#x1B[33m: #x1B[39;49;00m#x1B[33m{#x1B[39;49;00mcm.unraisable.object#x1B[33m!r}#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
                    msg += #x1B[33m"#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m.join(#x1B[90m#x1B[39;49;00m
                        traceback.format_exception(#x1B[90m#x1B[39;49;00m
                            cm.unraisable.exc_type,#x1B[90m#x1B[39;49;00m
                            cm.unraisable.exc_value,#x1B[90m#x1B[39;49;00m
                            cm.unraisable.exc_traceback,#x1B[90m#x1B[39;49;00m
                        )#x1B[90m#x1B[39;49;00m
                    )#x1B[90m#x1B[39;49;00m
>                   warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE                   pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=1, type=1, proto=0>#x1B[0m
#x1B[1m#x1B[31mE                   #x1B[0m
#x1B[1m#x1B[31mE                   Traceback (most recent call last):#x1B[0m
#x1B[1m#x1B[31mE                     File "........./Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 271, in __enter__#x1B[0m
#x1B[1m#x1B[31mE                       def __enter__(self):#x1B[0m
#x1B[1m#x1B[31mE                       #x1B[0m
#x1B[1m#x1B[31mE                   ResourceWarning: unclosed <socket.socket fd=26, family=1, type=1, proto=0>#x1B[0m

cm         = <_pytest.unraisableexception.catch_unraisable_exception object at 0x10bd372d0>
err_msg    = 'Exception ignored in'
msg        = 'Exception ignored in: <socket.socket fd=-1, family=1, type=1, proto=0>\n\nTraceback (most recent call last):\n  File ..._enter__\n    def __enter__(self):\n    \nResourceWarning: unclosed <socket.socket fd=26, family=1, type=1, proto=0>\n'

#x1B[1m#x1B[............/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11........./site-packages/_pytest/unraisableexception.py#x1B[0m:80: PytestUnraisableExceptionWarning

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

@bdraco
Copy link
Member Author

bdraco commented Oct 26, 2024

The code is nearly identical. I wonder if a pxd file would make more sense. Then we only have to maintain that and keep it in sync with the python code.

@bdraco
Copy link
Member Author

bdraco commented Oct 26, 2024

Maybe move WebsocketReader into http_websocket_reader.py then we have http_websocket_reader.pxd which has the cython defs and we don't have two copies of the code and cython will add the types from the pxd

@bdraco
Copy link
Member Author

bdraco commented Oct 26, 2024

It looks like that would work. Would have to move around some code first as http_websocket.py would need to import WebsocketReader from http_websocket_reader.py and since WebsocketReader needs ``WSMsgTypeandWSMessage` that would be a circular imports.

So maybe move them all to http_websocket_models.py first

@bdraco
Copy link
Member Author

bdraco commented Oct 26, 2024

I like that design better because we end up with ~100 lines of cython typing to maintain and the python code is the same and we don't have a copy

@bdraco
Copy link
Member Author

bdraco commented Oct 26, 2024

pxd is probably the way to do since the time in _feed_data and parse_frame are both high

This was referenced Oct 26, 2024
Merged
Merged
@bdraco
Copy link
Member Author

bdraco commented Oct 26, 2024

PXD only approach seems to be more than good enough and has far less maint burden #9543

@bdraco bdraco closed this Oct 26, 2024
@bdraco bdraco deleted the websocket_reader_cython branch October 26, 2024 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov will be requested when the pull request is marked ready for review asvetlov is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bdraco