Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a PXD only Cython extension for the WebSocket reader to improve performance #9543

Merged
merged 64 commits into from
Oct 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
689536f
Seperate http_websocket into multiple files
bdraco Oct 26, 2024
de38d0c
Seperate http_websocket into multiple files
bdraco Oct 26, 2024
49150fe
Seperate http_websocket into multiple files
bdraco Oct 26, 2024
3121b6d
Seperate http_websocket into multiple files
bdraco Oct 26, 2024
fc598bc
fix location
bdraco Oct 26, 2024
ae26d08
more cleanups
bdraco Oct 26, 2024
dcc924e
keep compat
bdraco Oct 26, 2024
18752e8
Add pxd file for _websocket_reader
bdraco Oct 26, 2024
05696fb
Test building from pxd
bdraco Oct 26, 2024
975e80b
subclass to allow patching
bdraco Oct 26, 2024
9164179
more cleanups
bdraco Oct 26, 2024
106adcb
more types
bdraco Oct 26, 2024
4717a06
more types
bdraco Oct 26, 2024
68acafc
type cleanups
bdraco Oct 26, 2024
133b50c
changelog
bdraco Oct 26, 2024
0fc268e
Relocate to _websocket
bdraco Oct 27, 2024
33149b0
fix path
bdraco Oct 27, 2024
26ae66f
Merge branch 'websocket_pxd' into websocket_pxd_create
bdraco Oct 27, 2024
ddd73db
fix patch target
bdraco Oct 27, 2024
f6a76f9
Merge branch 'websocket_pxd' into websocket_pxd_create
bdraco Oct 27, 2024
e077a36
rework so NO_EXTENSIONS is considered
bdraco Oct 27, 2024
b74b874
namespace cleanups
bdraco Oct 27, 2024
6e0fdcb
Revert "namespace cleanups"
bdraco Oct 27, 2024
5fb55e8
Merge branch 'websocket_pxd' into websocket_pxd_create
bdraco Oct 27, 2024
439101c
fix build path
bdraco Oct 27, 2024
8cda99a
add missing pxd file for mask
bdraco Oct 27, 2024
3b690f5
fix some double conversion
bdraco Oct 27, 2024
b6d256a
objects are immutable
bdraco Oct 27, 2024
d8d3c3a
match defs since we use the cython code insidepython code
bdraco Oct 27, 2024
f05f3bf
naming
bdraco Oct 27, 2024
9d9a886
Merge branch 'master' into websocket_pxd_create
bdraco Oct 27, 2024
9fb530d
must be named reader_c because PyInit is named
bdraco Oct 27, 2024
7fed656
reduce makefile
bdraco Oct 27, 2024
34acf92
fix namespace
bdraco Oct 27, 2024
df1e219
namespace fix
bdraco Oct 27, 2024
097899f
fix type, its not base exception
bdraco Oct 27, 2024
2be1609
reduce opcode type to unsigned int
bdraco Oct 27, 2024
3697714
reduce opcode type to unsigned int
bdraco Oct 27, 2024
54dab0a
lint
bdraco Oct 27, 2024
c03fde4
fix type
bdraco Oct 27, 2024
e3ecce8
changelog
bdraco Oct 27, 2024
55b6ad2
Update aiohttp/_websocket/reader_py.py
bdraco Oct 27, 2024
ffba3db
Revert "fix type"
bdraco Oct 27, 2024
0f9bea9
Revert "lint"
bdraco Oct 27, 2024
4ae06ce
Revert "reduce opcode type to unsigned int"
bdraco Oct 27, 2024
f1c83a0
Revert "reduce opcode type to unsigned int"
bdraco Oct 27, 2024
4ad464a
Merge remote-tracking branch 'upstream/websocket_pxd_create' into web…
bdraco Oct 27, 2024
6432c93
no need to convert it since its being packed into a tuple
bdraco Oct 27, 2024
d194295
cleanup type on masking
bdraco Oct 27, 2024
f101ef1
cleanup signature on mask
bdraco Oct 27, 2024
35100b6
cleanup signature on _feed_data
bdraco Oct 27, 2024
3eb183e
handle data not being bytes
bdraco Oct 27, 2024
107fb26
return types
bdraco Oct 27, 2024
c1ccb7b
type tuple
bdraco Oct 27, 2024
c30ac29
Revert "type tuple"
bdraco Oct 27, 2024
4e44c85
coverage
bdraco Oct 27, 2024
1ea8cae
add the missing coverage
bdraco Oct 27, 2024
a8322b5
fix missing type
bdraco Oct 27, 2024
3d0dd11
use fastcall
bdraco Oct 27, 2024
26d0e8c
Make sure memoryview works as well
bdraco Oct 27, 2024
5947443
Update aiohttp/_websocket/reader_py.py
bdraco Oct 27, 2024
a15e2eb
cleanup refs
bdraco Oct 27, 2024
cc025d3
cleanup refs
bdraco Oct 27, 2024
2165de1
comments
bdraco Oct 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES/9543.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improved performance of reading WebSocket messages with a Cython implementation -- by :user:`bdraco`.
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ endif
aiohttp/_find_header.c: $(call to-hash,aiohttp/hdrs.py ./tools/gen.py)
./tools/gen.py

# Special case for reader since we want to be able to disable
# the extension with AIOHTTP_NO_EXTENSIONS
aiohttp/_websocket/reader_c.c: aiohttp/_websocket/reader_c.py
cython -3 -o $@ $< -I aiohttp -Werror

# _find_headers generator creates _headers.pyi as well
aiohttp/%.c: aiohttp/%.pyx $(call to-hash,$(CYS)) aiohttp/_find_header.c
cython -3 -o $@ $< -I aiohttp -Werror
Expand All @@ -74,7 +79,7 @@ vendor/llhttp/node_modules: vendor/llhttp/package.json
generate-llhttp: .llhttp-gen

.PHONY: cythonize
cythonize: .install-cython $(PYXS:.pyx=.c)
cythonize: .install-cython $(PYXS:.pyx=.c) aiohttp/_websocket/reader_c.c

.install-deps: .install-cython $(PYXS:.pyx=.c) $(call to-hash,$(CYS) $(REQS))
@python -m pip install -r requirements/dev.in -c requirements/dev.txt
Expand Down Expand Up @@ -157,6 +162,7 @@ clean:
@rm -f aiohttp/_http_parser.c
@rm -f aiohttp/_http_writer.c
@rm -f aiohttp/_websocket.c
@rm -f aiohttp/_websocket/reader_c.c
@rm -rf .tox
@rm -f .develop
@rm -f .flake
Expand Down
3 changes: 3 additions & 0 deletions aiohttp/_websocket/mask.pxd
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
"""Cython declarations for websocket masking."""

cpdef void _websocket_mask_cython(bytes mask, bytearray data)
10 changes: 1 addition & 9 deletions aiohttp/_websocket/mask.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ cdef extern from "Python.h":
from libc.stdint cimport uint32_t, uint64_t, uintmax_t


def _websocket_mask_cython(object mask, object data):
cpdef void _websocket_mask_cython(bytes mask, bytearray data):
"""Note, this function mutates its `data` argument
"""
cdef:
Expand All @@ -21,14 +21,6 @@ def _websocket_mask_cython(object mask, object data):

assert len(mask) == 4

if not isinstance(mask, bytes):
mask = bytes(mask)

if isinstance(data, bytearray):
data = <bytearray>data
else:
data = bytearray(data)

data_len = len(data)
in_buf = <unsigned char*>PyByteArray_AsString(data)
mask_buf = <const unsigned char*>PyBytes_AsString(mask)
Expand Down
Loading
Loading