Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: serialize (named)tuple when using orjson #612

Merged

Conversation

zevisert
Copy link
Contributor

Description

When using orjson, namedtuples are not serialized. For example:

>>> import faust.utils.json, orjson, inspect, aiokafka.structs
>>> print(inspect.getsource(aiokafka.structs.TopicPartition))
class TopicPartition(NamedTuple):
    """A topic and partition tuple"""

    topic: str
    "A topic name"

    partition: int
    "A partition id"

>>> tp = aiokafka.structs.TopicPartition(topic='test', partition=0)
>>> orjson.dumps(tp)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: Type is not JSON serializable: TopicPartition
>>> faust.utils.json.dumps(tp)
# TypeError: JSON cannot serialize 'TopicPartition': TopicPartition(topic='test', partition=0)
# 
# The above exception was the direct cause of the following exception:
# 
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/code/oss/github/faust-streaming/faust/faust/utils/json.py", line 176, in dumps
#     return json_dumps(
#            ^^^^^^^^^^^
# TypeError: Type is not JSON serializable: TopicPartition
>>> # With this patch applied
>>> import faust.utils.json, orjson, inspect, aiokafka.structs
>>> tp = aiokafka.structs.TopicPartition(topic='test', partition=0)
>>> orjson.dumps(tp)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: Type is not JSON serializable: TopicPartition
>>> faust.utils.json.dumps(tp)
b'["test",0]'
Original traceback encountered in the [page_views example](https://faust-streaming.github.io/faust/playbooks/pageviews.html) when testing within an application we're integrating faust with
> faust -A faust_test send page_views '{"id": "foo", "user": "bar"}'                                                                                         
[2024-02-29 10:48:32,390] [20569] [ERROR] [^Worker]: Error: TypeError('Type is not JSON serializable: TopicPartition') 
TypeError: JSON cannot serialize 'TopicPartition': TopicPartition(topic='page_views', partition=0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/worker.py", line 279, in execute_from_commandline
    self.loop.run_until_complete(self._starting_fut)
  File "/opt/tooling/pyenv/versions/3.11.7/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 800, in start
    await self._default_start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 807, in _default_start
    await self._actually_start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 831, in _actually_start
    await child.maybe_start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 859, in maybe_start
    await self.start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 800, in start
    await self._default_start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 807, in _default_start
    await self._actually_start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 824, in _actually_start
    await self.on_start()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 1134, in on_start
    await self._fut
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/faust/cli/base.py", line 607, in execute
    await self.run(*args, **kwargs)
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/faust/cli/send.py", line 84, in run
    self.say(self.dumps(meta._asdict()))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/faust/cli/base.py", line 746, in dumps
    return json.dumps(obj)
           ^^^^^^^^^^^^^^^
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/faust/utils/json.py", line 176, in dumps
    return json_dumps(
           ^^^^^^^^^^^
TypeError: Type is not JSON serializable: TopicPartition
[2024-02-29 10:48:32,392] [20569] [ERROR] [^Worker]: Error while stopping child <send: stopping >: TypeError('Type is not JSON serializable: TopicPartition') 
TypeError: JSON cannot serialize 'TopicPartition': TopicPartition(topic='page_views', partition=0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 927, in _default_stop_children
    await asyncio.shield(child.stop())
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 902, in stop
    await self.on_stop()
  File "/opt/tooling/pyenv/versions/3.11.7/envs/hula-env/lib/python3.11/site-packages/mode/services.py", line 1145, in on_stop
    fut.result()
TypeError: Type is not JSON serializable: TopicPartition

The reason for this is that unlike stdlib json, orjson does not serialize named tuples by default, so we're add the tuple baseclass to the list of sequence types that faust.utils.json.on_default should serialize.

Copy link

codecov bot commented Mar 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.72%. Comparing base (ebf66ae) to head (22abce9).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #612   +/-   ##
=======================================
  Coverage   93.72%   93.72%           
=======================================
  Files         102      102           
  Lines       11122    11122           
  Branches     1545     1545           
=======================================
  Hits        10424    10424           
  Misses        607      607           
  Partials       91       91           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@wbarnha wbarnha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks harmless and adds more functionality, approved!

@wbarnha wbarnha merged commit 5330c45 into faust-streaming:master Mar 1, 2024
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants