Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka support #1506

Merged
merged 16 commits into from
Apr 21, 2022
Merged

Kafka support #1506

merged 16 commits into from
Apr 21, 2022

Conversation

CountRedClaw
Copy link
Contributor

Based on #600 but switched from pykafka to https://github.com/confluentinc/confluent-kafka-python as pykafka project was archived.
Also aiokafka was mentioned here, but I didn't find any usages of asyncio libraries in Kombu. Thus I assumed that the best option for me would be confluent-kafka-python.

Integration tests are green, but had to exclude BaseTimeToLive and BasePriority tests because Kafka does not allow to set expiration time per message, as well as it does not provide message priority feature (Kafka partition is an immutable ordered log of messages).

I'm not sure about the relevance of my PR (maybe you meant to use another library), so let me know if it makes sense.

@lgtm-com
Copy link

lgtm-com bot commented Mar 28, 2022

This pull request introduces 1 alert when merging 52fb571 into aab2588 - view on LGTM.com

new alerts:

  • 1 for Missing call to `__init__` during object initialization

ilbo0518 added 2 commits March 28, 2022 20:47
@lgtm-com
Copy link

lgtm-com bot commented Mar 28, 2022

This pull request introduces 1 alert when merging 39da6bb into aab2588 - view on LGTM.com

new alerts:

  • 1 for Missing call to `__init__` during object initialization

@lgtm-com
Copy link

lgtm-com bot commented Mar 28, 2022

This pull request introduces 1 alert when merging 6c6acb5 into aab2588 - view on LGTM.com

new alerts:

  • 1 for Missing call to `__init__` during object initialization

@auvipy auvipy self-requested a review April 3, 2022 04:39
@auvipy auvipy added this to the 5.3 milestone Apr 3, 2022
@matusvalo
Copy link
Member

Can you add also a documentation to head of file - see redis transport as an example:

"""Redis transport module for Kombu.
Features
========
* Type: Virtual
* Supports Direct: Yes
* Supports Topic: Yes
* Supports Fanout: Yes
* Supports Priority: Yes
* Supports TTL: No
Connection String
=================
Connection string has the following format:
.. code-block::
redis://[USER:PASSWORD@]REDIS_ADDRESS[:PORT][/VIRTUALHOST]
rediss://[USER:PASSWORD@]REDIS_ADDRESS[:PORT][/VIRTUALHOST]
To use sentinel for dynamic Redis discovery,
the connection string has following format:
.. code-block::
sentinel://[USER:PASSWORD@]SENTINEL_ADDRESS[:PORT]
Transport Options
=================
* ``sep``
* ``ack_emulation``: (bool) If set to True transport will
simulate Acknowledge of AMQP protocol.
* ``unacked_key``
* ``unacked_index_key``
* ``unacked_mutex_key``
* ``unacked_mutex_expire``
* ``visibility_timeout``
* ``unacked_restore_limit``
* ``fanout_prefix``
* ``fanout_patterns``
* ``global_keyprefix``: (str) The global key prefix to be prepended to all keys
used by Kombu
* ``socket_timeout``
* ``socket_connect_timeout``
* ``socket_keepalive``
* ``socket_keepalive_options``
* ``queue_order_strategy``
* ``max_connections``
* ``health_check_interval``
* ``retry_on_timeout``
* ``priority_steps``
"""

Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a rebase & docs update needed

@lgtm-com
Copy link

lgtm-com bot commented Apr 19, 2022

This pull request introduces 2 alerts when merging 3b7d903 into 14d395a - view on LGTM.com

new alerts:

  • 1 for Unused import
  • 1 for Missing call to `__init__` during object initialization

@lgtm-com
Copy link

lgtm-com bot commented Apr 19, 2022

This pull request introduces 1 alert when merging 9a1fe95 into 14d395a - view on LGTM.com

new alerts:

  • 1 for Missing call to `__init__` during object initialization

@CountRedClaw CountRedClaw requested a review from auvipy April 19, 2022 13:56
@CountRedClaw
Copy link
Contributor Author

Seems like confluent-kafka-python doesn't support PyPy. So had to disable PyPy tests for kafka.

@auvipy
Copy link
Member

auvipy commented Apr 19, 2022

Seems like confluent-kafka-python doesn't support PyPy. So had to disable PyPy tests for kafka.

that's OK, we just need to mention it on docs/release notes

@lgtm-com
Copy link

lgtm-com bot commented Apr 20, 2022

This pull request introduces 1 alert when merging 6cec1e9 into 14d395a - view on LGTM.com

new alerts:

  • 1 for Missing call to `__init__` during object initialization

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants