Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[EventHubs] Add keep_alive_interval to producer client #23361

Closed
wants to merge 1 commit into from
Closed

[EventHubs] Add keep_alive_interval to producer client #23361

wants to merge 1 commit into from

Conversation

yunhaoling
Copy link
Contributor

@yunhaoling yunhaoling commented Mar 7, 2022 •
edited
Loading

Expose keep alive config on producer client

potentially addressing issue: #21849

two issues:

  1. connection idletimeout
  • The connection between EH client and EH service is a long bidirectional amqp connection.
  • The EH service would disconnect the client for 240s idle timeout.
  • azure function is event driven, which means producer would stay idle until next incoming request
  • by expose keep alive, the client would send heartbeat every keep_alive_interval seconds
  1. execution hangs
    It is still unclear why the execution hangs after connection closed
  • probably because of segmentation fault?

addressing issue 1 might help the end to end scenario, however, azure function's behavior is unclear -- whether it's stateful or stateless:

  • by keep alive, we turn azure function into sort of a long operation and assume azure function is stateful, it could keep the producer, won't suspend the execution of background thread/coroutine.
    • ask azure function team/test the behavior
    • migrate to durable function/azure web app?

@ghost ghost added the Event Hubs label Mar 7, 2022
@azure-sdk
Copy link
Collaborator

API changes have been detected in azure-eventhub. You can review API changes here

API changes

+             keep_alive_interval: Union[int, None] = ..., 
+             keep_alive_interval: Union[int, None] = ..., 
+             keep_alive_interval: Union[int, None] = ..., 
+             keep_alive_interval: Union[int, None] = ...,

@ceng-p
Copy link

ceng-p commented Mar 7, 2022

Could you potentially extend this to cover comsumer client as well?

@yunhaoling
Copy link
Contributor Author

hey @98christianp, may I ask why would you like the support for EventHubConsumerClient?

The EventHubConsumerClient is supposed to be running forever which should not encounter the issue.
Do you observe any issue on your side?

@ceng-p
Copy link

ceng-p commented Apr 27, 2022

hey @98christianp, may I ask why would you like the support for EventHubConsumerClient?

The EventHubConsumerClient is supposed to be running forever which should not encounter the issue. Do you observe any issue on your side?

Thanks for your reply @yunhaoling
Yes I am, seems like the connection is closed and not able to reconnect after some time.

2022-04-27 11:55:44,465-INFO-link.py-detach:276-An error occurred when detaching the link: AMQPConnectionError('Error condition: ErrorCondition.SocketError\n Error Description: Can not send empty frame due to exception: [Errno 104] Connection reset by peer')
2022-04-27 11:55:44,465-INFO-link.py-_set_state:144-Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACHED: 0>
2022-04-27 11:55:44,465-INFO-link.py-detach:276-An error occurred when detaching the link: AMQPConnectionError('Error condition: ErrorCondition.SocketError\n Error Description: Can not send empty frame due to exception: [Errno 104] Connection reset by peer')
2022-04-27 11:55:44,465-INFO-link.py-_set_state:144-Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACHED: 0>
2022-04-27 11:55:44,465-INFO-management_link.py-_on_receiver_state_change:97-Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACHED: 0>
2022-04-27 11:55:44,465-INFO-link.py-detach:276-An error occurred when detaching the link: AMQPConnectionError('Error condition: ErrorCondition.SocketError\n Error Description: Can not send empty frame due to exception: [Errno 104] Connection reset by peer')
2022-04-27 11:55:44,465-INFO-link.py-_set_state:144-Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACHED: 0>
2022-04-27 11:55:44,465-INFO-management_link.py-_on_sender_state_change:72-Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACHED: 0>
2022-04-27 11:55:44,465-INFO-session.py-end:345-An error occurred when ending the session: AMQPConnectionError('Error condition: ErrorCondition.SocketError\n Error Description: Can not send empty frame due to exception: [Errno 104] Connection reset by peer')
2022-04-27 11:55:44,465-INFO-session.py-_set_state:98-Session state changed: <SessionState.MAPPED: 3> -> <SessionState.UNMAPPED: 0>
2022-04-27 11:55:44,465-INFO-_connection.py-close:743-An error occurred when closing the connection: AMQPConnectionError('Error condition: ErrorCondition.SocketError\n Error Description: Can not send empty frame due to exception: [Errno 104] Connection reset by peer')
2022-04-27 11:55:44,465-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: <ConnectionState.OPENED: 9> -> <ConnectionState.END: 13>
2022-04-27 11:55:44,526-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: None -> <ConnectionState.START: 0>
2022-04-27 11:55:44,531-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: <ConnectionState.START: 0> -> <ConnectionState.HDR_SENT: 2>
2022-04-27 11:55:44,531-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: <ConnectionState.HDR_SENT: 2> -> <ConnectionState.HDR_SENT: 2>
2022-04-27 11:55:44,532-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: <ConnectionState.HDR_SENT: 2> -> <ConnectionState.OPEN_PIPE: 4>
2022-04-27 11:55:44,532-INFO-session.py-_set_state:98-Session state changed: <SessionState.UNMAPPED: 0> -> <SessionState.BEGIN_SENT: 1>
2022-04-27 11:55:44,532-INFO-link.py-_set_state:144-Link state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2022-04-27 11:55:44,532-INFO-management_link.py-_on_receiver_state_change:97-Management link receiver state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2022-04-27 11:55:44,532-INFO-link.py-_set_state:144-Link state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2022-04-27 11:55:44,532-INFO-management_link.py-_on_sender_state_change:72-Management link sender state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2022-04-27 11:55:44,533-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: <ConnectionState.OPEN_PIPE: 4> -> <ConnectionState.OPEN_SENT: 7>
2022-04-27 11:55:44,583-INFO-_connection.py-_set_state:134-Connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' state changed: <ConnectionState.OPEN_SENT: 7> -> <ConnectionState.OPENED: 9>
2022-04-27 11:55:44,634-INFO-session.py-_set_state:98-Session state changed: <SessionState.BEGIN_SENT: 1> -> <SessionState.MAPPED: 3>
2022-04-27 11:55:44,684-INFO-link.py-_set_state:144-Link state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2022-04-27 11:55:44,684-INFO-management_link.py-_on_receiver_state_change:97-Management link receiver state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2022-04-27 11:55:44,734-INFO-link.py-_set_state:144-Link state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2022-04-27 11:55:44,734-INFO-management_link.py-_on_sender_state_change:72-Management link sender state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2022-04-27 11:55:44,735-INFO-cbs.py-_on_amqp_management_open_complete:102-CBS for connection 'EHConsumer-0b653ddd-2dff-426a-8b8d-229601c3511c-partition4' completed opening with status: <ManagementOpenResult.OK: 1>
2022-04-27 11:55:44,886-INFO-cbs.py-_on_execute_operation_complete:125-CBS Put token result (<ManagementExecuteOperationResult.OK: 0>), status code: 202, status_description: b'Accepted'.
2022-04-27 11:55:44,936-INFO-link.py-_set_state:144-Link state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2022-04-27 11:55:45,185-INFO-link.py-_incoming_attach:194-Cannot get source or target. Detaching link

This puts my process into a permenant loop where it tries to connect and ends with "Cannot get source or target. Detaching link"

@ghost ghost added the no-recent-activity There has been no recent activity on this issue. label Jul 1, 2022
@ghost
Copy link

ghost commented Jul 1, 2022

Hi @yunhaoling. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

@ghost ghost closed this Jul 8, 2022
@ghost
Copy link

ghost commented Jul 8, 2022

Hi @yunhaoling. Thank you for your contribution. Since there hasn't been recent engagement, we're going to close this out. Feel free to respond with a comment containing "/reopen" if you'd like to continue working on these changes. Please be sure to use the command to reopen or remove the "no-recent-activity" label; otherwise, this is likely to be closed again with the next cleanup pass.

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@annatisch annatisch Awaiting requested review from annatisch annatisch will be requested when the pull request is marked ready for review annatisch is a code owner

@kashifkhan kashifkhan Awaiting requested review from kashifkhan kashifkhan will be requested when the pull request is marked ready for review kashifkhan is a code owner

@swathipil swathipil Awaiting requested review from swathipil swathipil will be requested when the pull request is marked ready for review swathipil is a code owner

@l0lawrence l0lawrence Awaiting requested review from l0lawrence l0lawrence will be requested when the pull request is marked ready for review l0lawrence is a code owner

Assignees
No one assigned
Labels
Event Hubs no-recent-activity There has been no recent activity on this issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yunhaoling @azure-sdk @ceng-p