[ServiceBus] Resetting TCL opts in async transport with custom endpoint causes error

[swathipil]

To repro:

• Run the any async sample in parallel with the amqp proxy. This requires setting SSLContext cert_verify to None and local custom endpoint.
• We see the following error:

Traceback (most recent call last):
  File "/home/swathip/repos/priv-amqpfaultinjector/samples/faultinjectorsample_python/auto_lock_renew_async.py", line 85, in <module>
    asyncio.run(renew_lock_on_message_received_from_non_sessionful_entity())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/swathip/repos/priv-amqpfaultinjector/samples/faultinjectorsample_python/auto_lock_renew_async.py", line 60, in renew_lock_on_message_received_from_non_sessionful_entity
    async with servicebus_client.get_queue_sender(queue_name=QUEUE_NAME) as sender:
  File "/home/swathip/repos/azure-sdk-for-python/sdk/servicebus/azure-servicebus/azure/servicebus/aio/_servicebus_sender_async.py", line 155, in __aenter__
    await self._open_with_retry()
  File "/home/swathip/repos/azure-sdk-for-python/sdk/servicebus/azure-servicebus/azure/servicebus/aio/_base_handler_async.py", line 386, in _open_with_retry
    return await self._do_retryable_operation(self._open)
  File "/home/swathip/repos/azure-sdk-for-python/sdk/servicebus/azure-servicebus/azure/servicebus/aio/_base_handler_async.py", line 261, in _do_retryable_operation
    raise last_exception from None
azure.servicebus.exceptions.ServiceBusConnectionError: Failed to initiate the connection due to exception: [Errno 22] Invalid argument Error condition: amqp:socket-error.
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f9f444c3730>
transport: <_SelectorSocketTransport closing fd=7>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f9f43f6b5e0>
transport: <_SelectorSocketTransport closing fd=8>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f9f43f6b070>
transport: <_SelectorSocketTransport closing fd=9>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f9f43f68790>
transport: <_SelectorSocketTransport closing fd=10>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

• This error does not happen when [this line] is commented out.

TODO:

• also investigate why stack trace doesn't show point of failure/error