Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Types in poller #29228

Merged
merged 60 commits into from
Jul 6, 2023
Merged
Show file tree
Hide file tree
Changes from 57 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
cfafd30
Types in poller
lmazuel Mar 8, 2023
dbe3d32
Black it right
lmazuel Mar 17, 2023
6fc87de
Saving this week's work
lmazuel Mar 18, 2023
1facfcf
Move typing
lmazuel Mar 20, 2023
7f7f560
Split base polling in two
lmazuel Mar 27, 2023
3f041d8
Merge branch 'main' into polling_types
lmazuel Mar 27, 2023
7d0acfb
Merge branch 'main' into polling_types
lmazuel Apr 14, 2023
1316897
Typing fixes
lmazuel Apr 14, 2023
f3c42e9
Typing update
lmazuel Apr 14, 2023
bfd8432
Black
lmazuel Apr 14, 2023
bae1d5f
Unecessary Generic
lmazuel Apr 14, 2023
e494ee0
Stringify types
lmazuel Apr 14, 2023
c7d8aa8
Fix import
lmazuel Apr 15, 2023
bbe53f2
Spellcheck
lmazuel Apr 15, 2023
e39cccd
Weird typo...
lmazuel Apr 15, 2023
61e1371
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Apr 21, 2023
03a6d5f
PyLint
lmazuel Apr 21, 2023
bd00b83
More types
lmazuel Apr 22, 2023
e149c70
Update sdk/core/azure-core/azure/core/polling/async_base_polling.py
lmazuel May 2, 2023
368572e
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel May 4, 2023
ef9f186
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel May 22, 2023
bdff940
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel May 23, 2023
11a36f2
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel May 24, 2023
27a6045
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel May 24, 2023
34a6340
Missing type
lmazuel May 24, 2023
80b422c
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Jun 13, 2023
6465b24
Typing of the day
lmazuel Jun 15, 2023
17787c3
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Jun 17, 2023
2a5f3ac
Re-enable verifytypes
lmazuel Jun 17, 2023
b3fb232
Simplify the expectations async pipeline has on the response
lmazuel Jun 19, 2023
4f94c88
Async Cxt Manager
lmazuel Jun 19, 2023
ca61aa0
Final Typing?
lmazuel Jun 20, 2023
c82c946
More covariant
lmazuel Jun 20, 2023
4dd11c7
Upside down
lmazuel Jun 20, 2023
a049070
Fix tests
lmazuel Jun 20, 2023
e55ffbd
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Jun 20, 2023
bf3aa73
Messed up merge
lmazuel Jun 20, 2023
7644c15
Pylint
lmazuel Jun 20, 2023
f4f825e
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Jun 21, 2023
d229aae
Better Typing
lmazuel Jun 22, 2023
7a8c137
Final typing?
lmazuel Jun 22, 2023
48872c9
Pylint
lmazuel Jun 22, 2023
36e1e78
Simplify translation typing for now
lmazuel Jun 22, 2023
33e6125
Fix backcompat with azure-mgmt-core
lmazuel Jun 23, 2023
8aea6b1
Revert renaming private methods
lmazuel Jun 23, 2023
aa9c89c
Black
lmazuel Jun 23, 2023
188d7ab
Feedback from @kristapratico
lmazuel Jun 26, 2023
dee0db5
Docstrings part 1
lmazuel Jun 26, 2023
9127c89
Polling pylint part 2
lmazuel Jun 26, 2023
1467a21
Black
lmazuel Jun 26, 2023
5b83d9c
All LRO impl should use TypeVar
lmazuel Jun 29, 2023
f4fde67
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Jun 29, 2023
f559db9
Feedback
lmazuel Jul 5, 2023
5035836
Convert some Anyu after feedback
lmazuel Jul 5, 2023
a7ab020
Merge remote-tracking branch 'origin/main' into polling_types
lmazuel Jul 5, 2023
e7c52a0
Spellcheck
lmazuel Jul 5, 2023
3501c22
Black
lmazuel Jul 6, 2023
c5b06c1
Update sdk/core/azure-core/azure/core/polling/_async_poller.py
kashifkhan Jul 6, 2023
d620445
Update sdk/core/azure-core/azure/core/polling/_async_poller.py
kashifkhan Jul 6, 2023
c7a6d93
Update sdk/core/azure-core/azure/core/polling/_poller.py
kashifkhan Jul 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .vscode/cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@
"ints",
"iohttp",
"IOHTTP",
"IOLRO",
"inprogress",
"ipconfiguration",
"ipconfigurations",
Expand Down
22 changes: 3 additions & 19 deletions sdk/core/azure-core/azure/core/_pipeline_client_async.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,7 @@
Generic,
Optional,
cast,
TYPE_CHECKING,
)
from typing_extensions import Protocol
from .configuration import Configuration
from .pipeline import AsyncPipeline
from .pipeline.transport._base import PipelineClientBase
Expand All @@ -51,17 +49,8 @@
)


if TYPE_CHECKING: # Protocol and non-Protocol can't mix in Python 3.7

class _AsyncContextManagerCloseable(AsyncContextManager, Protocol):
"""Defines a context manager that is closeable at the same time."""

async def close(self):
...


HTTPRequestType = TypeVar("HTTPRequestType")
AsyncHTTPResponseType = TypeVar("AsyncHTTPResponseType", bound="_AsyncContextManagerCloseable")
AsyncHTTPResponseType = TypeVar("AsyncHTTPResponseType", bound="AsyncContextManager")
lmazuel marked this conversation as resolved.
Show resolved Hide resolved

_LOGGER = logging.getLogger(__name__)

Expand All @@ -80,11 +69,9 @@ class _Coroutine(Awaitable[AsyncHTTPResponseType]):
This allows the dev to either use the "async with" syntax, or simply the object directly.
It's also why "send_request" is not declared as async, since it couldn't be both easily.

"wrapped" must be an awaitable that returns an object that:
- has an async "close()"
- has an "__aexit__" method (IOW, is an async context manager)
"wrapped" must be an awaitable object that returns an object implements the async context manager protocol.

This permits this code to work for both requests.
This permits this code to work for both following requests.

```python
from azure.core import AsyncPipelineClient
Expand Down Expand Up @@ -124,9 +111,6 @@ async def __aenter__(self) -> AsyncHTTPResponseType:
async def __aexit__(self, *args) -> None:
await self._response.__aexit__(*args)

async def close(self) -> None:
await self._response.close()


class AsyncPipelineClient(
PipelineClientBase,
Expand Down
4 changes: 2 additions & 2 deletions sdk/core/azure-core/azure/core/pipeline/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@

from typing import TypeVar, Generic, Dict, Any

HTTPResponseType = TypeVar("HTTPResponseType")
HTTPRequestType = TypeVar("HTTPRequestType")
HTTPResponseType = TypeVar("HTTPResponseType", covariant=True)
HTTPRequestType = TypeVar("HTTPRequestType", covariant=True)


class PipelineContext(Dict[str, Any]):
Expand Down
37 changes: 8 additions & 29 deletions sdk/core/azure-core/azure/core/pipeline/policies/_universal.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,46 +35,25 @@
import types
import re
import uuid
from typing import IO, cast, Union, Optional, AnyStr, Dict, MutableMapping, Any, Set, Mapping
from typing import IO, cast, Union, Optional, AnyStr, Dict, Any, Set, Mapping
import urllib.parse
from typing_extensions import Protocol

from azure.core import __version__ as azcore_version
from azure.core.exceptions import DecodeError

from azure.core.pipeline import PipelineRequest, PipelineResponse
from ._base import SansIOHTTPPolicy

from ..transport import HttpRequest as LegacyHttpRequest
from ..transport._base import _HttpResponseBase as LegacySansIOHttpResponse
from ...rest import HttpRequest
from ...rest._rest_py3 import _HttpResponseBase as SansIOHttpResponse

_LOGGER = logging.getLogger(__name__)


class HTTPRequestType(Protocol):
"""Protocol compatible with new rest request and legacy transport request"""

headers: MutableMapping[str, str]
url: str
method: str
body: bytes


class HTTPResponseType(Protocol):
"""Protocol compatible with new rest response and legacy transport response"""

@property
def headers(self) -> MutableMapping[str, str]:
...

@property
def status_code(self) -> int:
...

@property
def content_type(self) -> Optional[str]:
...

def text(self, encoding: Optional[str] = None) -> str:
...
HTTPRequestType = Union[LegacyHttpRequest, HttpRequest]
HTTPResponseType = Union[LegacySansIOHttpResponse, SansIOHttpResponse]
PipelineResponseType = PipelineResponse[HTTPRequestType, HTTPResponseType]


class HeadersPolicy(SansIOHTTPPolicy[HTTPRequestType, HTTPResponseType]):
Expand Down
14 changes: 10 additions & 4 deletions sdk/core/azure-core/azure/core/pipeline/policies/_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,28 +25,34 @@
# --------------------------------------------------------------------------
import datetime
import email.utils
from typing import Optional, cast

from urllib.parse import urlparse
from ...utils._utils import _FixedOffset, case_insensitive_dict


def _parse_http_date(text):
def _parse_http_date(text: str) -> datetime.datetime:
"""Parse a HTTP date format into datetime.

:param str text: Text containing a date in HTTP format
:rtype: datetime.datetime
:return: The parsed datetime
"""
parsed_date = email.utils.parsedate_tz(text)
return datetime.datetime(*parsed_date[:6], tzinfo=_FixedOffset(parsed_date[9] / 60))
if not parsed_date:
raise ValueError("Invalid HTTP date")
tz_offset = cast(int, parsed_date[9]) # Look at the code, tz_offset is always an int, at worst 0
return datetime.datetime(*parsed_date[:6], tzinfo=_FixedOffset(tz_offset / 60))


def parse_retry_after(retry_after: str):
def parse_retry_after(retry_after: str) -> float:
"""Helper to parse Retry-After and get value in seconds.

:param str retry_after: Retry-After header
:rtype: float
lmazuel marked this conversation as resolved.
Show resolved Hide resolved
:return: Value of Retry-After in seconds.
"""
delay: float # Using the Mypy recommendation to use float for "int or float"
try:
delay = int(retry_after)
except ValueError:
Expand All @@ -56,7 +62,7 @@ def parse_retry_after(retry_after: str):
return max(0, delay)


def get_retry_after(response):
def get_retry_after(response) -> Optional[float]:
"""Get the value of Retry-After in seconds.

:param response: The PipelineResponse object
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def _iterate_response_content(iterator):
raise _ResponseStopIteration() # pylint: disable=raise-missing-from


class AsyncHttpResponse(_HttpResponseBase): # pylint: disable=abstract-method
class AsyncHttpResponse(_HttpResponseBase, AbstractAsyncContextManager): # pylint: disable=abstract-method
"""An AsyncHttpResponse ABC.

Allows for the asynchronous streaming of data from the response.
Expand Down Expand Up @@ -93,6 +93,9 @@ def parts(self) -> AsyncIterator:

return _PartGenerator(self, default_http_response_type=AsyncHttpClientTransportResponse)

async def __aexit__(self, exc_type, exc_value, traceback):
return None


class AsyncHttpClientTransportResponse( # pylint: disable=abstract-method
_HttpClientTransportResponse, AsyncHttpResponse
Expand Down
18 changes: 15 additions & 3 deletions sdk/core/azure-core/azure/core/polling/_async_poller.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,12 @@ async def run(self): # pylint:disable=invalid-overridden-method
"""


async def async_poller(client, initial_response, deserialization_callback, polling_method):
async def async_poller(
client: Any,
initial_response: Any,
deserialization_callback: Callable[[Any], PollingReturnType_co],
polling_method: AsyncPollingMethod[PollingReturnType_co],
) -> PollingReturnType_co:
"""Async Poller for long running operations.

.. deprecated:: 1.5.0
Expand All @@ -86,6 +91,8 @@ async def async_poller(client, initial_response, deserialization_callback, polli
:type deserialization_callback: callable or msrest.serialization.Model
:param polling_method: The polling strategy to adopt
:type polling_method: ~azure.core.polling.PollingMethod
:returns: The final resource at the end of the polling.
kashifkhan marked this conversation as resolved.
Show resolved Hide resolved
:rtype: any or None
"""
poller = AsyncLROPoller(client, initial_response, deserialization_callback, polling_method)
return await poller
Expand All @@ -109,7 +116,7 @@ def __init__(
self,
client: Any,
initial_response: Any,
deserialization_callback: Callable,
deserialization_callback: Callable[[Any], PollingReturnType_co],
polling_method: AsyncPollingMethod[PollingReturnType_co],
):
self._polling_method = polling_method
Expand All @@ -124,7 +131,11 @@ def __init__(
self._polling_method.initialize(client, initial_response, deserialization_callback)

def polling_method(self) -> AsyncPollingMethod[PollingReturnType_co]:
"""Return the polling method associated to this poller."""
"""Return the polling method associated to this poller.

:returns: The polling method associated to this poller.
kashifkhan marked this conversation as resolved.
Show resolved Hide resolved
:rtype: ~azure.core.polling.AsyncPollingMethod
"""
return self._polling_method

def continuation_token(self) -> str:
Expand Down Expand Up @@ -158,6 +169,7 @@ async def result(self) -> PollingReturnType_co:
"""Return the result of the long running operation.

:returns: The deserialized resource of the long running operation, if one is available.
:rtype: any or None
:raises ~azure.core.exceptions.HttpResponseError: Server problem with the query.
"""
await self.wait()
Expand Down
46 changes: 29 additions & 17 deletions sdk/core/azure-core/azure/core/polling/_poller.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,35 +64,44 @@ def from_continuation_token(cls, continuation_token: str, **kwargs) -> Tuple[Any
raise TypeError("Polling method '{}' doesn't support from_continuation_token".format(cls.__name__))


class NoPolling(PollingMethod):
class NoPolling(PollingMethod[PollingReturnType_co]):
"""An empty poller that returns the deserialized initial response."""

_deserialization_callback: Callable[[Any], PollingReturnType_co]
"""Deserialization callback passed during initialization"""

def __init__(self):
self._initial_response = None
self._deserialization_callback = None

def initialize(self, _: Any, initial_response: Any, deserialization_callback: Callable) -> None:
def initialize(
self,
_: Any,
initial_response: Any,
deserialization_callback: Callable[[Any], PollingReturnType_co],
) -> None:
self._initial_response = initial_response
self._deserialization_callback = deserialization_callback

def run(self) -> None:
"""Empty run, no polling."""

def status(self) -> str:
"""Return the current status as a string.
"""Return the current status.

:rtype: str
:return: The current status
"""
return "succeeded"

def finished(self) -> bool:
"""Is this polling finished?

:rtype: bool
:return: Whether this polling is finished
"""
return True

def resource(self) -> Any:
def resource(self) -> PollingReturnType_co:
return self._deserialization_callback(self._initial_response)

def get_continuation_token(self) -> str:
Expand All @@ -105,7 +114,7 @@ def from_continuation_token(cls, continuation_token: str, **kwargs) -> Tuple[Any
try:
deserialization_callback = kwargs["deserialization_callback"]
except KeyError:
raise ValueError("Need kwarg 'deserialization_callback' to be recreated from continuation_token")
raise ValueError("Need kwarg 'deserialization_callback' to be recreated from continuation_token") from None
import pickle

initial_response = pickle.loads(base64.b64decode(continuation_token)) # nosec
Expand All @@ -130,7 +139,7 @@ def __init__(
self,
client: Any,
initial_response: Any,
deserialization_callback: Callable,
deserialization_callback: Callable[[Any], PollingReturnType_co],
polling_method: PollingMethod[PollingReturnType_co],
) -> None:
self._callbacks: List[Callable] = []
Expand All @@ -147,10 +156,11 @@ def __init__(

# Prepare thread execution
self._thread = None
self._done = None
self._done = threading.Event()
self._exception = None
if not self._polling_method.finished():
self._done = threading.Event()
if self._polling_method.finished():
self._done.set()
else:
self._thread = threading.Thread(
target=with_current_context(self._start),
name="LROPoller({})".format(uuid.uuid4()),
Expand All @@ -161,9 +171,6 @@ def __init__(
def _start(self):
"""Start the long running operation.
On completion, runs any callbacks.

:param callable update_cmd: The API request to check the status of
the operation.
"""
try:
self._polling_method.run()
Expand All @@ -189,7 +196,11 @@ def _start(self):
callbacks, self._callbacks = self._callbacks, []

def polling_method(self) -> PollingMethod[PollingReturnType_co]:
"""Return the polling method associated to this poller."""
"""Return the polling method associated to this poller.

:returns: The polling method
kashifkhan marked this conversation as resolved.
Show resolved Hide resolved
:rtype: ~azure.core.polling.PollingMethod
"""
return self._polling_method

def continuation_token(self) -> str:
Expand Down Expand Up @@ -223,8 +234,9 @@ def result(self, timeout: Optional[float] = None) -> PollingReturnType_co:
"""Return the result of the long running operation, or
the result available after the specified timeout.

:returns: The deserialized resource of the long running operation,
if one is available.
:param float timeout: Period of time to wait before getting back control.
:returns: The deserialized resource of the long running operation, if one is available.
:rtype: any or None
:raises ~azure.core.exceptions.HttpResponseError: Server problem with the query.
"""
self.wait(timeout)
Expand Down Expand Up @@ -266,7 +278,7 @@ def add_done_callback(self, func: Callable) -> None:
argument, a completed LongRunningOperation.
"""
# Still use "_done" and not "done", since CBs are executed inside the thread.
if self._done is None or self._done.is_set():
if self._done.is_set():
func(self._polling_method)
# Let's add them still, for consistency (if you wish to access to it for some reasons)
self._callbacks.append(func)
Expand Down
Loading