Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Storage] [Datalake] Fixed next pylint errors in datalake storage #35167

Merged
merged 6 commits into from
Apr 21, 2024
Merged

[Storage] [Datalake] Fixed next pylint errors in datalake storage #35167

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f573e7b
[Storage] [Datalake] Fixed next pylint errors in file datalake
weirongw23-msft Apr 9, 2024
86abc21
fixed rest of next pylint errors
weirongw23-msft Apr 11, 2024
8a68c7b
resolved merge conflicts and adopted new type docstring conventions
weirongw23-msft Apr 18, 2024
6a3197f
fixed pylint issues after merge conflicts
weirongw23-msft Apr 18, 2024
0661fb4
aesthetic fixes to be consistent with main branch docstring conventions
weirongw23-msft Apr 18, 2024
9c4ff2c
added back content_settings and metadata keywords
weirongw23-msft Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions ...age/azure-storage-file-datalake/azure/storage/filedatalake/_data_lake_directory_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,8 @@ def delete_directory(self, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: None
:returns: None.
:rtype: None

.. admonition:: Example:

Expand Down Expand Up @@ -303,7 +304,10 @@ def get_directory_properties(self, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:rtype: DirectoryProperties
:returns:
DirectoryProperties with all user-defined metadata, standard HTTP properties,
and system properties for the directory. It does not return the content of the directory.
:rtype: ~azure.storage.filedatalake.DirectoryProperties

.. admonition:: Example:

Expand Down Expand Up @@ -396,7 +400,8 @@ def rename_directory(self, new_name, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeDirectoryClient
:returns: A DataLakeDirectoryClient with the renamed directory.
:rtype: ~azure.storage.filedatalake.DataLakeDirectoryClient

.. admonition:: Example:

Expand Down Expand Up @@ -496,7 +501,8 @@ def create_sub_directory(self, sub_directory, # type: Union[DirectoryProperties
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeDirectoryClient for the subdirectory.
:returns: DataLakeDirectoryClient for the subdirectory.
:rtype: ~azure.storage.filedatalake.DataLakeDirectoryClient
"""
subdir = self.get_sub_directory_client(sub_directory)
subdir.create_directory(metadata=metadata, **kwargs)
Expand Down Expand Up @@ -540,7 +546,8 @@ def delete_sub_directory(self, sub_directory, # type: Union[DirectoryProperties
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeDirectoryClient for the subdirectory
:returns: DataLakeDirectoryClient for the subdirectory.
:rtype: ~azure.storage.filedatalake.DataLakeDirectoryClient
"""
subdir = self.get_sub_directory_client(sub_directory)
subdir.delete_directory(**kwargs)
Expand Down Expand Up @@ -632,7 +639,8 @@ def create_file(self, file, # type: Union[FileProperties, str]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeFileClient
:returns: A DataLakeFileClient with newly created file.
:rtype: ~azure.storage.filedatalake.DataLakeFileClient
"""
file_client = self.get_file_client(file)
file_client.create_file(**kwargs)
Expand Down
24 changes: 16 additions & 8 deletions sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_data_lake_file_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,9 +213,10 @@ def create_file(self, content_settings=None, # type: Optional[ContentSettings]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: response dict (Etag and last modified).
:keyword str encryption_context:
Specifies the encryption context to set on the file.
:returns: response dict (Etag and last modified).
:rtype: dict[str, str] or dict[str, ~datetime.datetime]

.. admonition:: Example:

Expand Down Expand Up @@ -261,7 +262,8 @@ def delete_file(self, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: None
:returns: None.
:rtype: None

.. admonition:: Example:

Expand Down Expand Up @@ -426,6 +428,7 @@ def upload_data(
Upload data to a file.

:param data: Content to be uploaded to file
:type data: bytes, str, Iterable[AnyStr], or IO[AnyStr]
:param int length: Size of the data in bytes.
:param bool overwrite: to overwrite an existing file or not.
:keyword ~azure.storage.filedatalake.ContentSettings content_settings:
Expand Down Expand Up @@ -490,7 +493,8 @@ def upload_data(
Defaults to 100*1024*1024, or 100MB.
:keyword str encryption_context:
Specifies the encryption context to set on the file.
:return: response dict (Etag and last modified).
:returns: response dict (Etag and last modified).
:rtype: dict[str, Any]
"""
options = self._upload_options(
data,
Expand Down Expand Up @@ -541,8 +545,10 @@ def append_data(self, data, # type: Union[bytes, str, Iterable[AnyStr], IO[AnyS
"""Append data to the file.

:param data: Content to be appended to file
:param offset: start position of the data to be appended to.
:type data: bytes, str, Iterable[AnyStr], or IO[AnyStr]
:param int offset: start position of the data to be appended to.
:param length: Size of the data in bytes.
:type length: int or None
:keyword bool flush:
If true, will commit the data after it is appended.
:keyword bool validate_content:
Expand Down Expand Up @@ -576,7 +582,8 @@ def append_data(self, data, # type: Union[bytes, str, Iterable[AnyStr], IO[AnyS
:keyword ~azure.storage.filedatalake.CustomerProvidedEncryptionKey cpk:
Encrypts the data on the service-side with the given key.
Use of customer-provided keys must be done over HTTPS.
:return: dict of the response header
:returns: dict of the response header.
:rtype: dict[str, str], dict[str, ~datetime.datetime], or dict[str, int]

.. admonition:: Example:

Expand Down Expand Up @@ -637,8 +644,8 @@ def flush_data(self, offset, # type: int
# type: (...) -> Dict[str, Union[str, datetime]]
""" Commit the previous appended data.

:param offset: offset is equal to the length of the file after commit the
previous appended data.
:param int offset: offset is equal to the length of the file after commit
the previous appended data.
:param bool retain_uncommitted_data: Valid only for flush operations. If
"true", uncommitted data is retained after the flush operation
completes; otherwise, the uncommitted data is deleted after the flush
Expand Down Expand Up @@ -703,7 +710,8 @@ def flush_data(self, offset, # type: int
:keyword ~azure.storage.filedatalake.CustomerProvidedEncryptionKey cpk:
Encrypts the data on the service-side with the given key.
Use of customer-provided keys must be done over HTTPS.
:return: response header in dict
:returns: response header in dict
:rtype: dict[str, str] or dict[str, ~datetime.datetime]

.. admonition:: Example:

Expand Down
5 changes: 4 additions & 1 deletion ...orage/azure-storage-file-datalake/azure/storage/filedatalake/_data_lake_service_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,7 @@ def create_file_system(self, file_system, # type: Union[FileSystemProperties, s
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:returns: A FileSystemClient with newly created file system.
:rtype: ~azure.storage.filedatalake.FileSystemClient

.. admonition:: Example:
Expand Down Expand Up @@ -329,6 +330,7 @@ def _rename_file_system(self, name, new_name, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:returns: A FileSystemClient with the specified file system renamed.
:rtype: ~azure.storage.filedatalake.FileSystemClient
"""
self._blob_service_client._rename_container(name, new_name, **kwargs) # pylint: disable=protected-access
Expand Down Expand Up @@ -406,7 +408,8 @@ def delete_file_system(self, file_system, # type: Union[FileSystemProperties, s
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:rtype: None
:returns: A FileSystemClient with the specified file system deleted.
:rtype: ~azure.storage.filedatalake.FileSystemClient

.. admonition:: Example:

Expand Down
18 changes: 9 additions & 9 deletions sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_file_system_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,7 @@ def _rename_file_system(self, new_name, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:returns: FileSystemClient with renamed properties.
:rtype: ~azure.storage.filedatalake.FileSystemClient
"""
self._container_client._rename_container(new_name, **kwargs) # pylint: disable=protected-access
Expand Down Expand Up @@ -674,7 +675,8 @@ def create_directory(self, directory, # type: Union[DirectoryProperties, str]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeDirectoryClient
:returns: DataLakeDirectoryClient with new directory and metadata.
:rtype: ~azure.storage.file.datalake.DataLakeDirectoryClient

.. admonition:: Example:

Expand Down Expand Up @@ -727,7 +729,8 @@ def delete_directory(self, directory, # type: Union[DirectoryProperties, str]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeDirectoryClient
:returns: DataLakeDirectoryClient after deleting specified directory.
:rtype: ~azure.storage.file.datalake.DataLakeDirectoryClient

.. admonition:: Example:

Expand All @@ -753,11 +756,6 @@ def create_file(self, file, # type: Union[FileProperties, str]
The file with which to interact. This can either be the name of the file,
or an instance of FileProperties.
:type file: str or ~azure.storage.filedatalake.FileProperties
:param ~azure.storage.filedatalake.ContentSettings content_settings:
ContentSettings object used to set path properties.
:param metadata:
Name-value pairs associated with the file as metadata.
:type metadata: dict(str, str)
weirongw23-msft marked this conversation as resolved.
Show resolved Hide resolved
:keyword lease:
Required if the file has an active lease. Value can be a DataLakeLeaseClient object
or the lease ID as a string.
Expand Down Expand Up @@ -825,7 +823,8 @@ def create_file(self, file, # type: Union[FileProperties, str]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeFileClient
:returns: DataLakeFileClient with new file created.
:rtype: ~azure.storage.file.datalake.DataLakeFileClient

.. admonition:: Example:

Expand Down Expand Up @@ -878,7 +877,8 @@ def delete_file(self, file, # type: Union[FileProperties, str]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:return: DataLakeFileClient
:return: DataLakeFileClient after deleting specified file.
:rtype: azure.storage.file.datalake.DataLakeFileClient

.. admonition:: Example:

Expand Down
17 changes: 12 additions & 5 deletions sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_path_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,8 @@ def set_access_control(self, owner=None, # type: Optional[str]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:keyword: response dict (Etag and last modified).
:returns: response dict containing access control options (Etag and last modified).
:rtype: dict[str, str] or dict[str, ~datetime.datetime]
"""
if not any([owner, group, permissions, acl]):
raise ValueError("At least one parameter should be set for set_access_control API")
Expand Down Expand Up @@ -519,7 +520,8 @@ def get_access_control(self, upn=None, # type: Optional[bool]
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:keyword: response dict.
:returns: response dict containing access control options with no modifications.
:rtype: dict[str, Any]
"""
options = self._get_access_control_options(upn=upn, **kwargs)
try:
Expand Down Expand Up @@ -583,7 +585,7 @@ def set_access_control_recursive(self, acl, **kwargs):
#other-client--per-operation-configuration>`_.
:return: A summary of the recursive operations, including the count of successes and failures,
as well as a continuation token in case the operation was terminated prematurely.
:rtype: :class:`~azure.storage.filedatalake.AccessControlChangeResult`
:rtype: ~azure.storage.filedatalake.AccessControlChangeResult
:raises ~azure.core.exceptions.AzureError:
User can restart the operation using continuation_token field of AzureError if the token is available.
"""
Expand Down Expand Up @@ -637,7 +639,7 @@ def update_access_control_recursive(self, acl, **kwargs):
#other-client--per-operation-configuration>`_.
:return: A summary of the recursive operations, including the count of successes and failures,
as well as a continuation token in case the operation was terminated prematurely.
:rtype: :class:`~azure.storage.filedatalake.AccessControlChangeResult`
:rtype: ~azure.storage.filedatalake.AccessControlChangeResult
:raises ~azure.core.exceptions.AzureError:
User can restart the operation using continuation_token field of AzureError if the token is available.
"""
Expand Down Expand Up @@ -690,7 +692,7 @@ def remove_access_control_recursive(self, acl, **kwargs):
#other-client--per-operation-configuration>`_.
:return: A summary of the recursive operations, including the count of successes and failures,
as well as a continuation token in case the operation was terminated prematurely.
:rtype: :class:`~azure.storage.filedatalake.AccessControlChangeResult`
:rtype: ~azure.storage.filedatalake.AccessControlChangeResult
:raises ~azure.core.exceptions.AzureError:
User can restart the operation using continuation_token field of AzureError if the token is available.
"""
Expand Down Expand Up @@ -879,6 +881,8 @@ def _rename_path(self, rename_source, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:returns: response dict containing information about the renamed path.
:rtype: dict[str, Any]
"""
options = self._rename_path_options(
rename_source,
Expand Down Expand Up @@ -933,6 +937,9 @@ def _get_path_properties(self, **kwargs):
This value is not tracked or validated on the client. To configure client-side network timesouts
see `here <https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-file-datalake
#other-client--per-operation-configuration>`_.
:returns:
Information including user-defined metadata, standard HTTP properties,
and system properties for the file or directory.
:rtype: DirectoryProperties or FileProperties

.. admonition:: Example:
Expand Down
7 changes: 3 additions & 4 deletions sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_shared/policies.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import uuid
from io import SEEK_SET, UnsupportedOperation
from time import time
from typing import Any, Dict, Optional, TYPE_CHECKING, Union
from typing import Any, Dict, Optional, TYPE_CHECKING
from urllib.parse import (
parse_qsl,
urlencode,
Expand Down Expand Up @@ -411,9 +411,8 @@ def _set_next_host_location(self, settings: Dict[str, Any], request: "PipelineRe
"""
A function which sets the next host location on the request, if applicable.

:param dict[str, Any]] settings: The configurable values pertaining to the next host location.
:param request: A pipeline request object.
:type request: ~azure.core.pipeline.PipelineRequest
:param Dict[str, Any]] settings: The configurable values pertaining to the next host location.
:param PipelineRequest request: A pipeline request object.
"""
if settings['hosts'] and all(settings['hosts'].values()):
url = urlparse(request.url)
Expand Down
Loading