-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Storage] Added DataLake projects to Storage #7772
Merged
seanmcc-msft
merged 1 commit into
Azure:master
from
seanmcc-msft:storage/features/DataLake
Sep 26, 2019
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| Breaking Changes | ||
| ================ | ||
|
|
||
| Next Preview | ||
| -------------------------- | ||
| - New Azure.Storage.Files.DataLake client library. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| Release History | ||
| =============== | ||
|
|
||
| Next Preview | ||
| -------------------------- | ||
| This preview is the first release supporting DataLake for Azure | ||
| Storage files. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| # Azure Storage Files Data Lake client library for .NET | ||
|
|
||
| > Server Version: 2019-02-02 | ||
| Azure Data Lake includes all the capabilities required to make it easy for developers, data scientists, | ||
| and analysts to store data of any size, shape, and speed, and do all types of processing and analytics | ||
| across platforms and languages. It removes the complexities of ingesting and storing all of your data | ||
| while making it faster to get up and running with batch, streaming, and interactive analytics. | ||
|
|
||
| [Source code][source] | [Package (NuGet)][package] | [API reference documentation][docs] | [REST API documentation][rest_docs] | [Product documentation][product_docs] | ||
|
|
||
| ## Getting started | ||
|
|
||
| ### Install the package | ||
|
|
||
| Install the Azure Storage Files Data Lake client library for .NET with [NuGet][nuget]: | ||
|
|
||
| ```Powershell | ||
| dotnet add package Azure.Storage.Files.DataLake --version 12.0.0-preview.3 | ||
| ``` | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| You need an [Azure subscription][azure_sub] and a | ||
| [Storage Account][storage_account_docs] to use this package. | ||
|
|
||
| To create a new Storage Account, you can use the [Azure Portal][storage_account_create_portal], | ||
| [Azure PowerShell][storage_account_create_ps], or the [Azure CLI][storage_account_create_cli]. | ||
| Here's an example using the Azure CLI: | ||
|
|
||
| ```Powershell | ||
| az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS | ||
| ``` | ||
|
|
||
| ## Key concepts | ||
|
|
||
| TODO: Add Key Concepts | ||
|
|
||
| ## Examples | ||
|
|
||
| TODO: Add Examples | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| All File DataLake service operations will throw a | ||
| [StorageRequestFailedException][StorageRequestFailedException] on failure with | ||
| helpful [`ErrorCode`s][error_codes]. Many of these errors are recoverable. | ||
|
|
||
| TODO: Update sample | ||
|
|
||
| ```c# | ||
| string connectionString = "<connection_string>"; | ||
| // Try to create a container named "sample-container" and avoid any potential race | ||
| // conditions that might arise by checking if the container exists before creating | ||
| BlobContainerClient container = new BlobContainerClient(connectionString, "sample-container"); | ||
| try | ||
| { | ||
| container.Create(); | ||
| } | ||
| catch (StorageRequestFailedException ex) | ||
| when (ex.ErrorCode == BlobErrorCode.ContainerAlreadyExists) | ||
| { | ||
| // Ignore any errors if the container already exists | ||
| } | ||
| ``` | ||
|
|
||
| ## Next steps | ||
|
|
||
| TODO: Link Samples | ||
|
|
||
| ## Contributing | ||
|
|
||
| See the [Storage CONTRIBUTING.md][storage_contrib] for details on building, | ||
| testing, and contributing to this library. | ||
|
|
||
| This project welcomes contributions and suggestions. Most contributions require | ||
| you to agree to a Contributor License Agreement (CLA) declaring that you have | ||
| the right to, and actually do, grant us the rights to use your contribution. For | ||
| details, visit [cla.microsoft.com][cla]. | ||
|
|
||
| This project has adopted the [Microsoft Open Source Code of Conduct][coc]. | ||
| For more information see the [Code of Conduct FAQ][coc_faq] | ||
| or contact [[email protected]][coc_contact] with any | ||
| additional questions or comments. | ||
|
|
||
|  | ||
|
|
||
| <!-- LINKS --> | ||
| [source]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/storage/Azure.Storage.Files.DataLake/src | ||
| [package]: https://www.nuget.org/packages/Azure.Storage.Files.DataLake/ | ||
| [docs]: https://azure.github.io/azure-sdk-for-net/api/Azure.Storage.Files.DataLake.html | ||
| [rest_docs]: https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem | ||
| [product_docs]: https://docs.microsoft.com/en-us/azure/storage/blobs/?toc=%2fazure%2fstorage%2fblobs%2ftoc.json | ||
| [nuget]: https://www.nuget.org/ | ||
| [storage_account_docs]: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview | ||
| [storage_account_create_ps]: https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-powershell | ||
| [storage_account_create_cli]: https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-cli | ||
| [storage_account_create_portal]: https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal | ||
| [azure_cli]: https://docs.microsoft.com/cli/azure | ||
| [azure_sub]: https://azure.microsoft.com/free/ | ||
| [identity]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/identity/Azure.Identity/README.md | ||
| [StorageRequestFailedException]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/storage/Azure.Storage.Common/src/StorageRequestFailedException.cs | ||
| [error_codes]: https://docs.microsoft.com/en-us/rest/api/storageservices/blob-service-error-codes | ||
| [storage_contrib]: ../CONTRIBUTING.md | ||
| [cla]: https://cla.microsoft.com | ||
| [coc]: https://opensource.microsoft.com/codeofconduct/ | ||
| [coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/ | ||
| [coc_contact]: mailto:[email protected] |
19 changes: 19 additions & 0 deletions
19
...ge/Azure.Storage.Files.DataLake/samples/Azure.Storage.Files.DataLake.Samples.Tests.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
| <PropertyGroup> | ||
| <TargetFrameworks>$(RequiredTargetFrameworks)</TargetFrameworks> | ||
| <AssemblyTitle>Microsoft Azure.Storage.Files.DataLake client library samples</AssemblyTitle> | ||
| <IsPackable>false</IsPackable> | ||
| </PropertyGroup> | ||
| <ItemGroup> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\src\Azure.Storage.Files.DataLake.csproj" /> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\src\Azure.Storage.Common.csproj" /> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\..\identity\Azure.Identity\src\Azure.Identity.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <Compile Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\tests\Shared\**\*.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" /> | ||
| <None Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\tests\Shared\*.xml"> | ||
| <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> | ||
| </None> | ||
| </ItemGroup> | ||
| </Project> |
11 changes: 11 additions & 0 deletions
11
sdk/storage/Azure.Storage.Files.DataLake/src/AssemblyInfo.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT License. | ||
|
|
||
| using System.Runtime.CompilerServices; | ||
|
|
||
| [assembly: InternalsVisibleTo("Azure.Storage.Files.DataLake.Tests, PublicKey=" + | ||
| "0024000004800000940000000602000000240000525341310004000001000100d15ddcb2968829" + | ||
| "5338af4b7686603fe614abd555e09efba8fb88ee09e1f7b1ccaeed2e8f823fa9eef3fdd60217fc" + | ||
| "012ea67d2479751a0b8c087a4185541b851bd8b16f8d91b840e51b1cb0ba6fe647997e57429265" + | ||
| "e85ef62d565db50a69ae1647d54d7bd855e4db3d8a91510e5bcbd0edfbbecaa20a7bd9ae74593d" + | ||
| "aa7b11b4")] |
24 changes: 24 additions & 0 deletions
24
sdk/storage/Azure.Storage.Files.DataLake/src/Azure.Storage.Files.DataLake.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
| <PropertyGroup> | ||
| <TargetFrameworks>$(RequiredTargetFrameworks)</TargetFrameworks> | ||
| </PropertyGroup> | ||
| <PropertyGroup> | ||
| <AssemblyTitle>Microsoft Azure.Storage.Files.DataLake client library</AssemblyTitle> | ||
| <DefineConstants>FileSDK;$(DefineConstants)</DefineConstants> | ||
| <PackageTags>Microsoft Azure Storage Files, Microsoft, Azure, File, Files, Data Lake, Storage, StorageScalable, azureofficial</PackageTags> | ||
| <PackageReleaseNotes> | ||
| For release notes and more information please visit https://aka.ms/azure-sdk-preview3-net | ||
| </PackageReleaseNotes> | ||
| <Description> | ||
| This client library enables working with the Microsoft Azure Storage File service for storing binary and text data. | ||
| For this release see notes - https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Files.DataLake/README.md and https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Files.DataLake/Changelog.txt | ||
| in addition to the breaking changes https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Files.DataLake/BreakingChanges.txt | ||
| Microsoft Azure Storage quickstarts and tutorials - https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction | ||
| Microsoft Azure Storage REST API Reference - https://docs.microsoft.com/en-us/rest/api/storageservices/ | ||
| REST API Reference for Data Lake - https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem | ||
| </Description> | ||
| </PropertyGroup> | ||
| <ItemGroup> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\src\Azure.Storage.Common.csproj" /> | ||
| </ItemGroup> | ||
| </Project> |
106 changes: 106 additions & 0 deletions
106
sdk/storage/Azure.Storage.Files.DataLake/swagger/readme.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # File Storage | ||
seanmcc-msft marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| > see https://aka.ms/autorest | ||
|
|
||
| ## Configuration | ||
| ``` yaml | ||
| # Generate file storage | ||
| input-file: https://raw.githubusercontent.com/Azure/azure-rest-api-specs/storage-dataplane-preview/specification/storage/data-plane/Microsoft.StorageDataLake/stable/2018-11-09/DataLakeStorage.json | ||
| output-folder: ../src/Generated | ||
| clear-output-folder: false | ||
|
|
||
| # Use the Azure C# Track 2 generator | ||
| # use: C:\src\Storage\Swagger\Generator | ||
| # We can't use relative paths here, so use a relative path in generate.ps1 | ||
| azure-track2-csharp: true | ||
| ``` | ||
|
|
||
| ## Customizations for Track 2 Generator | ||
| See the [AutoRest samples](https://github.com/Azure/autorest/tree/master/Samples/3b-custom-transformations) | ||
| for more about how we're customizing things. | ||
|
|
||
| ### x-ms-code-generation-settings | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $.info["x-ms-code-generation-settings"] | ||
| transform: > | ||
| $.namespace = "Azure.Storage.Files.DataLake"; | ||
| $["client-name"] = "DataLakeRestClient"; | ||
| $["client-extensions-name"] = "FilesDataLakeExtensions"; | ||
| $["client-model-factory-name"] = "FilesDataLakeModelFactory"; | ||
| $["x-az-skip-path-components"] = true; | ||
| $["x-az-include-sync-methods"] = true; | ||
| $["x-az-public"] = false; | ||
| ``` | ||
|
|
||
| ### Url | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $.parameters.Url | ||
| transform: > | ||
| $["x-ms-client-name"] = "resourceUri"; | ||
| $.format = "url"; | ||
| $["x-az-trace"] = true; | ||
| ``` | ||
|
|
||
| ### Ignore common headers | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $["x-ms-paths"]..responses..headers["x-ms-request-id"] | ||
| transform: > | ||
| $["x-az-demote-header"] = true; | ||
| - from: swagger-document | ||
| where: $["x-ms-paths"]..responses..headers["x-ms-version"] | ||
| transform: > | ||
| $["x-az-demote-header"] = true; | ||
| - from: swagger-document | ||
| where: $["x-ms-paths"]..responses..headers["Date"] | ||
| transform: > | ||
| $["x-az-demote-header"] = true; | ||
| ``` | ||
|
|
||
| ### Clean up Failure response | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $["x-ms-paths"]..responses.default | ||
| transform: > | ||
| $["x-az-response-name"] = "StorageErrorResult"; | ||
| $["x-az-create-exception"] = true; | ||
| $["x-az-public"] = false; | ||
| $.headers["x-ms-error-code"]["x-az-demote-header"] = true; | ||
| ``` | ||
|
|
||
| ### ApiVersionParameter | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $.parameters.ApiVersionParameter | ||
| transform: $.enum = ["2019-02-02"] | ||
| ``` | ||
|
|
||
| ### MD5 to Hash | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $.parameters.ContentMD5["x-ms-client-name"] | ||
| transform: return "contentHash"; | ||
| - from: swagger-document | ||
| where: $.parameters.FileContentMD5["x-ms-client-name"] | ||
| transform: return "fileContentHash"; | ||
| - from: swagger-document | ||
| where: $.parameters.GetRangeContentMD5["x-ms-client-name"] | ||
| transform: return "rangeGetContentHash"; | ||
| ``` | ||
|
|
||
| ### StorageError | ||
| ``` yaml | ||
| directive: | ||
| - from: swagger-document | ||
| where: $.definitions.StorageError | ||
| transform: > | ||
| $["x-az-public"] = false; | ||
| $.properties.Code = { "type": "string" }; | ||
| ``` | ||
21 changes: 21 additions & 0 deletions
21
sdk/storage/Azure.Storage.Files.DataLake/tests/Azure.Storage.Files.DataLake.Tests.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
| <PropertyGroup> | ||
| <TargetFrameworks>$(RequiredTargetFrameworks)</TargetFrameworks> | ||
| </PropertyGroup> | ||
| <PropertyGroup> | ||
| <AssemblyTitle>Microsoft Azure.Storage.Files.DataLake client library tests</AssemblyTitle> | ||
| <IsPackable>false</IsPackable> | ||
| </PropertyGroup> | ||
| <ItemGroup> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\src\Azure.Storage.Files.DataLake.csproj" /> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\src\Azure.Storage.Common.csproj" /> | ||
| <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\..\identity\Azure.Identity\src\Azure.Identity.csproj" /> | ||
| </ItemGroup> | ||
| <ItemGroup> | ||
| <Folder Include="SessionRecords\" /> | ||
| <Compile Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\tests\Shared\**\*.cs" Link="Shared\%(RecursiveDir)\%(Filename)%(Extension)" /> | ||
| <None Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Common\tests\Shared\*.xml"> | ||
| <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> | ||
| </None> | ||
| </ItemGroup> | ||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I ask why we need internal access? It is generally a bad idea to couple these libraries like this. It will force use to always ship them together to ensure we don't break the internal api usage as well as means customers will always have to use the matching versions of these libraries as well to avoid breaking.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might not, we'll see later during the implementation. The intention is to only be dependent on the public blob interfaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it would be best if we can depend only on the public apis.