Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated README.md for Storage DataLake #8604

Merged
merged 4 commits into from
Nov 5, 2019
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 164 additions & 24 deletions sdk/storage/Azure.Storage.Files.DataLake/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Azure Storage Files Data Lake client library for .NET

> Server Version: 2019-02-02

Azure Data Lake includes all the capabilities required to make it easy for developers, data scientists,
and analysts to store data of any size, shape, and speed, and do all types of processing and analytics
across platforms and languages. It removes the complexities of ingesting and storing all of your data
while making it faster to get up and running with batch, streaming, and interactive analytics.

[Source code][source] | [Package (NuGet)][package] | [API reference documentation][docs] | [REST API documentation][rest_docs] | [Product documentation][product_docs]
amnguye marked this conversation as resolved.
Show resolved Hide resolved
[Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs] | [Product documentation][product_docs]

## Getting started

Expand All @@ -15,7 +16,7 @@ while making it faster to get up and running with batch, streaming, and interact
Install the Azure Storage Files Data Lake client library for .NET with [NuGet][nuget]:

```Powershell
dotnet add package Azure.Storage.Files.DataLake --version 12.0.0-preview.4
dotnet add package Azure.Storage.Files.DataLake --version 12.0.0-preview.5
```

### Prerequisites
Expand All @@ -33,39 +34,177 @@ az storage account create --name MyStorageAccount --resource-group MyResourceGro

## Key concepts

TODO: Add Key Concepts
This preview package for .NET includes ADLS Gen2 specific API support made available in Blob SDK. This includes:
amnguye marked this conversation as resolved.
Show resolved Hide resolved
1. New directory level operations (Create, Rename/Move, Delete) for both hierarchical namespace enabled (HNS) storage accounts and HNS disabled storage accounts. For HNS enabled accounts, the rename/move operations are atomic.
amnguye marked this conversation as resolved.
Show resolved Hide resolved
2. Permission related operations (Get/Set ACLs) for hierarchical namespace enabled (HNS) accounts.

HNS enabled accounts in ADLS Gen2 can also now leverage all of the operations available in Blob SDK. Support for File level semantics for ADLS Gen2 is planned to be made available in Blob SDK in a later release. In the meantime, please find below mapping for ADLS Gen2 terminology to Blob terminology
amnguye marked this conversation as resolved.
Show resolved Hide resolved

|ADLS Gen2 | Blob |
| ---------- | ---------- |
|Filesystem | Container |
|Folder | Directory |
|File | Blob |
amnguye marked this conversation as resolved.
Show resolved Hide resolved

## Examples

TODO: Add Examples
### Create a DataLakeServiceClient
```C# Snippet:SampleSnippetDataLakeServiceClient_Create
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have corresponding #regions? I don't see them in the PR and it's breaking the CI builds.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by regions?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need a #region in your C# samples named Snippet:SampleSnippetDataLakeServiceClient_Create and the code will get pulled in when you run C:\src\azure-sdk-for-net\eng\Update-Snippets.ps1.

See #7928 for an overview and the README/Samples for Queues. It's helping us ensure our README code snippets are always up to date and compiling.

// Make StorageSharedKeyCredential to pass to the serviceClient
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(<storage-account-name>, <storage-account-key>);

## Troubleshooting
// Create DataLakeServiceClient using StorageSharedKeyCredentials
DataLakeServiceClient serviceClient = new DataLakeServiceClient(<endpoint-storage-dfs-url>, sharedKeyCredential);
```

All File DataLake service operations will throw a
[RequestFailedException][RequestFailedException] on failure with
helpful [`ErrorCode`s][error_codes]. Many of these errors are recoverable.
### Create a DataLakeFileSystemClient
```C# Snippet:SampleSnippetDataLakeFileSystemClient_Create
// Make StorageSharedKeyCredential to pass to the serviceClient
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(<storage-account-name>, <storage-account-key>);

TODO: Update sample
// Create DataLakeServiceClient using StorageSharedKeyCredentials
DataLakeServiceClient serviceClient = new DataLakeServiceClient(<endpoint-storage-dfs-url>, sharedKeyCredential);

```c#
string connectionString = "<connection_string>";
// Try to create a container named "sample-container" and avoid any potential race
// conditions that might arise by checking if the container exists before creating
BlobContainerClient container = new BlobContainerClient(connectionString, "sample-container");
try
{
container.Create();
}
catch (RequestFailedException ex)
when (ex.ErrorCode == BlobErrorCode.ContainerAlreadyExists)
// Create a DataLakeFileSystemClient
DataLakeFileSystemClient filesystem = serviceClient.GetFileSystemClient("sample-filesystem");
filesystem.Create();
```

### Create a DataLakeDirectoryClient
```C# Snippet:SampleSnippetDataLakeDirectoryClient_Create
// Make StorageSharedKeyCredential to pass to the serviceClient
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(<storage-account-name>, <storage-account-key>);

// Create DataLakeServiceClient using StorageSharedKeyCredentials
DataLakeServiceClient serviceClient = new DataLakeServiceClient(<endpoint-storage-dfs-url>, sharedKeyCredential);

// Get a reference to a filesystem named "sample-filesystem" and then create it
DataLakeFileSystemClient filesystem = serviceClient.GetFileSystemClient("sample-filesystem");
filesystem.Create();

// Create a DataLakeDirectoryClient
DataLakeDirectoryClient directory = filesystem.CreateDirectory("sample-directory");
directory.Create();
```

### Create a DataLakeFileClient

Create DataLakeFileClient from a DataLakeDirectoryClient

```C# Snippet:SampleSnippetDataLakeFileClient_Create
// Make StorageSharedKeyCredential to pass to the serviceClient
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(<storage-account-name>, <storage-account-key>);

// Create DataLakeServiceClient using StorageSharedKeyCredentials
DataLakeServiceClient serviceClient = new DataLakeServiceClient(<endpoint-storage-dfs-url>, sharedKeyCredential);

// Get a reference to a filesystem named "sample-filesystem" and then create it
DataLakeFileSystemClient filesystem = serviceClient.GetFileSystemClient("sample-filesystem");
filesystem.Create();

// Create a DataLakeDirectoryClient
DataLakeDirectoryClient directory = filesystem.CreateDirectory("sample-directory");
directory.Create();

// Create a DataLakeFileClient
DataLakeFileClient file = directory.CreateFile("sample-file");
file.Create();
```

Create DataLakeFileClient from a DataLakeFileSystemClient
```C# Snippet:SampleSnippetDataLakeFileClient_Create
// Make StorageSharedKeyCredential to pass to the serviceClient
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(storageAccountName, storageAccountKey);

// Create DataLakeServiceClient using StorageSharedKeyCredentials
DataLakeServiceClient serviceClient = new DataLakeServiceClient(<endpoint-storage-dfs-url>, sharedKeyCredential);

// Get a reference to a filesystem named "sample-filesystem" and then create it
DataLakeFileSystemClient filesystem = serviceClient.GetFileSystemClient("sample-filesystem");
filesystem.Create();

// Create a DataLakeFileClient
DataLakeFileClient file = filesystem.CreateDirectory("sample-file");
file.Create();
```

### Appending Data to a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_Append
// FileAppend usage - e.g. file.Append(<stream-to-file-content>, <offset>)
// Create a DataLakeFileClient
DataLakeFileClient file = filesystem.CreateDirectory("sample-file");
file.Create();

// Append data to the DataLake File
file.Append(File.OpenRead(<path-to-file>, 0);
file.Flush(<length-of-file>);
```

### Reading Data from a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_Read
// Reading data to the DataLake File
Response<FileDownloadInfo> fileContents = file.Read();
```

### Listing/Traversing through a DataLake Filesystem
```C# Snippet:SampleSnippetDataLakeFileClient_List
// Listing/Traversing through a DataLake Filesystem
foreach (PathItem pathItem in filesystem.ListPaths(recursive: true))
{
// Ignore any errors if the container already exists
Console.WriteLine(pathItem.Name);
}
```
### Set Permissions on a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_SetPermissions
// Create a DataLake file so we can set the Access Controls on the files
DataLakeFileClient fileClient = filesystem.GetFileClient(Randomize("sample-file"));
fileClient.Create();

// Set the Permissions of the file
fileClient.SetPermissions(permissions: "rwxrwxrwx");
```
### Set Access Controls (ACLs) on a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_SetAcls
// Set the Permissions of the file
fileClient.SetAccessControl("user::rwx,group::r--,mask::rwx,other::---");
```
### Get Access Controls (ACLs) on a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_GetAcls
// Get the Permissions of the file
PathAccessControl accessControlResponse = fileClient.GetAccessControl();
```
### Rename a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_RenameFile
// Rename File Client
DataLakeDirectoryClient renamedDirectoryClient = fileClient.Rename("new-file-name");
```
### Rename a DataLake Directory
```C# Snippet:SampleSnippetDataLakeFileClient_RenameDirectory
// Rename File Client
DataLakeDirectoryClient renamedDirectoryClient = directoryClient.Rename("new-directory-name");
```
### Get Properties on a DataLake File
```C# Snippet:SampleSnippetDataLakeFileClient_GetProperties
// Get Properties on DataLake File
PathProperties pathProperties = FileClient.GetProperties();
```
### Get Properties on a DataLake Directory
```C# Snippet:SampleSnippetDataLakeDirectoryClient_GetProperties
// Get Properties on DataLake Directory
PathProperties pathProperties = DirectoryClient.GetProperties();
```
## Troubleshooting

All File DataLake service operations will throw a
[RequestFailedException][RequestFailedException] on failure with
helpful [`ErrorCode`s][error_codes]. Many of these errors are recoverable.
amnguye marked this conversation as resolved.
Show resolved Hide resolved

## Next steps

TODO: Link Samples
Get started with our [DataLake samples][samples]:

1. [Hello World](samples/Sample01a_HelloWorld.cs): Append, Read, and List DataLake Files (or [asynchronously](samples/Sample01b_HelloWorldAsync.cs))
2. [Auth](samples/Sample02_Auth.cs): Authenticate with public access, shared keys, shared access signatures, and Azure Active Directory.

## Contributing

Expand All @@ -82,7 +221,7 @@ For more information see the [Code of Conduct FAQ][coc_faq]
or contact [[email protected]][coc_contact] with any
additional questions or comments.

![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-net%2Fsdk%2Fstorage%2FAzure.Storage.Blobs.Cryptography%2FREADME.png)
![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-net%2Fsdk%2Fstorage%2FAzure.Storage.Files.DataLake%2FREADME.png)

<!-- LINKS -->
[source]: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/storage/Azure.Storage.Files.DataLake/src
Expand All @@ -104,4 +243,5 @@ additional questions or comments.
[cla]: https://cla.microsoft.com
[coc]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:[email protected]
[coc_contact]: mailto:[email protected]
[samples]: samples