diff --git a/sdk/batch/Azure.Compute.Batch/README.md b/sdk/batch/Azure.Compute.Batch/README.md index ea90bd78149ee..ec5de56e61b49 100644 --- a/sdk/batch/Azure.Compute.Batch/README.md +++ b/sdk/batch/Azure.Compute.Batch/README.md @@ -1,23 +1,18 @@ # Azure Batch client library for .NET -Azur Batch is a managed service that helps developers get secret simply and securely. +Azure Batch allows users to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. Use the client library for to: -* [Get secret](https://docs.microsoft.com/azure) +* Create and manage Batch jobs and tasks +* View and perform operations on nodes in a Batch pool -[Source code][source_root] | [Package (NuGet)][package] | [API reference documentation][reference_docs] | [Product documentation][azconfig_docs] | [Samples][source_samples] - - [Source code](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/batch/Azure.Compute.Batch/src) | [Package (NuGet)](https://www.nuget.org/packages) | [API reference documentation](https://azure.github.io/azure-sdk-for-net) | [Product documentation](https://docs.microsoft.com/azure) + [Source code](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/batch/Azure.Compute.Batch/src) | [Package (NuGet)](https://www.nuget.org/packages) | [API reference documentation](https://learn.microsoft.com/dotnet/api/overview/azure/batch?view=azure-dotnet) | [Product documentation](https://learn.microsoft.com/azure/batch/) ## Getting started -This section should include everything a developer needs to do to install and create their first client connection *very quickly*. - ### Install the package -First, provide instruction for obtaining and installing the package or library. This section might include only a single line of code, like `dotnet add package package-name`, but should enable a developer to successfully install the package from NuGet, npm, or even cloning a GitHub repository. - Install the client library for .NET with [NuGet](https://www.nuget.org/ ): ```dotnetcli @@ -26,21 +21,46 @@ dotnet add package Azure.Compute.Batch --prerelease ### Prerequisites -Include a section after the install command that details any requirements that must be satisfied before a developer can [authenticate](#authenticate-the-client) and test all of the snippets in the [Examples](#examples) section. For example, for Cosmos DB: +- An Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F). -> You must have an [Azure subscription](https://azure.microsoft.com/free/dotnet/) and [Cosmos DB account](https://docs.microsoft.com/azure/cosmos-db/account-overview) (SQL API). In order to take advantage of the C# 8.0 syntax, it is recommended that you compile using the [.NET Core SDK](https://dotnet.microsoft.com/download) 3.0 or higher with a [language version](https://docs.microsoft.com/dotnet/csharp/language-reference/configure-language-version#override-a-default) of `latest`. It is also possible to compile with the .NET Core SDK 2.1.x using a language version of `preview`. +- A Batch account with a linked Azure Storage account. You can create the accounts by using any of the following methods: [Azure CLI](https://learn.microsoft.com/azure/batch/quick-create-cli) | [Azure portal](https://learn.microsoft.com/azure/batch/quick-create-portal) | [Bicep](https://learn.microsoft.com/azure/batch/quick-create-bicep) | [ARM template](https://learn.microsoft.com/azure/batch/quick-create-template) | [Terraform](https://learn.microsoft.com/azure/batch/quick-create-terraform). +- [Visual Studio 2019](https://www.visualstudio.com/vs) or later, or [.NET 6.0](https://dotnet.microsoft.com/download/dotnet) or later, for Linux or Windows. ### Authenticate the client -If your library requires authentication for use, such as for Azure services, include instructions and example code needed for initializing and authenticating. +Batch account access supports two methods of authentication: Shared Key and Microsoft Entra ID. -For example, include details on obtaining an account key and endpoint URI, setting environment variables for each, and initializing the client object. +We strongly recommend using Microsoft Entra ID for Batch account authentication. Some Batch capabilities require this method of authentication, including many of the security-related features discussed here. The service API authentication mechanism for a Batch account can be restricted to only Microsoft Entra ID using the [allowedAuthenticationModes](https://learn.microsoft.com/rest/api/batchmanagement/batch-account/create?view=rest-batchmanagement-2024-02-01&tabs=HTTP) property. When this property is set, API calls using Shared Key authentication will be rejected. -## Key concepts +#### Authenticate using Microsoft Entra ID + +Azure Batch provides integration with Microsoft Entra ID for identity-based authentication of requests. With Azure AD, you can use role-based access control (RBAC) to grant access to your Azure Batch resources to users, groups, or applications. The [Azure Identity library](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/identity/Azure.Identity/README.md) provides easy Microsoft Entra ID support for authentication. + + +```C# +BatchClient client = new BatchClient( + new Uri("https://examplebatchaccount.eastus.batch.azure.com"), + new DefaultAzureCredential()); +``` + +#### Authenticate using Shared Key + +You can also use Shared Key authentication to sign into your Batch account. This method uses your Batch account access keys to authenticate Azure commands for the Batch service. You can find your batch account shared keys in the portal under the "keys" section or you can run the following [CLI command](https://learn.microsoft.com/cli/azure/batch/account/keys?view=azure-cli-latest) -The *Key concepts* section should describe the functionality of the main classes. Point out the most important and useful classes in the package (with links to their reference pages) and explain how those classes work together. Feel free to use bulleted lists, tables, code blocks, or even diagrams for clarity. +```bash +az batch account keys list --name --resource-group +``` + +```C# +var credential = new AzureNamedKeyCredential("examplebatchaccount", "BatchAccountKey"); +BatchClient client = new BatchClient( + new Uri("https://examplebatchaccount.eastus.batch.azure.com"), + new credential()); +``` + +## Key concepts -Include the *Thread safety* and *Additional concepts* sections below at the end of your *Key concepts* section. You may remove or add links depending on what your library makes use of: +[Azure Batch Overview](https://learn.microsoft.com/azure/batch/batch-technical-overview) ### Thread safety @@ -59,28 +79,21 @@ We guarantee that all client instance methods are thread-safe and independent of ## Examples -You can familiarize yourself with different APIs using [Samples](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/batch/Azure.Compute.Batch/). - +You can familiarize yourself with different APIs using [Samples](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/batch/Azure.Compute.Batch/samples/README.md). ## Troubleshooting - -Describe common errors and exceptions, how to "unpack" them if necessary, and include guidance for graceful handling and recovery. - -Provide information to help developers avoid throttling or other service-enforced errors they might encounter. For example, provide guidance and examples for using retry or connection policies in the API. - -If the package or a related package supports it, include tips for logging or enabling instrumentation to help them debug their code. - ## Next steps -* Provide a link to additional code examples, ideally to those sitting alongside the README in the package's `/samples` directory. -* If appropriate, point users to other packages that might be useful. -* If you think there's a good chance that developers might stumble across your package in error (because they're searching for specific functionality and mistakenly think the package provides that functionality), point them to the packages they might be looking for. +View more https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/batch/Azure.Compute.Batch/samples here for common usages of the Batch client library: [Batch Samples](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/batch/Azure.Compute.Batch/samples). ## Contributing -This is a template, but your SDK readme should include details on how to contribute code to the repo/package. +This project welcomes contributions and suggestions. +Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. +For details, visit [Contributor License Agreements](https://opensource.microsoft.com/cla/). - -[style-guide-msft]: https://docs.microsoft.com/style-guide/capitalization -[style-guide-cloud]: https://aka.ms/azsdk/cloud-style-guide +When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). +Simply follow the instructions provided by the bot. +You will only need to do this once across all repos using our CLA. -![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-net/sdk/batch/Azure.Compute.Batch/README.png) +This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). +For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. \ No newline at end of file diff --git a/sdk/batch/Azure.Compute.Batch/samples/README.md b/sdk/batch/Azure.Compute.Batch/samples/README.md new file mode 100644 index 0000000000000..2935760b753a5 --- /dev/null +++ b/sdk/batch/Azure.Compute.Batch/samples/README.md @@ -0,0 +1,15 @@ +--- +page_type: sample +languages: +- csharp +products: +- azure +- azure-batch +name: Azure Batch samples for .NET +description: Samples for the Azure.Compute.Batch client library +--- + +# Azure Batch SDK Samples +Before starting, take a look at the Azure Batch [README](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/batch/Azure.Compute.Batch/README.md) for more information on how to create an Batch Account using the Azure portal/Azure CLI, and retrieving the designated endpoint and credential. + +- [Creating a Pool, Job, and Task](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/batch/Azure.Compute.Batch/samples/Sample1_CreatePool_Job_Task.md) diff --git a/sdk/batch/Azure.Compute.Batch/samples/Sample1_CreatePool_Job_Task.md b/sdk/batch/Azure.Compute.Batch/samples/Sample1_CreatePool_Job_Task.md new file mode 100644 index 0000000000000..0be6dc5710f3d --- /dev/null +++ b/sdk/batch/Azure.Compute.Batch/samples/Sample1_CreatePool_Job_Task.md @@ -0,0 +1,241 @@ +# Creating a Batch Pool, Job, and Tasks + +This sample demonstrates how to create a pool, job, and tasks for a Batch account. + +## Special Note for Batch Pools + +Both the **Azure.Compute.Batch** and the ARM based **Azure.Management.Batch** sdk's support the creating and managing of Batch Pools with the only difference being that **Azure.Management.Batch** supports creating [Batch Pools with Managed Idenities](https://learn.microsoft.com/azure/batch/managed-identity-pools), for this reason creating pools via **Azure.Management.Batch** is considered a best practice. This example will use **Azure.Management.Batch** for pool creation and thus will need to create an ArmClient to do so. + +## Authenticating the Azure.ResourceManager `ArmClient` + +In order to create Batch Pool from the Azure.Managment.Batch library you will need to instantiate an Armclient. To create an authenticated client and start interacting with Microsoft Azure resources, see the [quickstart guide here](https://github.com/Azure/azure-sdk-for-net/blob/main/doc/dev/mgmt_quickstart.md). + +```C# +var credential = new DefaultAzureCredential(); +ArmClient _armClient = new ArmClient(credential); +``` + + +### Pool Creation + +Batch operations in the **Azure.Management.Batch** sdk are preformed from a BatchAccountResource object, to get a BatchAccountResource object you can query the armclient for the resource id of your Batch account. +```C# +BatchAccountResource batchAccount = await _armClient.GetBatchAccountResource("your-batch-account-resource-id").GetAsync(); +``` + +With the BatchAccountResource you can create a pool with the [batchAccount.GetBatchAccountPools().CreateOrUpdateAsync](https://learn.microsoft.com/dotnet/api/azure.resourcemanager.batch.batchaccountpoolcollection.createorupdateasync?view=azure-dotnet) command +```C# + +var poolName = "HelloWorldPool"; +var imageReference = new BatchImageReference() +{ + Publisher = "canonical", + Offer = "0001-com-ubuntu-server-jammy", + Sku = "22_04-lts", + Version = "latest" +}; +string nodeAgentSku = "batch.node.ubuntu 22.04"; + +BatchAccountPoolResource pool = (await batchAccount.GetBatchAccountPools().CreateOrUpdateAsync(WaitUntil.Completed, poolName, new BatchAccountPoolData() +{ + VmSize = "Standard_DS1_v2", + DeploymentConfiguration = new BatchDeploymentConfiguration() + { + VmConfiguration = new BatchVmConfiguration(imageReference, nodeAgentSku) + }, + ScaleSettings = new BatchAccountPoolScaleSettings() + { + FixedScale = new BatchAccountFixedScaleSettings() + { + TargetDedicatedNodes = 1 + } + } +})).Value; +``` + +## Authenticating the Azure.Compute.Batch `BatchClient` +Creation of Batch jobs and tasks can only be preformed with the `Azure.Compute.Batch` sdk. A `BatchClient` object is needed to preform Batch operations and can be created using [Microsoft Entra ID authtentication](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/identity/Azure.Identity/README.md) and the Batch account endpoint + +```C# +var credential = new DefaultAzureCredential(); +BatchClient _batchClient = new BatchClient( + new Uri("https://examplebatchaccount.eastus.batch.azure.com"), + credential); +``` + +### Job Creation + +Before we can create Batch Tasks we first need to create a Job for the tasks to be associatd with, this can be done via the `CreateJobAsync` command. The basic elements needed are a Id for job itself and the name of the pool that this job will run against. +```C# +await _batchClient.CreateJobAsync(new BatchJobCreateContent(jobId, new BatchPoolInfo() { PoolId = poolName })); +``` +### Task Creation +Batch tasks can be created from the BatchClient via the `CreateTaskAsync`. The basic elements needed are the name of the job the task will be assigned to, and id for the task itself, and a command to run. +```C# +await _batchClient.CreateTaskAsync(jobId, new BatchTaskCreateContent(taskId, $"echo Hello world from {taskId}")); +``` +### Full Sample + +```C# +// Copyright (c) Microsoft Corporation. All rights reserved. + +using Azure.Core; +using Azure.Identity; +using Azure.Compute.Batch; +using Azure.ResourceManager; +using Azure.ResourceManager.Batch; +using Azure.ResourceManager.Batch.Models; +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; + +namespace Batch.Samples.HelloWorld +{ + public class HelloWorldSample + { + private ArmClient _armClient; + private BatchClient _batchClient; + + /// + /// Creates a pool with a configurable number of nodes, then submits tasks which print a 'Hello world' message. + /// The resulting stdout.txt or stderr.txt (depending on each task's exit code) is then printed to the console. + /// + /// After running, the job will be terminated and the pool will be deleted. + /// + /// The ARM resource ID of the Batch account. + /// A task which completes when the sample has finished running. + public async Task Run(string batchAccountResourceId) + { + var batchAccountIdentifier = ResourceIdentifier.Parse(batchAccountResourceId); + + var credential = new DefaultAzureCredential(); + _armClient = new ArmClient(credential); + + BatchAccountResource batchAccount = await _armClient.GetBatchAccountResource(batchAccountIdentifier).GetAsync(); + + _batchClient = new BatchClient(new Uri($"https://{batchAccount.Data.AccountEndpoint}"), credential); + + var poolName = GenerateUniqueName("HelloWorldPool"); + var imageReference = new BatchImageReference() + { + Publisher = "canonical", + Offer = "0001-com-ubuntu-server-jammy", + Sku = "22_04-lts", + Version = "latest" + }; + string nodeAgentSku = "batch.node.ubuntu 22.04"; + + BatchAccountPoolResource pool = (await batchAccount.GetBatchAccountPools().CreateOrUpdateAsync(WaitUntil.Completed, poolName, new BatchAccountPoolData() + { + VmSize = "Standard_DS1_v2", + DeploymentConfiguration = new BatchDeploymentConfiguration() + { + VmConfiguration = new BatchVmConfiguration(imageReference, nodeAgentSku) + }, + ScaleSettings = new BatchAccountPoolScaleSettings() + { + FixedScale = new BatchAccountFixedScaleSettings() + { + TargetDedicatedNodes = 1 + } + } + })).Value; + + string jobId = GenerateUniqueName("HelloWorldJob"); + + try + { + await _batchClient.CreateJobAsync(new BatchJobCreateContent(jobId, new BatchPoolInfo() { PoolId = poolName })); + + for (int i = 0; i < 5; i++) + { + string taskId = $"task-{i}"; + Console.WriteLine("Submitting {0}", taskId); + await _batchClient.CreateTaskAsync(jobId, new BatchTaskCreateContent(taskId, $"echo Hello world from {taskId}")); + } + + Console.WriteLine("Waiting for all tasks to complete on job: {0} ...", jobId); + await waitForTasksToComplete(jobId); + + var completedTasks = _batchClient.GetTasksAsync(jobId, filter: "state eq 'completed'"); + await foreach (BatchTask t in completedTasks) + { + var outputFileName = t.ExecutionInfo.ExitCode == 0 ? "stdout.txt" : "stderr.txt"; + + Console.WriteLine("Task {0} exited with code {1}. Output ({2}):", + t.Id, t.ExecutionInfo.ExitCode, outputFileName); + + BinaryData fileContents = await _batchClient.GetTaskFileAsync(jobId, t.Id, outputFileName); + using (var reader = new StreamReader(fileContents.ToStream())) + { + Console.WriteLine(await reader.ReadLineAsync()); + } + } + } + finally + { + Console.WriteLine("Terminating job {0} and deleting pool {1}", jobId, poolName); + await Task.WhenAll([ + _batchClient.TerminateJobAsync(jobId), + pool.DeleteAsync(WaitUntil.Completed)]); + } + } + + /// + /// Poll all the tasks in the given job and wait for them to reach the completed state. + /// + /// The ID of the job to poll + /// A task that will complete when all Batch tasks have completed. + /// Thrown if all tasks haven't reached the completed state after a certain period of time + private async Task waitForTasksToComplete(String jobId) + { + // Note that this timeout should take into account the time it takes for the pool to scale up + var timeoutAfter = DateTime.Now.AddMinutes(10); + while (DateTime.Now < timeoutAfter) + { + var allComplete = true; + var tasks = _batchClient.GetTasksAsync(jobId, select: ["id", "state"]); + await foreach (BatchTask task in tasks) + { + if (task.State != BatchTaskState.Completed) + { + allComplete = false; + break; + } + } + + if (allComplete) + { + return; + } + + await Task.Delay(10000); + } + + throw new TimeoutException("Task(s) did not complete within the specified time"); + } + + /// + /// Generate a unique name with the given prefix using the current user name and a timestamp. + /// + /// The name's prefix. + /// The generated name. + private static string GenerateUniqueName(string prefix) + { + string currentUser = new string(Environment.UserName.Where(char.IsLetterOrDigit).ToArray()); + return string.Format("{0}-{1}-{2}", prefix, currentUser, DateTime.Now.ToString("yyyyMMdd-HHmmss")); + } + } + + public async static Task Main(string[] args) + { + string batchAccountResourceId = "your-batch-account-resource-id"; + + await new HelloWorldSample().Run(batchAccountResourceId); + + Console.WriteLine("Press return to exit..."); + Console.ReadLine(); + } +} +``` diff --git a/sdk/batch/Azure.Compute.Batch/src/Azure.Compute.Batch.csproj b/sdk/batch/Azure.Compute.Batch/src/Azure.Compute.Batch.csproj index 44d8016cad956..546e43a7e8200 100644 --- a/sdk/batch/Azure.Compute.Batch/src/Azure.Compute.Batch.csproj +++ b/sdk/batch/Azure.Compute.Batch/src/Azure.Compute.Batch.csproj @@ -1,6 +1,6 @@ - This is the Azure.Compute.Batch client library for developing .NET applications with rich experience. + This is the Azure.Compute.Batch client library for developing .NET applications with rich experience. This client library allows for the creation and management of Batch objects such as Pools, Jobs, and Tasks. For more information about Azure Batch, see https://learn.microsoft.com/azure/batch/batch-technical-overview Azure SDK Code Generation Azure.Compute.Batch for Azure Data Plane 1.0.0-beta.1 Azure.Compute.Batch diff --git a/sdk/batch/Azure.Compute.Batch/src/Custom/BatchNamedKeyCredentialPolicy.cs b/sdk/batch/Azure.Compute.Batch/src/Custom/BatchNamedKeyCredentialPolicy.cs index e1bf5ba97114b..1ae28452e339b 100644 --- a/sdk/batch/Azure.Compute.Batch/src/Custom/BatchNamedKeyCredentialPolicy.cs +++ b/sdk/batch/Azure.Compute.Batch/src/Custom/BatchNamedKeyCredentialPolicy.cs @@ -98,7 +98,7 @@ private string BuildStringToSign(HttpMessage message) stringBuilder.Append(contentLengthString == "0" ? "" : contentLengthString ?? "").Append('\n'); stringBuilder.Append(contentMD5 ?? "");// todo: fix base 64 VALUE stringBuilder.Append('\n'); - stringBuilder.Append(contentType ?? "").Append('\n'); // Empty date because ocp-date is expected (as per web page above)) + stringBuilder.Append(contentType ?? "").Append('\n'); stringBuilder.Append('\n'); stringBuilder.Append(ifModifiedSince ?? "").Append('\n'); stringBuilder.Append(ifMatch ?? "").Append('\n');