From 81d7891042c16c0c1a2e8c99774a0901653c90e6 Mon Sep 17 00:00:00 2001 From: Jesse Squire Date: Fri, 3 Jan 2020 15:29:39 -0500 Subject: [PATCH] [Event Hubs Client] Track Two (Release Documentation) (#9296) The focus of these changes is to revise and enhance the documentation in the README files for each of the Event Hubs packages and to provide details for the change log. Versions have also been changed in the projects to match the change log. Adding the generated public API surface for analysis during CI. --- .../CHANGELOG.md | 34 +- .../README.md | 58 +- ...ging.EventHubs.Processor.netstandard2.0.cs | 46 ++ ...Azure.Messaging.EventHubs.Processor.csproj | 3 +- .../Azure.Messaging.EventHubs/CHANGELOG.md | 24 +- .../Azure.Messaging.EventHubs/README.md | 83 +- ...zure.Messaging.EventHubs.netstandard2.0.cs | 358 ++++++++ .../proposal-event-hubs-second-preview.md | 778 +++++++++--------- .../src/Azure.Messaging.EventHubs.csproj | 3 +- 9 files changed, 963 insertions(+), 424 deletions(-) create mode 100755 sdk/eventhub/Azure.Messaging.EventHubs.Processor/api/Azure.Messaging.EventHubs.Processor.netstandard2.0.cs create mode 100755 sdk/eventhub/Azure.Messaging.EventHubs/api/Azure.Messaging.EventHubs.netstandard2.0.cs diff --git a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/CHANGELOG.md b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/CHANGELOG.md index 3b27421206c1c..d742ab8273312 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/CHANGELOG.md +++ b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/CHANGELOG.md @@ -1,8 +1,34 @@ # Release History -## 5.0.0-preview.7 (Unreleased) +## 5.0.0 -Release notes will be provided here when the final set of features for the release is available. +Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release: + +- Christopher Scott _([GitHub](https://github.com/christothes))_ + +### Changes + +#### General + +- A migration guide is now available for those moving from the 4.x version of the `Microsoft.Azure.EventHubs` libraries to the 5.0.0 version under the `Azure.Messaging.EventHubs` namespace. + +#### Organization and naming + +- Namespaces have been reorganized to align types to their functional area, reducing the number of types in the root namespace and offering better context for where a type is used. Cross-functional types have been left in the root while specialized types were moved to the `Producer`, `Consumer`, or `Processor` namespaces. + +#### Event processing + +- Load balancing has been tuned for better performance and lower resource use. (A community contribution, courtesy of [christothes](https://github.com/christothes)) + +- Reduction of reliance on Azure resources for Event Processor tests. (A community contribution, courtesy of [christothes](https://github.com/christothes)) + +- Logging has been implemented for Event Processor operations interacting with storage. (A community contribution, courtesy of [christothes](https://github.com/christothes)) + +- Logging has been implemented for general Event Processing operations, including background execution. + +- A bug with resuming from a storage checkpoint was fixed, ensuring that processing resumes from the next available event rather than reprocessing the event from which the checkpoint was created. + +- The protected `On[EventName]` members have been marked private to reduce the public surface and reduce confusion. They provided no benefit over providing a handler and the cognative cost was not justified. ## 5.0.0-preview.6 @@ -20,7 +46,7 @@ Thank you to our developer community members who helped to make the Event Hubs c - A large portion of the public API surface, including members and parameters, have had adjustments to their naming in order to improve discoverability, provide better context to developers, and better conform to the [Azure SDK Design Guidelines for .NET](https://azure.github.io/azure-sdk/dotnet_introduction.html). -#### Event Processing +#### Event processing - The API for the `EventProcessorClient` has been revised, adopting an event-driven model that aligns to many of the .NET base class library types and reduces complexity when constructing the client. @@ -28,7 +54,7 @@ Thank you to our developer community members who helped to make the Event Hubs c - The mechanism for creating checkpoints has been bundled with the event dispatched for processing; this allows a checkpoint to be created with a clear association with a given event, removing the need for developers to explicitly pass an event to a dedicated checkpoint manager. -#### Storage Operations +#### Storage operations - The concept of a plug-in model for the durable storage used by the processor has been removed; this package has taken a strong dependency on Azure Storage Blobs and no longer requires developers to manipulate an abstraction on top of storage. diff --git a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/README.md b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/README.md index c63f61e08ead6..33c5a0d403d40 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/README.md +++ b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/README.md @@ -35,7 +35,7 @@ To quickly create the needed resources in Azure and to receive connection string Install the Azure Event Hubs Event Processor client library for .NET using [NuGet](https://www.nuget.org/): ```PowerShell -Install-Package Azure.Messaging.EventHubs.Processor -Version 5.0.0-preview.6 +Install-Package Azure.Messaging.EventHubs.Processor -Version 5.0.0 ``` ### Obtain an Event Hubs connection string @@ -62,9 +62,9 @@ For more concepts and deeper discussion, see: [Event Hubs Features](https://docs ## Examples -### Creating an Event Processor Client +### Creating an Event Processor client -Since the `EventProcessorClient` as a dependency on Azure Storage blobs for persistence of its state, you'll need to provide a `BlobContainerClient` for the processor, which has been configured for the storage account and container that should be used. +Since the `EventProcessorClient` has a dependency on Azure Storage blobs for persistence of its state, you'll need to provide a `BlobContainerClient` for the processor, which has been configured for the storage account and container that should be used. ```csharp string storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>"; @@ -85,7 +85,7 @@ EventProcessorClient processor = new EventProcessorClient ); ``` -### Configure the Event and Error Handlers +### Configure the event and error handlers In order to use the `EventProcessorClient`, handlers for event processing and errors must be provided. These handlers are considered self-contained and developers are responsible for ensuring that exceptions within the handler code are accounted for. @@ -129,7 +129,7 @@ processor.ProcessEventAsync += processEventHandler; processor.ProcessErrorAsync += processErrorHandler; ``` -### Start and Stop Procesing +### Start and stop processing The `EventProcessorClient` will perform its processing in the background once it has been explicitly started and continue doing so until it has been explicitly stopped. While this allows the application code to perform other tasks, it also places the responsibility of ensuring that the process does not terminate during processing if there are no other tasks being performed. @@ -171,21 +171,55 @@ private async Task ProcessUntilCanceled(CancellationToken cancellationToken) } ``` +### Using an Active Directory principal with the Event Processor client + +The [Azure Identity library](https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/identity/Azure.Identity/README.md) provides Azure Active Directory authentication support which can be used for the Azure client libraries, including Event Hubs and Azure Storage. + +To make use of an Active Directory principal, one of the available identity tokens from the `Azure.Identity` library is also provided when creating the Event Processor client. In addition, the fully qualified Event Hubs namespace and the name of the desired Event Hub are supplied in lieu of the Event Hubs connection string. + +```csharp +Uri blobStorageUrl = new Uri("<< FULLY-QUALIFIED BLOBS CONTAINER URL >>"); + +string fullyQualifiedNamespace = "<< FULLY-QUALIFIED EVENT HUBS NAMESPACE (like something.servicebus.windows.net)>>" +string eventHubName = "<< NAME OF THE EVENT HUB >>"; +string consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>"; + +TokenCredential credential = new DefaultAzureIdentity(); +BlobContainerClient storageClient = new BlobContainerClient(blobStorageUrl, credential); + +EventProcessorClient processor = new EventProcessorClient +( + storageClient, + consumerGroup, + fullyQualifiedNamespace, + eventHubName, + credential +); +``` + +When using Azure Active Directory with Event Hubs, your principal must be assigned a role which allows reading from Event Hubs, such as the `Azure Event Hubs Data Receiver` role. For more information about using Azure Active Directory authorization with Event Hubs, please refer to [the associated documentation](https://docs.microsoft.com/en-us/azure/event-hubs/authorize-access-azure-active-directory). + +When using Azure Active Directory with Azure Storage, your principal must be assigned a role which allows read, write, and delete access to blobs, such as the `Storage Blob Data Contributor` role. For more information about using Active Directory Authorization with Azure Storage, please refer to the [the associated documentation](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad) and the [Azure Storage authorization sample](https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Blobs/samples/Sample02_Auth.cs). + ## Troubleshooting -### Common exceptions +### Event Processor client exceptions + +The Event Processor client makes every attempt to be resilient in the face of exceptions and will take the necessary actions to continue processing unless it is impossible to do so. No action from developers is needed for this to take place; it is natively part of the processor's behavior. + +In order to allow developers the opportunity to inspect and react to exceptions that occur within the Event Processor client operations, they are surfaced via the `ProcessError` event. The arguments for this event offer details about the exception and the context in which it was observed. Developers may perform normal operations on the Event Processor client from within this event handler, such as stopping and/or restarting it in response to errors, but may not otherwise influence the processor's exception behavior. -#### Timeout +For a basic example of implementing the error handler, please see the sample: [Manage the Event Processor when an error is encountered](./samples/Sample07_RestartProcessingOnError.cs). -This indicates that the Event Hubs service did not respond to an operation within the expected amount of time. This may have been caused by a transient network issue or service problem. The Event Hubs service may or may not have successfully completed the request; the status is not known. It is recommended to attempt to verify the current state and retry if necessary. +### Exceptions in event handlers -#### Quota Exceeded +Because the Event Processor client lacks the appropriate context to understand the severity of exceptions within the event handlers that developers provide, it cannot assume what actions would be a reasonable response to them. As a result, developers are considered responsible for exceptions that occur within the event handlers they provide using `try/catch` blocks and other standard language constructs. -This typically indicates that there are too many active event processors for a consumer group. This limit depends on the tier of the Event Hubs namespace, and moving to a higher tier may be desired. An alternative would be do create additional consumer groups and ensure that the number of active processors for any group is within the limit. Please see [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for more information. +The Event Processor client will not attempt to detect exceptions in developer code nor surface them explicitly. The resulting behavior will depend on the processor's hosting environment and the context in which the event handler was called. Because this may vary between different scenarios, it is strongly recommended that developers code their event handlers defensively and account for potential exceptions. -### Other exceptions +### Exception details -For detailed information about these and other exceptions that may occur, please refer to [Event Hubs messaging exceptions](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-messaging-exceptions). +For detailed information about exceptions that may occur, please refer to the Event Hubs client library [README]( https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/eventhub/Azure.Messaging.EventHubs/README.md#event-hubs-exception) and the service documentation for [Event Hubs messaging exceptions](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-messaging-exceptions). ## Next steps diff --git a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/api/Azure.Messaging.EventHubs.Processor.netstandard2.0.cs b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/api/Azure.Messaging.EventHubs.Processor.netstandard2.0.cs new file mode 100755 index 0000000000000..9792760067ac5 --- /dev/null +++ b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/api/Azure.Messaging.EventHubs.Processor.netstandard2.0.cs @@ -0,0 +1,46 @@ +namespace Azure.Messaging.EventHubs +{ + public partial class EventProcessorClient + { + protected EventProcessorClient() { } + public EventProcessorClient(Azure.Storage.Blobs.BlobContainerClient checkpointStore, string consumerGroup, string connectionString) { } + public EventProcessorClient(Azure.Storage.Blobs.BlobContainerClient checkpointStore, string consumerGroup, string connectionString, Azure.Messaging.EventHubs.EventProcessorClientOptions clientOptions) { } + public EventProcessorClient(Azure.Storage.Blobs.BlobContainerClient checkpointStore, string consumerGroup, string connectionString, string eventHubName) { } + public EventProcessorClient(Azure.Storage.Blobs.BlobContainerClient checkpointStore, string consumerGroup, string fullyQualifiedNamespace, string eventHubName, Azure.Core.TokenCredential credential, Azure.Messaging.EventHubs.EventProcessorClientOptions clientOptions = null) { } + public EventProcessorClient(Azure.Storage.Blobs.BlobContainerClient checkpointStore, string consumerGroup, string connectionString, string eventHubName, Azure.Messaging.EventHubs.EventProcessorClientOptions clientOptions) { } + public string ConsumerGroup { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string EventHubName { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string FullyQualifiedNamespace { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string Identifier { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public bool IsRunning { get { throw null; } protected set { } } + public event System.Func PartitionClosingAsync { add { } remove { } } + public event System.Func PartitionInitializingAsync { add { } remove { } } + public event System.Func ProcessErrorAsync { add { } remove { } } + public event System.Func ProcessEventAsync { add { } remove { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + public virtual void StartProcessing(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { } + public virtual System.Threading.Tasks.Task StartProcessingAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual void StopProcessing(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { } + public virtual System.Threading.Tasks.Task StopProcessingAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public partial class EventProcessorClientOptions + { + public EventProcessorClientOptions() { } + public Azure.Messaging.EventHubs.EventHubConnectionOptions ConnectionOptions { get { throw null; } set { } } + public string Identifier { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public System.TimeSpan? MaximumWaitTime { get { throw null; } set { } } + public Azure.Messaging.EventHubs.EventHubsRetryOptions RetryOptions { get { throw null; } set { } } + public bool TrackLastEnqueuedEventProperties { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } +} diff --git a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/src/Azure.Messaging.EventHubs.Processor.csproj b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/src/Azure.Messaging.EventHubs.Processor.csproj index 9b6e86976a00f..2872ad0273980 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs.Processor/src/Azure.Messaging.EventHubs.Processor.csproj +++ b/sdk/eventhub/Azure.Messaging.EventHubs.Processor/src/Azure.Messaging.EventHubs.Processor.csproj @@ -1,11 +1,10 @@  Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This library extends its Event Processor with durable storage for checkpoint information using Azure Blob storage. For more information about Event Hubs, see https://azure.microsoft.com/en-us/services/event-hubs/ - 5.0.0-preview.7 + 5.0.0 Azure;Event Hubs;EventHubs;.NET;Event Processor;EventProcessor;$(PackageCommonTags) $(RequiredTargetFrameworks) false - false diff --git a/sdk/eventhub/Azure.Messaging.EventHubs/CHANGELOG.md b/sdk/eventhub/Azure.Messaging.EventHubs/CHANGELOG.md index aef7b65d94262..285656db620d5 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs/CHANGELOG.md +++ b/sdk/eventhub/Azure.Messaging.EventHubs/CHANGELOG.md @@ -1,8 +1,28 @@ # Release History -## 5.0.0-preview.7 (Unreleased) +## 5.0.0 -Release notes will be provided here when the final set of features for the release is available. +### Acknowledgements + +Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release: + +- Alberto De Natale _([GitHub](https://github.com/albertodenatale))_ + +### Changes + +#### General + +- A migration guide is now available for those moving from the 4.x version of the `Microsoft.Azure.EventHubs` libraries to the 5.0.0 version under the `Azure.Messaging.EventHubs` namespace. + +- A bug was fixed that would intermittently cause a failure that caused retries to abort, potentially preventing recovery from transient failures. + +- Several minor performance and efficiency improvements have been implemented. + +#### Organization and naming + +- Namespaces have been reorganized to align types to their functional area, reducing the number of types in the root namespace and offering better context for where a type is used. Cross-functional types have been left in the root while specialized types were moved to the `Producer`, `Consumer`, or `Processor` namespaces. + +- The hierarchy of custom exceptions has been flattened, with only the `EventHubsException` remaining. The well-known failure scenarios that had previously been represented as stand-alone types are now exposed by a new `Reason` property to allow for applying exception filtering and other logic where inspecting the text of an exception message wouldn't be ideal. ## 5.0.0-preview.6 diff --git a/sdk/eventhub/Azure.Messaging.EventHubs/README.md b/sdk/eventhub/Azure.Messaging.EventHubs/README.md index be93bd68f616d..d4f25ea4f4003 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs/README.md +++ b/sdk/eventhub/Azure.Messaging.EventHubs/README.md @@ -37,7 +37,7 @@ If you'd like to run samples that use [Azure.Identity](https://github.com/Azure/ Install the Azure Event Hubs client library for .NET with [NuGet](https://www.nuget.org/): ```PowerShell -Install-Package Azure.Messaging.EventHubs -Version 5.0.0-preview.6 +Install-Package Azure.Messaging.EventHubs -Version 5.0.0 ``` ### Obtain a connection string @@ -147,34 +147,91 @@ await using (var consumer = new EventHubConsumerClient(consumerGroup, connection } ``` -### Process events using an Event Processor Client +### Process events using an Event Processor client For the majority of production scenarios, it is recommended that the [Event Processor Client](./../Azure.Messaging.EventHubs.Processor) be used for reading and processing events. The processor is intended to provide a robust experience for processing events across all partitions of an Event Hub in a performant and fault tolerant manner while providing a means to persist its state. Event Processor clients are also capable of working cooperatively within the context of a consumer group for a given Event Hub, where they will automatically manage distribution and balancing of work as instances become available or unavailable for the group. +Since the `EventProcessorClient` has a dependency on Azure Storage blobs for persistence of its state, you'll need to provide a `BlobContainerClient` for the processor, which has been configured for the storage account and container that should be used. + +```csharp +var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>"; +var blobContainerName = "<< NAME OF THE BLOBS CONTAINER >>"; + +var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>"; +var eventHubName = "<< NAME OF THE EVENT HUB >>"; +var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>"; + +BlobContainerClient storageClient = new BlobContainerClient(storageConnectionString, blobContainerName); + +EventProcessorClient processor = new EventProcessorClient +( + storageClient, + consumerGroup, + eventHubsConnectionString, + eventHubName +); +``` + More details can be found in the Event Processor Client [README](./../Azure.Messaging.EventHubs.Processor/README.md) and the accompanying [samples](./../Azure.Messaging.EventHubs.Processor/samples). -## Troubleshooting +### Using an Active Directory principal with the Event Hub clients -### Common exceptions +The [Azure Identity library](https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/identity/Azure.Identity/README.md) provides Azure Active Directory authentication support which can be used for the Azure client libraries, including Event Hubs. -#### Event Hub Client Closed +To make use of an Active Directory principal, one of the available identity tokens from the `Azure.Identity` library is also provided when creating the Event Hub client. In addition, the fully qualified Event Hubs namespace and the name of desired Event Hub are supplied in lieu of the Event Hubs connection string. -This occurs when an operation has been requested on an Event Hub client that has already been closed or disposed of. It is recommended to check the application code and ensure that objects from the Event Hubs client library are created and closed/disposed in the intended scope. +```csharp +var fullyQualifiedNamespace = "<< FULLY-QUALIFIED EVENT HUBS NAMESPACE (like something.servicebus.windows.net)>>" +var eventHubName = "<< NAME OF THE EVENT HUB >>"; -#### Timeout +TokenCredential credential = new DefaultAzureIdentity(); -This indicates that the Event Hubs service did not respond to an operation within the expected amount of time. This may have been caused by a transient network issue or service problem. The Event Hubs service may or may not have successfully completed the request; the status is not known. It is recommended to attempt to verify the current state and retry if necessary. +await using (var producer = new EventHubProducerClient(fullyQualifiedNamespace, eventHubName, credential)) +{ + // Publish events using the producer +} +``` -#### Quota Exceeded +When using Azure Active Directory, your principal must be assigned a role which allows access to Event Hubs, such as the `Azure Event Hubs Data Owner` role. For more information about using Azure Active Directory authorization with Event Hubs, please refer to [the associated documentation](https://docs.microsoft.com/en-us/azure/event-hubs/authorize-access-azure-active-directory). -This typically indicates that there are too many active read operations for a single consumer group. This limit depends on the tier of the Event Hubs namespace, and moving to a higher tier may be desired. An alternative would be do create additional consumer groups and ensure that the number of consumer client reads for any group is within the limit. Please see [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for more information. +## Troubleshooting + +### Event Hubs Exception + +An `EventHubsException` is triggered when an operation specific to Event Hubs has encountered an issue, including both errors within the service and specific to the client. The exception includes some contextual information to assist in understanding the context of the error and its relative severity. These are: + +- `IsTransient` : This identifies whether or not the exception is considered recoverable. In the case where it was deemed transient, the appropriate retry policy has already been applied and retries were unsuccessful. + +- `Reason` : Provides a set of well-known reasons for the failure that help to categorize and clarify the root cause. These are intended to allow for applying exception filtering and other logic where inspecting the text of an exception message wouldn't be ideal. Some key failure reasons are: + + - **Client Closed** : This occurs when an operation has been requested on an Event Hub client that has already been closed or disposed of. It is recommended to check the application code and ensure that objects from the Event Hubs client library are created and closed/disposed in the intended scope. + + - **Service Timeout** : This indicates that the Event Hubs service did not respond to an operation within the expected amount of time. This may have been caused by a transient network issue or service problem. The Event Hubs service may or may not have successfully completed the request; the status is not known. It is recommended to attempt to verify the current state and retry if necessary. -#### Message Size Exceeded + - **Quota Exceeded** : This typically indicates that there are too many active read operations for a single consumer group. This limit depends on the tier of the Event Hubs namespace, and moving to a higher tier may be desired. An alternative would be to create additional consumer groups and ensure that the number of consumer client reads for any group is within the limit. Please see [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for more information. -Event data, both individual and in batches, have a maximum size allowed. This includes the data of the event, as well as any associated metadata and system overhead. The best approach for resolving this error is to reduce the number of events being sent in a batch or the size of data included in the message. Because size limits are subject to change, please refer to [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for specifics. + - **Message Size Exceeded** : Event data as a maximum size allowed for both an individual event and a batch of events. This includes the data of the event, as well as any associated metadata and system overhead. The best approach for resolving this error is to reduce the number of events being sent in a batch or the size of data included in the message. Because size limits are subject to change, please refer to [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for specifics. + + - **Consumer Disconnected** : A consumer client was disconnected by the Event Hub service from the Event Hub instance. This typically occurs when a consumer with a higher owner level asserts ownership over a partition and consumer group pairing. + + - **Resource Not Found**: An Event Hubs resource, such as an Event Hub, consumer group, or partition, could not be found by the Event Hubs service. This may indicate that it has been deleted from the service or that there is an issue with the Event Hubs service itself. + +Reacting to a specific failure reason for the `EventHubException` can be accomplished in several ways, such as by applying an exception filter clause as part of the `catch` block: +```csharp +try +{ + // Read events using the consumer client +} +catch (EventHubsException ex) where + (ex.Reason == EventHubsException.FailureReason.ConsumerDisconnected) +{ + // Take action based on a consumer being disconnected +} +``` + ### Other exceptions -For detailed information about these and other exceptions that may occur, please refer to [Event Hubs messaging exceptions](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-messaging-exceptions). +For detailed information about the failures represented by the `EventHubsException` and other exceptions that may occur, please refer to [Event Hubs messaging exceptions](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-messaging-exceptions). ## Next steps diff --git a/sdk/eventhub/Azure.Messaging.EventHubs/api/Azure.Messaging.EventHubs.netstandard2.0.cs b/sdk/eventhub/Azure.Messaging.EventHubs/api/Azure.Messaging.EventHubs.netstandard2.0.cs new file mode 100755 index 0000000000000..add481308de9a --- /dev/null +++ b/sdk/eventhub/Azure.Messaging.EventHubs/api/Azure.Messaging.EventHubs.netstandard2.0.cs @@ -0,0 +1,358 @@ +namespace Azure.Messaging.EventHubs +{ + public partial class EventData + { + public EventData(System.ReadOnlyMemory eventBody) { } + protected EventData(System.ReadOnlyMemory eventBody, System.Collections.Generic.IDictionary properties = null, System.Collections.Generic.IReadOnlyDictionary systemProperties = null, long sequenceNumber = (long)-9223372036854775808, long offset = (long)-9223372036854775808, System.DateTimeOffset enqueuedTime = default(System.DateTimeOffset), string partitionKey = null) { } + public System.ReadOnlyMemory Body { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.IO.Stream BodyAsStream { get { throw null; } } + public System.DateTimeOffset EnqueuedTime { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public long Offset { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string PartitionKey { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.Collections.Generic.IDictionary Properties { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public long SequenceNumber { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.Collections.Generic.IReadOnlyDictionary SystemProperties { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public partial class EventHubConnection : System.IAsyncDisposable + { + protected EventHubConnection() { } + public EventHubConnection(string connectionString) { } + public EventHubConnection(string connectionString, Azure.Messaging.EventHubs.EventHubConnectionOptions connectionOptions) { } + public EventHubConnection(string connectionString, string eventHubName) { } + public EventHubConnection(string fullyQualifiedNamespace, string eventHubName, Azure.Core.TokenCredential credential, Azure.Messaging.EventHubs.EventHubConnectionOptions connectionOptions = null) { } + public EventHubConnection(string connectionString, string eventHubName, Azure.Messaging.EventHubs.EventHubConnectionOptions connectionOptions) { } + public string EventHubName { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string FullyQualifiedNamespace { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public bool IsClosed { get { throw null; } } + public virtual void Close(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { } + public virtual System.Threading.Tasks.Task CloseAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.ValueTask DisposeAsync() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public partial class EventHubConnectionOptions + { + public EventHubConnectionOptions() { } + public System.Net.IWebProxy Proxy { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public Azure.Messaging.EventHubs.EventHubsTransportType TransportType { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public partial class EventHubProperties + { + protected internal EventHubProperties(string name, System.DateTimeOffset createdOn, string[] partitionIds) { } + public System.DateTimeOffset CreatedOn { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string Name { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string[] PartitionIds { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } + public partial class EventHubsException : System.Exception + { + public EventHubsException(bool isTransient, string eventHubName) { } + public EventHubsException(bool isTransient, string eventHubName, Azure.Messaging.EventHubs.EventHubsException.FailureReason reason) { } + public EventHubsException(bool isTransient, string eventHubName, string message) { } + public EventHubsException(bool isTransient, string eventHubName, string message, Azure.Messaging.EventHubs.EventHubsException.FailureReason reason) { } + public EventHubsException(bool isTransient, string eventHubName, string message, Azure.Messaging.EventHubs.EventHubsException.FailureReason reason, System.Exception innerException) { } + public EventHubsException(bool isTransient, string eventHubName, string message, System.Exception innerException) { } + public EventHubsException(string eventHubName, string message, Azure.Messaging.EventHubs.EventHubsException.FailureReason reason) { } + public string EventHubName { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public bool IsTransient { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public override string Message { get { throw null; } } + public Azure.Messaging.EventHubs.EventHubsException.FailureReason Reason { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public enum FailureReason + { + GeneralError = 0, + ClientClosed = 1, + ConsumerDisconnected = 2, + ResourceNotFound = 3, + MessageSizeExceeded = 4, + QuotaExceeded = 5, + ServiceBusy = 6, + ServiceTimeout = 7, + ServiceCommunicationProblem = 8, + } + } + public enum EventHubsRetryMode + { + Fixed = 0, + Exponential = 1, + } + public partial class EventHubsRetryOptions + { + public EventHubsRetryOptions() { } + public Azure.Messaging.EventHubs.EventHubsRetryPolicy CustomRetryPolicy { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public System.TimeSpan Delay { get { throw null; } set { } } + public System.TimeSpan MaximumDelay { get { throw null; } set { } } + public int MaximumRetries { get { throw null; } set { } } + public Azure.Messaging.EventHubs.EventHubsRetryMode Mode { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public System.TimeSpan TryTimeout { get { throw null; } set { } } + } + public abstract partial class EventHubsRetryPolicy + { + protected EventHubsRetryPolicy() { } + public abstract System.TimeSpan? CalculateRetryDelay(System.Exception lastException, int attemptCount); + public abstract System.TimeSpan CalculateTryTimeout(int attemptCount); + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public enum EventHubsTransportType + { + AmqpTcp = 0, + AmqpWebSockets = 1, + } + public partial class PartitionProperties + { + protected internal PartitionProperties(string eventHubName, string partitionId, bool isEmpty, long beginningSequenceNumber, long lastSequenceNumber, long lastOffset, System.DateTimeOffset lastEnqueuedTime) { } + public long BeginningSequenceNumber { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string EventHubName { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string Id { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public bool IsEmpty { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public long LastEnqueuedOffset { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public long LastEnqueuedSequenceNumber { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.DateTimeOffset LastEnqueuedTime { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } +} +namespace Azure.Messaging.EventHubs.Consumer +{ + public partial class EventHubConsumerClient : System.IAsyncDisposable + { + public const string DefaultConsumerGroupName = "$Default"; + protected EventHubConsumerClient() { } + public EventHubConsumerClient(string consumerGroup, Azure.Messaging.EventHubs.EventHubConnection connection, Azure.Messaging.EventHubs.Consumer.EventHubConsumerClientOptions clientOptions = null) { } + public EventHubConsumerClient(string consumerGroup, string connectionString) { } + public EventHubConsumerClient(string consumerGroup, string connectionString, Azure.Messaging.EventHubs.Consumer.EventHubConsumerClientOptions clientOptions) { } + public EventHubConsumerClient(string consumerGroup, string connectionString, string eventHubName) { } + public EventHubConsumerClient(string consumerGroup, string fullyQualifiedNamespace, string eventHubName, Azure.Core.TokenCredential credential, Azure.Messaging.EventHubs.Consumer.EventHubConsumerClientOptions clientOptions = null) { } + public EventHubConsumerClient(string consumerGroup, string connectionString, string eventHubName, Azure.Messaging.EventHubs.Consumer.EventHubConsumerClientOptions clientOptions) { } + public string ConsumerGroup { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string EventHubName { get { throw null; } } + public string FullyQualifiedNamespace { get { throw null; } } + public bool IsClosed { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] protected set { } } + public virtual System.Threading.Tasks.Task CloseAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.ValueTask DisposeAsync() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + public virtual System.Threading.Tasks.Task GetEventHubPropertiesAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + public virtual System.Threading.Tasks.Task GetPartitionIdsAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.Task GetPartitionPropertiesAsync(string partitionId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Collections.Generic.IAsyncEnumerable ReadEventsAsync(Azure.Messaging.EventHubs.Consumer.ReadEventOptions readOptions, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Collections.Generic.IAsyncEnumerable ReadEventsAsync(bool startReadingAtEarliestEvent, Azure.Messaging.EventHubs.Consumer.ReadEventOptions readOptions = null, [System.Runtime.CompilerServices.EnumeratorCancellationAttribute] System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Collections.Generic.IAsyncEnumerable ReadEventsAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Collections.Generic.IAsyncEnumerable ReadEventsFromPartitionAsync(string partitionId, Azure.Messaging.EventHubs.Consumer.EventPosition startingPosition, Azure.Messaging.EventHubs.Consumer.ReadEventOptions readOptions, [System.Runtime.CompilerServices.EnumeratorCancellationAttribute] System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Collections.Generic.IAsyncEnumerable ReadEventsFromPartitionAsync(string partitionId, Azure.Messaging.EventHubs.Consumer.EventPosition startingPosition, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public partial class EventHubConsumerClientOptions + { + public EventHubConsumerClientOptions() { } + public Azure.Messaging.EventHubs.EventHubConnectionOptions ConnectionOptions { get { throw null; } set { } } + public Azure.Messaging.EventHubs.EventHubsRetryOptions RetryOptions { get { throw null; } set { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)] + public partial struct EventPosition : System.IEquatable + { + private object _dummy; + private int _dummyPrimitive; + public static Azure.Messaging.EventHubs.Consumer.EventPosition Earliest { get { throw null; } } + public static Azure.Messaging.EventHubs.Consumer.EventPosition Latest { get { throw null; } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public bool Equals(Azure.Messaging.EventHubs.Consumer.EventPosition other) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + public static Azure.Messaging.EventHubs.Consumer.EventPosition FromEnqueuedTime(System.DateTimeOffset enqueuedTime) { throw null; } + public static Azure.Messaging.EventHubs.Consumer.EventPosition FromOffset(long offset, bool isInclusive = true) { throw null; } + public static Azure.Messaging.EventHubs.Consumer.EventPosition FromSequenceNumber(long sequenceNumber, bool isInclusive = true) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + public static bool operator ==(Azure.Messaging.EventHubs.Consumer.EventPosition first, Azure.Messaging.EventHubs.Consumer.EventPosition second) { throw null; } + public static bool operator !=(Azure.Messaging.EventHubs.Consumer.EventPosition first, Azure.Messaging.EventHubs.Consumer.EventPosition second) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)] + public partial struct LastEnqueuedEventProperties + { + public LastEnqueuedEventProperties(long? lastSequenceNumber, long? lastOffset, System.DateTimeOffset? lastEnqueuedTime, System.DateTimeOffset? lastReceivedTime) { throw null; } + public System.DateTimeOffset? EnqueuedTime { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.DateTimeOffset? LastReceivedTime { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public long? Offset { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public long? SequenceNumber { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } + public partial class PartitionContext + { + protected internal PartitionContext(string partitionId) { } + public string PartitionId { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public virtual Azure.Messaging.EventHubs.Consumer.LastEnqueuedEventProperties ReadLastEnqueuedEventProperties() { throw null; } + } + [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)] + public partial struct PartitionEvent + { + private object _dummy; + private int _dummyPrimitive; + public PartitionEvent(Azure.Messaging.EventHubs.Consumer.PartitionContext partition, Azure.Messaging.EventHubs.EventData data) { throw null; } + public Azure.Messaging.EventHubs.EventData Data { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public Azure.Messaging.EventHubs.Consumer.PartitionContext Partition { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } + public partial class ReadEventOptions + { + public ReadEventOptions() { } + public System.TimeSpan? MaximumWaitTime { get { throw null; } set { } } + public long? OwnerLevel { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public bool TrackLastEnqueuedEventProperties { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } +} +namespace Azure.Messaging.EventHubs.Processor +{ + public partial class PartitionClosingEventArgs + { + public PartitionClosingEventArgs(string partitionId, Azure.Messaging.EventHubs.Processor.ProcessingStoppedReason reason, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { } + public System.Threading.CancellationToken CancellationToken { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string PartitionId { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public Azure.Messaging.EventHubs.Processor.ProcessingStoppedReason Reason { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } + public partial class PartitionInitializingEventArgs + { + public PartitionInitializingEventArgs(string partitionId, Azure.Messaging.EventHubs.Consumer.EventPosition defaultStartingPosition, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { } + public System.Threading.CancellationToken CancellationToken { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public Azure.Messaging.EventHubs.Consumer.EventPosition DefaultStartingPosition { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public string PartitionId { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } + [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)] + public partial struct ProcessErrorEventArgs + { + private object _dummy; + private int _dummyPrimitive; + public ProcessErrorEventArgs(string partitionId, string operation, System.Exception exception, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public System.Threading.CancellationToken CancellationToken { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.Exception Exception { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string Operation { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public string PartitionId { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + } + [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)] + public partial struct ProcessEventArgs + { + private object _dummy; + private int _dummyPrimitive; + public ProcessEventArgs(Azure.Messaging.EventHubs.Consumer.PartitionContext partition, Azure.Messaging.EventHubs.EventData data, System.Func updateCheckpointImplementation, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public System.Threading.CancellationToken CancellationToken { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public Azure.Messaging.EventHubs.EventData Data { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public bool HasEvent { get { throw null; } } + public Azure.Messaging.EventHubs.Consumer.PartitionContext Partition { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } } + public System.Threading.Tasks.Task UpdateCheckpointAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + } + public enum ProcessingStoppedReason + { + Shutdown = 0, + OwnershipLost = 1, + } +} +namespace Azure.Messaging.EventHubs.Producer +{ + public partial class CreateBatchOptions + { + public CreateBatchOptions() { } + public long? MaximumSizeInBytes { get { throw null; } set { } } + public string PartitionId { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + public string PartitionKey { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] set { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public sealed partial class EventDataBatch : System.IDisposable + { + internal EventDataBatch() { } + public int Count { get { throw null; } } + public long MaximumSizeInBytes { get { throw null; } } + public long SizeInBytes { get { throw null; } } + public void Dispose() { } + public bool TryAdd(Azure.Messaging.EventHubs.EventData eventData) { throw null; } + } + public partial class EventHubProducerClient : System.IAsyncDisposable + { + protected EventHubProducerClient() { } + public EventHubProducerClient(Azure.Messaging.EventHubs.EventHubConnection connection, Azure.Messaging.EventHubs.Producer.EventHubProducerClientOptions clientOptions = null) { } + public EventHubProducerClient(string connectionString) { } + public EventHubProducerClient(string connectionString, Azure.Messaging.EventHubs.Producer.EventHubProducerClientOptions clientOptions) { } + public EventHubProducerClient(string connectionString, string eventHubName) { } + public EventHubProducerClient(string fullyQualifiedNamespace, string eventHubName, Azure.Core.TokenCredential credential, Azure.Messaging.EventHubs.Producer.EventHubProducerClientOptions clientOptions = null) { } + public EventHubProducerClient(string connectionString, string eventHubName, Azure.Messaging.EventHubs.Producer.EventHubProducerClientOptions clientOptions) { } + public string EventHubName { get { throw null; } } + public string FullyQualifiedNamespace { get { throw null; } } + public bool IsClosed { [System.Runtime.CompilerServices.CompilerGeneratedAttribute] get { throw null; } [System.Runtime.CompilerServices.CompilerGeneratedAttribute] protected set { } } + public virtual System.Threading.Tasks.Task CloseAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.ValueTask CreateBatchAsync(Azure.Messaging.EventHubs.Producer.CreateBatchOptions options, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.ValueTask CreateBatchAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.ValueTask DisposeAsync() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + public virtual System.Threading.Tasks.Task GetEventHubPropertiesAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + public virtual System.Threading.Tasks.Task GetPartitionIdsAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.Task GetPartitionPropertiesAsync(string partitionId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + public virtual System.Threading.Tasks.Task SendAsync(Azure.Messaging.EventHubs.Producer.EventDataBatch eventBatch, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } + public partial class EventHubProducerClientOptions + { + public EventHubProducerClientOptions() { } + public Azure.Messaging.EventHubs.EventHubConnectionOptions ConnectionOptions { get { throw null; } set { } } + public Azure.Messaging.EventHubs.EventHubsRetryOptions RetryOptions { get { throw null; } set { } } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override bool Equals(object obj) { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override int GetHashCode() { throw null; } + [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)] + public override string ToString() { throw null; } + } +} +namespace Microsoft.Extensions.Azure +{ + public static partial class EventHubClientBuilderExtensions + { + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubConsumerClientWithNamespace(this TBuilder builder, string consumerGroup, string fullyQualifiedNamespace, string eventHubName) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilderWithCredential { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubConsumerClient(this TBuilder builder, string consumerGroup, string connectionString) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilder { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubConsumerClient(this TBuilder builder, string consumerGroup, string connectionString, string eventHubName) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilder { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubConsumerClient(this TBuilder builder, TConfiguration configuration) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilderWithConfiguration { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubProducerClientWithNamespace(this TBuilder builder, string fullyQualifiedNamespace, string eventHubName) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilderWithCredential { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubProducerClient(this TBuilder builder, string connectionString) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilder { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubProducerClient(this TBuilder builder, string connectionString, string eventHubName) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilder { throw null; } + public static Azure.Core.Extensions.IAzureClientBuilder AddEventHubProducerClient(this TBuilder builder, TConfiguration configuration) where TBuilder : Azure.Core.Extensions.IAzureClientFactoryBuilderWithConfiguration { throw null; } + } +} diff --git a/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-second-preview.md b/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-second-preview.md index 787c9e54087e7..238019020d5ad 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-second-preview.md +++ b/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-second-preview.md @@ -1,389 +1,389 @@ -# .NET Event Hubs Client: Track Two Proposal (Second Preview) - -Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This lets you process and analyze the massive amounts of data produced by your connected devices and applications. Once Event Hubs has collected the data, you can retrieve, transform and store it by using any real-time analytics provider or with batching/storage adapters. If you would like to know more about Azure Event Hubs, you may wish to review: [What is Event Hubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about)? - -## Design Overview - -This design is focused on the second preview of the track two Event Hubs client library, and limits the scope of discussion to those areas with active development for the second preview. For wider context and more general discussion of the design goals for the track two Event Hubs client, please see the [.NET Event Hubs Client: Track Two Proposal (First Preview)](https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-first-preview.md). - -## Goals for the Second Preview - -- Adhere to, and advance, the goals outline in the the [.NET Event Hubs Client: Track Two Proposal (First Preview)](https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-first-preview.md). - -- Continue to align the public API surface area to the guidance outlined in the [Azure SDK Design Guidelines for .NET](https://azuresdkspecs.z5.web.core.windows.net/DotNetSpec.html). - -- Design an exception hierarchy that follows the overall pattern used by the track one client, limiting changes and allowing the published [exception guidance](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-messaging-exceptions) to remain as relevant as possible. - -- Streamline the design around timeouts and retries, making the common scenarios easier while allowing developers with advanced scenarios to customize. - -- Redesign the "streaming consumer" pattern of receiving events, continuing to offer a streamlined experience where the client library owns the message handling loop. - -- Reintroduce the `EventDataBatch` concept, allowing event producers to create a size-limited batch and control its contents. - -## Non-Goals for the Second Preview - -- Rewriting of the relevant areas of track one codebase; where possible, existing code will be used with as few modifications as possible, allowing the new API surface to be built on proven, reliable, and well-tested code. - -- Support for scenarios outside the identified targets or revisions to associated packages, such as the Event Hubs Processor; the initial efforts will be focused on the API surface for key operations in the core client library. - -- Ensuring that cancellation tokens are honored throughout the implementation; they will be accepted as part of the API but may not have an effect depending on acceptance and treatment by the infrastructure provided by the track one client library. - -## High Level Scenarios - -### Producers can Predict Event Batch Sizes - -Bob's Apple Farm has deployed a series of sensors into their orchard to help monitor soil conditions and make more efficient use of irrigation. The sensors submit their data for centralized analysis by publishing events to Azure Event Hubs. Because these sensors have been deployed to fields in a rural area, connectivity options are limited; the sensors are making use of a shared cellular service with limited bandwidth. - -To make efficient use of the available bandwidth, avoid unplanned data usage charges, and ensure that sensors are able to share the network fairly, they have been configured to publish events on a scheduled basis and would like to limit the event batches to a known, predictable, maximum size. - -### Consumers can Subscribe to Partition Events - -Bob's Apple Farm uses the events published by its sensors to feed a continuous analytics pipeline responsible for making informed decisions for managing the orchard in real-time, including when to irrigate, when soil conditions require fertilizer, and, most importantly, when there is a situation that requires human intervention. - -Because the processing of this event data is critical for managing the orchard, those responsible for development at Bob's Apple Farm are focused on reducing potential sources of errors, such as boilerplate code for an messaging loop, and prefer to keep their code focused on logic specific to their business. They would like to consume event data published by their sensors without the need to focus on controlling the size of batches, applying explicit back pressure, or worrying about exceptions that occur outside of their core processing logic. - -## .NET API for the Second Preview - -### Key Types - -#### `RetryOptions` - -The set of options available for configuring retry behaviors. This includes specifying a timeout per-try, replacing the concept of "operation timeouts" that were offered as part of client options types in the first preview. - -#### `RetryPolicy` - -Serves as the abstract base for retry policies, allowing developers with advanced retry needs to created customized retry behaviors. - -#### `OperationCancelledException` - -This occurs when an operation has been requested on a client, producer, or consumer that has already been closed or disposed of. It is recommended to check the application code and ensure that objects from the Event Hubs client library are created and closed/disposed in the intended scope. - -#### `TimeoutException` - -This indicates that the Event Hubs service did not respond to an operation within the expected amount of time. This may have been caused by a transient network issue or service problem. The Event Hubs service may or may not have successfully completed the request; the status is not known. It is recommended to attempt to verify the current state and retry if necessary. - -#### `MessageSizeExceededException` - -Event data, both individual and in batches, have a maximum size allowed. This includes the data of the event, as well as any associated metadata and system overhead. The best approach for resolving this error is to reduce the number of events being sent in a batch or the size of data included in the message. Because size limits are subject to change, please refer to [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for specifics. - -#### `QuotaExceededException` - -The messaging entity has reached its maximum allowable size. This exception can happen if the maximum number of consumers have already been created for a given partition and consumer group. Because limits are subject to change, please refer to [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for specifics. - -#### `DateTimeOffset` - -An intrinsic .NET type, the `DateTimeOffset` represents a point in time relative to Coordinated Universal Time (UTC). Event Hubs client library members representing date/time data are normalized to UTC and will change to using `DateTimeOffset` in the second preview to avoid the ambiguity around time zones associated with the use of `DateTime`. - -#### `IAsyncEnumerable` - -An intrinsic .NET type currently in preview, the `IAsyncEnumerable` enables iterating over an enumerable in an asynchronous way, allowing for an infinite sequence to be generated from a source requiring asynchronous communication. The second preview of the Event Hubs client library makes use of this concept in its approach for allowing a "streaming consumer." - -### Examples: Configuring Retries - -#### Creating an Event Hub client with default retry configuration - -```csharp -var connectionString = "<< CONNECTION STRING WITH EVENT HUB >>"; -var client = new EventHubClient(connectionString); -``` - -#### Creating an Event Hub client with custom retry configuration - -```csharp -var clientOptions = new EventHubClientOptions(); - -clientOptions.RetryMode = RetryMode.Fixed; -clientOptions.MaxRetries = 5; -clientOptions.Delay = TimeSpan.FromMilliseconds(250); -clientOptions.MaxDelay = TimeSpan.FromSeconds(2); -clientOptions.TryTimeout = TimeSpan.FromSeconds(90); - -var connectionString = "<< CONNECTION STRING WITH EVENT HUB >>"; -var client = new EventHubClient(connectionString); -``` - -#### Use a custom retry policy with an Event Hub client - -```csharp -public class CustomRetryPolicy : EventHubsRetryPolicy -{ - public override TimeSpan? CalculateRetryDelay(Exception lastException, - int retryCount - { - // CUSTOM LOGIC... - } -} - -var connectionString = "<< CONNECTION STRING WITH EVENT HUB >>"; -var client = new EventHubClient(connectionString); - -client.RetryPolicy = new CustomRetryPolicy(): -``` - -#### Create a producer with default retry configuration - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); -``` - -#### Create a producer with custom retry configuration - -```csharp -var producerOptions = new EventHubProducerOptions(); - -producerOptions.RetryMode = RetryMode.Fixed; -producerOptions.MaxRetries = 5; -producerOptions.Delay = TimeSpan.FromMilliseconds(250); -producerOptions.MaxDelay = TimeSpan.FromSeconds(2); -producerOptions.TryTimeout = TimeSpan.FromSeconds(90); - -var client = CreateClient(); -var producer = client.CreateProducer(producerOptions); -``` - -#### Use a custom retry policy with an Event Hub producer - -```csharp -public class CustomRetryPolicy : EventHubsRetryPolicy -{ - public override TimeSpan? CalculateRetryDelay(Exception lastException, - int retryCount - { - // CUSTOM LOGIC... - } -} - -var client = CreateClient(); -var producer = client.CreateProducer(); - -producer.RetryPolicy = new CustomRetryPolicy(): -``` - -#### Create a consumer with default retry configuration - -```csharp -var client = CreateClient(); -var consumer = client.CreateConsumer(EventHubConsumer.DefaultConsumerGroupName, "fcbac12-43cda", EventPosition.Earliest); -``` - -#### Create a consumer with custom retry configuration - -```csharp -var consumerOptions = new EventHubConsumerOptions -{ - RetryOptions = new RetryOptions - { - RetryMode = RetryMode.Exponential, - MaxRetries = 5, - Delay = TimeSpan.FromMilliseconds(250), - MaxDelay = TimeSpan.FromSeconds(2), - TryTimeout = TimeSpan.FromSeconds(90) - } -}; - -var client = CreateClient(); -var consumer = client.CreateConsumer("NotTheDefault", "fcbac12-43cda", EventPosition.Earliest, consumerOptions); -``` - -#### Use a custom retry policy with an Event Hub consumer - -```csharp -public class CustomRetryPolicy : EventHubsRetryPolicy -{ - public override TimeSpan? CalculateRetryDelay(Exception lastException, - int retryCount - { - // CUSTOM LOGIC... - } -} - -var client = CreateClient(); -var consumer = client.CreateConsumer("NotTheDefault", "fcbac12-43cda", EventPosition.Earliest); - -consumer.RetryPolicy = new CustomRetryPolicy(): -``` - -### Examples: Publishing with Event Data Batches - -#### Create an Event Data Batch with default options - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); - -using (var eventBatch = producer.CreateEventDataBatch()) -{ - // Use the batch -} -``` - -#### Create an Event Data Batch with custom options - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); - -var batchOptions = new BatchOptions -{ - MaximumSizeInBytes = 4096, - PartitionKey = "these-go-to-the-same-partition-with-service-choice" -} - -using (var eventBatch = producer.CreateEventDataBatch(batchOptions)) -{ - // Use the batch -} -``` - -#### Fill a batch - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); - -using (var eventBatch = producer.CreateEventDataBatch()) -{ - var eventData = GetNextEvent(); - - while (eventBatch.TryAdd(eventData)) - { - eventData = GetNextEvent(); - } - - // Use the event batch -} -``` - -#### Determine if an event is in an batch - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); - -using (var eventBatch = CreateAndFillBatch(producer)) -{ - var specialEvent = GetSpecialEvent(); - var eventInBatch = eventBatch.Contains(specialEvent); -} -``` - -#### Determine the current size of an batch - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); - -using (var eventBatch = CreateAndFillBatch(producer)) -{ - var maximumSizeInBytes = eventBatch.MaximumSize; - var batchSizeInBytes = eventBatch.CurrentSize; -} -``` - -#### Publish a batch - -```csharp -var client = CreateClient(); -var producer = client.CreateProducer(); - -using (var eventBatch = CreateAndFillBatch(producer)) -{ - // Options when sending an EventBatch are allowed only - // on the batch itself, they may not be also provided during - // the Send. - - await producer.Send(eventBatch); -} -``` - -### Examples: Consuming Events from a Partition - -#### Subscribe to events as they are available, waiting as needed - -```csharp -var client = CreateClient(); -var consumer = client.CreateConsumer(EventHubConsumer.DefaultConsumerGroupName, "fcbac12-43cda", EventPosition.Earliest); - -await foreach(var eventData in consumer.SubscribeToPartition(someCancellationToken)) -{ - ProcessEvent(eventData); -} -``` - -#### Subscribe to events as they are available, using a maximum wait time to return control - -```csharp -var client = CreateClient(); -var consumer = client.CreateConsumer(EventHubConsumer.DefaultConsumerGroupName, "fcbac12-43cda", EventPosition.Earliest); -var maximumWaitTime = TimeSpan.FromSeconds(2); - -await foreach(var eventData in consumer.SubscribeToPartition(maximumWaitTime, someCancellationToken)) -{ - // If the maximum wait time elapsed before an event was - // available, it will be null. This returns control and allows - // for breaking out of the loop. - - if (eventData != null) - { - ProcessEvent(eventData); - } -} -``` - -### Packages - -#### `Azure.Messaging.EventHubs` - -The main client library package, containing the core components for interacting with the Azure Event Hubs service. - -### Namespaces - -#### `Azure.Messaging.EventHubs` - -The top-level container for the types in the client library intended to be used by Event Hubs client library users. It is intended that types most frequently used by users for basic operations belong to this namespace to ensure ease of discovery. - -##### _Example types:_ - -- `RetryOptions` -- `EventHubsRetryPolicy` -- `BatchOptions` -- `EventDataBatch` - -#### `Azure.Messaging.EventHubs.Errors` - -The location for exceptions and error-related information and operations. Many of these types are surfaced as information for consumers in response to conditions encountered during a basic operation. It is not intended that consumers need to create these types directly. - -##### _Example types:_ - -- `OperationCancelledException` -- `TimeoutException` -- `MessageSizeExceededException` -- `QuotaExceededException` - -#### `Azure.Messaging.EventHubs.Core` - -The location for internal types used by the Event Hubs library to facilitate operations; these constructs are not intended to be consumed externally. - -##### _Example types:_ - -- `DefaultEventHubsRetryPolicy` -- `SystemMessagePropertyName` - -#### `Azure.Messaging.EventHubs.Amqp` - -The location for internal types used by the Event Hubs library to facilitate AMQP protocol-related activities; these constructs are not intended to be consumed externally. - -##### _Example types:_ - -- `AmqpMessageConverter` -- `AmqpEventDataBatch` - -#### `Azure.Messaging.EventHubs.Diagnostics` - -The location for internal types used by the Event Hubs library for diagnostics and logging activities; these constructs are not intended to be consumed externally. - -##### _Example types:_ - -- `EventHubsEventSource` -- `EventHubsDiagnosticSource` +# .NET Event Hubs Client: Track Two Proposal (Second Preview) + +Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This lets you process and analyze the massive amounts of data produced by your connected devices and applications. Once Event Hubs has collected the data, you can retrieve, transform and store it by using any real-time analytics provider or with batching/storage adapters. If you would like to know more about Azure Event Hubs, you may wish to review: [What is Event Hubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about)? + +## Design Overview + +This design is focused on the second preview of the track two Event Hubs client library, and limits the scope of discussion to those areas with active development for the second preview. For wider context and more general discussion of the design goals for the track two Event Hubs client, please see the [.NET Event Hubs Client: Track Two Proposal (First Preview)](https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-first-preview.md). + +## Goals for the Second Preview + +- Adhere to, and advance, the goals outline in the the [.NET Event Hubs Client: Track Two Proposal (First Preview)](https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/eventhub/Azure.Messaging.EventHubs/design/proposal-event-hubs-first-preview.md). + +- Continue to align the public API surface area to the guidance outlined in the [Azure SDK Design Guidelines for .NET](https://azuresdkspecs.z5.web.core.windows.net/DotNetSpec.html). + +- Design an exception hierarchy that follows the overall pattern used by the track one client, limiting changes and allowing the published [exception guidance](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-messaging-exceptions) to remain as relevant as possible. + +- Streamline the design around timeouts and retries, making the common scenarios easier while allowing developers with advanced scenarios to customize. + +- Redesign the "streaming consumer" pattern of receiving events, continuing to offer a streamlined experience where the client library owns the message handling loop. + +- Reintroduce the `EventDataBatch` concept, allowing event producers to create a size-limited batch and control its contents. + +## Non-Goals for the Second Preview + +- Rewriting of the relevant areas of track one codebase; where possible, existing code will be used with as few modifications as possible, allowing the new API surface to be built on proven, reliable, and well-tested code. + +- Support for scenarios outside the identified targets or revisions to associated packages, such as the Event Hubs Processor; the initial efforts will be focused on the API surface for key operations in the core client library. + +- Ensuring that cancellation tokens are honored throughout the implementation; they will be accepted as part of the API but may not have an effect depending on acceptance and treatment by the infrastructure provided by the track one client library. + +## High Level Scenarios + +### Producers can Predict Event Batch Sizes + +Bob's Apple Farm has deployed a series of sensors into their orchard to help monitor soil conditions and make more efficient use of irrigation. The sensors submit their data for centralized analysis by publishing events to Azure Event Hubs. Because these sensors have been deployed to fields in a rural area, connectivity options are limited; the sensors are making use of a shared cellular service with limited bandwidth. + +To make efficient use of the available bandwidth, avoid unplanned data usage charges, and ensure that sensors are able to share the network fairly, they have been configured to publish events on a scheduled basis and would like to limit the event batches to a known, predictable, maximum size. + +### Consumers can Subscribe to Partition Events + +Bob's Apple Farm uses the events published by its sensors to feed a continuous analytics pipeline responsible for making informed decisions for managing the orchard in real-time, including when to irrigate, when soil conditions require fertilizer, and, most importantly, when there is a situation that requires human intervention. + +Because the processing of this event data is critical for managing the orchard, those responsible for development at Bob's Apple Farm are focused on reducing potential sources of errors, such as boilerplate code for an messaging loop, and prefer to keep their code focused on logic specific to their business. They would like to consume event data published by their sensors without the need to focus on controlling the size of batches, applying explicit back pressure, or worrying about exceptions that occur outside of their core processing logic. + +## .NET API for the Second Preview + +### Key Types + +#### `RetryOptions` + +The set of options available for configuring retry behaviors. This includes specifying a timeout per-try, replacing the concept of "operation timeouts" that were offered as part of client options types in the first preview. + +#### `RetryPolicy` + +Serves as the abstract base for retry policies, allowing developers with advanced retry needs to created customized retry behaviors. + +#### `OperationCancelledException` + +This occurs when an operation has been requested on a client, producer, or consumer that has already been closed or disposed of. It is recommended to check the application code and ensure that objects from the Event Hubs client library are created and closed/disposed in the intended scope. + +#### `TimeoutException` + +This indicates that the Event Hubs service did not respond to an operation within the expected amount of time. This may have been caused by a transient network issue or service problem. The Event Hubs service may or may not have successfully completed the request; the status is not known. It is recommended to attempt to verify the current state and retry if necessary. + +#### `MessageSizeExceededException` + +Event data, both individual and in batches, have a maximum size allowed. This includes the data of the event, as well as any associated metadata and system overhead. The best approach for resolving this error is to reduce the number of events being sent in a batch or the size of data included in the message. Because size limits are subject to change, please refer to [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for specifics. + +#### `QuotaExceededException` + +The messaging entity has reached its maximum allowable size. This exception can happen if the maximum number of consumers have already been created for a given partition and consumer group. Because limits are subject to change, please refer to [Azure Event Hubs quotas and limits](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) for specifics. + +#### `DateTimeOffset` + +An intrinsic .NET type, the `DateTimeOffset` represents a point in time relative to Coordinated Universal Time (UTC). Event Hubs client library members representing date/time data are normalized to UTC and will change to using `DateTimeOffset` in the second preview to avoid the ambiguity around time zones associated with the use of `DateTime`. + +#### `IAsyncEnumerable` + +An intrinsic .NET type currently in preview, the `IAsyncEnumerable` enables iterating over an enumerable in an asynchronous way, allowing for an infinite sequence to be generated from a source requiring asynchronous communication. The second preview of the Event Hubs client library makes use of this concept in its approach for allowing a "streaming consumer." + +### Examples: Configuring Retries + +#### Creating an Event Hub client with default retry configuration + +```csharp +var connectionString = "<< CONNECTION STRING WITH EVENT HUB >>"; +var client = new EventHubClient(connectionString); +``` + +#### Creating an Event Hub client with custom retry configuration + +```csharp +var clientOptions = new EventHubClientOptions(); + +clientOptions.RetryMode = RetryMode.Fixed; +clientOptions.MaxRetries = 5; +clientOptions.Delay = TimeSpan.FromMilliseconds(250); +clientOptions.MaxDelay = TimeSpan.FromSeconds(2); +clientOptions.TryTimeout = TimeSpan.FromSeconds(90); + +var connectionString = "<< CONNECTION STRING WITH EVENT HUB >>"; +var client = new EventHubClient(connectionString); +``` + +#### Use a custom retry policy with an Event Hub client + +```csharp +public class CustomRetryPolicy : EventHubsRetryPolicy +{ + public override TimeSpan? CalculateRetryDelay(Exception lastException, + int retryCount + { + // CUSTOM LOGIC... + } +} + +var connectionString = "<< CONNECTION STRING WITH EVENT HUB >>"; +var client = new EventHubClient(connectionString); + +client.RetryPolicy = new CustomRetryPolicy(): +``` + +#### Create a producer with default retry configuration + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); +``` + +#### Create a producer with custom retry configuration + +```csharp +var producerOptions = new EventHubProducerOptions(); + +producerOptions.RetryMode = RetryMode.Fixed; +producerOptions.MaxRetries = 5; +producerOptions.Delay = TimeSpan.FromMilliseconds(250); +producerOptions.MaxDelay = TimeSpan.FromSeconds(2); +producerOptions.TryTimeout = TimeSpan.FromSeconds(90); + +var client = CreateClient(); +var producer = client.CreateProducer(producerOptions); +``` + +#### Use a custom retry policy with an Event Hub producer + +```csharp +public class CustomRetryPolicy : EventHubsRetryPolicy +{ + public override TimeSpan? CalculateRetryDelay(Exception lastException, + int retryCount + { + // CUSTOM LOGIC... + } +} + +var client = CreateClient(); +var producer = client.CreateProducer(); + +producer.RetryPolicy = new CustomRetryPolicy(): +``` + +#### Create a consumer with default retry configuration + +```csharp +var client = CreateClient(); +var consumer = client.CreateConsumer(EventHubConsumer.DefaultConsumerGroupName, "fcbac12-43cda", EventPosition.Earliest); +``` + +#### Create a consumer with custom retry configuration + +```csharp +var consumerOptions = new EventHubConsumerOptions +{ + RetryOptions = new RetryOptions + { + RetryMode = RetryMode.Exponential, + MaxRetries = 5, + Delay = TimeSpan.FromMilliseconds(250), + MaxDelay = TimeSpan.FromSeconds(2), + TryTimeout = TimeSpan.FromSeconds(90) + } +}; + +var client = CreateClient(); +var consumer = client.CreateConsumer("NotTheDefault", "fcbac12-43cda", EventPosition.Earliest, consumerOptions); +``` + +#### Use a custom retry policy with an Event Hub consumer + +```csharp +public class CustomRetryPolicy : EventHubsRetryPolicy +{ + public override TimeSpan? CalculateRetryDelay(Exception lastException, + int retryCount + { + // CUSTOM LOGIC... + } +} + +var client = CreateClient(); +var consumer = client.CreateConsumer("NotTheDefault", "fcbac12-43cda", EventPosition.Earliest); + +consumer.RetryPolicy = new CustomRetryPolicy(): +``` + +### Examples: Publishing with Event Data Batches + +#### Create an Event Data Batch with default options + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); + +using (var eventBatch = producer.CreateEventDataBatch()) +{ + // Use the batch +} +``` + +#### Create an Event Data Batch with custom options + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); + +var batchOptions = new BatchOptions +{ + MaximumSizeInBytes = 4096, + PartitionKey = "these-go-to-the-same-partition-with-service-choice" +} + +using (var eventBatch = producer.CreateEventDataBatch(batchOptions)) +{ + // Use the batch +} +``` + +#### Fill a batch + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); + +using (var eventBatch = producer.CreateEventDataBatch()) +{ + var eventData = GetNextEvent(); + + while (eventBatch.TryAdd(eventData)) + { + eventData = GetNextEvent(); + } + + // Use the event batch +} +``` + +#### Determine if an event is in an batch + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); + +using (var eventBatch = CreateAndFillBatch(producer)) +{ + var specialEvent = GetSpecialEvent(); + var eventInBatch = eventBatch.Contains(specialEvent); +} +``` + +#### Determine the current size of an batch + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); + +using (var eventBatch = CreateAndFillBatch(producer)) +{ + var maximumSizeInBytes = eventBatch.MaximumSize; + var batchSizeInBytes = eventBatch.CurrentSize; +} +``` + +#### Publish a batch + +```csharp +var client = CreateClient(); +var producer = client.CreateProducer(); + +using (var eventBatch = CreateAndFillBatch(producer)) +{ + // Options when sending an EventBatch are allowed only + // on the batch itself, they may not be also provided during + // the Send. + + await producer.Send(eventBatch); +} +``` + +### Examples: Consuming Events from a Partition + +#### Subscribe to events as they are available, waiting as needed + +```csharp +var client = CreateClient(); +var consumer = client.CreateConsumer(EventHubConsumer.DefaultConsumerGroupName, "fcbac12-43cda", EventPosition.Earliest); + +await foreach(var eventData in consumer.SubscribeToPartition(someCancellationToken)) +{ + ProcessEvent(eventData); +} +``` + +#### Subscribe to events as they are available, using a maximum wait time to return control + +```csharp +var client = CreateClient(); +var consumer = client.CreateConsumer(EventHubConsumer.DefaultConsumerGroupName, "fcbac12-43cda", EventPosition.Earliest); +var maximumWaitTime = TimeSpan.FromSeconds(2); + +await foreach(var eventData in consumer.SubscribeToPartition(maximumWaitTime, someCancellationToken)) +{ + // If the maximum wait time elapsed before an event was + // available, it will be null. This returns control and allows + // for breaking out of the loop. + + if (eventData != null) + { + ProcessEvent(eventData); + } +} +``` + +### Packages + +#### `Azure.Messaging.EventHubs` + +The main client library package, containing the core components for interacting with the Azure Event Hubs service. + +### Namespaces + +#### `Azure.Messaging.EventHubs` + +The top-level container for the types in the client library intended to be used by Event Hubs client library users. It is intended that types most frequently used by users for basic operations belong to this namespace to ensure ease of discovery. + +##### _Example types:_ + +- `RetryOptions` +- `EventHubsRetryPolicy` +- `BatchOptions` +- `EventDataBatch` + +#### `Azure.Messaging.EventHubs.Errors` + +The location for exceptions and error-related information and operations. Many of these types are surfaced as information for consumers in response to conditions encountered during a basic operation. It is not intended that consumers need to create these types directly. + +##### _Example types:_ + +- `OperationCancelledException` +- `TimeoutException` +- `MessageSizeExceededException` +- `QuotaExceededException` + +#### `Azure.Messaging.EventHubs.Core` + +The location for internal types used by the Event Hubs library to facilitate operations; these constructs are not intended to be consumed externally. + +##### _Example types:_ + +- `DefaultEventHubsRetryPolicy` +- `SystemMessagePropertyName` + +#### `Azure.Messaging.EventHubs.Amqp` + +The location for internal types used by the Event Hubs library to facilitate AMQP protocol-related activities; these constructs are not intended to be consumed externally. + +##### _Example types:_ + +- `AmqpMessageConverter` +- `AmqpEventDataBatch` + +#### `Azure.Messaging.EventHubs.Diagnostics` + +The location for internal types used by the Event Hubs library for diagnostics and logging activities; these constructs are not intended to be consumed externally. + +##### _Example types:_ + +- `EventHubsEventSource` +- `EventHubsDiagnosticSource` diff --git a/sdk/eventhub/Azure.Messaging.EventHubs/src/Azure.Messaging.EventHubs.csproj b/sdk/eventhub/Azure.Messaging.EventHubs/src/Azure.Messaging.EventHubs.csproj index a47cf47e358fa..d1eff85a966b0 100755 --- a/sdk/eventhub/Azure.Messaging.EventHubs/src/Azure.Messaging.EventHubs.csproj +++ b/sdk/eventhub/Azure.Messaging.EventHubs/src/Azure.Messaging.EventHubs.csproj @@ -1,11 +1,10 @@  Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This client library allows for both publishing and consuming events using Azure Event Hubs. For more information about Event Hubs, see https://azure.microsoft.com/en-us/services/event-hubs/ - 5.0.0-preview.7 + 5.0.0 Azure;Event Hubs;EventHubs;.NET;AMQP;IoT;$(PackageCommonTags) $(RequiredTargetFrameworks) false - false