[event-hubs] adds loadBalancingOptions and greedy load balancing strategy #9706

chradek · 2020-06-25T01:39:58Z

Fixes #6108 and implements #9140

Description

The change adds loadBalancingOptions to the EventHubConsumerClient options passed to the constructor. These options allow the user to configure the load balancing behavior when the client is using a CheckpointStore.

Breakdown of loadBalancingOptions

There are 3 fields in loadBalancingOptions:

strategy: this can be set to either "greedy" or "balanced".
updateIntervalInMs: Controls the frequency with which the client checks if the distribution of partitions across clients is balanced.
partitionOwnershipExpirationIntervalInMs: Controls how long a PartitionOwnership can be unmodified (claimed) before it is considered expired.

Options 2 and 3 will likely be set together (e.g. if the updateIntervalInMs is increased, then the partitionOwnershipExpirationIntervalInMs will likely need to be increased as well, otherwise the client might not get a change to reclaim a PartitionOwnership before it is considered expired.

Balanced strategy

This was the default strategy used before this change. Each updateIntervalInMs, the client will claim 1 partition until the partitions are evenly distributed across clients.

Greedy strategy

With this strategy, each updateIntervalInMs, the client will determine how many partitions it will need for all partitions to be evenly distributed across clients and claim them all.

PartitionLoadBalancer vs LoadBalancingStrategy

Previously, we had a PartitionLoadBalancer interface with 2 implementations: GreedyPartitionLoadBalancer and FairPartitionLoadBalancer .
With this change, I've redefined PartitionLoadBalancer as LoadBalancingStrategy and created 3 implementations: BalancedLoadBalancingStrategy, GreedyLoadBalancingStrategy, and UnbalancedLoadBalancedStrategy (open to other names for this one).

The FairPartitionLoadBalancer is roughly the same as the BalancedLoadBalancingStrategy. The main difference is that the BalancedLoadBalancingStrategy only returns the new partition to claim, whereas the FairPartitionLoadBalancer also returned all the partitions the client already owned.

The GreedyPartitionLoadBalancer is roughly the same as the UnbalancedLoadBalancingStrategy. Both of these will just return all the partition ids they're handed (they do no actual load balancing).

The GreedyLoadBalancingStrategy is brand new as part of this feature. It is similar to the BalancedLoadBalancingStrategy, except instead of returning just 1 partition, it returns all the partitions that are needed to get the client to a balanced state.

Why don't LoadBalancingStrategies return partitions already owned by the client?

This turned out to be unnecessary because the client already knows which partitions it owns. It also makes it easier to ensure that when we claim the partitions, we claim the ones we don't already own first (which decreases the likelihood that another client claims it first).

chradek · 2020-06-25T01:40:34Z

/azp run js - eventhubs-client - tests

azure-pipelines · 2020-06-25T01:40:44Z

Azure Pipelines successfully started running 1 pipeline(s).

chradek · 2020-06-25T01:44:28Z

/azp run js - eventhubs-client - tests

azure-pipelines · 2020-06-25T01:44:38Z

Azure Pipelines successfully started running 1 pipeline(s).

chradek · 2020-06-25T03:18:51Z

/azp run js - eventhubs-client - tests

azure-pipelines · 2020-06-25T03:19:01Z

Azure Pipelines successfully started running 1 pipeline(s).

xirzec

Seems like a righteous change, just a few little comments.

xirzec · 2020-06-25T19:49:16Z

sdk/eventhub/event-hubs/src/eventHubConsumerClient.ts

      }

      this._context = createConnectionContext(
        connectionStringOrFullyQualifiedNamespace2,
        this._clientOptions
      );
    }
+    this._loadBalancingOptions = this._clientOptions?.loadBalancingOptions ?? {


Consider defaulting all the options here rather than maybe defaulting the entire option bag just to re-default the keys later. Then you'll know they have real values when you need to use them.

Good call, updating.

xirzec · 2020-06-25T19:51:35Z

sdk/eventhub/event-hubs/src/eventProcessor.ts

-          }
-        }
+        // Renew the EventProcessor's partition ownerships that are still active.
+        // await this._renewPartitionOwnership(abortSignal);


delete commented code if it's not necessary

Woops, thought I had removed it, thanks for spotting!

xirzec · 2020-06-25T19:53:08Z

sdk/eventhub/event-hubs/src/eventProcessor.ts

-        // swallow the error since it's fine to exit early from delay
-        await delayWithoutThrow(this._loopIntervalInMs, abortSignal);
+        // Swallow the error since it's fine to exit early from the delay.
+        await delayWithoutThrow(delayDurationInMs, abortSignal);


in the case of delay 0, are we intentionally yielding here so at least microtasks can be processed? I think this behavior will be different in the browser if it's using setTimeout under the hood.

I'm not sure I understand what the concern you're getting at is.

In general this delay is in place just so that we have a gap between each iteration of our load balancing loop. If the duration happens to be 0 (because this iteration of the loop happened to exceed the loopIntervalInMs) it's still ok to yield, but we don't strictly speaking have to.

Oh sorry I didn't explain this very well.

My first thought was "can we skip the delay when it's 0?" but then I thought that not yielding might not let background tasks complete, so it's useful.

The other remark is that in the browser sometimes setTimeout(0) acts like setTimeout(16) due to various implementation details.

We do want to make sure that the loop yields at some point since there will be background tasks running.

Yeah, 16 ms isn't a big deal here. I'd expect that most of the time, the delayDurationInMs will be larger than 0 (the default delay is 10 seconds, and so we're subtracting the time it takes to make 1 call to event hubs and then some calls potentially to a service like storage if the user is using storage to checkpoint, which I wouldn't expect to take 10 seconds.) We're ultimately trying to limit how many service calls we make while still doing it often enough that we're able to detect when we need to make changes to reach a balanced state, so if it takes just a little longer that's fine.

I like having this yield even if delay is 0 because then we don't have to do extra checks just to handle the '0' case, and because then we're guaranteed at least 1 yield for background tasks 😄

xirzec · 2020-06-25T19:58:26Z

sdk/eventhub/event-hubs/src/eventProcessor.ts

+    for (const partitionToClaim of uniquePartitionsToClaim) {
+      let partitionOwnershipRequest: PartitionOwnership;
+
+      if (abandonedPartitionOwnershipMap.has(partitionToClaim)) {


since abandonedPartitionOwnershipMap is only used as an argument to _createPartitionOwnershipRequest I wonder if you could get away with having a map of all partitions (instead of a separate abandoned only one) and then you could use that map to call _createPartitionOwnershipRequest with both abandoned and claimed partitions.

You'd still need partitionOwnershipMap, but only for the call to identifyPartitionsToClaim

I made the change. Still have 2 maps, but get to remove this branching logic so that is nice.

xirzec · 2020-06-25T20:03:25Z

sdk/eventhub/event-hubs/src/models/public.ts

+ */
+export interface EventHubConsumerClientOptions extends EventHubClientOptions {
+  /**
+   * An options bag to configure load balancing settings.


are the option comments necessary or can we comment what each option does in the doc comments for LoadBalancingOptions?

These comments should be on the EventHubConsumerClientOptions interface, not the loadBalancingOptions field, will fix!

chradek · 2020-06-25T22:33:12Z

/azp run js - eventhubs-client - tests

azure-pipelines · 2020-06-25T22:33:24Z

Azure Pipelines successfully started running 1 pipeline(s).

chradek · 2020-06-29T23:31:00Z

/azp run js - eventhubs-client - tests

azure-pipelines · 2020-06-29T23:31:09Z

Azure Pipelines successfully started running 1 pipeline(s).

chradek · 2020-06-30T00:07:41Z

The single test failure in event hubs looks like it might just be a flaky test. I saw it failed once on Friday off of master (so without the changes from this PR). The core test failure looks like a flaky keyvaults test, which this PR definitely does not touch.

richardpark-msft

It looks good to me. Assuming all tests pass you're good to go.

chradek · 2020-07-02T18:53:47Z

/azp run js - eventhubs-client - tests

azure-pipelines · 2020-07-02T18:53:57Z

Azure Pipelines successfully started running 1 pipeline(s).

chradek · 2020-07-02T20:17:43Z

/check-enforcer reset

chradek · 2020-07-02T20:18:16Z

/check-enforcer override

* [FormRecognizer] whitelist header names and query parameters for logging (#9149) * [FormRecognizer] Update CHANGELOG for preview.3 (#9117) * [FormRecognizer] Update CHANGELOG for preview.3 * parameter names for preview.3 should be `formUrl` and `receiptUrl` * More renaming * data => form/receipt * Update sdk/formrecognizer/ai-form-recognizer/CHANGELOG.md Co-authored-by: Krista Pratico <[email protected]> * logging whitelist * useTrainingLabels parameter now required * intervalInMs => updateIntervalInMs * Remove USReceipt for preview.3 Co-authored-by: Krista Pratico <[email protected]> * Modified changelog for the June Release (#9317) * Modified changelog for the June Release * Response to PR Comments * Dummy Commit I * Revert Dummy Commit * [event-hubs] doc improvements (#9342) * [event-hubs] explain empty events * [event-hubs] readme update for 8798 * [event-hubs] migration-guide checkpointing 8419 * update changelog for 5.2.1 * Update sdk/eventhub/event-hubs/migrationguide.md Co-authored-by: Richard Park <[email protected]> * Decided to be more ambitious for the release date Co-authored-by: Richard Park <[email protected]> * [Service Bus] Token Credential Overload for ATOM Mgmt Client (#9316) * Update Queue APIs aper the .NET proposal * remove @internal @ignore tags wherever required and generate API report * more exports * more and more exports * Add token credential constructor overlaod * doesExist() for queue, topic and subscription * generate API report * api extractor and ts-config (TO BE REVERTED) * feedback from the service team * update index.ts * update API report * Update queue related imports in tests * revert api-extractor and tsconfig changes to allow compiling tests * fix build failures of ATOM API tests * changes to rule options and interfaces * fix test failures * changes to Topic APIs * fix build failures * generate API report * subscription APIs updated * fix build failures * generate API report * fix API report warnings * changes to rule APIs * fix build failures * generate API report * getNamespaceProperties API * namespace info improvement + API report * fix warnings in the API report * generatre API report * resolve merge conflicts and organize imports * fix the warnings from API report * API report * tests for getRuntimeInfo API for a single entity * remove messageCount from *Description APIs * remove unused helper * test for Get namespace properties * tests for entityExists * tests for getEntitiesRuntimeInfo * Add changelog * Changelog * tokenProvider attempt to fix * added servicebus scope in the getToken attribute * mark "@azure/identity" as external * Token credential * test for Token credential - DefaultAzureCredential from `@azure/identity` * update the name * Update sdk/servicebus/service-bus/src/serviceBusAtomManagementClient.ts Co-authored-by: chradek <[email protected]> * Update sdk/servicebus/service-bus/src/serviceBusAtomManagementClient.ts Co-authored-by: chradek <[email protected]> * enter one more code path in the tests for gettoken invocation * refactoring the constructor as suggested by Chris and fixing tests * fix trailing slash issue * Update the test to cover more cases * fix test for AAD * remove unneeded ! * fix endpoint and endpointWithProtocol attributes * Add getToken for SasServiceCredential * merge tokenProvider into credentials * remove "@azure/ms-rest-nodeauth" from the external list in rollup config for tests * Update sdk/servicebus/service-bus/rollup.base.config.js Co-authored-by: Ramya Rao <[email protected]> * private sharedKeyCredential Co-authored-by: chradek <[email protected]> Co-authored-by: Ramya Rao <[email protected]> * [ai-text-analytics] Various documentation-related changes (#9333) * [ai-text-analytics] Various documentation-related fixes * Fixed use of showStats instead of includeStatistics * Added alternative TA product slug * [ai-text-analytics] Change versions in package.json and constants to 1.0.0 (#9346) * Removed a couple of old recordings * Changed all preview version indicators to 1.0.0 * Switch to using the CredentialUnavailable error (#8127) * Switch to using the CredentialUnavailable error * exporting CredentialUnavailable and fixing tests * updating api spec * fixed missed test update Co-authored-by: Scott Schaab <[email protected]> * [Cosmos] Update Containers createIfNotExists() docs (#9355) The content of the comment in the method is the same as that described in `Databases.createIfNotExists()`, and it does not match the content of `Containers.createIfNotExists()`, so it was corrected. * Remove pipeline config for eventgrid, graphrbac, operationalinsights and servicefabric. (#9334) * Sdk automation/@azure arm mediaservices (#9192) * Update from master * version Co-authored-by: SDK Automation <[email protected]> * New aka.ms/azsdk links (#9034) * Duplicated ThirdPartyNotices in eventhub * no more https://azure.github.io/azure-sdk-for-js links * other link fixes * new aka.ms links * docs/<service> to <service>/docs, as agreed upon over Teams * Changed https://aka.ms/azure-sdk-preview1-js to https://aka.ms/azsdk/releases/July2019Preview * removed the versions * Update README.md * lowercase july2019preview * lowercase july2019preview * [Cosmos] Use constants for auth headers returned by generateHeaders function (#9354) * [event-hubs] fix flaky disconnect test (#9350) * [Event Hubs] Update Identity tests to use public clients (#9343) * [service-bus] T2 preview.3 readme and changelog update with date and version. (#9340) * [event-hubs] migration guide and test updates (#9367) * [event-hubs] improve migration guide checkpoint section * [event-hubs] make receciver disconnect even more tolerant! * Apply suggestions from code review Co-authored-by: Harsha Nalluru <[email protected]> Co-authored-by: Harsha Nalluru <[email protected]> * [FormRecognizer] Update June release date in CHANGELOG (#9368) * Increment package version after release of azure-search-documents (#9370) * Increment package version after release of azure-service-bus (#9372) * Minor change in the sample (#9373) * Minor change in the sample * Additional changes * [Event Hubs] Use EventHubConsumerClient in misc.spec.ts test file (#9341) * Increment package version after release of azure-event-hubs (#9377) Co-authored-by: azure-sdk <[email protected]> * [Service Bus] Sample for SBManagementClient (#9369) * ts sample * js sample * update samples readmes * readme updates for ATOM * Update sdk/servicebus/service-bus/samples/javascript/README.md Co-authored-by: Richard Park <[email protected]> Co-authored-by: Richard Park <[email protected]> * [core-http] Fix constant flattened property serialization (#8658) * Fix constant flattened property serialization * Update changelog * Update changelog * [Identity] Default credential error msg (#9374) * 123 * update * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Simplify azure-identity error messaging * Azure.Identity simplify exception messaging * Azure.Identity simplify exception messaging * Azure.Identity simplify exception messaging * Simple change to try to re-run CI * Get building and test passing * Simplify error message and properly use CredentialUnavailable * Fix test Co-authored-by: Ziheng Zhou <[email protected]> Co-authored-by: zzhxiaofeng <[email protected]> * [Service Bus] Tests for SBClient token credential overload (#9347) * Add "@azure/identity" to the external list for test rollup * undo AAD test comments * Update sdk/servicebus/service-bus/test/serviceBusClient.spec.ts Co-authored-by: Richard Park <[email protected]> * remove unnecessary check Co-authored-by: Richard Park <[email protected]> * [ai-text-analytics] Final doc fixes before GA (#9382) * [ai-text-analytics] Remove dangling code fence * [ai-text-analytics] De-localize links. * [ai-text-analytics] add caveat note about TS 3.1 (#9407) * [ai-text-analytics] add caveat note about TS 3.1 * patch * [service-bus] Sender.createBatch cancellation support (#9233) - Instrumenting abortSignal futher in the sender (going through init()). - Replacing the _checkAndSetupAbortSignalCleanup() pattern in _trySend() with the newer shared utility waitForTimeoutOrAbortOrResolve function. - Update launch.json so you can just hit F5 and run the unit tests target. Part of #4375 * [service-bus] Fixing some issues with the samples (#9412) - Needed to add in AsyncIterable - `useProxy.ts` was incorrectly loading ws, causing it to not actually use a proxy. - `useProxy.ts` wasn't compiling against later versions of https-proxy-agent * [service-bus] Fix formatting issues for samples (#9421) * Sync eng/common directory with azure-sdk-tools repository (#9411) * [FormRecognizer] Update release date in CHANGELOG (#9422) * [FormRecognizer] Update sample in README and ref docs (#9441) * Update changelog for identity 1.1.0-preview.4 (#9356) * [Event Hubs] Use consumer & producer client in disconnects.spec.ts (#9427) * Fix sample optimisticConcurrencyViaEtag (#9451) * Min-Max Dependency testing (#8919) * Delete unused file "eng/pipelines/docs.yml" (#9453) * [Event Hubs] Update startingFromHere helper to stop using EventHubClient (#9431) * Update default Node version from 10 to 12 (#9454) * Update dev dependency "rollup-plugin-visualizer" (#9330) * Update runtime dependency "tslib" (#9322) * [appconfig] Updating JavaScript sample to include changes from #9451. (#9461) - Adding in a check that'll at least print when the results are incorrect so we don't run into this again. - Fixing package.json - we don't have a next tag now that AppConfig is GA. * Update dev dependency "puppeteer" (#9328) * Increment package version after release of azure-ai-form-recognizer (#9447) * Release for mgmt hdinsight (#9464) * [service-bus] Throw an error if the user tries to just send a string (sender.send(string)) (#9381) Throw an error if a user accidentally passes something that isn't a ServiceBusMessage (or it's accepted variations) Also: - Fixing some fairly obvious issues that eslint found - Making it we use the various methods for sending, not just single message. Mostly just to get some simple test coverage in our smoke testing. Fixes #9167 * rush update --full (#9463) * [FormRecognizer] rename includeTextDetails to includeTextContent (#9486) * [FormRecognizer] rename includeTextDetails to includeTextContent in the custom form/receipt recognition options. This was missed in the consistency push. * Update changelog * [event-hubs] fix subscription deadlock (#9491) * [service-bus] T2: Enabling abort in receiveBatch and subscribe (#9284) - Plumbing abortSignal through so it can be used to cancel the init() call in subscribe() and any part of receiveBatch() - Standardizing our AbortError message for all the new abortSignal work for Sender and Receiver. - Factoring out the cleanup code. Part of #4375 * [core-amqp] ensure sendRequest timeout is cleared for all responses (#9494) * [core-amqp] ensure sendRequest timeout is cleared for all responses * Update sdk/core/core-amqp/test/requestResponse.spec.ts Co-authored-by: Richard Park <[email protected]> * address feedback Co-authored-by: Richard Park <[email protected]> * release for mgmt cosmosdb (#9465) * Increment package version after release of azure-cosmos (#9366) * [FormRecognizer] Update live test resource region (#9419) This is to avoid busy region like westus2 * Verify node version after install (#9498) * [Cosmos] 3.7.1 Fixes null authToken for deleting items and IE browser crypto bug (#9512) * 3.7.1 * Remove throw * Restore old auth token behavior * Add second bugfix * Update sdk/cosmosdb/cosmos/test/functional/authorization.spec.ts Co-authored-by: Zachary Foster <[email protected]> * Assert status code * Remove only Co-authored-by: Zachary Foster <[email protected]> * Increment package version after release of azure-cosmos (#9517) * [Event Hubs] Use consumer & producer client in hubruntime.spec.ts (#9428) * Update swagger_to_sdk_config.json (#9527) upgrade typescript code-gen to latest version * [Event Hubs] Remove EventHubClient dependency in ProducerClient (#9520) * Update rush (#9542) * Release for mgmt datafactory (#9437) * [Cosmos] Fix bug where abort signal was being called immediately if passed by the user (#9540) * Increment package version after release of azure-cosmos (#9561) * Sync eng/common directory with azure-sdk-tools repository (#9489) * Update pnpm to v5 (#9547) * rush update --full (#9566) * private feed publish (#9306) * Renames and Refactorings (#9553) * Change analyzer-analyzerName in AnalyzeTextOptions * Change tokenizer-tokenizerName to CustomAnalyzer * Chg includeTotalResultCount-includeTotalCount * Minor Comment Added * [Cosmos] Remove references to PK on options objects (#9546) * Move text from method paramter to options bag (#9570) * Reference tools repo version (#9577) * Add initial Refs for repositiory * Initial Changes to refs JS * Switch template referencing to use extends * Remove resources from archetype-sdk-integration.yml * [Event Hubs] Test err scenarios against EventHubConsumerClient, not EventHubConsumer (#9521) * [Event Hubs] Remove the middle layer of EventHubProducer (#9548) * Update codeowners (#9576) * [KeyVault] Common folder (#8866) * Update challengeBasedAuthenticationPolicy.ts * Common folder for the challenge based auth During testing, I get this error: Error: Dynamic requires are not currently supported by @rollup/plugin-commonjs * deleting this from certificates again * wip * fixes from https://github.com/ljian3377/azure-sdk-for-js/commit/0ce33f73d14516e9abe2206e113381c98275c882 * other changes that make sense after rebasing * making sure keyvault-common is up to date * same changes on keys and secrets * formatting * this .gitignore is not needed * bringing the hotfix to this PR and making sure it passes CI * new pnpm-lock after merging master * package.json and eslintrc updates * fixed end of file on keyvault-common/src/index.ts * some of the feedback * standard tsconf on keyvault-common * tsconfig.json cleanup * removed unnecessary scripts from keyvault-common * [event-hubs] update EventHubConsumerClient.close to close subscriptions (#9567) * [event-hubs] update EventHubConsumerClient.close to close subscriptions * keep references to Subscriptions instead of EventProcessors and update test * fix EventHubConsumerClient waiting 1 min before exit (#8598) * add test * update changelog * [Key Vault] Separating public and internal folders (and fixing the sample paths) (#9584) * [Key Vault] Separating public and internal folders * TestMinMax: true * cleanups after merging the latest master * broke the build by mistake while fixing conflicts * aligning secrets to the path changes to keys and certificates * samples path fix * [core-https] Port request policies to new pipeline model (#9468) Port over existing policies from core-http. * [Service Bus] ATOM API - Merging the return types (#9432) * merge response interfaces * update exports * API report * extends Response and update DOC comments * NamespaceResponse -> NamespacePropertiesResponse * API Report * listRequestOptions -> options for Richard! * API report * Pin dev dependency "follow-redirects" (#9609) - [email protected] adds a peer dependency on "debug" - [email protected] depends on follow-redirects@^1.0.0, but does not depend on debug, which causes an error due to missing peer dependencies - rush update --full * [event-hubs] Fixing websockets sample (typescript one needed actual changes to both tsconfig.json and websockets.ts). (#9613) Propagating the fixes we found for #9412 (websockets/proxy samples were broken) to the sample here. Some small issues: - The tsconfig.json needed to have esModulesInterop turned on - package.json for both the JS and TS samples were referencing a tag we no longer publish (changed to 'latest' rather than 'next') - Small changes for the TS so it can compile properly using `import`, rather than `require`. Fixes #9415 * Unpin dev dependency "follow-redirects" (#9614) - [email protected] removes peer dependency "debug" - Fixes #9612 * [Service Bus] Convert ISO-8601 timestamp string into Date type (#9434) * Helper method to convert ISO-8601 time into Date type * createdAt, accessedAt, updatedAt * update Tests * API Report * getDateOrUndefined -> getDate * API report * "At" -> "On" * API Report * release for arm synapse (#9573) Co-authored-by: Shixun Guan <[email protected]> * [Identity] Remove unneeded devDependency "test-utils-recorder" (#9617) * remove unneeded devDependency * fix karma.conf * Update azure-sdk-tools Repository Resource Refs in Yaml files (#9606) * Update azure-sdk-build-tools Repository Resource Refs in Yaml files (#9605) * release for arm vmware (#9502) update client name for python and js Co-authored-by: SDK Automation <[email protected]> * [FormRecognizer] fix JavaScript copy model sample (#9460) to remove the TypeScript keyword from copy-n-paste * Increment package version after release of azure-ai-text-analytics (#9414) * Add @sarangan12 as a search owner (#9636) * [FormRecognizer] Simplify the samples to get poller results (#9488) * [core-http] Run formatter (#9631) * [core-http] Update copyright text (#9632) * Sync eng/common directory with azure-sdk-tools repository (#9610) * update logic to registry check (#9640) * Remove unneeded awaits for PagedAsyncIterableIterators (#9065) * [storage-blob] clean up container iteration readme The async iterator is returned synchronously, so no need for the `await`. * Fix other storage readmes * Fix a bunch of needless awaits * Fix overzealous deawaiting Co-authored-by: Ramya Rao <[email protected]> * [Event Hub] Update ConsumerClient, Processor, PumpManager & Pump to not use EventHubClient (#9600) * [core-http] change browser telemetry header key to x-ms-useragent (#9490) Based on the analysis at https://github.com/Azure/azure-sdk-for-js/issues/6530#issuecomment-642326789 * KeyVault service doesn't support CORS yet which means no browser support * Storage doesn't send x-ms-command-name or x-ms-useragent for browsers as they are not using core-http user agent policy * Event Hubs/Service Bus don't use http * Rest of services allows the x-ms-useragent header key it is safe to change the header key for browser user agent information from `x-ms-command-name` to `x-ms-useragent` in core-http. Updates to AppConfiguration and Storage libraries will be in separate PRs. * [Service Bus] Bug Fix - Same seq number returned for parallel schedule requests (#9503) * Add message_id to the request to be sent * backup message_id * test for parallel scheduled messages * backup correlation_id * Add a test - request without `message_id` should not match a response with `undefined` correlationId * remove reqst.correlation_id check * Update sdk/core/core-amqp/src/requestResponseLink.ts Co-authored-by: Ramya Rao <[email protected]> * update test with an additional check * Update the test as Ramya suggested * check message bodies too * update the _makeManagementRequest to add message_id Co-authored-by: Ramya Rao <[email protected]> * [core-http] Fix simple linting errors & warnings (#9663) * Code changes to nest user defined fields into a document property (#9655) * Code changes to nest user defined fields into a document property * Code changes to address PR Comments * [core-client] Create package for AutoRest specific code (#9620) Create a new package we can use for helpers for generated code. * [search-documents] Add sentinel value to connection string (#9672) * Add sentinel value to connection string * Update sdk/search/search-documents/src/searchIndexerClient.ts Co-authored-by: Jeff Fisher <[email protected]> Co-authored-by: Jeff Fisher <[email protected]> * [Service Bus] Fix "disconnect" tests in browser with rhea's bug-fix update (#9586) * unskip tests * update rhea versions * lock file * revert rhea caret updates * pnpm-lock from master * Only verify error.message if it exists * remove .only * Modified Readme file similar to .NET Readme (#9687) * Modified Readme file similar to .NET Readme * Minor change * Minor changes * Change Readme sample (#9697) * [Tables] Add the initial files for the track 2 Tables library (#9565) * Add the initial files for the track 2 tables library * Update the package.json file * Update pnpm-lock.yaml file * Fix small nits * Remove articats from ci.yml * Add downlevel types Co-authored-by: Mahdi Varposhti <[email protected]> * [Service Bus] Renames to use the messages suffix and remove singular alternatives (#9678) * [Event Hubs] Remove the middle layer of EventHubConsumer (#9623) * [Event Hubs] Run formatter (#9650) * add docs/rules back for eslint-plugin (#9705) * Avoid passing unused credentials to generated client constructors (#9635) * Add missing single quote (#9709) * Ignore src/generated folders when running check-format/format (#9670) * [Service Bus] Add changelog for recent service bus PRs (#9703) * Add changelog for recent service bus PRs * minor updates * request -> send request * Update sdk/core/core-amqp/CHANGELOG.md Co-authored-by: Ramya Rao <[email protected]> Co-authored-by: Ramya Rao <[email protected]> * [app-configuration] Fix issue with usages of 'crypto' package for browsers. (#9648) Our usage of the `crypto` package in appconfig causes an issue in browsers. This adds functions to hash and hmac that use the appropriate implementation for browser (WebCrypto) and the node.js crypto package for node. Fixes #9585 * [Service Bus] More info in the doc strings for ISO-8601 duration format (#9723) * Add link regarding ISO-8601 duration format * prefix the statement with "More on " * [Service Bus] Renaming ListRequestOptions {top, skip} -> {maxCount, skip} (#9664) * top -> maxCount * update tests * API Report * improve helper method for test * changelog * Update sdk/servicebus/service-bus/CHANGELOG.md Co-authored-by: Ramya Rao <[email protected]> * fix changelog format Co-authored-by: Ramya Rao <[email protected]> * Regeneration and Other changes (#9711) * Regeneration and Other changes * Test Build Fix * Minor Regenerate Change * [Core AMQP] Remove ThirdPartyNotices.txt for core-amqp (#9728) * Bump dependency core-http version to 1.1.4 (#9539) * Bump dependency core-http version to 1.1.4 to ensure that all packages have the `x-ms-useragent` browser header key change for telemetry. * Make exception for storage * [event-hubs] ensure error stack traces only logged in verbose mode (#9738) * [appconfig] Use core-http userAgentPolicy() (#9534) * [core-amqp] ensure error stack traces only logged in verbose mode (#9736) * [service-bus] Missing the necessary link for "what is service bus?" in the readme (#9740) We were missing the necessary link for "what is service bus?". Leading people to wonder "what is what is service bus?". Fixes #9683 * [core-auth] Update CHANGELOG for July release (#9741) * [core-amqp] prepare for 1.1.4 release (#9743) * [event-hubs] prepare for 5.2.2 release (#9745) * [keyvault-common,table] bump core-http dep version to 1.1.4 (#9739) * [keyvault-common,table] bump core-http dep version to 1.1.4 These two libraries were not updated in #9539, possibly due to the exception and tables change comes after the the exception was made. * Disable PR validation for Tables for now * Remove SkillSetCounter Py in ServiceCounters model (#9747) * [event-hubs][service-bus] update core-amqp to 1.1.4 (#9750) * [FormRecognizer] Renames based on Arch Board feedback (#9744) * [FR] Rename in swagger transformations * [FormRecognizer] Renames based on Arch Board feedback - Rename properties `requestedOn` to `trainingStartedOn` and `completedOn` to `trainingCompletedOn` in `CustomFormModel` and `CustomFormModelInfo` types. * Minor tweaks on log output * Update doc comments * [FormRecognizer] Add test coverage for Azure AD credential (#9752) * Enable AAD credential for recorded clients * Add a couple of recognition test to cover AAD credential * Add one training client test to cover AAD credential * [KeyVault] Some name cleanups and formattings (#9693) * [KeyVault] Some name cleanups and formattings * package.json types cleanup * keyvault-keys node recordings * keyvault-secrets browser and node recordings * keyvault-certificates node recordings * keys recordings * karma port changes * cleaning things up * keys recordings * browser recordings for certificates * missing keys recordings * fixing keyvault-keys tests by not running the crypto tests in the browser * better comment about local cryptography on the local crypto test * [KeyVault-Keys] The CryptographyClient indeed works with a KeyVaultKey (#9647) * [KeyVault-Keys] The CryptographyClient indeed works with a KeyVaultKey * sample readme fixes * undoing irrelevant change * no need for another sample file * https://github.com/Azure/azure-sdk-for-js/pull/9647#discussion_r444006990 * formatting * I forgot to check this in * removed unnecessary comment * documentation on the key paramter of the CryptographyClient constructor * I collapsed File Name into FileName in one of the READMEs by mistake * undoing the sample readme changes, theyre unnecessary * Update sdk/keyvault/keyvault-keys/src/cryptographyClient.ts Co-authored-by: Ramya Rao <[email protected]> * Update sdk/keyvault/keyvault-keys/samples/javascript/cryptography.js Co-authored-by: Ramya Rao <[email protected]> * Update cryptography.js * Update sdk/keyvault/keyvault-keys/samples/javascript/cryptography.js Co-authored-by: Ramya Rao <[email protected]> * Update sdk/keyvault/keyvault-keys/samples/typescript/src/cryptography.ts Co-authored-by: Ramya Rao <[email protected]> Co-authored-by: Ramya Rao <[email protected]> * [KeyVault] apiVersion documentation (#9656) * [KeyVault-Keys] apiVersion documentation * undid the samples changes * Update sdk/keyvault/keyvault-keys/README.md Co-authored-by: Ramya Rao <[email protected]> * Update sdk/keyvault/keyvault-keys/README.md Co-authored-by: Ramya Rao <[email protected]> * other clients Co-authored-by: Ramya Rao <[email protected]> * [KeyVault Admin] Generated client (#9668) * base files * rush.json addition * rbac generated code and other relevant changes * small fix * wip * no api-extractor for now * it builds * cleanups * prettier, eslint and file path fixes * two secret matches * moving things from core to generated Related to https://github.com/Azure/azure-sdk-for-js/issues/9069 * add-credentials to false * autorest fixes, from https://github.com/Azure/azure-sdk-for-js/pull/9668#discussion_r445201408 * One last instance of "Secret" in the README * no more build:minify * feedback from https://github.com/Azure/azure-sdk-for-js/pull/9668#discussion_r445149145 * 1.0.0-preview.1 since this preview is so far behind * copyright headers * tests.yml * Delete ThirdPartyNotice.txt * [KeyVault] Documenting the lack of browser support (#9716) * [KeyVault] Documenting the lack of browser support Here's what comes to mind. Feedback appreciated. Once approved: Fixes #9033 * typo * feedback, and same note in other READMEs * colon * feedback by Ramya * release for arm digitaltwins (#9681) * Generated from 0859f6ca9d516e1ae64f47d13c417a898fa6281f Update Typescript readme for SDK generation As requested by Azure JS SDK team * update version Co-authored-by: SDK Automation <[email protected]> Co-authored-by: Shixun Guan <[email protected]> * [Service Bus] Update relevant Docs for the update methods and and remove internal get calls (#9751) * Add Docs and remove get calls from the update requests * minor fix in docs * Changelog * fix update tests * final fix * fix updateSubscription tests * remove .only * update docs for update methods as Ramya suggested * Update sdk/servicebus/service-bus/CHANGELOG.md Co-authored-by: Ramya Rao <[email protected]> Co-authored-by: Ramya Rao <[email protected]> * Bump core-auth dependency version to 1.1.3 (#9754) for the ES5 compatibility fix. * [FormRecognizer] Enable copy model test (#9749) * [FormRecognizer] Enable copy model test and adding recordings * Remove some creating polling state from recording to shorten playback test run duration. * FR resource should also be created in westcentralus * Revert "Remove some creating polling state from recording" This reverts commit 669c0e44f27edd6d8a34547db9e126b6cf37717e. * Use model id from old recordings for copy model recorded tests * reverting what i accidentally changed in previoud PR (#7902) * Sync eng/common directory with azure-sdk-tools repository (#9720) * Update SECURITY.md (#9721) * Sync eng/common directory with azure-sdk-tools repository (#9790) * Generate id before extracting partition key (#9735) Co-authored-by: Abdou <[email protected]> Co-authored-by: Steve Faulkner <[email protected]> Co-authored-by: Zachary Foster <[email protected]> * [core-http] Share common code for exponential backoff (#9667) * [core-http] Refactor to share common code for exponential backoff for `ExponentialRetryPolicy` and `SystemErrorRetryPolicy` Address #4201 and #9569 * Update CHANGELOG * Address CR feedback * Update the release date in CHANGELOG * [FormRecognizer] address issues/feedbacks from UX study (#9673) * [FormRecognizer] address issues/feedback from UX study - Add ref doc to FormField.value and FormField.valueType * - Filter RecognizedForm.pages to those relevant to the recognized form * Default confidence to 1 if missing * - Remove `||` from sample snippets * [Cosmos] Increment package version after release of azure-cosmos (#9795) * [core-http] Upgrade to @azure/eslint-plugin-azure-sdk@^3.0.0 (#8687) * [core-http] Upgraded ESLint * Fixing the tests * formatting * These ESLint rules cant be applied in this repo * some of the feedback by Jeff * formatting * undoing this review API change * [event-hubs] Fixing flaky tests. (#8684) Fixing two flaky tests: - createBatch() was dumping out a Diagnostic-Id property into its messages. The issue was random tests overwriting the global tracer object with their test tracer. Now all those tests use the same utility function and cleanup after they're done. - There were a few tests that were using the date to choose their start position which ends up being unreliable in the face of clock skew. Changed over to use the same sequenceNumber positioning trick as other tests with a new testutils method. * [event-hubs] Fixing broken sample (#8695) Fixing missing 'async' keyword for snippet (and wrong client name) in readme. * undoing the eqeqeq changes on src/serializer * fixing the browser tests for core-http * formatting * no more jshint comments * Feedback from Jeremy regarding using Promise<never> * feedback from Jeff on test/mockHttp * stricter types on some serializers (from Jeremy) * comparison to undefined feedback (Ramya) * formatting * I couldnt do Jeffs advice * better solution for timeout * more improvements after updating from master * fixes after I applied eslint to the latest version of core-http * package.json fix after updating this branch based on the latest master * deserializeResponseBody is back as a non-async function, though it still returns a Promise * feedback from https://github.com/Azure/azure-sdk-for-js/pull/8687/files#r444542455 * Using RequestPolicyFactory in more places * eslint fixes after updating this branch based on master * very small lint fixes * Ignore unused arguments with _ prefix * headers.get() only returns string or undefined Co-authored-by: Richard Park <[email protected]> Co-authored-by: Ramya Achutha Rao <[email protected]> Co-authored-by: Jeremy Meng <[email protected]> * Release for arm network (#9529) Co-authored-by: SDK Automation <[email protected]> Co-authored-by: Shixun Guan <[email protected]> Co-authored-by: qiaozha <[email protected]> * Release for iotcentral 535 (#9803) Co-authored-by: SDK Automation <[email protected]> Co-authored-by: xichen <[email protected]> * [storage] add convenience methods exists, createIfNotExists, deleteIfExists (#8964) * [storage-blob] add CreateIfNotExists and DeleteIfExists for Container and Blob * [storage-flie-share] add CreateIfNotExists and DeleteIfExists * restore package.json * [storage-queue] add createIfNotExists and deleteIfExists * add exists for storage-file-share * add createIfNotExists and deleteIfExists for storage-file-datalake * record tests * fix typo * change return type - blob * change return type - datalake * change return type - file * change return type - queue Co-authored-by: Lin Jian <[email protected]> * Make sure we properly handle processError being treated as async. (#9797) Errors thrown _from_ processError will only be printed, not routed back into processError to prevent possible cascading failures. Fixes #7836 * Added Typescript Samples for Search Documents (#9809) * [FormRecognizer] Support passing in continuation token for next page (#9235) * Add mocked tests for listing multiple pages * [FormRecognizer] Fix broken ref docs about supported content types (#9518) * Mention file formats instead. * [Cosmos] Properly escape DEL character in partition key (#9822) * [Service Bus] Remove check for session exists when creating session receiver (#9766) * [Service Bus] Tracing expired sessions is not needed (#9765) * [app-configuration] Fixing readme with feedback from Cala. (#9823) * Fixing headings for the readme to match the standard. * prettier made some minor formatting changes that looked good. * [Service Bus] peekMessages to have maxMessageCount mandatory (#9798) * Increment package version after release of azure-cosmos (#9825) * [Service Bus] Add isServiceBusMessage checks to all methods that take messages (#9708) * [Service Bus] Change suffix for time stamp attributes from "On" to "At" (#9807) * remove duplicate copyright header * "On" -> "At" * Changelog * API Report * [ai-form-recognizer] Fix a silent merge conflict (#9827) * [Service Bus] Remove redundant docs between interfaces and implementations (#9826) * [storage] STG73 features preview (#9532) * storage-internal-avro parser (#8893) * IReadable * Avro compiles * use storage-internal-avro as shared source in storage-blob * local build config * test and bug fix * rename IReadable to AvroReadable, separate the stream implementation with the interface declaration we need different implementation in browsers * add "private": true to pacakge.json * use Record * me * fix * tentative: use string enum instead of string union to avoid two copies of constant string Co-authored-by: Lin Jian <[email protected]> * [Storage][Blob] Regenerate with Dec19 swagger * avor parser offset fix; Readable interface update (#9088) * [Storage] Regenerate API review * [storage] avro parser fix (#9154) * fix * resolve PR comments: remove useless assignment * AvroReadableFromStream: removeListener when read succeeds * remove listener when error callback triggered Co-authored-by: Lin Jian <[email protected]> * [Storage] Dec19 - Quickquery (#9112) * [Storage] Quick query * [Storage] Quick query API review * [Storage] Quick query recording * [Storage] Quick query test case update * [Storage] Resolve comments * regenerate storage-blob * re-generate storage-file-share * regenrate datalake and undo adding mode to storageClient and storageClientContext * update datalake service version to 2019-12-12 * [Storage] QuickQuery - Make BlobQueryCsvTextConfiguration properties optional (#9484) * [storage][blob] blob versioning (#7886) * blob versioning * support 'bv' SAS * add response headers to getProperties & api-extract * fix comments * add versionId to getBlob response headers * fix broken undelete test * undo 'add snapshot to options' * replace withVersionId() with getURLWithVersionId() * Revert "replace withVersionId() with getURLWithVersionId()" This reverts commit eb1d90d34783b7717ca0a6a651ba2c53a637be48. * hide the versionId option in convenience layer * add more tests * fix comments * change version to preview * fix merge error Co-authored-by: Lin Jian <[email protected]> * [storage] added change feed (#9151) * fix * change feed with mininal live tests * mocking tests * format code & rename buildAvroReader parameter name * add README.md * change eventTime type from string to Date * Update sdk/storage/storage-blob-changefeed/README.md Co-authored-by: Jeremy Meng <[email protected]> * Update sdk/storage/storage-blob-changefeed/README.md Co-authored-by: Jeremy Meng <[email protected]> * Update sdk/storage/storage-blob-changefeed/README.md Co-authored-by: Jeremy Meng <[email protected]> * PR comments * Rename eTag to etag in BlobChangeFeedEventData * remove hns only properties in change event record * fix minor bug in avro * Update sdk/storage/storage-blob-changefeed/README.md Co-authored-by: Brian Terlson <[email protected]> * Update sdk/storage/storage-blob-changefeed/README.md Co-authored-by: Brian Terlson <[email protected]> * rename to BlobChangeFeedGetChangesOptions and fix comments * update api Co-authored-by: Lin Jian <[email protected]> Co-authored-by: Jeremy Meng <[email protected]> Co-authored-by: Brian Terlson <[email protected]> * [storage][blob & datalake] Jumbo blob (#9480) * Jumbo putBlock, putBlob, datalake file append * undo upload ArrayBuffer * minor edit Co-authored-by: Lin Jian <[email protected]> * [Storage] Support blob tags (recording will be added later) (#9440) * [Storage] Support blob tags (recording will be added later) * [Storage] Unify blob tags type Tags everywhere in options and responses * [Storage] Blob tags - resolve comments * update service version to 2019-12-12 * blob tags test record * fix listContainer options * wip * record skipped test for Jumbo * set default formatter * me * wip * blob record node * record * add version check for x permission * record for browser * wip * wip * wip * wip * add f for account SAS permission * add permission f, and x for container * wip * me * me * PR comments * me * record changefeed * record with blobversioning disabled * record skipped test for browser * wip * avro compatible with Node v8 * fix test record * fix record * fix ae-forgotten-export * lint * fall back to util for old node.js TextEncoder * release pipeline for change feed * [storage][file] large smb files (#9624) * create large smb files * wip * record test * me Co-authored-by: Lin Jian <[email protected]> * [storage] added set Expiry (#9560) * add set expiry * add check for input * use Date instead of a time string * call blob endpoint directly in datalake * undo evenhub package.json * add alternative version and use number instead of decimal string for expiresOn relative * add timeToExpireInMs to option * use string literals instead of enum * keep old enum for backward compatibility * nit Co-authored-by: Lin Jian <[email protected]> * merge fix * omit doc in storage-internal-avro * Revert "[storage] added set Expiry (#9560)" This reverts commit c3b4414b09539c02266cc16b192f87f3d686dd8e. * nit * skip live tests for blob tags and quick query * change path for ts-samples Co-authored-by: Lin Jian <[email protected]> Co-authored-by: XiaoningLiu <[email protected]> Co-authored-by: Xiaoning Liu <[email protected]> Co-authored-by: Jeremy Meng <[email protected]> Co-authored-by: Brian Terlson <[email protected]> * [storage] version bump up for stg73 preview release (#9838) * use "rc" instead of preview and bump up versions for queue * nit Co-authored-by: Lin Jian <[email protected]> * add 'pack' script for avro as js - test - utils CI fail (#9841) Co-authored-by: Lin Jian <[email protected]> * Fix release artifacts parameter name for passing to stage-artifacts.yml (#9820) * rush update --full (#9848) * Sync eng/common directory with azure-sdk-tools repository (#9816) * [Service Bus] Bug Fix: userProperties aren't populated while using createRule() from ATOM API (#9794) * fix ruleResourceSerializer w.r.t userProperties * add test for userProperties * changelog * update error messages * remove console.log * Update sdk/servicebus/service-bus/CHANGELOG.md Co-authored-by: Ramya Rao <[email protected]> * remove extra types * Update enum names to TypeMaps * fix the 3 failed tests * keyValueTypeXMLTag * buildRawSqlParameter -> buildRawKeyValuePairFromSqlParameter * more docs and renames * Add test for array * InternalRawKeyValuePairs, refactoring and much more * scope the test to user-properties * refactor parameter.type * update error message * remove .only * fix error * unneeded .valueOf * Add boolean type * Validations * tests for Type validation errors on Correlation user property inputs * Update sdk/servicebus/service-bus/src/serializers/ruleResourceSerializer.ts * Update sdk/servicebus/service-bus/src/serializers/ruleResourceSerializer.ts * fix test failures Co-authored-by: Ramya Rao <[email protected]> * Release for appconfiguration 551 (#9832) * Generated from fbb8071c4a5769e8365fe4722a6ab3ef97720346 Update go readme since it was using 2019-11-01-preview API version. * release for appconfiguration 551 Co-authored-by: SDK Automation <[email protected]> Co-authored-by: xichen <[email protected]> * [Service Bus] Support for OperationOptions in management operations (#9654) * OperationOptionsForHTTP * API Report * OperationOptionsForHTTP * plug in OperationOptionsForHTTP * Make OperationOptionsForHTTP optional * fix build failures * API report * Fixes plugging in the options to webResource * TYPO * OperationOptions -> OperationOptionsForAMQP, OperationOptionsForHTTP -> OperationOptions * API Report * add some tests * OperationOptionsForAMQP -> AMQPOperationOptions * revert core-http changes * API Report * tests * fix build failures * Typos * API Report * AMQPOperationOptions -> OperationOptionsBase * API Report * Update sdk/core/core-http/src/webResource.ts * Changelog * fix operOptions for createSubscription * export { OperationOptions } from "@azure/core-http"; * API Report * Options description * Update docs as per the feedback * Update sdk/servicebus/service-bus/src/modelsToBeSharedWithEventHubs.ts Co-authored-by: chradek <[email protected]> * Update custom headers test, ensure the custom headers are populated as expected * fix test failures * API Report * fix test from merge resolution Co-authored-by: chradek <[email protected]> * Adding JS Samples (#9837) * Adding JS Samples * Dummy Commit * Revert Dummy Commit * [storage][stg73preview] use preview version (#9844) * undo .only * use preview version * common-version Co-authored-by: Lin Jian <[email protected]> * [Cosmos] Use tslint & fail build on lint errors (#9861) * [core-http] Fix a regression in exponential retry policy (#9852) The refactoring in PR #9667 introduce a behavior change: when the response is undefined we retry. This is different from the previous behavior that we don't retry on undefined response or response.status. Also the `delay()` and `_nextPolicy.sendRequest()` call should be inside the try-block. * Add tests address CR feedback * Update eslint plugin version to 3.0 (#9862) * [service-bus] Adding in support for stopping a Subscription for non-session `subscribe` (#9802) This first PR is just going to handle non-session closing of the result of a `subscribe()`* call. When close()'d we drain the receiver so no new messages will be received and leave the link _open_ so the user can still settle against it. To close the link they must close the outer `Receiver` object. *Normally we'd call it a subscription but that's already the name of an entity within service bus * Enable ChangeLog Verification in Repo (#9819) * [event-hubs] adds loadBalancingOptions and greedy load balancing strategy (#9706) * [event-hubs] adds EventHubConsumerClientOptions * fix conflicts from rebase * refactor load balancer to get all claimable partitions * add balanced and greedy load balancer strategies * add unbalancedLoadBalancingStrategy * update pumpManager to expose receivingFromPartitions * updates EventProcessor to use LoadBalancingStrategies * add tests for the partitionOwnershipExpirationIntervalInMs * add functional load balancing tests * update docs * update version to 5.3.0-preview.1 * add changelog * update pnpm-lock * address feedback * add explicity existance check to partitionOwnership.lastModifiedTimeInMs * trashing -> thrashing * be smarter about else-if statements * explain the magic number 6 * identifyPartitionsToClaim -> getPartitionsToClaim * identifyClaimablePartitions -> listAvailablePartitions * add better summary for EventHubConsumerClientOptions * add better docs around greedy and balanced strategies * remove superfluous doc from CommonEventProcessorOptions * add comment around why we have abandoned partitions * throw AbortError instead of silent return * remove unneeded receivingFrom method * update pnpm-lock.yaml * rush update * [Service Bus] Removing `isReceivingMessages` from the Receiver API Surface (#9875) * Remove `isReceivingMessages` from the API surface * API Report * fix build failures * remove `as any` * Add ### Breaking Changes sub-title * Update sdk/servicebus/service-bus/CHANGELOG.md Co-authored-by: Ramya Rao <[email protected]> Co-authored-by: Ramya Rao <[email protected]> * [service-bus] Adding in support for stopping a Subscription for session `subscribe` (#9849) Adding in support for stopping a session-based subscribe() call. Fixes #9493. * [Service Bus] Minor changelog updates (#9884) * Minor changelog updates * release date = Tuesday * [ai-form-recognizer] Anticipating support for non-text field elements (#9882) * [ai-form-recognizer] Renames to support non-text elements. * Samples * Fixed some missing renames in exception texts, fixed a typo in changelog. * [ai-form-recognizer] Use Date intead of number for expiresOn (#9876) * [app-configuration] updating changelog with release date (#9886) * Use subscription configuration template pattern that .NET and Java use (#9891) * [Cosmos] Add missing copyright text (#9887) * [Identity] Fix warning on unresolved dependencies (#9855) * [Identity] Fix warning on unresolved dependencies when building tests (#9897) * [arm-recoveryservicesbackup] Re-generate to fix missing nextLink (#9851) * [Key Vault Certificates] Happy path to avoid unstable live tests (#9879) * [Key Vault Certificates] Happy path to avoid variable behavior in live tests * forgot these .only * added timeout on the package.json somewhere I missed before * Sync eng/common directory with azure-sdk-tools repository (#9856) * [event-hubs] prepare for 5.3.0-preview.1 release (#9911) * [event-hubs] prepare for 5.3.0-preview.1 release * add unreleased entry for checkpointstore * [ai-form-recognizer] Remove RecognizedReceipt type (#9853) * [ai-form-recognizer] De-specialize receipt recognition * Add sample showing how to coerce the type of Form repsonses * Format * Fix ae-missing-export warnings * [Service Bus] Update preview.3 to preview.4 (#9913) * [Service Bus] Update preview.3 to preview.4 * update versions and hyperlinks * migration guide updates * Release Custom Vision Prediction & Training - July 2020 (#9906) * Changes to Prediction SDK * Code changes for TRaining SDK * Change version of Training to 5.1.0 * Increment package version after release of azure-core-http (#9907) * [Storage] avoid import fs directly in exporting modules (#9877) Put them in utils.node instead, and add stubs for browsers. * Increment package version after release of azure-event-hubs (#9930) * [ai-form-recognizer] Samples usability improvements (#9917) * [ai-form-recognizer] Samples usability improvements * Tweaked instructions * Fixes * Increment version for core releases (#9815) * Increment package version after release of azure-core-auth * Increment package version after release of azure-core-amqp * Increment version for storage releases (#9893) * Increment package version after release of azure-storage-queue * Increment package version after release of azure-storage-blob * Increment package version after release of azure-storage-file-datalake * Increment package version after release of azure-storage-file-share * Increment package version after release of azure-storage-blob-changefeed * use GA versions * update versions in build:autorest script Co-authored-by: Lin Jian <[email protected]> Co-authored-by: Jeremy Meng <[email protected]> * Increment package version after release of azure-search-documents (#9908) * [ai-form-recognizer] Last minute samples fixes from release testing. CHANGELOG release date. (#9936) * Increment package version after release of azure-service-bus (#9931) * [Cosmos] Remove Lazy Index Tests (#9932) Co-authored-by: Steve Faulkner <[email protected]> * [Storage] bump core-http dependency version to 1.1.4 (#9737) Revert "Make exception for storage" This reverts commit a3035f8256e11105d9d16615f7e7809aa40fd1ff. * [servicebus] docs: fix type in doc comment (#9941) There was disagreement about the returned type between the doc comment and the code itself. * Clean up Changelog and ci files (#9915) * Sync eng/common directory with azure-sdk-tools repository (#9934) * [Service Bus] atomXml.spec.ts has internal tests (#9946) * [storage] enalbe live tests for new features (#9892) * queue: use ./.env * file: skip large file test except playback * blob: un-skip live tests ex. browser, quick query * record undelete * fix intermittent test failure: wait 30s after containerClient.setAccessPolicy() * Add enableVersioning parameter to test-resources.json * Add ARM template parameter to enable versioning * "boolean" -> "bool" Co-authored-by: Lin Jian <[email protected]> Co-authored-by: Daniel Jurek <[email protected]> * Fix issue when debugging storage recorded tests (#9940) There's some difference between `npm run unit-test:node`/`npx _mocha ...` and `node ./node_modules/mocha/bin/_mocha` which we used to launch the debugger. The former did a probe to find a `mocha` executable in a set of paths and in our case it finds `./node_modules/.bin/mocha` and execute it. `./node_modules/.bin/mocha` is actually a shell script wrapping around `./node_modules/mocha/bin/mocha` and setting the `NODE_PATH` environment variable. One of the path included in this variable is particularly important: `<project-root>/common/temp/node_modules/.pnpm/node_modules`. It helped to resolve the indirect dependency of `nock` which otherwise wouldn't resolve under storage libraries' `./node_modules/`. So just launching `node ./node_modules/mocha/bin/_mocha` would lead to an error of `Cannot find module 'nock'`. This change adds a `NODE_PATH` environment variable to the VS Code launch configuration using `ts-node`. It should be safe to only include the `common/temp/node_modules/.pnpm/node_modules` because in our repository, all libraries' dependencies are symbol links to packages under pnpm's `node_modules` directory. The other configuration of debugging unit tests does not have this issue, possibly because the test file is a bundle. Fixes #9901. * [Service Bus] Clean up SubscriptionRuleManager (#9952) * fixing the logic to add to the common-versions.json (#9967) * Update rush and pnpm (#9965) * rush update --full (#9969) * Clean up resources (#9945) * Sync eng/common directory with azure-sdk-tools repository (#9964) * fix unstable test case (#9957) Co-authored-by: Lin Jian <[email protected]> * update readme of keyvault (#9978) * Remove exporting the DAC credentials list (#9972) * [Service Bus] Remove unused helper methods (#9990) * [Service Bus] Avoid using Dictionary (#9943) * release-for-mgmt-dns (#9979) * [core-amqp] close session after sender and receiver in requestResponseLink (#9998) * Preventing dev feed for private package on script level and with artifact parameter option in pipeline (#9948) * Sync eng/common directory with azure-sdk-tools repository (#10012) * [Service Bus] Renaming `userProperties` to `properties` - Both ServiceBusMessage and ATOM API (#10003) * Rename userProperties under ManagementClient too * fix test utils * rename userProps in the tests * one leftover * API Report * Changelog * fix build failure * Increment package version after release of azure-identity (#10017) Co-authored-by: azure-sdk <[email protected]> * verification of links pipelines (#9216) * fix eventhub link (#10019) * [Test Recorder] Fix browser custom console logging when recording (#10018) In function `customConsoleLog()` we save `window.console.log` to a local variable and use this variable to log to real console. The problem is that when `customConsoleLog()` is called multiple times, `window.console.log` is no longer the original one, and we end up nesting the modified `log` function again and again. This PR saves the original console log function to a module level variable instead so it's not wrapped multiple times. * Allow overriding consoleLog for testing * [Test Recorder] Fix an issue where not all browser requests are recorded (#10013) We use an async function to add a recorded request to an array then save the array to files when recorder stops. However we did not wait for the async function to complete, which leads to recording missing requests. This issue only happens in the browser case. In Node it is done in synchronous way. This change makes recorder to wait until all the recorded requests are added before saving them. All the tests have been updated to await for `recorder.stop()`. Co-authored-by: Harsha Nalluru <[email protected]> * [Key Vault Certificates] Test timeout need to be increased since now we wait for all pollers (#10031) * [Key Vault Certificates] Test timeout need to be increased since now we wait for all pollers * dependency-testing timeout * Adding dev-tool to the repo (#7872) * Azure SDK dev-tool first pass * [dev-tool] dev-tool dev-samples and fixes to all commands * Nested command structure * Better argument parsing, type-checking, and recursive command structure. Added support for running a single sample. * Added dev-tool README * [ai-text-analytics] Update package.json to use new script. * Removed some development cruft * prettier + eslint * Quick fix to ParsedOptions type * WIP * Command framework improvements * Basic unit-tests for package resolution. * One more test, assorted changes * README update * Migrated all packages with sample code to use dev-tool * Added dummy integration-test stub * Added dummy integration-test stub for eslint plugin * Added dummy integration-test:browser stubs * Added dev-tool dependency to packages using it * Corrected build:samples step in package.json * WIP * [dev-tool] ts-to-js command * [dev-tool] leaf command test * Fixed more deeply nested samples due to shared code * [Service Bus] Invalid param tests don't need live resources (#9947) * [Service Bus] Pick TestClientType randomly (#9950) * generate and versioning (#9198) Co-authored-by: Jeremy Meng <[email protected]> Co-authored-by: Krista Pratico <[email protected]> Co-authored-by: Sarangan Rajamanickam <[email protected]> Co-authored-by: chradek <[email protected]> Co-authored-by: Richard Park <[email protected]> Co-authored-by: Harsha Nalluru <[email protected]> Co-authored-by: Ramya Rao <[email protected]> Co-authored-by: Will Temple <[email protected]> Co-authored-by: Jonathan Turner <[email protected]> Co-authored-by: Scott Schaab <[email protected]> Co-authored-by: まっぴぃ⊿ <[email protected]> Co-authored-by: Mitch Denny <[email protected]> Co-authored-by: changlong-liu <[email protected]> Co-authored-by: SDK Automation <[email protected]> Co-authored-by: Daniel Rodríguez <[email protected]> Co-authored-by: Steve Faulkner <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: KarishmaGhiya <[email protected]> Co-authored-by: azure-sdk <[email protected]> Co-authored-by: Jose Manuel Heredia Hidalgo <[email protected]> Co-authored-by: Ziheng Zhou <[email protected]> Co-authored-by: zzhxiaofeng <[email protected]> Co-authored-by: Andreas Bjørn Hassing <[email protected]> Co-authored-by: Mike Harder <[email protected]> Co-authored-by: Qiaoqiao Zhang <[email protected]> Co-authored-by: Zachary Foster <[email protected]> Co-authored-by: chunyu3 <[email protected]> Co-authored-by: Chidozie Ononiwu <[email protected]> Co-authored-by: Jeff Fisher <[email protected]> Co-authored-by: Abyssknight <[email protected]> Co-authored-by: Shixun Guan <[email protected]> Co-authored-by: Brian Terlson <[email protected]> Co-authored-by: Jeff Fisher <[email protected]> Co-authored-by: Mahdi Varposhti <[email protected]> Co-authored-by: Mahdi Varposhti <[email protected]> Co-authored-by: Jeremy Meng <[email protected]> Co-authored-by: Josh Free <[email protected]> Co-authored-by: abouroubi <[email protected]> Co-authored-by: Abdou <[email protected]> Co-authored-by: Ramya Achutha Rao <[email protected]> Co-authored-by: qiaozha <[email protected]> Co-authored-by: xichen <[email protected]> Co-authored-by: xichen <[email protected]> Co-authored-by: Lin Jian <[email protected]> Co-authored-by: Lin Jian <[email protected]> Co-authored-by: XiaoningLiu <[email protected]> Co-authored-by: Xiaoning Liu <[email protected]> Co-authored-by: Daniel Jurek <[email protected]> Co-authored-by: Steve Faulkner <[email protected]> Co-authored-by: Matt Ellis <[email protected]>

chradek added 11 commits June 24, 2020 17:50

[event-hubs] adds EventHubConsumerClientOptions

faad867

fix conflicts from rebase

daa5920

refactor load balancer to get all claimable partitions

6b1e029

add balanced and greedy load balancer strategies

13cb669

add unbalancedLoadBalancingStrategy

5a0a56e

update pumpManager to expose receivingFromPartitions

aed4006

updates EventProcessor to use LoadBalancingStrategies

c3656e2

add tests for the partitionOwnershipExpirationIntervalInMs

f2205b8

add functional load balancing tests

ed1ff3b

update docs

793d95d

update version to 5.3.0-preview.1

d5f15e5

chradek added Client This issue points to a problem in the data-plane of the library. Event Hubs labels Jun 25, 2020

chradek requested review from bterlson, HarshaNalluru, ramya-rao-a, richardpark-msft and xirzec as code owners June 25, 2020 01:39

add changelog

961795e

update pnpm-lock

41202b5

xirzec reviewed Jun 25, 2020

View reviewed changes

address feedback

dc03b06

chradek added 12 commits June 29, 2020 14:42

be smarter about else-if statements

9a97ab4

explain the magic number 6

0a6a354

identifyPartitionsToClaim -> getPartitionsToClaim

ceb7c9a

identifyClaimablePartitions -> listAvailablePartitions

62add8a

add better summary for EventHubConsumerClientOptions

a313b66

add better docs around greedy and balanced strategies

5bb4827

remove superfluous doc from CommonEventProcessorOptions

2420c2b

add comment around why we have abandoned partitions

f203204

throw AbortError instead of silent return

e1a9fdb

remove unneeded receivingFrom method

12c3cd6

Merge remote-tracking branch 'upstream/master' into eh-load-balance-v2

7d84e15

update pnpm-lock.yaml

c2c31a7

This was referenced Jun 30, 2020

Increment version for eventhub releases #9824

Closed

[event-hubs] Load balancers can end up in an eternal fight for partitions #7017

Closed

[event-hubs] Load balancing can starve out new consumers #6945

Closed

Merge remote-tracking branch 'upstream/master' into eh-load-balance-v2

ea14b6e

chradek force-pushed the eh-load-balance-v2 branch from fd5091d to ea14b6e Compare July 1, 2020 22:57

rush update

db43e37

richardpark-msft approved these changes Jul 2, 2020

View reviewed changes

Merge remote-tracking branch 'upstream/master' into eh-load-balance-v2

52e3e0d

chradek merged commit 9b0d772 into Azure:master Jul 2, 2020

chradek deleted the eh-load-balance-v2 branch July 2, 2020 20:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[event-hubs] adds loadBalancingOptions and greedy load balancing strategy #9706

[event-hubs] adds loadBalancingOptions and greedy load balancing strategy #9706

chradek commented Jun 25, 2020

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

xirzec left a comment

xirzec Jun 25, 2020

chradek Jun 25, 2020

chradek Jun 25, 2020

xirzec Jun 25, 2020

chradek Jun 25, 2020

chradek Jun 25, 2020

xirzec Jun 25, 2020

chradek Jun 25, 2020

xirzec Jun 25, 2020

chradek Jun 25, 2020

xirzec Jun 25, 2020

chradek Jun 25, 2020

xirzec Jun 25, 2020

chradek Jun 25, 2020

chradek Jun 25, 2020

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

chradek commented Jun 29, 2020

azure-pipelines bot commented Jun 29, 2020

chradek commented Jun 30, 2020

richardpark-msft left a comment

chradek commented Jul 2, 2020

azure-pipelines bot commented Jul 2, 2020

chradek commented Jul 2, 2020

chradek commented Jul 2, 2020

[event-hubs] adds loadBalancingOptions and greedy load balancing strategy #9706

[event-hubs] adds loadBalancingOptions and greedy load balancing strategy #9706

Conversation

chradek commented Jun 25, 2020

Description

Breakdown of loadBalancingOptions

Balanced strategy

Greedy strategy

PartitionLoadBalancer vs LoadBalancingStrategy

Why don't LoadBalancingStrategies return partitions already owned by the client?

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

xirzec left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chradek commented Jun 25, 2020

azure-pipelines bot commented Jun 25, 2020

chradek commented Jun 29, 2020

azure-pipelines bot commented Jun 29, 2020

chradek commented Jun 30, 2020

richardpark-msft left a comment

Choose a reason for hiding this comment

chradek commented Jul 2, 2020

azure-pipelines bot commented Jul 2, 2020

chradek commented Jul 2, 2020

chradek commented Jul 2, 2020