Python EventHubs load balancing #6901
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
bryevdv
reviewed
Aug 22, 2019
sdk/eventhub/azure-eventhubs/azure/eventhub/eventprocessor/event_processor.py
Outdated
Show resolved
Hide resolved
bryevdv
reviewed
Aug 22, 2019
sdk/eventhub/azure-eventhubs/azure/eventhub/eventprocessor/event_processor.py
Outdated
Show resolved
Hide resolved
bryevdv
reviewed
Aug 22, 2019
| @@ -17,16 +17,16 @@ class CloseReason(Enum): | |||
| EVENTHUB_EXCEPTION = 2 # Exception happens during receiving events | |||
|
|
|||
|
|
|||
| class PartitionProcessor(ABC): | |||
| class PartitionProcessor(Protocol): | |||
There was a problem hiding this comment.
@johanste I am fairly new to MyPy, I may have some gaps in my understanding. From what I know, Protocol is useful to move runtime (type or structure) checks to something that MyPy can report on statically. But that's only useful to the people that are using MyPy, i.e. the developers of a library, more or less. To me this suggests that Protocol is most useful for things that end-users are not expected to implement themselves. If end-users are expected to make their own subclasses or implement a protocol, they can't be assumed to be using MyPy, so then you will still need all the explicit runtime checks (see all the hasattr checks above). In which case, isn't an ABC is better, so that the library devs don't have to perform these checks explicitly everywhere?
There was a problem hiding this comment.
When I changed from ABC to Protocol I was also thinking using a subclass looks more natural to me. Or did I use the Protocol in a wrong way? @johanste
There was a problem hiding this comment.
Changed back to use ABC instead of Protocol
johanste
reviewed
Aug 23, 2019
sdk/eventhub/azure-eventhubs/azure/eventhub/eventprocessor/_ownership_manager.py
Outdated
Show resolved
Hide resolved
johanste
reviewed
Aug 23, 2019
| self.all_parition_ids = [] | ||
| self.eventhub_client = eventhub_client | ||
| self.eventhub_name = eventhub_client.eh_name | ||
| self.consumer_group_name = event_processor._consumer_group_name |
There was a problem hiding this comment.
Code style; I prefer to not pass in a complex type (e.g. event_processor) in order to grab two properties from it. Especially since the properties in this case are private! It would be better to pass in consumer_group_name and partition_manager as separate parameters. This will avoid unnecessary coupling between the classes.
There was a problem hiding this comment.
removed event_processor and added consumer_group_name and owner_id as arguments
johanste
reviewed
Aug 23, 2019
| def __init__(self, event_processor, eventhub_client: EventHubClient, ownership_timeout: int): | ||
| self.all_parition_ids = [] | ||
| self.eventhub_client = eventhub_client | ||
| self.eventhub_name = eventhub_client.eh_name |
There was a problem hiding this comment.
Is there a specific reason that we make the eventhub_name an attribute on the instance, but that we also continue to get eh_name from the client instance later? Presumably, the values should stay the same....
There was a problem hiding this comment.
eventhub_name, consumer_group_name are two important attributes for an ownership. They usually appear together. They don't change anywhere in an ownership manager. Putting them together has better readability, I think.
There was a problem hiding this comment.
Let me rephrase, why are you both assigning the name to an attribute and accessing the name from self.eventhub_client.eh_name in other methods?
There was a problem hiding this comment.
That's a mistake. Changed all to self.eventhub_name
johanste
reviewed
Aug 23, 2019
| """ | ||
| self.all_parition_ids = await self.eventhub_client.get_partition_ids() | ||
|
|
||
| async def _balance_ownership(self): |
There was a problem hiding this comment.
_balance_ownership should take the partition ids as a parameter. This avoids using semi-global state. Presumably, the only reason we have to keep the partition_ids as an instance attribute is that we don't want to repeatedly query the values.
There was a problem hiding this comment.
Yes, partition_ids as an instance attribute is to avoid repeatedly querying. Changed _balance_ownership to take partition ids as a parameter.
johanste
reviewed
Aug 23, 2019
| if not ownership: # no ownership found for this partition. So it is claimable | ||
| claimable_partition_ids.append(partition_id) | ||
| else: | ||
| last_modified_time = ownership["last_modified_time"] |
There was a problem hiding this comment.
What is the value of last_modified_time supposed to be? Seconds since Jan 1, 1970?
There was a problem hiding this comment.
Yes, it's time.time( )
johanste
reviewed
Aug 23, 2019
| else: # the ownership is still active | ||
| if owner_id == self.owner_id: # partition is actively owned by this running EventProcessor | ||
| active_ownership_self.append(ownership) | ||
| active_ownership_count_group_by_owner[owner_id] = active_ownership_count_group_by_owner.get(owner_id, 0) + 1 # all active owners |
There was a problem hiding this comment.
This is really odd usage of collections.Counter. It does it's own bookkeeping of counts (that is its purpose in life). You may just as well use a set if you are going to use it yourself...
There was a problem hiding this comment.
I used set before Counter. I changed to Counter because I want to use Counter.most_common( ) @ line 122. Counter receives an iterable in the constructor but it doesn't have a method to gradually receive individual values.
johanste
reviewed
Aug 23, 2019
| while self._running: | ||
| claimed_ownership_list = await ownership_manager.claim_ownership() | ||
| if claimed_ownership_list: | ||
| claimed_partition_ids = [x["partition_id"] for x in claimed_ownership_list] |
There was a problem hiding this comment.
I've seen enough dict access (by key) into the ownership data to say that we really should make it into a class at this point.
There was a problem hiding this comment.
Will consider this after this release. Too much stuff has higher priority than this change at this time.
johanste
reviewed
Aug 23, 2019
| if not self.all_parition_ids: | ||
| await self._retrieve_partition_ids() | ||
| to_claim = await self._balance_ownership() | ||
| claimed_list = await self._claim_ownership(to_claim) |
There was a problem hiding this comment.
Having the same name (except for leading underscore), but with slightly different functionality is somewhat confusing. The claim_ownership method actually updates (or rebalances) owerships. _claim_ownership actually does the claim...
There was a problem hiding this comment.
I removed _claim_ownership since it's a very short function.
It can be implemented with one line of code.
johanste
reviewed
Aug 23, 2019
| except sqlite3.OperationalError: | ||
| raise | ||
| pass |
There was a problem hiding this comment.
Removed it. Sorry for pushing this code accidentally. I was thinking of raising a different type of exceptions but gave up.
YijunXieMS
removed request for
danieljurek,
iscai-msft,
mayurid,
mitchdenny,
rakshith91,
scbedd,
schaabs,
shurd,
weshaggard and
zikalino
added 10 commits
September 5, 2019 18:53
This reverts commit a9446de.
johanste
reviewed
Sep 7, 2019
| await partition_processor.initialize(partition_context) | ||
| events = await partition_consumer.receive() | ||
| await partition_processor.process_events(events, partition_context) | ||
| except asyncio.CancelledError: |
There was a problem hiding this comment.
I am very concerned about the fact that we are not re-raising the asyncio.CancelledError here...
There was a problem hiding this comment.
I changed "break" to "raise" for this "except asyncio.CancelledError:" block
|
Merged to feature branch "eventhubs_preview3" |
YijunXieMS
pushed a commit
that referenced
this pull request
Sep 9, 2019
* Small changes from code review * change EventData.msg_properties to private attribute * remove abstract method * code clean 1 * code clean 2 * Fix pylint * Fix pylint * Use properties EventData.partition_key * Small changes from code review * change EventData.msg_properties to private attribute * remove abstract method * code clean 1 * code clean 2 * Fix pylint * Fix pylint * Use properties EventData.partition_key * Use properties EventData.partition_key * Temporarily disable pylint errors that need refactoring * fix pylint errors * fix pylint errors * ignore eventprocessor pylint temporarily * small pylint adjustment * Add typing for Python2.7 * [EventHub] IoTHub management operations improvement and bug fixing (#6894) * Fix bug that iothub hub can't receive * Support direct mgmt ops of iothub * Improve mgmt ops and update livetest * Small fix * Improvement of iothub mgmt * [EventHub] Retry refactor (#7026) * Retry refactor * Refactor retry, delay and handle exception * Remove unused module * Small fix * Small fix * add system_properties to EventData * Fix a small bug * Refine example code * Update receive method (#7064) * Update accessibility of class (#7091) * Fix pylint * Update accessibility of of class * Small fix in livetest * Wait longer in iothub livetest * Small updates in livetest * Update samples and codes according to the review (#7098) * Update samples and codes according to the review * Small update * Python EventHubs load balancing (#6901) * Draft EventProcessor Loadbalancing * EventProcessor Load balancing * small changes from bryan's review * remove checkpoint manager from initialize * small changes * Draft EventProcessor Loadbalancing * EventProcessor Load balancing * small changes from bryan's review * remove checkpoint manager from initialize * small changes * Fix code review feedback * Packaging update of azure-mgmt-datalake-analytics * Packaging update of azure-loganalytics * Packaging update of azure-mgmt-storage * code review fixes and pylint error * reduce dictionary access * Revert "Packaging update of azure-mgmt-storage" This reverts commit cf22c7c. * Revert "Packaging update of azure-loganalytics" This reverts commit 40c7f03. * Revert "Packaging update of azure-mgmt-datalake-analytics" This reverts commit c126bea. * Trivial code change * Refine exception handling for eventprocessor * Enable pylint for eventprocessor * Expose OwnershipLostError * Move eventprocessor to aio rename Sqlite3PartitionManager to SamplePartitionManager * change checkpoint_manager to partition context * fix pylint error * fix a small issue * Catch list_ownership/claim_ownership exceptions and retry * Fix code review issues * fix event processor long running test * Remove utils.py * Remove close() method * Updated docstrings * add pytest * small fixes * Revert "Remove utils.py" This reverts commit a9446de. * change asyncio.create_task to 3.5 friendly code * Remove Callable * raise CancelledError instead of break * Fix a pylint error * Eventhubs blobstorage checkpointstore merge to preview3 (#7109) * exclude eventprocessor test for python27 * exclude eventprocessor test * Revert "Eventhubs blobstorage checkpointstore merge to preview3 (#7109)" This reverts commit 13a8fe7. * Fix small problem in consumer iterator (#7110) * Fixed an issue that initializes partition processor multiple times * Update release history for 5.0.0b3 * Update README for 5.0.0b3
rakshith91
pushed a commit
to rakshith91/azure-sdk-for-python
that referenced
this pull request
Sep 9, 2019
* Small changes from code review * change EventData.msg_properties to private attribute * remove abstract method * code clean 1 * code clean 2 * Fix pylint * Fix pylint * Use properties EventData.partition_key * Small changes from code review * change EventData.msg_properties to private attribute * remove abstract method * code clean 1 * code clean 2 * Fix pylint * Fix pylint * Use properties EventData.partition_key * Use properties EventData.partition_key * Temporarily disable pylint errors that need refactoring * fix pylint errors * fix pylint errors * ignore eventprocessor pylint temporarily * small pylint adjustment * Add typing for Python2.7 * [EventHub] IoTHub management operations improvement and bug fixing (Azure#6894) * Fix bug that iothub hub can't receive * Support direct mgmt ops of iothub * Improve mgmt ops and update livetest * Small fix * Improvement of iothub mgmt * [EventHub] Retry refactor (Azure#7026) * Retry refactor * Refactor retry, delay and handle exception * Remove unused module * Small fix * Small fix * add system_properties to EventData * Fix a small bug * Refine example code * Update receive method (Azure#7064) * Update accessibility of class (Azure#7091) * Fix pylint * Update accessibility of of class * Small fix in livetest * Wait longer in iothub livetest * Small updates in livetest * Update samples and codes according to the review (Azure#7098) * Update samples and codes according to the review * Small update * Python EventHubs load balancing (Azure#6901) * Draft EventProcessor Loadbalancing * EventProcessor Load balancing * small changes from bryan's review * remove checkpoint manager from initialize * small changes * Draft EventProcessor Loadbalancing * EventProcessor Load balancing * small changes from bryan's review * remove checkpoint manager from initialize * small changes * Fix code review feedback * Packaging update of azure-mgmt-datalake-analytics * Packaging update of azure-loganalytics * Packaging update of azure-mgmt-storage * code review fixes and pylint error * reduce dictionary access * Revert "Packaging update of azure-mgmt-storage" This reverts commit cf22c7c. * Revert "Packaging update of azure-loganalytics" This reverts commit 40c7f03. * Revert "Packaging update of azure-mgmt-datalake-analytics" This reverts commit c126bea. * Trivial code change * Refine exception handling for eventprocessor * Enable pylint for eventprocessor * Expose OwnershipLostError * Move eventprocessor to aio rename Sqlite3PartitionManager to SamplePartitionManager * change checkpoint_manager to partition context * fix pylint error * fix a small issue * Catch list_ownership/claim_ownership exceptions and retry * Fix code review issues * fix event processor long running test * Remove utils.py * Remove close() method * Updated docstrings * add pytest * small fixes * Revert "Remove utils.py" This reverts commit a9446de. * change asyncio.create_task to 3.5 friendly code * Remove Callable * raise CancelledError instead of break * Fix a pylint error * Eventhubs blobstorage checkpointstore merge to preview3 (Azure#7109) * exclude eventprocessor test for python27 * exclude eventprocessor test * Revert "Eventhubs blobstorage checkpointstore merge to preview3 (Azure#7109)" This reverts commit 13a8fe7. * Fix small problem in consumer iterator (Azure#7110) * Fixed an issue that initializes partition processor multiple times * Update release history for 5.0.0b3 * Update README for 5.0.0b3
xiangyan99
added a commit
that referenced
this pull request
Sep 16, 2019
* Certs Final PR (#7076) * keyvault certificate implementation * got rid of error mapping, README uses HttpResponseError instead * updated README to conform to Charles' new README standard * added samples for issuers and contacts * switched from PFX to PKCS12 for SecretContentType enum * fixed key vault overview * got rid of adding RUN_IDENTIFIER to vault name * got rid of erroneous references to keys * Delete seed.txt * incorporated Charles' and Johan's comments * Perform encrypt/verify/wrap locally when possible (#6939) * [EventHubs] Track2 Preview3 (#7059) * Small changes from code review * change EventData.msg_properties to private attribute * remove abstract method * code clean 1 * code clean 2 * Fix pylint * Fix pylint * Use properties EventData.partition_key * Small changes from code review * change EventData.msg_properties to private attribute * remove abstract method * code clean 1 * code clean 2 * Fix pylint * Fix pylint * Use properties EventData.partition_key * Use properties EventData.partition_key * Temporarily disable pylint errors that need refactoring * fix pylint errors * fix pylint errors * ignore eventprocessor pylint temporarily * small pylint adjustment * Add typing for Python2.7 * [EventHub] IoTHub management operations improvement and bug fixing (#6894) * Fix bug that iothub hub can't receive * Support direct mgmt ops of iothub * Improve mgmt ops and update livetest * Small fix * Improvement of iothub mgmt * [EventHub] Retry refactor (#7026) * Retry refactor * Refactor retry, delay and handle exception * Remove unused module * Small fix * Small fix * add system_properties to EventData * Fix a small bug * Refine example code * Update receive method (#7064) * Update accessibility of class (#7091) * Fix pylint * Update accessibility of of class * Small fix in livetest * Wait longer in iothub livetest * Small updates in livetest * Update samples and codes according to the review (#7098) * Update samples and codes according to the review * Small update * Python EventHubs load balancing (#6901) * Draft EventProcessor Loadbalancing * EventProcessor Load balancing * small changes from bryan's review * remove checkpoint manager from initialize * small changes * Draft EventProcessor Loadbalancing * EventProcessor Load balancing * small changes from bryan's review * remove checkpoint manager from initialize * small changes * Fix code review feedback * Packaging update of azure-mgmt-datalake-analytics * Packaging update of azure-loganalytics * Packaging update of azure-mgmt-storage * code review fixes and pylint error * reduce dictionary access * Revert "Packaging update of azure-mgmt-storage" This reverts commit cf22c7c. * Revert "Packaging update of azure-loganalytics" This reverts commit 40c7f03. * Revert "Packaging update of azure-mgmt-datalake-analytics" This reverts commit c126bea. * Trivial code change * Refine exception handling for eventprocessor * Enable pylint for eventprocessor * Expose OwnershipLostError * Move eventprocessor to aio rename Sqlite3PartitionManager to SamplePartitionManager * change checkpoint_manager to partition context * fix pylint error * fix a small issue * Catch list_ownership/claim_ownership exceptions and retry * Fix code review issues * fix event processor long running test * Remove utils.py * Remove close() method * Updated docstrings * add pytest * small fixes * Revert "Remove utils.py" This reverts commit a9446de. * change asyncio.create_task to 3.5 friendly code * Remove Callable * raise CancelledError instead of break * Fix a pylint error * Eventhubs blobstorage checkpointstore merge to preview3 (#7109) * exclude eventprocessor test for python27 * exclude eventprocessor test * Revert "Eventhubs blobstorage checkpointstore merge to preview3 (#7109)" This reverts commit 13a8fe7. * Fix small problem in consumer iterator (#7110) * Fixed an issue that initializes partition processor multiple times * Update release history for 5.0.0b3 * Update README for 5.0.0b3 * [AutoPR] reservations/resource-manager (#6861) * Generated from 631cc8ef9ba25eddb465ba207157ff92ab4fa641 (#6302) Task 4855805: ARM swagger update from lockedPriceTotal to billingCurrencyTotal & pricingCurrencyTotal. * Packaging update of azure-mgmt-reservations * Generated from 9c61d69a19c13e40a8468ea21c909aa677ac4678 (#6502) Task 4957070: ARM swagger update to include term in get reservation response. * [AutoPR reservations/resource-manager] [Hub Generated] Review request for Microsoft.Capacity to add version preview/2019-04-01 (#6860) * Generated from ea7847bd3538ded2d30c5c436a3f9f0464637d90 update * Generated from ecb712b6e6535b86e9283f57a1a65d63c3cbb04a update * Generated from a65461546a2e2b7d3391cda3ecbe437938540d32 update * updated test * fix test * updated history and version * setting a maxparallel paramter for the storage livetests. we need to run them in serial for now (#7134) * Enable SSO on Windows (#7006) * adjusting where inconsistency is set. if we are inconsistent, but the… (#7152) * adjusting where inconsistency is set. if we are inconsistent, but there is no lock file, we know that we have to exit with code 1 * updating logic to account for edge case where frozen file exists, but doesn't have any contents. we will falsely miss setting exitcode(1) * Fix for bailing out from writing a lockfile with inconsistencies * Fix writing a lockfile on Python2 * Improve verbose output formatting * Display dependency consistency msg in all cases * revert changes to doc build host machine. we need to push some docs out and I can debug it just the same on a forked branch (#7163) * Fix tracing if method raise exception (#7133) * fix link to azure-identity readme (#7169) * Eventhubs preview3 documentation update (#7139) * msal_extensions -> msal-extensions (#7172) * update doc (#7131) * [Cosmos] Preview2 (#7140) * [Cosmos] Core pipeline integration (#6961) * Updated dependencies * Added core pipeline * Ignore test config * Fixed indexes test * Refactored request creation * Fixed index test * Added trace decorators * Bumped version * Updated policies * Renamed request_options -> request_params * [Cosmos] Applying track 2 SDK guidelines (#7021) * Updated dependencies * Added core pipeline * Ignore test config * Fixed indexes test * Refactored request creation * Fixed index test * Added trace decorators * Bumped version * Updated policies * Renamed request_options -> request_params * Renamed clients * Updated with azure-core errors * Fixed test warnings * Updated config * PR fixes * Fixed init user import * Fixed init clients * Started revising constructors * Test conn str constructor * Update iterables with core paging * Added context managers * Reverted storage changes * Updated constructor * Mypy and Pylint * Renamed all listing operations * Some mypy fixes * Cleaned up method signatures * Fix pylint * Propagate kwargs * Fix pylint * Some mypy fixes * Updated readme and release notes * Fix for passing in extra headers * Reverted credentials * Review feedback * Fix pylint * Fixed samples * Updated docstrings * Fixed whitespace and imports * Some mypy fixes * Mypy fixes * Removed continuation token support * Pylint fix * Docs tweaks * Updated continuation token * Updated response header * [Cosmos] Bumped dependency (#7147) * Bumped dependency * Update reqs * Misc fixes for Cosmos SDK (#7157) * Made offer extend object instead of dict * added support for urllib3 connection retry * [Cosmos] docs updates (#7167) * Bumped dependency * Update reqs * Fixed dependency, replaced changelog * Updated readme and examples * Updated docs URLs * [Cosmos] Fix example formatting (#7171) * Bumped dependency * Update reqs * Fixed dependency, replaced changelog * Updated readme and examples * Updated docs URLs * Fixed f-strings * Another f-string * [Cosmos] Snippet references (#7174) * Bumped dependency * Update reqs * Fixed dependency, replaced changelog * Updated readme and examples * Updated docs URLs * Fixed f-strings * Another f-string * Snippet references * Key Vault: added example and fixed link in Certificates README (#7151) * added example and fixed link * added missing punctuation * Key Vault: fixed certificate docstring errors (#7132) * fixed certificate docstring errors * implemented Charles' comments * specified issuer must be unknown for merge_certificate * implmented Charles' comments pt 2 * Key Vault: added certificate changelog (#7128) * added certificate changelog * removed reference to azure core versioning * updated versioning and release date * implemented Charles' comments * remove duplicate enum key (#7173) * azure-keyvault-nspkg (#7125) * Storage Preview 3 Release (#7154) * Storage swagger (#6777) * Python swagger update This only changes to the official swagger version and regenerates * Upgrade blobs swagger to 2019-02-02 * Fixed namespace dir and header * Enable XML for blobs * Generate vanilla blobs * Fixed global parameters in swagger spec * Regenerated blobs * Revert "Fixed global parameters in swagger spec" This reverts commit 75bd21b. * Moved parameter fix to readme * Fixed blob list serialization * Fix for new metadata model * Fixed datetime formatting * Fixed required access policy * Fixed readme analysis * Regenerated queues * Regenerated files * Fixed files datetime * Skip tests pending msrest fix * Fixed for queue XML * Fix for docsettings * [storage] Fix for Files upload return type (#6772) * Fix missing import * Fix for Files inconsistent return types * Blob async trace decorators * [SnapshotSAS]Add Snapshot SAS (#6819) * [SnapshotSAS]Add Snapshot SAS * [SnapshotSAS]Extract SharedAccessSignature for Blob File and Queue * [SnapshotSAS]Fix Typo * [SnapshotSAS]Delete Redundant code The removed code is for another feature. * [SnapshotSAS]Stylistic Tweak * Updating Storage Files swagger to 2019-02-02 (#6872) * Updating Storage Files swagger to 2019-02-02 This includes a few transform changes to set default values for new required parameters and marked certain 8601 dates as strings when they use more precision that Python's dates. I'm seeing a few test failures that I'm hoping someone on the Python side can investigate (mostly "The specified share is being deleted. Try operation later." that I was seeing before my changes). There are a few other failures related to new service features that we can fix if easy or disable and file a work item to get these changes in. The recorded tests will also have to be updated, but we don't need to block on that right now. * skip file permission related tests * [Storage] File swagger updates (#6926) * Updating Storage Files swagger to 2019-02-02 This includes a few transform changes to set default values for new required parameters and marked certain 8601 dates as strings when they use more precision that Python's dates. I'm seeing a few test failures that I'm hoping someone on the Python side can investigate (mostly "The specified share is being deleted. Try operation later." that I was seeing before my changes). There are a few other failures related to new service features that we can fix if easy or disable and file a work item to get these changes in. The recorded tests will also have to be updated, but we don't need to block on that right now. * swagger changes * [storage] make storage pylint-clean (#6929) * make storage pylint-clean * feedback * Remove Storage swagger files and reference azure-rest-api-specs (#6943) Also includes some changes to the generated code updated to the latest version of blobs * Append block from url sync and async (#6925) * [Sync][AppendBlock] * [Async][AppendBlock]Add append_block_from_url * [Blob][AppendBlock]add recordings for append_block_from_url * [Blob][AppendBlock]fix failing CI * [Blob][AppendBlock]fix Pylint * [Blob][AppendBlock]Adjust Stylistic Things * [UpdatePage]add upload_pages_from_url sync and async (#6960) * [UpdatePage]add upload_pages_from_url sync and async * [UploadPage]Adjust Stylistic Things * Fix Pylint * Fix Pylint * [Queues] Migrate to devtools test framework (#6882) * initial migration * samples * comments * queues test fixes * recodings update * update * comment address * pylint fix * comments address * Put range from url (#7081) * [File]Upload Range From Url Sync * [File]Upload Range From Url ASync * Fix Pylint * Fix Pylint * CPK--Client Provided Encryption Key (#7104) * [Blob]CPK without test * Add Generated Code * Fix Pylint * [Blob][CPK]Test * Add Missing Recordings * [Blob][CPK]use existing field self.scheme * File rest parity (#7001) * [File][RestParity]Rest Parity Sync Add Rest Parity Sync part(except create permission) tweak _shared package a bit * [File][RestParity]Rest Parity Async Add Rest Parity Async part(except create permission) * [File][RestParity]Add Rest Parity Async Recording Files Add recording files for both sync and async * [File][RestParity]Fix CI * [File][RestParity]Recording again to fix CI * Add Generated Code * Stylistic Things and Record * [Swagger][BugFix]workaround to fix swagger generated code * [File][RestParity]Add Create_Permission API and Test * Fix Test * Fix Pylint * Revert the workaround * [File][RestParity]Tweak Documentation and Tests * delete .dat file * [Blob][SAS] Added support for identity SAS (#7020) * [Storage] Incremented version for preview 3 and updated change log (#7156) * Update Azure Core and msrest * Add Workaround for SharePermissions,revert msrest * Update version and history.md * updates on history.md * minor change * more changes * [Storage] Fix flaky tests (#7179) * Update HISTORY.md * Fix some typo in error message * Fix Some Tests * some more fixes * Skip OAUTH Test * update readme and history (#7188) * Skip Building on incompatible Python Versions (#7160) * removing --universal argument in build_packages.py * extending common_tasks.py. now we filter package set ALSO by CI compatibility * adding additional dependencies during build to allow targeting to successfully work * Disable dev-setup "Develop" mode in CI checks (#7162) * update sdist to not install in develop * allowing CI to run dev_setup without develop mode * updating appconfiguration manifest so that the sdist can properly install * update applicationinsights manifest so that the sdist can properly install * bugfix in dev_setup.py, bad syntax. adding 'develop' environment to the tox.ini * datalake analytics packages have bad manifests * repair manifest for sdist azure-eventhub * adding manifest for identity * updating manifest for scheduler * repair manifest for servermanager * Eventhubs preview3 doc update * Final Key Vault 4.0.0b3 updates (#7189) * add cryptography dependency (#7207) * Eventhubs blobstorage merge to master (#7150) * Increment Key Vault versions (#7211) * Increment azure-identity version (#7176) * Doc build resolution for extension package (#7217) * support for special casing in the doc build * update the docs.yml to take advantage of the new feature * removing stray == (#7218) * handle 0 packages being passed to dev_setup from setup_execute_tests.py (#7220) * adding line item entry (#7221) * KeyVault: Removing JsonWeb prefix from enums (#7212) * removed JsonWeb prefix from enums in certificates * removed JsonWeb prefix from enums in keys * updated changelog with breaking enum change * Eventhubs blobstorage docs update (#7216) * fixed certificates 4.0.0b3 changelog (#7214) * [cognitive-services] cog services devtools test framework (#7087) * cog sci test framework * wait for account creation * add fixme for luis authoring key * have mgmt testcase inherit from azure testcase * move text analytics to new test framework * add back code in constructor for mgmt testcase * Key Vault: reworked certificates samples, all passing (#7225) * reworked certificates samples, all passing * automatically run sync samples instead of within a try catch block * adding azure-eventhubs to the exception list. should result in what we expect unless another package installs it in dev mode (#7240) * add ability to specify job timeout (#7237) * add ability to specify manual timeout * use accepted spelling of "parameters" * Cosmos Pipeline Refactor (#7227) * add emulator to ci.yml * mark globaldb tests, try removing emulator from live tests * use the accepted gating cosmos emulator url * live tests * remove redundant BuildTargetingString * make coverage arg into a variable because of build-test.yml's construction * remove extra comment, extra templates don't add value here given that this file is cosmos-specific already * remove old cosmos pipelines * remove extra comment * Fix spelling mistakes in cosmos_db.py (#6980) * Fix doc link (#6458) It didn't have https causing a 404 * class links with extra text need :class: (#7215) * class links with extra text need :class: * use explicit syntax in azure-keyvault-certificates * Key Vault: added certificate rst files to sphinx doc (#7248) * added certificate rst files to sphinx doc * added reference to generate certificate docs * Address nspkg Failures (#7243) * increase timeout for pypy, adjust error output in create_install_wheel.py * updating build-artifacts to generate python2 nspkgs if any exist * Update Dependency Script to Key of Package Name for SDIST, update overrides to match (#7253) * adjusted analyze_deps to pull sdist package name from the setup.py, not folder * updated the overrides to match real package name, rather than sdist package name (from folder) * Un-use deprecated method in devtools (#7222) * Un-use deprecated method * make it python 2 compatible * [AutoPR compute/resource-manager] added missing packages to compute release (#7165) * Fix error creating EnvironmentCredential with username/password (#7260) * update sphinx sources for certificates (#7290) * update sphinx sources for certificates * updating conf to pull in additional sample pattern * Resolve faulty package filter when building for release (#7297) * originally removed this method due to a 'brilliant simplification' only to realize that kinda missed an essential filter during a build for a release * Read env variables and not settings if available (#7101) * Read env variables and not settings if available * Remove ADLA_JOB_ID from main MgmtTestCase * Missing self * Fix incorrect settings usage * Remove AD_DOMAIN from mgmt testcase * Add set_value_to_scrub * Fix create_mgmt_client * Remove CLIENT_OID from main mgmttestcase * No packaging update for KV certificates * Packaging update of azure-mgmt-datalake-analytics * Packaging update of azure-graphrbac * Fix missing return * update requirements versions and fix breaks * update to September 2019 release
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Python EventHubs load balancing 1st version.
It implements the design of https://gist.github.com/srnagar/7ef8566cfef0673288275c450dc99590
The requirement is still under discussion and changing.
Data storage is sqlite as an sample. We haven't decided which storage to use for preview 3.