Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cosmos] Add max_integrated_cache_staleness param option to item methods #22946

Merged
merged 47 commits into from
Mar 10, 2022
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
51664b6
consistency level gets set to default found in database account
simorenoh Jan 27, 2022
edb78a4
async client default change
simorenoh Jan 27, 2022
fd98dae
updated docs based on finding and updated samples to reflect best pra…
simorenoh Jan 27, 2022
df02ee4
Update CHANGELOG.md
simorenoh Jan 27, 2022
dcac0b2
Update README.md
simorenoh Jan 27, 2022
68d9a97
Update README.md
simorenoh Jan 27, 2022
8b006a6
Update README.md
simorenoh Jan 27, 2022
36cf3be
Update CHANGELOG.md
simorenoh Jan 27, 2022
6f7151e
formatting
simorenoh Jan 27, 2022
69df86d
formatting
simorenoh Jan 27, 2022
5b4f406
updated consistency for first request to Eventual (lowest latency)
simorenoh Jan 27, 2022
c19f655
pylint
simorenoh Jan 27, 2022
ce12951
from_connection_string methods
simorenoh Jan 27, 2022
84987ce
from_connection_string2
simorenoh Jan 27, 2022
2d56ebd
Update sdk/cosmos/azure-cosmos/README.md
simorenoh Jan 28, 2022
f018c89
Apply suggestions from code review
simorenoh Jan 28, 2022
2732fa1
Update README.md
simorenoh Jan 31, 2022
7116654
removed forceful header usage, changed setup to only check for Sessio…
simorenoh Feb 2, 2022
c4d688b
need to set header if Session consistency for updating session if ne…
simorenoh Feb 2, 2022
c126d33
Apply suggestions from code review
simorenoh Feb 2, 2022
00fcfd2
added test for session token
simorenoh Feb 2, 2022
9a80b00
Merge branch 'integrated-cache' of https://github.com/simorenoh/azure…
simorenoh Feb 2, 2022
7c098ea
Update CHANGELOG.md
simorenoh Feb 2, 2022
ae64f96
Update _cosmos_client_connection_async.py
simorenoh Feb 3, 2022
b121337
added max_integrated_cache_staleness to item methods in containers
simorenoh Feb 7, 2022
fb92392
Merge branch 'main' into integrated-cache
simorenoh Feb 7, 2022
01b494a
added validation and provisional comments
simorenoh Feb 7, 2022
dea6d19
pylint
simorenoh Feb 8, 2022
03ad4ab
only applied to read-only operations
simorenoh Feb 8, 2022
3378dce
Update container.py
simorenoh Feb 8, 2022
a79e9dd
Update CHANGELOG.md
simorenoh Feb 8, 2022
ed2af41
Apply suggestions from code review
simorenoh Feb 15, 2022
b48e68f
Update _base.py
simorenoh Feb 15, 2022
1d65490
Merge branch 'integrated-cache' of https://github.com/simorenoh/azure…
simorenoh Feb 15, 2022
6476a74
updated param comments to mention integrated cache configuration
simorenoh Feb 15, 2022
88788ac
Merge branch 'main' into integrated-cache
simorenoh Feb 17, 2022
2d55627
moved to kwargs
simorenoh Feb 22, 2022
7fc9572
added tests to verify functionality
simorenoh Feb 23, 2022
b0df13f
Update test_integrated_cache.py
simorenoh Feb 23, 2022
257196f
Update test_integrated_cache.py
simorenoh Feb 23, 2022
1dc7554
updates to test to ensure it works with setup
simorenoh Feb 24, 2022
ae2923d
added headers test and new way to track client headers before sending…
simorenoh Feb 25, 2022
cadf7bc
Update test_integrated_cache.py
simorenoh Mar 4, 2022
01774cf
Create test_axq.py
simorenoh Mar 8, 2022
6363ce7
Added mocking tests for max integrated cache staleness. Fixed issue w…
kushagraThapar Mar 9, 2022
aa518be
Merge branch 'main' into integrated-cache
kushagraThapar Mar 9, 2022
0f24bed
upgrade version for release
simorenoh Mar 9, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added tests to verify functionality
simorenoh committed Feb 23, 2022
commit 7fc957227d6e0dbb3d7be5adf6a312a9c0ca5c84
1 change: 1 addition & 0 deletions sdk/cosmos/azure-cosmos/azure/cosmos/http_constants.py
Original file line number Diff line number Diff line change
@@ -201,6 +201,7 @@ class HttpHeaders(object):

# Dedicated Gateway headers
DedicatedGatewayCacheStaleness = "x-ms-dedicatedgateway-max-age"
IntegratedCacheHit = "x-ms-cosmos-cachehit"


class HttpHeaderPreferenceTokens(object):
109 changes: 109 additions & 0 deletions sdk/cosmos/azure-cosmos/test/test_integrated_cache.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# The MIT License (MIT)
# Copyright (c) 2022 Microsoft Corporation

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import unittest

import azure.cosmos.cosmos_client as cosmos_client
from azure.cosmos.http_constants import HttpHeaders as headers
import pytest
import random
import uuid
import test_config

# This class tests the integrated cache, which only works against accounts with the dedicated gateway configured
# Since it doesn't work against the emulator

pytestmark = pytest.mark.cosmosEmulator

endpoint = ""
key = ""


@pytest.mark.usefixtures("teardown")
class TestIntegratedCache(unittest.TestCase):
configs = test_config._test_config
host = configs.host
masterKey = configs.masterKey
throughput = 400

@classmethod
def setUpClass(cls):
cls.client = cosmos_client.CosmosClient(cls.host, cls.masterKey)
cls.database = test_config._test_config.create_database_if_not_exist(cls.client)
cls.container = test_config._test_config.create_collection_if_not_exist_no_custom_throughput(cls.client)

def test_RU_cost(self):
if self.host.endswith(":8081/"):
print("Skipping; this test only works for accounts with the dedicated gateway configured, or the emulator" +
" running with the proper setup flags, which should run on port 8082.")
return

body = self.get_test_item()
self.container.create_item(body)
item_id = body['id']

# Initialize cache for item point read, and verify there is a cost to the read call
self.container.read_item(item=item_id, partition_key=item_id, max_integrated_cache_staleness_in_ms=(1000 * 30))
print(self.client.client_connection.last_response_headers)
self.assertEqual(self.client.client_connection.last_response_headers[headers.IntegratedCacheHit], 'False')
self.assertTrue(int(self.client.client_connection.last_response_headers[headers.RequestCharge]) > 0)

# Verify that cache is being hit for item read and that there's no RU consumption for this second read
self.container.read_item(item=item_id, partition_key=item_id)
print(self.client.client_connection.last_response_headers)
self.assertEqual(self.client.client_connection.last_response_headers[headers.IntegratedCacheHit], 'True')
self.assertTrue(int(self.client.client_connection.last_response_headers[headers.RequestCharge]) == 0)

body = self.get_test_item()
self.container.create_item(body)
item_id = body["id"]
query = 'SELECT * FROM c'

# Initialize cache for single partition query read, and verify there is a cost to the query call
self.container.query_items(query=query, partition_key=item_id, max_integrated_cache_staleness_in_ms=(1000 * 30))
print(self.client.client_connection.last_response_headers)
self.assertEqual(self.client.client_connection.last_response_headers[headers.IntegratedCacheHit], 'False')
self.assertTrue(int(self.client.client_connection.last_response_headers[headers.RequestCharge]) > 0)

# Verify that cache is being hit for item query and that there's no RU consumption for this second query
self.container.query_items(query=query, partition_key=item_id)
print(self.client.client_connection.last_response_headers)
self.assertEqual(self.client.client_connection.last_response_headers[headers.IntegratedCacheHit], 'True')
self.assertTrue(int(self.client.client_connection.last_response_headers[headers.RequestCharge]) == 0)

# Verify that reading all items does not have a cost anymore, since all items have been populated into cache
self.container.read_all_items()
self.assertEqual(self.client.client_connection.last_response_headers[headers.IntegratedCacheHit], 'True')
self.assertTrue(int(self.client.client_connection.last_response_headers[headers.RequestCharge]) == 0)


def get_test_item(self):
test_item = {
'id': 'Item_' + str(uuid.uuid4()),
'test_object': True,
'lastName': 'Smith',
'attr1': random.randint(0, 10)
}
return test_item


if __name__ == "__main__":
unittest.main()
11 changes: 3 additions & 8 deletions sdk/cosmos/azure-cosmos/test/test_partition_split_query.py
Original file line number Diff line number Diff line change
@@ -89,18 +89,13 @@ def run_queries(self, container, iterations):
print("validation succeeded for all query results")

def get_test_item(self):
async_item = {
'id': 'Async_' + str(uuid.uuid4()),
'address': {
'state': 'WA',
'city': 'Redmond',
'street': '1 Microsoft Way'
},
test_item = {
'id': 'Item_' + str(uuid.uuid4()),
'test_object': True,
'lastName': 'Smith',
'attr1': random.randint(0, 10)
}
return async_item
return test_item


if __name__ == "__main__":