Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[formrecognizer] support Copy API #11372

Merged
merged 26 commits into from
May 20, 2020
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f73471a
add copy feature gen code
kristapratico May 5, 2020
3903319
add copy implementation (wip)
kristapratico May 5, 2020
d7160bb
update copy implementation
kristapratico May 5, 2020
24a1636
add copy model samples
kristapratico May 5, 2020
957cce1
add back get copy auth method and update samples
kristapratico May 11, 2020
3ff9b0b
add CopyAuthorization to init
kristapratico May 11, 2020
d127002
changes from feedback on design
kristapratico May 12, 2020
a8ac624
raise better error messages in polling for failed copying
kristapratico May 12, 2020
19c77c7
update readme/changelog
kristapratico May 12, 2020
40ac006
updating sample snippets
kristapratico May 12, 2020
967cff4
Merge branch 'master' into fr-copy-api
kristapratico May 12, 2020
464877f
renames to align with .net
kristapratico May 12, 2020
ff525cb
change copy target to dict
kristapratico May 13, 2020
d65ab3e
add copy sync/async tests
kristapratico May 14, 2020
cb25fc7
merge master
kristapratico May 14, 2020
23b1d3a
make mypy happy
kristapratico May 14, 2020
ed4d8e3
update docstrings
kristapratico May 14, 2020
0e689f8
merge master
kristapratico May 14, 2020
bce5031
Merge branch 'master' into fr-copy-api
kristapratico May 18, 2020
a2a5dac
review feedback
kristapratico May 18, 2020
4736418
rename authorize_copy_target -> get_copy_authorization
kristapratico May 18, 2020
ff26bc6
feedback + add test for copy authorization
kristapratico May 19, 2020
ca64271
fix to testcase
kristapratico May 20, 2020
c100972
change description in samples
kristapratico May 20, 2020
688c3eb
hardcode region
kristapratico May 20, 2020
58a2210
construct resource id in testcase
kristapratico May 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdk/formrecognizer/azure-ai-formrecognizer/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

**New features**

- Copy API is now supported to copy a model from one Form Recognizer resource to another
- Support to copy a custom model from one Form Recognizer resource to another
- Authentication using `azure-identity` credentials now supported
- see the [Azure Identity documentation](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/identity/azure-identity/README.md) for more information

Expand Down
2 changes: 1 addition & 1 deletion sdk/formrecognizer/azure-ai-formrecognizer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ See the full details regarding [authentication][cognitive_authentication] of cog
- Training custom models to recognize all fields and values found in your custom forms. A `CustomFormModel` is returned indicating the form types the model will recognize, and the fields it will extract for each form type. See the [service's documents][fr-train-without-labels] for a more detailed explanation.
- Training custom models to recognize specific fields and values you specify by labeling your custom forms. A `CustomFormModel` is returned indicating the fields the model will extract, as well as the estimated accuracy for each field. See the [service's documents][fr-train-with-labels] for a more detailed explanation.
- Managing models created in your account.
- Copying a model from one Form Recognizer resource to another.
- Copying a custom model from one Form Recognizer resource to another.

Please note that models can also be trained using a graphical user interface such as the [Form Recognizer Labeling Tool][fr-labeling-tool].

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -249,15 +249,15 @@ def get_custom_model(self, model_id, **kwargs):
return CustomFormModel._from_generated(response)

@distributed_trace
def authorize_copy_target(self, resource_id, resource_region, **kwargs):
def get_copy_authorization(self, resource_id, resource_region, **kwargs):
# type: (str, str, Any) -> Dict[str, Union[str, int]]
"""Generate authorization for copying a model into the target Form Recognizer resource.
"""Generate authorization for copying a custom model into the target Form Recognizer resource.
This should be called by the target resource (where the model will be copied to)
and the output can be passed as the `target` parameter into :func:`~begin_copy_model()`.

:param str resource_id: Azure Resource Id of the target Form Recognizer resource
where the model will be copied to.
:param str resource_region: Location of the target Azure resource. A valid Azure
:param str resource_region: Location of the target Form Recognizer resource. A valid Azure
region name supported by Cognitive Services.
:return: A dictionary with values for the copy authorization -
"modelId", "accessToken", "resourceId", "resourceRegion", and "expirationDateTimeTicks".
Expand All @@ -267,8 +267,8 @@ def authorize_copy_target(self, resource_id, resource_region, **kwargs):
.. admonition:: Example:

.. literalinclude:: ../samples/sample_copy_model.py
:start-after: [START authorize_copy_target]
:end-before: [END authorize_copy_target]
:start-after: [START get_copy_authorization]
:end-before: [END get_copy_authorization]
:language: python
:dedent: 8
:caption: Authorize the target resource to receive the copied model
Expand All @@ -295,17 +295,17 @@ def begin_copy_model(
"""Copy a custom model stored in this resource (the source) to the user specified
target Form Recognizer resource. This should be called with the source Form Recognizer resource
(with the model that is intended to be copied). The `target` parameter should be supplied from the
target resource's output from calling the :func:`~authorize_copy_target()` method.
target resource's output from calling the :func:`~get_copy_authorization()` method.

:param str model_id: Model identifier of the model to copy to target resource.
:param dict target:
The copy authorization generated from the target resource's call to
:func:`~authorize_copy_target()`.
:func:`~get_copy_authorization()`.
:keyword int polling_interval: Default waiting time between two polls for LRO operations if
no Retry-After header is present.
:return: An instance of an LROPoller. Call `result()` on the poller
object to return a :class:`~azure.ai.formrecognizer.CustomFormModelInfo`.
:rtype: ~azure.core.polling.LROPoller[CustomFormModelInfo]
:rtype: ~azure.core.polling.LROPoller[~azure.ai.formrecognizer.CustomFormModelInfo]
:raises ~azure.core.exceptions.HttpResponseError:

.. admonition:: Example:
Expand All @@ -322,7 +322,7 @@ def begin_copy_model(

def _copy_callback(raw_response, _, headers): # pylint: disable=unused-argument
copy_result = self._client._deserialize(CopyOperationResult, raw_response)
return CustomFormModelInfo._from_generated_copy(copy_result, target["modelId"])
return CustomFormModelInfo._from_generated(copy_result, target["modelId"])

return self._client.begin_copy_custom_model( # type: ignore
model_id=model_id,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,6 @@ def _from_generated(cls, field, value, read_result):
page_number=value.page if value else None,
)


@classmethod
def _from_generated_unlabeled(cls, field, idx, page, read_result):
return cls(
Expand Down Expand Up @@ -456,6 +455,7 @@ def __repr__(self):
self.text, self.bounding_box, repr(self.words), self.page_number
)[:1024]


class FormWord(FormContent):
"""Represents a word recognized from the input document.

Expand Down Expand Up @@ -848,23 +848,14 @@ def __init__(self, **kwargs):
self.last_modified = kwargs.get("last_modified", None)

@classmethod
def _from_generated(cls, model):
def _from_generated(cls, model, model_id=None):
return cls(
model_id=model.model_id,
model_id=model_id if model_id else model.model_id,
status=model.status,
created_on=model.created_date_time,
last_modified=model.last_updated_date_time
)

@classmethod
def _from_generated_copy(cls, copy, model_id):
return cls(
model_id=model_id,
status=copy.status,
created_on=copy.created_date_time,
last_modified=copy.last_updated_date_time
)

def __repr__(self):
return "CustomFormModelInfo(model_id={}, status={}, created_on={}, last_modified={})".format(
self.model_id, self.status, self.created_on, self.last_modified
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,5 @@ def get_status(self, pipeline_response): # pylint: disable=no-self-use
if copy_result:
errors = copy_result.get("errors")
if errors:
message = ""
for err in errors:
message += "({}) {}\n".format(err.get("code"), err.get("message"))
raise HttpResponseError(message)
raise_error(response, errors, message="")
return status
Original file line number Diff line number Diff line change
Expand Up @@ -257,19 +257,19 @@ async def get_custom_model(self, model_id: str, **kwargs: Any) -> CustomFormMode
return CustomFormModel._from_generated(response)

@distributed_trace_async
async def authorize_copy_target(
async def get_copy_authorization(
self,
resource_id: str,
resource_region: str,
**kwargs: Any
) -> Dict[str, Union[str, int]]:
"""Generate authorization for copying a model into the target Form Recognizer resource.
"""Generate authorization for copying a custom model into the target Form Recognizer resource.
This should be called by the target resource (where the model will be copied to)
and the output can be passed as the `target` parameter into :func:`~copy_model()`.

:param str resource_id: Azure Resource Id of the target Form Recognizer resource
where the model will be copied to.
:param str resource_region: Location of the target Azure resource. A valid Azure
:param str resource_region: Location of the target Form Recognizer resource. A valid Azure
region name supported by Cognitive Services.
:return: A dictionary with values for the copy authorization -
"modelId", "accessToken", "resourceId", "resourceRegion", and "expirationDateTimeTicks".
Expand All @@ -279,8 +279,8 @@ async def authorize_copy_target(
.. admonition:: Example:

.. literalinclude:: ../samples/async_samples/sample_copy_model_async.py
:start-after: [START authorize_copy_target_async]
:end-before: [END authorize_copy_target_async]
:start-after: [START get_copy_authorization_async]
:end-before: [END get_copy_authorization_async]
:language: python
:dedent: 8
:caption: Authorize the target resource to receive the copied model
Expand All @@ -306,12 +306,12 @@ async def copy_model(
"""Copy a custom model stored in this resource (the source) to the user specified
target Form Recognizer resource. This should be called with the source Form Recognizer resource
(with the model that is intended to be copied). The `target` parameter should be supplied from the
target resource's output from calling the :func:`~authorize_copy_target()` method.
target resource's output from calling the :func:`~get_copy_authorization()` method.

:param str model_id: Model identifier of the model to copy to target resource.
:param dict target:
The copy authorization generated from the target resource's call to
:func:`~authorize_copy_target()`.
:func:`~get_copy_authorization()`.
:keyword int polling_interval: Default waiting time between two polls for LRO operations if
no Retry-After header is present.
:return: CustomFormModelInfo
Expand All @@ -331,7 +331,7 @@ async def copy_model(

def _copy_callback(raw_response, _, headers): # pylint: disable=unused-argument
copy_result = self._client._deserialize(CopyOperationResult, raw_response)
return CustomFormModelInfo._from_generated_copy(copy_result, target["modelId"])
return CustomFormModelInfo._from_generated(copy_result, target["modelId"])

return await self._client.copy_custom_model( # type: ignore
model_id=model_id,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ All of these samples need the endpoint to your Form Recognizer resource ([instru
|[sample_train_model_without_labels.py][sample_train_model_without_labels] and [sample_train_model_without_labels_async.py][sample_train_model_without_labels_async]|Train a custom model with unlabeled data|
|[sample_train_model_with_labels.py][sample_train_model_with_labels] and [sample_train_model_with_labels_async.py][sample_train_model_with_labels_async]|Train a custom model with labeled data|
|[sample_manage_custom_models.py][sample_manage_custom_models] and [sample_manage_custom_models_async.py][sample_manage_custom_models_async]|Manage the custom models in your account|
|[sample_copy_model.py][sample_copy_model] and [sample_copy_model_async.py][sample_copy_model_async]|Copy a model from one Form Recognizer resource to another|
|[sample_copy_model.py][sample_copy_model] and [sample_copy_model_async.py][sample_copy_model_async]|Copy a custom model from one Form Recognizer resource to another|


## Prerequisites
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,15 @@ async def copy_model_async(self):
target_region = os.environ["AZURE_FORM_RECOGNIZER_TARGET_REGION"]
target_resource_id = os.environ["AZURE_FORM_RECOGNIZER_TARGET_RESOURCE_ID"]

# [START authorize_copy_target_async]
# [START get_copy_authorization_async]
target_client = FormTrainingClient(endpoint=target_endpoint, credential=AzureKeyCredential(target_key))

async with target_client:
target = await target_client.authorize_copy_target(
target = await target_client.get_copy_authorization(
resource_region=target_region,
resource_id=target_resource_id
)
# [END authorize_copy_target_async]
# [END get_copy_authorization_async]

# [START copy_model_async]
source_client = FormTrainingClient(endpoint=source_endpoint, credential=AzureKeyCredential(source_key))
Expand All @@ -66,7 +66,8 @@ async def copy_model_async(self):

async with target_client:
copied_over_model = await target_client.get_custom_model(copy.model_id)
print(copied_over_model)
print("Model ID: {}".format(copied_over_model.model_id))
print("Status: {}".format(copied_over_model.status))
# [END copy_model_async]


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,14 @@ def copy_model(self):
target_region = os.environ["AZURE_FORM_RECOGNIZER_TARGET_REGION"]
target_resource_id = os.environ["AZURE_FORM_RECOGNIZER_TARGET_RESOURCE_ID"]

# [START authorize_copy_target]
# [START get_copy_authorization]
target_client = FormTrainingClient(endpoint=target_endpoint, credential=AzureKeyCredential(target_key))

target = target_client.authorize_copy_target(
target = target_client.get_copy_authorization(
resource_region=target_region,
resource_id=target_resource_id
)
# [END authorize_copy_target]
# [END get_copy_authorization]

# [START begin_copy_model]
source_client = FormTrainingClient(endpoint=source_endpoint, credential=AzureKeyCredential(source_key))
Expand All @@ -63,7 +63,8 @@ def copy_model(self):
copy = poller.result()

copied_over_model = target_client.get_custom_model(copy.model_id)
print(copied_over_model)
print("Model ID: {}".format(copied_over_model.model_id))
print("Status: {}".format(copied_over_model.status))
# [END begin_copy_model]


Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
interactions:
- request:
body: null
headers:
Accept:
- application/json
Accept-Encoding:
- gzip, deflate
Connection:
- keep-alive
Content-Length:
- '0'
User-Agent:
- azsdk-python-ai-formrecognizer/1.0.0b3 Python/3.7.3 (Windows-10-10.0.18362-SP0)
Python/3.7.3 (Windows-10-10.0.18362-SP0)
method: POST
uri: https://westus.api.cognitive.microsoft.com/formrecognizer/v2.0-preview/custom/models/copyAuthorization
response:
body:
string: '{"modelId": "3a1af284-1187-402d-9869-39ad09181bf5", "accessToken":
"redacted", "expirationDateTimeTicks": 637255865516657423}'
headers:
apim-request-id:
- be03b750-1177-47fa-9473-028489a9ce67
content-type:
- application/json; charset=utf-8
date:
- Tue, 19 May 2020 15:49:11 GMT
location:
- https://westus.api.cognitive.microsoft.com/formrecognizer/v2.0-preview/custom/models/3a1af284-1187-402d-9869-39ad09181bf5
strict-transport-security:
- max-age=31536000; includeSubDomains; preload
transfer-encoding:
- chunked
x-content-type-options:
- nosniff
x-envoy-upstream-service-time:
- '231'
status:
code: 201
message: Created
version: 1
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
interactions:
- request:
body: null
headers:
Accept:
- application/json
User-Agent:
- azsdk-python-ai-formrecognizer/1.0.0b3 Python/3.7.3 (Windows-10-10.0.18362-SP0)
Python/3.7.3 (Windows-10-10.0.18362-SP0)
method: POST
uri: https://westus.api.cognitive.microsoft.com/formrecognizer/v2.0-preview/custom/models/copyAuthorization
response:
body:
string: '{"modelId": "adcd2786-9aa5-4098-9a55-7d68eaeaaf2f", "accessToken":
"redacted", "expirationDateTimeTicks": 637255866617034918}'
headers:
apim-request-id: 99a50a5e-a423-47ae-8e9e-9369a4ac8d90
content-type: application/json; charset=utf-8
date: Tue, 19 May 2020 15:51:01 GMT
location: https://westus.api.cognitive.microsoft.com/formrecognizer/v2.0-preview/custom/models/adcd2786-9aa5-4098-9a55-7d68eaeaaf2f
strict-transport-security: max-age=31536000; includeSubDomains; preload
transfer-encoding: chunked
x-content-type-options: nosniff
x-envoy-upstream-service-time: '226'
status:
code: 201
message: Created
url: https://westus.api.cognitive.microsoft.com//formrecognizer/v2.0-preview/custom/models/copyAuthorization
version: 1
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def test_copy_model_successful(self, client, container_sas_url, location, resour
poller = client.begin_train_model(container_sas_url)
model = poller.result()

target = client.authorize_copy_target(resource_region=location, resource_id=resource_id)
target = client.get_copy_authorization(resource_region=location, resource_id=resource_id)

poller = client.begin_copy_model(model.model_id, target=target)
copy = poller.result()
Expand All @@ -46,7 +46,7 @@ def test_copy_model_fail(self, client, container_sas_url, location, resource_id)
model = poller.result()

# give an incorrect region
target = client.authorize_copy_target(resource_region="eastus", resource_id=resource_id)
target = client.get_copy_authorization(resource_region="eastus", resource_id=resource_id)

with self.assertRaises(HttpResponseError):
poller = client.begin_copy_model(model.model_id, target=target)
Expand All @@ -59,13 +59,13 @@ def test_copy_model_transform(self, client, container_sas_url, location, resourc
poller = client.begin_train_model(container_sas_url)
model = poller.result()

target = client.authorize_copy_target(resource_region=location, resource_id=resource_id)
target = client.get_copy_authorization(resource_region=location, resource_id=resource_id)

raw_response = []

def callback(response, _, headers):
copy_result = client._client._deserialize(CopyOperationResult, response)
model_info = CustomFormModelInfo._from_generated_copy(copy_result, target["modelId"])
model_info = CustomFormModelInfo._from_generated(copy_result, target["modelId"])
raw_response.append(copy_result)
raw_response.append(model_info)

Expand All @@ -78,3 +78,15 @@ def callback(response, _, headers):
self.assertEqual(copy.status, actual.status)
self.assertEqual(copy.last_modified, actual.last_updated_date_time)
self.assertEqual(copy.model_id, target["modelId"])

@GlobalFormRecognizerAccountPreparer()
@GlobalTrainingAccountPreparer(copy=True)
def test_copy_authorization(self, client, container_sas_url, location, resource_id):

target = client.get_copy_authorization(resource_region="eastus", resource_id=resource_id)

self.assertIsNotNone(target["modelId"])
self.assertIsNotNone(target["accessToken"])
self.assertIsNotNone(target["expirationDateTimeTicks"])
self.assertEqual(target["resourceRegion"], "eastus")
self.assertEqual(target["resourceId"], resource_id)
Loading