ManagedIdentityCredential fails DefaultAzureCredential on other cloud provider's compute instances #20554

EpicWink · 2021-09-06T08:36:25Z

Package Name: azure-identity
Package Version: 1.6.1
Operating System: Linux, Ubuntu 20.04 Focal
Python Version: 3.8.10

Describe the bug
When using azure.identity.DefaultAzureCredential on another cloud provider's compute instance, when http://169.254.169.254 is used by azure.identity.ImdsCredential (for azure.identity.ManagedIdentityCredential), the cloud provider responds with something ImdsCredential can't handle, which it then throws a ClientAuthenticationError which is indistinguishable from a true authentication failure.

To Reproduce

Start a basic HTTP server

$ sudo python

>>> import werkzeug
>>> @werkzeug.wrappers.Request.application
... def app(request):
...     return werkzeug.wrappers.Response(
...         "<html><body><p>spam</p></body></html>",
...         mimetype="text/html")
>>> werkzeug.serving.run_simple("10.70.131.160", 80, app)

Forward address

$ sudo iptables -t nat -A OUTPUT -d 169.254.169.254 -j DNAT --to-destination 10.70.131.160

Use DefaultAzureCredential

>>> import azure.identity
>>> c = azure.identity.DefaultAzureCredential()
>>> c.get_token("https://storage.azure.com/.default")

Expected behavior
azure.identity.ManagedIdentityCredential is soft-failed in the chain, allowing subsequent credential types to be tried.

Actual behavior

[ WARNING] azure.identity._internal.decorators: EnvironmentCredential.get_token failed: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Traceback (most recent call last):
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_internal/decorators.py", line 27, in wrapper
    token = fn(*args, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/environment.py", line 115, in get_token
    raise CredentialUnavailableError(message=message)
azure.identity._exceptions.CredentialUnavailableError: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
[    INFO] azure.identity._credentials.chained: DefaultAzureCredential - EnvironmentCredential is unavailable
[    INFO] azure.core.pipeline.policies.http_logging_policy: Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED'
[    INFO] azure.core.pipeline.policies.http_logging_policy: Request method: 'GET'
[    INFO] azure.core.pipeline.policies.http_logging_policy: Request headers:
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'User-Agent': 'azsdk-python-identity/1.6.1 Python/3.8.10 (Linux-5.11.0-27-generic-x86_64-with-glibc2.29)'
[    INFO] azure.core.pipeline.policies.http_logging_policy: No body was attached to the request
[   DEBUG] urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80
[   DEBUG] urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com HTTP/1.1" 200 37
[    INFO] azure.core.pipeline.policies.http_logging_policy: Response status: 200
[    INFO] azure.core.pipeline.policies.http_logging_policy: Response headers:
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Content-Type': 'text/html; charset=utf-8'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Content-Length': '37'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Server': 'Werkzeug/2.0.1 Python/3.8.10'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Date': 'Mon, 06 Sep 2021 08:34:13 GMT'
[    INFO] azure.core.pipeline.policies.http_logging_policy: Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED'
[    INFO] azure.core.pipeline.policies.http_logging_policy: Request method: 'GET'
[    INFO] azure.core.pipeline.policies.http_logging_policy: Request headers:
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Metadata': 'REDACTED'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'User-Agent': 'azsdk-python-identity/1.6.1 Python/3.8.10 (Linux-5.11.0-27-generic-x86_64-with-glibc2.29)'
[    INFO] azure.core.pipeline.policies.http_logging_policy: No body was attached to the request
[   DEBUG] urllib3.connectionpool: Resetting dropped connection: 169.254.169.254
[   DEBUG] urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com HTTP/1.1" 200 37
[    INFO] azure.core.pipeline.policies.http_logging_policy: Response status: 200
[    INFO] azure.core.pipeline.policies.http_logging_policy: Response headers:
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Content-Type': 'text/html; charset=utf-8'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Content-Length': '37'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Server': 'Werkzeug/2.0.1 Python/3.8.10'
[    INFO] azure.core.pipeline.policies.http_logging_policy:     'Date': 'Mon, 06 Sep 2021 08:34:13 GMT'
[ WARNING] azure.identity._internal.get_token_mixin: ImdsCredential.get_token failed: Unexpected content type "text/html; charset=utf-8"
Traceback (most recent call last):
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/html; charset=utf-8"
[ WARNING] azure.identity._internal.decorators: ManagedIdentityCredential.get_token failed: Unexpected content type "text/html; charset=utf-8"
Traceback (most recent call last):
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_internal/decorators.py", line 27, in wrapper
    token = fn(*args, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/managed_identity.py", line 88, in get_token
    return self._credential.get_token(*scopes, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/html; charset=utf-8"
[ WARNING] azure.identity._credentials.chained: DefaultAzureCredential.get_token failed: ManagedIdentityCredential raised unexpected error "Unexpected content type "text/html; charset=utf-8""
Traceback (most recent call last):
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/chained.py", line 67, in get_token
    token = credential.get_token(*scopes, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_internal/decorators.py", line 27, in wrapper
    token = fn(*args, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/managed_identity.py", line 88, in get_token
    return self._credential.get_token(*scopes, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/html; charset=utf-8"
[ WARNING] azure.identity._credentials.chained: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
	ManagedIdentityCredential: Unexpected content type "text/html; charset=utf-8"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/default.py", line 150, in get_token
    return super(DefaultAzureCredential, self).get_token(*scopes, **kwargs)
  File "/media/linux-data/venvs/azauth/lib/python3.8/site-packages/azure/identity/_credentials/chained.py", line 90, in get_token
    raise ClientAuthenticationError(message=message)
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
	ManagedIdentityCredential: Unexpected content type "text/html; charset=utf-8"

Workaround

import azure.identity

class ManagedIdentityCredential(azure.identity.ManagedIdentityCredential):
    def get_token(self, *scopes, **kwargs):
        try:
            return super().get_token(*scopes, **kwargs)
        except azure.core.exceptions.ClientAuthenticationError as e:
            if str(e).lower().startswith("unexpected"):
                raise azure.identity.CredentialUnavailableError(
                    f"[{e.__class__.__name__}] {e}"
                ) from e
            raise

credential = azure.identity.DefaultAzureCredential(
    exclude_managed_identity_credential=True,
)
credential.credentials += (ManagedIdentityCredential(),)

The text was updated successfully, but these errors were encountered:

chlowell · 2021-09-08T15:58:11Z

Thanks for opening this issue. DefaultAzureCredential and ManagedIdentityCredential are designed for applications that deploy to Azure, so this isn't unexpected. I don't want to change ManagedIdentityCredential's error behavior in this scenario because I think doing so without breaking code deployed to Azure would require a fragile special case and unnecessarily complicate the implementation. I suggest working around this by either constructing a custom credential chain for your application (see the readme) or simply excluding managed identity from DefaultAzureCredential:

DefaultAzureCredential(exclude_managed_identity_credential=True)

ghost · 2021-09-08T15:58:49Z

Hi @EpicWink. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

EpicWink · 2021-09-08T23:04:56Z

@chlowell could I suggest having ImdsCredential raise a subclass of ClientAuthenticationError so my workaround doesn't use fragile inspection of messages? This would be a 100% backwards-compatible change

chlowell · 2021-09-08T23:39:07Z

In your example the exception is ClientAuthenticationError, are you seeing something else in deployment? As I wrote above, there are simpler workarounds that don't involve inspecting error messages. Do these not work in your application?

EpicWink · 2021-09-09T23:26:40Z

ClientAuthenticationError is the error I am seeing in the current version. Disabling the managed-identity would mean I can't use identity credentials when my app is deployed to Azure

chlowell · 2021-09-10T16:07:04Z

That's true. How do you want to authenticate to Azure services when deployed to the other cloud?

EpicWink · 2021-09-11T00:22:27Z

We get a client credential from a secrets store. This is made transparent to users by a TokenCredential implementation we add to DefaultAzureCredential's credentials.

chlowell · 2021-09-14T18:28:46Z

Modifying DefaultAzureCredential.credentials is fragile because its elements and their ordering may change between versions. I suggest moving away from DefaultAzureCredential. It's designed for simpler scenarios. You can create similar composite credentials with ChainedTokenCredential or, if it makes sense for your application, use configuration to choose a specific credential type.

EpicWink · 2021-09-15T06:27:49Z

Thanks for the advice, good idea. Is there any chance azure.identity._internal.managed_identity_client.ManagedIdentityClientBase._process_response could raise a specific subclass of azure.core.exceptions.ClientAuthenticationError (eg UnexpectedAuthenticationResponse) when it receives a response it can't handle? This would allow users of azure.identity.ManagedIdentityCredential to manually handle this scenario themselves, and would be a minor and backwards-compatible change. I'm happy to make the PR myself

ghost · 2021-09-22T10:02:36Z

Hi @EpicWink, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

[Hub Generated] Review request for Microsoft.MachineLearningServices to add version preview/2022-10-01-preview (Azure#20818) * Adds base for updating Microsoft.MachineLearningServices from version preview/2022-06-01-preview to version 2022-10-01-preview * Updates readme * Updates API version in new specs and examples * Registries public preview 2022-10-01-preview (Azure#20200) * Registries public preview 2022-10-01-preview * Fix validation errors * Fix LintDiff and Avocado errors * Fixing prettier errors * Fixing more LintDiff errors * Another LintDiff error * Another Lint error Co-authored-by: Fahd Kamal <[email protected]> * Added schedule api (Azure#20273) * added schedule api * add updateSchedules to examples parent folder * fix typo * change position of allOf * remove subnet from example compute create * after prettier run * Description change for recurrence freq * point to mfe for common objects * add hash operator * removed triggertype as well * change year * get rid of breaking changes since we didn't change the api Co-authored-by: Naman Agarwal <[email protected]> * Added Compute Instance OS Patching Properties (Azure#20284) * Added CI Os Patching related properties * Updated descriptions as per Dennis's suggestions Co-authored-by: Srivatsa Sinha <[email protected]> * Maulik/sync preview (Azure#20554) * Added changes to sync with main * Fixed Prettier Check Co-authored-by: maulikjoshi <[email protected]> * Remove update schedules REST call for compute instance (Azure#20699) * change post call to put * change put to patch * remove updateschedules rest call Co-authored-by: Naman Agarwal <[email protected]> * Update connection type and auth type (Azure#20493) * Update connection type and auth type * update connection category * update x-ms-secret tag * Add autologger_settings for CI (Azure#20302) * Add autologger_settings for CI Add settings to opt-in/out of default mlflow autologger on CIs. * update machinelearningservices.json * Add for command job * fix comma * Add autologger for command job * fix CI failures * Remove mfe.json from PR * Remove whitespace * Update custom-words.txt * Delete incorrect custom-words.txt * Fixing auto-rest issues in registries (Azure#20772) * Fixing auto-rest issues in registries * Removing location from the required list Co-authored-by: Fahd Kamal <[email protected]> * Maulik/mfe preview changes (Azure#20372) * Added mfe.json preview changes * Added resources to Registries and Workspaces folder * removed unwanted resources * removed unwanted resources * Ran prettier check * removed unwanted resources * removed unwanted resources * Added ManagedOnlineDeployment resource * Added files to OnlineDeplyment * Added Model Validation changes * Added mfe.json changes to handle duplicate operation id changes * Updated mfe to fix underscore error * Update title of mfe.json * Renamed name path parameter * Updated examples * Fixed Semantic validations and examples * Fixed Semantic validations and examples * Fixed Semantic validations and examples * Removed name changes from workspaces * Changes related to Model Validation * Changes related to Model validation * Added consumes amd produces in registries.json * Removed changes from registries * Updated parameter mlFlowAutologger to mlflowAutologger in mfe.json * Changed property name to mlflowAutologger Co-authored-by: maulikjoshi <[email protected]> * Updated custom words to resolve merge conflict (Azure#20833) * Updated custom words to resolve merge conflict * Updated custom words to resolve merge conflict Co-authored-by: maulikjoshi <[email protected]> * Resolved merge conflict (Azure#20845) Co-authored-by: maulikjoshi <[email protected]> * Fixed LintDiff checks (Azure#20847) * Fixed LintDiff checks * Fixed LintDiff checks * Fixed LintDiff checks * Fixed LintDiff checks * Updated examples Co-authored-by: maulikjoshi <[email protected]> * Resolved merge conflict of cutsom-words.txt (Azure#20956) * Resolved merge conflict * Resolved merge conflict Co-authored-by: maulikjoshi <[email protected]> * [Hub Generated] Review request for Microsoft.MachineLearningServices to add version stable/2022-10-01 (Azure#20800) (Azure#20982) * Adds base for updating Microsoft.MachineLearningServices from version stable/2022-05-01 to version 2022-10-01 * Updates readme * Updates API version in new specs and examples * Added readOnly property (Azure#20506) * Added readOnly property * Added changes to sync with main * Fixed Prettier check Co-authored-by: maulikjoshi <[email protected]> * Add schedule related schemas for Compute Instance (Azure#20697) * support for update schedules on Compute Instance * change put to patch * remove update schedules rest call Co-authored-by: Naman Agarwal <[email protected]> * Added mfe.json changes (Azure#20514) * Added mfe.json changes * Added checks related changes * Updated mfe.json to handle online endpoint and deployment changes * Updated mfe.json with AutoRebuild changes Co-authored-by: maulikjoshi <[email protected]> * Removed duplicate definitions and updated with mfe references (Azure#20840) * Removed duplicate definitions and updated with mfe references * Replaced Cron to CronTrigger, Recurrence to RecurrenceTrigger from mfe Co-authored-by: maulikjoshi <[email protected]> Co-authored-by: maulikjoshi <[email protected]> Co-authored-by: Naman Agarwal <[email protected]> Co-authored-by: Naman Agarwal <[email protected]> Co-authored-by: maulikjoshi <[email protected]> Co-authored-by: Naman Agarwal <[email protected]> Co-authored-by: Naman Agarwal <[email protected]> * Revert "[Hub Generated] Review request for Microsoft.MachineLearningS… (Azure#20983) * Revert "[Hub Generated] Review request for Microsoft.MachineLearningServices to add version stable/2022-10-01 (Azure#20800) (Azure#20982)" This reverts commit 649ef8334aca13558e534babdcf53cef6831f22a. * Updated readme.md to resolve merge conflict Co-authored-by: maulikjoshi <[email protected]> * Maulik/fix preview (Azure#20994) * Merged readme from main * Added preview tag changes Co-authored-by: maulikjoshi <[email protected]> * shuffle readme tags (Azure#21005) * conflict resolution (Azure#21007) Co-authored-by: fahdkmsft <[email protected]> Co-authored-by: Fahd Kamal <[email protected]> Co-authored-by: Naman Agarwal <[email protected]> Co-authored-by: Naman Agarwal <[email protected]> Co-authored-by: srivatsasinha <[email protected]> Co-authored-by: Srivatsa Sinha <[email protected]> Co-authored-by: maulikjoshi <[email protected]> Co-authored-by: ZhidaLiu <[email protected]> Co-authored-by: prakharg-msft <[email protected]> Co-authored-by: Teddy Todorov <[email protected]>

chlowell added Azure.Identity customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. labels Sep 8, 2021

chlowell assigned xiangyan99 Sep 8, 2021

ghost closed this as completed Sep 22, 2021

github-actions bot locked and limited conversation to collaborators Apr 11, 2023

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ManagedIdentityCredential fails DefaultAzureCredential on other cloud provider's compute instances #20554

ManagedIdentityCredential fails DefaultAzureCredential on other cloud provider's compute instances #20554

EpicWink commented Sep 6, 2021 •

edited

Loading

chlowell commented Sep 8, 2021

ghost commented Sep 8, 2021

EpicWink commented Sep 8, 2021

chlowell commented Sep 8, 2021

EpicWink commented Sep 9, 2021 •

edited

Loading

chlowell commented Sep 10, 2021

EpicWink commented Sep 11, 2021

chlowell commented Sep 14, 2021

EpicWink commented Sep 15, 2021

ghost commented Sep 22, 2021

ManagedIdentityCredential fails DefaultAzureCredential on other cloud provider's compute instances #20554

ManagedIdentityCredential fails DefaultAzureCredential on other cloud provider's compute instances #20554

Comments

EpicWink commented Sep 6, 2021 • edited Loading

chlowell commented Sep 8, 2021

ghost commented Sep 8, 2021

EpicWink commented Sep 8, 2021

chlowell commented Sep 8, 2021

EpicWink commented Sep 9, 2021 • edited Loading

chlowell commented Sep 10, 2021

EpicWink commented Sep 11, 2021

chlowell commented Sep 14, 2021

EpicWink commented Sep 15, 2021

ghost commented Sep 22, 2021

EpicWink commented Sep 6, 2021 •

edited

Loading

EpicWink commented Sep 9, 2021 •

edited

Loading