Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Blob creation key lookup doesn't respect non-commercial endpoints #3991

Open
mruge-shr opened this issue Feb 26, 2025 · 1 comment
Open

Blob creation key lookup doesn't respect non-commercial endpoints #3991

mruge-shr opened this issue Feb 26, 2025 · 1 comment
Labels
awaiting-feedback Blocked on input from the author kind/bug Some behavior is incorrect or out of spec

Comments

@mruge-shr
Copy link

What happened?

Creating a blob inside of a container on GCC-High Azure environment.

azure-native:storage:Blob (adminKubeconfigBlob):
    error: POST https://management.azure.com/subscriptions/__redacted_GCCH_SubscriptionID__/resourceGroups/resource_group508e9b80/providers/Microsoft.Storage/storageAccounts/shredinfsecrets/listKeys
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: SubscriptionNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "SubscriptionNotFound",
        "message": "The subscription '__redacted_GCCH_SubscriptionID__' could not be found."
      }
    }

It should be looking for the subscription in GCC High, not Commercial

Example

## This creates successfully
secrets_container = storage.BlobContainer(
    "kube-secrets-container",
    account_name=secrets_storage_account.name,
    container_name="secrets",
    resource_group_name=resource_group.name,
    public_access="None",
)

## This returns an the error in the description
admin_kubeconfig_blob = storage.Blob(
    "adminKubeconfigBlob",
    account_name=secrets_storage_account.name,
    container_name='secrets',
    blob_name="admin-kubeconfig.yaml",
    resource_group_name=resource_group.name,
    source=admin_kubeconfig.apply(lambda config: StringAsset(config)),  # Securely store kubeconfig
    content_type="text/yaml",
)

Output of pulumi about

CLI
Version 3.152.0
Go Version go1.23.6
Go Compiler gc

Plugins
KIND NAME VERSION
resource azure-native 2.88.0
resource azuread 6.2.0
resource kubernetes 4.21.1
language python 3.152.0
resource random 4.17.0

Host
OS debian
Version 11.11
Arch x86_64

This project is written in python: executable='/workspaces/shred-apps/deploy/aks_cluster/venv/bin/python' version='3.12.8'

Current Stack: organization/shred-k8s/dev

TYPE URN
pulumi:pulumi:Stack urn:pulumi:dev::shred-k8s::pulumi:pulumi:Stack::shred-k8s-dev
pulumi:providers:random urn:pulumi:dev::shred-k8s::pulumi:providers:random::default_4_17_0
random:index/randomPassword:RandomPassword urn:pulumi:dev::shred-k8s::random:index/randomPassword:RandomPassword::harbor-admin-password
pulumi:providers:azure-native urn:pulumi:dev::shred-k8s::pulumi:providers:azure-native::default_2_88_0
azure-native:resources:ResourceGroup urn:pulumi:dev::shred-k8s::azure-native:resources:ResourceGroup::resource_group
azure-native:network:VirtualNetwork urn:pulumi:dev::shred-k8s::azure-native:network:VirtualNetwork::virtual_network
azure-native:network:Subnet urn:pulumi:dev::shred-k8s::azure-native:network:Subnet::subnet-1
azure-native:containerservice:ManagedCluster urn:pulumi:dev::shred-k8s::azure-native:containerservice:ManagedCluster::managed_cluster
pulumi:providers:kubernetes urn:pulumi:dev::shred-k8s::pulumi:providers:kubernetes::k8s-provider
kubernetes:core/v1:Namespace urn:pulumi:dev::shred-k8s::kubernetes:core/v1:Namespace::harbor-ns
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:core/v1:Secret::harbor-pull-secret
kubernetes:helm.sh/v3:Chart urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart::harbor
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-registryctl
kubernetes:core/v1:ConfigMap urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::default/harbor-jobservice-env
kubernetes:core/v1:ConfigMap urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::default/harbor-core
kubernetes:core/v1:ConfigMap urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::default/harbor-portal
kubernetes:core/v1:ConfigMap urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::default/harbor-jobservice
kubernetes:core/v1:PersistentVolumeClaim urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:PersistentVolumeClaim::default/harbor-jobservice
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-core
kubernetes:core/v1:ConfigMap urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::default/harbor-registryctl
kubernetes:core/v1:PersistentVolumeClaim urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:PersistentVolumeClaim::default/harbor-registry
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-jobservice
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-portal
kubernetes:core/v1:ConfigMap urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::default/harbor-registry
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-registry
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-trivy
kubernetes:apps/v1:StatefulSet urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:StatefulSet::default/harbor-database
kubernetes:apps/v1:StatefulSet urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:StatefulSet::default/harbor-redis
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-database
kubernetes:apps/v1:StatefulSet urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:StatefulSet::default/harbor-trivy
kubernetes:core/v1:Service urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::default/harbor-redis
kubernetes:apps/v1:Deployment urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::default/harbor-portal
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-database
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-trivy
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-core
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-ingress
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-jobservice
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-registry
kubernetes:core/v1:Secret urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/harbor-registry-htpasswd
azure-native:storage:StorageAccount urn:pulumi:dev::shred-k8s::azure-native:storage:StorageAccount::shredinf-secrets
azure-native:storage:BlobContainer urn:pulumi:dev::shred-k8s::azure-native:storage:BlobContainer::kube-secrets-container
kubernetes:apps/v1:Deployment urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::default/harbor-registry
kubernetes:apps/v1:Deployment urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::default/harbor-core
kubernetes:apps/v1:Deployment urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::default/harbor-jobservice
kubernetes:networking.k8s.io/v1:Ingress urn:pulumi:dev::shred-k8s::kubernetes:helm.sh/v3:Chart$kubernetes:networking.k8s.io/v1:Ingress::default/harbor-ingress
pulumi:providers:azure-native urn:pulumi:dev::shred-k8s::pulumi:providers:azure-native::azure-provider
pulumi:providers:kubernetes urn:pulumi:dev::shred-k8s::pulumi:providers:kubernetes::default_4_21_1
azure-native:network:Subnet urn:pulumi:dev::shred-k8s::azure-native:network:Subnet::subnet-2
azure-native:network:Subnet urn:pulumi:dev::shred-k8s::azure-native:network:Subnet::subnet-3

Found no pending operations associated with dev

Backend
Name c8d8c4f20687
URL azblob://shred-apps-cluster
User vscode
Organizations
Token type personal

Dependencies:
NAME VERSION
kubernetes 32.0.1
pulumi_azure_native 2.88.0
pulumi_azuread 6.2.0
pulumi_kubernetes 4.21.1
pulumi_random 4.17.0
setuptools 75.8.0
wheel 0.45.1

Pulumi locates its logs in /tmp by default

Additional context

{
	"name": "Pulumi",
	"image": "mcr.microsoft.com/devcontainers/python:1-3.12-bullseye",
	"runArgs": ["--env-file",".devcontainer/.env"],
	"features": {
		"ghcr.io/devcontainers/features/git-lfs:1": {},
		"ghcr.io/devcontainers-extra/features/pulumi:1": {},
		"ghcr.io/devcontainers/features/azure-cli:1": {},
		"ghcr.io/audacioustux/devcontainers/k9s:1": {},
		"ghcr.io/devcontainers/features/docker-in-docker:2": {}
	},
	"postCreateCommand": "bash .devcontainer/setup.sh"
}
AZURE_STORAGE_DOMAIN=blob.core.usgovcloudapi.net
AZURE_STORAGE_ACCOUNT=shredinf
AZURE_ENVIRONMENT=usgovernment
AZURE_CLOUD_ENVIRONMENT=usgovernment

ARM_ENVIRONMENT=usgovernment
ARM_TENANT_ID=__REDACTED__
ARM_SUBSCRIPTION_ID=__REDACTED__
AZURE_RESOURCE_GROUP=SHRED-INFRASTRUCTURE
PULUMI_CONFIG_PASSPHRASE=""
sudo az aks install-cli && sudo curl -sSL -o /usr/local/bin/kubelogin https://github.com/Azure/kubelogin/releases/latest/download/kubelogin-linux-amd64 && sudo chmod +x /usr/local/bin/kubelogin
az config set core.login_experience_v2=off
az cloud set --name AzureUSGovernment
echo "az login"
echo "pulumi login azblob://shred-apps-cluster"

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@mruge-shr mruge-shr added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Feb 26, 2025
@thomas11
Copy link
Contributor

Hi @mruge-shr, as documented in the setup guide, you should set the ARM_ENVIRONMENT variable. The provider doesn't read AZURE_ENVIRONMENT or AZURE_CLOUD_ENVIRONMENT at this point.

Confusingly, it seems like AZURE_CLOUD is supposed to be the new standard for the Azure SDKs. I'll track checking a few more of these variables and accepting a few more different values like AzureUSGovernment to make this easier.

@thomas11 thomas11 added awaiting-feedback Blocked on input from the author and removed needs-triage Needs attention from the triage team labels Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting-feedback Blocked on input from the author kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thomas11 @mruge-shr