Sovereign cloud support for AzureGermanCloud and AzureUSGovCloud.

[wangtt03]

Changes:

• Extract all the sovereign cloud configurations to defaults.go
• Cleanup hard coded URL suffix in shell scripts.
• Remove FQDN suffix related code from azureconst.go
• Fix SwarmMode China deployment failure bug by extracting docker engine download repo and docker compose download URL.
• Update testcases.

---

This change is 

Fixes #1066

[msftclas]

@wangtt03,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

[acs-bot]

Can one of the admins verify this patch?

[wangtt03]

@colemickens Could you please review this PR? Thanks!

[raulfiru]

I'm back in the office only on Tuesday - at the moment i have crappy internet so the latest on Tuesday morning (monday evening for you) i'll regen and deploy

[raulfiru]

might be wrong but i guess you are making that pull request from a private repo so i can't get the branch... and i can't merge into a new brach here since i don't have access (obviously)... So i'm stuck - can get the changes locally.

git remote add --does not work
git fetch origin pull/ID/head:BRANCHNAME --does not work either

another question: do you use the location to determine the right FQDN or have another attribute?

this is what i set:

"location": "germanycentral",

[colemickens]

The repo is open, something like this should work:

git remote add ccg [email protected]:/ChinaCloudGroup/acs-engine.git
git remote update
git checkout sovereign_cloud
git reset --hard ccg/sovereign_cloud

[raulfiru]

worked!

didn't have to change vm image name and now works like a charm... i still ssh-ed on the master (11PM... will try remote tomorrow)

azureuser@k8s-master-12051246-0:~$ kubectl get services
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   10.0.0.1     <none>        443/TCP   7m
azureuser@k8s-master-12051246-0:~$ kubectl get nodes
NAME                        STATUS                     AGE
k8s-agentpool1-12051246-0   Ready                      7m
k8s-agentpool1-12051246-1   Ready                      7m
k8s-agentpool1-12051246-2   Ready                      7m
k8s-master-12051246-0       Ready,SchedulingDisabled   7m

great - tnx guys! you can merge from my point of view ;)

PS: didn't do any regression for non-Azure.de

[wangtt03]

@raulfiru I am so happy to hear that! Thanks! 👍

[benkoller]

I had to change the osImageVersion as 16.04.201703070 seems not to be available in germanycentral (?):

~ ❯❯❯ az vm image list --all --location germanycentral | jq  '[ .[] | select( .sku | contains ("16.04")) ]'
[
  {
    "offer": "UbuntuServer",
    "publisher": "Canonical",
    "sku": "16.04-LTS",
    "urn": "Canonical:UbuntuServer:16.04-LTS:16.04.201701130",
    "version": "16.04.201701130"
  },
  {
    "offer": "UbuntuServer",
    "publisher": "Canonical",
    "sku": "16.04.0-LTS",
    "urn": "Canonical:UbuntuServer:16.04.0-LTS:16.04.201604203",
    "version": "16.04.201604203"
  },
  {
    "offer": "UbuntuServer",
    "publisher": "Canonical",
    "sku": "16.04.0-LTS",
    "urn": "Canonical:UbuntuServer:16.04.0-LTS:16.04.201605161",
    "version": "16.04.201605161"
  },
  {
    "offer": "UbuntuServer",
    "publisher": "Canonical",
    "sku": "16.04.0-LTS",
    "urn": "Canonical:UbuntuServer:16.04.0-LTS:16.04.201609071",
    "version": "16.04.201609071"
  },
  {
    "offer": "UbuntuServer",
    "publisher": "Canonical",
    "sku": "16.04.0-LTS",
    "urn": "Canonical:UbuntuServer:16.04.0-LTS:16.04.201610200",
    "version": "16.04.201610200"
  }
]

After correcting the image version the master vm extension deployment fails as the custom script exits with 1.

Checking /var/log/azure/cluster-provision.log I get a lot of these:

+ /usr/local/bin/kubectl cluster-info
The connection to the server localhost:8080 was refused - did you specify the right host or port?
+ '[' 1 = 0 ']'
+ sleep 1
+ for i in '{1..600}'
+ '[' -e /usr/local/bin/kubectl ']'

Any hints where I should look to troubleshoot further?

[wangtt03]

It seems that the apiserver is not started correctly. Please check if hyperkube docker image is pulled, could you please attach the docker ps output?

[benkoller]

I suspect I mangled the location configuration. Which points are relevant for setting the target environment?

[benkoller]

So, now I manually adjusted the targetEnvironment and fqdnEndpointSuffix in azuredeploy.parameters.json to get this to work as I still lack the understanding how and where you derive containerService.Location. I really would like to jump a bit deeper in this, but as of right now I lack the time to do so - sorry about that.

How did you plan on setting the location during the template generation?

[raulfiru]

@benkoller it worked like a charm for me... did you get the pull request as @colemickens wrote above (i had to change some paths) or you used the master branch?

You seam to know what you are doing (much more then me to be honest), but what i did is to get the pull locally (overwrite master), then follow the install guide where i've overwritten:

%GOPATH%\src\github.com\Azure\acs-engine

with the git folder that has the pull

then followed all the other steps and worked...

[benkoller]

I am on the sovereign_cloud-Branch and followed the above instructions to the letter. In the end all I had to do was the manual change of both FQDN and TargetEnv after template generation, but it doesn't seem like its the method intended by @wangtt03

I unfortunately can't deploy the first workloads today but I'll deploy some stuff tomorrow. Until now all seems fine though.

[raulfiru]

that is strange indeed... i tried on 2 azure germany subscriptions.

on one worked every time (tried 3 times) - cluster-provision.log:

+ for i in '{1..600}'
+ '[' -e /usr/local/bin/kubectl ']'
+ /usr/local/bin/kubectl cluster-info
The connection to the server localhost:8080 was refused - did you specify the right host or port?
+ '[' 1 = 0 ']'
+ sleep 1
+ for i in '{1..600}'
+ '[' -e /usr/local/bin/kubectl ']'
+ /usr/local/bin/kubectl cluster-info
Kubernetes master is running at http://localhost:8080

on another..also tried 3 times... side by side with different dns names and then i reused the exact same file and deleted the one in the other subscription (i thought it could be the names/certs)

+ sleep 1
+ for i in '{1..600}'
+ '[' -e /usr/local/bin/kubectl ']'
+ /usr/local/bin/kubectl cluster-info
The connection to the server localhost:8080 was refused - did you specify the right host or port?
+ '[' 1 = 0 ']'

failed every time... same as @benkoller

[colemickens]

It's almost surely because of ServicePrincipal problems. I'm guessing the SP that you are using only has permissions on one of the subscriptions.

[raulfiru]

i'm using different SP for each.

and everything else gets provisioned except: k8s-master-12051246-0/cse0 - and that looks like a script that run on the vm:

"commandToExecute": "[concat('/usr/bin/nohup /bin/bash -c \"/bin/bash /opt/azure/containers/provision.sh ',variables('tenantID'),' ',variables('subscriptionId'),' ',variables('resourceGroup'),' ',variables('location'),' ',variables('subnetName'),' ',variables('nsgName'),' ',variables('virtualNetworkName'),' ',variables('routeTableName'),' ',variables('primaryAvailablitySetName'),' ',variables('servicePrincipalClientId'),' ',variables('servicePrincipalClientSecret'),' ',variables('clientPrivateKey'),' ',variables('targetEnvironment'),' ',variables('networkPolicy'),' ',variables('fqdnEndpointSuffix'), ' >> /var/log/azure/cluster-provision.log 2>&1 &\" &')]"

they run fine on agents but not on the master. It hangs there until the loop (600 sec) timesout

[colemickens]

@raulfiru Yes, it will fail if the SP is invalid. Please follow the troubleshooting steps here to rule out the SP credentials being invalid or having wrong permissions: https://github.com/Azure/acs-engine/blob/master/docs/kubernetes.md#misconfigured-service-principal

[raulfiru]

you were right

Apr 18 23:42:39 k8s-master-12051246-0 docker[6315]: E0418 23:42:39.796008    6659 kubelet_node_status.go:70] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: autorest#WithErrorUnlessStatusCode: POST https://login.microsoftonline.de/b1904b97-e54a-4cb1-8031-6e578f3ad607/oauth2/token?api-version=1.0 failed with 400 Bad Request: StatusCode=400

i'll check the SP tomorrow and retry.

tnx and good night for now

[raulfiru]

works!

It was my mistake as i was using in the other case the SP name and not the ID.

now, both subscriptions work

[colemickens]

@raulfiru and to confirm, you didn't have to edit anything for FQDN or ubuntu image? I'd like to understand why this isn't working for @benkoller, if that is indeed still the case.

[raulfiru]

regarding the azure VM image - while the default image in the azuredeply.json is 16.04.201703070, the one is azuredeply.parameters.json is 16.04.201701130 - so it worked and i didn't change anything.

[benkoller]

@colemickens I have to power down the current deployment and will be out of the office until Monday so I can't check again. Until then I'd like to rule out I missed something during template generation, there are no additional steps to perform apart from ./acs-engine examples/kubernetes.json, correct? Do I need to add a "location": "AzureGermanCloud" or similar to the kubernetes.json?

[wangtt03]

There should be a location field target to 'germanycentral' or 'germanynortheast'

[benkoller]

Can you share / add an updated example/Kubernetes.json?

[wangtt03]

{
"apiVersion": "vlabs",
"location":"germanycentral",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "germany-m01",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "<your_ssh_public_key>"
}
]
}
},
"servicePrincipalProfile": {
"servicePrincipalClientID": "<your_client_id>",
"servicePrincipalClientSecret": "<your_client secret>"
}
}
}

Hope it helps!😊

[benkoller]

Thx @wangtt03, deployment went without a hitch with the location field. I had used AzureGermanCloud instead of germanycentral, thx for your example.

[jackfrancis]

@wangtt03 could you rebase this on master, or give me admin perms on ChinaCloudGroup:sovereign_cloud and I'd be happy to do it myself. Thanks either way!

[wangtt03]

@jackfrancis ,I will rebase this PR, and I will give you the admin permission also.

[jackfrancis]

@wangtt03 I checked out the ChinaCloudGroup fork of acs-engine, but don't have permissions to push (I have some minor changes that fix make test-style warnings).

[jackfrancis]

@wangtt03 false alarm, I hadn't accepted the collaboration invite, all works now! thanks!

[anhowe]

one change remaining, otherwise looks good.

[anhowe]

all "apt" commands should have retries around them. Please see other areas of code where we do this.

[jackfrancis]

@anhowe See this function and the below invocations to address your retry apt commands suggestion.

retry for apt commands in place

[ghost]

🚀

[raulfiru]

checking it today/tomorrow awesome!