feat: Support Non-Azure Stack Custom Clouds with custom endpoints/root certs/sources.list #3063

ericsuhong · 2020-04-12T00:29:16Z

Reason for Change:
Currently, when CustomCloudProfile is specified, aks-engine treats it as Azure Stack and executes many Azure Stack-specific logic (i.e. loading azure stack specific image, setting lower arm api versions, disabling instance metadata, etc).

This makes impossible for users to specify custom cloud endpoints without triggering Azure Stack specific logic.

This PR decouples Azure Stack logic from Custom Cloud logic by introducing an internally computed property "IsCustomCloudProfile".

The main idea is to use “IsAzureStackCloud” property for doing any Azure Stack specific works, and use “IsCustomCloudProfile” property for generating actual azure.json and azurestackcloud.json files in VMs in the end.

I also added following two additional fields to support specifying custom root certificates and sources.list:

"customCloudRootCertificates": comma-separated list of base64-encoded pem certificates, which will be installed to /usr/local/shared/ca-certificates folder
"customCloudSourcesList": base64-encoded custom sources.list file. It will overwrite /etc/apt/sources.list (while taking its backup)

Sample customCloudProfile section:

"customCloudProfile": {
      "environment": {
        "name": "MyCustomCloud",
        "resourceManagerEndpoint": "https://management.customcloud/",
        "activeDirectoryEndpoint": "https://login.customcloud/",
        ...
        }
      },
      "customCloudRootCertificates": "LS0tLS1CRUdJTiBDRV...",
      "customCloudSourcesList": "ZGViIGh0dHBzOi8vcmVwb2RlcG90LmF6dXJlLmVhZ2..."
    },

Produces azure.json:

{
    "cloud":"AzureStackCloud",
    "tenantId": "...",
    "subscriptionId": "...",
    ...
}

and produces azurestackcloud.json:

{
    "name": "MyCustomCloud",
    "resourceManagerEndpoint": "https://management.customcloud/",
    "activeDirectoryEndpoint": "https://login.customcloud/",
    ...
}

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Notes:

ericsuhong · 2020-04-12T00:31:21Z

@devigned @jadarsie @honcao I closed old PR and created a new one. Would really appreciate if you can review this PR again.

jackfrancis · 2020-04-13T18:22:52Z

pkg/api/types.go

@@ -2071,13 +2094,14 @@ func (p *Properties) GetKubernetesHyperkubeSpec() string {

 // IsAzureStackCloud return true if the cloud is AzureStack
 func (p *Properties) IsAzureStackCloud() bool {
-	return p.CustomCloudProfile != nil
+	// For backward compatibility, treat nil Environment and empty Environment name as AzureStackCloud as well
+	return p.CustomCloudProfile != nil && (p.CustomCloudProfile.Environment == nil || p.CustomCloudProfile.Environment.Name == "" || strings.EqualFold(p.CustomCloudProfile.Environment.Name, "AzureStackCloud"))


nit:

return p.IsCustomCloudProfile() instead of copying the p.CustomCloudProfile != nil implementation from the IsCustomCloudProfile method (i.e., let's just re-use the method)

@jadarsie do we want the p.CustomCloudProfile.Environment == nil back-compat foo for AzureStackCloud? It would be nice to have IsAzureStackCloud actually require that the Environment.Name string property be equal to "AzureStackCloud".

parts/k8s/cloud-init/artifacts/cse_customcloud.sh

pkg/api/defaults-custom-cloud-profile.go

pkg/api/types.go

pkg/api/vlabs/types.go

jackfrancis · 2020-04-13T18:36:48Z

pkg/api/vlabs/types.go

 // IsAzureStackCloud return true if the cloud is AzureStack
 func (p *Properties) IsAzureStackCloud() bool {
-	return p.CustomCloudProfile != nil
+	// For backward compatibility, treat nil Environment and empty Environment name as AzureStackCloud as well
+	return p.CustomCloudProfile != nil && (p.CustomCloudProfile.Environment == nil || p.CustomCloudProfile.Environment.Name == "" || strings.EqualFold(p.CustomCloudProfile.Environment.Name, "AzureStackCloud"))


same comments/questions in api/types.go as well

pkg/engine/engine_test.go

jackfrancis · 2020-04-13T20:48:27Z

/azp run pr-e2e

azure-pipelines · 2020-04-13T20:48:38Z

Azure Pipelines successfully started running 1 pipeline(s).

parts/k8s/cloud-init/artifacts/cse_customcloud.sh

jackfrancis · 2020-04-13T21:58:44Z

/azp run pr-e2e

azure-pipelines · 2020-04-13T21:58:53Z

Azure Pipelines successfully started running 1 pipeline(s).

jackfrancis · 2020-04-13T22:27:47Z

/azp run pr-e2e

azure-pipelines · 2020-04-13T22:27:57Z

Azure Pipelines successfully started running 1 pipeline(s).

jackfrancis · 2020-04-13T22:51:01Z

/azp run pr-e2e

azure-pipelines · 2020-04-13T22:51:10Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov · 2020-04-13T23:06:05Z

Codecov Report

Merging #3063 into master will increase coverage by 0.26%.
The diff coverage is 79.19%.

@@            Coverage Diff             @@
##           master    #3063      +/-   ##
==========================================
+ Coverage   70.58%   70.85%   +0.26%     
==========================================
  Files         145      145              
  Lines       25219    25274      +55     
==========================================
+ Hits        17802    17907     +105     
+ Misses       6312     6251      -61     
- Partials     1105     1116      +11

Impacted Files	Coverage Δ
cmd/addpool.go	`18.91% <0.00%> (ø)`
cmd/scale.go	`12.39% <0.00%> (ø)`
cmd/upgrade.go	`45.25% <0.00%> (ø)`
pkg/engine/templates_generated.go	`33.60% <ø> (+2.61%)`	⬆️
pkg/api/vlabs/types.go	`73.30% <27.27%> (-2.72%)`	⬇️
cmd/deploy.go	`62.61% <50.00%> (ø)`
pkg/engine/engine.go	`86.04% <50.00%> (-0.27%)`	⬇️
pkg/api/types.go	`93.98% <75.00%> (-0.16%)`	⬇️
pkg/api/defaults-custom-cloud-profile.go	`85.71% <87.83%> (-3.58%)`	⬇️
pkg/api/addons.go	`97.74% <100.00%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1612db...1d74e95. Read the comment docs.

parts/k8s/cloud-init/artifacts/cse_customcloud.sh

jackfrancis · 2020-04-14T16:35:58Z

/azp run pr-e2e

azure-pipelines · 2020-04-14T16:36:08Z

Azure Pipelines successfully started running 1 pipeline(s).

jackfrancis

lgtm

@jadarsie for approval as well

jadarsie

So far so good, let me run e2e against Azure Stack

pkg/api/vlabs/validate.go

jackfrancis · 2020-04-15T22:52:52Z

/azp run pr-e2e

azure-pipelines · 2020-04-15T22:53:01Z

Azure Pipelines successfully started running 1 pipeline(s).

jackfrancis · 2020-04-16T00:23:29Z

/azp run pr-e2e

azure-pipelines · 2020-04-16T00:23:39Z

Azure Pipelines successfully started running 1 pipeline(s).

jadarsie · 2020-04-16T16:17:10Z

/lgtm

…t certs/sources.list (Azure#3063)