Add etcd encryption key

[CecileRobertMichon]

What this PR does / why we need it:

Re-implement #2694 because CLA was not signed.

In @croeck 's words:

Bugfix to deploy a common etcd encryption key on all master nodes.

• Before this commit, all masters (if there should be more than one) would end up with different encryption keys, resulting in all but one masters to not be ready at all time.
  • calico did not start, complain about invalid padding in the secret
  • /etc/cni/net.d was not present on all but one master
• This key can either be specified as an option to kubeConfig, or will be generated before deployment.

Test setup

• Kubernetes 1.10.1
• custom acs-engine build (this fix on top of Update Calico to 3.1 #2521, which was also rebased on the latest master)
  • Mac OS X 10.12.6
• 3 masters
• 2 agent nodes

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.1",
      "kubernetesConfig": {
        "enableDataEncryptionAtRest": true,
        "useManagedIdentity": true,
        "enableRbac": true,
        "networkPolicy": "calico",
        "privateCluster": {
            "enabled": true
        },
        "dnsServiceIP": "10.11.96.96",
        "serviceCidr": "10.11.64.0/18",
        "clusterSubnet": "10.11.128.0/17",
        "etcdDiskSizeGB": "256"
      }
    },
    "aadProfile": {
      "serverAppID": "****",
      "clientAppID": "****",
      "tenantID": "****"
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "dev2",
      "vmSize": "Standard_A4_v2",
      "vnetSubnetId": "/subscriptions/****/resourceGroups/****/providers/Microsoft.Network/virtualNetworks/dev2-k8s-network/subnets/dev2-k8s-master-subnet",
      "firstConsecutiveStaticIP": "10.11.24.10",
      "vnetCidr": "10.11.0.0/16"
    },
    "agentPoolProfiles": [
      {
        "name": "agent",
        "count": 2,
        "vmSize": "Standard_A4_v2",
        "vnetSubnetId": "/subscriptions/****/resourceGroups/****/providers/Microsoft.Network/virtualNetworks/dev2-k8s-network/subnets/dev2-k8s-worker-subnet",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "****",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa ****"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "****",
      "secret": "****"
    }
  }
}

Which issue this PR fixes

relates to #2521 and #2587
In this PR it is stated that Kubernetes 1.10 was not yet achieved with Calico 3. This was however possible for me.

relates to #2202
Most likely fixes this issue, but I did not test with 1.9 and calico 2.x

• documentation
• unit tests
• tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

[CecileRobertMichon]

@ritazh I added base64 encoding validation here as follow up to your comment in #2694, let me know if you know of better way of checking if a string is base64 encoded...

[ritazh]

👍

[codecov]

Codecov Report

Merging #2756 into master will decrease coverage by 0.01%.
The diff coverage is 37.5%.

@@            Coverage Diff             @@
##           master    #2756      +/-   ##
==========================================
- Coverage   46.93%   46.92%   -0.02%     
==========================================
  Files          86       86              
  Lines       12786    12802      +16     
==========================================
+ Hits         6001     6007       +6     
- Misses       6231     6240       +9     
- Partials      554      555       +1

Impacted Files	Coverage Δ
pkg/api/types.go	28.88% <ø> (ø)	⬆️
pkg/api/vlabs/types.go	18.57% <ø> (ø)	⬆️
pkg/api/vlabs/validate.go	42.93% <0%> (-0.3%)	⬇️
pkg/api/converterfromapi.go	9.85% <0%> (-0.02%)	⬇️
pkg/api/convertertoapi.go	56.56% <100%> (+0.05%)	⬆️
pkg/acsengine/engine.go	62.78% <33.33%> (-0.05%)	⬇️
pkg/acsengine/defaults.go	62.74% <57.14%> (-0.09%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 244055e...a4c1297. Read the comment docs.

[jackfrancis]

lgtm

[CecileRobertMichon]

Back compat test running at https://jenkins.azure-containers.io/view/acs-engine%20ad-hoc/job/k8s-upgrade/83

[ritazh]

LGTM