AKS Engine is designed to be used as a CLI tool (aks-engine-azurestack). This document outlines the functionality that aks-engine-azurestack provides to create and maintain a Kubernetes cluster on Azure.
To get a quick overview of the commands available via the aks-engine-azurestack CLI tool, just run aks-engine-azurestack with no arguments (or include the --help argument):
$ aks-engine
Usage:
aks-engine-azurestack [flags]
aks-engine-azurestack [command]
Available Commands:
addpool Add a node pool to an existing AKS Engine-created Kubernetes cluster
completion Generates bash completion scripts
deploy Deploy an Azure Resource Manager template
generate Generate an Azure Resource Manager template
get-logs Collect logs and current cluster nodes configuration.
get-versions Display info about supported Kubernetes versions
help Help about any command
rotate-certs (experimental) Rotate certificates on an existing AKS Engine-created Kubernetes cluster
scale Scale an existing AKS Engine-created Kubernetes cluster
upgrade Upgrade an existing AKS Engine-created Kubernetes cluster
version Print the version of aks-engine
Flags:
--debug enable verbose debug logs
-h, --help help for aks-engine
--show-default-model Dump the default API model to stdout
Use "aks-engine-azurestack [command] --help" for more information about a command.These commands are provided by AKS Engine in order to create and maintain Kubernetes clusters. Note: there is no aks-engine-azurestack command to delete a cluster; to delete a Kubernetes cluster created by AKS Engine, you must delete the resource group that contains cluster resources. If the resource group can't be deleted because it contains other, non-Kubernetes-relate Azure resources, then you must manually delete the Virtual Machine and/or Virtual Machine Scale Set (VMSS), Disk, Network Interface, Network Security Group, Public IP Address, Virtual Network, Load Balancer, and all other resources specified in the aks-engine-generated ARM template. Because manually deleting resources is tedious and requires following serial dependencies in the correct order, it is recommended that you dedicate a resource group for the Azure resources that AKS Engine will create to run your Kubernetes cluster. If you're running more than one cluster, we recommend a dedicated resource group per cluster.
The aks-engine-azurestack deploy command will create a new cluster from scratch, using an API model (cluster definition) file as input to define the desired cluster configuration and shape, in the subscription, region, and resource group you provide, using credentials that you provide. Use this command to create a new cluster.
$ aks-engine-azurestack deploy --help
Deploy an Azure Resource Manager template, parameters file and other assets for a cluster
Usage:
aks-engine-azurestack deploy [flags]
Flags:
-m, --api-model string path to your cluster definition file
--auth-method client_secret auth method (default:client_secret, `client_certificate`)
--auto-suffix automatically append a compressed timestamp to the dnsPrefix to ensure unique cluster name automatically
--azure-env string the target Azure cloud (default "AzurePublicCloud")
--ca-certificate-path string path to the CA certificate to use for Kubernetes PKI assets
--ca-private-key-path string path to the CA private key to use for Kubernetes PKI assets
--certificate-path string path to client certificate (used with --auth-method=client_certificate)
--client-id string client id (used with --auth-method=[client_secret|client_certificate])
--client-secret string client secret (used with --auth-method=client_secret)
-p, --dns-prefix string dns prefix (unique name for the cluster)
-f, --force-overwrite automatically overwrite existing files in the output directory
-h, --help help for deploy
--identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad")
--language string language to return error messages in (default "en-us")
-l, --location string location to deploy to (required)
-o, --output-directory string output directory (derived from FQDN if absent)
--private-key-path string path to private key (used with --auth-method=client_certificate)
-g, --resource-group string resource group to deploy to (will use the DNS prefix from the apimodel if not specified)
--set stringArray set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2)
-s, --subscription-id string azure subscription id (required)
Global Flags:
--debug enable verbose debug logsDetailed documentation on aks-engine-azurestack deploy can be found here.
The aks-engine-azurestack scale command will scale (in or out) a specific node pool participating in a Kubernetes cluster created by AKS Engine. Use this command to manually scale a node pool to a specific number of nodes.
$ aks-engine-azurestack scale --help
Scale an existing AKS Engine-created Kubernetes cluster by specifying a new desired number of nodes in a node pool
Usage:
aks-engine-azurestack scale [flags]
Flags:
-m, --api-model string path to the generated apimodel.json file
--apiserver string apiserver endpoint (required to cordon and drain nodes)
--auth-method client_secret auth method (default:client_secret, `client_certificate`)
--azure-env string the target Azure cloud (default "AzurePublicCloud")
--certificate-path string path to client certificate (used with --auth-method=client_certificate)
--client-id string client id (used with --auth-method=[client_secret|client_certificate])
--client-secret string client secret (used with --auth-method=client_secret)
-h, --help help for scale
--identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad")
--language string language to return error messages in (default "en-us")
-l, --location string location the cluster is deployed in
-c, --new-node-count int desired number of nodes
--node-pool string node pool to scale
--private-key-path string path to private key (used with --auth-method=client_certificate)
-g, --resource-group string the resource group where the cluster is deployed
-s, --subscription-id string azure subscription id (required)
Global Flags:
--debug enable verbose debug logsThe scale command has limitations for scaling in (reducing the number of nodes in a node pool):
- It accepts a new, desired node count; it does not accept a list of specific nodes to remove from the pool.
- For VMSS-backed node pools, the removed nodes will not be cordoned and drained prior to being removed, which means any running workloads on nodes-to-be-removed will be disrupted without warning, and temporary operational impact is to be expected.
We generally recommend that you manage node pool scaling dynamically using the cluster-autoscaler project. More documentation about cluster-autoscaler is here, including how to automatically install and configure it at cluster creation time as an AKS Engine addon.
Detailed documentation on aks-engine-azurestack scale can be found here.
The aks-engine-azurestack addpool command will add a new node pool to an existing AKS Engine-created cluster. Using a JSON file to define a the new node pool's configuration, and referencing the aks-engine-generated apimodel.json, you can add new nodes to your cluster. Use this command to add a specific number of new nodes using a discrete configuration compared to existing nodes participating in your cluster.
$ aks-engine-azurestack addpool --help
Add a node pool to an existing AKS Engine-created Kubernetes cluster by referencing a new agentpoolProfile spec
Usage:
aks-engine-azurestack addpool [flags]
Flags:
-m, --api-model string path to the generated apimodel.json file
--auth-method client_secret auth method (default:client_secret, `client_certificate`)
--azure-env string the target Azure cloud (default "AzurePublicCloud")
--certificate-path string path to client certificate (used with --auth-method=client_certificate)
--client-id string client id (used with --auth-method=[client_secret|client_certificate])
--client-secret string client secret (used with --auth-method=client_secret)
-h, --help help for addpool
--identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad")
--language string language to return error messages in (default "en-us")
-l, --location string location the cluster is deployed in
-p, --node-pool string path to a JSON file that defines the new node pool spec
--private-key-path string path to private key (used with --auth-method=client_certificate)
-g, --resource-group string the resource group where the cluster is deployed
-s, --subscription-id string azure subscription id (required)
Global Flags:
--debug enable verbose debug logsDetailed documentation on aks-engine-azurestack addpool can be found here.
The aks-engine-azurestack upgrade command orchestrates a Kubernetes version upgrade across your existing cluster nodes. Use this command to upgrade the Kubernetes version running your control plane, and optionally on all your nodes as well.
$ aks-engine-azurestack upgrade --help
Upgrade an existing AKS Engine-created Kubernetes cluster, one node at a time
Usage:
aks-engine-azurestack upgrade [flags]
Flags:
-m, --api-model string path to the generated apimodel.json file
--auth-method client_secret auth method (default:client_secret, `client_certificate`)
--azure-env string the target Azure cloud (default "AzurePublicCloud")
--certificate-path string path to client certificate (used with --auth-method=client_certificate)
--client-id string client id (used with --auth-method=[client_secret|client_certificate])
--client-secret string client secret (used with --auth-method=client_secret)
--control-plane-only upgrade control plane VMs only, do not upgrade node pools
--cordon-drain-timeout int how long to wait for each vm to be cordoned in minutes (default -1)
-f, --force force upgrading the cluster to desired version. Allows same version upgrades and downgrades.
-h, --help help for upgrade
--identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad")
-b, --kubeconfig string the path of the kubeconfig file
--language string language to return error messages in (default "en-us")
-l, --location string location the cluster is deployed in (required)
--private-key-path string path to private key (used with --auth-method=client_certificate)
-g, --resource-group string the resource group where the cluster is deployed (required)
-s, --subscription-id string azure subscription id (required)
-k, --upgrade-version string desired kubernetes version (required)
--upgrade-windows-vhd upgrade image reference of the Windows nodes (default true)
--vm-timeout int how long to wait for each vm to be upgraded in minutes (default -1)
Global Flags:
--debug enable verbose debug logsDetailed documentation on aks-engine-azurestack upgrade can be found here.
AKS Engine also provides a command to generate a reusable ARM template only, without creating any actual Azure resources.
The aks-engine-azurestack generate command is similar to aks-engine-azurestack deploy: it uses an API model (cluster definition) file as input to define the desired cluster configuration and shape of a new Kubernetes cluster. Unlike deploy, aks-engine-azurestack generate does not actually submit any operational requests to Azure, but is instead used to generate a reusable ARM template which may be deployed at a later time. Use this command as a part of a workflow that creates one or more Kubernetes clusters via an ARM group deployment that takes an ARM template as input (e.g., az deployment group create using the standard az Azure CLI).
$ aks-engine-azurestack generate --help
Generates an Azure Resource Manager template, parameters file and other assets for a cluster
Usage:
aks-engine-azurestack generate [flags]
Flags:
-m, --api-model string path to your cluster definition file
--ca-certificate-path string path to the CA certificate to use for Kubernetes PKI assets
--ca-private-key-path string path to the CA private key to use for Kubernetes PKI assets
--client-id string client id
--client-secret string client secret
-h, --help help for generate
--no-pretty-print skip pretty printing the output
-o, --output-directory string output directory (derived from FQDN if absent)
--parameters-only only output parameters files
--set stringArray set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2)
Global Flags:
--debug enable verbose debug logsDetailed documentation on aks-engine-azurestack generate can be found here.
Detailed documentation on aks-engine-azurestack rotate-certs can be found here.
The aks-engine-azurestack get-logs can conveniently collect host VM logs from your Linux node VMs for local troubleshooting. This command does not support Windows nodes. The command assumes that your node VMs have an SSH daemon listening on port 22, that all nodes share a common SSH keypair for interactive login, and that a public endpoint exists on one of the control plane VMs for accommodating SSH agent key forwarding.
$ aks-engine-azurestack get-logs --help
Usage:
aks-engine-azurestack get-logs [flags]
Flags:
-m, --api-model string path to the generated apimodel.json file (required)
--control-plane-only get logs from control plane VMs only
-h, --help help for get-logs
--linux-script string path to the log collection script to execute on the cluster's Linux nodes (required)
--linux-ssh-private-key string path to a valid private SSH key to access the cluster's Linux nodes (required)
-l, --location string Azure location where the cluster is deployed (required)
-o, --output-directory string collected logs destination directory, derived from --api-model if missing
--ssh-host string FQDN, or IP address, of an SSH listener that can reach all nodes in the cluster (required)
Global Flags:
--debug enable verbose debug logsThe aks-engine-azurestack codebase contains a working log retrieval script in scripts/collect-logs.sh, so you can use it to quickly gather logs from your node VMs:
$ git clone https://github.com/Azure/aks-engine-azurestack.git && cd aks-engine
Cloning into 'aks-engine'...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 92107 (delta 13), reused 15 (delta 1), pack-reused 92063
Receiving objects: 100% (92107/92107), 92.86 MiB | 7.27 MiB/s, done.
Resolving deltas: 100% (64711/64711), done.
$ export LATEST_AKS_ENGINE_RELEASE=v0.56.0
$ git checkout $LATEST_AKS_ENGINE_RELEASE
Note: checking out 'v0.56.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 666073d49 chore: updating Windows VHD with new cached artifacts (#3843)
$ bin/aks-engine-azurestack get-logs --api-model _output/$CLUSTER_NAME/apimodel.json --location $CLUSTER_NAME --linux-ssh-private-key _output/$CLUSTER_NAME-ssh --linux-script ./scripts/collect-logs.sh --ssh-host $CLUSTER_NAME.$LOCATION.cloudapp.azure.com
...
INFO[0062] Logs downloaded to _output/<name of cluster>/_logsThe following example assumes that the $CLUSTER_NAME environment variable is assigned to the value of the cluster name (properties.masterProfile.dnsPrefix in the cluster API model), and that $LOCATION is assigned to the location string of the resource group that your cluster was created into.