From e43c5c2da89da7ba03305ec0131c39882d5fe77b Mon Sep 17 00:00:00 2001 From: Yusuke Kuoka Date: Wed, 14 Dec 2016 18:32:42 +0900 Subject: [PATCH] Documentation for v0.9.2 fix #110 #120 #135 #138 #119 --- Documentation/kube-aws-cluster-updates.md | 12 +- Documentation/kubernetes-on-aws-destroy.md | 8 + Documentation/kubernetes-on-aws-launch.md | 12 +- .../kubernetes-on-aws-limitations.md | 20 ++ Documentation/kubernetes-on-aws-node-pool.md | 156 ++++++++++ .../kubernetes-on-aws-prerequisites.md | 26 ++ Documentation/kubernetes-on-aws-render.md | 86 +++++- Documentation/kubernetes-on-aws.md | 24 ++ README.md | 266 +++++++----------- ROADMAP.md | 35 +++ 10 files changed, 454 insertions(+), 191 deletions(-) create mode 100644 Documentation/kubernetes-on-aws-destroy.md create mode 100644 Documentation/kubernetes-on-aws-limitations.md create mode 100644 Documentation/kubernetes-on-aws-node-pool.md create mode 100644 Documentation/kubernetes-on-aws-prerequisites.md create mode 100644 ROADMAP.md diff --git a/Documentation/kube-aws-cluster-updates.md b/Documentation/kube-aws-cluster-updates.md index 673210e34..5ce03faae 100644 --- a/Documentation/kube-aws-cluster-updates.md +++ b/Documentation/kube-aws-cluster-updates.md @@ -1,4 +1,4 @@ -# kube-aws cluster updates +# Updating the Kubernetes cluster ## Types of cluster update There are two distinct categories of cluster update. @@ -35,7 +35,11 @@ Fortunately, CoreOS update engine will take care of keeping the members of the e In the (near) future, etcd will be hosted on Kubernetes and this problem will no longer be relevant. Rather than concocting overly complex band-aid, we've decided to "punt" on this issue of the time being. +Once you have successfully updated your cluster, you are ready to [add node pools to your cluster][aws-step-5]. - - - +[aws-step-1]: kubernetes-on-aws.md +[aws-step-2]: kubernetes-on-aws-render.md +[aws-step-3]: kubernetes-on-aws-launch.md +[aws-step-4]: kube-aws-cluster-updates.md +[aws-step-5]: kubernetes-on-aws-node-pool.md +[aws-step-6]: kubernetes-on-aws-destroy.md diff --git a/Documentation/kubernetes-on-aws-destroy.md b/Documentation/kubernetes-on-aws-destroy.md new file mode 100644 index 000000000..f0261391f --- /dev/null +++ b/Documentation/kubernetes-on-aws-destroy.md @@ -0,0 +1,8 @@ +## Destroy the cluster + +When you are done with your cluster, run `kube-aws node-pools destroy` and `kube-aws destroy` then all cluster components will be destroyed. + +If you created any node pool, you must delete these first by running `kube-aws node-pools destroy`, or `kube-aws destroy` will end up failing because node pools still references +AWS resources managed by the main cluster. + +If you created any Kubernetes Services of type `LoadBalancer`, you must delete these first, as the CloudFormation cannot be fully destroyed if any externally-managed resources still exist. diff --git a/Documentation/kubernetes-on-aws-launch.md b/Documentation/kubernetes-on-aws-launch.md index 061f95854..057ff9894 100644 --- a/Documentation/kubernetes-on-aws-launch.md +++ b/Documentation/kubernetes-on-aws-launch.md @@ -7,7 +7,7 @@ This is the [third step of running Kubernetes on AWS][aws-step-1]. We're ready t Now for the exciting part, creating your cluster: ```sh -$ kube-aws up +$ kube-aws up --s3-uri s3:/// ``` **NOTE**: It can take some time after `kube-aws up` completes before the cluster is available. When the cluster is first being launched, it must download all container images for the cluster components (Kubernetes, dns, heapster, etc). Depending on the speed of your connection, it can take a few minutes before the Kubernetes api-server is available. @@ -18,7 +18,7 @@ If you configured Route 53 settings in your configuration above via `createRecor Otherwise, navigate to the DNS registrar hosting the zone for the provided external DNS name. Ensure a single A record exists, routing the value of `externalDNSName` defined in `cluster.yaml` to the externally-accessible IP of the master node instance. -You can invoke `kube-aws status` to get the cluster API IP address after cluster creation, if necessary. This command can take a while. +You can invoke `kube-aws status` to get the cluster API endpoint after cluster creation, if necessary. This command can take a while. ## Access the cluster @@ -59,11 +59,11 @@ If you want to share, audit or back up your stack, use the export flag: $ kube-aws up --export ``` -## Destroy the cluster - -When you are done with your cluster, simply run `kube-aws destroy` and all cluster components will be destroyed. -If you created any Kubernetes Services of type `LoadBalancer`, you must delete these first, as the CloudFormation cannot be fully destroyed if any externally-managed resources still exist. +Once you have successfully launched your cluster, you are ready to [update your cluster][aws-step-4]. [aws-step-1]: kubernetes-on-aws.md [aws-step-2]: kubernetes-on-aws-render.md [aws-step-3]: kubernetes-on-aws-launch.md +[aws-step-4]: kube-aws-cluster-updates.md +[aws-step-5]: kubernetes-on-aws-node-pool.md +[aws-step-6]: kubernetes-on-aws-destroy.md diff --git a/Documentation/kubernetes-on-aws-limitations.md b/Documentation/kubernetes-on-aws-limitations.md new file mode 100644 index 000000000..27db92d03 --- /dev/null +++ b/Documentation/kubernetes-on-aws-limitations.md @@ -0,0 +1,20 @@ +# Known Limitations + +## hostPort doesn't work + +This isn't really an issue of kube-aws but rather Kubernetes and/or CNI issue. +Anyways, it doesn't work if `hostNetwork: false`. + +If you want to deploy `nginx-ingress-controller` which requires `hostPort`, just set `hostNetwork: true`: + +``` + spec: + hostNetwork: true + containers: + - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3 + name: nginx-ingress-lb +``` + +Relevant kube-aws issue: [does hostPort not work on kube-aws/CoreOS?](https://github.com/coreos/kube-aws/issues/91) + +See [the upstream issue](https://github.com/kubernetes/kubernetes/issues/23920#issuecomment-254918942) for more information. diff --git a/Documentation/kubernetes-on-aws-node-pool.md b/Documentation/kubernetes-on-aws-node-pool.md new file mode 100644 index 000000000..2e80986ce --- /dev/null +++ b/Documentation/kubernetes-on-aws-node-pool.md @@ -0,0 +1,156 @@ +# Node Pool + +Node Pool allows you to bring up additional pools of worker nodes each with a separate configuration including: + +* Instance Type +* Storage Type/Size/IOPS +* Instance Profile +* Additional, User-Provided Security Group(s) +* Spot Price +* AWS service to manage your EC2 instances: [Auto Scaling](http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html) or [Spot Fleet](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html) +* [Node labels](http://kubernetes.io/docs/user-guide/node-selection/) +* [Taints](https://github.com/kubernetes/kubernetes/issues/17190) + +## Deploying a Multi-AZ cluster with cluster-autoscaler support with Node Pools + +Edit the `cluster.yaml` file to decrease `workerCount`, which is meant to be number of worker nodes in the "main" cluster, down to zero: + +```yaml +workerCount: 0 +subnets: + - availabilityZone: us-west-1a + instanceCIDR: "10.0.0.0/24" +``` + +Update the main cluster to catch up changes made in `cluster.yaml`: + +``` +$ kube-aws update \ + --s3-uri s3:/// +``` + +Create two node pools, each with a different subnet and an availability zone: + +``` +$ kube-aws node-pools init --node-pool-name first-pool-in-1a \ + --availability-zone us-west-1a \ + --key-name ${KUBE_AWS_KEY_NAME} \ + --kms-key-arn ${KUBE_AWS_KMS_KEY_ARN} + +$ kube-aws node-pools init --node-pool-name second-pool-in-1b \ + --availability-zone us-west-1a \ + --key-name ${KUBE_AWS_KEY_NAME} \ + --kms-key-arn ${KUBE_AWS_KMS_KEY_ARN} +``` + +Edit the `cluster.yaml` for the first zone: + +``` +$ $EDITOR node-pools/first-pool-in-1a/cluster.yaml +``` + +```yaml +workerCount: 1 +subnets: + - availabilityZone: us-west-1a + instanceCIDR: "10.0.1.0/24" +``` + +Edit the `cluster.yaml` for the second zone: + +``` +$ $EDITOR node-pools/second-pool-in-1b/cluster.yaml +``` + +```yaml +workerCount: 1 +subnets: + - availabilityZone: us-west-1b + instanceCIDR: "10.0.2.0/24" +``` + +Launch the node pools: + +``` +$ kube-aws node-pools up --node-pool-name first-pool-in-1a \ + --s3-uri s3:/// + +$ kube-aws node-pools up --node-pool-name second-pool-in-1b \ + --s3-uri s3:/// +``` + +Deployment of cluster-autoscaler is currently out of scope of this documentation. +Please read [cluster-autoscaler's documentation](https://github.com/kubernetes/contrib/blob/master/cluster-autoscaler/cloudprovider/aws/README.md) for instructions on it. + +## Customizing min/max size of the auto scaling group + +If you've chosen to power your worker nodes in a node pool with an auto scaling group, you can customize `MinSize`, `MaxSize`, `MinInstancesInService` in `cluster.yaml`: + +Please read [the AWS documentation](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#aws-properties-as-group-prop) for more information on `MinSize`, `MaxSize`, `MinInstancesInService` for ASGs. + +``` +worker: + # Auto Scaling Group definition for workers. If only `workerCount` is specified, min and max will be the set to that value and `rollingUpdateMinInstancesInService` will be one less. + autoScalingGroup: + minSize: 1 + maxSize: 3 + rollingUpdateMinInstancesInService: 2 +``` + +See [the detailed comments in `cluster.yaml`](https://github.com/coreos/kube-aws/blob/master/nodepool/config/templates/cluster.yaml) for further information. + +## Deploying a node pool powered by Spot Fleet + +Utilizing Spot Fleet gives us chances to dramatically reduce cost being spent on EC2 instances powering Kubernetes worker nodes while achieving reasonable availability. +AWS says cost reduction is up to 90% but the cost would slightly vary among instance types and other users' bids. + +Spot Fleet support may change in backward-incompatible ways as it is still an experimenta feature. +So, please use this feature at your own risk. +However, we'd greatly appreciate your feedbacks because they do accelerate improvements in this area! + +This feature assumes you already have the IAM role with ARN like "arn:aws:iam::youraccountid:role/aws-ec2-spot-fleet-role" in your own AWS account. +It implies that you've arrived "Spot Requests" in EC2 Dashboard in the AWS console at least once. +See [the AWS documentation describing pre-requisites for Spot Fleet](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-requests.html#spot-fleet-prerequisites) for details. + +To add a node pool powered by Spot Fleet, edit node pool's `cluster.yaml`: + +```yaml +worker: + spotFleet: + targetCapacity: 3 +``` + +To customize your launch specifications to diversify your pool among instance types other than the defaults, edit `cluster.yaml`: + +```yaml +worker: + spotFleet: + targetCapacity: 5 + launchSpecifications: + - weightedCapacity: 1 + instanceType: m3.medium + - weightedCapacity: 2 + instanceType: m3.large + - weightedCapacity: 2 + instanceType: m4.large +``` + +This configuration would normally result in Spot Fleet to bring up 3 instances to meet your target capacity: + +* 1x m3.medium = 1 capacity +* 1x m3.large = 2 capacity +* 1x m4.large = 2 capacity + +This is achieved by the `diversified` strategy of Spot Fleet. +Please read [the AWS documentation describing Spot Fleet Allocation Strategy](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html#spot-fleet-allocation-strategy) for more details. + +Please also see [the detailed comments in `cluster.yaml`](https://github.com/coreos/kube-aws/blob/master/nodepool/config/templates/cluster.yaml) and [the GitHub issue summarizing the initial implementation](https://github.com/coreos/kube-aws/issues/112) of this feature for further information. + +When you are done with your cluster, [destroy your cluster][aws-step-6] + +[aws-step-1]: kubernetes-on-aws.md +[aws-step-2]: kubernetes-on-aws-render.md +[aws-step-3]: kubernetes-on-aws-launch.md +[aws-step-4]: kube-aws-cluster-updates.md +[aws-step-5]: kubernetes-on-aws-node-pool.md +[aws-step-6]: kubernetes-on-aws-destroy.md diff --git a/Documentation/kubernetes-on-aws-prerequisites.md b/Documentation/kubernetes-on-aws-prerequisites.md new file mode 100644 index 000000000..d7eb11974 --- /dev/null +++ b/Documentation/kubernetes-on-aws-prerequisites.md @@ -0,0 +1,26 @@ +# Pre-requisites + +If you're deploying a cluster with kube-aws: + +* [EC2 instances whose types are larger than or equal to `m3.medium` should be chosen for the cluster to work reliably](https://github.com/coreos/kube-aws/issues/138) +* [At least 3 etcd, 2 controller, 2 worker nodes are required to achieve high availability](https://github.com/coreos/kube-aws/issues/138#issuecomment-266432162) + +## Deploying to an existing VPC + +If you're deploying a cluster to an existing VPC: + +* Internet Gateway needs to be added to VPC before cluster can be created + * Or [all the nodes will fail to launch because they can't pull docker images or ACIs required to run essential processes like fleet, hyperkube, etcd, awscli, cfn-signal, cfn-init.](https://github.com/coreos/kube-aws/issues/120) +* Existing route tables to be reused by kube-aws must be tagged with the key `KubernetesCluster` and your cluster's name for the value. + * Or [Kubernetes will fail to create ELBs correspond to Kubernetes services with `type=LoadBalancer`](https://github.com/coreos/kube-aws/issues/135) +* ["DNS Hostnames" must be turned on before cluster can be created](https://github.com/coreos/kube-aws/issues/119) + * Or etcd nodes are unable to communicate each other thus the cluster doesn't work at all + +Once you understand pre-requisites, you are [ready to launch your first Kubernetes cluster][aws-step-1]. + +[aws-step-1]: kubernetes-on-aws.md +[aws-step-2]: kubernetes-on-aws-render.md +[aws-step-3]: kubernetes-on-aws-launch.md +[aws-step-4]: kube-aws-cluster-updates.md +[aws-step-5]: kubernetes-on-aws-node-pool.md +[aws-step-6]: kubernetes-on-aws-destroy.md diff --git a/Documentation/kubernetes-on-aws-render.md b/Documentation/kubernetes-on-aws-render.md index 4a0cc5c2c..3df8b9666 100644 --- a/Documentation/kubernetes-on-aws-render.md +++ b/Documentation/kubernetes-on-aws-render.md @@ -32,7 +32,7 @@ $ aws kms --region= create-key --description="kube-aws assets" } ``` -You will use the KeyMetadata.Arn string to identify your KMS key in the init step. +You will use the `KeyMetadata.Arn` string to identify your KMS key in the init step. ### External DNS name @@ -42,6 +42,19 @@ When the cluster is created, the controller will expose the TLS-secured API on a Alternatively, kube-aws can automatically create this A record in an *existing* [Route 53][route53] hosted zone. If you have a DNS zone hosted in Route 53, you can configure for it below. +### S3 bucket + +[Amazon S3](http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) buckets are used to transfer large stack templates generated by kube-aws to CloudFormation. If you already have a S3 bucket that you would like to use, you can skip creating a new bucket and provide the URI for your existing bucket. + +You can create a S3 bucket in the [AWS console](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#create-bucket-intro), or with the `aws` command line tool: + +```sh +$ aws s3api --region= create-bucket --bucket +{ + "Location": "/" +} +``` + ## Initialize an asset directory Create a directory on your local machine to hold the generated assets: @@ -85,6 +98,19 @@ There will now be a `cluster.yaml` file in the asset directory. This is the main * In certain cases, such as users with advanced pre-existing PKI infrastructure, the operator may wish to pre-generate all cluster TLS assets. In this case, you can run `kube-aws render stack` and copy in your TLS assets into the `credentials/` folder before running `kube-aws up`. + ```sh + ls -R credentials/ + credentials/: + admin-key.pem apiserver-key.pem ca.pem etcd-client.pem etcd.pem worker.pem + admin.pem apiserver.pem etcd-client-key.pem etcd-key.pem worker-key.pem + ``` + +The next command generates the default set of cluster assets in your asset directory. + + ```sh + $ kube-aws render stack + ``` + Here's what the directory structure looks like: ```sh @@ -113,7 +139,7 @@ $ tree These assets (templates and credentials) are used to create, update and interact with your Kubernetes cluster. -At this point you should be ready to create your cluster. You can also check the `my-cluster` asset directory into version control if you desire. The contents of this directory are your reproducible cluster assets. Please take care not to commit the `my-cluster/credentials` directory, as it contains your TLS secrets. If you're using git, the `credentials` directory will already be ignored for you. +At this point you should be ready to create your cluster. You can also now check the `my-cluster` asset directory into version control if you desire. The contents of this directory are your reproducible cluster assets. Please take care not to commit the `my-cluster/credentials` directory but rather to encrypt and/or put it to more secure storage, as it contains your TLS secrets. If you're using git, the `credentials` directory will already be ignored for you. **PRODUCTION NOTE**: the TLS keys and certificates generated by `kube-aws` should *not* be used to deploy a production Kubernetes cluster. Each component certificate is only valid for 90 days, while the CA is valid for 365 days. @@ -174,7 +200,7 @@ You can now customize your cluster by editing asset files. Any changes to these * **credentials/** - This directory contains the **unencrypted** TLS assets for your cluster, along with a pre-configured `kubeconfig` file which provides access to your cluster api via kubectl. + This directory contains both encryped and **unencrypted** TLS assets for your cluster, along with a pre-configured `kubeconfig` file which provides access to your cluster api via kubectl. [mount-disks]: https://coreos.com/os/docs/latest/mounting-storage.html [insecure-registry]: https://coreos.com/os/docs/latest/registry-authentication.html#using-a-registry-without-ssl-configured @@ -188,14 +214,14 @@ Edit the `cluster.yaml` file: ```yaml containerRuntime: rkt -releaseChannel: alpha +releaseChannel: stable ``` -Note that while using rkt as the runtime is now supported, it is still a new option as of the Kubernetes v1.3 release and has a few [known issues](http://kubernetes.io/docs/getting-started-guides/rkt/notes/). +Note that while using rkt as the runtime is now supported, it is still a new option as of the Kubernetes v1.4 release and has a few [known issues](http://kubernetes.io/docs/getting-started-guides/rkt/notes/). ### Calico network policy -The cluster can be optionally configured to use Calico to provide network policy. These policies limit and control how different pods, namespaces, etc can communicate with each other. These rules can be managed after the cluster is launched, but the feature needs to be turned on beforehand. +The cluster can be optionally configured to use Calico to provide [network policy](http://kubernetes.io/docs/user-guide/networkpolicies/). These policies limit and control how different pods, namespaces, etc can communicate with each other. These rules can be managed after the cluster is launched, but the feature needs to be turned on beforehand. Edit the `cluster.yaml` file: @@ -205,22 +231,50 @@ useCalico: true ### Route53 Host Record -`kube-aws` can optionally create an A record for the controller IP in an existing route53 hosted zone. +`kube-aws` can optionally create an ALIAS record for the controller's ELB in an existing Route53 hosted zone. Edit the `cluster.yaml` file: ```yaml externalDNSName: kubernetes.staging.example.com createRecordSet: true -hostedZone: staging.example.com +hostedZoneId: A12B3CDE4FG5HI +# DEPRECATED: use hostedZoneId instead +#hostedZone: staging.example.com ``` -If `createRecordSet` is not set to true, the deployer will be responsible for making externalDNSName routable to the controller IP after the cluster is created. +If `createRecordSet` is not set to true, the deployer will be responsible for making externalDNSName routable to the the ELB managing the controller nodes after the cluster is created. ### Multi-AZ Clusters Kube-aws supports "spreading" a cluster across any number of Availability Zones in a given region. +__A word of caution about EBS and Persistent Volumes__: Any pods deployed to a Multi-AZ cluster must mount EBS volumes via [Persistent Volume Claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims). Specifying the ID of the EBS volume directly in the pod spec will not work consistently if nodes are spread across multiple zones. + +Read more about Kubernetes Multi-AZ cluster support [here](http://kubernetes.io/docs/admin/multiple-zones/). + +#### A common pitfall when deploying multi-AZ clusters in combination with cluster-autoscaler + +[cluster-autoscaler](https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler) is a tool that automatically adjusts the number of Kubernetes worker nodes when: + +> * there is a pod that doesn’t have enough space to run in the cluster +> * some nodes in the cluster are so underutilized, for an extended period of time, that they can be deleted and their pods will be easily placed on some other, existing nodes. +> https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler + +A common pitfall in deploying cluster-autoscaler to a multi-AZ cluster is that you have to instruct an Auto Scaling Group not to spread over multiple availability zones or cluster-autoscaler results in instability while scaling out the nodes i.e. it takes unnecessary much time to finally bring up a node in the insufficient zone. + +> The autoscaling group should span 1 availability zone for the cluster autoscaler to work. If you want to distribute workloads evenly across zones, set up multiple ASGs, with a cluster autoscaler for each ASG. At the time of writing this, cluster autoscaler is unaware of availability zones and although autoscaling groups can contain instances in multiple availability zones when configured so, the cluster autoscaler can't reliably add nodes to desired zones. That's because AWS AutoScaling determines which zone to add nodes which is out of the control of the cluster autoscaler. For more information, see https://github.com/kubernetes/contrib/pull/1552#discussion_r75533090. +> https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler/cloudprovider/aws#deployment-specification + +Please read the following guides carefully and select the appropriate deployment according to your requirement regarding auto-scaling. + +#### For production cluster not requiring cluster-autoscaler + +If you don't need auto-scaling at all, or you need only the AWS-native auto-scaling i.e. the a combination of auto scaling groups, scaling policies, CloudWatch alarms, +theoretically, you can safely go this way. + +Edit the `cluster.yaml` file to define multiple subnets, each with a different availability zone hence multi-AZ: + ```yaml subnets: - availabilityZone: us-west-1a @@ -228,9 +282,15 @@ Kube-aws supports "spreading" a cluster across any number of Availability Zones - availabilityZone: us-west-1b instanceCIDR: "10.0.1.0/24" ``` -__A word of caution about EBS and Persistent Volumes__: Any pods deployed to a Multi-AZ cluster must mount EBS volumes via [Persistent Volume Claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims). Specifying the ID of the EBS volume directly in the pod spec will not work consistently if nodes are spread across multiple zones. -Read more about Kubernetes Multi-AZ cluster support [here](http://kubernetes.io/docs/admin/multiple-zones/). +This implies that you rely on AWS AutoScaling for selecting which subnet hence which availability zone to add a node when an auto-scaling group's `DesiredCapacity` is increased. + +Please read [the AWS documentation for more details about AWS Auto Scaling](http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html). + +#### For production cluster requiring cluster-autoscaler + +You must utilize [an experimental feature called Node Pool](aws-experimental-features) to achieve this deployment. +Please read [the documentation for experimental features](aws-experimental-features) for more instructions. ### Certificates and Keys @@ -289,7 +349,7 @@ The `validate` command check the validity of your changes to the cloud-config us This is an important step to make sure your stack will launch successfully: ```sh -$ kube-aws validate +$ kube-aws validate --s3-uri s3:/// ``` If your files are valid, you are ready to [launch your cluster][aws-step-3]. @@ -297,6 +357,8 @@ If your files are valid, you are ready to [launch your cluster][aws-step-3]. [aws-step-1]: kubernetes-on-aws.md [aws-step-2]: kubernetes-on-aws-render.md [aws-step-3]: kubernetes-on-aws-launch.md +[aws-step-4]: kube-aws-cluster-updates.md +[aws-step-5]: kubernetes-on-aws-node-pool.md [k8s-openssl]: openssl.md [tls-note]: #certificates-and-keys [route53]: https://aws.amazon.com/route53/ diff --git a/Documentation/kubernetes-on-aws.md b/Documentation/kubernetes-on-aws.md index 3d6bba7cc..b927e584e 100644 --- a/Documentation/kubernetes-on-aws.md +++ b/Documentation/kubernetes-on-aws.md @@ -19,11 +19,23 @@ Each of the steps will cover: * [Step 3: Launch][aws-step-3] * Create the CloudFormation stack and start our EC2 machines * Set up CLI access to the new cluster +* [Step 4: Update][aws-step-4] + * Update the CloudFormation stack +* [Step 5: Add Node Pool][aws-step-5] + * Create the additional pool of worker nodes + * Adjust template configuration for each pool of worker nodes + * Required to support [cluster-autoscaler](https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler) +* [Step 6: Destroy][aws-step-6] + * Destroy the cluster Let's get started. ## Download kube-aws +Go to the [releases](https://github.com/coreos/kube-aws/releases) and download the latest release tarball for your architecture. + +Currently, binaries coming from the final release for each version are signed so you should download the corresponding signature (.sig) as well. If you've decided to download a non-final release you can skip down to "Extract the binary:". + Import the [CoreOS Application Signing Public Key](https://coreos.com/security/app-signing-key/): ```sh @@ -86,6 +98,15 @@ aws_access_key_id = AKID1234567890 aws_secret_access_key = MY-SECRET-KEY ``` +### Method 3: Environment variables + +Provide AWS credentials to kube-aws by exporting the following environment variables: + +```sh +export AWS_ACCESS_KEY_ID=AKID1234567890 +export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY +``` + ## Test Credentials Test that your credentials work by describing any instances you may already have running on your account: @@ -104,3 +125,6 @@ $ aws ec2 describe-instances [aws-step-1]: kubernetes-on-aws.md [aws-step-2]: kubernetes-on-aws-render.md [aws-step-3]: kubernetes-on-aws-launch.md +[aws-step-4]: kube-aws-cluster-updates.md +[aws-step-5]: kubernetes-on-aws-node-pool.md +[aws-step-6]: kubernetes-on-aws-destroy.md diff --git a/README.md b/README.md index 07903601a..0b5229608 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,8 @@ -# Kubernetes on AWS +# Kubernetes on AWS (kube-aws) This is the source of the `kube-aws` tool and the installation artifacts used by the official Kubernetes on AWS documentation. -View the full instructions at https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html. + +View the full instructions at [GitHub](/Documentation/kubernetes-on-aws.md) or at [the CoreOS documentation website](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html). --- @@ -11,82 +12,45 @@ This survey is meant for those who are currently running at least some workloads --- -### Download pre-built binary - -Go to the [releases](https://github.com/coreos/kube-aws/releases) and download the latest release tarball for your architecture. - -Currently, binaries coming from the final release for each version are signed so you should download the corresponding signature (.sig) as well. If you've decided to download a non-final release you can skip down to "Extract the binary:". - -Import the [CoreOS Application Signing Public Key](https://coreos.com/security/app-signing-key/): - -```sh -gpg2 --keyserver pgp.mit.edu --recv-key FC8A365E -``` - -Validate the key fingerprint: - -```sh -gpg2 --fingerprint FC8A365E -``` -The correct key fingerprint is `18AD 5014 C99E F7E3 BA5F 6CE9 50BD D3E0 FC8A 365E` +## Features + +* Create, update and destroy Kubernetes clusters on AWS +* Highly available and scalable Kubernetes clusters backed by multi-AZ deployment and Node Pools +* Deployment to an existing VPC +* Powered on various AWS services including CloudFormation, KMS, Auto Scaling, Spot Fleet, EC2, ELB, S3, etc. + +## Getting Started + +Check out our getting started tutorial on launching your first Kubernetes cluster in AWS. + +* [Pre-requisites](/Documentation/kubernetes-on-aws-prerequisites.md) +* [Step 1: Configure](/Documentation/kubernetes-on-aws.md) + * Download the latest release of kube-aws + * Define account and cluster settings +* [Step 2: Render](/Documentation/kubernetes-on-aws-render.md) + * Compile a re-usable CloudFormation template for the cluster + * Optionally adjust template configuration + * Validate the rendered CloudFormation stack +* [Step 3: Launch](/Documentation/kubernetes-on-aws-launch.md) + * Create the CloudFormation stack and start our EC2 machines + * Set up CLI access to the new cluster +* [Step 4: Update](/Documentation/kube-aws-cluster-updates.md) + * Update the CloudFormation stack +* [Step 5: Add Node Pool](/Documentation/kubernetes-on-aws-node-pool.md) + * Create the additional pool of worker nodes + * Adjust template configuration for each pool of worker nodes + * Required to support [cluster-autoscaler](https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler) +* [Step 6: Destroy](/Documentation/kubernetes-on-aws-destroy.md) + * Destroy the cluster + +## Examples + +Generate `cluster.yaml`: -Validate the tarball's GPG signature: - -```sh -PLATFORM=linux-amd64 -# Or -PLATFORM=darwin-amd64 - -gpg2 --verify kube-aws-${PLATFORM}.tar.gz.sig kube-aws-${PLATFORM}.tar.gz -``` -Extract the binary: - -```sh -tar zxvf kube-aws-${PLATFORM}.tar.gz -``` - -Add kube-aws to your path: - -```sh -mv ${PLATFORM}/kube-aws /usr/local/bin ``` - -### AWS Credentials -The supported way to provide AWS credentials to kube-aws is by exporting the following environment variables: - -```sh -export AWS_ACCESS_KEY_ID=AKID1234567890 -export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY -``` - -### Create a KMS Key - -[Amazon KMS](http://docs.aws.amazon.com/kms/latest/developerguide/overview.html) keys are used to encrypt and decrypt cluster TLS assets. If you already have a KMS Key that you would like to use, you can skip this step. - -Creating a KMS key can be done via the AWS web console or via the AWS cli tool: - -```shell -$ aws kms --region=us-west-1 create-key --description="kube-aws assets" -{ - "KeyMetadata": { - "CreationDate": 1458235139.724, - "KeyState": "Enabled", - "Arn": "arn:aws:kms:us-west-1:xxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx", - "AWSAccountId": "xxxxxxxxxxxxx", - "Enabled": true, - "KeyUsage": "ENCRYPT_DECRYPT", - "KeyId": "xxxxxxxxx", - "Description": "kube-aws assets" - } -} -``` -You'll need the `KeyMetadata.Arn` string for the next step: - -## Initialize an asset directory -```sh $ mkdir my-cluster $ cd my-cluster -$ kube-aws init --cluster-name= \ +$ kube-aws init --cluster-name=my-cluster \ --external-dns-name= \ --region=us-west-1 \ --availability-zone=us-west-1c \ @@ -94,140 +58,104 @@ $ kube-aws init --cluster-name= \ --kms-key-arn="arn:aws:kms:us-west-1:xxxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx" ``` -There will now be a cluster.yaml file in the asset directory. - -## Render contents of the asset directory - -* In the simplest case, you can have kube-aws generate both your TLS identities and certificate authority for you. - - ```sh - $ kube-aws render credentials --generate-ca - ``` - - This is not recommended for production. - -* It is recommended that, for production, you supply your own immediate certificate signing authority. - - ```sh - $ kube-aws render credentials --ca-cert-path=/path/to/ca-cert.pem --ca-key-path=/path/to/ca-key.pem - ``` - - For more information on operating your own CA, check out this [awesome guide](https://jamielinux.com/docs/openssl-certificate-authority/). - -* In certain cases, such as users with advanced pre-existing PKI infrastructure, you may wish to pre-generate all cluster TLS assets. In this case, make sure the file tree below exists in your cluster assets directory before running `kube-aws up`. - - ```sh - ls -R credentials/ - credentials/: - admin-key.pem apiserver-key.pem ca.pem etcd-client.pem etcd.pem worker.pem - admin.pem apiserver.pem etcd-client-key.pem etcd-key.pem worker-key.pem - ``` +Generate assets: -The next command generates the default set of cluster assets in your asset directory. These assets are templates that are used to create, update and interact with your Kubernetes cluster. - - ```sh - $ kube-aws render stack - ``` - -You can now customize your cluster by editing asset files: - -* **cluster.yaml** - - This is the configuration file for your cluster. It contains the configuration parameters that are templated into your userdata and cloudformation stack. - -* **userdata/** - - * `cloud-config-worker` - * `cloud-config-controller` - - This directory contains the [cloud-init](https://github.com/coreos/coreos-cloudinit) cloud-config userdata files. The CoreOS operating system supports automated provisioning via cloud-config files, which describe the various files, scripts and systemd actions necessary to produce a working cluster machine. These files are templated with your cluster configuration parameters and embedded into the cloudformation stack template. - -* **stack-template.json** - - This file describes the [AWS cloudformation](https://aws.amazon.com/cloudformation/) stack which encompasses all the AWS resources associated with your cluster. This JSON document is templated with configuration parameters, we well as the encoded userdata files. - -* **credentials/** - - This directory contains the **unencrypted** TLS assets for your cluster, along with a pre-configured `kubeconfig` file which provides access to your cluster api via kubectl. - -You can also now check the `my-cluster` asset directory into version control if you desire. The contents of this directory are your reproducible cluster assets. Please take care not to commit the `my-cluster/credentials` directory, as it contains your TLS secrets. If you're using git, the `credentials` directory will already be ignored for you. - -## Route53 Host Record (optional) - -`kube-aws` can optionally create an A record for the controller IP in an existing hosted zone. - -Edit the `cluster.yaml` file: - -```yaml -externalDNSName: my-cluster.staging.core-os.net -createRecordSet: true -hostedZone: staging.core-os.net +``` +$ kube-aws render credentials --generate-ca +$ kube-aws render stack ``` -If `createRecordSet` is not set to true, the deployer will be responsible for making externalDNSName routable to the controller IP after the cluster is created. +Validate configuration: -## Validate your cluster assets +``` +$ kube-aws validate --s3-uri s3:/// +``` -The `validate` command check the validity of the cloud-config userdata files and the cloudformation stack description: +Launch: -```sh -$ kube-aws validate ``` +$ kube-aws up --s3-uri s3:/// -## Create a cluster from asset directory +# Or export your cloudformation stack +$ kube-aws up --export -```sh -$ kube-aws up +# Access the cluster +$ KUBECONFIG=kubeconfig kubectl get nodes --show-labels ``` -This command can take a while. - -## Access the cluster +Update: -```sh -$ kubectl --kubeconfig=kubeconfig get nodes +``` +$ $EDITOR cluster.yaml +$ kube-aws update --s3-uri s3:/// ``` -It can take some time after `kube-aws up` completes before the cluster is available. Until then, you will have a `connection refused` error. - -## Export your cloudformation stack +Node Pool: -```sh -$ kube-aws up --export +``` +$ kube-aws node-pools init --node-pool-name my-pool +$ kube-aws node-pools render --node-pool-name my-pool +$ kube-aws node-pools validate --node-pool-name my-pool \ + --s3-uri s3:/// +$ kube-aws node-pools up --node-pool-name my-pool \ + --s3-uri s3:/// +$ $EDITOR node-pools/my-pool/cluster.yaml +$ kube-aws node-pools update --node-pool-name my-pool \ + --s3-uri s3:/// ``` -## Update an existing kube-aws cluster +Destroy: -Read the [cluster update](Documentation/kube-aws-cluster-updates.md) documentation. +``` +$ kube-aws node-pools destroy --node-pool-name my-pool +$ kube-aws destroy +``` ## Development ### Build -Run the `./build` script to compile `kube-aws` locally. +Clone this repository to the appropriate path under the GOPATH. + +``` +$ export GOPATH=$HOME/go +$ mkdir -p $GOPATH/src/github.com/coreos/ +$ git clone git@github.com:coreos/kube-aws.git $GOPATH/src/github.com/coreos/kube-aws +``` + +Run `make build` to compile `kube-aws` locally. This depends on having: -* golang >= 1.5 +* golang >= 1.7 The compiled binary will be available at `bin/kube-aws`. ### Run Unit Tests ```sh -go test $(go list ./... | grep -v '/vendor/') +make test ``` -### Modifying templates +### Reformat Code -The various templates are located in the `pkg/config/templates/` folder of the source repo. `go generate` is used to pack these templates into the source code. In order for changes to templates to be reflected in the source code: +```sh +make format +``` + +### Modifying Templates + +The various templates are located in the `config/templates/` and the `nodepool/config/templates/` directory of the source repo. `go generate` is used to pack these templates into the source code. In order for changes to templates to be reflected in the source code: ```sh -go generate ./pkg/config +make build ``` -This command is run automatically as part of the `build` script. +## Other Resources + +Extra or advanced topics in for kube-aws: -### Useful Resources +* [Known Limitations](/Documentation/kubernetes-on-aws-limitations.md) +* [Roadmap](/ROADMAP.md) The following links can be useful for development: diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 000000000..9a1654dda --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,35 @@ +# Roadmap + +This document is meant to provide high-level but actionable objectives for future kube-aws deveploment. +Please file an issue to make suggestions on this roadmap! + +## Every stage + + * Provide article walking users through: + * setting up a cluster from scratch + * using/enabling new features + * (breaking changes) + +## Stage 1: v0.9.2 + + * Node Pools + * Worker nodes optionally powered by Spot Fleet + * Clean cluster upgrades (preventing downtime, make sure they succeed) + +## Stage 2: v0.9.3 + + * Cluster Auto Scaling + * Including partial support for auto-scaling worker nodes, kube-dns + * Self-hosted Calico + +# Stage 3: v0.9.4 + + * etcd improvements + * Backups + * Recovery + * YAML CloudFormation templates + +## Stage N: v0.9.x + + * Bootkube switch + * `kube-aws` can largely go into maintenance mode when k8s upgrades can be safely achieved on self-hosted clusters.