EKS Workshop update to EKS 1.16 (#85)

* Changed initial part of the toolset to 1.16

* Added changes up to helm and commented kube-report-ops section

* changed from spot instace advisor to ec2-instance-selector

* modified ec2-instance-selector and aws-termination-handler sections

* changes to cluster autoscaler and jenkins

* updated cleanup sections

content/using_ec2_spot_instances_with_eks/cleanup.md

@@ -18,7 +18,7 @@ Before you clean up the resources and complete the workshop, you may want to rev
 kubectl delete hpa monte-carlo-pi-service
 kubectl delete -f ~/environment/cluster-autoscaler/cluster_autoscaler.yml
 kubectl delete -f monte-carlo-pi-service.yml
-helm delete --purge kube-ops-view kube-resource-report metrics-server
+helm delete kube-ops-view metrics-server
 ```
 
 ## Removing eks nodegroups

content/using_ec2_spot_instances_with_eks/eksctl/launcheks.md

@@ -41,7 +41,7 @@ The following command will create an eks cluster with the name `eksworkshop-eksc
 .It will also create a nodegroup with 2 on-demand instances.
 
 ```
-eksctl create cluster --version=1.15 --name=eksworkshop-eksctl --node-private-networking  --managed --nodes=2 --alb-ingress-access --region=${AWS_REGION} --node-labels="lifecycle=OnDemand,intent=control-apps" --asg-access
+eksctl create cluster --version=1.16 --name=eksworkshop-eksctl --node-private-networking  --managed --nodes=2 --alb-ingress-access --region=${AWS_REGION} --node-labels="lifecycle=OnDemand,intent=control-apps" --asg-access
 ```
 
 eksctl allows us to pass parameters to initialize the cluster. While initializing the cluster, eksctl does also allow us to create nodegroups.

content/using_ec2_spot_instances_with_eks/eksctl/prerequisites.md

@@ -6,7 +6,7 @@ weight: 10
 
 For this module, we need to download the [eksctl](https://eksctl.io/) binary:
 ```
-export EKSCTL_VERSION=0.18.0
+export EKSCTL_VERSION=0.23.0
 curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/${EKSCTL_VERSION}/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp
 sudo mv -v /tmp/eksctl /usr/local/bin
 ```

content/using_ec2_spot_instances_with_eks/helm_root/deploy_metric_server.md

@@ -8,8 +8,9 @@ weight: 20
 Metrics Server is a cluster-wide aggregator of resource usage data. These metrics will drive the scaling behavior of the [deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). We will deploy the metrics server using `Helm` configured earlier in this workshop.
 
 ```
-helm install stable/metrics-server \
-    --name metrics-server \
+kubectl create namespace metrics
+helm install metrics-server \
+    stable/metrics-server \
     --version 2.10.0 \
     --namespace metrics
 ```

content/using_ec2_spot_instances_with_eks/helm_root/helm_deploy.md

@@ -4,63 +4,43 @@ date: 2018-08-07T08:30:11-07:00
 weight: 10
 ---
 
-Before we can get started configuring `helm` we'll need to first install the command line tools that you will interact with. To do this run the following.
+## Install the Helm CLI
 
-```
-cd ~/environment
-curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
-chmod +x get_helm.sh
-./get_helm.sh
+Before we can get started configuring Helm, we'll need to first install the
+command line tools that you will interact with. To do this, run the following:
+
+```sh
+curl -sSL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
 ```
 
-{{% notice info %}}
-Once you install helm, the command will prompt you to run 'helm init'. **Do not run 'helm init'.** Follow the instructions to configure helm using **Kubernetes RBAC** and then install tiller as specified below
-If you accidentally run 'helm init', you can safely uninstall tiller by running 'helm reset --force'
-{{% /notice %}}
+We can verify the version
 
-### Configure Helm access with RBAC
+```sh
+helm version --short
+```
 
-Helm relies on a service called **tiller** that requires special permission on the
-kubernetes cluster, so we need to build a _**Service Account**_ for **tiller**
-to use. We'll then apply this to the cluster.
+Let's configure our first Chart repository. Chart repositories are similar to
+APT or yum repositories that you might be familiar with on Linux, or Taps for
+Homebrew on macOS.
 
-To create a new service account manifest:
-```
-cat <<EoF > ~/environment/rbac.yaml
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: tiller
-  namespace: kube-system
----
-apiVersion: rbac.authorization.k8s.io/v1beta1
-kind: ClusterRoleBinding
-metadata:
-  name: tiller
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: cluster-admin
-subjects:
-  - kind: ServiceAccount
-    name: tiller
-    namespace: kube-system
-EoF
-```
+Download the `stable` repository so we have something to start with:
 
-Next apply the config:
-```
-kubectl apply -f ~/environment/rbac.yaml
+```sh
+helm repo add stable https://kubernetes-charts.storage.googleapis.com/
+helm repo update
 ```
 
-Then we can install **tiller** using the **helm** tooling
+Once this is installed, we will be able to list the charts you can install:
 
-```
-helm init --service-account tiller
+```sh
+helm search repo stable
 ```
 
-This will install **tiller** into the cluster which gives it access to manage
-resources in your cluster.
-
+Finally, let's configure Bash completion for the `helm` command:
 
+```sh
+helm completion bash >> ~/.bash_completion
+. /etc/profile.d/bash_completion.sh
+. ~/.bash_completion
+source <(helm completion bash)
+```

content/using_ec2_spot_instances_with_eks/helm_root/install_kube_ops_view.md

@@ -10,9 +10,8 @@ that will help with understanding our cluster setup in a visual way. The first o
 The following line updates the stable helm repository and then installs kube-ops-view using a LoadBalancer Service type and creating a RBAC (Resource Base Access Control) entry for the read-only service account to read nodes and pods information from the cluster.
 
 ```
-helm repo update
-helm install stable/kube-ops-view \
---name kube-ops-view \
+helm install kube-ops-view \
+stable/kube-ops-view \
 --set service.type=LoadBalancer \
 --set nodeSelector.intent=control-apps \
 --set rbac.create=True
@@ -60,6 +59,15 @@ Spend some time checking the state and properties of your EKS cluster.
 
 ![kube-ops-view](/images/using_ec2_spot_instances_with_eks/helm/kube-ops-view-legend.png)
 
+<!--  
+
+# I'm commenting this section temporarily The ClusterRole associated with
+# the chart does not provide all the permissions for kube-report-ops
+# to work well and instead we are getting an error at the moment on EKS 1.16
+# this will require either a change in the kube-report-ops or changes to modify
+# The clusterrole once the helm chart is installed; I'll contribute this to the
+# upstream project and then get this section enabled back again.
+
 ### Exercise
  
 {{% notice info %}}
@@ -73,7 +81,7 @@ In this exercise we will install and explore another great tool, **[kube-resourc
 Execute the following command in your Cloud9 terminal
 ```
 git clone https://github.com/hjacobs/kube-resource-report
-helm install --name kube-resource-report \
+helm install kube-resource-report \
 --set service.type=LoadBalancer \
 --set service.port=80 \
 --set container.port=8080 \
@@ -100,5 +108,4 @@ Kube-resource-reports will keep track in time of the cluster. Further more, it i
 
 The result of this exercise should show kube-resource-report estimated cost of your cluster as well as the utilization of different components.
 
-
-
+-->

content/using_ec2_spot_instances_with_eks/jenkins/autoscaling_nodes.md

@@ -4,24 +4,13 @@ date: 2018-08-07T08:30:11-07:00
 weight: 80
 ---
 
-In a previous module in this workshop, we saw that we can use Kubernetes cluster-autoscaler to automatically increase the size of our nodegroups (EC2 Auto Scaling groups) when our Kubernetes deployment scaled out, and some of the pods remained in `pending` state due to lack of resources on the cluster. Let's implement the same concept for our Jenkins worker nodes and see this in action.
+In a previous module in this workshop, we saw that we can use Kubernetes cluster-autoscaler to automatically increase the size of our nodegroups (EC2 Auto Scaling groups) when our Kubernetes deployment scaled out, and some of the pods remained in `pending` state due to lack of resources on the cluster. Let's check the same concept applies for our Jenkins worker nodes and see this in action.
+
+If you recall, Cluster Autoscaler was configured to Auto-Discover Auto Scaling groups created with the tags : k8s.io/cluster-autoscaler/enabled, and k8s.io/cluster-autoscaler/eksworkshop-eksctl. You can find out in the AWS Console section for **EC2 -> Auto Scaling Group**, that the new jenkins node group does indeed have the right tags defined.
 
-#### Configuring cluster-autoscaler to use our new Jenkins dedicated nodegroup
-1\. Edit the cluster-autoscaler deployment configuration\
-```bash
-kubectl edit deployment cluster-autoscaler -n kube-system
-```
-2\. Under the two `--nodes=` lines where you configured your EC2 Auto Scaling group names in the previous module, add another line with the name of the new Jenkins dedicated nodegroup, so your file looks like this (but with different ASG names which you collected from the EC2 Management Console)\
-```
---nodes=0:5:eksctl-eksworkshop-eksctl10-nodegroup-dev-8vcpu-32gb-spot-NodeGroup-16XJ6GMZCT3XQ
---nodes=0:5:eksctl-eksworkshop-eksctl10-nodegroup-dev-4vcpu-16gb-spot-NodeGroup-1RBXH0I6585MX
---nodes=0:5:eksctl-eksworkshop-eksctl10-nodegroup-jenkins-agents-2vcpu-8gb-spot-2-NodeGroup-7GE4LS6B34DK
-```
-3\. Once you save/quit the file with `:x!`, the new configuration will apply\
 
 {{% notice tip %}}
-CI/CD workloads can benefit of Cluster Autoscaler ability to scale down to 0! Capacity will 
-be provided just when needed, which increases further cost savings.
+CI/CD workloads can benefit of Cluster Autoscaler ability to scale down to 0! Capacity will be provided just when needed, which increases further cost savings.
 {{% /notice %}}
 
 #### Running multiple Jenkins jobs to reach a Pending pods state

content/using_ec2_spot_instances_with_eks/jenkins/jenkins_cleanup.md

@@ -14,12 +14,6 @@ If you're running in your own account, make sure you run through these steps to
 helm delete cicd
 ```
 
-### Removing the Jenkins nodegroup from cluster-autoscaler
-```
-kubectl edit deployment cluster-autoscaler -n kube-system
-```
-Delete the third **\-\-nodes=** line that contains the Jenkins nodegroup name.
-
 ### Removing the Jenkins nodegroup
 ```
 eksctl delete nodegroup -f spot_nodegroup_jenkins.yml --approve

content/using_ec2_spot_instances_with_eks/jenkins/setup_jenkins.md

@@ -7,7 +7,7 @@ weight: 30
 #### Install Jenkins
 
 ```
-helm install stable/jenkins --set rbac.create=true,master.servicePort=80,master.serviceType=LoadBalancer --name cicd
+helm install cicd stable/jenkins --set rbac.create=true,master.servicePort=80,master.serviceType=LoadBalancer
 ```
 
 The output of this command will give you some additional information such as the
@@ -27,7 +27,7 @@ Once the pod status changes to `running`, we can get the load balancer address w
 
 ```
 export SERVICE_IP=$(kubectl get svc --namespace default cicd-jenkins --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
-echo http://$SERVICE_IP/login
+echo "Jenkins running at : http://$SERVICE_IP/login"
 ```
 
 The expected result should be similar to:

content/using_ec2_spot_instances_with_eks/prerequisites/at_an_aws_validaterole.md

@@ -18,6 +18,6 @@ If the _Arn_ contains the role name from above and an Instance ID, you may proce
 {
     "Account": "123456789012", 
     "UserId": "AROA1SAMPLEAWSIAMROLE:i-01234567890abcdef", 
-    "Arn": "arn:aws:sts::123456789012:assumed-role/TeamRole/MasterRole"
+    "Arn": "arn:aws:sts::216876048363:assumed-role/TeamRole/i-0dd09eac19be01448"
 }
 ```

content/using_ec2_spot_instances_with_eks/prerequisites/k8stools.md

@@ -15,14 +15,14 @@ for the download links.](https://docs.aws.amazon.com/eks/latest/userguide/gettin
 
 #### Install kubectl
 ```
-export KUBECTL_VERSION=v1.15.10
+export KUBECTL_VERSION=v1.16.12
 sudo curl --silent --location -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl
 sudo chmod +x /usr/local/bin/kubectl
 ```
 
 #### Install JQ and envsubst
 ```
-sudo yum -y install jq gettext
+sudo yum -y install jq gettext bash-completion
 ```
 
 #### Verify the binaries are in the path and executable

content/using_ec2_spot_instances_with_eks/prerequisites/sshkey.md

@@ -20,12 +20,6 @@ Press `enter` 3 times to take the default choices
 
 Upload the public key to your EC2 region:
 
-```bash
-aws ec2 import-key-pair --key-name "eksworkshop" --public-key-material file://~/.ssh/id_rsa.pub
-```
-
-If you got an error similar to `An error occurred (InvalidKey.Format) when calling the ImportKeyPair operation: Key is not in valid OpenSSH public key format` then you can try this command instead:
-
 ```bash
 aws ec2 import-key-pair --key-name "eksworkshop" --public-key-material fileb://~/.ssh/id_rsa.pub
-```
+```

content/using_ec2_spot_instances_with_eks/scaling/deploy_ca.files/cluster_autoscaler.yml

@@ -47,6 +47,9 @@ rules:
 - apiGroups: ["storage.k8s.io"]
   resources: ["storageclasses"]
   verbs: ["watch","list","get"]
+- apiGroups: ["storage.k8s.io"]
+  resources: ["csinodes"]
+  verbs: ["watch","list","get"]
 - apiGroups: ["batch"]
   resources: ["jobs"]
   verbs: ["watch","list","get"]
@@ -126,7 +129,7 @@ spec:
       nodeSelector:
         intent: control-apps      
       containers:
-        - image: k8s.gcr.io/cluster-autoscaler:v1.15.5
+        - image: us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.16.5
           name: cluster-autoscaler
           resources:
             limits:
@@ -141,8 +144,7 @@ spec:
             - --stderrthreshold=info
             - --cloud-provider=aws
             - --skip-nodes-with-local-storage=false
-            - --nodes=0:5:<AUTOSCALING GROUP NAME 4vCPUS 16GB RAM>
-            - --nodes=0:5:<AUTOSCALING GROUP NAME 8vCPUS 32GB RAM>
+            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/eksworkshop-eksctl
             - --expander=random
             - --expendable-pods-priority-cutoff=-10
             - --scale-down-unneeded-time=2m0s

content/using_ec2_spot_instances_with_eks/scaling/deploy_ca.md

@@ -7,13 +7,21 @@ weight: 10
 We will start by deploying [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler). Cluster Autoscaler for AWS provides integration with Auto Scaling groups. It enables users to choose from four different options of deployment:
 
 * One Auto Scaling group 
-* **Multiple Auto Scaling groups** - This is what we will use
-* Auto-Discovery
+* Multiple Auto Scaling groups
+* **Auto-Discovery** - This is what we will use
 * Master Node setup
 
-In this workshop we will configure Cluster Autoscaler to scale using the Autoscaling groups associated with the nodegroups that we created in the [Adding Spot Workers with eksctl]({{< ref "/using_ec2_spot_instances_with_eks/spotworkers/workers_eksctl.md" >}}) section.
+In this workshop we will configure Cluster Autoscaler to scale using **[Cluster Autoscaler Auto-Discovery functionality](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md)**. When configured in Auto-Discovery mode on AWS, Cluster Autoscaler will look for Auto Scaling Groups that match a set of pre-set AWS tags. As a convention we use the tags : `k8s.io/cluster-autoscaler/enabled`, and `k8s.io/cluster-autoscaler/eksworkshop-eksctl` .
+
+This will select the two Auto Scaling groups that have been created for Spot instances.
+
+{{% notice note %}}
+The  **[following link](https://console.aws.amazon.com/ec2/autoscaling/home?#AutoScalingGroups:filter=eksctl-eksworkshop-eksctl-nodegroup-dev;view=details)** Should take you to the
+Auto Scaling Group console and select the two spot node-group we have previously created; You should check that
+the tags `k8s.io/cluster-autoscaler/enabled`, and `k8s.io/cluster-autoscaler/eksworkshop-eksctl` are present 
+in both groups. This has been done automatically by **eksctl** upon creation of the groups.
+{{% /notice %}}
 
-### Configure the Cluster Autoscaler (CA)
 We have provided a manifest file to deploy the CA. Copy the commands below into your Cloud9 Terminal. 
 
 ```
@@ -22,41 +30,13 @@ curl -o ~/environment/cluster-autoscaler/cluster_autoscaler.yml https://raw.gith
 sed -i "s/--AWS_REGION--/${AWS_REGION}/g" ~/environment/cluster-autoscaler/cluster_autoscaler.yml
 ```
 
-### Configure the ASG
-We will need to provide the names of the Autoscaling Groups that we want CA to manipulate.  
-
-Your next task is to collect the names of the Auto Scaling Groups (ASGs) containing your Spot worker nodes. Record the names somewhere. We will use this later in the manifest file.
-
-You can find the names in the console by **[following this link](https://console.aws.amazon.com/ec2/autoscaling/home?#AutoScalingGroups:filter=eksctl-eksworkshop-eksctl-nodegroup-dev;view=details)**. 
-
-![ASG](/images/using_ec2_spot_instances_with_eks/scaling/scaling-asg-spot-groups.png)
-
-You will need to save both ASG names for the next section.
-
-### Configure the Cluster Autoscaler
-
-Using the file browser on the left, open **cluster-autoscaler/cluster_autoscaler.yml** and amend the file:
-
- * Search for the block in the file containing this two lines.
- ```
-            - --nodes=0:5:<AUTOSCALING GROUP NAME 4vCPUS 16GB RAM>
-            - --nodes=0:5:<AUTOSCALING GROUP NAME 8vCPUS 32GB RAM>
- ```
-
- * Replace the content **<AUTOSCALING GROUP NAME xVPUS xxGB RAM>** with the actual names of the two nodegroups. The following shows an example configuration.
- ```
-            - --nodes=0:5:eksctl-eksworkshop-eksctl-nodegroup-dev-4vcpu-16gb-spot-NodeGroup-1V6PX51MY0KGP
-            - --nodes=0:5:eksctl-eksworkshop-eksctl-nodegroup-dev-8vcpu-32gb-spot-NodeGroup-S0A0UGWAH5N1
- ```
-
- * **Save** the file
-
-This command contains all of the configuration for the Cluster Autoscaler. Each `--nodes` entry defines a new Autoscaling Group mapping to a Cluster Autoscaler nodegroup. Cluster Autoscaler will consider the nodegroups selected when scaling the cluster. The syntax of the line is minimum nodes **(0)**, max nodes **(5)** and **ASG Name**.
-
-
 ### Deploy the Cluster Autoscaler
 
-Cluster Autoscaler gets deploy like any other pod. In this case we will use the **[kube-system namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)**, similar to what we do with other management pods.
+{{% notice info %}}
+You are encouraged to look at the configuration that you downloaded for cluster autoscaler in the directory `cluster-autoscaler` and find out about some of the parameter we are passing to it. The full list of parameters can be found in **[Cluster Autoscaler documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca)**. 
+{{% /notice %}}
+
+Cluster Autoscaler gets deployed like any other pod. In this case we will use the **[kube-system namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)**, similar to what we do with other management pods.
 
 ```
 kubectl apply -f ~/environment/cluster-autoscaler/cluster_autoscaler.yml