[ENG-2729] Savings Recommendations: Node group right-sizing & allow lists

using-kubecost/navigating-the-kubecost-ui/savings/cluster-right-sizing-recommendations.md

@@ -1,14 +1,18 @@
 # Cluster Right-Sizing Recommendations
 
-Kubecost can provide and implement recommendations for right-sizing your supported clusters to ensure they are configured in the most cost-effective way. Recommendations are available for any and all clusters. Kubecost in certain configurations is also capable of taking a recommendation and applying it directly to your cluster in one moment. These two processes should be distinguished respectively as viewing cluster recommendations vs. adopting cluster recommendations.
+Kubecost can provide and implement recommendations for right-sizing your supported clusters to ensure they are configured in the most cost-effective way. Recommendations are available for any and all clusters. In certain configurations, Kubecost is also capable of taking a recommendation and applying it directly to your cluster in one moment. These two processes should be distinguished respectively as viewing cluster recommendations vs. adopting cluster recommendations.
 
 Kubecost is also able to implement cluster sizing recommendations on a user-scheduled interval, known as continuous cluster right-sizing.
 
 ## Viewing cluster right-sizing recommendations
 
-You can access cluster right-sizing by selecting _Savings_ in the left navigation, then select the _Right-size your cluster nodes_ panel.
+You can access cluster right-sizing by selecting _Savings_ in the left navigation, then select the _Right-size your cluster nodes_ panel. 
 
-Kubecost will offer two recommendations: simple (uses one node type) and complex (uses two or more node types). Kubecost may hide the complex recommendation when it is more expensive than the simple recommendation, and present a single recommendation instead. These recommendations and their metrics will be displayed in a chart next to your existing configuration in order to compare values like total cost, node count, and usage.
+You can choose to view recommendations for the cluster as a whole, or for each individual node group in the cluster.
+
+For the cluster as a whole, Kubecost will offer two recommendations: simple (returns one node group) and complex (returns two or more node groups). Kubecost may hide the complex recommendation when it is more expensive than the simple recommendation, and present a single recommendation instead. For individual node groups, Kubecost will always provide a recommendation with one node group.  
+
+These recommendations and their metrics will be displayed in a chart next to your existing configuration in order to compare values like total cost, node count, and usage.
 
 ### Configuring your cluster right-sizing recommendations
 
@@ -17,6 +21,7 @@ Kubecost provides its right-sizing recommendations based on the characteristics
 There are multiple dropdown menus to consider:
 
 * In the Cluster dropdown, you can select the individual cluster you wish to apply right-sizing recommendations to.
+* (Optional) In the Node Group dropdown, you can select the individual node group you wish to apply right-sizing recommendations to.
 * In the Window dropdown, select the number of days to query for your cluster's most recent activity. Options range from 1 day to 7 days. If your cluster has varying performance on different days of the week, it's better to select a longer interval for the most consistent recommendations.
 
 You can toggle on _Show optimization inputs_ to view resources which will determine the minimum size of your nodes. These resources are:
@@ -81,3 +86,42 @@ If you are using Persistent Volumes (PVs) with AWS's Elastic Block Store (EBS) C
 Kubecost mitigates this problem by ensuring continuous cluster right-sizing creates at least one node per AZ by forcing NodeGroups to have a node count greater than or equal to the number of AZs of the EKS cluster. This will also prevent you from setting a minimum node count for your recommendation below the number of AZs for your cluster. If the EBS CSI continues to be problematic, you can consider switching your CSI to services like Elastic File System (EFS) or FSx for Lustre.
 
 Using Cluster Autoscaler on AWS may result in a similar error. See more [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#common-notes-and-gotchas).
+
+## Instance Selection 
+
+Cluster right-sizing recommendations are generated by simulating placing your cluster workload requirements on multiple different instance type configurations and choosing the most resource- and cost-optimal configuration. Kubecost queries public pricing APIs of the relevant cloud provider to build and maintain a ‘general pool’ of instance type options that the cluster-sizing algorithm can choose from.
+Kubecost also supports limiting the instance types that the cluster-sizing algorithm will consider so that your organization can generate recommendations using the instance types most relevant to your use case.
+
+### Configuration
+
+You can supply your desired list of node types in an allow list. Kubecost will only consider the specified instance types when generating recommendations. We support allow lists for AWS, GCP and Azure instance types.
+Each of the allow lists can be configured by setting the related [Helm chart parameters](https://github.com/kubecost/cost-analyzer-helm-chart/blob/e98d38e7e99a329719e6c814560b71ee83cc7b82/cost-analyzer/values.yaml#L3451-L3454). For example:
+
+```yaml
+kubecostProductConfigs:
+  ...
+  savingsRecommendationsAllowLists:
+    AWS:
+      - t3.2xlarge
+      - t3.xlarge
+      - t3.large
+      - t3.medium
+      - t3.small
+      - t3.micro
+      - t3.nano
+      - ...
+    GCP: 
+      - e2-standard-2
+      - ...
+    Azure:
+      - B1ms
+      - ...
+```
+
+### Supported instance types
+
+The complete lists of supported instance types currently available for each of the supported cloud service providers (AWS, GCP, Azure) can be found in the Helm chart:
+- [AWS](https://github.com/kubecost/cost-analyzer-helm-chart/blob/develop/cost-analyzer/savings-recommendations-allow-lists-aws.yaml)
+- [Azure](https://github.com/kubecost/cost-analyzer-helm-chart/blob/develop/cost-analyzer/savings-recommendations-allow-lists-azure.yaml)
+- [GCP](https://github.com/kubecost/cost-analyzer-helm-chart/blob/develop/cost-analyzer/savings-recommendations-allow-lists-gcp.yaml)
+