Resolve issue kubernetes-sigs#144

update gofmt Apply suggestions from code review Apply some changes in readme.md and the naming issues according to the code review. Co-authored-by: Abdul Qadeer <[email protected]> change the metricProviderSpec for TargetLoadPackingArgs update gofmt Add example yamls for both library and service client approach Add Operator Field in the WatcherMetrics for both plugin and testing update gofmt add operator in node metrics for integration test update gofmt revert pkg/apis/config/v1beta1/zz_generated.defaults.go Update the comments for MetricProvider
k8stopologyawareschedwg · May 31, 2021 · c506b97 · c506b97
1 parent d2e4544
commit c506b97
Show file tree

Hide file tree

Showing 11 changed files with 403 additions and 73 deletions.
diff --git a/go.mod b/go.mod
@@ -3,11 +3,10 @@ module sigs.k8s.io/scheduler-plugins
 go 1.15
 
 require (
-	github.com/francoispqt/gojay v1.2.13
 	github.com/google/go-cmp v0.5.2
 	github.com/google/uuid v1.1.1
 	github.com/patrickmn/go-cache v2.1.0+incompatible
-	github.com/paypal/load-watcher v0.0.0-20210120084711-f521a95b7915
+	github.com/paypal/load-watcher v0.1.1
 	github.com/spf13/pflag v1.0.5
 	github.com/stretchr/testify v1.5.1
 	k8s.io/api v0.19.8

diff --git a/go.sum b/go.sum
diff --git a/pkg/apis/config/types.go b/pkg/apis/config/types.go
@@ -75,6 +75,25 @@ type CapacitySchedulingArgs struct {
 	KubeConfigPath string
 }
 
+// MetricProviderType is a "string" type.
+type MetricProviderType string
+
+const (
+	KubernetesMetricsServer MetricProviderType = "KubernetesMetricsServer"
+	Prometheus              MetricProviderType = "Prometheus"
+	SignalFx                MetricProviderType = "SignalFx"
+)
+
+// Denote the spec of the metric provider
+type MetricProviderSpec struct {
+	// Types of the metric provider
+	Type MetricProviderType
+	// The address of the metric provider
+	Address string
+	// The authentication token of the metric provider
+	Token string
+}
+
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
 
 // TargetLoadPackingArgs holds arguments used to configure TargetLoadPacking plugin.
@@ -87,6 +106,8 @@ type TargetLoadPackingArgs struct {
 	DefaultRequestsMultiplier string
 	// Node target CPU Utilization for bin packing
 	TargetUtilization int64
+	// Metric Provider to use when using load watcher as a library
+	MetricProvider MetricProviderSpec
 	// Address of load watcher service
 	WatcherAddress string
 }
diff --git a/pkg/apis/config/v1beta1/types.go b/pkg/apis/config/v1beta1/types.go
@@ -75,6 +75,25 @@ type CapacitySchedulingArgs struct {
 	KubeConfigPath *string `json:"kubeConfigPath,omitempty"`
 }
 
+// MetricProviderType is a "string" type.
+type MetricProviderType string
+
+const (
+	KubernetesMetricsServer MetricProviderType = "KubernetesMetricsServer"
+	Prometheus              MetricProviderType = "Prometheus"
+	SignalFx                MetricProviderType = "SignalFx"
+)
+
+// Denote the spec of the metric provider
+type MetricProviderSpec struct {
+	// Types of the metric provider
+	Type MetricProviderType `json:"type,omitempty"`
+	// The address of the metric provider
+	Address *string `json:"address,omitempty"`
+	// The authentication token of the metric provider
+	Token *string `json:"token,omitempty"`
+}
+
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
 // +k8s:defaulter-gen=true
 
@@ -88,6 +107,8 @@ type TargetLoadPackingArgs struct {
 	DefaultRequestsMultiplier *string `json:"defaultRequestsMultiplier,omitempty"`
 	// Node target CPU Utilization for bin packing
 	TargetUtilization *int64 `json:"targetUtilization,omitempty"`
+	// Specify the metric provider type, address and token using MetricProviderSpec
+	MetricProvider MetricProviderSpec `json:"metricProvider,omitempty"`
 	// Address of load watcher service
 	WatcherAddress *string `json:"watcherAddress,omitempty"`
 }
diff --git a/pkg/apis/config/v1beta1/zz_generated.conversion.go b/pkg/apis/config/v1beta1/zz_generated.conversion.go
diff --git a/pkg/apis/config/v1beta1/zz_generated.deepcopy.go b/pkg/apis/config/v1beta1/zz_generated.deepcopy.go
diff --git a/pkg/apis/config/zz_generated.deepcopy.go b/pkg/apis/config/zz_generated.deepcopy.go
diff --git a/pkg/trimaran/targetloadpacking/README.md b/pkg/trimaran/targetloadpacking/README.md
@@ -12,16 +12,64 @@ This folder holds the `TargetLoadPacking` plugin implementation based on [Trimar
 - [ ] 👨 Stable (used in companies for production workloads)
 
 ## TargetLoadPacking Plugin
-`TargetLoadPacking` depends on [Load Watcher](https://github.com/paypal/load-watcher) service. Instructions to build and deploy load watcher can be found [here](https://github.com/paypal/load-watcher/blob/master/README.md).
-`watcherAddress` argument below must be setup for `TargetLoadPacking` to work.
+`TargetLoadPacking` depends on [Load Watcher](https://github.com/paypal/load-watcher).
+It uses `load-watcher` in two modes.
+1. Using `load-watcher` as a service.
+   You can run `load-watcher` service separately to provide real time node resource usage metrics for `TargetLoadPacking` to consume.
+   Instructions to build and deploy load watcher can be found [here](https://github.com/paypal/load-watcher/blob/master/README.md).
+   In this way, you just need to configure `watcherAddress: http://xxxx.svc.cluster.local:2020` to your `load-watcher` service. You can 
+   also deploy `load-watcher` as a service in the same scheduler pod, 
+   following the tutorial [here](https://medium.com/paypal-engineering/real-load-aware-scheduling-in-kubernetes-with-trimaran-a8efe14d51e2).
+
+2. Using `load-watcher` as a library to fetch metrics from other providers, such as Prometheus, SignalFx and Kubernetes metric server.
+   In this mode, you need to configure three parameters: `metricProviderType`, `metricProviderAddress` and `metricProviderToken` if authentication is needed.
+
+   By default, `metricProviderType` is ` KubernetesMetricsServer` if not set. Now it supports `KubernetesMetricsServer`, `Prometheus` and `SignalFx`.
+    - `metricProviderType: KubernetesMetricsServer` use `load-watcher` as a client library to retrieve metrics from Kubernetes metric
+      server.
+    - `metricProviderType: Prometheus` use `load-watcher` as a client library to retrieve metrics from Prometheus directly.
+    - `metricProviderType: SignalFx` use `load-watcher` as a client library to retrieve metrics from SignalFx directly.
+
+   `metricProviderAddress` and `metricProviderToken` should be configured according to `metricProviderType`.
+    - You can ignore `metricProviderAddress` when using `metricProviderType: KubernetesMetricsServer`
+    - Configure the prometheus endpoint for `metricProviderAddress` when using `metricProviderType: Prometheus`.
+      An example could be `http://prometheus-k8s.monitoring.svc.cluster.local:9090`.
 
 Apart from `watcherAddress`, you can configure the following in `TargetLoadPackingArgs`:
 
 1) `targetUtilization` : CPU Utilization % target you would like to achieve in bin packing. It is recommended to keep this value 10 less than what you desire. Default if not specified is 40.
 2) `defaultRequests` : This configures CPU requests for containers without requests or limits i.e. Best Effort QoS. Default is 1 core.
 3) `defaultRequestsMultiplier` : This configures multiplier for containers without limits i.e. Burstable QoS. Default is 1.5
 
-Following is an example config to achieve around 80% CPU utilization, with default CPU requests as 2 cores and requests multiplier as 2:
+The following is an example config to use `load-watcher` as a library to retrieve metrics from pre-installed prometheus, achieve around 80% CPU utilization, with default CPU requests as 2 cores and requests multiplier as 2.
+
+```yaml
+apiVersion: kubescheduler.config.k8s.io/v1beta1
+kind: KubeSchedulerConfiguration
+leaderElection:
+  leaderElect: false
+profiles:
+- schedulerName: trimaran
+  plugins:
+    score:
+      disabled:
+      - name: NodeResourcesBalancedAllocation
+      - name: NodeResourcesLeastAllocated
+      enabled:
+       - name: TargetLoadPacking
+  pluginConfig:
+  - name: TargetLoadPacking
+    args:
+      defaultRequests:
+        cpu: "2000m"
+      defaultRequestsMultiplier: "2"
+      targetUtilization: 70
+      metricProvider: 
+        type: Prometheus
+        address: http://prometheus-k8s.monitoring.svc.cluster.local:9090
+```
+
+Alternatively, you can use the `load-watcher` as a service in the config below.
 
 ```yaml
 apiVersion: kubescheduler.config.k8s.io/v1beta1
@@ -43,6 +91,6 @@ profiles:
       defaultRequests:
         cpu: "2000m"
       defaultRequestsMultiplier: "2"
-      targetUtilization: 70 
+      targetUtilization: 70
       watcherAddress: http://127.0.0.1:2020
 ```