Planning upgrade to EKS 1.28

[poornima-krishnasamy]

Go through the release notes of EKS 1.28 and create a plan to upgrade our clusters

Things to consider:

Review changelog & release notes ✅
EKS Module support at current version? ✅
Are there any API deprecations & removals? (Check insights) ✅
Are there new components being added? ✅
What changes are being introduced to current components? ✅
Are there changes to core infra of the CP required? i.e. Are all our current components compatible with ?
Are there changes users need to make? ✅
Do we need to expand any of our smoke/integration testing? ✅
Create additional tickets needed for any findings specific to this upgrade ✅

Cluster upgrade Runbook:
https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html

Related to: #5569

[jaskaransarkaria]

Deprecations and removals

Removals:

[Removal of CSI Migration for GCE PD](https://github.com/kubernetes/enhancements/issues/1488)

Deprecations:

[Ceph RBD in-tree plugin](https://github.com/kubernetes/kubernetes/pull/118303)
[Ceph FS in-tree plugin](https://github.com/kubernetes/kubernetes/pull/118143)

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#no-really-you-must-read-this-before-you-upgrade

CephFS volume plugin (kubernetes.io/cephfs) has been deprecated in this release and will be removed in a subsequent release. The alternative is to use the CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes cluster. (kubernetes/kubernetes#118143, @humblec)

Deprecated support for CSI migration of Ceph RBD volumes. Users who were relying on Kubernetes' ability to migrate to an out-of-tree storage driver should complete that migration before the support for it is removed. (kubernetes/kubernetes#118303, @carlory)

RBD volume plugin (kubernetes.io/rbd) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (kubernetes/kubernetes#118552, @humblec)

Changes by Kind

Deprecation

• Changed kubectl version default output to be identical to what kubectl version --short printed, and removed --short flag entirely. (Remove long/golang version information making short the default kubernetes/kubernetes#116720, @soltysh)
• Kube-controller-manager deprecate --volume-host-cidr-denylist and --volume-host-allow-local-loopback flags. (remove helper function for unused storage feature in pkg/proxy/util kubernetes/kubernetes#118128, @carlory) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Kubelet: The --azure-container-registry-config flag has been deprecated and will be removed in a future release, please use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. (kubelet: mark '--azure-container-registry-config' flag as deprecated kubernetes/kubernetes#118596, @SataQiu) [SIG Node]
• Removed tracking annotation from validation and defaulting. (remove tracking annotation from validation and webhooks kubernetes/kubernetes#117633, @kannon92)
• Removed withdrawn feature NetworkPolicyStatus. (Remove/Withdraw NetworkPolicy Status kubernetes/kubernetes#115843, @rikatz)
• The deprecated flag --lock-object-namespace and --lock-object-name have been removed from kube-scheduler. Please use --leader-elect-resource-namespace and --leader-elect-resource-name or ComponentConfig instead to configure those parameters. (Remove deprecated scheduler CLI flags kubernetes/kubernetes#119130, @SataQiu) [SIG Scheduling]
• KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In a future release, Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. ([KMSv2] Mark KMS v1beta1 as deprecated with no further fixes kubernetes/kubernetes#119007, @aramase)

[jaskaransarkaria]

What's New

• Changes to supported skew between control plane and node versions:

  • Kubernetes v1.28 expands the supported skew between core node and control plane components by one minor version, from n-2 to n-3, so that node components (kubelet and kube-proxy) for the oldest supported minor version work with control plane components (kube-apiserver, kube-scheduler, kube-controller-manager, cloud-controller-manager) for the newest supported minor version.

• Beta support for enabling swap space on Linux

  • This adds swap support to nodes in a controlled, predictable manner so that Kubernetes users can perform testing and provide data to continue building cluster capabilities on top of swap.

  There are two distinct types of users for swap, who may overlap:

  • Node administrators, who may want swap available for node-level performance tuning and stability/reducing noisy neighbor issues.

  • Application developers, who have written applications that would benefit from using swap memory.

• Source code reorganization for control plane components

New components / features

Graduations to stable

This release includes a total of 12 enhancements promoted to Stable:

• kubectl events
• Retroactive default StorageClass assignment
• Non-graceful node shutdown
• Support 3rd party device monitoring plugins
• Auth API to get self-user attributes
• Proxy Terminating Endpoints
• Expanded DNS Configuration
• Cleaning up IPTables Chain Ownership
• Minimizing iptables-restore input size
• Graduate the kubelet pod resources endpoint to GA
• Extend podresources API to report allocatable resources
• Move EndpointSlice Reconciler into Staging

Notable features

• Graduated the LegacyServiceAccountTokenTracking feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key kubernetes.io/legacy-token-last-used). (graduate LegacyServiceAccountTokenTracking to GA kubernetes/kubernetes#117591, @zshihang) [SIG API Machinery, Auth and Testing]
• Moved non-graceful node shutdown to GA. (move non-graceful node shutdown to GA kubernetes/kubernetes#118228, @carlory)
• Renamed PodHasNetwork to PodReadyToStartContainers. (feat: rename PodHasNetwork to PodReadyToStartContainers kubernetes/kubernetes#117702, @kannon92) [SIG Node and Testing]
• The short names vwc and mwc were introduced for the resources validatingwebhookconfigurations and mutatingwebhookconfigurations. (add shortnames for mutatingwebhookconfigurations and validatingwebhookconfigurations kubernetes/kubernetes#117535, @hysyeah)
• kube-proxy in iptables mode will now have separate sync_full_proxy_rules_duration_seconds\nand sync_partial_proxy_rules_duration_seconds (in addition to the existing\nsync_proxy_rules_duration_seconds), giving better information about the duration of each \nsync type, rather than only giving a weighted average of the two sync types together. (Add new partial/full sync time metrics for iptables kube-proxy kubernetes/kubernetes#117787, @danwinship)
• [Kube-proxy]: Implemented connection draining for terminating nodes. ([Kube-proxy]: Implement KEP-3836 kubernetes/kubernetes#116470, @alexanderConstantinescu)
• Added the implementation for PodRecreationPolicy to wait for the creation of pods once the existing ones are fully terminated. (Job: create replacement pods only after terminated kubernetes/kubernetes#117015, @kannon92)
• Added a container image for kubectl at registry.k8s.io/kubectl across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) (Add an image for kubectl kubernetes/kubernetes#116672, @dims) [SIG Architecture and Release]
• Added a new command line argument --interactive to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. (kubectl delete: Introduce new interactive flag for interactive deletion kubernetes/kubernetes#114530, @ardaguclu) [SIG CLI and Testing]
• Added '--concurrent-cron-job-syncs' flag for kube-controller-manager to set the number of workers for cron job controller. (add --concurrent-cron-job-syncs flag for kube-controller-manager kubernetes/kubernetes#117550, @borgerli)
• Added '--concurrent-job-syncs' flag for kube-controller-manager to set the number of job controller workers. (Add --concurrent-job-syncs flag to kube-controller-manager kubernetes/kubernetes#117138, @tosi3k)
• Added DisruptionTarget condition to the pod preempted by kubelet to make room for a critical pod. (Add DisruptionTarget condition when preempting for critical pod kubernetes/kubernetes#117586, @mimowo)

Other features

• A ValidatingAdmissionPolicy now has its messageExpression field checked against resolved types. (ValidatingAdmissionPolicy: expended type checking to messageExpression kubernetes/kubernetes#119209, @jiahuif) [SIG API Machinery]

• Added --concurrency flag to configure the concurrency of kubectl diff execution, defaults to 1. (kubectl: Allow parallelizing diffing kubernetes/kubernetes#118810, @brancz)

• Added ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache. (Implement Alpha state for KEP #2340 kubernetes/kubernetes#118508, @serathius)

• Added apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics. (matchCondition metrics for beta graduation kubernetes/kubernetes#119311, @ivelichkovich)

• Added a new feature gate, SchedulerQueueingHints (enabled by default). The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins. In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. (implement SchedulerQueueingHints feature gate kubernetes/kubernetes#119328, @sanposhiho) [SIG Scheduling]

• Added full cgroup v2 swap support for both Limited and Unlimited swap.

• When LimitedSwap is enabled the swap limit would be automatically calculated for Burstable QoS pods. For Best-Effort/Guaranteed QoS pods, swap would be disabled.

• Containers with memory requests equal to their memory limits also won't have swap access, and it is a way to opt-out of swap for a single container.

• The formula for the swap limit for Burstable QoS pods is: (/)*.

• Support for cgroup v1 is removed. ([KEP-2400] Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes/kubernetes#118764, @iholder101)

• Added handling for pods in podgc for PodReplacementPolicy or PodDisruption. (KEP-3939: pod gc changes for pod replacement policy kep kubernetes/kubernetes#118772, @kannon92)

• Added reason to metric attachdetach_controller_forced_detaches in the attach detach controller. (Add reason to force detach metric kubernetes/kubernetes#119185, @xing-yang)

• Added support for pod hostNetwork field selector (support fieldSelector spec.hostNetwork kubernetes/kubernetes#110477, @halfcrazy) [SIG Apps and Node]

• Added swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). ([KEP-2400] Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes/kubernetes#118865, @iholder101)

• Allow to monitor client-go DNS resolver latencies via rest_client_dns_resolution_duration_seconds Prometheus metric. (client-go: add DNS resolver latency metrics kubernetes/kubernetes#115357, @mfojtik)

• Apiserver adds two new metrics etcd_requests_total and etcd_request_errors_total that allow users to monitor requests to etcd storage, split by operation and resource type. (apiserver/etcd: add request and request_error metrics kubernetes/kubernetes#117222, @iyear) [SIG API Machinery]

• Bumped distroless-iptables to 0.2.6 based on Go 1.20.6. (Bump distroless-iptables to 0.2.6 kubernetes/kubernetes#119365, @xmudrii)

• Bumped metrics-server to v0.6.3. (Bump metrics-server to v0.6.3 kubernetes/kubernetes#117120, @dgrisonnet)

• CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". (CEL lib: Expose errors on authz decisions instead of raising them from check() kubernetes/kubernetes#118804, @benluddy) [SIG API Machinery]

• CRI: exposed commit memory bytes in container stats specific to Windows (Expose commit memory used in WindowsMemoryUsage struct kubernetes/kubernetes#119238, @kiashok)

• Client-go now exposes two new metrics to monitor the client-go logic that generate http.Transports for the clients.

  • rest_client_transport_cache_entries is a gauge metric with the number of existing entries in the internal cache
  • rest_client_transport_create_calls_total is a counter that increments each time a new transport is created, storing the result of the operation needed to generate it: hit, miss or uncacheable. (add new metric for the internal client-go cache size kubernetes/kubernetes#117295, @aojea)

• Cloud controller manager's node controller now emits timing metrics for initial Node synchronization. These metrics measure the delay between the creation of a new Node and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's --concurrent-node-syncs flag. (Add metrics for cloud controller manager initial node sync kubernetes/kubernetes#119241, @cartermckinnon) [SIG Cloud Provider and Instrumentation]

• Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation." (DRA: for delayed allocation, deallocate when no longer used kubernetes/kubernetes#118936, @pohly)

• Enabled use of pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport. (apis: drop check for volumes with user namespaces kubernetes/kubernetes#118691, @giuseppe)

• External credential provider plugins will now have their standard error output logged by kubelet upon failures. (Include stderr in cred provider plugin errors kubernetes/kubernetes#117448, @cartermckinnon)

• Faster scheduling when ResourceClaims are involved. (dra: scheduler queueing hints kubernetes/kubernetes#119078, @pohly)

• Fixed the alpha CloudDualStackNodeIPs feature. (Set the node-ips annotation correctly with CloudDualStackNodeIPs kubernetes/kubernetes#118329, @danwinship)

• Graduated the ProbeTerminationGracePeriod feature gate to GA. (ProbeTerminationGracePeriod promote to GA kubernetes/kubernetes#114307, @rphillips)

• Hashing of KeyID in Logs

  • This release adds a feature to hash the KeyID values in the logs. The KeyID values are sensitive information that should not be exposed in plain text in the logs. By hashing the KeyID values, we can protect the confidentiality of the data while still being able to log the necessary information. ([KMSv2] chore: hashes keyID being logged kubernetes/kubernetes#118988, @nilekhc) [SIG API Machinery, Auth and Testing]

• Implemented alpha support for a drop-in kubelet configuration directory. (cmd/kubelet: implement drop-in configuration directory for kubelet kubernetes/kubernetes#119390, @sohankunkerkar)

• In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. (Cache authz decisions within the scope of validating policy admission. kubernetes/kubernetes#116443, @benluddy) [SIG API Machinery and Testing]

• Introduce support for CEL optionals (see CEL spec proposal 246). This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable safe rollback on downgrade. (Bump cel go to latest version kubernetes/kubernetes#118339, @jpbetz) [SIG API Machinery, Auth, Cloud Provider and Testing]

• Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. (dra: pre-scheduled pods kubernetes/kubernetes#118209, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]

• Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim (promote ProxyTerminatingEndpoints to GA kubernetes/kubernetes#117718, @aojea) [SIG Network, Testing and Windows]

• Kube-proxy service health returns http header X-Load-Balancing-Endpoint-Weight with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. (kube-proxy service health: add new header with number of local endpoints kubernetes/kubernetes#118999, @cezarygerard)

• Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. (kubelet: support batched prepare/unprepare in v1alpha3 DRA plugin API kubernetes/kubernetes#119012, @pohly)

• Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. (node authorizer: limit kubelet access to ResourceClaim objects kubernetes/kubernetes#116254, @pohly) [SIG Auth and Testing]

• Kubelet: un-deprecated --provider-id flag. (undeprecate kubelet --provider-id flag kubernetes/kubernetes#116530, @pacoxu)

• Kubernetes is now built with Go 1.20.4. ([go] Bump images, dependencies and versions to go 1.20.4 kubernetes/kubernetes#117744, @xmudrii) [SIG Release and Testing]

• Kubernetes is now built with Go 1.20.5. ([go] Bump images, versions and deps to use Go 1.20.5 kubernetes/kubernetes#118507, @jeremyrickard)

• Kubernetes is now built with Go 1.20.6. ([go] Bump images, versions and deps to use Go 1.20.6 kubernetes/kubernetes#119324, @xmudrii)

• Metric scheduler_scheduler_goroutines is removed. Use scheduler_goroutines instead. (Remove old metric scheduler_goroutines kubernetes/kubernetes#117727, @kerthcet) [SIG Scheduling]

• Migrated pkg/controller/endpoint to contextual logging. (Migrated pkg/controller/endpoint to contextual logging kubernetes/kubernetes#116755, @my-git9)

• Migrated pkg/scheduler/framework/preemption to use contextual logging. (Migrated pkg/scheduler/framework/preemption & defaultpreemption to use contextual logging kubernetes/kubernetes#116835, @mengjiao-liu)

• Migrated pod-security-admission to use contextual logging. (Migrated pod-security-admission to contextual logging kubernetes/kubernetes#114471, @Namanl2001) [SIG Apps and Auth]

• Migrated controller functions to use contextual logging. (contextual logging cleanup kubernetes/kubernetes#116930, @fatsheep9146) [SIG API Machinery, Apps, Network, Node, Storage and Testing]

• Migrated the Job controller (within kube-controller-manager) to use contextual logging. (Migrated pkg/controller/job to contextual logging kubernetes/kubernetes#116910, @fatsheep9146) [SIG API Machinery, Apps and Testing]

• Migrated the EndpointSlice and EndpointSliceMirroring controllers (within kube-controller-manager) to use contextual logging. (Migrated pkg/controller/endpointslice and pkg/controller/endpointslicemirroring to contextual logging kubernetes/kubernetes#115295, @Namanl2001) [SIG API Machinery, Apps, Network and Testing]

• Migrated the certificate controller (within kube-controller-manager) to use contextual logging. (certificate controller: use contextual logging kubernetes/kubernetes#113994, @mengjiao-liu) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]

• Migrated the noderesources scheduler plugin to use contextual logging. (Migrated pkg/scheduler/framework/plugins/noderesources to contextual logging kubernetes/kubernetes#116748, @mengjiao-liu)

• Migrated the podtopologyspread scheduler plugins to use contextual logging. (Migrated pkg/scheduler/framework/plugins/podtopologyspread to contextual logging kubernetes/kubernetes#116797, @mengjiao-liu) [SIG Instrumentation and Scheduling]

• New CEL Library functions to support Kubernetes Quantities. (Add quantity library to CEL kubernetes/kubernetes#118803, @alexzielenski) [SIG API Machinery]

• New Metrics Added for Encryption Configuration Controller

• This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:

• apiserver_encryption_config_controller_automatic_reload_failures_total: Total number of failed automatic reloads of encryption configuration.

• apiserver_encryption_config_controller_automatic_reload_success_total: Total number of successful automatic reloads of encryption configuration.

• apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds: Timestamp of the last successful or failed automatic reload of encryption configuration.

• These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. ([KMSv2] feat: implements metrics for encryption config hot reload kubernetes/kubernetes#119008, @nilekhc)

• New staging repo has been created for the EndpointSlice reconciler. (Convert EndpointSlice Reconciler to a library in staging. kubernetes/kubernetes#118953, @mskrocki)

• Promoted ServiceNodePortStaticSubrange feature gate to beta, and it will be enabled by default. (KEP-3668: promote ServiceNodePortStaticSubrange to beta kubernetes/kubernetes#117877, @xuzhenglun)

• Promoted the following apiserver flowcontrol metrics to Beta:

• apiserver_flowcontrol_request_wait_duration_seconds

• apiserver_flowcontrol_current_executing_seats

• apiserver_flowcontrol_nominal_limit_seats

• apiserver_flowcontrol_rejected_requests_total

• apiserver_flowcontrol_dispatched_requests_total

• apiserver_flowcontrol_current_inqueue_requests

• apiserver_flowcontrol_current_executing_requests (Promote kube-apiserver flowcontrol metrics to Beta kubernetes/kubernetes#119110, @andrewsykim)

• Replaced apiserver_storage_db_total_size_in_bytes with apiserver_storage_size_bytes metric. (Improve apiserver storage size metric kubernetes/kubernetes#118812, @serathius)

• Scheduler now waits for handlers to finish syncing before the scheduling cycles start. ([Scheduler] Make sure handlers have synced before scheduling kubernetes/kubernetes#116729, @AxeZhan)

• Set metrics-server's metric-resolution to 15s. (Set metrics-server's metric-resolution to 15s kubernetes/kubernetes#117121, @dgrisonnet) [SIG Cloud Provider and Instrumentation]

• SubjectAccessReview requests sent to webhook authorizers now default spec.resourceAttributes.version to * if unset. (Ensure version "*" is passed instead of "" for all authz checks kubernetes/kubernetes#116937, @AxeZhan) [SIG Apps and Auth]

• Supported specifying a custom retry period for cloud load-balancer operations. (Support specifying custom LB retry period from cloud provider kubernetes/kubernetes#94021, @timoreimann)

• The "value" part in the wait --for=jsonpath='{expression}'[=value] is now optional. If the value is not provided i.e., the command looks like wait --for=jsonpath='{expression}' then the wait condition is interpreted as matched when the expression returns any single JSON value like object or a literal. (Support JSONPath condition without value kubernetes/kubernetes#118160, @minherz)

• The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. (feature(pkg/api): warning for Pod with null labelSelector in PodAffinity and TopologySpread kubernetes/kubernetes#117025, @sanposhiho) [SIG Scheduling]

• The AdvancedAuditing feature gate that graduated to GA in v1.12 (and was unconditionally enabled) has been removed. (Remove AdvancedAuditing feature gate. kubernetes/kubernetes#118763, @Shubham82)

• The ExpandedDNSConfig feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. (Promote ExpandedDNSConfig feature to the GA stage kubernetes/kubernetes#116741, @gjkim42) [SIG Apps, Network and Node]

• The apiserver debug endpoint /debug/api_priority_and_fairness/dump_requests has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. (Track executing requests kubernetes/kubernetes#119009, @MikeSpreitzer) [SIG API Machinery]

• The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of kubectl --help ...

• Changed kubectl create secret --help description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. (kubectl: optimize usage message of commands which have subcommands. kubernetes/kubernetes#117930, @LronDC)

• The scheduler skips the InterPodAffinity Score plugin when nothing to do with the Pod. It will affect some metrics values related to the InterPodAffinity Score plugin. (Return Skip in InterPodAffinity#PreScore under specific conditions kubernetes/kubernetes#117794, @utam0k) [SIG Scheduling]

• The scheduler skips the PodTopologySpread Filter plugin if no spread constraints. It will affect some metrics values related to the PodTopologySpread Filter plugin. (Add check to skip PodTopologySpread PreFilter if no constraints are specified kubernetes/kubernetes#117683, @utam0k)

• The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod. It will affect some metrics values related to the PodTopologySpread Score plugin. (Return Skip in PodTopologySpread#PreScore under specific conditions kubernetes/kubernetes#118608, @utam0k)

• Updated etcd image to 3.5.9-0. (etcd: Build image 3.5.9-0 kubernetes/kubernetes#117999, @kkkkun) [SIG API Machinery]

• Updated cAdvisor to v0.47.2 and fixed metrics in cri-o when a container restarts. (deps: Bump to cAdvisor v0.47.2 kubernetes/kubernetes#118774, @harche)

• Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 (bump distroless-iptables to v0.2.5 kubernetes/kubernetes#118541, @jeremyrickard) [SIG Testing]

• Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.4 (Update distroless-iptables to v0.2.4 kubernetes/kubernetes#117746, @xmudrii) [SIG Testing]

• Updated the scheduler interface and cache methods to use contextual logging. (scheduler: update the scheduler interface and cache methods to use contextual logging kubernetes/kubernetes#116849, @mengjiao-liu)

• ValidatingAdmissionPolicy type checking now correctly handles authorizer variable. (add support for authorizer to type checking. kubernetes/kubernetes#118540, @jiahuif) [SIG API Machinery]

• When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. (DRA: improve handling of completed pods kubernetes/kubernetes#118817, @pohly)

• With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating the need to specify the cgroupDriver configuration option (or --cgroup-driver flag) of kubelet. (kubelet: get cgroup driver config from CRI kubernetes/kubernetes#118770, @marquiz)

• --version=v1.X.Y... can now be used to set the prerelease and buildID portions of the version reported by components (Allow override of prerelease/buildID portions of version at runtime kubernetes/kubernetes#117688, @liggitt) [SIG API Machinery, Architecture and Release]

• RetroactiveDefaultStorageClass feature made stable and enabled by default. (graduate RetroactiveDefaultStorageClass feature to GA in 1.28 kubernetes/kubernetes#118102, @RomanBednar)

• TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. (topologymanager: Promote support for improved multi-numa alignment in Topology Manager to beta kubernetes/kubernetes#118816, @PiotrProkop)

• force_delete_pods_total and force_delete_pod_errors_total metrics count all pod deletion behaviors. (podgc metrics should count all pod deletion behaviors kubernetes/kubernetes#118480, @carlory)

• klog text output now uses JSON as encoding for structs, maps and slices. (dependencies: klog v2.100.1 kubernetes/kubernetes#117687, @pohly)

• kubeadm: added a new "kubeadm config validate" command that can be used to validate any input config file. Use the --config flag to pass a config file to it. See the command --help screen for more information. As a result of adding this new command, enhance the validation capabilities of the existing "kubeadm config migrate" command. For both commands unknown APIs or fields will throw errors. (kubeadm: add the "config validate" subcommand kubernetes/kubernetes#118013, @neolit123)

• kubeadm: added the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP/experimental APIs in the future. (kubeadm: add v1beta4 to scheme; add --allow-experimental-api flag kubernetes/kubernetes#118866, @neolit123)

• kubeadm: generate CA certificates with a start time that is offset 5 minutes in the past relative to the current system time to workaround cases of clock desync. (kubeadm: backdate generated CAs kubernetes/kubernetes#118922, @champtar)

• plugin_evaluation_total metric supports prescore/score extension point. The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming pod. (feature(scheduler): plugin_evaluation_total metric support preScore/score kubernetes/kubernetes#118025, @AxeZhan)

[jaskaransarkaria]

API Changes

Notable changes

• ACTION_REQUIRED When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. (Add warnings for big number of completions and parallelism kubernetes/kubernetes#118420, @alculquicondor) [SIG Apps]
• Added IP mode field to load balancer status ingress. (Make Kubernetes aware of the LoadBalancer behaviour kubernetes/kubernetes#118895, @RyanAoh)
• Added new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (feat: Append job creation timestamp to cronjob annotations kubernetes/kubernetes#118137, @helayoty)
• Graduated AdmissionWebhookMatchCondition feature to beta. (Graduate AdmissionWebhookMatchCondition to beta kubernetes/kubernetes#119380, @a-hilaly)
• If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via memory.oom.group . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. (use the cgroup aware OOM killer if available kubernetes/kubernetes#117793, @tzneal) [SIG Apps, Node and Testing]
• Indexed Job pods now have the pod completion index set as a pod label. (Add completion index as pod label for indexed jobs kubernetes/kubernetes#118883, @danielvegamyhre) [SIG Apps]
• NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. (feat(NodeVolumeLimits): return Skip in PreFilter kubernetes/kubernetes#115398, @tangwz) [SIG Scheduling]
• PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. (PersistentVolume last phase transition time kubernetes/kubernetes#116469, @RomanBednar)
• Pods which set hostNetwork: true and declare ports, get the hostPort field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now hostPort will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. (Do hostNet Pod-ports -> hostPorts in Pod defaults kubernetes/kubernetes#117696, @thockin) [SIG Apps]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (Add StatefulSet pod index as pod label kubernetes/kubernetes#119232, @danielvegamyhre) [SIG Apps]
• • Supported BackoffLimitPerIndex in Jobs. (Support BackoffLimitPerIndex in Jobs kubernetes/kubernetes#118009, @mimowo)
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer creates the KUBE-MARK-DROP chain (which has been unused for several releases) or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (move KEP-3178 IPTablesOwnershipCleanup to GA kubernetes/kubernetes#119374, @danwinship)
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (Add SidecarContainers feature kubernetes/kubernetes#116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Updated the comment about the feature-gate level for PodFailurePolicy from alpha to beta (Update podFailurePolicy comment from alpha-level to beta kubernetes/kubernetes#118278, @mimowo)
• kube-controller-manager: The LegacyServiceAccountTokenCleanUp feature gate is now available as alpha (off by default). When enabled, the legacy-service-account-token-cleaner controller loop removes service account token secrets that have not been used in the time specified by --legacy-service-account-token-clean-up-period (defaulting to one year), and are referenced from the .secrets list of a ServiceAccount object, and are not referenced from pods. (LegacyServiceAccountTokenCleanUp alpha kubernetes/kubernetes#115554, @yt2985)
• kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (scheduler: remove deprecated v1beta2 KubeSchedulerConfiguration component config kubernetes/kubernetes#117649, @SataQiu)
• Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta. (Update podFailurePolicy comments kubernetes/kubernetes#117802, @kerthcet) [SIG API Machinery and Apps]

Other changes

• A CDIDevice field is included in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (Add CDI devices to device plugin API kubernetes/kubernetes#118254, @elezar) [SIG Node and Testing]

• Added ServedVersions field to StorageVersion API. (Add servedVersions info in StorageVersion API kubernetes/kubernetes#118386, @Richabanker)

• Added podReplacementPolicy and terminating field to job api. (add podReplacementPolicy and terminating field to job api kubernetes/kubernetes#119301, @kannon92)

• Added a new namespaceParamRef field to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy. (KEP-3488: Per namespace policy params kubernetes/kubernetes#119215, @alexzielenski) [SIG API Machinery and Testing]

• Added a warning that TLS 1.3 ciphers are not configurable. (Add note about TLS 1.3 cipher suites kubernetes/kubernetes#115399, @3u13r) [SIG API Machinery and Node]

• Added error handling for seccomp localhost configurations that do not properly set a localhostProfile. (Fix seccomp localhost error handling kubernetes/kubernetes#117020, @cji)

• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. ([KEP-2876]Add reason and fieldPath into CRD validation rules kubernetes/kubernetes#118041, @cici37) [SIG API Machinery]

• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject variable with expressions. (ValidatingAdmissionPolicy: support namespace access kubernetes/kubernetes#118267, @cici37) [SIG API Machinery and Testing]

• Added new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (KEP-4008: CRD Validation Ratcheting alpha implementation kubernetes/kubernetes#118990, @alexzielenski)

• Added new config option delayCacheUntilActive to KubeSchedulerConfiguration that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in kube-scheduler (Start informers after leader election kubernetes/kubernetes#115754, @linxiulei) [SIG API Machinery and Scheduling]

• Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the KMSv2KDF feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. (kmsv2: KDF based nonce extension kubernetes/kubernetes#118828, @enj)

• Exposed rest.DefaultServerUrlFor function. (Export DefaultServerUrlFor utility function kubernetes/kubernetes#118055, @timofurrer)

• Extended the Job API for alpha version of BackoffLimitPerIndex. (Extend the Job API for BackoffLimitPerIndex kubernetes/kubernetes#119294, @mimowo)

• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (Exempt borrowing implementation kubernetes/kubernetes#118782, @MikeSpreitzer) [SIG API Machinery]

• Kube-proxy: added --logging-format flag to support structured logging. (Add '--logging-format' flag to kube-proxy kubernetes/kubernetes#117800, @cyclinder)

• Promoted API groups ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding to v1beta1. (KEP-3488: Promote ValidatingAdmissionPolicy to Beta kubernetes/kubernetes#118644, @alexzielenski) [SIG API Machinery, Apps and Testing]

• Promoted the feature gate ValidtaingAdmissionPolicy to beta, and it is turned off by default. (Add test cases for ValidatingAdmissionPolicy kubernetes/kubernetes#119409, @alexzielenski)

• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (promote beta metrics kubernetes/kubernetes#119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]

• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus. (Update api recovery apis kubernetes/kubernetes#116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]

• Removed WindowsHostProcessContainers feature-gate. (Removing WindowsHostProcessContainers feature-gate kubernetes/kubernetes#117570, @marosset) [SIG API Machinery, Apps, Auth, Node and Windows]

• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ( Unknown Version Interoperability Proxy Impl kubernetes/kubernetes#117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]

• The SelfSubjectReview API is promoted to authentication.k8s.io/v1 and the kubectl auth whoami command is GA. (KEP-3325: Promote SelfSubjectReview to GA kubernetes/kubernetes#117713, @nabokihms) [SIG API Machinery, Architecture, Auth, CLI and Testing]

• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still -, but a random suffix will avoid name collisions. (DRA: generated resource claim names kubernetes/kubernetes#117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently. (Faster ExtractList. Add ExtractListWithAlloc variant. kubernetes/kubernetes#113362, @sxllwx)

• component-base/logs is now stricter about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. (component-base/logs: improve handling of re-applying a configuration kubernetes/kubernetes#117108, @pohly)

[jaskaransarkaria]

kubent output

→  kubent --target-version 1.28
11:01AM INF >>> Kube No Trouble `kubent` <<<
11:01AM INF version 0.7.2 (git sha 25eb8a3757d1db39a04e94bb97a3f099fb5c9fb6)
11:01AM INF Initializing collectors and retrieving data
11:01AM INF Target K8s version is 1.28.0
11:01AM INF Retrieved 5643 resources from collector name=Cluster
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for autoscaling/v2beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for autoscaling/v2beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:09AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:12AM WRN failed to discover supported resources for networking.k8s.io/v1beta1: the server could not find the requested resource
11:12AM WRN failed to discover supported resources for networking.k8s.io/v1beta1: the server could not find the requested resource
11:21AM WRN failed to discover supported resources for batch/v1beta1: the server could not find the requested resource
11:30AM INF Retrieved 9890 resources from collector name="Helm v3"
11:30AM INF Loaded ruleset name=custom.rego.tmpl
11:30AM INF Loaded ruleset name=deprecated-1-16.rego
11:30AM INF Loaded ruleset name=deprecated-1-22.rego
11:30AM INF Loaded ruleset name=deprecated-1-25.rego
11:30AM INF Loaded ruleset name=deprecated-1-26.rego
11:30AM INF Loaded ruleset name=deprecated-1-27.rego
11:30AM INF Loaded ruleset name=deprecated-1-29.rego
11:30AM INF Loaded ruleset name=deprecated-future.rego
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.22 <<<
------------------------------------------------------------------------------------------
KIND      NAMESPACE                       NAME                                        API_VERSION                 REPLACE_WITH (SINCE)
Ingress   <undefined>                     hmpps-interventions-onboarding              networking.k8s.io/v1beta1   networking.k8s.io/v1 (1.19.0)
Ingress   <undefined>                     hmpps-delius-interventions-event-listener   networking.k8s.io/v1beta1   networking.k8s.io/v1 (1.19.0)
Ingress   polygraph-offender-management   poms-ingress                                networking.k8s.io/v1beta1   networking.k8s.io/v1 (1.19.0)
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.25 <<<
------------------------------------------------------------------------------------------
KIND                      NAMESPACE                     NAME                                                              API_VERSION           REPLACE_WITH (SINCE)
PodDisruptionBudget       <undefined>                   hmpps-community-accommodation-wiremock                            policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   create-and-vary-a-licence-wiremock-hmpps-community-api-wiremock   policy/v1beta1        policy/v1 (1.21.0)
HorizontalPodAutoscaler   <undefined>                   court-list-splitter                                               autoscaling/v2beta1   autoscaling/v2 (1.23.0)
HorizontalPodAutoscaler   <undefined>                   court-hearing-event-receiver                                      autoscaling/v2beta1   autoscaling/v2 (1.23.0)
CronJob                   c100-application-production   c100-application-cronjob-production                               batch/v1beta1         batch/v1 (1.21.0)
CronJob                   c100-application-production   c100-application-cronjob-payments-production                      batch/v1beta1         batch/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   pre-sentence-service-gotenberg                                    policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   pre-sentence-service-wproofreader                                 policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   pre-sentence-service                                              policy/v1beta1        policy/v1 (1.21.0)
CronJob                   <undefined>                   dlq-transfer-cronjob                                              batch/v1beta1         batch/v1 (1.21.0)
PodDisruptionBudget       c100-application-production   c100-application-pdb-production                                   policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   court-list-splitter                                               policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   court-hearing-event-receiver                                      policy/v1beta1        policy/v1 (1.21.0)
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.26 <<<
------------------------------------------------------------------------------------------