CVE-2022-0185: heap overflow bug in legacy_parse_param()

[miwithro]

William Liu and Jamie Hill-Daniel discovered that the file system context functionality in the Linux kernel contained an integer underflow vulnerability, leading to an out-of-bounds write. A local attacker could use this to cause a denial of service (system crash) or execute arbitrary code.

Mitigation

AKS has applied the patch to node image version 2022.01.24

AKS Information:

Upgrade your node image to 2022.01.24.

AKS
-- | --

[zohebs341]

Scanning Report:

[control: CVE-2022-0185-linux-kernel-container-escape - https://hub.armo.cloud/docs/c-0079] failed
Description: CVE-2022-0185 is a kernel vulnerablity enabling privilege escalation and it can lead attackers to escape containers and take control over nodes. This control alerts on vulnerable kernel versions of Kubernetes nodes
Failed:
Node - aks-agentpool-22415207-vmss000003
Node - aks-agentpool-22415207-vmss000004
Node - aks-agentpool-22415207-vmss000005

Environment We are using:

Azure AKS 1.21.7 version OS Image- 18.04.6 TLS
Kernel-version: 5.4.0-1065-azure

[miwithro]

this is remediated in

Kernel-version:5.4.0-1067-azure which we are working to deploy in AKS.

[zohebs341]

@miwithro Thanks for sharing details.

Do we need to upgrade AKS cluster from 1.21.7 to 1.22.X ? Or It will be only AKS Nodes/Image upgrade in K8S 1.21.7

Appreciate your response

[miwithro]

No this will not require an AKS upgrade but node image upgrade. Once we have the exact VHD details ready I will update the issue, so everyone knows which node image to upgrade too.

[zohebs341]

@miwithro Thanks for your time and updates.

[davidrsnd]

Until the patched image is release I was able to deploy the mitigation "sysctl -w kernel.unprivileged_userns_clone=0" on all current and new created nodes through a daemonset.

Example:
https://github.com/davidrsnd/public/blob/main/CVE20220185_temp_wa.yaml

Please don't forget to delete Daemonset after patched image is deployed.

[miwithro]

See updated Mitigations applied in the intro for everyone. Thank you.

[adamshawvipps]

How do I get my AKS nodeimage to 1.24? The latest nodepool image I can update to from the portal is 2022.01.08

[miwithro]

@adamshawvipps we are pushing the 2022.1.24 VHD out next week.

[michaelschmit]

Any update on when the new image is going to be pushed out this week?

[miwithro]

@michaelschmit Yes it will be rolled out to all Azure regions by 2022-02-07.

[maria-pronin]

We are running v1.20.7, once you are roll out the fix would we require to upgrade the AKS?

[miwithro]

@maria-pronin no just need to upgrade the node image not the AKS version

[ninkaninus]

@maria-pronin no just need to upgrade the node image not the AKS version

What will be the recommended way of doing this trough az cli?

[miwithro]

@ninkaninus az aks upgrade -n <> -g <> --node-image-only

[sosammy91]

May I know when will AKSUbuntu-1804-2022.01.24 be made available to West Europe region?

When running az aks nodepool get-upgrades, I still get as "latestNodeImageVersion": "AKSUbuntu-1804containerd-2022.01.19".

[miwithro]

@sosammy91 it should be available in "westeurope" by Wednesday. So please give it a try on Thursday to be safe.

[zohebs341]

@miwithro Thanks for your updates on fix.

How about UAE North/Central Region? I

[zohebs341]

@miwithro In UAE North/Central, I can see the below image as the latest Image.

"kubernetesVersion": "1.21.7",
"latestNodeImageVersion": "AKSUbuntu-1804gen2containerd-2022.01.19"

Can you please check and let me know when the 2022.01.24 image will be available for UAE Region.

[nnellanspdl]

this is remediated in

Kernel-version:5.4.0-1068-azure which we are working to deploy in AKS.

According to the release notes for the 2022-01-24 image it is using 5.4.0-1067 and not the remediated version you mentioned above. Is this version (5.4.0-1067) still vulnerable?

Snippet from the release notes:

Using kernel:
Linux version 5.4.0-1067-azure (buildd@lcy02-amd64-037) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #70~18.04.1-Ubuntu SMP Thu Jan 13 19:46:01 UTC 2022

[saltera]

The Ubuntu CVE appears to suggest it may have been fixed in 5.4.0-1067.70~18.04.1. However I'm not sure how to correlate that with this thread given the slightly different format.

Source: https://ubuntu.com/security/CVE-2022-0185

[miwithro]

@nnellanspdl the 2022-01-24 image is using the remediated kernel (5.4.0-1067.70). Will fix the Release Notes.

@zohebs341 "uaenorth" is Wednesday as well.

[tkent]

Just for those waiting for this, the image does not appear to be available in at least westus2 region either. Does Wednesday (or 2022-02-09) apply there as well?

[miwithro]

@tkent that is correct. Looking like Wednesday that image will hit "westus2"

[dauntlessXXI]

@miwithro when will the image hit the US Gov regions?

[miwithro]

@dauntlessXXI by this weekend.

[vinzim]

Still waiting on uscentral, do we have a time on when this might be available?

[tfbubu111]

@sosammy91 it should be available in "westeurope" by Wednesday. So please give it a try on Thursday to be safe.

@miwithro

Today I was trying to upgrade node image in 'westeurope', the latest node image is 20220119. Kindly help confirm the release time, many thanks.

[miwithro]

@tfbubu111 looks like I was about 1 day off. I see westeurope in the queue for the deployment in the next few hours.

[michaelspinks]

Interesting that I spun up a node today which is AKSUbuntu-1804gen2containerd-2022.01.24 running 5.4.0-1067-azure which is billed as patched. However the Azure Diagnostics for my cluster is warning me that I have CVE-2022-0185 - Linux Kernel Vulnerability on my nodes.

[miwithro]

@michaelspinks looks like an Azure Diagnostics issue. I am looking into it. Short answer is if you are running 2022.01.24 you are patched for CVE-2022-0185

[michaelspinks]

Thanks for that @miwithro

[Cristiano-Rosa]

@miwithro

Before and after the image node upgrade I still see the same kernel version:

root@aks-vmss2-73514419-vmss000000:/# uname -a
Linux aks-vmss2-73514419-vmss000000 5.4.0-1067-azure #70~18.04.1-Ubuntu SMP Thu Jan 13 19:46:01 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I understand it should show 5.4.0-1067.70-azure rather than 5.4.0-1067-azure after the node image upgrade to "AKSUbuntu-1804gen2containerd-2022.01.24".

[Cristiano-Rosa]

@miwithro

According to this source:
https://ubuntu.com/security/CVE-2022-0185

linux-azure-5.4 for Ubuntu 18.04 should have 5.4.0-1067.70 to fix CVE-2022-0185

My nodes are linux-azure-5.4 for Ubuntu 18.04:

kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-vmss2-73514419-vmss000000 Ready agent 31m v1.20.9 10.240.0.62 Ubuntu 18.04.6 LTS 5.4.0-1067-azure containerd://1.4.9+azure
aks-vmss2-73514419-vmss000001 Ready agent 27m v1.20.9 10.240.0.91 Ubuntu 18.04.6 LTS 5.4.0-1067-azure containerd://1.4.9+azure

I have already upgraded to "AKSUbuntu-1804gen2containerd-2022.01.24":

az aks nodepool show -g BereAKS1 -n vmss2 --cluster-name BereAKS1 --query nodeImageVersion
"AKSUbuntu-1804gen2containerd-2022.01.24"

However the kernel version still shows 5.4.0-1067-azure rather than 5.4.0-1067.70:

root@aks-vmss2-73514419-vmss000000:/# uname -a
Linux aks-vmss2-73514419-vmss000000 5.4.0-1067-azure #70~18.04.1-Ubuntu SMP Thu Jan 13 19:46:01 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[miwithro]

That is still 5.4.0-1067.70 notice the -azure #70. This is all captured in the commit.

https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/+git/bionic/commit/?h=azure-5.4&id=dda3369b4d3e88090f1ea28f15b24bc2eec620f8

[ruijarimba]

Same as @Cristiano-Rosa, shouldn't the kernel be 5.4.0-1067.70-azure instead of 5.4.0-1067-azure?

Our cluster is in West Europe. This is the output of kubectl get nodes -o wide:

NAME                              STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
aks-default-xxxxxxxx-vmss000000   Ready    agent   23m   v1.22.4   xx.xx.52.4     <none>        Ubuntu 18.04.6 LTS               5.4.0-1067-azure   containerd://1.5.5+azure
aks-ll-xxxxxxxx-vmss000000        Ready    agent   20m   v1.22.4   xx.xx.60.4     <none>        Ubuntu 18.04.6 LTS               5.4.0-1067-azure   containerd://1.5.5+azure
aks-lm-xxxxxxxx-vmss000000        Ready    agent   19m   v1.22.4   xx.xx.56.4     <none>        Ubuntu 18.04.6 LTS               5.4.0-1067-azure   containerd://1.5.5+azure
aks-mon-xxxxxxxx-vmss000000       Ready    agent   20m   v1.22.4   xx.xx.50.4     <none>        Ubuntu 18.04.6 LTS               5.4.0-1067-azure   containerd://1.5.5+azure

Node image version: AKSUbuntu-1804gen2containerd-2022.01.24

[marcelocerri]

Hi! I'm part of the Ubuntu kernel team and I would like to add some explanation on how our kernel versions work.

We basically have two different version numbers: the kernel release number (5.4.0-1067-azure) and the debian package version (5.4.0-1067.70). The Ubuntu CVE and USN pages will list the debian package version that fixes the vulnerability, not the kernel release number.

The debian package version is composed by the upstream major version (5.4.0), the ABI number (1067) and the upload number (70), while the kernel release number is composed by upstream major version (5.4.0) the ABI number (1067) and kernel flavor (azure) (no upload number).

The upload number is only used to allow us to upload a new version of the same debian kernel package when it has a build failure. So for example, if 5.4.0-1067.69 fails to build, we will fix that version and upload it again as 5.4.0-1067.70. That means that only a single kernel is released for a given ABI number (ie, 1067).

In this particular case, that means that the kernel with release number 5.4.0-1067-azure contains the fix for the CVE 2022-0185.

If you have any questions or concerns, please let me know.

Thank you!

[phealy]

If you want to easily validate VHD image version and kernel version at the same time, you can use kubectl's custom-columns feature:

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,IMAGE:.metadata.labels."kubernetes\.azure\.com/node-image-version",KERNEL:.status.nodeInfo.kernelVersion
NAME                                IMAGE                                     KERNEL
aks-nodepool1-28048431-vmss000000   AKSUbuntu-1804gen2containerd-2022.01.24   5.4.0-1067-azure
aks-nodepool1-28048431-vmss000001   AKSUbuntu-1804gen2containerd-2022.01.24   5.4.0-1067-azure
aks-nodepool1-28048431-vmss000002   AKSUbuntu-1804gen2containerd-2022.01.24   5.4.0-1067-azure

[zohebs341]

@miwithro @marcelocerri Hi Team, For the UAE North region - Still even after upgrading the node image, I can see the same versions.

kubectl get nodes -o custom-columns=NAME:.metadata.name,IMAGE:.metadata.labels."kubernetes.azure.com/node-image-version",KERNEL:.status.nodeInfo.kernelVersion
NAME IMAGE KERNEL
aks-agentpool-30873716-vmss00000f AKSUbuntu-1804gen2containerd-2022.01.19 5.4.0-1067-azure
aks-agentpool-30873716-vmss00000g AKSUbuntu-1804gen2containerd-2022.01.19 5.4.0-1067-azure
aks-agentpool-30873716-vmss00000h AKSUbuntu-1804gen2containerd-2022.01.19 5.4.0-1067-azure

And from the portal, seems Microsoft has removed that vulnerability. As I don't see this URL and vulnerability from Microsoft Portal. Few days back, I was able to see this CVE in "Known Issues, Availability and Performance" tab.

Why it has been removed from the portal and when can we expect an updated remediated version for UAE North region.

[miwithro]

@zohebs341 uaenorth was completed a while ago. 2022.01.24 should be the latest VHD image in that region. Please try to upgrade again.

[zohebs341]

@miwithro Thank you. Upgraded again and this time it reflected.

Hopefully, this image is free of vulnerabilities(Remediated one). (AKSUbuntu-1804gen2containerd-2022.01.24)

kubectl get nodes -o custom-columns=NAME:.metadata.name,IMAGE:.metadata.labels."kubernetes.azure.com/node-image-version",KERNEL:.status.nodeInfo.kernelVersionNAME IMAGE KERNEL
aks-agentpool-30873716-vmss00000i AKSUbuntu-1804gen2containerd-2022.01.24 5.4.0-1067-azure
aks-agentpool-30873716-vmss00000j AKSUbuntu-1804gen2containerd-2022.01.24 5.4.0-1067-azure
aks-agentpool-30873716-vmss00000k AKSUbuntu-1804gen2containerd-2022.01.24 5.4.0-1067-azure

[zohebs341]

@miwithro

In the latest version of AKS Nodes, the value is "1". Again you will create a new image for this [CVE-2022-0185]?

sysctl status kernel.unprivileged_userns_clone

sysctl: cannot stat /proc/sys/status: No such file or directory
kernel.unprivileged_userns_clone = 1

AKS: 1.21.7
Node Image Version: AKSUbuntu-1804gen2containerd-2022.02.07
Container runtime version: containerd://1.4.12+azure-2

[rouke-broersma]

@zohebs341 I'm fairly sure that kernel.unprivileged_userns_clone=0 is only a mitigation and not the fix. The fix is a patch that removes the underflow vulnerability. This patch is part of the updated node image.

[ghost]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.