-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SONiC Container Hardening #1364
Conversation
Signed-off-by: Mai Bui <[email protected]>
Signed-off-by: Mai Bui <[email protected]>
@Yarden-Z I made some changes directly to the HLD, please check again |
|
||
### 3.2 net=host | ||
|
||
Removing the `net=host` is required to prevent the container from accessing the full network scope of the host and system. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe, BGP container needs full network access in order to function properly, do we have the list of containers that can be migrated without net=host? Some of the container (DHCP relay, SNMP..etc) directly work on the Linux networking, removing net=host, might require additional changes at the application level.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On the SNMP container we have managed to remove the net=host.
For BGP - we also have a PR up with @maipbui changes that we can review.
For DHCP - we'll need to check this.
I have a composed list of containers and the suggestions on whether we can remove the flags or not, I'll add it to the HLD as an appendix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please help review code PRs which we included in this HLD PR's description. @venkatmahalingam Yes, BGP container needs host network mode.
Signed-off-by: Mai Bui <[email protected]>
|
||
### 4.2 net=host optimization | ||
|
||
Here we will provide a detailed example of how to switch from the `--net=host` configuration (host network) to the `--net=bridge` configuration paired with port forwarding in a specific container. We are using the database container as an example for this item. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this design cover SDK change? SONiC containers are massively interactive with Redis database, it is required to let our SDK adapt bridge mode container
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does this require SDK changes?
Regarding the SDK - since it resides in a single container, we have decided against reducing the privileges and capabilities of this container, as this will require massive SDK re-design (and this is not our purpose).
Also - the SDK container is not at a high risk of compromise since it does not interact with the outside network.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Blueve Which SDK? Is it switch ASIC vendor SDK or swss-common library?
Could you give specific sample function in the SDK you are worrying about?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@qiluo-msft swss-common, do we still have a unified way to interact with Redis database with this design?
Current we can use two approaches to interact Redis in sdk:
- local IP
- Unix sock file
With this design, both of them are not available I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both are still applicable with this design (tested).
UDS should not be affected, since reducing the privileges of the redis container does not affect the Unix socket (on a different level of permissions - file access permissions).
Local IP might require a slight change, but you can check it in the relevant PR and it is currently working.
Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): 14807420 How I did it Reduce linux capabilities in privileged flag, retain NET_ADMIN and SYS_ADMIN capabilities How to verify it Install new image to DUT, verify bgp container is up Run bgp sonic-mgmt kvmtest
Description of PR HLD implementation: Container Hardening (sonic-net/SONiC#1364) Dependency: sonic-net/sonic-buildimage#14932 #### What is the motivation for this PR? Check bgp container has access to /dev/sda* or /dev/vda* after limiting privileged flag to less Linux capabilities. #### How did you do it? #### How did you verify/test it? ``` container_hardening/test_container_hardening.py::test_bgp_dev PASSED [100%] ``` Signed-off-by: Mai Bui <[email protected]>
#### Why I did it Reduce docker privilege This is part of HLD sonic-net/SONiC#1364 #### How I did it Remove flag --privileged #### How to verify it docker exec -it database bash root@0048b82b460b:/# ip link add dummy0 type dummy RTNETLINK answers: Operation not permitted
#### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag, retain NET_ADMIN capability
| SYS_PTRACE | Trace arbitrary processes using ptrace(2). | | ||
| SYS_RAWIO | Perform I/O port operations (iopl(2) and ioperm(2)). | | ||
| SYS_RESOURCE | Override resource Limits | | ||
| SYS_TIME | Set system clock (settimeofday(2), stime(2), adjtimex(2)); set real-time (hardware) clock. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how do we know if the appliation does not use SYS_TIME?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't, but if a specific container requires this - it can add it as a specific flag only for this container.
## 10. Open/Action items - if any | ||
|
||
Currently, Nvidia and MSFT have scoped commitment for specific containers. | ||
Redis and SNMP already have these adjustments. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if SNMP container wants to listen on some particular interface on the host? container hardening will be disabled by default for now? can this be driven based on FEATURE table config?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the SNMP container would like to listen to a specific interface - you can check this:
https://github.com/sonic-net/sonic-buildimage/pull/15176/files
I think that this PR will showcase your specific use-case.
Community review recording https://zoom.us/rec/share/jwTKj7zNviREe488pYQugvH4dIaUFXVYs1N1iDv5jzjnFbVq1TT09vqbrwDIIski.Z6KNg_je7AawlH2V |
-p 6379:6379 \ | ||
{%- endif %} | ||
|
||
#### How to check? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the user experience with this change with the current container capabilities? What is the plan to mitigate this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
User should not experience any changes.
If there are, then one of the following should occur:
- Adjust the user action/container action so that it does not require the elevated privileges or specific net host capabilities.
- Add to the container the required capabilities it needs to perform.
We should go with option 1 first, and only after we have decided option 1 is not feasible - option 2 is on the table.
N/A, this feature will be checked on a system level. | ||
|
||
### 9.2 System Test cases | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any impact w.r.t memory and cpu as it is running in different network name spaces?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Negligible, as it is part of the infra.
I'll be more precise - the impact is as much of how much the Linux ACL takes, but this should not be visible to any user.
|
||
### 3.1 Root privileges | ||
|
||
When removing the root privileges from a specific container - we are required to remove the `--privileged` flag and add the required missing Linux capabilities to the docker, or alternatively adjust the container so that it does not require root privileges to perform any action. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is similar work done with Multi ASIC design, it is better to converge before making any decision changing net=host -> bridge etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not familiar with it, what is the work being done there?
N/A | ||
|
||
### 6.2. CLI/YANG model Enhancements | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a tool that knows what containers are running privileged mode?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There might be, requires a bit of research.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe you can check by docker inspect container_name | grep Privileged
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be applicable for the --privileged flag, but this HLD is also discussing host network.
So while this might work for one of the HLD item (removing containers from privilege mode) it does not scope the whole feature list.
So this command might be a good start, but we should look at a more encompassing approach to check this (and not stop here).
Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): 14807420 How I did it Reduce linux capabilities in privileged flag How to verify it Run snmp sonic-mgmt tests Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. admin@vlab-01:~$ docker inspect snmp | grep Privi "Privileged": false, admin@vlab-01:~$ docker exec -it snmp bash root@vlab-01:/# capsh --print Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_se
Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): 14807420 How I did it Reduce linux capabilities in privileged flag How to verify it Run restapi sonic-mgmt tests on sn4600c Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps.
…k container (#17217) Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): 14807420 How I did it Reduce linux capabilities in privileged flag How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps.
@qiluo-msft @lguohan any further comments or we can approve and merge this HLD? |
…k container (#17217) Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): 14807420 How I did it Reduce linux capabilities in privileged flag How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps.
@zhangyanzhao could you please help to close the loop and get this HLD PR merged? |
### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-01:~$ docker inspect iccpd | grep Privi "Privileged": false, admin@vlab-01:~$ docker exec -it iccpd bash root@vlab-01:/# capsh --print Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep ```
### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-01:~$ docker inspect p4rt | grep Privi "Privileged": false, admin@vlab-01:~$ docker exec -it p4rt bash root@vlab-01:/# capsh --print Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep ```
Description of PR HLD implementation: Container Hardening (sonic-net/SONiC#1364) Dependency: sonic-net/sonic-buildimage#14932 #### What is the motivation for this PR? Check bgp container has access to /dev/sda* or /dev/vda* after limiting privileged flag to less Linux capabilities. #### How did you do it? #### How did you verify/test it? ``` container_hardening/test_container_hardening.py::test_bgp_dev PASSED [100%] ``` Signed-off-by: Mai Bui <[email protected]>
### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-01:~$ docker inspect nat | grep Privi "Privileged": false, admin@vlab-01:~$ docker exec -it nat bash root@vlab-01:/# capsh --print Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep ```
@Blueve @lguohan @venkatmahalingam @maipbui @madhupalu @qiluo-msft please let us know if everything looks good and approve if so |
…iner [docker-acms] limit privileged flag for acms container #### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Run acms sonic-mgmt tests Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-01:~$ docker inspect acms | grep Privilege "Privileged": false, admin@vlab-01:~$ docker exec -it acms bash root@vlab-01:/# capsh --print Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep ``` Signed-off-by: Mai Bui <[email protected]>
@zhangyanzhao this HLD has been shared long time ago and most of the code PRs were already merged. |
Added note regarding user-defined bridges. Not in the scope of this HLD at the moment
#### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-08:~$ docker inspect database0 | jq '.[0] | {CapAdd: .HostConfig.CapAdd, Privileged: .HostConfig.Privileged, AppArmor: .AppArmorProfile, Security Opt: .HostConfig.SecurityOpt, Devices: .HostConfig.Devices, MaskedPath: .HostConfig.MaskedPaths, ReadonlyPaths: .HostConfig.ReadonlyPaths}' { "CapAdd": null, "Privileged": false, "AppArmor": "unconfined", "SecurityOpt": [ "apparmor=unconfined" ], "Devices": [], "MaskedPath": [], "ReadonlyPaths": [] } ```
…-net#19248) #### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-08:~$ docker inspect database0 | jq '.[0] | {CapAdd: .HostConfig.CapAdd, Privileged: .HostConfig.Privileged, AppArmor: .AppArmorProfile, Security Opt: .HostConfig.SecurityOpt, Devices: .HostConfig.Devices, MaskedPath: .HostConfig.MaskedPaths, ReadonlyPaths: .HostConfig.ReadonlyPaths}' { "CapAdd": null, "Privileged": false, "AppArmor": "unconfined", "SecurityOpt": [ "apparmor=unconfined" ], "Devices": [], "MaskedPath": [], "ReadonlyPaths": [] } ```
…-net#19248) #### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-08:~$ docker inspect database0 | jq '.[0] | {CapAdd: .HostConfig.CapAdd, Privileged: .HostConfig.Privileged, AppArmor: .AppArmorProfile, Security Opt: .HostConfig.SecurityOpt, Devices: .HostConfig.Devices, MaskedPath: .HostConfig.MaskedPaths, ReadonlyPaths: .HostConfig.ReadonlyPaths}' { "CapAdd": null, "Privileged": false, "AppArmor": "unconfined", "SecurityOpt": [ "apparmor=unconfined" ], "Devices": [], "MaskedPath": [], "ReadonlyPaths": [] } ```
Create the container hardening initial proposal for SONiC.
During this PR - we would like to present the container hardening criteria and items for SONiC and future requirements in the future for all SONiC containers
Preview:
Containers is a method of creating virtualization and abstraction of an OS for a subset of processes/service on top of a single host with the purpose of giving it an environment to run and execute its tasks without effect of nearby containers/processes.
In SONiC, we are deploying container with full visibility and capabilities as the host Linux.
This poses a security risk and vulnerability as 1 breached container means that the whole system is breached.
Addressing this issue – we have composed this doc for container hardening, containing the security hardening requirements and definitions for all containers on top of SONiC
sonic-mgmt PRs
Revert related PR
syncd, swss cannot be reduced privileged