Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stack: Kyverno #38

Merged
merged 9 commits into from
Oct 2, 2024
Merged

Stack: Kyverno #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
95eea40
kyverno: add initial implementation
kurktchiev Sep 23, 2024
336a15f
kyverno: update structure so we can deploy enforce and audit modes as…
kurktchiev Sep 23, 2024
25eff37
kyverno: rework how the Apps work so user can better choose to use Au…
kurktchiev Sep 23, 2024
63fed0c
make the README clearer
kurktchiev Sep 23, 2024
0eba4f3
fix formatting of commands
kurktchiev Sep 23, 2024
8e92bb4
kyverno: break out the two apps in seperate files
kurktchiev Sep 23, 2024
50824be
kyverno: install latest version of the policy charts
kurktchiev Sep 23, 2024
bee24ff
kyverno: lock to version
kurktchiev Oct 1, 2024
6a9b95e
kyverno: drop ess- since its not a default provider
kurktchiev Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions kyverno-integration/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Kyverno Stack

Implementation of Kyverno for CNOE

## Components

The Stack installs `Kyverno` and optionally `Kyverno Pod Security Policies - Restricted` implementation. By default users should use:
- `module/audit` - for testing and understanding of the impact
- `module/enforce` - once the proper state of platform is understood and all necessary workload exceptions or violations have been accounted for.
- If you chose to enable `Enforce` mode. Exceptions for the following `ref-implementation` components are included, to ensure proper operability:
- [ArgoCD](modules/enforce/exceptions/argocd.yaml)
- [Crossplane](modules/enforce/exceptions/crossplane.yaml)
- [Backstage](modules/enforce/exceptions/backstage.yaml)
- [Ingress-Nginx](modules/enforce/exceptions/ingress-nginx.yaml)
- [Kind cluster](modules/enforce/exceptions/kind.yaml), this should mainly be needed when testing `ref-implementation` on a `kind` installation

*NOTE* - enabling `Enforce` mode without prior testing will most likely cause issues for NEW workloads, already existing workloads will not be affected immediately, always start with `Audit` unless you are completely sure of the impact enabling blocking policies will have on your platform.

## Installation

You can use and test out this stack without using any policies, using the `ref-implementation` as follows:

```bash
idpbuilder create --use-path-routing \
-p https://github.com/cnoe-io/stacks//ref-implementation \
-p https://github.com/cnoe-io/stacks//kyverno-integration
```

Depending on your use case, install the Kubernetes PSS Policies in `Audit`, implemented in Kyverno as follows:

```bash
idpbuilder create --use-path-routing \
-p https://github.com/cnoe-io/stacks//ref-implementation \
-p https://github.com/cnoe-io/stacks//kyverno-integration \
-p https://github.com/cnoe-io/stacks//kyverno-integration/modules/audit
```

If you would like to change to `Enforce` mode:

```bash
idpbuilder create --use-path-routing \
-p https://github.com/cnoe-io/stacks//ref-implementation \
-p https://github.com/cnoe-io/stacks//kyverno-integration \
-p https://github.com/cnoe-io/stacks//kyverno-integration/modules/enforce
```

31 changes: 31 additions & 0 deletions kyverno-integration/kyverno.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
name: kyverno
namespace: argocd
spec:
project: default
source:
chart: kyverno
repoURL: https://kyverno.github.io/kyverno/
targetRevision: 3.2.7
helm:
releaseName: kyverno
valuesObject:
kyverno.fullname: kyverno
destination:
server: "https://kubernetes.default.svc"
namespace: kyverno
syncPolicy:
syncOptions:
- Replace=true
- CreateNamespace=true
automated:
selfHeal: true
prune: true
retry:
limit: 30
backoff:
duration: 5s
factor: 2
maxDuration: 3m0s
33 changes: 33 additions & 0 deletions kyverno-integration/modules/audit/kyverno-pss-policies-audit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
name: kyverno-pss-policies-audit
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/kyverno/kyverno
targetRevision: 3.2.7
path: charts/kyverno-policies
helm:
releaseName: "kyverno-policies"
parameters:
- name: "podSecurityStandard"
value: restricted
- name: "validationFailureAction"
value: Audit
- name: "podSecuritySeverity"
value: High
destination:
server: "https://kubernetes.default.svc"
syncPolicy:
syncOptions:
- Replace=true
automated:
selfHeal: true
retry:
limit: 30
backoff:
duration: 5s
factor: 2
maxDuration: 3m0s
35 changes: 35 additions & 0 deletions kyverno-integration/modules/enforce/exceptions/argocd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
apiVersion: kyverno.io/v2beta1
kind: PolicyException
metadata:
name: argocd-cnoe-operation
namespace: kyverno
spec:
exceptions:
kurktchiev marked this conversation as resolved.
Show resolved Hide resolved
- policyName: disallow-privilege-escalation
ruleNames:
- privilege-escalation
- autogen-privilege-escalation
- policyName: disallow-capabilities-strict
ruleNames:
- require-drop-all
- autogen-require-drop-all
- policyName: require-run-as-nonroot
kurktchiev marked this conversation as resolved.
Show resolved Hide resolved
ruleNames:
- run-as-non-root
- autogen-run-as-non-root
- policyName: restrict-seccomp-strict
ruleNames:
- check-seccomp-strict
- autogen-check-seccomp-strict
match:
any:
- resources:
kinds:
- Pod
- Deployment
- ReplicaSet
namespaces:
- argocd
names:
# TODO: this should be more targeted than blanket *
- argocd-*
35 changes: 35 additions & 0 deletions kyverno-integration/modules/enforce/exceptions/backstage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
apiVersion: kyverno.io/v2beta1
kind: PolicyException
metadata:
name: backstage-cnoe-operation
namespace: kyverno
spec:
exceptions:
kurktchiev marked this conversation as resolved.
Show resolved Hide resolved
- policyName: disallow-privilege-escalation
ruleNames:
- privilege-escalation
- autogen-privilege-escalation
- policyName: disallow-capabilities-strict
ruleNames:
- require-drop-all
- autogen-require-drop-all
- policyName: require-run-as-nonroot
ruleNames:
- run-as-non-root
- autogen-run-as-non-root
- policyName: restrict-seccomp-strict
ruleNames:
- check-seccomp-strict
- autogen-check-seccomp-strict
match:
any:
- resources:
kinds:
- Pod
- Deployment
- ReplicaSet
namespaces:
- backstage
names:
# TODO: this should be more targeted than blanket *
- backstage*
36 changes: 36 additions & 0 deletions kyverno-integration/modules/enforce/exceptions/crossplane.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
apiVersion: kyverno.io/v2beta1
kind: PolicyException
metadata:
name: crossplane-system-cnoe-operation
namespace: kyverno
spec:
exceptions:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install crossplane different to make it more align with best practice policies?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am not sure what you mean here can you elaborate

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds exceptions to allow crossplane to work, is there some changes we can do the crossplane installation here to not need these many exceptions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could, but since Stacks have no conditionals, if the exceptions are installed without Kyverno, the installation will fail as k8s won't know what object that is. Let me know.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nabuskey ^^ let me know if you have any thoughts

Copy link
Contributor

@nabuskey nabuskey Oct 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a path forward for this without native conditionals and parameter sharing in packages. Best we can do is to make changes to the Crossplane stack but it's outside scope of this PR. This kind of things is exactly why we need a way to support templating or layering somehow. I don't think we can expect people to wrap everything in a helm chart.

- policyName: disallow-capabilities-strict
ruleNames:
- require-drop-all
- autogen-require-drop-all
- policyName: disallow-privilege-escalation
ruleNames:
- privilege-escalation
- autogen-privilege-escalation
- policyName: require-run-as-nonroot
ruleNames:
- run-as-non-root
- autogen-run-as-non-root
- policyName: restrict-seccomp-strict
ruleNames:
- check-seccomp-strict
- autogen-check-seccomp-strict
match:
any:
- resources:
kinds:
- Pod
- Deployment
- ReplicaSet
namespaces:
- crossplane-system
names:
# TODO: this should be more targeted than blanket *
- crossplane*
- upbound-provider-*
22 changes: 22 additions & 0 deletions kyverno-integration/modules/enforce/exceptions/ingress-nginx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: kyverno.io/v2beta1
kind: PolicyException
metadata:
name: ingress-nginx-cnoe-operation
namespace: kyverno
spec:
exceptions:
- policyName: disallow-host-ports
ruleNames:
- host-ports-none
- autogen-host-ports-none
match:
any:
- resources:
kinds:
- Pod
- Deployment
- ReplicaSet
namespaces:
- ingress-nginx
names:
- ingress-nginx*
66 changes: 66 additions & 0 deletions kyverno-integration/modules/enforce/exceptions/kind.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
apiVersion: kyverno.io/v2beta1
kurktchiev marked this conversation as resolved.
Show resolved Hide resolved
kind: PolicyException
metadata:
name: system-cnoe-operation
namespace: kyverno
spec:
exceptions:
- policyName: disallow-host-path
ruleNames:
- host-path
- autogen-host-path
- policyName: disallow-privilege-escalation
ruleNames:
- privilege-escalation
- autogen-privilege-escalation
- policyName: disallow-privileged-containers
ruleNames:
- privileged-containers
- autogen-privileged-containers
- policyName: disallow-capabilities-strict
ruleNames:
- require-drop-all
- autogen-require-drop-all
- adding-capabilities-strict
- autogen-adding-capabilities-strict
- adding-capabilities
- autogen-adding-capabilities
- policyName: disallow-capabilities
ruleNames:
- adding-capabilities
- autogen-adding-capabilities
- policyName: require-run-as-nonroot
kurktchiev marked this conversation as resolved.
Show resolved Hide resolved
ruleNames:
- run-as-non-root
- autogen-run-as-non-root
- policyName: restrict-seccomp-strict
ruleNames:
- check-seccomp-strict
- autogen-check-seccomp-strict
- policyName: restrict-volume-types
ruleNames:
- restricted-volumes
- autogen-restricted-volumes
- policyName: disallow-host-namespaces
ruleNames:
- host-namespaces
- autogen-host-namespaces
match:
any:
- resources:
kinds:
- Pod
- Deployment
- ReplicaSet
- StatefulSet
- DaemonSet
namespaces:
- kube-system
- local-path-storage
names:
# TODO: this should be more targeted than blanket *
- kube-*
- kindnet*
- local-path*
- coredns*
- etcd-*
26 changes: 26 additions & 0 deletions kyverno-integration/modules/enforce/kyverno-pss-exceptions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
name: kyverno-pss-policies-enforce-exceptions
namespace: argocd
spec:
project: default
source:
repoURL: cnoe://exceptions
targetRevision: HEAD
path: "."
directory:
recurse: true
destination:
server: "https://kubernetes.default.svc"
syncPolicy:
syncOptions:
- Replace=true
automated:
selfHeal: true
retry:
limit: 30
backoff:
duration: 5s
factor: 2
maxDuration: 3m0s
33 changes: 33 additions & 0 deletions kyverno-integration/modules/enforce/kyverno-pss-policies-enforce.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
name: kyverno-pss-policies-enforce
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/kyverno/kyverno
targetRevision: HEAD
path: charts/kyverno-policies
helm:
releaseName: "kyverno-policies"
parameters:
- name: "podSecurityStandard"
value: restricted
- name: "validationFailureAction"
value: Enforce
- name: "podSecuritySeverity"
value: High
destination:
server: "https://kubernetes.default.svc"
syncPolicy:
syncOptions:
- Replace=true
automated:
selfHeal: true
retry:
limit: 30
backoff:
duration: 5s
factor: 2
maxDuration: 3m0s
Loading