Skip to content

Commit

Permalink
Redesign of C2P for pipeline use case (#3)
Browse files Browse the repository at this point in the history
* update k8s library version

Signed-off-by: Takumi Yanagawa <[email protected]>

* decomposer outputs should contain only successfully decomposed policies

Signed-off-by: Takumi Yanagawa <[email protected]>

* refactor

Signed-off-by: Takumi Yanagawa <[email protected]>

* decomposed resources include policy-generator manifest

Signed-off-by: Takumi Yanagawa <[email protected]>

* reuse policy-generator manifest generated at decomposition

Signed-off-by: Takumi Yanagawa <[email protected]>

* composer takes OSCAL

Signed-off-by: Takumi Yanagawa <[email protected]>

* composer generates policySet

Signed-off-by: Takumi Yanagawa <[email protected]>

* composer takes c2pcr

Signed-off-by: Takumi Yanagawa <[email protected]>

* refactor

Signed-off-by: Takumi Yanagawa <[email protected]>

* add templatized parameter filling

Signed-off-by: Takumi Yanagawa <[email protected]>

* add default namespace selector

Signed-off-by: Takumi Yanagawa <[email protected]>

* refactor

Signed-off-by: Takumi Yanagawa <[email protected]>

* use kustomize type provided by sigs.k8s.io

Signed-off-by: Takumi Yanagawa <[email protected]>

* add component-title annotation

Signed-off-by: Takumi Yanagawa <[email protected]>

* add reporter

Signed-off-by: Takumi Yanagawa <[email protected]>

* add placement_type and helpers

Signed-off-by: Takumi Yanagawa <[email protected]>

* convert policy to policy report

Signed-off-by: Takumi Yanagawa <[email protected]>

* compliance-report from policy reports

Signed-off-by: Takumi Yanagawa <[email protected]>

* do not ommit the InformGatekeeper/KyvernoPolicies field

Signed-off-by: Takumi Yanagawa <[email protected]>

* refactor

Signed-off-by: Takumi Yanagawa <[email protected]>

* create c2pcli

Signed-off-by: Takumi Yanagawa <[email protected]>

* enable gorelease

Signed-off-by: Takumi Yanagawa <[email protected]>

* move policyResultDir parameter to c2pcr

Signed-off-by: Takumi Yanagawa <[email protected]>

* generate reports as default

Signed-off-by: Takumi Yanagawa <[email protected]>

* add metadata to compliance report

Signed-off-by: Takumi Yanagawa <[email protected]>

* add scripts

Signed-off-by: Takumi Yanagawa <[email protected]>

* fix

Signed-off-by: Takumi Yanagawa <[email protected]>

* separate generated OCM manifests and policy generator sources

Signed-off-by: Takumi Yanagawa <[email protected]>

* fix NPE when unable to get statuses

Signed-off-by: Takumi Yanagawa <[email protected]>

* add scripts for setting up argocd

Signed-off-by: Takumi Yanagawa <[email protected]>

* ocm status collector

Signed-off-by: Takumi Yanagawa <[email protected]>

* fix

Signed-off-by: Takumi Yanagawa <[email protected]>

* generate md file

Signed-off-by: Takumi Yanagawa <[email protected]>

* fix

Signed-off-by: Takumi Yanagawa <[email protected]>

* add report-utils

Signed-off-by: Takumi Yanagawa <[email protected]>

* use OCM gitops

Signed-off-by: Takumi Yanagawa <[email protected]>

* remove the dependency on hub namepsace from Reporter

Signed-off-by: Takumi Yanagawa <[email protected]>

* allow to use PolicyGenerator with empty namespace in PolicyDefaults

Signed-off-by: Takumi Yanagawa <[email protected]>

* upsate collector script

Signed-off-by: Takumi Yanagawa <[email protected]>

---------

Signed-off-by: Takumi Yanagawa <[email protected]>
  • Loading branch information
yana1205 authored Jul 25, 2023
1 parent e9d9f53 commit fba0dd8
Show file tree
Hide file tree
Showing 159 changed files with 225,152 additions and 865 deletions.
59 changes: 59 additions & 0 deletions .github/workflows/create-and-publish-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

# GitHub recommends pinning actions to a commit SHA.
# To get a newer version, you will need to update the SHA.
# You can also reference a tag or branch, but the action may change without warning.

name: Create and publish a Docker image

on:
push:
tags:
- 'v*.*.*'

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}-ocm-status-collector

jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up QEMU
uses: docker/setup-qemu-action@v1
# https://github.com/docker/setup-buildx-action
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1

- name: Log in to the Container registry
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

- name: Build and push Docker image
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
with:
context: ./scripts/docker
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
53 changes: 53 additions & 0 deletions .github/workflows/goreleaser-workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: goreleaser

on:
push:
tags:
- 'v*'
paths:
- .github/workflows/goreleaser.yml
- .goreleaser.yaml

permissions:
contents: write
packages: write

env:
REGISTRY: ghcr.io
ARCHS: linux/amd64,linux/arm64

jobs:
goreleaser:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-go@v4
with:
go-version: v1.19
- name: Set env
run: echo "RELEASE_VERSION=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
- name: Delete non-semver tags
run: 'git tag -d $(git tag -l | grep -v "^v")'
- name: Set LDFLAGS
run: echo LDFLAGS="$(make ldflags)" >> $GITHUB_ENV
- name: Run GoReleaser on tag
if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
uses: goreleaser/goreleaser-action@v4
with:
distribution: goreleaser
version: latest
args: release --timeout 60m
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Run GoReleaser on push without tag
if: github.event_name == 'push' && !contains(github.ref, 'refs/tags/')
uses: goreleaser/goreleaser-action@v4
with:
distribution: goreleaser
version: latest
args: release --timeout 60m --snapshot --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ Dockerfile.cross
.vscode
.idea

# Output of gorelease
dist

# ignore output by test
/**/_test

Expand Down
21 changes: 21 additions & 0 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# .goreleaser.yaml
builds:
- id: c2pcli
main: ./cmd/c2pcli
binary: c2pcli
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ignore:
- goos: linux
goarch: arm64

release:
disable: false
skip_upload: false
github:
owner: yana1205
name: compliance-to-policy
16 changes: 16 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -277,3 +277,19 @@ catalog-build: opm ## Build a catalog image.
.PHONY: catalog-push
catalog-push: ## Push a catalog image.
$(MAKE) docker-push IMG=$(CATALOG_IMG)

###
.PHONY: compose-v2
compose-v2: bin/compose-v2.linux_amd64 bin/compose-v2.darwin_amd64 bin/compose-v2.darwin_arm64

bin/compose-v2.linux_amd64:
GOOS=linux GOARCH=amd64 go build -o bin/compose-v2.linux_amd64 ./cmd/compose-v2

bin/compose-v2.darwin_amd64:
GOOS=darwin GOARCH=amd64 go build -o bin/compose-v2.darwin_amd64 ./cmd/compose-v2

bin/compose-v2.darwin_arm64:
GOOS=darwin GOARCH=arm64 go build -o bin/compose-v2.darwin_arm64 ./cmd/compose-v2

bin/compose-v2.%.gz: bin/compose-v2.%
gzip ./bin/compose-v2.$*
127 changes: 101 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,70 @@
# compliance-to-policy
Compliance-to-Policy (C2P) provides the framework to bridge the gap between compliance and policy administration.

## setup pipeline
1. Create two repositories (one is configuration repository that's used for pipeline from OSCAL to Policy and another is evidence repository that's used for pipeline from OCM statuses to Compliance result)
- For example, c2p-for-ocm-pipeline01-config and c2p-for-ocm-pipeline01-evidence
1. Create Github Personal Access Token having following permissions
- Repository permission of `Contents`, `Pull Requests`, and `Workflows` with read-and-write against both the configuration repository and the evidence repository.
1. Fork C2P repository (yana1205/compliance-to-policy.git) and checkout `template`
1. Set required parameters for github action to initialize your configuration and evidence repo
1. Go to Settings tab
1. Go to `Actions` under `Secrets and variables`
1. Create `New repository secret`
- Name: PAT
- Secret: Created Github Personal Access Token
1. Go to `Variables` tab to create `New repository variable`
1. Create `CONFIGURATION_REPOSITORY` variable
- Name: CONFIGURATION_REPOSITORY
- Value: `<configuration repository org>/<configuration repository name> (e.g. yana1205/c2p-for-ocm-pipeline01-config)`
1. Create `EVIDENCE_REPOSITORY` variable
- Name: EVIDENCE_REPOSITORY
- Value: `<evidence repository org>/<evidence repository name> (e.g. yana1205/c2p-for-ocm-pipeline01-evidence)`
1. Run Action `Initialize repositories` with branch `template`
1. Go to the configuration repository and create `New repository secret`
- Name: PAT
- Secret: Created Github Personal Access Token
1. Go to the evidence repository and create `New repository secret`
- Name: PAT
- Secret: Created Github Personal Access Token

## run oscal-to-pocliy
1. Go to the configuration repository
1. Go to `Actions` tab
1. Run `OSCAL to Policy`
1. This action generates manifests from OSCAL and then generate a PR of changes for a directory `ocm-policy-manifests` containing the generated manifests.
1. Merge the PR

## integrate with GitOps
1. Sync `ocm-policy-manifests` directory with your OCM Hub by OCM GitOps (OCM Channel and Subscription addon)

## deploy collector to your OCM Hub
1. Apply RBAC for collector
```
kubectl apply -f https://raw.githubusercontent.com/yana1205/compliance-to-policy/redesign.0622/scripts/collect/rbac.yaml
```
1. Create Secret for Github access
```
kubectl -n c2p create secret generic --save-config collect-ocm-status-secret --from-literal=user=<github user> --from-literal=token=<github PAT> --from-literal=org=<evidence org name> --from-literal=repo=<evidence repo name>
```
e.g.
```
kubectl -n c2p create secret generic --save-config collect-ocm-status-secret --from-literal=user=yana1205 --from-literal=token=github_pat_xxx --from-literal=org=yana1205 --from-literal=repo=c2p-for-ocm-pipeline01-evidence
```
1. Deploy collector cronjob
```
kubectl apply -f https://raw.githubusercontent.com/yana1205/compliance-to-policy/redesign.0622/scripts/collect/cronjob.yaml
```

## cleanup
```
kubectl delete -f https://raw.githubusercontent.com/yana1205/compliance-to-policy/redesign.0622/scripts/collect/cronjob.yaml
kubectl -n c2p delete secret collect-ocm-status-secret
kubectl delete -f https://raw.githubusercontent.com/yana1205/compliance-to-policy/redesign.0622/scripts/collect/rbac.yaml
```

---
# Controller pattern
## Prerequisites
1. Install [Policy Generator Plugin](https://github.com/open-cluster-management-io/policy-generator-plugin#as-a-kustomize-plugin)

Expand All @@ -22,37 +86,48 @@ Decompose OCM poicy collection to kubernetes resources composing each OCM policy
├── _sources
└── resources
```

Individual decomposed resource contains k8s manifests and configuration files (policy-generator.yaml and kustomization.yaml) for PolicyGenerator.
```
$ tree -L 3 /tmp/c2p-output/decomposed/resources
/tmp/c2p-output/decomposed/resources
├── add-chrony
│   ├── add-chrony-worker
│   │   └── MachineConfig.50-worker-chrony.0.yaml
│   ├── kustomization.yaml
│   └── policy-generator.yaml
├── add-tvk-license
│   ├── add-tvk-license
│   │   └── License.triliovault-license.0.yaml
│   ├── kustomization.yaml
```
## C2P Composer
Compose OCM Policy from policy resources from compliance information (for example, [compliance.yaml](cmd/compose/compliance.yaml))

1. Run C2P Composer
```
go run ./cmd/compose/compose.go --policy-resources-dir=/tmp/c2p-output/decomposed/resources --compliance-yaml=./cmd/compose/compliance.yaml --out=/tmp/c2p-output
```
1. Composed OCM policies are output in `/tmp/c2p-output/composed`
```
$ tree -L 1 /tmp/c2p-output/composed
/tmp/c2p-output/composed
├── add-chrony.yaml
└── install-odf-lvm-operator.yaml
```
1. If you want to see the intermidiate files to generate OCM Policy, please set `--temp-dir=<something to directory>` in the previous C2P Composer command.
```
$ mkdir -p /tmp/c2p-temp
$ go run ./cmd/compose/compose.go --policy-resources-dir=/tmp/c2p-output/decomposed/resources --compliance-yaml=./cmd/compose/compliance.yaml --out=/tmp/c2p-output --temp-dir=/tmp/c2p-temp
$ tree -L 4 /tmp/c2p-temp
/tmp/c2p-temp
└── tmp-747478669
└── CM-2 Baseline Configuration
├── add-chrony
│ ├── kustomization.yaml
│ ├── policy-generator.yaml
│ └── resources
└── install-odf-lvm-operator
├── kustomization.yaml
├── policy-generator.yaml
└── resources
go run cmd/compose-by-c2pcr/main.go --c2pcr ./cmd/compose-by-c2pcr/c2pcr.yaml --out /tmp/c2p-output
```
1. Composed OCM policies are output in `/tmp/c2p-output`
```
$ tree /tmp/c2p-output
/tmp/c2p-output
├── add-chrony
│ ├── add-chrony-worker
│ │ └── MachineConfig.50-worker-chrony.0.yaml
│ ├── kustomization.yaml
│ └── policy-generator.yaml
├── install-odf-lvm-operator
│ ├── kustomization.yaml
│ ├── odf-lvmcluster
│ │ └── LVMCluster.odf-lvmcluster.0.yaml
│ ├── policy-generator.yaml
│ └── policy-odf-lvm-operator
│ ├── Namespace.openshift-storage.0.yaml
│ ├── OperatorGroup.openshift-storage-operatorgroup.0.yaml
│ └── Subscription.lvm-operator.0.yaml
├── kustomization.yaml
├── policy-generator.yaml
└── policy-sets.yaml
```

## C2P Controller
Expand Down
53 changes: 53 additions & 0 deletions cmd/c2pcli/cmd/cmd.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/*
Copyright 2023 IBM Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package cmd

import (
"github.com/spf13/cobra"

"github.com/IBM/compliance-to-policy/cmd/c2pcli/options"
composecmd "github.com/IBM/compliance-to-policy/cmd/compose/cmd"
reportutilscmd "github.com/IBM/compliance-to-policy/cmd/report-utils/cmd"
reportcmd "github.com/IBM/compliance-to-policy/cmd/report/cmd"
)

func New() *cobra.Command {
opts := options.NewOptions()

command := &cobra.Command{
Use: "c2pcli",
Short: "C2P CLI",
RunE: func(cmd *cobra.Command, args []string) error {
if err := opts.Complete(); err != nil {
return err
}

if err := opts.Validate(); err != nil {
return err
}
return nil
},
}

opts.AddFlags(command.Flags())

command.AddCommand(composecmd.New())
command.AddCommand(reportcmd.New())
command.AddCommand(reportutilscmd.New())

return command
}
30 changes: 30 additions & 0 deletions cmd/c2pcli/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
Copyright 2023 IBM Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package main

import (
"os"

"github.com/IBM/compliance-to-policy/cmd/c2pcli/cmd"
)

func main() {
err := cmd.New().Execute()
if err != nil {
os.Exit(1)
}
}
Loading

0 comments on commit fba0dd8

Please sign in to comment.