From 922554dd63ebc874f1ec5230080ebb1856ac423e Mon Sep 17 00:00:00 2001 From: Qi Wang Date: Fri, 6 Oct 2023 00:50:10 -0400 Subject: [PATCH] Add ClusterImagePolicy Signed-off-by: Qi Wang --- ...ImagePolicy-for-Signature-Verification.md} | 216 +++++++++--------- 1 file changed, 108 insertions(+), 108 deletions(-) rename enhancements/api-review/{add-ImagePolicy-to-config.openshift.io-v1alpha1.md => add-ClusterImagePolicy-and-ImagePolicy-for-Signature-Verification.md} (64%) diff --git a/enhancements/api-review/add-ImagePolicy-to-config.openshift.io-v1alpha1.md b/enhancements/api-review/add-ClusterImagePolicy-and-ImagePolicy-for-Signature-Verification.md similarity index 64% rename from enhancements/api-review/add-ImagePolicy-to-config.openshift.io-v1alpha1.md rename to enhancements/api-review/add-ClusterImagePolicy-and-ImagePolicy-for-Signature-Verification.md index f1857d1a787..b09eac52546 100644 --- a/enhancements/api-review/add-ImagePolicy-to-config.openshift.io-v1alpha1.md +++ b/enhancements/api-review/add-ClusterImagePolicy-and-ImagePolicy-for-Signature-Verification.md @@ -1,5 +1,5 @@ --- -title: add-ImagePolicy-to-config.openshift.io-v1alpha1 +title: add-ClusterImagePolicy-and-ImagePolicy-for-Signature-Verification authors: - "@QiWang19" reviewers: # Include a comment about what domain expertise a reviewer is expected to bring and what area of the enhancement you expect them to focus on. For example: - "@networkguru, for networking aspects, please look at IP bootstrapping aspect" @@ -25,7 +25,7 @@ tracking-link: # link to the tracking ticket (for example: Jira Feature or Epic ## Summary -This enhancement propose an ImagePolicy CRD to provide a way to set sigstore image signature verification configuration on Openshift. +This enhancement introduces ClusterImagePolicy and ImagePolicy CRDs to independently manage configurations at the cluster and namespace scopes. ## Motivation @@ -36,15 +36,15 @@ tools, so that I can utilize the increased security of my software supply chain. ### Goals +- ClusterImagePolicy is defined as cluster scoped CRD. ImagePolicy is defined as namespaced CRD. - The user can create an ImagePolicy instance specifying the images/repositories to be verified and their policy. The MCO will write the configuration for signature verification. Once this is done, CRI-O can verify the images/repositories. - -- MCO container runtime config controller watches ImagePolicy instance in different kubernetes namespaces and merges the instances for each namespace into a single [containers-policy.json](https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md) in a predefined `/path/to/policies/\.json`. - -- CRI-O can verify Cosign signature signed images using configuration from ImagePolicy by using the namespace from the sandbox config on the `PullImage` RPC. +- MCO container runtime config controller watches ImagePolicy instance in different kubernetes namespaces and merges the instances for each namespace into a single [containers-policy.json](https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md) in a predefined `/path/to/policies/.json`. +- MCO container runtime config controller watches ClusterImagePolicy instance and merges the instances into a single [/etc/containers/policy.json](https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md). +- CRI-O can verify Cosign signature signed images using configuration from ClusterImagePolicy and ImagePolicy by matching the namespace from the sandbox config on the `PullImage` RPC. ### Non-Goals -- Allow configure policy for OCP product repo +- Configuring policies for the images in the OCP payload is not within the scope of this enhancement. - Providing a tool to mirror the signatures is out of the scope of this enhancement. In order to verify the signature, the disconnected users need to mirror signatures together with the application images. - Grant the application administrator the ability to weaken cluster-scoped policies, to avoid expanding the set of administrators capable of increasing cluster exposure to vulnerable images. - Grant the application administrator the ability to tighten cluster-scoped policies. This could be useful in the future, but we are deferring it to limit the amount of work needed for an initial implementation. @@ -53,26 +53,24 @@ tools, so that I can utilize the increased security of my software supply chain. ### Workflow Description -**cluster administrator** is a human user responsible for adding signature verification configurations in -cluster scope. -1. The cluster administrator creates ImagePolicy yaml file with `namespace: openshift-config`. -2. The cluster administrator creates ImagePolicy instance using `oc create -f imagepolicy.yaml`. -3. The cluster administrator can get the merged cluster scope policies from `status` of ImagePolicy instances. -4. The cluster administrator can delete the signature verification configuration by deleting its ImagePolicy instances. - -**application administrator** is a human user responsible for -adding signature verification configurations for specific namespace. +**cluster administrator** Signature Verification Configuration Workflow: +1. The cluster administrator requests the addition of signature verification configurations at the cluster scope. +2. The cluster administrator writes the verification certification to the ClusterImagePolicy YAML file and creates a ClusterImagePolicy CR using `oc create -f imagepolicy.yaml`. +3. The cluster administrator can retrieve the merged cluster-scoped policies by checking `/etc/containers/policy.json` within the `99--generated-registries` machine-config. +4. The cluster administrator has the option to delete the signature verification configuration by removing its ClusterImagePolicy instances. -1. The application administrator creates ImagePolicy yaml file for namespace that is not `namespace: openshift-config` -2. The application administrator creates ImagePolicy instance using `oc create -f imagepolicy.yaml`. -3. The application administrator can get the merged cluster policy and policy for certain namespace from `status` of ImagePolicy instances. -4. The application administrator can delete the signature verification configuration by deleting its ImagePolicy instances. -5. The application administrator cannot override cluster scoped policies because they will treated with a higher priority. +**application administrator** Signature Verification Configuration Workflow: +1. The application administrator requests the addition of signature verification configurations at the namespace scope. +2. The application administrator writes the verification certification to the ImagePolicy YAML file and creates a ImagePolicy CR using `oc create -f imagepolicy.yaml`. Please note that the application administrator cannot override cluster-scoped policies, as they are treated with higher priority. +3. The application administrator can retrieve the cluster override and merged policies by checking `.json` within the `99--generated-imagepolicies` machine-config. +4. The application administrator has the option to remove the signature verification configuration by deleting its ImagePolicy instances. ### API Extensions #### Type definitions +Type definitions of ImagePolicy. ClusterImagePolicy is expected to have a similar structure to ImagePolicy. + ```go // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object @@ -112,17 +110,13 @@ type ImagePolicySpec struct { // +kubebuilder:validation:Required // +required // +listType=set - Scopes []ImageScope `json:"scopes"` + Scopes []string `json:"scopes"` // policy defines the verification policy for the items in the scopes list // +kubebuilder:validation:Required // +required Policy Policy `json:"policy"` } -// ImageScope is the item of the scopes list. -// +kubebuilder:validation:Pattern=`^\*(?:\.(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]))+$|^((((?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])(?:\.(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]))+(?::[0-9]+)?)|(localhost(?::[0-9]+)?))(?:(?:/[a-z0-9]+(?:(?:(?:[._]|__|[-]*)[a-z0-9]+)+)?)+)?)(?::([\w][\w.-]{0,127}))?(?:@([A-Za-z][A-Za-z0-9]*(?:[-_+.][A-Za-z][A-Za-z0-9]*)*[:][[:xdigit:]]{32,}))?$` -type ImageScope string - // Policy defines the verification policy for the items in the scopes list. type Policy struct { // rootOfTrust specifies the public key, the Fulcio certificate and the Rekor public key. @@ -131,7 +125,7 @@ type Policy struct { RootOfTrust PolicyRootOfTrust `json:"rootOfTrust"` // fulcioSubject specifies OIDC issuer and the email of the fulcio authentication configuration. - // Required if rootOfTrust.fulcioCAData is set. + // Required if rootOfTrust is based on fulcioCA. // +optional FulcioSubject PolicyFulcioSubject `json:"fulcioSubject,omitempty"` @@ -140,22 +134,33 @@ type Policy struct { SignedIdentity PolicyIdentity `json:"signedIdentity,omitempty"` } -// PolicyRootOfTrust defines the public key, the Fulcio certificate and the Rekor public key. +// PolicyRootOfTrust defines the root of trust based on the policyType. +// +union type PolicyRootOfTrust struct { - // keyData contains inline base64 encoded data of the public key. - // May be omitted when the image was signed with a keyless signature. - // Requires only one of keyData and fulcioCAData must be set. - // +optional - KeyData string `json:"keyData,omitempty"` + // +kubebuilder:validation:Required + // +kubebuilder:validation:Enum=PublicKey;FulcioCAWithRekor + PolicyType string `json:"policyType,omitempty"` + PublicKey PublicKey `json:"publicKey,omitempty"` + FulcioCAWithRekor FulcioCAWithRekor `json:"fulcioCAWithRekor,omitempty"` +} - // fulcioCAData contains inline base64 data for the fulcio CA certificate. - // Requires only one of keyData and fulcioCAData must be set. - // +optional - FulcioCAData string `json:"fulcioCAData,omitempty"` +// PublicKey defines the root of trust based on public key. +type PublicKey struct { + // +kubebuilder:validation:Required + // keyData contains inline base64 encoded data of the public key. + KeyData string `json:"keyData,omitempty"` + // rekorKeyData contains inline base64 data of the Rekor public key. + // +optional + RekorKeyData string `json:"rekorKeyData,omitempty"` +} - // rekorKeyData contains inline base64 data of the Rekor public key. - // Required if fulcioCAData is set. - // +optional +// FulcioCAWithRekor defines the root of trust based on the Fulcio CA certificate and the Rekor public key. +type FulcioCAWithRekor struct { + // +kubebuilder:validation:Required + // fulcioCAData contains inline base64 data for the fulcio CA certificate. + FulcioCAData string `json:"fulcioCAData,omitempty"` + // +kubebuilder:validation:Required + // rekorKeyData contains inline base64 data of the Rekor public key. RekorKeyData string `json:"rekorKeyData,omitempty"` } @@ -191,10 +196,10 @@ type PolicyIdentity struct { MatchPolicy IdentityMatchPolicy `json:"matchPolicy,omitempty"` // exactRepository is required if matchPolicy is set to "ExactRepository". // +optional - PolicyMatchExactRepository *ExactRepository `json:"exactRepository,omitempty"` + PolicyMatchExactRepository *PolicyMatchExactRepository `json:"exactRepository,omitempty"` // remapIdentity is required if matchPolicy is set to "RemapIdentity". // +optional - PolicyMatchRemapIdentity *RemapIdentity `json:"remapIdentity,omitempty"` + PolicyMatchRemapIdentity *PolicyMatchRemapIdentity `json:"remapIdentity,omitempty"` } // IdentityMatchPolicy defines the type of matching for setting "matchPolicy". @@ -230,12 +235,6 @@ const ( // +k8s:deepcopy-gen=true type ImagePolicyStatus struct { - - // policyJSON contains the whole policy applied to the kubernetes namespace - // which got written to disk. This includes cluster-wide policies - // from the `openshift-config` namespace as well. - PolicyJSON string `json:"policyJSON,omitempty"` - // conditions provide details on the status of this API Resource. // +listType=map // +listMapKey=type @@ -245,16 +244,16 @@ type ImagePolicyStatus struct { ### Implementation Details/Notes/Constraints [optional] -#### Update container runtime config controller to watch ImagePolicy +#### Update container runtime config controller to watch ClusterImagePolicy and ImagePolicy -MCO container runtime config controller ImagePolicy syncHandler updates the signature verification configurations: -- Retrieves all the ImagePolicy instances on the cluster. -- Rre-validation: parse the images to validate the policy is not for scopes under OCP product repo. Report the error to machine config logs and the `status` field of that ImagePolicy CR. -- Validates conflicting data from ImagePolicy instances. Ignore the policies from non global namespace CRs if the sigstore policy for the same image scope has been configured under global(openshift-config) namespace. Log this to the `status` field of both global and non global CRs. -- Adds following configurations to a single machine config - - merges the instances for each namespace into a single [containers-policy.json](https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md) in a predefined `/path/to/policies/\.json`. - ImagePolicy CR should not change the `default` field of the cluster-wide policy.json. And the `default` of each \.json will be the same as the cluster-wide `default` in /etc/containers/policy.json. The `default` might be `insecureAcceptAnything` or `reject`, the ImagePolicy handler needs to inherits the `default` value from Image CR handler. - - adds [/etc/containers/registries.d/*.yaml](https://github.com/containers/image/blob/main/docs/containers-registries.d.5.md) to allow matching sigstore signatures, for example: +Enhance the MCO container runtime config controller to manage ClusterImagePolicy and ImagePolicy CRs, and update the signature verification configurations: +- Retrieves all the ClusterImagePolicy and ImagePolicy instances on the cluster. +- Pre-validation: + - Ensure that there are no scopes under the OCP image payload. If any are found, add an info-level log message to the machine config controller about the error encountered while adding an OCP image payload. Update the `status` of the CR to indicate that the policy will not be applied. + - Check for conflicts between cluster scope and namespace scope policies. If the namespaced ImagePolicy scope is equal to or nests inside an existing cluster-scoped ClusterImagePolicy CR, do not deploy the namespaced policy. + Update the `status` of both CRs and the machine config controller logs to indicate that the ClusterImagePolicy will be applied, while the non-global ImagePolicy will not be applied. +- Adds following configurations to machine configs + - machine config `99--generated-imagepolicies` adds [/etc/containers/registries.d/*.yaml](https://github.com/containers/image/blob/main/docs/containers-registries.d.5.md) to allow matching sigstore signatures, for example: ```yaml docker: @@ -268,70 +267,72 @@ MCO container runtime config controller ImagePolicy syncHandler updates the sign default-docker: use-sigstore-attachments: true ``` -- The functions ImagePolicy syncHandler will be insert to current syncImageConfig() to coordinate with Image CR on editing `/etc/containers/policy.json`. -- Merge rules for CRs and json string global policy: - - when the syncHandler processing an image scope of an `openshift-config` namespace CR - - the scope exists in an existing `openshift-config` namespace CR: - append the policy to existing policy; the policy will be written to `/etc/containers/policy.json` and each `\.json` if that namespace has ImagePolicy CR can successfully rollout. - - the scope already exists in a non-global namespace CR: do not roll out the non-global policy to `\.json`. The policy for `openshift-config` will be written to `/etc/containers/policy.json` and `\.json`; update the `status` of both CRs and machine config controller logs that the policy for `openshift-config` is applied, the non-global namespace policy is ignored. - - if the above cases do not apply: the policy will be written to `/etc/containers/policy.json` and each `\.json` if that namespace has ImagePolicy CR can successfully rollout. - - when the syncHandler processing an image scope of a non-global namespaced CR - - the scope exists in an existing `openshift-config` namespace CR: do not roll out the non-global policy in `\.json`. the policy for `openshift-config` will be written to `/etc/containers/policy.json` and `\.json`; update the `status` of both CRs and machine config controller logs that the policy for `openshift-config` is applied, the non-global namespace policy is ignored. - - the scope exists in another non-global namespace CR: - append the policy to existing policy; the policy will be written to `\.json` - - the above cases do not apply: +- Merge rule for ClusterImagePolicy and ImagePolicy CRs: + - when the syncHandler processing an image scope within a ClusterImagePolicy CR + - the scope exists in an existing ClusterImagePolicy CR: + append the policy to existing cluster policy; the policy will be written to `/etc/containers/policy.json` and each `\.json` if that namespace has an ImagePolicy CR that can be successfully rolled out. + - if the scope is either equal to or a broader scope than one already present in an ImagePolicy CR: do not roll out the non-global policy in `.json`. The policy for the cluster will be written to `/etc/containers/policy.json` and `.json`; + update the `status` of both CRs and machine config controller logs to indicate that the policy for cluster has been applied, while the non-global namespace policy is ignored. + - if none of the above cases apply: the policy will be written to `/etc/containers/policy.json` and each `\.json` if that namespace has an ImagePolicy CR that can be successfully rolled out. + - when the syncHandler processing an image scope within an ImagePolicy CR + - if the policy scope is equal to or nests inside an existing ClusterImagePolicy CR: do not roll out the non-global policy in `\.json`. the policy for the cluster will be written to `/etc/containers/policy.json` and `\.json`; + update the `status` of both CRs and machine config controller logs to indicate that the policy for the cluster has been applied, while the non-global namespace policy is ignored. + - the scope exists in another ImagePolicy CR: + append the policy to existing policy; the policy will be written to `.json` + - if none of the above cases apply: the policy will be written to `/path/to/policies/\.json` - the policies will be coordinated with the base [/etc/containers/policy.json](https://github.com/openshift/machine-config-operator/blob/master/templates/master/01-master-container-runtime/_base/files/policy.yaml) file or the Image CR, inheriting the `default` policy from them. If the signature policy scope is also configured as `insecureAcceptAnything` or `reject`, the rollout will fail. In such a case, the error will be reported to the machine config logs. - the `/etc/containers/policy.json` holds the cluster wide policy. `\.json` holds the merged cluster override policy and namespaced policy. -- For namespaced ImagePolicy CRs, updates the merged cluster override policy and namespaced policy existed in `\.json` to `status` of each ImagePolicy CR as a json string. For openshift-config namespace ImagePolicy CRs, updates the merged cluster override policy existed in `/etc/containers/policy.json` to `status` of each ImagePolicy CR as a json string. -- Image policies that are written to `/etc/containers/policy.json` will be rolled out by machine config `99--generated-registries`. Merged policies that are written to `\.json` will be rolled out by machine config `99--generated-imagepolicies`. -- Once an ImagePolicy object gets created/updated, container runtime config controller will call the syncHandler, create machine config `99--generated-imagepolicies` if the namespaced CR can successfully rollout. -Once the last ImagePolicy object gets deleted, container runtime config controller will delete the machine config `99-[pool]-generated-imagepolicies` and run the syncHandler to coordinate policy updates with the Image CR. +- Image policies that are written to `/etc/containers/policy.json` will be rolled out by machine config `99--generated-registries`. Merged policies that are written to `.json` will be rolled out by machine config `99--generated-imagepolicies`. +- Once an ImagePolicy or ClusterImagePolicy object gets created/updated, container runtime config controller will call the syncHandler, create machine config `99--generated-imagepolicies` if the namespaced CR can successfully rollout. +Once the last ImagePolicy or ClusterImagePolicy object get deleted, container runtime config controller will delete the machine config `99-[pool]-generated-imagepolicies`. It then runs the syncHandler to coordinate policy updates with the Image CR. #### Example of ImagePolicy CRs -Example imagepolicies for `openshift-config` and `testnamespace`. +Example of ClusterImagePolicy and ImagePolicy. ```yaml -kind: ImagePolicy +kind: ClusterImagePolicy metadata: name: mypolicy-0 - namespace: openshift-config spec: scopes: - test0.com policy: rootoftrust: - fulciocadata: dGVzdC1jYS1kYXRhLWRhdGE= - rekorkeydata: dGVzdC1yZWtvci1rZXktZGF0YQ== + policyType: FulcioCAWithRekor + fulcioCAWithRekor: + fulciocadata: dGVzdC1jYS1kYXRhLWRhdGE= + rekorkeydata: dGVzdC1yZWtvci1rZXktZGF0YQ== fulciosubject: oidcissuer: https://OIDC.example.com signedemail: test-user@example.com signedidentity: matchpolicy: RemapIdentity remapIdentity: - remapprefix: test-remap-prefix - remapsignedprefix: test-remap-signed-prefix + prefix: test-remap-prefix + signedPrefix: test-remap-signed-prefix ``` ```yaml -kind: ImagePolicy +kind: ClusterImagePolicy metadata: name: mypolicy-1 - namespace: openshift-config spec: scopes: - - test0.com # this policy for test0.com will be append to the mypolicy-0 + - test0.com # this policy for test0.com and the policy from mypolicy-0 will be appended together - test1.com policy: rootoftrust: - keydata: dGVzdC1rZXktZGF0YQ== - rekorkeydata: dGVzdC1yZWtvci1rZXktZGF0YQ== + policyType: PublicKey + publicKey: + keydata: dGVzdC1rZXktZGF0YQ== + rekorkeydata: dGVzdC1yZWtvci1rZXktZGF0YQ== signedidentity: matchpolicy: RemapIdentity remapIdentity: - remapprefix: test-remap-prefix - remapsignedprefix: test-remap-signed-prefix + prefix: test-remap-prefix + signedPrefix: test-remap-signed-prefix ``` ```yaml @@ -341,11 +342,13 @@ metadata: namespace: testnamespace spec: scopes: - - test0.com # for test0.com, openshift-config policy will overwrite this policy + - test0.com # for test0.com, cluster policy will overwrite this policy - test2.com policy: rootoftrust: - keydata: dGVzdC1rZXktZGF0YQ== + policyType: PublicKey + publicKey: + keydata: dGVzdC1rZXktZGF0YQ== ``` Apply the above CRs, if no Image CRs changes the policy.json. The below `/etc/containers/policy.json` will be rolled out. The condensed json string of the file will be updated to the `status.policyJSON` of `openshift-config` CR: @@ -408,7 +411,8 @@ Apply the above CRs, if no Image CRs changes the policy.json. The below `/etc/co } ``` -The merged cluster override policy and namespaced policy in the below `/path/to/policies/testnamespace.json` will be rolled out. The condensed json string of the file will be updated to the `status.policyJSON` of `mypolicy-2` CR: +The merged cluster override policy and namespaced policy in the below `/path/to/policies/testnamespace.json` will be rolled out. + ```json { "default": [ @@ -477,15 +481,11 @@ The merged cluster override policy and namespaced policy in the below `/path/to/ } ``` -#### Notes - -ImagePolicy is defined as namespaced CRD. `namespace: openshift-config` is used for cluster policy. - ### Risks and Mitigations -Risk: The cluster scoped and namespace policies overwrite +Risk: The ClusterImagePolicy and ImagePolicy policies overwrite -Mitigation: MCO container runtime config controller makes sure namespaced policies for an image cannot override global definitions when merging cluster openshift-config policies and namespaced policies into status field of ImagePolicy instances. +Mitigation: The [Implementation Details](#Update-container-runtime-config-controller-to-watch-ClusterImagePolicy-and-ImagePolicy) merge rule makes sure ImagePolicy for an image cannot override global definitions when merging ClusterImagePolicy and ImagePolicy. ### Drawbacks @@ -498,7 +498,7 @@ Mitigation: MCO container runtime config controller makes sure namespaced polici **Note:** *Section not required until targeted at a release.* MCO container runtime config controller can add unit tests and e2e tests. -- unit test: verify the policies from ImagePolicy instances merged correctly and the controller writes correct format policy.json +- unit test: verify the policies from ClusterImagePolicy and ImagePolicy instances merged correctly and the controller writes correct format policy.json - e2e test: - verify the MCO writes configuration to the correct location. - test demonstrating that an unsigned image in violation of the policy is rejected @@ -535,45 +535,45 @@ Not applicable. ### Upgrade / Downgrade Strategy Upgrade expectations: -- In order to use ImagePolicy, an existing cluster will have to make an upgrade to the version that the `ImagePolicy` CRD is available. -- If the existing cluster manages `/etc/containers/policy.json` via the MachineConfig, the configuration will continue to work if no new `ImagePolicy` resources created during the upgrade. If `ImagePolicy` resources created and resulting `/path/to/policies/\.json` exists, CRI-O will use the config from `/path/to/policies/\.json`. +- In order to use ClusterImagePolicy and ImagePolicy, an existing cluster will have to make an upgrade to the version that the `ClusterImagePolicy` and `ImagePolicy` CRDs are available. +- If the existing cluster manages `/etc/containers/policy.json` via the MachineConfig, the configuration will continue to work if no new `ClusterImagePolicy` or `ImagePolicy` resources created during the upgrade. If `ImagePolicy` resources created and resulting `/path/to/policies/.json` exists, CRI-O will use the config from `/path/to/policies/.json`. Downgrade expectations: - Not applicable. - If `N` does not support ImagePolicy CRD, `N+1`supports IamgePolicy. It is not expected that the ImagePolicy related to the failure `N->N+1`. New resources should not be created during `N->N+1`. + If `N` does not support ClusterImagePolicy and ImagePolicy CRD, `N+1`supports ClusterImagePolicy and ImagePolicy. It is not expected that the CRDs related to the failure `N->N+1`. New resources should not be created during `N->N+1`. ### Version Skew Strategy -The `ImagePolicy` involve coordinating behavior in the CRI-O v1.28. The node team will not expose the `ImagePolicy` CRD if the CRI-O/MCO does not support the namespaced signature to avoid the version skew with CRI-O. +The implementation of `ClusterImagePolicy` and `ImagePolicy` is synchronized with CRI-O v1.28. To prevent version mismatches with CRI-O, the node team plans to target the release of CRI-O/MCO simultaneously. During an upgrade, the CRs will take effect only after both CRIO and MCO have been upgraded to versions that support the CRDs. ### Operational Aspects of API Extensions #### Failure Modes - CR field syntax error failure reported from CLI -- CR conflicting value failure: rollout failure according to the merge rules [Implementation Details](#Update-container-runtime-config-controller-to-watch-ImagePolicy). The failure will be reported by machine config controller logs +- CR conflicting value failure: rollout failure according to the merge rules [Implementation Details](#Update-container-runtime-config-controller-to-watch-ClusterImagePolicy-and-ImagePolicy). The failure will be reported by machine config controller logs - MCO rolling out configuration file failure reported by MCO - If the signature validation fails, then CRI-O will report that in the same way as for any other pull failure. Further enhancements to the kubelet, CRI and CRI-O error handling can be achieved in future Kubernetes releases. -The above errors should not impact the overall cluster health -OCP Node team is likely to be called upon in case of escalation with one of the failure modes. +The above errors should not impact the overall cluster health since configuring policies for the images in the OCP payload is prohibited. +The OCP Node team is likely to be called upon in case of escalation with one of the failure modes. #### Support Procedures - detect the failure modes in a support situation, describe possible symptoms (events, metrics, alerts, which log output in which component) - - If the ImagePolicy CR does not follow the syntax requirements, api-server will fail when creating the objects. - - If the ImagePolicy CR is not rolled out by MCO, machine-config-controller logs will shows error with the reason. + - If the ClisterImagePolicy, ImagePolicy CR does not follow the syntax requirements, api-server will fail when creating the objects. + - If the ClusterImagePolicy, ImagePolicy CR is not rolled out by MCO, machine-config-controller logs will shows error with the reason. - disable the API extension (e.g. remove MutatingWebhookConfiguration `xyz`, remove APIService `foo`) - Disabling/removing the CRD is not possible without removing the CR instances. Customer will lose data. - What consequences does it have on existing, running workloads? - - No impact. CRI-O does not verify signature verification for running containers. + - No impact. CRI-O does not verify the signatures of container images while they are running. - What consequences does it have for newly created workloads? - - In order to verify signature for new pods, ImagePolicy instances should be successfully rolled out before creating new workloads. + - In order to verify signature for new pods, ClusterImagePolicy, ImagePolicy instances should be successfully rolled out before creating new workloads. - Does functionality fail gracefully and will work resume when re-enabled without risking - User can create another objects if previous objects failed to roll out.