fix: make notation verifier installation optional on ratify installation

[shahramk64]

Description

What this PR does / why we need it:

It enables Ratify to not install notation by default. It gives the option that the notation verifier installation is disabled at the time Ratify installs.

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):

Fixes #1451

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Helm Chart Change (any edit/addition/update that is necessary for changes merged to the main branch)
• This change requires a documentation update

How Has This Been Tested?

The following scenarios are being tested to make sure the right report is generated:
a) When both notation and cosign are enabled:
1. when the image is signed by both notation and cosign:
2. when the image is signed by just notation:
3. when the image is signed by just cosign:
4. when the image is unsigned:

b) When only notation is enabled:
1. when the image is signed by both notation and cosign:
2. when the image is signed by just notation:
3. when the image is signed by just cosign:
4. when the image is unsigned

c) When only cosign is enabled:
1. when the image is signed by both notation and cosign:
2. when the image is signed by just notation:
3. when the image is signed by just cosign:
4. when the image is unsigned

d) When neither notation nor cosign is enabled:
1. when the image is signed by both notation and cosign:
2. when the image is signed by just notation:
3. when the image is signed by just cosign:
4. when the image is unsigned

Checklist:

• Does the affected code have corresponding tests?
• Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
• Does this introduce breaking changes that would require an announcement or bumping the major version?
• Do all new files have appropriate license header?

Post Merge Requirements

• MAINTAINERS: manually trigger the "Publish Package" workflow after merging any PR that indicates Helm Chart Change

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

see 12 files with indirect coverage changes

[susanshi]

discussed in the PR reviewing meeting, we should have a warning ( if possible) if there is no verifier is configured. Please also validate when there are no verifiers configured, Ratify produces a verification response with FAILed status.

[susanshi]

we don't want to block the installation on 0 verifiers, we will try our test to warn if possible.

[shahramk64]

Here is the result of testing different scenarios:

1. When both notation and cosign are enabled:

• when the image is signed by both notation and cosign:
  Result: success
  Verification report contains both notation and cosign result

• when the image is signed by just cosign:
  Result: Success
  Verification report contains the cosign result

• when the image is signed by just notation:
  Result: Success
  Verification report contains the notation result

• when the image is unsigned:
  Result: Denied
  Command line error:
  Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801
  Verification report: (from the logs)

{ 
"verifierReports": [
   {
     "subject": "skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801",
     "isSuccess": false,
     "message": "verification failed: Error: no verifier report, Code: NO_VERIFIER_REPORT, Component Type: executor, Description: No verifier report was generated. This might be due to various factors, such as lack of artifacts attached to the image, a misconfiguration in the Referrer Store preventing access to the registry, or the absence of appropriate verifiers corresponding to the referenced image artifacts."
   }
 ]
}

2. When only notation is enabled. Ratify was uninstalled, and the values.yaml file was updated to enable notation and disable cosign, then reinstalled Ratify.

• when the image is signed by both notation and cosign:
  Result: success
  Verification report contains both notation and cosign result

• when the image is signed by just notation:
  Result: Success
  Verification report contains the notation result

• when the image is signed by just cosign:
  Result: Denied:
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: skal21cr.azurecr.io/ratify-demo/net-monitor_cosign@sha256:64ff594285365b782b564581a1653236c5b1d989fb4c3547ad12c5d0eae2c3a8
  Verification report: (from the logs)

{
"verifierReports": [
   {
     "subject": "skal21cr.azurecr.io/ratify-demo/net-monitor_cosign@sha256:64ff594285365b782b564581a1653236c5b1d989fb4c3547ad12c5d0eae2c3a8",
     "isSuccess": false,
     "message": "verification failed: Error: no verifier report, Code: NO_VERIFIER_REPORT, Component Type: executor, Description: No verifier report was generated. This might be due to various factors, such as lack of artifacts attached to the image, a misconfiguration in the Referrer Store preventing access to the registry, or the absence of appropriate verifiers corresponding to the referenced image artifacts."
   }
 ]
}

• when the image is unsigned
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801
  Verification report: (from the logs)

{
"verifierReports": [
   {
     "subject": "skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801",
     "isSuccess": false,
     "message": "verification failed: Error: no verifier report, Code: NO_VERIFIER_REPORT, Component Type: executor, Description: No verifier report was generated. This might be due to various factors, such as lack of artifacts attached to the image, a misconfiguration in the Referrer Store preventing access to the registry, or the absence of appropriate verifiers corresponding to the referenced image artifacts."
   }
 ]
}

3. When only cosign is enabled:

• when the image is signed by both notation and cosign:
  Result: success
  Verification report contains both notation and cosign result

• when the image is signed by just cosign:
  Result: Success
  Verification report contains the cosign result

• when the image is signed by just notation:
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: skal21cr.azurecr.io/ratify-demo/net-monitor_notation@sha256:59c9f99c37a0bbf5266ad36cd37da1de911e4f196e574f68caa0d1e1937ea3cc
  Verification report (from the logs):

{
 "verifierReports": [
   {
     "subject": "skal21cr.azurecr.io/ratify-demo/net-monitor_notation@sha256:59c9f99c37a0bbf5266ad36cd37da1de911e4f196e574f68caa0d1e1937ea3cc",
     "isSuccess": false,
     "message": "verification failed: Error: no verifier report, Code: NO_VERIFIER_REPORT, Component Type: executor, Description: No verifier report was generated. This might be due to various factors, such as lack of artifacts attached to the image, a misconfiguration in the Referrer Store preventing access to the registry, or the absence of appropriate verifiers corresponding to the referenced image artifacts."
   }
 ]
}

• when the image is unsigned
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801
  Verification report (from the logs):

{
"verifierReports": [
    {
      "subject": "skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801",
      "isSuccess": false,
      "message": "verification failed: Error: no verifier report, Code: NO_VERIFIER_REPORT, Component Type: executor, Description: No verifier report was generated. This might be due to various factors, such as lack of artifacts attached to the image, a misconfiguration in the Referrer Store preventing access to the registry, or the absence of appropriate verifiers corresponding to the referenced image artifacts."
    }
  ]
}

4. When neither notation nor cosign is enabled:

• when the image is signed by both notation and cosign:
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Error validating one or more images: ["skal21cr.azurecr.io/ratify-demo/net-monitor@sha256:6ac5db7cf3f731fd817643eeccd2b28650324660eb69dfc670a88fd644f07d9b", "Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier"]
  Verification report (from the logs):
  time=2024-08-27T07:28:55.372130209Z level=error msg=Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier component-type=server go.version=go1.21.10 namespace= trace-id=e2757580-8155-4309-ab77-71c2a3e33091

• when the image is signed by just notation:
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Error validating one or more images: ["skal21cr.azurecr.io/ratify-demo/net-monitor_notation@sha256:59c9f99c37a0bbf5266ad36cd37da1de911e4f196e574f68caa0d1e1937ea3cc", "Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier"]
  Verification report (from the logs):
  time=2024-08-27T07:30:03.010427881Z level=error msg=Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier component-type=server go.version=go1.21.10 namespace= trace-id=185b69e4-166e-4a87-9037-dc3c88121548

• when the image is signed by just cosign:
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Error validating one or more images: ["skal21cr.azurecr.io/ratify-demo/net-monitor_cosign@sha256:64ff594285365b782b564581a1653236c5b1d989fb4c3547ad12c5d0eae2c3a8", "Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier"]
  Verification report (from the logs):
  time=2024-08-27T07:31:01.203722761Z level=error msg=Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier component-type=server go.version=go1.21.10 namespace= trace-id=93d46277-c139-4a5e-a8ab-09c609226ed7

• when the image is unsigned
  Result: Denied
  Command line error: Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Error validating one or more images: ["skal21cr.azurecr.io/ratify-demo/net-watcher@sha256:13e4eaae14654cfb02b7975b74f8af42a1d89e5c79cc1eebb56b66b8516dc801", "Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier"]
  Verification report (from the logs):
  time=2024-08-27T07:31:55.420565189Z level=error msg=Error: config invalid, Code: CONFIG_INVALID, Component Type: verifier, Detail: verifiers config should have at least one verifier component-type=server go.version=go1.21.10 namespace= trace-id=0e7db5af-e668-4dbc-9ee5-8cee4f981e02

[shahramk64]

@susanshi Should we only check for cosign and notation here? Or sbom should also be included?

[susanshi]

all external plugin should be included too

[susanshi]

curious: does this block print out a warning on helm install?

[shahramk64]

Yes, Helm doesn't provide a straightforward way to issue a warning that doesn't halt execution, initially I was just printing a string based on the same condition on top of the verifier.yaml file. But it doesn't work, because Helm treated the printed warning message as part of the Kubernetes resource definition.
WE can leverage Helm's NOTES.txt file, which is specifically designed for displaying messages after a successful installation or rendering. While this doesn't show up directly during the template rendering phase, it will ensure the user sees the message right after installation.

[binbin-li]

nit: NOTES can be removed as this would be auto-generated by helm. Otherwise it will look like:

[binbin-li]

we can add check on vulnerabilityreport as well

[binbin-li]

lgtm