Add e2e support for setting backend via Policy #630

mszostok · 2022-02-07T19:06:03Z

Description

Changes proposed in this pull request:

Changes

Logger aware Argo Renderer

Update CRD:

output:
  typeInstances:
    backend:
    id: "123" # backend ref in which TI is stored
    abstract: true/false # whether ref. backend is abstract or not

Propagate backend data into TypeInstance output under Action CRD

Update Engine GraphQL to show backend information for output TypeInstance
- update Go clients
- add unit-test coverage
Update Local Hub GraphQL schemas and examples.
- Add TypeInstanceBackend and handle it properly in Cypher. This is a draft implementation but already gives us a huge benefit as it enabled an option to create e2e test and protect current contract. If we rewrite/refactor Local Hub, we can protect ourselves from regressions.
Ensure that core storage is registered when Local Hub starts.

Add support for setting backend based on TypeRefs via Global Policy and Action Policy (support custom merging):

typeInstance:
  rules:
    - typeRef:
        path: "cap.type.aws.auth.credentials"
        revision: "0.1.0" # optional, if not provided, applies to any revision
      backend:
        id: "00fd161c-01bd-47a6-9872-47490e11f996" # Vault backend storage

Pattern matching:

// GetByTypeRef returns storage backend for a given TypeRef.
// If backend for an explicit TypeRef is not found, the pattern matching is used.
//
// For example, if TypeRef is `cap.type.capactio.examples.message:0.1.0`:
// - cap.type.capactio.examples.*:0.1.0
// - cap.type.capactio.examples.*
// - cap.type.capactio.*:0.1.0
// - cap.type.capactio.*
// - cap.type.*:0.1.0
// - cap.type.*
// - cap.*:0.1.0
// - cap.*
//

add Policy validation.

Extends workflow syntax with capact-outputTypeInstance[*].backend
Update Argo renderer to support backends based on TypeRef or capact-outputTypeInstance[*].backend alias. Once rendered the delegated backend storage is deterministic as it's "hardcoded" to a given ID.
Add integration test for:
Success:
1. Create action with default storage
  - Empty Policy -> uses builtin storage
  - Policy with TypeRef defaults -> uses a given backend ID
2. Create action with Helm storage (requires)
  - Policy with requires under interface.
  - Policy with default requires + override with interface specific requires (Will be done in follow-up PR when global default will be implemented)

Other:

now make dev-cluster(-update) accepts COMPONENTS and BUILD_IMAGES_LIST envs. TODO: update dev doc on website.
fix setting Hub manifests during capact install.

Testing

Integration tests prove that it works but here is also a “manual” opt to run a few examples:

Build Capact CLI: make build-tool-cli and use for all below commands.

Create TypeInstances for update and download:

cat > /tmp/download-ti.yaml << ENDOFFILE
typeInstances:
  - alias: "download"
    typeRef:
      path: cap.type.capactio.capact.validation.download
      revision: 0.1.0
    value:
      key: true
ENDOFFILE
export DOWNLOAD_TI=$(capact ti create -f /tmp/download-ti.yaml -ojson | jq -r '.[] | select(.alias == "download") | .id')
cat > /tmp/update-ti.yaml << ENDOFFILE
typeInstances:
  - alias: "update"
    typeRef:
      path: cap.type.capactio.capact.validation.update
      revision: 0.1.0
    value:
      key: true
ENDOFFILE
export UPDATE_TI=$(capact ti create -f /tmp/update-ti.yaml -ojson | jq -r '.[] | select(.alias == "update") | .id')

Create Helm storage TypeInstance:

cat > /tmp/helm-storage.yaml << ENDOFFILE
typeInstances:
  - alias: "helm"
    typeRef:
      path: cap.type.helm.storage
      revision: 0.1.0
    value:
      url: "ever" # currently, not needed
      acceptValues: false # just for demo purposes
ENDOFFILE
export HELM_STORAGE_TI=$(capact ti create -f /tmp/helm-storage.yaml -ojson | jq -r '.[] | select(.alias == "helm") | .id')

Scenarios

Success:

Create action with default (built-in) storage as Policy is empty and Implementation doesn't use any required backend.

Create Action input:

cat > /tmp/act-input-ti.yaml << ENDOFFILE
typeInstances:
  - name: "testUpdate"
    id: ${UPDATE_TI}
  - name: "testInput"
    id: ${DOWNLOAD_TI}
ENDOFFILE

Run Action

capact act create cap.interface.capactio.capact.validation.action.passing --name test --type-instances-from-file /tmp/act-input-ti.yaml

     `capact act run & capact act watch`

Get Action output TypeInstances:

capact act get test -ojson | jq '.Actions[0].output.typeInstances'

Example output:

[
  {
    "backend": {
      "abstract": true,
      "id": "318b99bd-9b26-4bc1-8259-0a7ff5dae61c"
    },
    "id": "12d18143-49b9-4604-9a4e-eb407ca76c78",
    "typeRef": {
      "path": "cap.type.capactio.capact.validation.upload",
      "revision": "0.1.0"
    }
  },
  {
    "backend": {
      "abstract": true,
      "id": "318b99bd-9b26-4bc1-8259-0a7ff5dae61c"
    },
    "id": "b06c97bc-0546-450f-a34d-8f38966dbea0",
    "typeRef": {
      "path": "cap.type.capactio.capact.validation.update",
      "revision": "0.1.0"
    }
  }
]

Create action with default storage based on typeinstace Policy.

Update Global Policy:

cat > /tmp/type-ref-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.*
    oneOf:
    - implementationConstraints:
        requires:
        - path: cap.core.type.platform.kubernetes
    - implementationConstraints: {}
typeInstance:
  rules:
    - typeRef:
        path: cap.type.capactio.capact.validation.upload
        revision: 0.1.0
      backend:
        id: ${HELM_STORAGE_TI}
        description: Default Hub backend storage via TypeRef
ENDOFFILE
capact policy apply -f /tmp/type-ref-policy.yaml

Create Action input:

cat > /tmp/act-input-ti.yaml << ENDOFFILE
typeInstances:
  - name: "testUpdate"
    id: ${UPDATE_TI}
  - name: "testInput"
    id: ${DOWNLOAD_TI}
ENDOFFILE

Run Action

capact act create cap.interface.capactio.capact.validation.action.passing --name test-type-ref --type-instances-from-file /tmp/act-input-ti.yaml

     `capact act run & capact act watch`

Get Action output TypeInstances:

capact act get test-type-ref -ojson | jq '.Actions[0].output.typeInstances'

Example output:

[
  {
    "backend": {
      "abstract": false,
      "id": "e9d424b3-c55d-4b02-9e92-9c736b96f98e"
    },
    "id": "4fd10ff0-5eca-46db-b95e-3c8bab3767ba",
    "typeRef": {
      "path": "cap.type.capactio.capact.validation.upload",
      "revision": "0.1.0"
    }
  },
  {
    "backend": {
      "abstract": true,
      "id": "318b99bd-9b26-4bc1-8259-0a7ff5dae61c"
    },
    "id": "b06c97bc-0546-450f-a34d-8f38966dbea0",
    "typeRef": {
      "path": "cap.type.capactio.capact.validation.update",
      "revision": "0.1.0"
    }
  }
]

Create Action with Helm storage, which is enforced in Implementation requires section.

Create required TypeInstance:

cat > /tmp/inject-ti.yaml << ENDOFFILE
typeInstances:
  - alias: "inject"
    typeRef:
      path: cap.type.capactio.capact.validation.single-key
      revision: 0.1.0
    value:
      key: true
ENDOFFILE
export INJECT_TI=$(capact ti create -f /tmp/inject-ti.yaml -ojson | jq -r '.[] | select(.alias == "inject") | .id')

Update Global Policy:

cat > /tmp/inject-storage-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.interface.capactio.capact.validation.action.passing
    oneOf:
    - implementationConstraints:
        attributes:
        - path: cap.attribute.capactio.capact.validation.policy.most-preferred
        requires:
        - path: cap.type.capactio.capact.validation.single-key
      inject:
        requiredTypeInstances:
        - description: Test TypeInstance
          id: ${INJECT_TI}
        - description: Helm backend TypeInstance
          id: ${HELM_STORAGE_TI}
    - implementationConstraints:
        path: cap.implementation.capactio.capact.validation.action.passing-a
  - interface:
      path: cap.*
    oneOf:
    - implementationConstraints: {}
typeInstance:
  rules: []
ENDOFFILE
capact policy apply -f /tmp/inject-storage-policy.yaml

Create Action input:

cat > /tmp/act-input-ti.yaml << ENDOFFILE
typeInstances:
  - name: "testUpdate"
    id: ${UPDATE_TI}
  - name: "testInput"
    id: ${DOWNLOAD_TI}
ENDOFFILE

Run Action

capact act create cap.interface.capactio.capact.validation.action.passing --name inject-storage --type-instances-from-file /tmp/act-input-ti.yaml

capact act run & capact act watch

Get Action output TypeInstances:

capact act get inject-storage -ojson | jq '.Actions[0].output.typeInstances'

Example output:

[
  {
    "backend": {
      "abstract": false,
      "id": "e9d424b3-c55d-4b02-9e92-9c736b96f98e"
    },
    "id": "cdf4ff38-f66e-4d03-a6ed-1093f154d3af",
    "typeRef": {
      "path": "cap.type.capactio.capact.validation.upload",
      "revision": "0.1.0"
    }
  }
]

Check with capact ti get that uses were properly set for a given Storage Backends.

Above scenarios can be repeated with Action Policy which will override the Global Policy.

Failures:

Create action with wrong type as storage for TypeRef

cat > /tmp/type-ref-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.*
    oneOf:
    - implementationConstraints:
        requires:
        - path: cap.core.type.platform.kubernetes
    - implementationConstraints: {}
typeInstance:
  rules:
    - typeRef:
        path: cap.type.capactio.capact.validation.upload
        revision: 0.1.0
      backend:
        id: ${DOWNLOAD_TI}
        description: Default Hub backend storage via TypeRef
ENDOFFILE
capact policy apply -f /tmp/type-ref-policy.yaml

capact act create cap.interface.capactio.capact.validation.action.passing --name test-v1 --type-instances-from-file /tmp/act-input-ti.yaml

capact act get test-v1 -oyaml

Expected output:

    message: |-
      Cannot render given action: while rendering Action: while listing ImplementationRevisions for Interface "cap.interface.capactio.capact.validation.action.passing:": while TypeInstance metadata validation after resolving TypeRefs:
      - Metadata for "BackendTypeInstance":
          * Type reference ID: "2f856900-1b03-488e-8212-f7d86c2ae3e2", description: "Default Hub backend storage via TypeRef" is not a Hub storage (will retry - 10/15)

Create action without storage injected as requires part

Update Policy:

cat > /tmp/inject-storage-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.interface.capactio.capact.validation.action.passing
    oneOf:
    - implementationConstraints:
        attributes:
        - path: cap.attribute.capactio.capact.validation.policy.most-preferred
        requires:
        - path: cap.type.capactio.capact.validation.single-key
      inject:
        requiredTypeInstances:
        - description: Test TypeInstance
          id: ${INJECT_TI}
typeInstance:
  rules: []
ENDOFFILE
capact policy apply -f /tmp/inject-storage-policy.yaml

Create action:

capact act create cap.interface.capactio.capact.validation.action.passing --name test-v2 --type-instances-from-file /tmp/act-input-ti.yaml

Get output:

capact act get test-v2 -oyaml

Expected output:

    message: 'Cannot render given action: while rendering Action: while picking ImplementationRevision
      for Interface "cap.interface.capactio.capact.validation.action.passing:": No
      Implementations found with current policy for given Interface (will retry -
      3/15)'

Create action with wrong type as storage injected as requires part

Update Policy

cat > /tmp/inject-storage-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.interface.capactio.capact.validation.action.passing
    oneOf:
    - implementationConstraints:
        attributes:
        - path: cap.attribute.capactio.capact.validation.policy.most-preferred
        requires:
        - path: cap.type.capactio.capact.validation.single-key
      inject:
        requiredTypeInstances:
        - description: Test TypeInstance
          id: ${INJECT_TI}
        - description: Helm backend TypeInstance
          id: ${DOWNLOAD_TI}
ENDOFFILE
capact policy apply -f /tmp/inject-storage-policy.yaml

Create action:

capact act create cap.interface.capactio.capact.validation.action.passing --name test-v3 --type-instances-from-file /tmp/act-input-ti.yaml

Get output:

capact act get test-v3 -oyaml

Expected output:

    message: 'Cannot render given action: while rendering Action: while picking ImplementationRevision
      for Interface "cap.interface.capactio.capact.validation.action.passing:": No
      Implementations found with current policy for given Interface (will retry -
      3/15)'

Wrong capact-outputTypeInstances[].backend type.

First change the backend to injected in https://github.com/mszostok/os-hub-manifests/blob/policy-syntax/type-instances/manifests/implementation/capactio/capact/validation/action/passing-b.yaml#L73. Next, populate updated manifests.

Update Policy:

cat > /tmp/inject-storage-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.interface.capactio.capact.validation.action.passing
    oneOf:
    - implementationConstraints:
        attributes:
        - path: cap.attribute.capactio.capact.validation.policy.most-preferred
        requires:
        - path: cap.type.capactio.capact.validation.single-key
      inject:
        requiredTypeInstances:
        - description: Test TypeInstance
          id: ${INJECT_TI}
        - description: Helm backend TypeInstance
          id: ${HELM_STORAGE_TI}
    - implementationConstraints:
        path: cap.implementation.capactio.capact.validation.action.passing-a
  - interface:
      path: cap.*
    oneOf:
    - implementationConstraints: {}
typeInstance:
  rules: []
ENDOFFILE
capact policy apply -f /tmp/inject-storage-policy.yaml

Create Action

capact act create cap.interface.capactio.capact.validation.action.passing --name test-v4 --type-instances-from-file /tmp/act-input-ti.yaml

Expected output:

    message: 'Cannot render given action: while rendering Action: while noting output
      artifacts: while resolving backend ID for testUpload: TypeInstance with "injected"
      alias is not a Hub storage (will retry - 0/15)'

Missing capact-outputTypeInstances[].backend type.

First change the backend to something in https://github.com/mszostok/os-hub-manifests/blob/policy-syntax/type-instances/manifests/implementation/capactio/capact/validation/action/passing-b.yaml#L73. Next, populate updated manifests.

Update Policy:

cat > /tmp/inject-storage-policy.yaml << ENDOFFILE
interface:
  rules:
  - interface:
      path: cap.interface.capactio.capact.validation.action.passing
    oneOf:
    - implementationConstraints:
        attributes:
        - path: cap.attribute.capactio.capact.validation.policy.most-preferred
        requires:
        - path: cap.type.capactio.capact.validation.single-key
      inject:
        requiredTypeInstances:
        - description: Test TypeInstance
          id: ${INJECT_TI}
        - description: Helm backend TypeInstance
          id: ${HELM_STORAGE_TI}
    - implementationConstraints:
        path: cap.implementation.capactio.capact.validation.action.passing-a
  - interface:
      path: cap.*
    oneOf:
    - implementationConstraints: {}
typeInstance:
  rules: []
ENDOFFILE
capact policy apply -f /tmp/inject-storage-policy.yaml

Create Action

capact act create cap.interface.capactio.capact.validation.action.passing --name test-v5 --type-instances-from-file /tmp/act-input-ti.yaml

Expected output:

    message: 'Cannot render given action: while rendering Action: while noting output
      artifacts: while resolving backend ID for testUpload: cannot find backend storage
      for specified something alias (will retry - 11/15)'

Related issue(s)

Add e2e support for setting backend.id (Policy, Workflow, GraphQL) #624

hack/lib/const.sh

pkg/hub/client/policy_enforced_client.go

pkosiec

Works flawlessly 👌 Awesome work! Just a few minor comments, but they are so minor that I can approve it right now.

test/e2e/action_test.go

pkg/engine/api/graphql/schema.graphql

pkg/engine/k8s/policy/metadata/metadata_resolver.go

pkg/sdk/renderer/argo/dedicated_renderer.go

test/e2e/action_test.go

pkg/sdk/validation/policy/policy.go

pkg/hub/client/public/facade/doc.go

pkosiec

👍 LGTM - I just posted very last comments as I noticed an improvement with quoting all the names (aliases, type references, etc.).

(remember about switching manifest location to the capactio/hub-manifests#main for make dev-cluster)

pkg/sdk/renderer/argo/dedicated_renderer.go

pkg/sdk/validation/policy/policy.go

First version adjusted after rebase with main

fc8b3c7

mszostok added enhancement New feature or request WIP Work in progress area/engine Relates to Engine area/hub Relates to Hub area/hub-manifests Relates to Hub manifests labels Feb 7, 2022

mszostok added this to the 0.7.0 milestone Feb 7, 2022

mszostok requested review from mkuziemko, pkosiec and tmitchell as code owners February 7, 2022 19:06

mszostok added 2 commits February 8, 2022 12:51

Remove setting default storage and relations in renderer

98c11a6

Fix indend in graphql file

3879e09

mszostok force-pushed the policy-syntax/type-instances branch from db40490 to 3879e09 Compare February 8, 2022 14:38

mszostok added 2 commits February 11, 2022 12:18

FINALLYYYY! Add e2e test coverage

90c39c9

Fix abstract propagation, remove custom matcher, add more assertions

31d4f96

mszostok force-pushed the policy-syntax/type-instances branch from 291ef2c to 99f99f1 Compare February 11, 2022 12:42

Extract values to const

c9f0c7f

mszostok force-pushed the policy-syntax/type-instances branch from 99f99f1 to c9f0c7f Compare February 11, 2022 12:49

mszostok mentioned this pull request Feb 11, 2022

Add built-in and Helm storage Types, adjust validation manifests capactio/hub-manifests#57

Merged

mszostok force-pushed the policy-syntax/type-instances branch from e78a8ae to 9848484 Compare February 11, 2022 14:12

mszostok commented Feb 11, 2022

View reviewed changes

hack/lib/const.sh Outdated Show resolved Hide resolved

mszostok changed the title ~~First version adjusted after rebase with main~~ Add e2e support for setting backend via Policy Feb 11, 2022

Self review, change hub-manifests repo

7066856

mszostok force-pushed the policy-syntax/type-instances branch from 9848484 to 7066856 Compare February 11, 2022 14:50

mszostok added 3 commits February 12, 2022 23:59

Adjust integration tests

18fe3d3

Fix e2e tests

777d7d6

Add seed to rand

5a4a9b9

mszostok force-pushed the policy-syntax/type-instances branch from 05e7b08 to 5a4a9b9 Compare February 13, 2022 19:02

Fix bug in install scripts

d862280

mszostok mentioned this pull request Feb 13, 2022

Add e2e support for setting backend.id (Policy, Workflow, GraphQL) #624

Closed

Use commit sha as branch with / is not supported

7c8a5e4

mszostok force-pushed the policy-syntax/type-instances branch from 0e1ff6b to 7c8a5e4 Compare February 13, 2022 22:08

mszostok removed the WIP Work in progress label Feb 13, 2022

Add more logging, fix panic

86f0be8

This was referenced Feb 15, 2022

Handle multiple backends for the TypeInstance upload from workflow #634

Closed

Add common TypeInstance injection #635

Closed

mszostok commented Feb 15, 2022

View reviewed changes

pkg/hub/client/policy_enforced_client.go Outdated Show resolved Hide resolved

pkosiec approved these changes Feb 16, 2022

View reviewed changes

Apply changes after review

020110a

mszostok force-pushed the policy-syntax/type-instances branch from cf144d8 to 020110a Compare February 16, 2022 17:35

pkosiec approved these changes Feb 17, 2022

View reviewed changes

pkg/sdk/renderer/argo/dedicated_renderer.go Outdated Show resolved Hide resolved

pkg/sdk/renderer/argo/dedicated_renderer.go Outdated Show resolved Hide resolved

pkg/sdk/validation/policy/policy.go Show resolved Hide resolved

Add quotes, change manifests to official ones

ca1c551

mszostok merged commit ee49421 into capactio:main Feb 17, 2022

mszostok deleted the policy-syntax/type-instances branch February 17, 2022 08:54

pkosiec mentioned this pull request Feb 28, 2022

Add common TypeInstance injection #646

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e support for setting backend via Policy #630

Add e2e support for setting backend via Policy #630

mszostok commented Feb 7, 2022 •

edited

Loading

pkosiec left a comment

pkosiec left a comment

Add e2e support for setting backend via Policy #630

Add e2e support for setting backend via Policy #630

Conversation

mszostok commented Feb 7, 2022 • edited Loading

Description

Changes

Testing

Scenarios

Above scenarios can be repeated with Action Policy which will override the Global Policy.

Related issue(s)

pkosiec left a comment

Choose a reason for hiding this comment

pkosiec left a comment

Choose a reason for hiding this comment

mszostok commented Feb 7, 2022 •

edited

Loading