Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Keys, Script and Signing Service to dev environment #1076

Closed
wants to merge 23 commits into from

Conversation

rochacbruno
Copy link
Member

@rochacbruno rochacbruno commented Dec 9, 2021

This PR adds the following features

  • Dev environment GPG keys
  • Dev environment Signing Script
  • Dev environment Signing Service creation
  • (the same as above for Github CI test runs)
  • Settings variables for signing configuration
  • Auto sign on collection approval
  • endpoints to perform on-demand signing
  • signatures exposed on v3/collection/ endpoint (for CLI)
  • signatures and sign states exposed on ui/v1/ endpoints (for UI)
  • settings and feature flags exposed on /settings/ and /feature-flags/ (for UI)
  • permissions exposed on /me/ (for UI)
  • Filtering by sign state (for UI endpoint)
  • Logging for collection signatures
  • Setup Signing Service on ephemeral (by @drodowic )
  • Functional testing for collection signatures
  • Sync signed_only content field on Remote configuration

metadata

Issue: AAH-1181
env:LOCK_REQUIREMENTS=0
env:PULP_ANSIBLE_REVISION=main

@rochacbruno rochacbruno marked this pull request as draft December 9, 2021 19:03
@rochacbruno
Copy link
Member Author

rochacbruno commented Dec 20, 2021

Signing

gpg --batch --import /tmp/ansible-sign.key &>/dev/null

django-admin shell_plus

ss = AsciiArmoredDetachedSigningService.objects.filter(name="ansible-default").first()
repo = AnsibleRepository.objects.filter(name="published").first()


from pulp_ansible.app.tasks.signature import sign
from pulpcore.plugin.tasking import dispatch

task = dispatch(
    sign, 
    exclusive_resources=[repo], 
    kwargs=dict(
        repository_href=repo.pk, 
        content_hrefs=["*"], 
        signing_service_href=ss.pk
   )
 )

Acessing signatures

Collection Version
GET /api/automation-hub/content/published/v3/collections/tosigntest/tosignf/versions/1.0.0/
HTTP 200 OK
Allow: GET, DELETE, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
{
    "version": "1.0.0",
    "href": "/api/automation-hub/content/published/v3/collections/tosigntest/tosignf/versions/1.0.0/",
    "created_at": "2021-12-16T18:43:21.748529Z",
    "updated_at": "2021-12-16T18:43:21.748551Z",
    "requires_ansible": ">=2",
    "artifact": {
        "filename": "tosigntest-tosignf-1.0.0.tar.gz",
        "sha256": "cd5eb8b5c089dbfff708b37b2bbbe60741fa0cedcb65a5a32ba9674306a74ac3",
        "size": 1574
    },
    "collection": {
        "id": "044ef4c9-7df4-4465-aedb-36c82d97095f",
        "name": "tosignf",
        "href": "/api/automation-hub/content/published/v3/collections/tosigntest/tosignf/"
    },
    "download_url": "http://localhost:5001/api/automation-hub/v3/artifacts/collections/published/tosigntest-tosignf-1.0.0.tar.gz",
    "name": "tosignf",
    "namespace": {
        "name": "tosigntest"
    },
    "signatures": [
        {
            "signature": "-----BEGIN PGP SIGNATURE-----\n\niQHIBAABCAAyFiEE6+0XDoyUgOIqHQWbFSUOnsCmJXcFAmHhg0EUHGdhbGF4eTNA\nYW5zaWJsZS5jb20ACgkQFSUOnsCmJXeKmgv/ed8eSOLr4SMbjP+cJILG5749UqfT\nmUz3+3eDfpe2YJpNGfWavWuMbuYAiujWDXsN46qhOdAFwvdkWxRj2GDK/V+1qB+P\nixtQQZ7Drpc8eEOVYYjFIH8+NoXmoeJCKL9ifOmIgtpapzBt09H1qVTIWuzPhmb0\nSt98nGLcEQeXHxzfebSx06NlKEt0n607hur9nZ+Lwy5I7jZtiEgcyezejQAzoQFv\nwcA52IgOnruyMLP6QdjVPJg/ZzqrvrlLe33zSIGr5271TW4YVgOHzV7aOcUVWpKr\nR4m2GXFEQ+fcEeZf65bprb4ndxFlNlpc4yXiDP3vTvBwKoxzYKdtUzlZmO2zN32A\nozRBClFiw0W++NUL/6zl/cUNGuA0CUBGzsRFF8uOijCFVTZ3pvnF+2hk4RpzUkcc\n52VGddFiFvcPhdMVLNitk2J3qhybGHETi0CP4opDUTHXkZC0hsIOLfRlqg7KxtXl\nl99SrE5bsWnpaVAFRinSGcopRWsd28iQbXvJ\n=8dHK\n-----END PGP SIGNATURE-----\n",
            "pubkey_fingerprint": "EBED170E8C9480E22A1D059B15250E9EC0A62577",
            "signing_service": "ansible-default",
            "pulp_created": "2022-01-14T14:05:53.835605Z"
        }
    ],
    "metadata": {
        "authors": [
            "Orion User 1"
        ],
        "contents": [],
        "dependencies": {
            "tosigntest.tosignh": "<=3.0.0"
        },
        "description": "a collection with some deps on other collections",
        "documentation": "",
        "homepage": "",
        "issues": "",
        "license": [
            "GPL-3.0-or-later"
        ],
        "repository": "http://github.example.com/orionuser1/skeleton",
        "tags": [
            "collectiontest"
        ]
    },
    "manifest": {},
    "files": {}
}

@rochacbruno
Copy link
Member Author

URL to sign collection on demand

curl -X POST \
  'http://0.0.0.0:8002/api/automation-hub/v3/sign/collections/' \
  -H 'Accept: */*' \
  -H 'Authorization: Token a64a0ab93948c92febf2c0611dc1b189f9f37cdd' \
  -H 'Content-Type: application/json' \
  -d '{
    "signing_service": "ansible-default",
    "repository": "published",
    "namespace": "bla",
    "collection": "zaz",
    "version": "1.5.0",
    "content_units": ["*"]
}'

URL (POST ONLY)

http://0.0.0.0:8002/api/automation-hub/v3/sign/collections/

Payloads

Required

{
    "signing_service": "ansible-default",
    "repository": "published",
}

Either content_units is specified (* or a list of UUIDS)

{
    "signing_service": "ansible-default",
    "repository": "published",
    "content_units": ["*"]
}

OR sign all collections and versions under a namespace

{
    "signing_service": "ansible-default",
    "repository": "published",
    "namespace": "bla",
}

sign all versions under a specific collection

{
    "signing_service": "ansible-default",
    "repository": "published",
    "namespace": "bla",
    "collection": "zaz",
}

Sign specific by version specifier

{
    "signing_service": "ansible-default",
    "repository": "published",
    "namespace": "bla",
    "collection": "zaz",
    "version": "1.2.3"
}

@rochacbruno
Copy link
Member Author

rochacbruno commented Jan 19, 2022

Added to _ui/v1 APIs

http://0.0.0.0:5001/api/automation-hub/_ui/v1/collection-versions/?limit=2

Signed collection versions gets a signatures field, this field exposes the fingerprint for the pub_key used in the signature, the SIGNATURE DATA hash is available only on the v3/ API.

"data": [
    {
      "namespace": "test",
      "name": "a",
      "version": "1.0.0",
      "requires_ansible": ">=2",
      "created_at": "2022-01-14T14:04:02.157731Z",
      "metadata": {
        "dependencies": {
          "test.b": "*"
        },
        "contents": [],
        "documentation": "",
        "homepage": "",
        "issues": "",
        "repository": "http://github.example.com/orionuser1/skeleton",
        "description": "a collection with some deps on other collections",
        "authors": [
          "Orion User 1"
        ],
        "license": [
          "GPL-3.0-or-later"
        ],
        "tags": [
          "collectiontest"
        ],
        "signatures": [
          "EBED170E8C9480E22A1D059B15250E9EC0A62577"
        ]
      },
      "contents": [],
      "repository_list": [
        "published"
      ]
    }

On http://0.0.0.0:5001/api/automation-hub/_ui/v1/repo/published/?deprecated=false

There is now 2 fields, the same signatures as above and also a sign_state

"data": [
    {
      "id": "5e094530-4439-4188-9f83-6c278583d981",
      "namespace": {
        "id": 1,
        "name": "test",
        "company": "",
        "email": "",
        "avatar_url": "",
        "description": "",
        "groups": [
          {
            "id": 1,
            "name": "system:partner-engineers",
            "object_permissions": [
              "upload_to_namespace",
              "change_namespace",
              "delete_namespace"
            ]
          }
        ]
      },
      "name": "a",
      "download_count": 0,
      "latest_version": {
        "namespace": "test",
        "name": "a",
        "version": "1.0.0",
        "requires_ansible": ">=2",
        "created_at": "2022-01-14T14:04:02.157731Z",
        "metadata": {
          "dependencies": {
            "test.b": "*"
          },
          "contents": [],
          "documentation": "",
          "homepage": "",
          "issues": "",
          "repository": "http://github.example.com/orionuser1/skeleton",
          "description": "a collection with some deps on other collections",
          "authors": [
            "Orion User 1"
          ],
          "license": [
            "GPL-3.0-or-later"
          ],
          "tags": [
            "collectiontest"
          ],
          "signatures": [
            "EBED170E8C9480E22A1D059B15250E9EC0A62577"
          ]
        },
        "contents": []
      },
      "deprecated": false,
      "sign_state": "signed"
    }

Values are:

  • unsigned: collection doesn't at least one version signed.
  • signed: collection has all versions signed.
  • partial: collection has some versions signed.

c/c @himdel

@himdel
Copy link
Collaborator

himdel commented Jan 19, 2022

Hey, checking the necessary endpoints, I think we're possibly missing a couple of changes:

  • namespace detail / collection list:

    • _ui/v1/repo/published/ (= CollectionAPI.list) - items have .sign_state
      • 🔴 is always partial now, despite the only available versions having been uploaded the same way, after signatures enabled
    • filtering in _ui/v1/repo/published/ by sign state
    • sign all collections can probably use the same endpoint to get the count, filtering for sign_state IN partial,unsigned with limit=1 and using meta.count (needs filtering)
  • collection detail

    • _ui/v1/repo/published/:namespace/:collection/ (= CollectionAPI.getCached) - .latest_version.metadata.signatures
      • ⚠️ this is enough if we're only showing signed state for the currently selected version - we would need a sign_state + all_versions[*].signature combo (or see below) to
        • indicate a partically signed collection (some versions yes some no)
        • show signedness state in the version selector
        • be able to disable the Sign all button
        •     count the number of affected versions for Sign all
      • ❓ is it possible to get multiple signatures in the array? Would we display all of them?
      • 🔴 is always an empty array now
    • _ui/v1/collection-versions/?namespace=...&name=...&repository=... (= CollectionVersionAPI.list with a name + namespace + repository filter - not used yet) (+ &sort=-version)
      • this can be used if we don't add this to all_versions above, at the cost of a second API request that downloads the contents of all versions, and needs disabled pagination to really get all versions (&offset=0&limit=100000)
      • (or, if we can't add it to all_versions, maybe exposing signature info on content/published/v3/collections/arista/avd/versions/ would be better? (less data on that one))
      • sign all versions can use this one to get the count
  • approval dashboard

    • _ui/v1/collection-versions/ (= CollectionVersionAPI.list) - items have .metadata.signatures
      • 🔴 is always an empty array now
  • feature exposed to UI

    • I'm seeing GALAXY_AUTO_SIGN_COLLECTIONS: true and GALAXY_COLLECTION_SIGNING_SERVICE: "ansible-default" returned from the _ui/v1/settings/ (= SettingsAPI.get) endpoint
    • @rochacbruno do I correctly assume that GALAXY_AUTO_SIGN_COLLECTIONS should affect the approval dashboard (Approve -> Sign and approve, and the color of Approved indicators in the list), while GALAXY_COLLECTION_SIGNING_SERVICE affects the rest? Or do we need both? What's a "not supporting signing" value for these please?
  • sign collection, version or namespace

the SIGNATURE DATA hash is available only on the v3/ API

I'm not sure what you mean, is that the hash in the signatures array?
Or is that some other hash the UI would want to consume? (And if so, which v3 api? :))

Add signed-only to remote configuration (API and UI)

Is this also a part of this? That's a UI change as well, right?

can_sign_collections

❓ some new permissions seem to be involved, should we also let admin assign these to groups in the UI? (or is this the other kind?)

This PR adds:

- secret and public keys on dev/common/
- imports and trusts keys on container build time
- add make docker/add-signing-service

TODO:

- [ ] Route URL to spawn sign task (subclass pulp_ansible#754
  serializer)
- [ ] Surface the signature on collectionversion serializer
- [ ] Add test to sign a collection

Issue: AAH-1181

Required PR: pulp/pulp_ansible#754

env:LOCK_REQUIREMENTS=0
env:PULP_CONTAINER_REVISION=39b3000150960c554d2124ab3654e3e7b4c54352
env:PULPCORE_REVISION=f8306ac5d3af1cf9936d39abb0568e86d18cd55f
env:GALAXY_IMPORTER_REVISION=7091519f38acb8e10b85baffe7c6074b02309598
Issue: AAH-1181

Required PR: pulp/pulp_ansible#754
env:LOCK_REQUIREMENTS=0
Issue: AAH-1181

Required PR: pulp/pulp_ansible#754

env:LOCK_REQUIREMENTS=0
env:PULP_CONTAINER_REVISION=39b3000150960c554d2124ab3654e3e7b4c54352
env:PULPCORE_REVISION=f8306ac5d3af1cf9936d39abb0568e86d18cd55f
env:GALAXY_IMPORTER_REVISION=7091519f38acb8e10b85baffe7c6074b02309598
Issue: AAH-1181

Required PR: pulp/pulp_ansible#754

env:LOCK_REQUIREMENTS=0
env:PULP_CONTAINER_REVISION=39b3000150960c554d2124ab3654e3e7b4c54352
env:PULPCORE_REVISION=f8306ac5d3af1cf9936d39abb0568e86d18cd55f
env:GALAXY_IMPORTER_REVISION=7091519f38acb8e10b85baffe7c6074b02309598
tasks on promotion e.g: copy + remove must run on the same
task (or task group) in sequence and using the same locks.

Required PR: pulp/pulp_ansible#754

env:LOCK_REQUIREMENTS=0
- Added tests taken from pulp_ansible PR
- Added Permissions
- Auto sign only when moving to published (avoid sign rejected content)
- Added permissions to /me/ endpoint
- Configs to run tests on CI

Issue: AAH-1181
Required PR: pulp/pulp_ansible#754
env:LOCK_REQUIREMENTS=0
Issue: AAH-1181
Required PR: pulp/pulp_ansible#754
env:LOCK_REQUIREMENTS=0
Issue: AAH-1181
env:LOCK_REQUIREMENTS=0
env:PULP_ANSIBLE_REVISION=master
@rochacbruno
Copy link
Member Author

@brumik

{
  "id": "f006b2b0-e044-4e0f-b167-8c169f0a6d4d",
  "namespace": {
    "id": 5,
    "name": "vittest",
    "company": "",
    "email": "",
    "avatar_url": "",
    "description": "",
    "groups": [
      {
        "id": 1,
        "name": "system:partner-engineers",
        "object_permissions": [
          "upload_to_namespace",
          "change_namespace",
          "delete_namespace"
        ]
      }
    ]
  },
  "name": "vith",
  "download_count": 0,
  "latest_version": {
    "namespace": "vittest",
    "name": "vith",
    "version": "4.0.0",
    "requires_ansible": ">=2",
    "created_at": "2022-01-24T17:04:33.344202Z",
    "metadata": {
      "dependencies": {},
      "contents": [],
      "documentation": "",
      "homepage": "",
      "issues": "",
      "repository": "http://github.example.com/orionuser1/skeleton",
      "description": "a collection with some deps on other collections",
      "authors": [
        "Orion User 1"
      ],
      "license": [
        "GPL-3.0-or-later"
      ],
      "tags": [
        "collectiontest"
      ],
      "signatures": []
    },
    "contents": [],
    "version_sign_state": "unsigned",
    "docs_blob": {
      "contents": [],
      "collection_readme": {
        "html": "<p>This is collection_reqs_test collection</p>",
        "name": "README.md"
      },
      "documentation_files": []
    }
  },
  "all_versions": [
    {
      "version": "4.0.0",
      "created": "2022-01-24 17:04:33.344202+00:00",
      "version_sign_state": "unsigned"
    },
    {
      "version": "3.0.0",
      "created": "2022-01-24 17:04:32.851448+00:00",
      "version_sign_state": "unsigned"
    },
    {
      "version": "2.0.0",
      "created": "2022-01-24 17:04:32.366363+00:00",
      "version_sign_state": "unsigned"
    },
    {
      "version": "1.0.0",
      "created": "2022-01-24 17:04:31.887908+00:00",
      "version_sign_state": "signed"
    }
  ],
  "sign_state": "partial",
  "total_versions": 4,
  "signed_versions": 1,
  "unsigned_versions": 3
}

and

{
  "id": "bd1f6b5c-cf75-4752-af08-c37ec166d38d",
  "namespace": {
    "id": 5,
    "name": "vittest",
    "company": "",
    "email": "",
    "avatar_url": "",
    "description": "",
    "groups": [
      {
        "id": 1,
        "name": "system:partner-engineers",
        "object_permissions": [
          "upload_to_namespace",
          "change_namespace",
          "delete_namespace"
        ]
      }
    ]
  },
  "name": "vith",
  "download_count": 0,
  "latest_version": {
    "namespace": "vittest",
    "name": "vith",
    "version": "5.0.0",
    "requires_ansible": ">=2",
    "created_at": "2022-01-24T17:04:33.904249Z",
    "metadata": {
      "dependencies": {},
      "contents": [],
      "documentation": "",
      "homepage": "",
      "issues": "",
      "repository": "http://github.example.com/orionuser1/skeleton",
      "description": "a collection with some deps on other collections",
      "authors": [
        "Orion User 1"
      ],
      "license": [
        "GPL-3.0-or-later"
      ],
      "tags": [
        "collectiontest"
      ],
      "signatures": [
        {
          "signature": "-----BEGIN PGP SIGNATURE-----\n\niQHIBAABCAAyFiEE6+0XDoyUgOIqHQWbFSUOnsCmJXcFAmHu3C8UHGdhbGF4eTNA\nYW5zaWJsZS5jb20ACgkQFSUOnsCmJXeCBwv+KRukj6gRuNxfx+buJvqztzFkNdQz\nm9M/nrAKrLZngOgYBE1oAw4wH9vrcayDay/WvOF3ESgZLBdP1pMTWam403zXHESL\nEzOK3SkikHw2kh2tnBO+3yimLf0EhPNBYbp9CGw4zQ0BhloKqwRODAIr5iOT3L0y\nB1dp6l7MvS068dbtLnHYaEpm7XYcnVH+mewHcmpcTwli6DPoTU7ff82fbunMRT1W\n3KiRjMeEbRqgMC9QazXTU9MREYQ5CSxBR8uCekKJRKw6bij5bX5LHj4KFy2HA2PI\nRDFdy+ir6cqaJgfmr2tfAN4c6dzUnxLDMmZvuywrbw3qnYaGjOUGHZbl3DFaG9Zp\no60dnseIhPjFaHeFfydV6IyaehGml5Http1gdv60e7qbBsto1CbEKigI5BNfqJTu\nRp9Cz34lbZLpSNZnq/XUMqsuqNcCP1mze4g5Xq6AvT98TG27jN3zPyMb+0dmkN+A\n21GAOPFzAjnFhoQp0LwrheTLYbMeRrg9qkPz\n=ZQbE\n-----END PGP SIGNATURE-----\n",
          "pubkey_fingerprint": "EBED170E8C9480E22A1D059B15250E9EC0A62577",
          "signing_service": "ansible-default"
        }
      ]
    },
    "contents": [],
    "version_sign_state": "signed",
    "docs_blob": {
      "contents": [],
      "collection_readme": {
        "html": "<p>This is collection_reqs_test collection</p>",
        "name": "README.md"
      },
      "documentation_files": []
    }
  },
  "all_versions": [
    {
      "version": "5.0.0",
      "created": "2022-01-24 17:04:33.904249+00:00",
      "version_sign_state": "signed"
    }
  ],
  "sign_state": "signed",
  "total_versions": 1,
  "signed_versions": 1,
  "unsigned_versions": 0
}

@rochacbruno
Copy link
Member Author

rochacbruno commented Jan 27, 2022

How to determine if the system has signing enabled?

/api/automation-hub/_ui/v1/feature-flags/

{
  "collection_auto_sign": true,    # this system signs during approval
  "collection_signing": true,    # there is a signing service and keys for collections
  "execution_environments": true
}

Or by checking on:

/api/automation-hub/_ui/v1/settings/

"GALAXY_COLLECTION_SIGNING_SERVICE": "ansible-default",
"GALAXY_AUTO_SIGN_COLLECTIONS": true

@rochacbruno rochacbruno force-pushed the AAH-1181 branch 2 times, most recently from 44da326 to f9a2554 Compare January 27, 2022 18:17
@rochacbruno
Copy link
Member Author

rochacbruno commented Jan 27, 2022

Filtering by sign state

/api/automation-hub/_ui/v1/repo/staging/?deprecated=false&sign_state=unsigned,partial
# options are [signed, unsigned, partial]

This endpoint accepts other filters such as namespace, name, version

@rochacbruno
Copy link
Member Author

Permissions on http://0.0.0.0:5001/api/automation-hub/_ui/v1/me/

"model_permissions": {
    "sign_collections_on_namespace": true,
    "sign_collections_on_repository": true,

on approval dashboard the permission is "move_collection": true, as if user can move the sign is automatic

@rochacbruno rochacbruno force-pushed the AAH-1181 branch 9 times, most recently from 5e30c7b to 9996656 Compare January 31, 2022 19:44
Issue: AAH-1181
env:LOCK_REQUIREMENTS=0
env:PULP_ANSIBLE_REVISION=main
@rochacbruno
Copy link
Member Author

/retest

drodowic and others added 2 commits February 1, 2022 09:13
…AH-1181

Issue: AAH-1181
env:LOCK_REQUIREMENTS=0
env:PULP_ANSIBLE_REVISION=main
@rochacbruno
Copy link
Member Author

/retest

Issue: AAH-1181
env:LOCK_REQUIREMENTS=0
env:PULP_ANSIBLE_REVISION=main
rochacbruno and others added 2 commits February 3, 2022 14:06
Issue: AAH-1181
env:LOCK_REQUIREMENTS=0
env:PULP_ANSIBLE_REVISION=main
@rochacbruno
Copy link
Member Author

@brumik @himdel this PR is being replaced by #1145

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants