Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSDF Origin Push/Pull Support added #182

Merged
merged 3 commits into from
Jun 18, 2024
Merged

OSDF Origin Push/Pull Support added #182

merged 3 commits into from
Jun 18, 2024

Conversation

atripathy86
Copy link
Contributor

@atripathy86 atripathy86 commented Jun 12, 2024

Related Issues / Pull Requests

Add support for reading and writing to OSDF Origin Remotes

Description

Enable Artifact Pushing/Pull from OSDF Origin

  • This functions with dvc[http] plugin for push except that it passes a
    bearer token in the header before any push/pull to a OSDF HTTP Remote.
  • Introduces new command cmf init osdfremote with new
    cmflib/commands/init/osdfremote.py
  • Handles artifact push when core.remote==osdf through dvc_push except
    that it updates core.remote.password in .dvc/config before doing so
  • Required OSDF Token is generated in function new generate_osdf_token()
    in utils/helper_functions.py. This uses scitokens and cryptography
    packages. This reads supplied private key, receives token from issuer and
    formats it as "Bearer + token".
  • Handles artifact pull through a new OSDFRemoteArtifacts Class in
    storage_backends/osdf_artifacts.py. This works with requests package
  • commands/artifacts/pull.py has new handling for core.remote==osdf
    which updates core.remote.password in .dvc/config and passes the
    newly generated dynamic password to download_artifacts() to use.
  • Also updated dvc version to 3.50.2 which is needed for dvc[http] to
    function properly. Also bumped ml-metadata to 1.15.0 which is needed for
    compatibility. External packages scitokens and cryptography also added
    as dependencies.
  • Added documentation for osdfremote in docs/cmf_client/cmf_client.md

What changes are proposed in this pull request?

  • Bug fix (non-breaking change which fixes an issue).
  • New feature (non-breaking change which adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected; for instance,
    examples in this repository need to be updated too).
  • This change requires a documentation update.

Checklist:

  • My code follows the style guidelines of this project (PEP-8 with Google-style docstrings).
  • My code modifies existing public API, or introduces new public API, and I updated or wrote docstrings that
    uses Google-style formatting and any other formatting that is supported by mkdocs and plugins this project
    uses.
  • I have commented my code.
  • My code requires documentation updates, and I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

- This functions with dvc[http] plugin for push except that it passes a
  bearer token in the header before any push/pull to a OSDF HTTP Remote.
- Introduces new command `cmf init osdfremote` with new
    cmflib/commands/init/osdfremote.py
- Handles artifact push when `core.remote==osdf` through dvc_push except
  that it updates `core.remote.password` in .dvc/config before doing so
- Required OSDF Token is generated in function new generate_osdf_token()
  in utils/helper_functions.py. This uses scitokens and cryptography
packages. This reads supplied private key, gets token from issuer and
formats it as "Bearer + token".
- Handles artifact pull through a new OSDFRemoteArtifacts Class in
`storage_backends/osdf_artifacts.py`. This works with requests package
- `commands/artifacts/pull.py` has new handling for `core.remote==osdf`
  which updates `core.remote.password` in .dvc/config and passes  the
newly generated dynamic password to download_artifacts to use.
- Also updated dvc version to 3.50.2 which is needed for dvc[ttp] to
  function. Also bumped ml-metadata to 1.15.0 which is needed for
compatibility. External packages scitokens and cryptography also added
as dependencies.
@annmary-roy annmary-roy merged commit e6c8858 into master Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants