-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat!: add data connectors #407
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
leafty
force-pushed
the
leafty/build-data-connectors
branch
from
September 13, 2024 11:31
9a7f277
to
093d7ef
Compare
leafty
force-pushed
the
leafty/build-data-connectors
branch
2 times, most recently
from
September 23, 2024 07:35
6dcb37a
to
739e439
Compare
leafty
force-pushed
the
leafty/build-data-connectors
branch
2 times, most recently
from
September 30, 2024 14:24
7ecf26e
to
7574d13
Compare
Moves the `project_slugs` table to `entity_slugs` and `project_slugs_old` to `entity_slugs_old`. This is done to prepare for more entity types to be namespaced. Note: the sequence migration has been tested: 1. Force migration revision to be `9058bf0a1a12` (before this). 2. Deploy with image tag `renku/renku-data-service:sha-345a1bb` (current `main`) and run `alembic check` and `alembic heads` to verify the DB is before this change. 3. Create some projects. 4. Re-deploy this PR and check that migration revision is `a11752a5afba` (after this). 5. Create more projects.
Introduce namespaced data connectors for Renku 2.0. Details: * Remove API routes for `/storages_v2` * Add API routes for `/data_connectors` * Add `data_connectors` table * Drop the `storage_secrets` table * Add `data_connectors_secrets` table
Fixes deleting storage configuration fields by setting them to `null`.
Adds authorization for data connectors in Renku 2.0. Details: * Authz changes to support data connectors The `authz` schema changes include future support for linking data connectors to projects, but doing so is not implemented in this PR.
Adds the possibility to link a data connector to a project. Details: * Add API to link a data connector to a project * Add API to list projects connected to a data connector * Add API to list data connectors connected to a project Note that `read` permissions are cascaded to direct members of the connected project. * Creating a link from a private data connector to a project requires the `editor` role on both sides of the link. * Creating a link from a public data connector to a project requires the `editor` role on the project. * Removing a link requires either the `editor` role on the project or the `owner` role on the data connector.
Add support for saving and managing secrets for data connectors. Details: * Add API endpoints to list, update and delete saved secrets for a given data connector.
leafty
force-pushed
the
leafty/build-data-connectors
branch
from
October 4, 2024 08:48
c5820ef
to
b772eef
Compare
…nectors # Conflicts: # components/renku_data_services/project/apispec.py
Fixes two issues with the migrations for data connectors: 1. The migrations cannot be rolled back after creating a data connector. This was caused by leftover entries in the `entity_slugs` table. 2. The entity slug constraint was not created. The change was not picked up by Alembic; it seems Alembic can only handle `CheckConstraint` creation together with table creation, but not new constraints on an existing table, updates to them, or removing them from a table.
…nectors # Conflicts: # components/renku_data_services/storage/api.spec.yaml # components/renku_data_services/storage/apispec.py # components/renku_data_services/storage/db.py # components/renku_data_services/storage/models.py
Add a background job to migrate "Data Sources" (`/storages_v2`) to "Data Connectors" (`/data_connectors`). Each storage item has a new data connector created and linked to the source project.
olevski
approved these changes
Oct 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feature branch to add support for data connectors.
See: SwissDataScienceCenter/renku#3751.
Changes:
In progress: