You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should unify our repositories under DID-based structure.
Additional Consideration:
Consider where to store dataset aliases in different types of repositories (including local CLI workspace). Ideally we want ID -> Alias, Alias -> ID resolutions to be O(1), but at the same time:
guarantee that aliases are unique
our "index" is resistant to concurrent updates
Storing datasets by DID means it will no longer be possible to have same dataset under different aliases in CLI (or any other repo). This currently is allowed, but consider how to handle this after the change:
kamu pull s3://datasets.kamu.dev/odf/v2/contrib/com.cryptocompare.ohlcv.eth-usd --as a
kamu pull s3://datasets.kamu.dev/odf/v2/contrib/com.cryptocompare.ohlcv.eth-usd --as b
Consider what to do when we push datasets to basic repositories (plain S3, Local FS), e.g. what the following operation does now and how it should work after this change:
kamu repo add myrepo /some/path
kamu push com.cryptocompare.ohlcv.eth-usd --to myrepo/com.cryptocompare.ohlcv.eth-usd
We have too many repository structures:
We should unify our repositories under DID-based structure.
Additional Consideration:
ID -> Alias, Alias -> ID
resolutions to be O(1), but at the same time:SearchServiceImpl
is currently making hard assumptions about the remote repository structure and will need to be updated. See Support ODF repositories inkamu search
command #508 for details.We have some python scripts that push datasets from local (multi-tenant) workspace directly to S3. We should update them after these changes.
Notes:
The text was updated successfully, but these errors were encountered: