Skip to content

Commit

Permalink
feat: File persistence definition and implementation (feast-dev#4742)
Browse files Browse the repository at this point in the history 
* File persistence definition and implementation

Signed-off-by: Daniele Martinoli <[email protected]>

* removed optional and default markers

Signed-off-by: Daniele Martinoli <[email protected]>

* removed global cfg variable

Signed-off-by: Daniele Martinoli <[email protected]>

* reviewed ApplyDefaultsToStatus

Signed-off-by: Daniele Martinoli <[email protected]>

* extended checks in ApplyDefaultsToStatus

Signed-off-by: Daniele Martinoli <[email protected]>

* removed var error

Signed-off-by: Daniele Martinoli <[email protected]>

* added IsValidOfflineStoreFilePersistenceType to the API definitions

Signed-off-by: Daniele Martinoli <[email protected]>

* moved IsValidOfflineStoreFilePersistenceType to services package and use it in deploy flow

Signed-off-by: Daniele Martinoli <[email protected]>

* renamed isValidOfflineStoreFilePersistenceType to checkOfflineStoreFilePersistenceType

Signed-off-by: Daniele Martinoli <[email protected]>

* adding controller tests for ephemeral stores

Signed-off-by: Daniele Martinoli <[email protected]>

* using slices package

Signed-off-by: Daniele Martinoli <[email protected]>

---------

Signed-off-by: Daniele Martinoli <[email protected]>
  • Loading branch information
@dmartinol @shuchu
dmartinol authored and shuchu committed Nov 21, 2024
1 parent 746bade commit 7493511
Show file tree
Hide file tree
Showing 15 changed files with 1,218 additions and 65 deletions.
39 changes: 39 additions & 0 deletions infra/feast-operator/api/v1alpha1/featurestore_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,16 +71,55 @@ type FeatureStoreServices struct {
// OfflineStore configures the deployed offline store service
type OfflineStore struct {
ServiceConfigs `json:",inline"`
Persistence *OfflineStorePersistence `json:"persistence,omitempty"`
}

// OfflineStorePersistence configures the persistence settings for the offline store service
type OfflineStorePersistence struct {
FilePersistence *OfflineStoreFilePersistence `json:"file,omitempty"`
}

// OfflineStorePersistence configures the file-based persistence for the offline store service
type OfflineStoreFilePersistence struct {
// +kubebuilder:validation:Enum=dask;duckdb
Type string `json:"type,omitempty"`
}

var ValidOfflineStoreFilePersistenceTypes = []string{
"dask",
"duckdb",
}

// OnlineStore configures the deployed online store service
type OnlineStore struct {
ServiceConfigs `json:",inline"`
Persistence *OnlineStorePersistence `json:"persistence,omitempty"`
}

// OnlineStorePersistence configures the persistence settings for the online store service
type OnlineStorePersistence struct {
FilePersistence *OnlineStoreFilePersistence `json:"file,omitempty"`
}

// OnlineStoreFilePersistence configures the file-based persistence for the offline store service
type OnlineStoreFilePersistence struct {
Path string `json:"path,omitempty"`
}

// LocalRegistryConfig configures the deployed registry service
type LocalRegistryConfig struct {
ServiceConfigs `json:",inline"`
Persistence *RegistryPersistence `json:"persistence,omitempty"`
}

// RegistryPersistence configures the persistence settings for the registry service
type RegistryPersistence struct {
FilePersistence *RegistryFilePersistence `json:"file,omitempty"`
}

// RegistryFilePersistence configures the file-based persistence for the registry service
type RegistryFilePersistence struct {
Path string `json:"path,omitempty"`
}

// Registry configures the registry service. One selection is required. Local is the default setting.
Expand Down
120 changes: 120 additions & 0 deletions infra/feast-operator/api/v1alpha1/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

80 changes: 80 additions & 0 deletions infra/feast-operator/config/crd/bases/feast.dev_featurestores.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,21 @@ spec:
description: PullPolicy describes a policy for if/when to
pull a container image
type: string
persistence:
description: OfflineStorePersistence configures the persistence
settings for the offline store service
properties:
file:
description: OfflineStorePersistence configures the file-based
persistence for the offline store service
properties:
type:
enum:
- dask
- duckdb
type: string
type: object
type: object
resources:
description: ResourceRequirements describes the compute resource
requirements.
Expand Down Expand Up @@ -361,6 +376,18 @@ spec:
description: PullPolicy describes a policy for if/when to
pull a container image
type: string
persistence:
description: OnlineStorePersistence configures the persistence
settings for the online store service
properties:
file:
description: OnlineStoreFilePersistence configures the
file-based persistence for the offline store service
properties:
path:
type: string
type: object
type: object
resources:
description: ResourceRequirements describes the compute resource
requirements.
Expand Down Expand Up @@ -546,6 +573,18 @@ spec:
description: PullPolicy describes a policy for if/when
to pull a container image
type: string
persistence:
description: RegistryPersistence configures the persistence
settings for the registry service
properties:
file:
description: RegistryFilePersistence configures the
file-based persistence for the registry service
properties:
path:
type: string
type: object
type: object
resources:
description: ResourceRequirements describes the compute
resource requirements.
Expand Down Expand Up @@ -780,6 +819,21 @@ spec:
description: PullPolicy describes a policy for if/when
to pull a container image
type: string
persistence:
description: OfflineStorePersistence configures the persistence
settings for the offline store service
properties:
file:
description: OfflineStorePersistence configures the
file-based persistence for the offline store service
properties:
type:
enum:
- dask
- duckdb
type: string
type: object
type: object
resources:
description: ResourceRequirements describes the compute
resource requirements.
Expand Down Expand Up @@ -962,6 +1016,19 @@ spec:
description: PullPolicy describes a policy for if/when
to pull a container image
type: string
persistence:
description: OnlineStorePersistence configures the persistence
settings for the online store service
properties:
file:
description: OnlineStoreFilePersistence configures
the file-based persistence for the offline store
service
properties:
path:
type: string
type: object
type: object
resources:
description: ResourceRequirements describes the compute
resource requirements.
Expand Down Expand Up @@ -1151,6 +1218,19 @@ spec:
description: PullPolicy describes a policy for if/when
to pull a container image
type: string
persistence:
description: RegistryPersistence configures the persistence
settings for the registry service
properties:
file:
description: RegistryFilePersistence configures
the file-based persistence for the registry
service
properties:
path:
type: string
type: object
type: object
resources:
description: ResourceRequirements describes the compute
resource requirements.
Expand Down
20 changes: 20 additions & 0 deletions infra/feast-operator/config/samples/v1alpha1_featurestore_ephemeral_persistence.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
name: sample-ephemeral-persistence
spec:
feastProject: my_project
services:
onlineStore:
persistence:
file:
path: /data/online_store.db
offlineStore:
persistence:
file:
type: dask
registry:
local:
persistence:
file:
path: /data/registry.db
Loading

0 comments on commit 7493511

Please sign in to comment.