Skip to content

Commit

Permalink
feat: Adding Python classes for new Feature View type SortedFeatureVi…
Browse files Browse the repository at this point in the history 
…ew (#175)

* Adding python classes for new Feature View type SortedFeatureView

* Adding python classes for new Feature View type SortedFeatureView

* Adding python classes for new Feature View type SortedFeatureView

* fixing formatting

* Adding tests and fixing formatting

* Adding go class

* Adding relevant go class

* Adding additional testing

* getting rid of dummy entity constants

* formatting file to fix linting errors

* Adding in new changes to support the new feature view type sortedFeatureView

* fixing circular dependency

* fixing circular dependency

* fixing circular dependency by fixing how things are imported

* fixing circular dependency by fixing how things are imported

* Fixing linting and missed additions of the SortedFeatureView

* fixing formatting issues

* fixing formatting issues

* fixing failing tests

* adding SortedFeatureView to the permission proto

* Fixing linting errors

* Fixing linting errors

* Fixing linting errors

* Fixing linting errors

* Fixing linting errors

* Adding in more changes

* Adding changes to the workflow to see if it fixes the build failures

* fixing linting issues

* Committing PR changes

* fixing failing tests

* removing setup tools from GHA unit_tests workflow

* Modifying go class and adding back build addition to workflow

* removing cast

* changing value type field for SortedFeatureViewSortKeyModel

* changing value type field for SortedFeatureViewSortKeyModel

* modified ensure_valid method in SortedFeatureView

* Added more changes based on PR comments
  • Loading branch information
@Manisha4
Manisha4 authored Feb 28, 2025
1 parent df265fc commit a6881dc
Show file tree
Hide file tree
Showing 28 changed files with 1,557 additions and 55 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/unit_tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ jobs:
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=22.1,<23"
- name: Upgrade setuptools and wheel
run: pip install --upgrade setuptools wheel
- name: Test
run: make test-go

Expand Down
78 changes: 40 additions & 38 deletions docs/reference/registry/registry-permissions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,43 +3,45 @@

## API Endpoints and Permissions

| Endpoint | Resource Type | Permission | Description |
| ------------------------ |---------------------|------------------------| -------------------------------------------------------------- |
| ApplyEntity | Entity | Create, Update, Delete | Apply an entity to the registry |
| GetEntity | Entity | Read | Get an entity from the registry |
| ListEntities | Entity | Read | List entities in the registry |
| DeleteEntity | Entity | Delete | Delete an entity from the registry |
| ApplyDataSource | DataSource | Create, Update, Delete | Apply a data source to the registry |
| GetDataSource | DataSource | Read | Get a data source from the registry |
| ListDataSources | DataSource | Read | List data sources in the registry |
| DeleteDataSource | DataSource | Delete | Delete a data source from the registry |
| ApplyFeatureView | FeatureView | Create, Update, Delete | Apply a feature view to the registry |
| GetFeatureView | FeatureView | Read | Get a feature view from the registry |
| ListFeatureViews | FeatureView | Read | List feature views in the registry |
| DeleteFeatureView | FeatureView | Delete | Delete a feature view from the registry |
| GetStreamFeatureView | StreamFeatureView | Read | Get a stream feature view from the registry |
| ListStreamFeatureViews | StreamFeatureView | Read | List stream feature views in the registry |
| GetOnDemandFeatureView | OnDemandFeatureView | Read | Get an on-demand feature view from the registry |
| ListOnDemandFeatureViews | OnDemandFeatureView | Read | List on-demand feature views in the registry |
| ApplyFeatureService | FeatureService | Create, Update, Delete | Apply a feature service to the registry |
| GetFeatureService | FeatureService | Read | Get a feature service from the registry |
| ListFeatureServices | FeatureService | Read | List feature services in the registry |
| DeleteFeatureService | FeatureService | Delete | Delete a feature service from the registry |
| ApplySavedDataset | SavedDataset | Create, Update, Delete | Apply a saved dataset to the registry |
| GetSavedDataset | SavedDataset | Read | Get a saved dataset from the registry |
| ListSavedDatasets | SavedDataset | Read | List saved datasets in the registry |
| DeleteSavedDataset | SavedDataset | Delete | Delete a saved dataset from the registry |
| ApplyValidationReference | ValidationReference | Create, Update, Delete | Apply a validation reference to the registry |
| GetValidationReference | ValidationReference | Read | Get a validation reference from the registry |
| ListValidationReferences | ValidationReference | Read | List validation references in the registry |
| DeleteValidationReference| ValidationReference | Delete | Delete a validation reference from the registry |
| ApplyPermission | Permission | Create, Update, Delete | Apply a permission to the registry |
| GetPermission | Permission | Read | Get a permission from the registry |
| ListPermissions | Permission | Read | List permissions in the registry |
| DeletePermission | Permission | Delete | Delete a permission from the registry |
| Commit | | None | Commit changes to the registry |
| Refresh | | None | Refresh the registry |
| Proto | | None | Get the proto representation of the registry |
| Endpoint | Resource Type | Permission | Description |
|---------------------------|---------------------|------------------------|-------------------------------------------------|
| ApplyEntity | Entity | Create, Update, Delete | Apply an entity to the registry |
| GetEntity | Entity | Read | Get an entity from the registry |
| ListEntities | Entity | Read | List entities in the registry |
| DeleteEntity | Entity | Delete | Delete an entity from the registry |
| ApplyDataSource | DataSource | Create, Update, Delete | Apply a data source to the registry |
| GetDataSource | DataSource | Read | Get a data source from the registry |
| ListDataSources | DataSource | Read | List data sources in the registry |
| DeleteDataSource | DataSource | Delete | Delete a data source from the registry |
| ApplyFeatureView | FeatureView | Create, Update, Delete | Apply a feature view to the registry |
| GetFeatureView | FeatureView | Read | Get a feature view from the registry |
| ListFeatureViews | FeatureView | Read | List feature views in the registry |
| DeleteFeatureView | FeatureView | Delete | Delete a feature view from the registry |
| GetSortedFeatureView | SortedFeatureView | Read | Get a sorted feature view from the registry |
| ListSortedFeatureViews | SortedFeatureView | Read | List sorted feature views in the registry |
| GetStreamFeatureView | StreamFeatureView | Read | Get a stream feature view from the registry |
| ListStreamFeatureViews | StreamFeatureView | Read | List stream feature views in the registry |
| GetOnDemandFeatureView | OnDemandFeatureView | Read | Get an on-demand feature view from the registry |
| ListOnDemandFeatureViews | OnDemandFeatureView | Read | List on-demand feature views in the registry |
| ApplyFeatureService | FeatureService | Create, Update, Delete | Apply a feature service to the registry |
| GetFeatureService | FeatureService | Read | Get a feature service from the registry |
| ListFeatureServices | FeatureService | Read | List feature services in the registry |
| DeleteFeatureService | FeatureService | Delete | Delete a feature service from the registry |
| ApplySavedDataset | SavedDataset | Create, Update, Delete | Apply a saved dataset to the registry |
| GetSavedDataset | SavedDataset | Read | Get a saved dataset from the registry |
| ListSavedDatasets | SavedDataset | Read | List saved datasets in the registry |
| DeleteSavedDataset | SavedDataset | Delete | Delete a saved dataset from the registry |
| ApplyValidationReference | ValidationReference | Create, Update, Delete | Apply a validation reference to the registry |
| GetValidationReference | ValidationReference | Read | Get a validation reference from the registry |
| ListValidationReferences | ValidationReference | Read | List validation references in the registry |
| DeleteValidationReference | ValidationReference | Delete | Delete a validation reference from the registry |
| ApplyPermission | Permission | Create, Update, Delete | Apply a permission to the registry |
| GetPermission | Permission | Read | Get a permission from the registry |
| ListPermissions | Permission | Read | List permissions in the registry |
| DeletePermission | Permission | Delete | Delete a permission from the registry |
| Commit | | None | Commit changes to the registry |
| Refresh | | None | Refresh the registry |
| Proto | | None | Get the proto representation of the registry |

## How to configure Authentication and Authorization
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
76 changes: 76 additions & 0 deletions go/internal/feast/model/sortedfeatureview.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package model

import (
"github.com/feast-dev/feast/go/protos/feast/core"
"github.com/feast-dev/feast/go/protos/feast/types"
)

type SortOrder struct {
Order core.SortOrder_Enum
}

func NewSortOrderFromProto(order core.SortOrder_Enum) *SortOrder {
return &SortOrder{
Order: order,
}
}

func (so *SortOrder) String() string {
switch so.Order {
case core.SortOrder_ASC:
return "ASC"
case core.SortOrder_DESC:
return "DESC"
default:
return "INVALID"
}
}

func (so *SortOrder) ToProto() core.SortOrder_Enum {
return so.Order
}

type SortKey struct {
FieldName string
Order *SortOrder
ValueType types.ValueType_Enum
}

func NewSortKeyFromProto(proto *core.SortKey) *SortKey {
return &SortKey{
FieldName: proto.GetName(),
Order: NewSortOrderFromProto(proto.GetDefaultSortOrder()),
ValueType: proto.GetValueType(),
}
}

type SortedFeatureView struct {
*FeatureView
SortKeys []*SortKey
}

func NewSortedFeatureViewFromProto(proto *core.SortedFeatureView) *SortedFeatureView {
baseFV := &FeatureView{
Base: NewBaseFeatureView(proto.GetSpec().GetName(), proto.GetSpec().GetFeatures()),
Ttl: proto.GetSpec().GetTtl(),
}

// Convert each sort key from the proto.
sortKeys := make([]*SortKey, len(proto.GetSpec().GetSortKeys()))
for i, skProto := range proto.GetSpec().GetSortKeys() {
sortKeys[i] = NewSortKeyFromProto(skProto)
}

return &SortedFeatureView{
FeatureView: baseFV,
SortKeys: sortKeys,
}
}

func (sfv *SortedFeatureView) NewSortedFeatureViewFromBase(base *BaseFeatureView) *SortedFeatureView {
newFV := sfv.FeatureView.NewFeatureViewFromBase(base)
return &SortedFeatureView{
FeatureView: newFV,
SortKeys: sfv.SortKeys,
}
}
3 changes: 2 additions & 1 deletion protos/feast/core/Permission.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ message PermissionSpec {
ON_DEMAND_FEATURE_VIEW = 1;
BATCH_FEATURE_VIEW = 2;
STREAM_FEATURE_VIEW= 3;
SORTED_FEATURE_VIEW= 15;
ENTITY = 4;
FEATURE_SERVICE = 5;
DATA_SOURCE = 6;
Expand Down Expand Up @@ -67,4 +68,4 @@ message PermissionSpec {
message PermissionMeta {
google.protobuf.Timestamp created_timestamp = 1;
google.protobuf.Timestamp last_updated_timestamp = 2;
}
}
2 changes: 2 additions & 0 deletions sdk/python/feast/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from .on_demand_feature_view import OnDemandFeatureView
from .project import Project
from .repo_config import RepoConfig
from .sorted_feature_view import SortedFeatureView
from .stream_feature_view import StreamFeatureView
from .value_type import ValueType

Expand All @@ -42,6 +43,7 @@
"FeatureService",
"FeatureStore",
"FeatureView",
"SortedFeatureView",
"OnDemandFeatureView",
"RepoConfig",
"StreamFeatureView",
Expand Down
10 changes: 9 additions & 1 deletion sdk/python/feast/base_feature_view.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@
from feast.protos.feast.core.OnDemandFeatureView_pb2 import (
OnDemandFeatureView as OnDemandFeatureViewProto,
)
from feast.protos.feast.core.SortedFeatureView_pb2 import (
SortedFeatureView as SortedFeatureViewProto,
)
from feast.protos.feast.core.StreamFeatureView_pb2 import (
StreamFeatureView as StreamFeatureViewProto,
)
Expand Down Expand Up @@ -98,7 +101,12 @@ def proto_class(self) -> Type[Message]:
@abstractmethod
def to_proto(
self,
) -> Union[FeatureViewProto, OnDemandFeatureViewProto, StreamFeatureViewProto]:
) -> Union[
FeatureViewProto,
OnDemandFeatureViewProto,
StreamFeatureViewProto,
SortedFeatureViewProto,
]:
pass

@classmethod
Expand Down
60 changes: 60 additions & 0 deletions sdk/python/feast/diff/registry_diff.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@
from feast.protos.feast.core.OnDemandFeatureView_pb2 import OnDemandFeatureViewSpec
from feast.protos.feast.core.Permission_pb2 import Permission as PermissionProto
from feast.protos.feast.core.SavedDataset_pb2 import SavedDataset as SavedDatasetProto
from feast.protos.feast.core.SortedFeatureView_pb2 import (
SortedFeatureView as SortedFeatureViewProto,
)
from feast.protos.feast.core.StreamFeatureView_pb2 import (
StreamFeatureView as StreamFeatureViewProto,
)
Expand Down Expand Up @@ -110,6 +113,7 @@ def tag_objects_for_keep_delete_update_add(
EntityProto,
FeatureViewProto,
FeatureServiceProto,
SortedFeatureViewProto,
OnDemandFeatureViewProto,
StreamFeatureViewProto,
ValidationReferenceProto,
Expand Down Expand Up @@ -180,6 +184,61 @@ def diff_registry_objects(
getattr(new_udf, _udf_field.name),
)
)
elif _field.name == "sort_keys":
# Special handling for sort_keys field (a repeated field)
current_sort_keys = getattr(current_spec, _field.name)
new_sort_keys = getattr(new_spec, _field.name)
if len(current_sort_keys) != len(new_sort_keys):
transition = TransitionType.UPDATE
property_diffs.append(
PropertyDiff(_field.name, current_sort_keys, new_sort_keys)
)
else:
# Iterate through each sort key pair
for idx, (cs, ns) in enumerate(
zip(current_sort_keys, new_sort_keys)
):
if cs.name != ns.name:
transition = TransitionType.UPDATE
property_diffs.append(
PropertyDiff(
f"sort_keys[{idx}].name", cs.name, ns.name
)
)
if cs.value_type != ns.value_type:
transition = TransitionType.UPDATE
property_diffs.append(
PropertyDiff(
f"sort_keys[{idx}].value_type",
cs.value_type,
ns.value_type,
)
)
if cs.default_sort_order != ns.default_sort_order:
transition = TransitionType.UPDATE
property_diffs.append(
PropertyDiff(
f"sort_keys[{idx}].default_sort_order",
cs.default_sort_order,
ns.default_sort_order,
)
)
if cs.tags != ns.tags:
transition = TransitionType.UPDATE
property_diffs.append(
PropertyDiff(
f"sort_keys[{idx}].tags", cs.tags, ns.tags
)
)
if cs.description != ns.description:
transition = TransitionType.UPDATE
property_diffs.append(
PropertyDiff(
f"sort_keys[{idx}].description",
cs.description,
ns.description,
)
)
else:
transition = TransitionType.UPDATE
property_diffs.append(
Expand Down Expand Up @@ -342,6 +401,7 @@ def apply_diff_to_registry(
FeastObjectType.FEATURE_VIEW,
FeastObjectType.ON_DEMAND_FEATURE_VIEW,
FeastObjectType.STREAM_FEATURE_VIEW,
FeastObjectType.SORTED_FEATURE_VIEW,
]:
feature_view_obj = cast(
BaseFeatureView, feast_object_diff.current_feast_object
Expand Down
Loading

0 comments on commit a6881dc

Please sign in to comment.