Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [ai-platform] add blocking_operation_ids to ImportFeatureValuesOperationMetadata #4284

Merged
merged 21 commits into from
Jun 6, 2023
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2d84abd
feat: add match service in aiplatform v1
gcf-owl-bot[bot] May 18, 2023
05b5751
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 18, 2023
b689c52
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 18, 2023
8334db4
Merge branch 'owl-bot-copy-packages-google-cloud-aiplatform' of https…
gcf-owl-bot[bot] May 18, 2023
859064f
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
ddelgrosso1 May 18, 2023
37ca3c5
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
ddelgrosso1 May 19, 2023
a05df07
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
danieljbruce May 23, 2023
4508bac
feat: support for Model Garden -- A single place to search, discover,…
gcf-owl-bot[bot] May 31, 2023
ecb59ba
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 31, 2023
9c89c16
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
sofisl May 31, 2023
ddbc40c
feat: add NVIDIA_A100_80GB to AcceleratorType
gcf-owl-bot[bot] Jun 1, 2023
eae2897
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 2, 2023
c20b722
feat: add blocking_operation_ids to ImportFeatureValuesOperationMetadata
gcf-owl-bot[bot] Jun 2, 2023
f914c0d
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 2, 2023
3f86aca
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 2, 2023
8f5b76e
Merge branch 'owl-bot-copy-packages-google-cloud-aiplatform' of https…
gcf-owl-bot[bot] Jun 2, 2023
67828b9
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
d-goog Jun 2, 2023
0d4f041
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
sofisl Jun 5, 2023
006f4a5
build: export new clients
sofisl Jun 6, 2023
40e7521
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 6, 2023
e419e31
Merge branch 'main' into owl-bot-copy-packages-google-cloud-aiplatform
sofisl Jun 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/google-cloud-aiplatform/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,8 @@ Samples are in the [`samples/`](https://github.com/googleapis/google-cloud-node/
| Job_service.resume_model_deployment_monitoring_job | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/job_service.resume_model_deployment_monitoring_job.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/job_service.resume_model_deployment_monitoring_job.js,packages/google-cloud-aiplatform/samples/README.md) |
| Job_service.search_model_deployment_monitoring_stats_anomalies | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/job_service.search_model_deployment_monitoring_stats_anomalies.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/job_service.search_model_deployment_monitoring_stats_anomalies.js,packages/google-cloud-aiplatform/samples/README.md) |
| Job_service.update_model_deployment_monitoring_job | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/job_service.update_model_deployment_monitoring_job.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/job_service.update_model_deployment_monitoring_job.js,packages/google-cloud-aiplatform/samples/README.md) |
| Match_service.find_neighbors | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/match_service.find_neighbors.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/match_service.find_neighbors.js,packages/google-cloud-aiplatform/samples/README.md) |
| Match_service.read_index_datapoints | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/match_service.read_index_datapoints.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/match_service.read_index_datapoints.js,packages/google-cloud-aiplatform/samples/README.md) |
| Metadata_service.add_context_artifacts_and_executions | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.add_context_artifacts_and_executions.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.add_context_artifacts_and_executions.js,packages/google-cloud-aiplatform/samples/README.md) |
| Metadata_service.add_context_children | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.add_context_children.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.add_context_children.js,packages/google-cloud-aiplatform/samples/README.md) |
| Metadata_service.add_execution_events | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.add_execution_events.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.add_execution_events.js,packages/google-cloud-aiplatform/samples/README.md) |
Expand Down Expand Up @@ -230,6 +232,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/google-cloud-node/
| Metadata_service.update_execution | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.update_execution.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/metadata_service.update_execution.js,packages/google-cloud-aiplatform/samples/README.md) |
| Migration_service.batch_migrate_resources | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/migration_service.batch_migrate_resources.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/migration_service.batch_migrate_resources.js,packages/google-cloud-aiplatform/samples/README.md) |
| Migration_service.search_migratable_resources | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/migration_service.search_migratable_resources.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/migration_service.search_migratable_resources.js,packages/google-cloud-aiplatform/samples/README.md) |
| Model_garden_service.get_publisher_model | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/model_garden_service.get_publisher_model.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/model_garden_service.get_publisher_model.js,packages/google-cloud-aiplatform/samples/README.md) |
| Model_service.batch_import_evaluated_annotations | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/model_service.batch_import_evaluated_annotations.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/model_service.batch_import_evaluated_annotations.js,packages/google-cloud-aiplatform/samples/README.md) |
| Model_service.batch_import_model_evaluation_slices | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/model_service.batch_import_model_evaluation_slices.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/model_service.batch_import_model_evaluation_slices.js,packages/google-cloud-aiplatform/samples/README.md) |
| Model_service.copy_model | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/model_service.copy_model.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-aiplatform/samples/generated/v1/model_service.copy_model.js,packages/google-cloud-aiplatform/samples/README.md) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ enum AcceleratorType {
// Nvidia Tesla A100 GPU.
NVIDIA_TESLA_A100 = 8;

// Nvidia A100 80GB GPU.
NVIDIA_A100_80GB = 9;

// Nvidia L4 GPU.
NVIDIA_L4 = 11;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -534,7 +534,7 @@ message ImportFeatureValuesRequest {
];

// Source column that holds entity IDs. If not provided, entity IDs are
// extracted from the column named `entity_id`.
// extracted from the column named entity_id.
string entity_id_field = 5;

// Required. Specifications defining which Feature values to import from the
Expand Down Expand Up @@ -1288,6 +1288,10 @@ message ImportFeatureValuesOperationMetadata {
// The number rows that weren't ingested due to having timestamps outside the
// retention boundary.
int64 timestamp_outside_retention_rows_count = 7;

// List of ImportFeatureValues operations running under a single EntityType
// that are blocking this operation.
repeated int64 blocking_operation_ids = 8;
}

// Details of operations that exports Features values.
Expand Down Expand Up @@ -1439,6 +1443,6 @@ message EntityIdSelector {
}

// Source column that holds entity IDs. If not provided, entity IDs are
// extracted from the column named `entity_id`.
// extracted from the column named entity_id.
string entity_id_field = 5;
}
Original file line number Diff line number Diff line change
Expand Up @@ -159,17 +159,16 @@ message DeployedIndex {
// Output only. The DeployedIndex may depend on various data on its original
// Index. Additionally when certain changes to the original Index are being
// done (e.g. when what the Index contains is being changed) the DeployedIndex
// may be asynchronously updated in the background to reflect this changes. If
// this timestamp's value is at least the
// may be asynchronously updated in the background to reflect these changes.
// If this timestamp's value is at least the
// [Index.update_time][google.cloud.aiplatform.v1.Index.update_time] of the
// original Index, it means that this DeployedIndex and the original Index are
// in sync. If this timestamp is older, then to see which updates this
// DeployedIndex already contains (and which not), one must
// [list][Operations.ListOperations] [Operations][Operation]
// [working][Operation.name] on the original Index. Only
// the successfully completed Operations with
// [Operations.metadata.generic_metadata.update_time]
// [google.cloud.aiplatform.v1.GenericOperationMetadata.update_time]
// DeployedIndex already contains (and which it does not), one must
// [list][google.longrunning.Operations.ListOperations] the operations that
// are running on the original Index. Only the successfully completed
// Operations with
// [update_time][google.cloud.aiplatform.v1.GenericOperationMetadata.update_time]
// equal or before this sync time are contained in this DeployedIndex.
google.protobuf.Timestamp index_sync_time = 6
[(google.api.field_behavior) = OUTPUT_ONLY];
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
// Copyright 2023 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

package google.cloud.aiplatform.v1;

import "google/api/annotations.proto";
import "google/api/client.proto";
import "google/api/field_behavior.proto";
import "google/api/resource.proto";
import "google/cloud/aiplatform/v1/index.proto";

option csharp_namespace = "Google.Cloud.AIPlatform.V1";
option go_package = "cloud.google.com/go/aiplatform/apiv1/aiplatformpb;aiplatformpb";
option java_multiple_files = true;
option java_outer_classname = "MatchServiceProto";
option java_package = "com.google.cloud.aiplatform.v1";
option php_namespace = "Google\\Cloud\\AIPlatform\\V1";
option ruby_package = "Google::Cloud::AIPlatform::V1";

// MatchService is a Google managed service for efficient vector similarity
// search at scale.
service MatchService {
option (google.api.default_host) = "aiplatform.googleapis.com";
option (google.api.oauth_scopes) =
"https://www.googleapis.com/auth/cloud-platform";

// Finds the nearest neighbors of each vector within the request.
rpc FindNeighbors(FindNeighborsRequest) returns (FindNeighborsResponse) {
option (google.api.http) = {
post: "/v1/{index_endpoint=projects/*/locations/*/indexEndpoints/*}:findNeighbors"
body: "*"
};
}

// Reads the datapoints/vectors of the given IDs.
// A maximum of 1000 datapoints can be retrieved in a batch.
rpc ReadIndexDatapoints(ReadIndexDatapointsRequest)
returns (ReadIndexDatapointsResponse) {
option (google.api.http) = {
post: "/v1/{index_endpoint=projects/*/locations/*/indexEndpoints/*}:readIndexDatapoints"
body: "*"
};
}
}

// The request message for
// [MatchService.FindNeighbors][google.cloud.aiplatform.v1.MatchService.FindNeighbors].
message FindNeighborsRequest {
// A query to find a number of the nearest neighbors (most similar vectors)
// of a vector.
message Query {
// Required. The datapoint/vector whose nearest neighbors should be searched
// for.
IndexDatapoint datapoint = 1 [(google.api.field_behavior) = REQUIRED];

// The number of nearest neighbors to be retrieved from database for each
// query. If not set, will use the default from the service configuration
// (https://cloud.google.com/vertex-ai/docs/matching-engine/configuring-indexes#nearest-neighbor-search-config).
int32 neighbor_count = 2;

// Crowding is a constraint on a neighbor list produced by nearest neighbor
// search requiring that no more than some value k' of the k neighbors
// returned have the same value of crowding_attribute.
// It's used for improving result diversity.
// This field is the maximum number of matches with the same crowding tag.
int32 per_crowding_attribute_neighbor_count = 3;

// The number of neighbors to find via approximate search before
// exact reordering is performed. If not set, the default value from scam
// config is used; if set, this value must be > 0.
int32 approximate_neighbor_count = 4;

// The fraction of the number of leaves to search, set at query time allows
// user to tune search performance. This value increase result in both
// search accuracy and latency increase. The value should be between 0.0
// and 1.0. If not set or set to 0.0, query uses the default value specified
// in
// NearestNeighborSearchConfig.TreeAHConfig.fraction_leaf_nodes_to_search.
double fraction_leaf_nodes_to_search_override = 5;
}

// Required. The name of the index endpoint.
// Format:
// `projects/{project}/locations/{location}/indexEndpoints/{index_endpoint}`
string index_endpoint = 1 [
(google.api.field_behavior) = REQUIRED,
(google.api.resource_reference) = {
type: "aiplatform.googleapis.com/IndexEndpoint"
}
];

// The ID of the DeployedIndex that will serve the request. This request is
// sent to a specific IndexEndpoint, as per the IndexEndpoint.network. That
// IndexEndpoint also has IndexEndpoint.deployed_indexes, and each such index
// has a DeployedIndex.id field.
// The value of the field below must equal one of the DeployedIndex.id
// fields of the IndexEndpoint that is being called for this request.
string deployed_index_id = 2;

// The list of queries.
repeated Query queries = 3;

// If set to true, the full datapoints (including all vector values and
// restricts) of the nearest neighbors are returned.
// Note that returning full datapoint will significantly increase the
// latency and cost of the query.
bool return_full_datapoint = 4;
}

// The response message for
// [MatchService.FindNeighbors][google.cloud.aiplatform.v1.MatchService.FindNeighbors].
message FindNeighborsResponse {
// A neighbor of the query vector.
message Neighbor {
// The datapoint of the neighbor.
// Note that full datapoints are returned only when "return_full_datapoint"
// is set to true. Otherwise, only the "datapoint_id" and "crowding_tag"
// fields are populated.
IndexDatapoint datapoint = 1;

// The distance between the neighbor and the query vector.
double distance = 2;
}

// Nearest neighbors for one query.
message NearestNeighbors {
// The ID of the query datapoint.
string id = 1;

// All its neighbors.
repeated Neighbor neighbors = 2;
}

// The nearest neighbors of the query datapoints.
repeated NearestNeighbors nearest_neighbors = 1;
}

// The request message for
// [MatchService.ReadIndexDatapoints][google.cloud.aiplatform.v1.MatchService.ReadIndexDatapoints].
message ReadIndexDatapointsRequest {
// Required. The name of the index endpoint.
// Format:
// `projects/{project}/locations/{location}/indexEndpoints/{index_endpoint}`
string index_endpoint = 1 [
(google.api.field_behavior) = REQUIRED,
(google.api.resource_reference) = {
type: "aiplatform.googleapis.com/IndexEndpoint"
}
];

// The ID of the DeployedIndex that will serve the request.
string deployed_index_id = 2;

// IDs of the datapoints to be searched for.
repeated string ids = 3;
}

// The response message for
// [MatchService.ReadIndexDatapoints][google.cloud.aiplatform.v1.MatchService.ReadIndexDatapoints].
message ReadIndexDatapointsResponse {
// The result list of datapoints.
repeated IndexDatapoint datapoints = 1;
}
Loading