Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add open lineage support #4793

Merged
merged 2 commits into from
Nov 6, 2023
Merged

feat: add open lineage support #4793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0834273
feat: add open lineage support
gcf-owl-bot[bot] Nov 6, 2023
5db7564
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/google-cloud-datacatalog-lineage/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/google-cloud-node/
| Lineage.list_lineage_events | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.list_lineage_events.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.list_lineage_events.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
| Lineage.list_processes | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.list_processes.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.list_processes.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
| Lineage.list_runs | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.list_runs.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.list_runs.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
| Lineage.process_open_lineage_run_event | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.process_open_lineage_run_event.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.process_open_lineage_run_event.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
| Lineage.search_links | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.search_links.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.search_links.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
| Lineage.update_process | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.update_process.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.update_process.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
| Lineage.update_run | [source code](https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.update_run.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/google-cloud-node&page=editor&open_in_editor=packages/google-cloud-datacatalog-lineage/samples/generated/v1/lineage.update_run.js,packages/google-cloud-datacatalog-lineage/samples/README.md) |
Expand Down
104 changes: 85 additions & 19 deletions ...google-cloud-datacatalog-lineage/protos/google/cloud/datacatalog/lineage/v1/lineage.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright 2022 Google LLC
// Copyright 2023 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -43,6 +43,19 @@ service Lineage {
option (google.api.oauth_scopes) =
"https://www.googleapis.com/auth/cloud-platform";

// Creates new lineage events together with their parents: process and run.
// Updates the process and run if they already exist.
// Mapped from Open Lineage specification:
// https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json.
rpc ProcessOpenLineageRunEvent(ProcessOpenLineageRunEventRequest)
returns (ProcessOpenLineageRunEventResponse) {
option (google.api.http) = {
post: "/v1/{parent=projects/*/locations/*}:processOpenLineageRunEvent"
body: "open_lineage"
};
option (google.api.method_signature) = "parent,open_lineage";
}

// Creates a new process.
rpc CreateProcess(CreateProcessRequest) returns (Process) {
option (google.api.http) = {
Expand Down Expand Up @@ -232,8 +245,11 @@ message Process {
// or numbers, spaces or characters like `_-:&.`
string display_name = 2 [(google.api.field_behavior) = OPTIONAL];

// Optional. The attributes of the process. Can be anything, for example,
// "author". Up to 100 attributes are allowed.
// Optional. The attributes of the process. Should only be used for the
// purpose of non-semantic management (classifying, describing or labeling the
// process).
//
// Up to 100 attributes are allowed.
map<string, google.protobuf.Value> attributes = 3
[(google.api.field_behavior) = OPTIONAL];

Expand Down Expand Up @@ -280,8 +296,10 @@ message Run {
// or numbers, spaces or characters like `_-:&.`
string display_name = 2 [(google.api.field_behavior) = OPTIONAL];

// Optional. The attributes of the run. Can be anything, for example, a string
// with an SQL request. Up to 100 attributes are allowed.
// Optional. The attributes of the run. Should only be used for the purpose of
// non-semantic management (classifying, describing or labeling the run).
//
// Up to 100 attributes are allowed.
map<string, google.protobuf.Value> attributes = 3
[(google.api.field_behavior) = OPTIONAL];

Expand Down Expand Up @@ -316,11 +334,11 @@ message LineageEvent {
// Optional. List of source-target pairs. Can't contain more than 100 tuples.
repeated EventLink links = 8 [(google.api.field_behavior) = OPTIONAL];

// Optional. The beginning of the transformation which resulted in this
// Required. The beginning of the transformation which resulted in this
// lineage event. For streaming scenarios, it should be the beginning of the
// period from which the lineage is being reported.
google.protobuf.Timestamp start_time = 6
[(google.api.field_behavior) = OPTIONAL];
[(google.api.field_behavior) = REQUIRED];

// Optional. The end of the transformation which resulted in this lineage
// event. For streaming scenarios, it should be the end of the period from
Expand All @@ -340,16 +358,9 @@ message EventLink {

// The soft reference to everything you can attach a lineage event to.
message EntityReference {
// Required. Fully Qualified Name of the entity. Useful for referencing
// entities that aren't represented as GCP resources, for example, tables in
// Dataproc Metastore API.
//
// Examples:
//
// * `bigquery:dataset.project_id.dataset_id`
// * `bigquery:table.project_id.dataset_id.table_id`
// * `pubsub:project_id.topic_id`
// * `dataproc_metastore:projectId.locationId.instanceId.databaseId.tableId`
// Required. [Fully Qualified Name
// (FQN)](https://cloud.google.com/data-catalog/docs/fully-qualified-names)
// of the entity.
string fully_qualified_name = 1 [(google.api.field_behavior) = REQUIRED];
}

Expand Down Expand Up @@ -380,6 +391,9 @@ message OperationMetadata {

// The resource deletion operation.
DELETE = 1;

// The resource creation operation.
CREATE = 2;
}

// Output only. The current operation state.
Expand All @@ -406,6 +420,48 @@ message OperationMetadata {
[(google.api.field_behavior) = OUTPUT_ONLY];
}

// Request message for
// [ProcessOpenLineageRunEvent][google.cloud.datacatalog.lineage.v1.ProcessOpenLineageRunEvent].
message ProcessOpenLineageRunEventRequest {
// Required. The name of the project and its location that should own the
// process, run, and lineage event.
string parent = 1 [(google.api.field_behavior) = REQUIRED];

// Required. OpenLineage message following OpenLineage format:
// https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json
google.protobuf.Struct open_lineage = 2
[(google.api.field_behavior) = REQUIRED];

// A unique identifier for this request. Restricted to 36 ASCII characters.
// A random UUID is recommended. This request is idempotent only if a
// `request_id` is provided.
string request_id = 3;
}

// Response message for
// [ProcessOpenLineageRunEvent][google.cloud.datacatalog.lineage.v1.ProcessOpenLineageRunEvent].
message ProcessOpenLineageRunEventResponse {
// Created process name.
// Format: `projects/{project}/locations/{location}/processes/{process}`.
string process = 1 [(google.api.resource_reference) = {
type: "datalineage.googleapis.com/Process"
}];

// Created run name.
// Format:
// `projects/{project}/locations/{location}/processes/{process}/runs/{run}`.
string run = 2 [
(google.api.resource_reference) = { type: "datalineage.googleapis.com/Run" }
];

// Created lineage event names.
// Format:
// `projects/{project}/locations/{location}/processes/{process}/runs/{run}/lineageEvents/{lineage_event}`.
repeated string lineage_events = 3 [(google.api.resource_reference) = {
type: "datalineage.googleapis.com/LineageEvent"
}];
}

// Request message for
// [CreateProcess][google.cloud.datacatalog.lineage.v1.CreateProcess].
message CreateProcessRequest {
Expand Down Expand Up @@ -542,6 +598,9 @@ message UpdateRunRequest {
// The list of fields to update. Currently not used. The whole message is
// updated.
google.protobuf.FieldMask update_mask = 2;

// If set to true and the run is not found, the request creates it.
bool allow_missing = 3;
}

// Request message for
Expand Down Expand Up @@ -693,7 +752,7 @@ message DeleteLineageEventRequest {
// Request message for
// [SearchLinks][google.cloud.datacatalog.lineage.v1.Lineage.SearchLinks].
message SearchLinksRequest {
// Required. The project and location you want search in the format `projects/*/locations/*`
// Required. The project and location you want search in.
string parent = 1 [
(google.api.field_behavior) = REQUIRED,
(google.api.resource_reference) = {
Expand Down Expand Up @@ -769,7 +828,7 @@ message Link {
// Request message for
// [BatchSearchLinkProcesses][google.cloud.datacatalog.lineage.v1.Lineage.BatchSearchLinkProcesses].
message BatchSearchLinkProcessesRequest {
// Required. The project and location you want search in the format `projects/*/locations/*`
// Required. The project and location where you want to search.
string parent = 1 [
(google.api.field_behavior) = REQUIRED,
(google.api.resource_reference) = {
Expand Down Expand Up @@ -861,9 +920,16 @@ message Origin {

// Looker Studio
LOOKER_STUDIO = 5;

// Dataproc
DATAPROC = 6;
}

// Type of the source.
//
// Use of a source_type other than `CUSTOM` for process creation
// or updating is highly discouraged, and may be restricted in the future
// without notice.
SourceType source_type = 1;

// If the source_type isn't CUSTOM, the value of this field should be a GCP
Expand Down
Loading