Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-13633] [Playground] Implement method to get a default example for each SDKs #16484

Merged
merged 46 commits into from
Mar 1, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
509b7ff
Implement method to get a default example for each SDKs
pavel-avilov Jan 12, 2022
e9cbe79
Add error handling
pavel-avilov Jan 12, 2022
68edd97
Added saving of precompiled objects catalog to cache at the server st…
Jan 12, 2022
ab16bea
Added caching of the catalog only in case of unspecified SDK
Jan 12, 2022
d181cce
Update regarding comments
Jan 13, 2022
5743386
Update regarding comments
Jan 13, 2022
38673a4
Simplified logging regarding comment
Jan 14, 2022
4071bdb
Get defaultExamplePath from the corresponding config
pavel-avilov Jan 14, 2022
3047469
Refactoring code
pavel-avilov Jan 14, 2022
d6e5d0b
Add the `link` field to response
pavel-avilov Jan 18, 2022
f80ebdd
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
pavel-avilov Jan 19, 2022
e035006
Remove gjson;
pavel-avilov Jan 19, 2022
c3c5c79
Refactoring code
pavel-avilov Jan 19, 2022
931b79e
Merge remote-tracking branch 'origin/BEAM-13632-cache-examples-data' …
pavel-avilov Jan 20, 2022
29e6f61
Getting default precompiled object from cache
pavel-avilov Jan 21, 2022
5688841
Refactoring code
pavel-avilov Jan 21, 2022
43fc19b
Added saving of precompiled objects catalog to cache at the server st…
Jan 12, 2022
27b0157
Added caching of the catalog only in case of unspecified SDK
Jan 12, 2022
b9b77b9
Update regarding comments
Jan 13, 2022
2187918
Update regarding comments
Jan 13, 2022
3655716
Simplified logging regarding comment
Jan 14, 2022
7fe3d8b
Updates regarding comments
Jan 24, 2022
6830670
Update for environment_service_test.go
Jan 24, 2022
af655a5
Merge remote-tracking branch 'origin/BEAM-13632-cache-examples-data' …
pavel-avilov Jan 24, 2022
b95103c
Get default example from catalog
pavel-avilov Jan 24, 2022
f4ae9a7
GetCatalogFromCacheOrStorage method
pavel-avilov Jan 25, 2022
276ddd1
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
pavel-avilov Jan 25, 2022
94a1613
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
pavel-avilov Jan 26, 2022
1550e3a
Update licenses
pavel-avilov Jan 26, 2022
8e6e1e5
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
pavel-avilov Feb 1, 2022
af3623c
Update licenses;
pavel-avilov Feb 1, 2022
a7d6ded
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
Feb 7, 2022
3c8bd03
[BEAM-13633][Playground]
Feb 7, 2022
f273192
[BEAM-13633][Playground]
Feb 7, 2022
ebf5888
[BEAM-13633][Playground]
Feb 8, 2022
29c28b5
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
Feb 8, 2022
e413153
[BEAM-13633][Playground]
Feb 8, 2022
da6baa0
Add code of the default example to response
pavel-avilov Feb 9, 2022
1ea1026
Revert "Add code of the default example to response"
pavel-avilov Feb 9, 2022
d8b7bdb
Refactoring code
pavel-avilov Feb 9, 2022
c189363
Refactoring code;
Feb 10, 2022
6e63cfb
Edit commentaries
pavel-avilov Feb 10, 2022
0bcb1c4
Refactoring code
pavel-avilov Feb 10, 2022
66a3ca5
Merge remote-tracking branch 'origin/master' into BEAM-13633_default_…
pavel-avilov Mar 1, 2022
52622a3
Merge branch 'master' into BEAM-13633_default_ex_for_sdks
pavel-avilov Mar 1, 2022
4f54530
Add bucket name to methods
pavel-avilov Mar 1, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion playground/api/v1/api.proto
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,12 @@ message GetPrecompiledObjectLogsRequest{
string cloud_path = 1;
}

// GetListOfPrecompiledObjectsResponse represent the map between sdk and categories for the sdk.
// GetDefaultPrecompiledObjectRequest contains information of the needed PrecompiledObject sdk.
message GetDefaultPrecompiledObjectRequest {
Sdk sdk = 1;
}

// GetPrecompiledObjectsResponse represent the map between sdk and categories for the sdk.
message GetPrecompiledObjectsResponse{
repeated Categories sdk_categories = 1;
}
Expand All @@ -204,6 +209,11 @@ message GetPrecompiledObjectLogsResponse {
string output = 1;
}

// GetDefaultPrecompiledObjectResponse represents the default PrecompiledObject and his category for the sdk.
message GetDefaultPrecompiledObjectResponse {
PrecompiledObject precompiled_object = 1;
}

service PlaygroundService {

// Submit the job for an execution and get the pipeline uuid.
Expand Down Expand Up @@ -244,4 +254,7 @@ service PlaygroundService {

// Get the logs of an PrecompiledObject.
rpc GetPrecompiledObjectLogs(GetPrecompiledObjectLogsRequest) returns (GetPrecompiledObjectLogsResponse);

// Get the default precompile object for the sdk.
rpc GetDefaultPrecompiledObject(GetDefaultPrecompiledObjectRequest) returns (GetDefaultPrecompiledObjectResponse);
}
27 changes: 26 additions & 1 deletion playground/backend/cmd/server/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ func (controller *playgroundController) GetPrecompiledObjects(ctx context.Contex
// GetPrecompiledObjectCode returns the code of the specific example
func (controller *playgroundController) GetPrecompiledObjectCode(ctx context.Context, info *pb.GetPrecompiledObjectCodeRequest) (*pb.GetPrecompiledObjectCodeResponse, error) {
cd := cloud_bucket.New()
codeString, err := cd.GetPrecompiledObject(ctx, info.GetCloudPath())
codeString, err := cd.GetPrecompiledObjectCode(ctx, info.GetCloudPath())
if err != nil {
logger.Errorf("GetPrecompiledObjectCode(): cloud storage error: %s", err.Error())
return nil, errors.InternalError("Error during getting Precompiled Object's code", "Error with cloud connection")
Expand Down Expand Up @@ -294,3 +294,28 @@ func (controller *playgroundController) GetPrecompiledObjectLogs(ctx context.Con
response := pb.GetPrecompiledObjectLogsResponse{Output: logs}
return &response, nil
}

// GetDefaultPrecompiledObject returns the default precompile object for sdk.
func (controller *playgroundController) GetDefaultPrecompiledObject(ctx context.Context, info *pb.GetDefaultPrecompiledObjectRequest) (*pb.GetDefaultPrecompiledObjectResponse, error) {
switch info.Sdk {
case pb.Sdk_SDK_UNSPECIFIED, pb.Sdk_SDK_SCIO:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can include SCIO as a supported SDK.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

logger.Errorf("GetDefaultPrecompiledObject(): unimplemented sdk: %s\n", info.Sdk)
return nil, errors.InvalidArgumentError("Error during preparing", "Sdk is not implemented yet: %s", info.Sdk.String())
}

bucket := cloud_bucket.New()
precompiledObject, err := bucket.GetDefaultPrecompiledObject(ctx, controller.env.BeamSdkEnvs.DefaultExamplePath())
if err != nil {
logger.Errorf("GetDefaultPrecompiledObject(): cloud storage error: %s", err.Error())
return nil, errors.InternalError("Error during getting default Precompiled Object", "Error with cloud connection")
}

response := pb.GetDefaultPrecompiledObjectResponse{PrecompiledObject: &pb.PrecompiledObject{
CloudPath: precompiledObject.CloudPath,
Name: precompiledObject.Name,
Description: precompiledObject.Description,
Type: precompiledObject.Type,
PipelineOptions: precompiledObject.PipelineOptions,
}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we return the code of the example as well? Or it will be requested from the frontend after receiving this response?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this information will be enough for frontend.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please also add the link field according to this one

return &response, nil
}
3 changes: 2 additions & 1 deletion playground/backend/configs/SDK_GO.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@
"test_args": [
"test",
"-v"
]
],
"default_example": "SDK_GO/MinimalWordCount"
}
3 changes: 2 additions & 1 deletion playground/backend/configs/SDK_JAVA.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,6 @@
"-cp",
"bin:",
"org.junit.runner.JUnitCore"
]
],
"default_example": "SDK_JAVA/MinimalWordCount"
}
3 changes: 2 additions & 1 deletion playground/backend/configs/SDK_PYTHON.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@
"test_cmd": "pytest",
"compile_args": [],
"run_args": [],
"test_args": []
"test_args": [],
"default_example": "SDK_PYTHON/WordCountWithMetrics"
}
1 change: 1 addition & 0 deletions playground/backend/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ require (
github.com/google/uuid v1.3.0
github.com/improbable-eng/grpc-web v0.14.1
github.com/rs/cors v1.8.0
github.com/tidwall/gjson v1.13.0
go.uber.org/goleak v1.1.12
google.golang.org/api v0.58.0
google.golang.org/grpc v1.41.0
Expand Down
6 changes: 6 additions & 0 deletions playground/backend/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,12 @@ github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tidwall/gjson v1.13.0 h1:3TFY9yxOQShrvmjdM76K+jc66zJeT6D3/VFFYCGQf7M=
github.com/tidwall/gjson v1.13.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
Expand Down
Loading