Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConfigService gRPC and JS API #2979

Merged
merged 13 commits into from
Oct 31, 2022
Merged
453 changes: 453 additions & 0 deletions go/internal/proto/config/config.pb.go

Large diffs are not rendered by default.

141 changes: 141 additions & 0 deletions go/internal/proto/config/config_grpc.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions props/configs/src/main/resources/dh-defaults.prop
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,16 @@ default.processEnvironmentFactory=io.deephaven.util.process.DefaultProcessEnviro

OperationInitializationThreadPool.threads=1


AuthHandlers=io.deephaven.auth.AnonymousAuthenticationHandler

# List of configuration properties to provide to unauthenticated clients, so that they can decide how best to prove their
# identity to the server.
authentication.client.configuration.list=AuthHandlers

# List of configuration properties to provide to authenticated clients, so they can interact with the server.
client.configuration.list=java.version,deephaven.version,barrage.version,
# Version list to add to the configuration property list. Each `=`-delimited pair denotes a short name for a versioned
# jar, and a class that is found in that jar. Any such keys will be made available to the client.configuration.list
# as <key>.version.
client.version.list=deephaven=io.deephaven.engine.table.Table,barrage=io.deephaven.barrage.flatbuf.BarrageMessageWrapper
6 changes: 5 additions & 1 deletion props/test-configs/src/main/resources/dh-tests.prop
Original file line number Diff line number Diff line change
Expand Up @@ -97,4 +97,8 @@ BarrageMessageProducer.maxSnapshotCellCount=50
BarrageStreamGenerator.batchSize=4

#BarrageTable.debug=true
#BarrageMessageProducer.debug=true
#BarrageMessageProducer.debug=true

AuthHandlers=io.deephaven.auth.AnonymousAuthenticationHandler
authentication.client.configuration.list=
client.version.list=
4 changes: 4 additions & 0 deletions proto/proto-backplane-grpc/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ RUN set -eux; \
/includes/deephaven/proto/application.proto \
/includes/deephaven/proto/inputtable.proto \
/includes/deephaven/proto/partitionedtable.proto \
/includes/deephaven/proto/config.proto \
/includes/deephaven/proto/storage.proto; \
/opt/protoc/bin/protoc \
--plugin=protoc-gen-ts=/usr/src/app/node_modules/.bin/protoc-gen-ts \
Expand All @@ -42,6 +43,7 @@ RUN set -eux; \
/includes/deephaven/proto/application.proto \
/includes/deephaven/proto/inputtable.proto \
/includes/deephaven/proto/partitionedtable.proto \
/includes/deephaven/proto/config.proto \
/includes/deephaven/proto/storage.proto; \
python3 -m grpc_tools.protoc \
--grpc_python_out=/generated/python \
Expand All @@ -55,6 +57,7 @@ RUN set -eux; \
/includes/deephaven/proto/application.proto \
/includes/deephaven/proto/inputtable.proto \
/includes/deephaven/proto/partitionedtable.proto \
/includes/deephaven/proto/config.proto \
/includes/deephaven/proto/storage.proto; \
/opt/protoc/bin/protoc \
--plugin=protoc-gen-go=/opt/protoc-gen-go \
Expand All @@ -72,4 +75,5 @@ RUN set -eux; \
/includes/deephaven/proto/application.proto \
/includes/deephaven/proto/inputtable.proto \
/includes/deephaven/proto/partitionedtable.proto \
/includes/deephaven/proto/config.proto \
/includes/deephaven/proto/storage.proto;
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Copyright (c) 2016-2022 Deephaven Data Labs and Patent Pending
*/
syntax = "proto3";

package io.deephaven.proto.backplane.grpc;

option java_multiple_files = true;
option optimize_for = SPEED;
option go_package = "github.com/deephaven/deephaven-core/go/internal/proto/config";

/**
* Provides simple configuration data to users. Unauthenticated users may call GetAuthenticationConstants
* to discover hints on how they should proceed with providing their identity, while already-authenticated
* clients may call GetConfigurationConstants for details on using the platform.
*/
service ConfigService {
rpc GetAuthenticationConstants(AuthenticationConstantsRequest) returns (AuthenticationConstantsResponse) {}
rpc GetConfigurationConstants(ConfigurationConstantsRequest) returns (ConfigurationConstantsResponse) {}
Comment on lines +18 to +19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering about the "Constants" part of it - at least based on the current impl, users could manually set and change these values at runtime...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, naming is here to be critiqued. I don't think we want the server to push changes to these configuration values, so from a given client's perspective, we should assume that config values received are functionally constants, but at the same time, there is nothing preventing the server from changing them at runtime without a restart.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On DHE the config values are certainly treated as constants. I'm alright if they change at runtime as long as there's a notification there's been a change, such that UI can re-load appropriately. It's definitely less complex if we just treat them as constants.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do have changes, sending a disconnect also would force the client to reconnect/fetch the server config values again as well. I don't think we're expecting any use cases where config values change on the fly, I'd think it's easier to just do disconnect/reload everything if the user does want to change config values (which shouldn't happen often?)

}

message AuthenticationConstantsRequest {}
message ConfigurationConstantsRequest {}
message AuthenticationConstantsResponse {
repeated ConfigPair config_values = 1;
}
message ConfigurationConstantsResponse {
repeated ConfigPair config_values = 1;
}
message ConfigPair {
string key = 1;
oneof value {//leave room for more types
string string_value = 2;
niloc132 marked this conversation as resolved.
Show resolved Hide resolved
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I do want to talk about pros and cons of our choices here wrt https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/struct.proto. Is there benefit to relying on Struct/Value instead? Or, we could potentially mirror it with our own type and leave out the parts we don't need right now:

message Value {
  // The kind of value.
  oneof kind {
    // Represents a string value.
    string string_value = 3;
  }
}

This might leave us room in the future if we did want to transition to upstream types.
Thoughts on repeated vs map<string, ? We get a bit stronger of typing w/ a map.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, in theory, will make the change.

I'm not sure what you mean by stronger typing though, the key is still a string the value is still whatever fits in the one-of/etc, which is still checked by tag. Indeed, the map<k,v> syntax is just sugar over a message with a 1 and 2 tag of the specified type, so key collisions can happen (though presumably the receiving side's runtime will prune the duplicates, like it would for non-repeated fields). Note especially the warning in the docs at https://developers.google.com/protocol-buffers/docs/proto3#backwards_compatibility:

Any protocol buffers implementation that supports maps must both produce and accept data that can be accepted by the above definition.

Functionally, the only difference here is that the oneof is moved out to its own message type, so we just pay for an extra tag "go decode a value now" "here's the type of the data" instead of just "here's the type of the data".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, great link. I didn't realize the semantics of map were equivalent to repeated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, PTAL.

4 changes: 4 additions & 0 deletions proto/raw-js-openapi/src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ require("deephaven/proto/inputtable_pb");
require("deephaven/proto/object_pb");
require("deephaven/proto/partitionedtable_pb");
require("deephaven/proto/storage_pb");
require("deephaven/proto/config_pb");
require("Flight_pb")
require("BrowserFlight_pb")
var sessionService = require("deephaven/proto/session_pb_service");
Expand All @@ -17,6 +18,7 @@ var inputTableService = require("deephaven/proto/inputtable_pb_service");
var objectService = require("deephaven/proto/object_pb_service");
var partitionedTableService = require("deephaven/proto/partitionedtable_pb_service");
var storageService = require("deephaven/proto/storage_pb_service");
var configService = require("deephaven/proto/config_pb_service");
var browserFlightService = require("BrowserFlight_pb_service");
var flightService = require("Flight_pb_service");

Expand Down Expand Up @@ -49,6 +51,8 @@ var io = { deephaven: {
partitionedtable_pb_service: partitionedTableService,
storage_pb: proto.io.deephaven.proto.backplane.grpc,
storage_pb_service: storageService,
config_pb: proto.io.deephaven.proto.backplane.grpc,
config_pb_service: configService,
},
barrage: {
"flatbuf": {
Expand Down
36 changes: 36 additions & 0 deletions py/client/pydeephaven/proto/config_pb2.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading