Add grpc investigation, change proto location, gen TS client, fix typo (

#649)

cmd/secret-storage-backend/README.md

@@ -17,7 +17,7 @@ By default, the Secret Storage Backend has the `aws_secretsmanager` provider ena
 
 1. Create AWS security credentials with `SecretsManagerReadWrite` policy.
 2. Export environment variables:
-    
+
    ```bash
     export AWS_ACCESS_KEY_ID="{accessKey}"
     export AWS_SECRET_ACCESS_KEY="{secretKey}"
@@ -28,7 +28,7 @@ By default, the Secret Storage Backend has the `aws_secretsmanager` provider ena
     APP_LOGGER_DEV_MODE=true go run ./cmd/secret-storage-backend/main.go
     ```
 
-The server listens to gRPC calls according to the [Storage Backend Protocol Buffers schema](../../pkg/hub/api/grpc/storage_backend.proto).
+The server listens to gRPC calls according to the [Storage Backend Protocol Buffers schema](../../hub-js/proto/storage_backend.proto).
 To perform such calls, you can use e.g. [Insomnia](https://insomnia.rest/) tool.
 
 ### Dotenv provider

docs/investigation/local-hub-v2/README.md

@@ -7,3 +7,4 @@ Ideally, we want to implement Local Hub in Go.
 There are the following proof of concept projects:
 - [PostgreSQL](./postresql/README.md) - The goal of this investigation is to find an efficient way to implement Local Hub backed with PostgreSQL.
 - [Dgraph](./dgraph/README.md) - The goal of this investigation is to check whether Dgraph can be used as a Local Hub replacement.
+- [Generate TypeScript gRPC client](./ts-grpc/README.md) - The goal of this investigation is to find which tools we should use to generate the gRPC client for delegated storage backend.

docs/investigation/local-hub-v2/ts-grpc/README.md

@@ -0,0 +1,54 @@
+## How to generate gRPC client for TypeScript
+
+This document aggregates raw notes from a short investigation as a part of the [#626](https://github.com/capactio/capact/issues/626) issue.
+
+## Goal
+
+The goal is to find which tools we should use to generate the gRPC client for delegated storage backend. We want to:
+- generate TypeScript Types
+- have support for async/await
+
+## Options
+
+There are numerous option to generate gRPC clients. The most popular ones:
+- [`@grpc/proto-loader`](https://www.npmjs.com/package/@grpc/proto-loader). This is an official library created by [gRPC](https://github.com/grpc) organization. Doesn't support [`async/await`](https://github.com/grpc/grpc-node/issues/54) natively.
+	**Stats:**
+	- Weekly downloads: `4,632,833`
+	- Last publish: `5/01/2022`
+	- Stars: `3.3k` (but it is in monorepo, so stars are not directly related to this package)
+
+- [`ts-protoc-gen`](https://www.npmjs.com/package/ts-protoc-gen). Community plugin that requires proto compiler usage.
+	**Stats:**
+	- Weekly downloads: `80,981`
+	- Last publish: `16/08/2021`
+	- Stars: `419`
+- [`grpc_tools_node_protoc_ts`](https://www.npmjs.com/package/grpc_tools_node_protoc_ts). Community plugin that requires proto compiler usage.
+	**Stats:**
+	- Weekly downloads: `101,496`
+	- Last publish: `27/04/2021`
+	- Stars: `1.1k`
+- [`ts-proto`](https://www.npmjs.com/package/ts-proto)
+	**Stats:**
+	- Weekly downloads: `44,656`
+	- Last publish: `27/02/2022`
+	- Stars: `752`
+
+All the above tools generate code without native support for `async/await` but there are other options:
+- [Promisify @grpc-js service client with typescript](https://gist.github.com/smnbbrv/f147fceb4c29be5ce877b6275018e294) - tested but didn't work, got error: `This expression is not callable. Type 'never' has no call signatures.`
+- [promisifyAll](https://docs.servicestack.net/grpc-nodejs) - works only for JavaScript.
+- [grpc-promise](https://github.com/carlessistare/grpc-promise) - seems to be not maintained anymore.
+- [Dapr approach](https://github.com/dapr/js-sdk/blob/18e46fed1b4f52589be667cfbdab577ddb238eb1/src/implementation/Client/GRPCClient/state.ts#L14) - they just write services by their own.
+- [nice-grpc](https://github.com/deeplay-io/nice-grpc) - works only with code generated by dedicated plugins.
+- [gRPC helper](https://github.com/xizhibei/grpc-helper) - not tested, enables more that we need.
+
+## Testing
+
+I tested two possible solutions:
+- use [`@grpc/proto-loader`](https://www.npmjs.com/package/@grpc/proto-loader) and create dedicated service, same as in [Dapr approach](https://github.com/dapr/js-sdk/blob/18e46fed1b4f52589be667cfbdab577ddb238eb1/src/implementation/Client/GRPCClient/state.ts#L14).
+- use [`ts-proto`](https://github.com/stephenh/ts-proto) and [nice-grpc](https://github.com/deeplay-io/nice-grpc) to automatically provide `async/await`
+
+Simple SPIKE is described in [grpc-gen](./grpc-gen/README.md).
+
+## Summary
+
+When we will use the gRPC client we still need to create dedicated services as we need to implement more that simple call. Because of that, the official solution seems to be the best one. On the other hand, there is already [gen-grpc-resources.sh](../../../../hack/gen-grpc-resources.sh) which uses the proto compiler with dedicated plugins. In my opinion, being consistent is more important in this case and usage of `ts-proto` seems to be the best. Additionally, we will get rid of manually generated promises because we can use  [nice-grpc](https://github.com/deeplay-io/nice-grpc) for that.

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/.gitignore

@@ -0,0 +1,2 @@
+node_modules
+/dist

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/.prettierignore

@@ -0,0 +1,4 @@
+proto
+dist
+package-lock.json
+node_modules

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/README.md

@@ -0,0 +1,85 @@
+# grpc-proto-loader example
+
+This example shows how to use different libraries to generate a gRPC client. Used libraries:
+
+- [`@grpc/proto-loader`](https://www.npmjs.com/package/@grpc/proto-loader)
+- [`ts-proto`](https://github.com/stephenh/ts-proto)
+
+## App layout
+
+- [client-official.ts](client-official.ts) - Showcase usage of gRPC client generated via [`@grpc/proto-loader`](https://www.npmjs.com/package/@grpc/proto-loader).
+- [client-official-await.ts](client-official-await.ts) - Showcase async/await usage of gRPC client generated via [`@grpc/proto-loader`](https://www.npmjs.com/package/@grpc/proto-loader). Promises are implemented manually.
+- [client-ts-plugin.ts](client-ts-plugin.ts) - Showcase async/await usage of gRPC client generated via [`ts-proto`](https://github.com/stephenh/ts-proto). Promises are provided via [`nice-grpc`](https://www.npmjs.com/package/nice-grpc).
+
+## Generating the clients
+
+Install dependencies:
+
+```bash
+npm install
+```
+
+### [`@grpc/proto-loader`](https://www.npmjs.com/package/@grpc/proto-loader)
+
+To generate the TypeScript files into [`./proto/official`](./proto/ts-plugin), run:
+
+```bash
+npx proto-loader-gen-types --longs=String --enums=String --defaults --oneofs --outDir=proto/official --grpcLib=@grpc/grpc-js ../../../../../hub-js/proto/storage_backend.proto
+```
+
+This is aliased as a npm script:
+
+```bash
+npm run build:official-proto
+```
+
+### [`ts-proto`](https://github.com/stephenh/ts-proto)
+
+To generate the TypeScript files into [`./proto/ts-plugin`](./proto/ts-plugin), run:
+
+> **NOTE:** The `./proto/ts-plugin` directory needs to exist.
+
+```bash
+npx grpc_tools_node_protoc \
+  --plugin=protoc-gen-ts_proto=$(npm bin)/protoc-gen-ts_proto \
+  --ts_proto_out=./proto/ts-plugin \
+  --ts_proto_opt=esModuleInterop=true,outputServices=generic-definitions,useExactTypes=false \
+  --proto_path='../../../../../hub-js/proto/' \
+  ./../../../../../hub-js/proto/storage_backend.proto
+```
+
+This is aliased as a npm script:
+
+```bash
+npm run build:plugin-proto
+```
+
+### Running example scenario
+
+This simple project demonstrates the different libraries you can use to perform gRPC calls.
+
+1. Build clients:
+
+   ````bash
+   npm run build
+   ````
+
+2. Start the [secret storage](./../../../../../cmd/secret-storage-backend/README.md) with `dotenv` provider enabled:
+
+   ````bash
+   APP_LOGGER_DEV_MODE=true APP_SUPPORTED_PROVIDERS="dotenv" go run ./cmd/secret-storage-backend/main.go
+   ````
+
+3. Now run the client by specifying which example you want to run:
+
+   ```bash
+   npm run start:client-official
+   ```
+
+   ```bash
+   npm run start:client-official-await
+   ```
+
+   ```bash
+   npm run start:client-ts-plugin
+   ```

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/client-official-await.ts

@@ -0,0 +1,117 @@
+import * as grpc from '@grpc/grpc-js';
+import * as protoLoader from '@grpc/proto-loader';
+
+import { ProtoGrpcType } from './proto/official/storage_backend';
+import { StorageBackendClient } from './proto/official/storage_backend/StorageBackend';
+import { GetValueRequest } from './proto/official/storage_backend/GetValueRequest';
+import { PROTO_PATH, TARGET } from './config';
+import { GetValueResponse__Output } from './proto/official/storage_backend/GetValueResponse';
+import { ServiceError } from '@grpc/grpc-js';
+import {
+  OnCreateRequest,
+  OnDeleteRequest,
+} from './proto/ts-plugin/storage_backend';
+import { OnDeleteResponse__Output } from './proto/official/storage_backend/OnDeleteResponse';
+import { OnCreateResponse__Output } from './proto/official/storage_backend/OnCreateResponse';
+
+async function main() {
+  const svc = new DelegatedStorageClient(PROTO_PATH, TARGET);
+
+  const provider = 'dotenv'; // or 'aws_secretsmanager';
+  const onCreate: OnCreateRequest = {
+    typeInstanceId: '1234',
+    context: Buffer.from(`{"provider":"${provider}"}`),
+    value: Buffer.from(`{"key":"${provider}"}`),
+  };
+  const { value, ...ti } = onCreate; // extract common id and context to `ti`
+
+  const createRes = await svc
+    .onCreate(onCreate)
+    .catch((err: ServiceError) => console.error(err));
+  if (createRes) {
+    console.log(`TypeInstance created: ${createRes.context}`);
+  }
+  const onGet: GetValueRequest = {
+    ...ti,
+    resourceVersion: 1,
+  };
+  const getRes = await svc
+    .getValue(onGet)
+    .catch((err: ServiceError) => console.error(err));
+  if (getRes) {
+    console.log(`Fetch TypeInstance: ${getRes.value}`);
+  }
+
+  const onDel: OnDeleteRequest = ti;
+  await svc
+    .onDelete(onDel)
+    .then(() => console.info('Deleted TypeInstance'))
+    .catch((err: ServiceError) => console.error(err));
+}
+
+export default class DelegatedStorageClient {
+  private client: StorageBackendClient;
+
+  constructor(protoPath: string, target: string) {
+    const packageDefinition = protoLoader.loadSync(protoPath);
+    const proto = grpc.loadPackageDefinition(
+      packageDefinition
+    ) as unknown as ProtoGrpcType;
+    this.client = new proto.storage_backend.StorageBackend(
+      target,
+      grpc.credentials.createInsecure()
+    );
+  }
+
+  async getValue(
+    req: GetValueRequest
+  ): Promise<GetValueResponse__Output | undefined> {
+    return new Promise((resolve, reject) =>
+      this.client.GetValue(req, (error, res) => {
+        if (error) return reject(error);
+        return resolve(res);
+      })
+    );
+  }
+
+  async onCreate(
+    req: OnCreateRequest
+  ): Promise<OnCreateResponse__Output | undefined> {
+    return new Promise((resolve, reject) =>
+      this.client.onCreate(req, (error, res) => {
+        if (error) return reject(error);
+        return resolve(res);
+      })
+    );
+  }
+
+  async onDelete(
+    req: OnDeleteRequest
+  ): Promise<OnDeleteResponse__Output | undefined> {
+    return new Promise((resolve, reject) =>
+      this.client.onDelete(req, (error, res) => {
+        if (error) return reject(error);
+        return resolve(res);
+      })
+    );
+  }
+}
+
+// TODO: Didn't get make the generic approach working.
+type Callback<A, B> = (err?: A, res?: B) => void;
+
+const promisify =
+  <T, A, B>(
+    fn: (req: T, cb: Callback<grpc.ServiceError, B>) => void
+  ): ((req: T) => Promise<B | undefined>) =>
+  (req: T) =>
+    new Promise((resolve, reject) => {
+      fn(req, (err?: grpc.ServiceError, res?: B) => {
+        if (err) return reject(err);
+        return resolve(res);
+      });
+    });
+
+(async () => {
+  await main();
+})();

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/client-official.ts

@@ -0,0 +1,42 @@
+import * as grpc from '@grpc/grpc-js';
+import * as protoLoader from '@grpc/proto-loader';
+
+import { ProtoGrpcType } from './proto/official/storage_backend';
+import { GetValueRequest } from './proto/official/storage_backend/GetValueRequest';
+import { PROTO_PATH, TARGET } from './config';
+
+function main() {
+  const packageDefinition = protoLoader.loadSync(PROTO_PATH);
+  const proto = grpc.loadPackageDefinition(
+    packageDefinition
+  ) as unknown as ProtoGrpcType;
+  const client = new proto.storage_backend.StorageBackend(
+    TARGET,
+    grpc.credentials.createInsecure()
+  );
+
+  const deadline = new Date();
+  deadline.setSeconds(deadline.getSeconds() + 5);
+  client.waitForReady(deadline, (error?: Error) => {
+    if (error) {
+      console.log(`Client connect error: ${error.message}`);
+    } else {
+      const provider = 'dotenv'; // or 'aws_secretsmanager';
+      const request: GetValueRequest = {
+        typeInstanceId: '123',
+        resourceVersion: 1,
+        context: Buffer.from(`{"provider":"${provider}"}`),
+      };
+      client.GetValue(request, (error, res) => {
+        if (error) {
+          console.error(error);
+          console.error('Is not found: ', error.code == grpc.status.NOT_FOUND);
+        } else if (res) {
+          console.log(`(client) Got server response: ${res.value}`);
+        }
+      });
+    }
+  });
+}
+
+main();

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/client-ts-plugin.ts

@@ -0,0 +1,53 @@
+import {
+  GetValueRequest,
+  OnCreateRequest,
+  OnDeleteRequest,
+  StorageBackendDefinition,
+} from './proto/ts-plugin/storage_backend';
+import { createChannel, createClient, Client } from 'nice-grpc';
+import { TARGET } from './config';
+import { ServiceError } from '@grpc/grpc-js';
+
+async function main() {
+  const channel = createChannel(TARGET);
+  const client: Client<typeof StorageBackendDefinition> = createClient(
+    StorageBackendDefinition,
+    channel
+  );
+
+  const provider = 'dotenv'; // or 'aws_secretsmanager';
+  const onCreate: OnCreateRequest = {
+    typeInstanceId: '1234',
+    context: Buffer.from(`{"provider":"${provider}"}`),
+    value: Buffer.from(`{"key":"${provider}"}`),
+  };
+  const { value, ...ti } = onCreate; // extract common id and context to `ti`
+
+  const createRes = await client
+    .onCreate(onCreate)
+    .catch((err: ServiceError) => console.error(err));
+  if (createRes) {
+    console.log(`TypeInstance created: ${createRes.context}`);
+  }
+
+  const onGet: GetValueRequest = {
+    ...ti,
+    resourceVersion: 1,
+  };
+  const getRes = await client
+    .getValue(onGet)
+    .catch((err: ServiceError) => console.error(err));
+  if (getRes) {
+    console.log(`Fetch TypeInstance: ${getRes.value}`);
+  }
+
+  const onDel: OnDeleteRequest = ti;
+  await client
+    .onDelete(onDel)
+    .then(() => console.info('Deleted TypeInstance'))
+    .catch((err: ServiceError) => console.error(err));
+}
+
+(async () => {
+  await main();
+})();

docs/investigation/local-hub-v2/ts-grpc/grpc-gen/config.ts

@@ -0,0 +1,2 @@
+export const TARGET = '0.0.0.0:50051';
+export const PROTO_PATH = '../../../../../hub-js/proto/storage_backend.proto';