Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

serialize & deserialize from connect server #2

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
57cd768
initial stab at ts connect server
hotpocket Nov 6, 2024
d103974
use schemas from buf registry instead of locally generated ones
hotpocket Nov 6, 2024
2eaed62
remove unnecessary cas4ts
hotpocket Nov 7, 2024
88d34d7
readme with working examples
hotpocket Nov 7, 2024
a3017f7
simplify: remove unused server code
hotpocket Nov 7, 2024
c7a1fae
a better description
hotpocket Nov 7, 2024
a8f668a
remove unnecessary tsconfig
hotpocket Nov 7, 2024
e40a98e
remove leftover proto gen config
hotpocket Nov 7, 2024
29f6779
connect serializer v0 implementation
hotpocket Nov 7, 2024
6b641df
seems correct. may not be.
hotpocket Nov 7, 2024
1f45b63
added questions section to address unknowns.
hotpocket Nov 7, 2024
6e09283
3.9.3-edge.2
Nov 7, 2024
ffe3fff
Merge branch 'stateful:main' into v1
hotpocket Nov 7, 2024
65133fe
test server being called from ConnectSerializer.
hotpocket Nov 7, 2024
4e2819f
Merge branch 'stateful:main' into v1
hotpocket Nov 8, 2024
4129af5
deserialize now using test server
hotpocket Nov 8, 2024
6da075b
verify address is not empty
hotpocket Nov 12, 2024
5a272d1
Merge branch 'main' into v1
hotpocket Nov 13, 2024
f3b977a
wrong port#
hotpocket Nov 14, 2024
6dbf112
refactor transport and catch connect errors
hotpocket Nov 16, 2024
020d20f
attempt to connect using a grpc transport. tried with and without ce…
hotpocket Nov 20, 2024
07c4b94
actually attempt to use the grpcTransport and correct address.
hotpocket Nov 20, 2024
5c05076
sort out how to connect using tls and start the server with the prope…
hotpocket Nov 24, 2024
4105664
Merge branch 'main' into v1
hotpocket Nov 24, 2024
a0b2645
ca is not needed for successful connection.
hotpocket Nov 24, 2024
ba89a69
more detail on recent tls issue.
hotpocket Nov 24, 2024
d3362e5
committing this prior to Grpc feature extractions
hotpocket Nov 25, 2024
9a413c9
beginning of refactor of tyhpe specific functionality.
hotpocket Nov 26, 2024
1a00cdb
replace marshal functionality in Connect and Grpc serializers.
hotpocket Nov 26, 2024
e8a29f3
using a manually started server for now - subject to change.
hotpocket Nov 26, 2024
7c96341
commit to save state before refactor of caching components of grpcSer…
hotpocket Nov 27, 2024
c3f1cdc
move caching into base class so ConnectSerializer gets it too.
hotpocket Nov 27, 2024
58ab0ba
need to fix the error about a missing toBinary method then this will …
hotpocket Nov 27, 2024
28270d5
attempting to track down frontmatter property not having toBinary fun…
hotpocket Nov 28, 2024
44c42f9
explicitly cast proto.frontmatter so it will have necessary serilizat…
hotpocket Nov 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 25 additions & 13 deletions src/extension/serializer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -425,14 +425,23 @@ export abstract class SerializerBase implements NotebookSerializer, Disposable {
return
}

if (GrpcSerializer.isDocumentSessionOutputs(doc.metadata)) {
if (SerializerBase.isDocumentSessionOutputs(doc.metadata)) {
this.togglePreviewButton(false)
return
}

this.cacheDocUriMapping.set(cacheId, doc.uri)
}

public static isDocumentSessionOutputs(metadata: { [key: string]: any } | undefined): boolean {
if (!metadata) {
// it's not session outputs unless known
return false
}
const sessionOutputId = metadata[RUNME_FRONTMATTER_PARSED]?.['runme']?.['session']?.['id']
return Boolean(sessionOutputId)
}

async handleCloseNotebook(doc: NotebookDocument) {
const cacheId = SerializerBase.getDocumentCacheId(doc.metadata)
/**
Expand Down Expand Up @@ -658,6 +667,11 @@ export class GrpcSerializer extends SerializerBase {
return Uri.parse(GrpcSerializer.getSourceFilePath(outputsUri.fsPath))
}

// should probably refactor other calls to this and/or @deprecate it
public static isDocumentSessionOutputs(metadata: { [key: string]: any } | undefined): boolean {
return SerializerBase.isDocumentSessionOutputs(metadata)
}

protected applyIdentity(data: typeof this.protoNotebookType): typeof this.protoNotebookType {
const identity = this.lifecycleIdentity
switch (identity) {
Expand All @@ -679,15 +693,6 @@ export class GrpcSerializer extends SerializerBase {
return data
}

public static isDocumentSessionOutputs(metadata: { [key: string]: any } | undefined): boolean {
if (!metadata) {
// it's not session outputs unless known
return false
}
const sessionOutputId = metadata[RUNME_FRONTMATTER_PARSED]?.['runme']?.['session']?.['id']
return Boolean(sessionOutputId)
}

public override async switchLifecycleIdentity(
notebook: NotebookDocument,
identity: RunmeIdentity,
Expand Down Expand Up @@ -866,7 +871,7 @@ export class GrpcSerializer extends SerializerBase {
metadata: { ['runme.dev/frontmatter']?: string },
kernel?: Kernel,
): Frontmatter {
return Marshal.frontmatter(metadata, kernel)
return Marshal.frontmatter(metadata, kernel) as Frontmatter
}

protected async reviveNotebook(
Expand Down Expand Up @@ -979,6 +984,9 @@ export class ConnectSerializer extends SerializerBase {
marshalFrontmatter,
}) as typeof this.protoNotebookType

// cast so when we serialize we don't get missing methods due to it not being a real proto
notebook.frontmatter = new es_proto.Frontmatter(notebook.frontmatter)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this cast and the one below for the maskedNotebook squash a very annoying bug.


if (marshalFrontmatter) {
data.metadata ??= {}
data.metadata[RUNME_FRONTMATTER_PARSED] = notebook.frontmatter
Expand All @@ -993,6 +1001,7 @@ export class ConnectSerializer extends SerializerBase {
const maskedNotebook = Marshal.notebook(data, maskedNotebookProto, {
marshalFrontmatter,
}) as typeof this.protoNotebookType
maskedNotebook.frontmatter = new es_proto.Frontmatter(maskedNotebook.frontmatter)
const cacheOutputs = this.cacheNotebookOutputs(notebook, maskedNotebook, cacheId)
const request = this.client!.serialize(serialRequest)

Expand Down Expand Up @@ -1024,7 +1033,7 @@ export class ConnectSerializer extends SerializerBase {
session = {
id: sid,
document: { relativePath },
} as any
} as es_proto.RunmeSession
}

const outputs = { enabled: true, summary: true }
Expand Down Expand Up @@ -1137,7 +1146,10 @@ class Marshal {
return notebookProto
}

static frontmatter(metadata: { ['runme.dev/frontmatter']?: string }, kernel?: Kernel) {
static frontmatter(
metadata: { ['runme.dev/frontmatter']?: string },
kernel?: Kernel,
): es_proto.Frontmatter | Frontmatter {
if (
!metadata.hasOwnProperty('runme.dev/frontmatter') ||
typeof metadata['runme.dev/frontmatter'] !== 'string'
Expand Down