Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

serialize & deserialize from connect server #2

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
57cd768
initial stab at ts connect server
hotpocket Nov 6, 2024
d103974
use schemas from buf registry instead of locally generated ones
hotpocket Nov 6, 2024
2eaed62
remove unnecessary cas4ts
hotpocket Nov 7, 2024
88d34d7
readme with working examples
hotpocket Nov 7, 2024
a3017f7
simplify: remove unused server code
hotpocket Nov 7, 2024
c7a1fae
a better description
hotpocket Nov 7, 2024
a8f668a
remove unnecessary tsconfig
hotpocket Nov 7, 2024
e40a98e
remove leftover proto gen config
hotpocket Nov 7, 2024
29f6779
connect serializer v0 implementation
hotpocket Nov 7, 2024
6b641df
seems correct. may not be.
hotpocket Nov 7, 2024
1f45b63
added questions section to address unknowns.
hotpocket Nov 7, 2024
6e09283
3.9.3-edge.2
Nov 7, 2024
ffe3fff
Merge branch 'stateful:main' into v1
hotpocket Nov 7, 2024
65133fe
test server being called from ConnectSerializer.
hotpocket Nov 7, 2024
4e2819f
Merge branch 'stateful:main' into v1
hotpocket Nov 8, 2024
4129af5
deserialize now using test server
hotpocket Nov 8, 2024
6da075b
verify address is not empty
hotpocket Nov 12, 2024
5a272d1
Merge branch 'main' into v1
hotpocket Nov 13, 2024
f3b977a
wrong port#
hotpocket Nov 14, 2024
6dbf112
refactor transport and catch connect errors
hotpocket Nov 16, 2024
020d20f
attempt to connect using a grpc transport. tried with and without ce…
hotpocket Nov 20, 2024
07c4b94
actually attempt to use the grpcTransport and correct address.
hotpocket Nov 20, 2024
5c05076
sort out how to connect using tls and start the server with the prope…
hotpocket Nov 24, 2024
4105664
Merge branch 'main' into v1
hotpocket Nov 24, 2024
a0b2645
ca is not needed for successful connection.
hotpocket Nov 24, 2024
ba89a69
more detail on recent tls issue.
hotpocket Nov 24, 2024
d3362e5
committing this prior to Grpc feature extractions
hotpocket Nov 25, 2024
9a413c9
beginning of refactor of tyhpe specific functionality.
hotpocket Nov 26, 2024
1a00cdb
replace marshal functionality in Connect and Grpc serializers.
hotpocket Nov 26, 2024
e8a29f3
using a manually started server for now - subject to change.
hotpocket Nov 26, 2024
7c96341
commit to save state before refactor of caching components of grpcSer…
hotpocket Nov 27, 2024
c3f1cdc
move caching into base class so ConnectSerializer gets it too.
hotpocket Nov 27, 2024
58ab0ba
need to fix the error about a missing toBinary method then this will …
hotpocket Nov 27, 2024
28270d5
attempting to track down frontmatter property not having toBinary fun…
hotpocket Nov 28, 2024
44c42f9
explicitly cast proto.frontmatter so it will have necessary serilizat…
hotpocket Nov 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions fake_server/client.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import { createPromiseClient, PromiseClient } from '@connectrpc/connect'
import { createConnectTransport } from '@connectrpc/connect-node'
import {
DeserializeRequest,
SerializeRequest,
Notebook
} from '@buf/stateful_runme.bufbuild_es/runme/parser/v1/parser_pb'
import { ParserService } from '@buf/stateful_runme.connectrpc_es/runme/parser/v1/parser_connect'

// OLD MANUAL IMPORT PATHS from manual generation
// import { DeserializeRequest, SerializeRequest, Notebook } from './src/gen/runme/parser/v1/service_pb'
// import { ParserService } from './src/gen/runme/parser/v1/service_connect'

const transport = createConnectTransport({
baseUrl: 'http://localhost:1234',
httpVersion: '1.1'
})

async function main() {
const client: PromiseClient<typeof ParserService>= createPromiseClient(ParserService, transport)
// serialize something
const notebook = new Notebook({
cells: [{ value: 'markdown data', kind: 1 }],
metadata: {},
frontmatter: {}
})
const serializeRequest = new SerializeRequest()
serializeRequest.notebook = notebook
const sr = await client.serialize(serializeRequest)
console.log('Serialization result:')
console.log(sr)
console.log('String: ', new TextDecoder().decode(sr.result))

// and deserialize it
const deserializeRequest = new DeserializeRequest()
deserializeRequest.source = sr.result
const dr = await client.deserialize(deserializeRequest)
console.log('DeSerialization result:')
console.log(dr)
console.log('String: ', JSON.stringify(dr.notebook))
}

void main()
178 changes: 178 additions & 0 deletions fake_server/client2.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
/* eslint-disable quotes */
import fs from 'node:fs'
import path from 'node:path'
import net from 'node:net'
import { spawn, ChildProcess } from 'node:child_process'

import {
createPromiseClient,
PromiseClient,
Transport,
} from '@connectrpc/connect'
import {
ConnectTransportOptions,
GrpcTransportOptions,
} from '@connectrpc/connect-node'
import {
createConnectTransport,
createGrpcTransport,
} from '@connectrpc/connect-node'
import {
DeserializeRequest,
SerializeRequest,
Notebook,
} from '@buf/stateful_runme.bufbuild_es/runme/parser/v1/parser_pb'
import { ParserService } from '@buf/stateful_runme.connectrpc_es/runme/parser/v1/parser_connect'

// Parse command-line arguments
const args = process.argv.slice(2)
let protocol = 'grpc' as 'grpc' | 'connect'
let useTls = false

args.forEach((arg) => {
if (arg.startsWith('--protocol=')) {
const value = arg.split('=')[1]
if (value === 'grpc' || value === 'connect') {
protocol = value as 'grpc' | 'connect'
} else {
console.warn(`Invalid protocol: ${value}. Using default 'grpc'.`)
}
} else if (arg.startsWith('--useTls=')) {
const value = arg.split('=')[1]
useTls = value === 'true'
}
})

// Config
let transport: Transport
const host = 'localhost'
const port = 9999
const tlsPath = path.normalize(__dirname + '/../tls')
const keyPath = `${tlsPath}/key.pem`
const certPath = `${tlsPath}/cert.pem`
let config = {
baseUrl: `http://${host}:${port}`,
httpVersion: '1.1',
nodeOptions: {},
} as ConnectTransportOptions | GrpcTransportOptions
const serverBinary = path.normalize('../bin/runme')
const serverArgs = [
'server',
'--address', `${host}:${port}`,
'--tls', `${tlsPath}`,
'--runner', ...(useTls ? [] : [`--insecure`])
]

let serverProcess: ChildProcess | null = null

async function isServerRunning(port: number, host: string = '127.0.0.1'): Promise<boolean> {
return new Promise((resolve) => {
// it's running if we can connect to it
const client = net.createConnection({ port, host }, () => {
client.end() // Close this test connection
resolve(true) // Server is running
})
// it's not running if we can't connect to it
client.on('error', () => { resolve(false) })
})
}

// Function to spawn the server process
function spawnServer(): void {
const fullCommand = `${serverBinary} ${serverArgs.join(' ')}`
console.log('Starting server with command:')
console.log(fullCommand)

serverProcess = spawn(serverBinary, serverArgs, { stdio: 'inherit' })

serverProcess.on('error', (error) => {
console.error('Failed to start server process:', error)
})

serverProcess.on('exit', (code) => {
console.log(`Server process exited with code ${code}`)
})

console.log('Server process spawned successfully.')
}

// Function to kill the server process
function killServer(): void {
if (serverProcess) {
console.log('Killing server process...')
serverProcess.kill()
serverProcess = null
}
}

// Function to initialize the transport configuration
function setupTransport() {
if (protocol === 'connect') {
transport = createConnectTransport(config)
} else if (protocol === 'grpc') {
config.httpVersion = '2'
if (useTls) {
config.baseUrl = `https://${host}:${port}`
const cert = fs.readFileSync(certPath)
const key = fs.readFileSync(keyPath)
config.nodeOptions = {
cert,
key,
rejectUnauthorized: false, // Use only in dev for self-signed certificates
}
}
transport = createGrpcTransport(config)
}
}

// Main function
async function main() {
try {
console.log(`Using protocol: ${protocol}`)
console.log(`Using TLS: ${useTls}`)

// Check if server is running
const running = await isServerRunning(port, host)
if (!running) {
console.log('Server is not running. Spawning the server...')
spawnServer()
// Wait a few seconds for the server to start
await new Promise((resolve) => setTimeout(resolve, 5000))
} else {
console.log('Server is already running.')
}

// Set up the transport
setupTransport()

const client: PromiseClient<typeof ParserService> = createPromiseClient(ParserService, transport)

// Serialize something
const notebook = new Notebook({
cells: [{ value: 'markdown data', kind: 1 }],
metadata: {},
frontmatter: {},
})
const serializeRequest = new SerializeRequest()
serializeRequest.notebook = notebook
const sr = await client.serialize(serializeRequest)
console.log('Serialization result:')
console.log(sr)
console.log('String: ', new TextDecoder().decode(sr.result))

// And deserialize it
const deserializeRequest = new DeserializeRequest()
deserializeRequest.source = sr.result
const dr = await client.deserialize(deserializeRequest)
console.log('DeSerialization result:')
console.log(dr)
console.log('String: ', JSON.stringify(dr.notebook))
} catch (error) {
console.error('Error during execution:', error)
} finally {
// Kill the server process if it was spawned by this script
killServer()
}
}

void main()
20 changes: 20 additions & 0 deletions fake_server/con_serv.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/* eslint-disable @typescript-eslint/no-unused-vars */
import { fastifyConnectPlugin } from '@connectrpc/connect-fastify'
import { fastify } from 'fastify'

import routes from './routes'

async function main() {
const server = fastify()
await server.register(fastifyConnectPlugin, {
routes,
})
server.get('/', (_, reply) => {
reply.type('text/plain')
reply.send('Hello World!')
})
await server.listen({ host: 'localhost', port: 1234 })
console.log('server is listening at', server.addresses())
}

void main()
Loading