Referencing external types

[Quantumplation]

Hello!

First, I'm profoundly impressed with Typebox, so great job!

I've been working on a project that does code-generation based on a json schema I receive from elsewhere, and typebox has been incredibly useful.

However, for a few of the types that are referenced in the schema, there are corresponding classes, rather than just simple JSON objects, and I'd like to allow the user to pass in either the "on the wire" representation (ex: hex encoded string), a raw Uint8Array, or handles to web-assembly representations of the decoded object.

I know how to do a union type with the first two, and I think I have something working for the last, but I wanted to check that I was using the library as intended and wasn't missing something more obvious.

My first approach was to try using the TypeRegistry to register a custom type, but that doesn't seem to be what I'm looking for.

My second approach was to use Type.Unsafe<WasmObj>(Type.Any()). For example:

const Configuration = Type.Object({
  Data: Type.Union([
    Type.String(),
    Type.Uint8Array(), 
    Type.Unsafe<WasmObj>(Type.Any())
  ])
});

There are a few other similar constructs where the "plain" representation would be more complex than a string/uint8array, but I figure the same principle applies.

This seems to give me the inferred type I want, but I'm not sure if I'm misinterpreting how this is supposed to use this and will run into trouble later, or if there's a simpler approach that I missed.

(I did search for similar discussions, and found this one, which is what gave me the idea, but didn't fully answer my question about whether this is the "intended" way to do something like this).

[sinclairzx81]

@Quantumplation Hi, thanks for the kind words.

However, for a few of the types that are referenced in the schema, there are corresponding classes, rather than just simple JSON objects, and I'd like to allow the user to pass in either the "on the wire" representation (ex: hex encoded string), a raw Uint8Array, or handles to web-assembly representations of the decoded object.

For complex objects / classes and other non-json serializable data, the best option is probably to use a Transform type. A transform allows you apply encode and decode logic to a json serializable value (like a string). The following implements a simple Transform that encodes a WASM module as a base64 string over the wire, but decodes as a number[] representing bytes. The decodeWasm function also attempts to test the validity of the data by instancing a WebAssembly.Module as part of the decode (there may be a better way to validate the bytes)

import { Value } from '@sinclair/typebox/value'
import { Type } from '@sinclair/typebox'

// ------------------------------------------------------------------
// Wasm Type: Encode as Base64 over the wire
// ------------------------------------------------------------------
function encodeWasm(bytes: Iterable<number>): string {
  return btoa(String.fromCharCode(...bytes))
}
function decodeWasm(data: string): number[] {
  const decoded = Array.from(atob(data), char => char.charCodeAt(0))
  new WebAssembly.Module(new Uint8Array(decoded)) // will throw if bytes are invalid. There may be
                                                  // a better way to test if bytes are valid
  return decoded
}

const Wasm = Type.Transform(Type.String())
  .Decode(value => decodeWasm(value))
  .Encode(value => encodeWasm(value))
 
// ------------------------------------------------------------------
// Test
// ------------------------------------------------------------------

const bytecode = 'AGFzbQEAAAABBwFgAnx8AXwDAgEABgEABwcBA2FkZAAACgoBCAAgACABoA8LABEEbmFtZQAHBm1vZHVsZQIBAQ=='

const decoded = Value.Decode(Wasm, bytecode)

const encoded = Value.Encode(Wasm, decoded)

// .. instance and test result

const instance: any = new WebAssembly.Instance(new WebAssembly.Module(new Uint8Array(decoded)), {})

const result = instance.exports.add(1000, 2000)

console.log({
  decoded,
  encoded,
  result
})

// prints: {
//   decoded: [
//       0,  97, 115, 109,   1,   0,   0,   0,   1,   7,   1,  96,
//       2, 124, 124,   1, 124,   3,   2,   1,   0,   6,   1,   0,
//       7,   7,   1,   3,  97, 100, 100,   0,   0,  10,  10,   1,
//       8,   0,  32,   0,  32,   1, 160,  15,  11,   0,  17,   4,
//     110,  97, 109, 101,   0,   7,   6, 109, 111, 100, 117, 108,
//     101,   2,   1,   1
//   ],
//   encoded: 'AGFzbQEAAAABBwFgAnx8AXwDAgEABgEABwcBA2FkZAAACgoBCAAgACABoA8LABEEbmFtZQAHBm1vZHVsZQIBAQ==',
//   result: 3000
// }

---

TypeBox includes a couple of extension types for Date and Uint8Array but would advise against these for this use case. TypeBox includes these types for applications that use other encoding formats (like MsgPack and Cbor) which offer facilities for Date and Uint8Array encoding. If you are just working with Json, I would recommend Transforms as these offer similar functionality (albeit with the requirement to manually encode byte arrays as base64 strings)

Hope this helps
S

[Quantumplation]

Ooo, ok, yes, Transform looks like exactly what I need! I didn't connect the dots on transform initially. I'll give it a shot and report back on how it goes!

[Quantumplation]

@sinclairzx81 a related question: is it possible to reference external types in a module? I want to pre-define some of these and import them in my generated code, something like:

import { TWasmObj } from `@package/standard-library`;

const Contracts = Type.Module({
  Configuration: Type.Object({
    Data: TWasmObj,
    // Data: Type.Unsafe<WasmObj>(Type.Any()) <-- Inlined
  })
});
const Configuration = Contracts.Import("Configuration");
type Configuration = Exact<typeof Configuration>;

let s: Configuration = { data: 10 }; // 

but when I do this, the type inference seems to break down; s seems to be inferred as { [x: string]: any } and accepts any type. I haven't yet tested if the same thing happens for Transform, and if this is just a quirk of Unsafe, but I figured I'd ask while I'm here 😅

[Quantumplation]

Ok, so I experimented with Transform a little bit, and it feels backwards from what I want; If I do

const WasmObj = Type.Transform(Type.String()).Decode(...).Encode(...);
type WasmObj = Static<typeof WasmObj>;

let s: WasmObj = ...;

Then s infers as string.

What I'm looking for is a way for the type to infer as WasmObj in the type hierarchy, so that when I have a function like:

serialize<T extends TSchema>(type: T, data: Exact<T>): Uint8Array {
 ...
}

then a user can invoke this like serialize(WasmObj, someObj);

(in practice, WasmObj would be one of the fields in a larger type hierarchy, etc., but I'm simplifying out some of the details)