Abi Generation V2 (#1459)

* naive first marshalling attemmpt * int 8-int64 supported * negative numbers support * support maps * funky benchmark * structs reflection caching attempt * some fuzz tests * fix speed comment * relocate implementation out of test * benchmark * rename test to TestMakeSureMarshalUnmarshalIsNotTooSlow * fix fuzz test * spec tests for js implant * update spec tests * remove fuzz test * simplify to 2 types * restore original logic * rewrite marshal with avalanchego's wrappers.Packer * pack bytes with uint32 and everything else with uint16 * check for long arrays and strings, marshal maps with uint16 * update benchmark results * move to codec * come back to codec.packer * speed up TestMakeSureMarshalUnmarshalIsNotTooSlow a bit * support pointer to a struct * auto marshaller integration * lint * remove a slow test breaking CI * fix linter errors * faster reflection cache * minimize test to exclude testing errors * unsafe type caching * simplify benchmark * update benchmarks * add benchmark results * add benchmem * add benchmem results * deprecate string operations * move empty address error * empty file * lint * remove .prof * correct 'marshall' to 'marshal' according to Go conventions * simplify codec.Packer * get chainid from tmpnet instead of the platform (#1458) * lint * change to linearcodec * lint * go mod tidy * add serialize tag to Transfer * abi generation * spec tests * ABI in RPC * HasTypeID as a separate iface * add TypeParser.GetRegisteredTypes method * move ABI to core API * remove unused errors * auto size calculation * rename LinearCodecInstance * lint * update from #1198 * from clean slate * return abi logic * nit: remove function name in panics * transaction test nit * catch up with main * lint * rename HasTypeID to Typed * separate package for abi * restore spec tests * treat codec.Address as a byte array while serializing * comments and nits * use set.Set instead of map[reflect.Type]bool * lint * remove memo field * require serialize=true * remove a breaker * calculate ABI in place * use ABI as struct in implementation * stable ABI hash * ABI wants its own ABI * rename abi to vmabi * remove ABI for ABI * redo map as an array * clean up test specs a bit * basic codegen and refactor tests WIP * trying different naming * spec simplification WIP * go generate * lower case json * proper codegen test * further simplify spec tests * transfer test * file based spec test * simplify tests * ABI of ABI * Outer/Inner struct tests for TS debug * lint * check ABI * lost in merge * remove comment lines in abi test * remove a debug statement * move mock abi file * rename to abigen * use cobra * Update codec/address.go Co-authored-by: aaronbuchwald <[email protected]> Signed-off-by: containerman17 <[email protected]> * require that the " characters in address string * rename ABI-related stuff * fix tests after renaming * test full marshal cycle * TestDescribeVM * Update abi/codegen.go Co-authored-by: aaronbuchwald <[email protected]> Signed-off-by: containerman17 <[email protected]> * remove StringAsBytes * add unicode package * lint * go mod tidy * flatten types def in abi * don't use mixed receivers * inline vm.Hash into a test * rename abi.VM to abi.ABI * remove embed * funish renaming * get rid of vm.getabi * nit avoid redundant import alias * DescribeVM -> NewABI * lint * mock gen * share typesAlreadyProcessed across describing multiple actions * put a comment on each test * Update abi/auto_marshal_abi_spec_test.go Co-authored-by: aaronbuchwald <[email protected]> Signed-off-by: containerman17 <[email protected]> * nit: objectBytes * remove mustPrintOrderedJSON * comment on empty names * comment on IsUpper * revert typealias * TODO here to switch to the new address format * We should follow the style of funcName does X * use rune in cobra * comment on Dereference * comment on serialize tag * use %s and t instead of t.String() * readme * rename mockabi_test * lint --------- Signed-off-by: containerman17 <[email protected]> Co-authored-by: aaronbuchwald <[email protected]>
ava-labs · Sep 12, 2024 · 4778979 · 4778979
1 parent cdf17ba
commit 4778979
Show file tree

Hide file tree

Showing 47 changed files with 1,218 additions and 5 deletions.
diff --git a/abi/README.md b/abi/README.md
@@ -0,0 +1,96 @@
+# ABI Package
+
+## Overview
+The ABI package provides functionality for marshaling and unmarshaling actions. It is designed to work across different language implementations.
+
+## ABI Format
+The ABI is defined in JSON format, as shown in the `abi.json` file:
+```json
+{
+    "actions": [
+        {
+            "id": 1,
+            "action": "MockObjectSingleNumber"
+        },
+    ],
+    "types": [
+        {
+            "name": "MockObjectSingleNumber",
+            "fields": [
+                {
+                    "name": "Field1",
+                    "type": "uint16"
+                }
+            ]
+        },
+    ]
+}
+```
+
+The ABI consists of two main sections:
+- actions: A list of action definitions, each with a typeID and action name (action name specifies the type)
+- types: A dictionary of types including their name and corresponding fields
+
+The type in each field must either be included in the ABI's `types` or in the list of [Supported Primitive Types](#supported-primitive-types).
+
+## Test Vectors
+This implementation provides `testdata/` for implementations in any other language.
+
+To verify correctness, an implementation can implement the following pseudocode:
+```
+abi = abi.json
+
+for filename in testdata/*.hex:
+    if filename.endswith(".hash.hex"):
+        continue
+    expectedHex = readFile(filename)
+    json = readFile(filename.replace(".hex", ".json"))
+
+    actualHex = Marshal(abi, json)
+    if actualHex != expectedHex:
+        raise "Hex values do not match"
+
+```
+
+## ABI Verification
+Frontends can use the ABI to display proper action and field names. For a wallet to verify it knows what it's signing, it must ensure that a canonical hash of the ABI is included in the message it signs.
+
+A correct VM will verify the signature against the same ABI hash, such that verification fails if the wallet signed an action against a different than expected ABI.
+
+This enables frontends to provide a verifiable display of what they are asking users to sign.
+
+## Constraints
+- Actions require an ID, other structs / types do not require one
+- Multiple structs with the same name from different packages are not supported
+- Maps are not supported; use slices (arrays) instead
+- Built-in types include the special case type aliases: `codec.Address` and `codec.Bytes`
+
+## Generating Golang Bindings
+Use cmd/abigen to automatically generate Go bindings from an ABI's JSON.
+
+For example, to auto-generate golang bindings for the test ABI provided in `./abi/testdata/abi.json` run:
+
+```sh
+go run ./cmd/abigen/ ./abi/testdata/abi.json ./example.go --package=testpackage
+```
+
+This should generate the same code that is present in `./abi/mockabi_test.go`.
+
+## Supported Primitive Types
+
+| Type     | Range/Description                                        | JSON Serialization | Binary Serialization                  |
+|----------|----------------------------------------------------------|--------------------|---------------------------------------|
+| `bool`   | true or false                                            | boolean            | 1 byte                                |
+| `uint8`  | numbers from 0 to 255                                    | number             | 1 byte                                |
+| `uint16` | numbers from 0 to 65535                                  | number             | 2 bytes                               |
+| `uint32` | numbers from 0 to 4294967295                             | number             | 4 bytes                               |
+| `uint64` | numbers from 0 to 18446744073709551615                   | number             | 8 bytes                               |
+| `int8`   | numbers from -128 to 127                                 | number             | 1 byte                                |
+| `int16`  | numbers from -32768 to 32767                             | number             | 2 bytes                               |
+| `int32`  | numbers from -2147483648 to 2147483647                   | number             | 4 bytes                               |
+| `int64`  | numbers from -9223372036854775808 to 9223372036854775807 | number             | 8 bytes                               |
+| `Address`| 33 byte array                                            | base64             | 33 bytes                              |
+| `Bytes`  | byte array                                               | base64             | uint32 length + bytes                 |
+| `string` | string                                                   | string             | uint16 length + bytes                 |
+| `[]T`    | for any `T` in the above list, serialized as an array    | array              | uint32 length + elements              |
+
diff --git a/abi/abi.go b/abi/abi.go
@@ -0,0 +1,179 @@
+// Copyright (C) 2024, Ava Labs, Inc. All rights reserved.
+// See the file LICENSE for licensing terms.
+
+package abi
+
+import (
+	"fmt"
+	"reflect"
+	"strings"
+
+	"github.com/ava-labs/avalanchego/utils/set"
+
+	"github.com/ava-labs/hypersdk/codec"
+)
+
+type ABI struct {
+	Actions []Action `serialize:"true" json:"actions"`
+	Types   []Type   `serialize:"true" json:"types"`
+}
+
+var _ codec.Typed = (*ABI)(nil)
+
+func (ABI) GetTypeID() uint8 {
+	return 0
+}
+
+type Field struct {
+	Name string `serialize:"true" json:"name"`
+	Type string `serialize:"true" json:"type"`
+}
+
+type Action struct {
+	ID     uint8  `serialize:"true" json:"id"`
+	Action string `serialize:"true" json:"action"`
+}
+
+type Type struct {
+	Name   string  `serialize:"true" json:"name"`
+	Fields []Field `serialize:"true" json:"fields"`
+}
+
+func NewABI(actions []codec.Typed) (ABI, error) {
+	vmActions := make([]Action, 0)
+	vmTypes := make([]Type, 0)
+	typesSet := set.Set[string]{}
+	typesAlreadyProcessed := set.Set[reflect.Type]{}
+
+	for _, action := range actions {
+		actionABI, typeABI, err := describeAction(action, typesAlreadyProcessed)
+		if err != nil {
+			return ABI{}, err
+		}
+		vmActions = append(vmActions, actionABI)
+		for _, t := range typeABI {
+			if !typesSet.Contains(t.Name) {
+				vmTypes = append(vmTypes, t)
+				typesSet.Add(t.Name)
+			}
+		}
+	}
+	return ABI{Actions: vmActions, Types: vmTypes}, nil
+}
+
+// describeAction generates the Action and Types for a single action.
+// It handles both struct and pointer types, and recursively processes nested structs.
+// Does not support maps or interfaces - only standard go types, slices, arrays and structs
+func describeAction(action codec.Typed, typesAlreadyProcessed set.Set[reflect.Type]) (Action, []Type, error) {
+	t := reflect.TypeOf(action)
+	if t.Kind() == reflect.Ptr {
+		t = t.Elem()
+	}
+
+	actionABI := Action{
+		ID:     action.GetTypeID(),
+		Action: t.Name(),
+	}
+
+	typesABI := make([]Type, 0)
+	typesLeft := []reflect.Type{t}
+
+	// Process all types, including nested ones
+	for {
+		var nextType reflect.Type
+		nextTypeFound := false
+		for _, anotherType := range typesLeft {
+			if !typesAlreadyProcessed.Contains(anotherType) {
+				nextType = anotherType
+				nextTypeFound = true
+				break
+			}
+		}
+		if !nextTypeFound {
+			break
+		}
+
+		fields, moreTypes, err := describeStruct(nextType)
+		if err != nil {
+			return Action{}, nil, err
+		}
+
+		typesABI = append(typesABI, Type{
+			Name:   nextType.Name(),
+			Fields: fields,
+		})
+		typesLeft = append(typesLeft, moreTypes...)
+
+		typesAlreadyProcessed.Add(nextType)
+	}
+
+	return actionABI, typesABI, nil
+}
+
+// describeStruct analyzes a struct type and returns its fields and any nested struct types it found
+func describeStruct(t reflect.Type) ([]Field, []reflect.Type, error) {
+	kind := t.Kind()
+
+	if kind != reflect.Struct {
+		return nil, nil, fmt.Errorf("type %s is not a struct", t)
+	}
+
+	fields := make([]Field, 0)
+	otherStructsSeen := make([]reflect.Type, 0)
+
+	for i := 0; i < t.NumField(); i++ {
+		field := t.Field(i)
+		fieldType := field.Type
+		fieldName := field.Name
+
+		// Skip any field that will not be serialized by the codec
+		serializeTag := field.Tag.Get("serialize")
+		if serializeTag != "true" {
+			continue
+		}
+
+		// Handle JSON tag for field name override
+		jsonTag := field.Tag.Get("json")
+		if jsonTag != "" {
+			parts := strings.Split(jsonTag, ",")
+			fieldName = parts[0]
+		}
+
+		if field.Anonymous && fieldType.Kind() == reflect.Struct {
+			// Handle embedded struct by flattening its fields
+			embeddedFields, moreTypes, err := describeStruct(fieldType)
+			if err != nil {
+				return nil, nil, err
+			}
+			fields = append(fields, embeddedFields...)
+			otherStructsSeen = append(otherStructsSeen, moreTypes...)
+		} else {
+			arrayPrefix := ""
+
+			// Here we assume that all types without a name are slices.
+			// We completely ignore the fact that maps exist as we don't support them.
+			// Types like `type Bytes = []byte` are slices technically, but they have a name
+			// and we need them to be named types instead of slices.
+			for fieldType.Name() == "" {
+				arrayPrefix += "[]"
+				fieldType = fieldType.Elem()
+			}
+
+			typeName := arrayPrefix + fieldType.Name()
+
+			// Add nested structs and pointers to structs to the list for processing
+			if fieldType.Kind() == reflect.Struct {
+				otherStructsSeen = append(otherStructsSeen, fieldType)
+			} else if fieldType.Kind() == reflect.Ptr {
+				otherStructsSeen = append(otherStructsSeen, fieldType.Elem())
+			}
+
+			fields = append(fields, Field{
+				Name: fieldName,
+				Type: typeName,
+			})
+		}
+	}
+
+	return fields, otherStructsSeen, nil
+}
diff --git a/abi/abi_test.go b/abi/abi_test.go
@@ -0,0 +1,45 @@
+// Copyright (C) 2024, Ava Labs, Inc. All rights reserved.
+// See the file LICENSE for licensing terms.
+
+package abi
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/ava-labs/hypersdk/codec"
+)
+
+func TestNewABI(t *testing.T) {
+	require := require.New(t)
+
+	actualABI, err := NewABI([]codec.Typed{
+		MockObjectSingleNumber{},
+		MockActionTransfer{},
+		MockObjectAllNumbers{},
+		MockObjectStringAndBytes{},
+		MockObjectArrays{},
+		MockActionWithTransfer{},
+		MockActionWithTransferArray{},
+		Outer{},
+	})
+	require.NoError(err)
+
+	expectedABIJSON := mustReadFile(t, "testdata/abi.json")
+	expectedABI := mustJSONParse[ABI](t, string(expectedABIJSON))
+
+	require.Equal(expectedABI, actualABI)
+}
+
+func TestGetABIofABI(t *testing.T) {
+	require := require.New(t)
+
+	actualABI, err := NewABI([]codec.Typed{ABI{}})
+	require.NoError(err)
+
+	expectedABIJSON := mustReadFile(t, "testdata/abi.abi.json")
+	expectedABI := mustJSONParse[ABI](t, string(expectedABIJSON))
+
+	require.Equal(expectedABI, actualABI)
+}