Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add blob scalar type #2091

Merged
merged 18 commits into from
Dec 7, 2023
Merged
3 changes: 2 additions & 1 deletion client/descriptions.go
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ const (
FieldKind_DATETIME FieldKind = 10
FieldKind_STRING FieldKind = 11
FieldKind_STRING_ARRAY FieldKind = 12
_ FieldKind = 13 // safe to repurpose (was never used)
FieldKind_BYTES FieldKind = 13
_ FieldKind = 14 // safe to repurpose (was never used)
_ FieldKind = 15 // safe to repurpose (was never used)

Expand Down Expand Up @@ -204,6 +204,7 @@ var FieldKindStringToEnumMapping = map[string]FieldKind{
"String": FieldKind_STRING,
"[String]": FieldKind_NILLABLE_STRING_ARRAY,
"[String!]": FieldKind_STRING_ARRAY,
"Bytes": FieldKind_BYTES,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: I think this is a strange way of representing this, and that [Byte] is more sensible and fits much better with the rest of the types, and any potential future Byte, [Byte!] or Byte! types.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to Blob as we discussed in the standup.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Keenan :)

}

// RelationType describes the type of relation between two types.
Expand Down
3 changes: 3 additions & 0 deletions db/collection_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,9 @@ func validateFieldSchema(val *fastjson.Value, field client.FieldDescription) (an

case client.FieldKind_FOREIGN_OBJECT, client.FieldKind_FOREIGN_OBJECT_ARRAY:
return nil, NewErrFieldOrAliasToFieldNotExist(field.Name)

case client.FieldKind_BYTES:
return getString(val)
}

return nil, client.NewErrUnhandledType("FieldKind", field.Kind)
Expand Down
3 changes: 3 additions & 0 deletions request/graphql/schema/collection.go
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,7 @@ func astTypeToKind(t ast.Type) (client.FieldKind, error) {
typeFloat string = "Float"
typeDateTime string = "DateTime"
typeString string = "String"
typeBytes string = "Bytes"
)

switch astTypeVal := t.(type) {
Expand Down Expand Up @@ -379,6 +380,8 @@ func astTypeToKind(t ast.Type) (client.FieldKind, error) {
return client.FieldKind_DATETIME, nil
case typeString:
return client.FieldKind_STRING, nil
case typeBytes:
return client.FieldKind_BYTES, nil
default:
return client.FieldKind_FOREIGN_OBJECT, nil
}
Expand Down
6 changes: 5 additions & 1 deletion request/graphql/schema/descriptions.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
gql "github.com/sourcenetwork/graphql-go"

"github.com/sourcenetwork/defradb/client"
schemaTypes "github.com/sourcenetwork/defradb/request/graphql/schema/types"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: Seeing this make me wonder if the type should be added to the graphql-go package.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to that. Should we move the other types there as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me nervous, I think we added DateTime to that package, but graphql-go is supposed to be a general purpose gql package, not a defra package. And we may/do want to get rid of it/replace it at somepoint

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we added DateTime to that package

No, DateTime was already defined as a scalar in the graphql-go package, but there were some problems with it as it relates to Defra integration, there was PR that got merged on the graphql-go package related to the DateTime stuff, but it wasn't the implementation/definition of the DateTime scalar. (reference: #931 and sourcenetwork/graphql-go#8)

I do agree though that we shouldn't have to define the scalar in the graphql-go package, it exposes the necessary types/functions for us to define our own scalars outside that package, exactly as they're being used here. Moreover, the fact that we have our own defined custom scalars implies that w.e package we may replace the graphql-go package with, should implement the "base" scalars as defined in the GQL spec, and all we have to worry about is the specific "custom" scalars that are neatly organized in a single place.

)

var (
Expand All @@ -31,9 +32,10 @@ var (
gql.String: client.FieldKind_STRING,
&gql.Object{}: client.FieldKind_FOREIGN_OBJECT,
&gql.List{}: client.FieldKind_FOREIGN_OBJECT_ARRAY,
// Custom scalars
schemaTypes.BytesScalarType: client.FieldKind_BYTES,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Related to the other comment from Fred, I think it should be made clear that these are custom scalars, as the comment on line 36 suggests ("More custom ones to come"). Just a quick comment separating base scalars from custom scalars for clarity 👍

// More custom ones to come
// - JSON
// - ByteArray
// - Counters
}

Expand All @@ -52,6 +54,7 @@ var (
client.FieldKind_STRING: gql.String,
client.FieldKind_STRING_ARRAY: gql.NewList(gql.NewNonNull(gql.String)),
client.FieldKind_NILLABLE_STRING_ARRAY: gql.NewList(gql.String),
client.FieldKind_BYTES: schemaTypes.BytesScalarType,
}

// This map is fine to use
Expand All @@ -70,6 +73,7 @@ var (
client.FieldKind_STRING: client.LWW_REGISTER,
client.FieldKind_STRING_ARRAY: client.LWW_REGISTER,
client.FieldKind_NILLABLE_STRING_ARRAY: client.LWW_REGISTER,
client.FieldKind_BYTES: client.LWW_REGISTER,
client.FieldKind_FOREIGN_OBJECT: client.NONE_CRDT,
client.FieldKind_FOREIGN_OBJECT_ARRAY: client.NONE_CRDT,
}
Expand Down
3 changes: 3 additions & 0 deletions request/graphql/schema/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,9 @@ func defaultTypes() []gql.Type {
gql.Int,
gql.String,

// Custom Scalar types
schemaTypes.BytesScalarType,

// Base Query types

// Sort/Order enum
Expand Down
63 changes: 63 additions & 0 deletions request/graphql/schema/types/scalars.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// Copyright 2023 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package types

import (
"encoding/hex"

"github.com/sourcenetwork/graphql-go"
"github.com/sourcenetwork/graphql-go/language/ast"
)

var BytesScalarType = graphql.NewScalar(graphql.ScalarConfig{
Name: "Bytes",
Description: "The `Bytes` scalar type represents an array of bytes.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: If it is an array of bytes, why does it only appear to accept string values?

todo: I'm not sure if it does only accept string values, could you add an integration test documenting the beheviour when provided an array of numbers:

testUtils.CreateDoc{
	Doc: `{
		"name": "John",
		"data": [0,1,2,3...]
	}`,
},

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is a bug in theclient.Document API. You can also do this to a string field:

testUtils.CreateDoc{
		Doc: `{
			"name":  12345
		}`,
}
testUtils.CreateDoc{
		Doc: `{
			"name":  false
		}`,
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite a bug, we just dont do any checking of the saved value. The read should fail though I think. Does the read of [0,1,2,3...] fail? Can we please document the behaviour, as it is a more likely question than assigning 1234 to a boolean :)

Serialize: func(value any) any {
switch value := value.(type) {
case []byte:
return hex.EncodeToString(value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Why are you encoding/decoding at all here (including in ParseValue)?

todo: The graphql library lacks any documentation here, but we can add some to our code to document why Serialize and ParseValue exist.

case *[]byte:
return hex.EncodeToString(*value)
default:
return nil
Copy link
Contributor

@AndrewSisley AndrewSisley Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Please document why you return nil here

}
},
ParseValue: func(value any) any {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: Thank you for the reminder as to how much I dislike the graphql library and it's totally rubbish way of (not) handling errors...

switch value := value.(type) {
case string:
data, err := hex.DecodeString(value)
if err != nil {
return nil
}
return data
case *string:
data, err := hex.DecodeString(*value)
if err != nil {
return nil
}
return data
default:
return nil
Copy link
Contributor

@AndrewSisley AndrewSisley Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Why does this return nil instead of []byte{}?

EDIT:
question: Is this a silent failure? Like the errors above:

if err != nil {
	return nil
}

todo: If so, please document

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just following how the other types work in the graphql-go library. It seems like nil is the return value when the type cannot be parsed.

}
},
ParseLiteral: func(valueAST ast.Value) any {
switch valueAST := valueAST.(type) {
case *ast.StringValue:
data, err := hex.DecodeString(valueAST.Value)
if err != nil {
return nil
}
return data
default:
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Similar to another comment, please document this - it looks like a silent error, but I only know that by guessing given the:

if err != nil {
	return nil
}

stuff in other code blocks

}
},
})
82 changes: 82 additions & 0 deletions request/graphql/schema/types/scalars_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
// Copyright 2023 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package types

import (
"testing"

"github.com/sourcenetwork/graphql-go/language/ast"
"github.com/stretchr/testify/assert"
)

func TestBytesScalarTypeSerialize(t *testing.T) {
input := []byte{0, 255}
output := "00ff"

cases := []struct {
input any
expect any
}{
{input, output},
{&input, output},
{nil, nil},
{0, nil},
{false, nil},
}
for _, c := range cases {
result := BytesScalarType.Serialize(c.input)
assert.Equal(t, c.expect, result)
}
}

func TestBytesScalarTypeParseValue(t *testing.T) {
input := "00ff"
output := []byte{0, 255}
invalid := "invalid"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Why is this invalid?

todo: Please document why this is invalid, if it remains invalid.


cases := []struct {
input any
expect any
}{
{input, output},
{&input, output},
{invalid, nil},
{&invalid, nil},
{nil, nil},
{0, nil},
{false, nil},
}
for _, c := range cases {
result := BytesScalarType.ParseValue(c.input)
assert.Equal(t, c.expect, result)
}
}

func TestBytesScalarTypeParseLiteral(t *testing.T) {
cases := []struct {
input ast.Value
expect any
}{
{&ast.StringValue{Value: "00ff"}, []byte{0, 255}},
{&ast.StringValue{Value: "invalid"}, nil},
{&ast.IntValue{}, nil},
{&ast.BooleanValue{}, nil},
{&ast.NullValue{}, nil},
{&ast.EnumValue{}, nil},
{&ast.FloatValue{}, nil},
{&ast.ListValue{}, nil},
{&ast.ObjectValue{}, nil},
}
for _, c := range cases {
result := BytesScalarType.ParseLiteral(c.input)
assert.Equal(t, c.expect, result)
}
}
60 changes: 60 additions & 0 deletions tests/integration/mutation/update/field_kinds/bytes_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
// Copyright 2023 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package field_kinds

import (
"testing"

testUtils "github.com/sourcenetwork/defradb/tests/integration"
)

func TestMutationUpdate_WithBytesField(t *testing.T) {
test := testUtils.TestCase{
Description: "Simple update of bytes field",
Actions: []any{
testUtils.SchemaUpdate{
Schema: `
type Users {
name: String
data: Bytes
}
`,
},
testUtils.CreateDoc{
Doc: `{
"name": "John",
"data": "00FE"
}`,
},
testUtils.UpdateDoc{
Doc: `{
"data": "00FF"
}`,
},
testUtils.Request{
Request: `
query {
Users {
data
}
}
`,
Results: []map[string]any{
{
"data": "00FF",
},
},
},
},
}

testUtils.ExecuteTestCase(t, test)
}
46 changes: 46 additions & 0 deletions tests/integration/schema/simple_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -271,3 +271,49 @@ func TestSchemaSimpleErrorsGivenNonNullManyRelationField(t *testing.T) {

testUtils.ExecuteTestCase(t, test)
}

func TestSchemaSimpleCreatesSchemaGivenTypeWithBytesField(t *testing.T) {
test := testUtils.TestCase{
Actions: []any{
testUtils.SchemaUpdate{
Schema: `
type Users {
data: Bytes
}
`,
},
testUtils.IntrospectionRequest{
Request: `
query {
__type (name: "Users") {
name
fields {
name
type {
name
kind
}
}
}
}
`,
ExpectedData: map[string]any{
"__type": map[string]any{
"name": "Users",
"fields": DefaultFields.Append(
Field{
"name": "data",
"type": map[string]any{
"kind": "SCALAR",
"name": "Bytes",
},
},
).Tidy(),
},
},
},
},
}

testUtils.ExecuteTestCase(t, test)
}
24 changes: 0 additions & 24 deletions tests/integration/schema/updates/add/field/kind/invalid_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,30 +64,6 @@ func TestSchemaUpdatesAddFieldKind9(t *testing.T) {
testUtils.ExecuteTestCase(t, test)
}

func TestSchemaUpdatesAddFieldKind13(t *testing.T) {
Copy link
Contributor

@AndrewSisley AndrewSisley Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Please add a tests (one by enum number, and one by String-name) in .../schema/updates/add/field/kind/foo.go that tests that this field works with PatchSchema

test := testUtils.TestCase{
Description: "Test schema update, add field with kind deprecated (13)",
Actions: []any{
testUtils.SchemaUpdate{
Schema: `
type Users {
name: String
}
`,
},
testUtils.SchemaPatch{
Patch: `
[
{ "op": "add", "path": "/Users/Fields/-", "value": {"Name": "foo", "Kind": 13} }
]
`,
ExpectedError: "no type found for given name. Type: 13",
},
},
}
testUtils.ExecuteTestCase(t, test)
}

func TestSchemaUpdatesAddFieldKind14(t *testing.T) {
test := testUtils.TestCase{
Description: "Test schema update, add field with kind deprecated (14)",
Expand Down