Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mismatched len of columns when inserting empty map into JSON column #1113

Closed
leodido opened this issue Oct 9, 2023 · 1 comment · Fixed by #1116
Closed

Mismatched len of columns when inserting empty map into JSON column #1113

leodido opened this issue Oct 9, 2023 · 1 comment · Fixed by #1116
Labels

Comments

@leodido
Copy link
Contributor

leodido commented Oct 9, 2023

Describe the bug

Steps to reproduce

  1. Enable the experimental JSON type (eg., SET allow_experimental_object_type = 1)
  2. Give to a column the experimental JSON type
  3. Insert an empty map (ie., map[string]interface{}{})

Expected behavior

I'd expect the JSON column to accept empty maps like map[string]interface{}{}.

The same way it works if inputting a string representing an empty JSON (eg., "{}").

Instead, it results in an error (see below) coming from lib/proto/block.go.

Code example

To prove this bug I've modified your example clickhouse_api/json.go by adding a fourth column (Col4) (code below).

package main

import (
	"context"
	"fmt"

	"github.com/ClickHouse/clickhouse-go/v2"
)

func InsertReadJSON() error {
	opts := &clickhouse.Options{
		Settings: clickhouse.Settings{
			"allow_experimental_object_type": 1,
		},
	}

	conn, err := clickhouse.Open(opts)
	if err != nil {
		return err
	}
	ctx := context.Background()

	conn.Exec(ctx, "DROP TABLE IF EXISTS example")

	if err = conn.Exec(ctx, `
		CREATE TABLE example (
				Col1 JSON,
				Col2 JSON,
				Col3 JSON,
				Col4 JSON
			)
			Engine Memory
		`); err != nil {
		return err
	}

	type User struct {
		Name     string `json:"name"`
		Age      uint8  `json:"age"`
		Password string `ch:"-"`
	}

	defer func() {
		conn.Exec(ctx, "DROP TABLE example")
	}()
	batch, err := conn.PrepareBatch(ctx, "INSERT INTO example")
	if err != nil {
		return err
	}
	// we can insert JSON as either a string, struct or map
	col1Data := `{"name": "Clicky McClickHouse", "age": 40, "password": "password"}`
	col2Data := User{
		Name:     "Clicky McClickHouse Snr",
		Age:      uint8(80),
		Password: "random",
	}
	col3Data := map[string]any{
		"name":     "Clicky McClickHouse Jnr",
		"age":      uint8(10),
		"password": "clicky",
	}
	col4Data := map[string]interface{}{} // Notice that `{}` works resulting in an output of `map[_dummy:0]`

	// both named and unnamed can be added with slices
	if err = batch.Append(col1Data, col2Data, col3Data, col4Data); err != nil {
		return err
	}

	if err = batch.Send(); err != nil {
		return err
	}
	// we can scan JSON into either a map or struct
	var (
		col1 map[string]any
		col2 map[string]any
		col3 User
		col4 map[string]any
	)
	// named tuples can be retrieved into a map or slices, unnamed just slices
	if err = conn.QueryRow(ctx, "SELECT * FROM example").Scan(&col1, &col2, &col3, &col4); err != nil {
		return err
	}
	fmt.Printf("row: col1=%v, col2=%v, col3=%v\n, col4=%v\n", col1, col2, col3, col4)

	return nil
}

func main() {
	if err := InsertReadJSON(); err != nil {
		fmt.Println(err)
	}
}

Error log

When running the code above you'll get:

clickhouse [Encode]:  mismatched len of columns - expected 1, received 0 for col Col4

Notice that using col4 = "{}" instead it works as expected - ie. outputting the following:

row: col1=map[age:40 name:Clicky McClickHouse password:password], col2=map[age:80 name:Clicky McClickHouse Snr], col3={Clicky McClickHouse Jnr 10 }
, col4=map[_dummy:0]

Configuration

Environment

  • Client version: 23.9.1.1854 (official build)
  • Language version: github.com/ClickHouse/clickhouse-go/v2 v2.14.2
  • OS: 23.9.1 revision 54466
  • Interface: ClickHouse API

ClickHouse server

  • ClickHouse Server version: 23.9.1 revision 54466
  • ClickHouse Server non-default settings, if any: NONE
  • CREATE TABLE statements for tables involved: NONE
@leodido leodido added the bug label Oct 9, 2023
@jkaflik
Copy link
Contributor

jkaflik commented Oct 10, 2023

Hi @leodido

Thanks for reporting this. I can confirm this issue exists:

package issues

import (
	"context"
	"github.com/ClickHouse/clickhouse-go/v2"
	clickhouse_tests "github.com/ClickHouse/clickhouse-go/v2/tests"
	"github.com/stretchr/testify/require"
	"testing"
)

func Test1113(t *testing.T) {
	var (
		conn, err = clickhouse_tests.GetConnection("issues", clickhouse.Settings{
			"max_execution_time":             60,
			"allow_experimental_object_type": true,
		}, nil, &clickhouse.Compression{
			Method: clickhouse.CompressionLZ4,
		})
	)
	ctx := context.Background()
	require.NoError(t, err)
	const ddl = `
		CREATE TABLE test_1113 (
			col_1 JSON,
			col_2 JSON
		) Engine MergeTree() ORDER BY tuple()
		`
	require.NoError(t, conn.Exec(ctx, ddl))
	defer func() {
		conn.Exec(ctx, "DROP TABLE IF EXISTS test_1113")
	}()

	batch, err := conn.PrepareBatch(context.Background(), "INSERT INTO test_1113")
	require.NoError(t, err)

	v1 := map[string]struct {
		Str string
	}{"a": {Str: "value"}}
	v2 := map[string]any{}

	require.NoError(t, batch.Append(v1, v2))
	require.NoError(t, batch.Send())
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants