encoding/json: mangled unmarshal string result

[AllenX2018]

What version of Go are you using (go version)?

$ go version
go version go1.14.1 windows/amd64

Does this issue reproduce with the latest release?

Yes, it reproduces with the latest release.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows

What did you do?

I create a type whose real type is string and implement encoding.TextUnmarshaler and encoding.TextUnmarshaler. I use this type as a map's key type. Then I just simply do a roundtrip test with encoding/json's Marshal and Unmarshal function. When my input json contains non-ascii characters, something unexpected happened.

type KeyString string

func (k KeyString) MarshalText() ([]byte, error) {
	return []byte("Prefix__" + k), nil
}

func (k *KeyString) UnmarshalText(text []byte) error {
	*k = KeyString(strings.TrimPrefix(string(text), "Prefix__"))
	return nil
}

func test() {
	b := []byte(`{"开源":"12345开源"}`)
	m := make(map[KeyString]string)

	if err := json.Unmarshal(b, &m); err == nil {
		fmt.Println(m)
		s, _ := json.Marshal(m)
		fmt.Println(string(s))
	}
}

What did you expect to see?

Output:
map[开源:12345开源]
{"开源":"12345开源"}

What did you see instead?

Output:
map[开��:12345开源]
{"开\ufffd\ufffd�":"12345开源"}

[dsnet]

\cc @mvdan

Git bisect indicates 5469770.

Since this is a regression for go1.14, the fix should be probably be cherry-pick worthy for the go1.14 releases branch.

[dsnet]

@gopherbot, please open a backport issue for 1.14.

[gopherbot]

Backport issue(s) opened: #38106 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[mvdan]

Thanks @dsnet - I'll try to have a look at it this weekend.

[gopherbot]

Change https://golang.org/cl/226218 mentions this issue: encoding/json: don't mangle strings in an edge case when decoding

[gopherbot]

Change https://golang.org/cl/233057 mentions this issue: [release-branch.go1.14] encoding/json: don't mangle strings in an edge case when decoding