We just upgraded to version v1.19.0 and are getting panics from the client

[fmstephe]

goroutine 15262745 [running]:
panic(0xb82ac0, 0xc4200120e0)
        /usr/lib/go/src/runtime/panic.go:500 +0x1a1
github.com/aerospike/aerospike-client-go.(*unpacker).unpackBlob(0xc43cc1bbb8, 0x0, 0x1, 0xc43383c810, 0xc43383c901, 0xc43cc1b968, 0x4090b1)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/unpacker.go:192 +0x165
github.com/aerospike/aerospike-client-go.(*unpacker).unpackObject(0xc43cc1bbb8, 0xc4317acc01, 0xc43cc1b9d8, 0xc43cc1b9c8, 0x0, 0x0)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/unpacker.go:294 +0x4cd
github.com/aerospike/aerospike-client-go.(*unpacker).unpackMapNormal(0xc43cc1bbb8, 0xa38, 0xc42036fba8, 0x1095ec0, 0xc435e45760)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/unpacker.go:116 +0x88
github.com/aerospike/aerospike-client-go.(*unpacker).unpackMap(0xc43cc1bbb8, 0xa38, 0x57a, 0x0, 0x0, 0x57a)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/unpacker.go:109 +0x52
github.com/aerospike/aerospike-client-go.(*unpacker).UnpackMap(0xc43cc1bbb8, 0x1094d40, 0xc42036fba8, 0x57a, 0x57a)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/unpacker.go:98 +0x6e
github.com/aerospike/aerospike-client-go.bytesToParticle(0x13, 0xc43c26e000, 0x1f4bb, 0x1f4bb, 0x0, 0x1f4bb, 0xb4f1c0, 0xc431012ef0, 0x0, 0x0)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/value.go:651 +0x6ae
github.com/aerospike/aerospike-client-go.(*batchCommandGet).parseRecord(0xc43391b980, 0xc43b524af0, 0x3, 0xf300e6d3000005c0, 0x14, 0x14, 0x1)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/batch_command_get.go:136 +0x224
github.com/aerospike/aerospike-client-go.(*batchCommandGet).parseRecordResults(0xc43391b980, 0x109f9a0, 0xc43391b980, 0x1f57a, 0x8, 0x8, 0x0)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/batch_command_get.go:103 +0x2ea
github.com/aerospike/aerospike-client-go.(*baseMultiCommand).parseResult(0xc42036fba0, 0x109f9a0, 0xc43391b980, 0xc4319785c0, 0x0, 0x0)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/batch_command.go:97 +0x164
github.com/aerospike/aerospike-client-go.(*baseCommand).execute(0xc43c496090, 0x109f9a0, 0xc43391b980, 0x1, 0xc43391b980)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/command.go:1068 +0x5e4
github.com/aerospike/aerospike-client-go.(*batchCommandGet).Execute(0xc43391b980, 0xc43c496030, 0x109f9a0)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/batch_command_get.go:148 +0x42
github.com/aerospike/aerospike-client-go.(*Client).batchExecute.func1(0xc43281d8c0, 0xc428388e60, 0xc43281d8a8, 0xc431672520, 0xc42031e780, 0xc43c496030)
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/client.go:1479 +0x84
created by github.com/aerospike/aerospike-client-go.(*Client).batchExecute
        /home/user/.go/src/github.com/aerospike/aerospike-client-go/client.go:1484 +0x236

We would like to update to 1.20 and would like a recent stable version to be able to upgrade from.

[khaf]

Is this surfacing in your tests as well, or only in production?

[fmstephe]

Only in production.

[khaf]

Can you retrieve the data via aql?

[fmstephe]

Can you clarify your question? We are using version 1.11 with no problems. So the data is retrievable. But we aren't able to retrieve the exact data which caused the panics, there is no indication what that data was in the panic.

[khaf]

This answers my question, I was trying to establish if the data is malformed or not. Let me go through the changes since 1.11 and see if I can pin down the change that might have caused the panic. Will get back to you in a day or two.

I assume the panic happens only occasionally and not on each request, is that correct?

[fmstephe]

It took about 10 minutes running in production before the servers started crashing.

[khaf]

This should have been taken care of in the latest release.