pebble backend panic in Kubo 0.33 during shutdown

[lidel]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

docker image

Version

0.33.0-rc3-4c23919

Config

https://github.com/ipshipyard/waterworks-infra/blob/master/ansible/inventory/host_vars/collab-cluster-am6-1.cluster.dwebops.net.yml

Description

We've been running 0.33.0-rc3-4c23919 at one of our collab cluster nodes pebble panic's during shutdown:

Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: Initializing daemon...
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: Kubo version: 0.33.0-rc3-4c23919
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: Repo version: 16
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: System version: amd64/linux
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: Golang version: go1.23.5
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: PeerID: 12D3KooWRNijznEQoXrxBeNLb2TqbSFm8gG8jKtfEsbC1C9nPqce
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: 2025-01-28T17:31:02.587Z        INFO        autotls        node/groups.go:178        appended AutoWSS listener: /ip4/0.0.0.0/tcp/4001/tls/sni/*.libp2p.direct/ws
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: 2025-01-28T17:31:02.587Z        INFO        autotls        node/groups.go:178        appended AutoWSS listener: /ip6/::/tcp/4001/tls/sni/*.libp2p.direct/ws
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: 2025-01-28T17:31:02.591Z        INFO        autotls.maintenance        [email protected]/maintain.go:63        started background certificate maintenance        {"cache": "0xc0004f9380"}
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: 2025-01-28T17:31:02.682Z        INFO        autotls.start        client/acme.go:396        found preexisting cert for "*.k51qzi5uqu5dly016rcpbuh6h5jc5xovwotgy3rqltt8l5pcxyhznqiy02xeqb.libp2p.direct" in local storage
Jan 28 17:31:02 collab-cluster-am6-1 ipfs[128960]: 2025-01-28T17:31:02.684Z        DEBUG        autotls        [email protected]/cache.go:271        added certificate to cache        {"subjects": ["*.k51qzi5uqu5dly016rcpbuh6h5jc5xovwotgy3rqltt8l5pcxyhznqiy02xeqb.libp2p.direct"], "expiration": "2025-04-28T15:42:39.000Z", "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "30e9e6c9fc95f05f9963fbd8cdde66e2254a4f9a5f6af8820dbf1ffde21c043e", "cache_size": 1, "cache_capacity": 0}
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on 10.12.124.131:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on 127.0.0.1:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on 147.75.87.211:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on 172.17.0.1:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on 209.94.78.1:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on [2602:fea2:3::1]:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on [2604:1380:4601:f600::3]:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Swarm listening on [::1]:4001 (TCP+UDP)
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Run 'ipfs id' to inspect announced and discovered multiaddrs of this node.
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: RPC API server listening on /ip4/127.0.0.1/tcp/5001
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: WebUI: http://127.0.0.1:5001/webui
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Gateway server listening on /ip4/127.0.0.1/tcp/8080
Jan 28 17:31:03 collab-cluster-am6-1 ipfs[128960]: Daemon is ready
[..]
Jan 28 17:31:55 collab-cluster-am6-1 ipfs[128960]: Received interrupt signal, shutting down...
Jan 28 17:31:55 collab-cluster-am6-1 ipfs[128960]: (Hit ctrl-c again to force-shutdown the daemon.)
Jan 28 17:31:55 collab-cluster-am6-1 systemd[1]: Stopping IPFS daemon...
Jan 28 17:32:03 collab-cluster-am6-1 ipfs[128960]: 2025-01-28T17:32:03.690Z        ERROR        cmd/ipfs        kubo/daemon.go:1160        initial version check failed, will not be run again        {"error": "could not perform version check due to missing or incompatible DHT configuration"}
[..]
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: panic: pebble: closed
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: goroutine 28489 [running]:
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/cockroachdb/pebble.(*DB).getInternal(0x1102aeb?, {0xc0593c6300?, 0x0?, 0x4962900?}, 0x40?, 0xc0593c6300?)
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/cockroachdb/[email protected]/db.go:537 +0x445
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/cockroachdb/pebble.(*DB).Get(...)
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/cockroachdb/[email protected]/db.go:520
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/go-ds-pebble.(*Datastore).get(0x10000000010?, {0xc0593c6300?, 0x3f?, 0xc000d776e8?}, 0x1)
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/datastore.go:78 +0x37
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/go-ds-pebble.(*Datastore).Get(0xc0031fe600, {0xc006ace308?, 0xc0593c62c0?}, {{0xc0593c62c0?, 0x0?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/datastore.go:100 +0x3d
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/go-ds-measure.(*measure).Get(0xc000372f08, {0x3696700, 0xc0396873e0}, {{0xc0593c62c0?, 0x11020ea?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/measure.go:199 +0x10f
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/go-ds-measure.(*measure).Get(0xc000373208, {0x3696700, 0xc0396873e0}, {{0xc0593c62c0?, 0xc0593c6240?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/measure.go:199 +0x10f
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/go-datastore/keytransform.(*Datastore).Get(0xc001714b00, {0x3696700, 0xc0396873e0}, {{0xc0593c6280?, 0x1749355?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/keytransform/keytransform.go:57 +0x65
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/blockstore.(*blockstore).Get(0xc001ea4a38, {0x3696700, 0xc0396873e0}, {{0xc008fd7560?, 0xc053b25230?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/blockstore/blockstore.go:187 +0x9f
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/kubo/thirdparty/verifbs.(*VerifBS).Get(0xc003d81550, {0x3696700, 0xc0396873e0}, {{0xc008fd7560?, 0x1?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/kubo/thirdparty/verifbs/verifbs.go:63 +0x84
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/blockstore.(*tqcache).Get(0xc001f81680, {0x3696700, 0xc0396873e0}, {{0xc008fd7560?, 0x12?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/blockstore/twoqueue_cache.go:236 +0x1aa
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/blockstore.(*idstore).Get(0xc001714c40, {0x3696700, 0xc0396873e0}, {{0xc008fd7560?, 0x20?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/blockstore/idstore.go:89 +0x93
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/blockservice.getBlock({0x3696700, 0xc0396873e0}, {{0xc008fd7560?, 0x1d?}}, {0x36a95b8, 0xc002dd8df8}, 0xc000d77d70)
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/blockservice/blockservice.go:258 +0x10b
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/blockservice.(*Session).GetBlock(0xc0058861c0, {0x3696700, 0xc008db57d0}, {{0xc008fd7560?, 0x24?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/blockservice/blockservice.go:464 +0x33e
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/ipld/merkledag.(*sesGetter).Get(0xc002845b40, {0x3696700, 0xc008db57d0}, {{0xc008fd7560?, 0xc00b4040c0?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/ipld/merkledag/merkledag.go:143 +0x31
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/ipld/merkledag.FetchGraphWithDepthLimit.GetLinksDirect.func5({0x3696700?, 0xc008db57d0?}, {{0xc008fd7560?, 0x1?}})
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/ipld/merkledag/merkledag.go:128 +0x3b
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: github.com/ipfs/boxo/ipld/merkledag.parallelWalkDepth.func1()
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/ipld/merkledag/merkledag.go:506 +0x2d2
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]: created by github.com/ipfs/boxo/ipld/merkledag.parallelWalkDepth in goroutine 21777
Jan 28 17:32:30 collab-cluster-am6-1 ipfs[128960]:         github.com/ipfs/[email protected]/ipld/merkledag/merkledag.go:488 +0x1c5
Jan 28 17:32:30 collab-cluster-am6-1 systemd[1]: ipfs.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 28 17:32:30 collab-cluster-am6-1 systemd[1]: ipfs.service: Failed with result 'exit-code'.
Jan 28 17:32:30 collab-cluster-am6-1 systemd[1]: Stopped IPFS daemon.
Jan 28 17:32:30 collab-cluster-am6-1 systemd[1]: ipfs.service: Consumed 9min 38.552s CPU time.

cc @gammazero @hsanjuan for visibility.

[hsanjuan]

Pebble was closed before something gave up reading from it... prob not a very easy fix...