v0.11.0-rc1 underreports RepoSize significantly

[ylempereur]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest go-ipfs version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.io

Version

go-ipfs version: 0.11.0-rc1
Repo version: 11
System version: amd64/darwin
Golang version: go1.16.8

Config

I only included the relevant part:

  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },

Description

My datastore contained over 2GB before I switched to the latest binary (and was reporting it correctly in v0.10.0), it now reports around 230MB (in repo stat and webui).

I have run some tests and the pinned data is still there and reachable. Adding and removing data (using add, pin rm and repo gc) do not appear to alter the reported repo size (while the data becomes reachable after add and unreachable after gc).

My install is mostly default, with the exception that I'm using the AcceleratedDHTClient.

➜  ~ du -hd0 .ipfs/blocks
2.2G	.ipfs/blocks
➜  ~ ipfs repo stat
NumObjects: 30360
RepoSize:   242893065
StorageMax: 10000000000
RepoPath:   /Users/yves/.ipfs
Version:    fs-repo@11
➜  ~ ipfs stats provide
TotalProvides:          60k (60,715)
AvgProvideDuration:     13.121ms
LastReprovideDuration:  8m41.179902s
LastReprovideBatchSize: 30k (30,360)

Please advise.

[aschmahmann]

Thanks for the report. It looks like a function got missed as part of #6803 (i.e. https://github.com/ipfs/go-ds-flatfs/blob/27a1f1eee3538be7ff43b7f1a5c73122427738fb/flatfs.go#L1094).

@guseggert could you take a look through the other datastores to:

1. Add interface checks for the interfaces a given datastore struct should/does implement
2. Fix the errors

This is going to look like ipfs/go-datastore#184.

I'd definitely check out FlatFS, Badger (1+2) and S3 datastores as those are the mostly commonly used ones.

[guseggert]

@ylempereur Could you upgrade to RC2 and verify that it fixes this issue for you? https://github.com/ipfs/go-ipfs/releases/tag/v0.11.0-rc2

[ylempereur]

@guseggert That did it! the correct size is now being reported when using rc2.

Thank you