IPFS Services Memory Increases with Data Addition

[smvdn]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

DockerImage: ipfs/go-ipfs:v0.19.0

Config

 
{
  "Identity": {
    "PeerID": "",
    "PrivKey": ""
  },
  "Datastore": {
    "StorageMax": "1000GB",
    "StorageGCWatermark": 90,
    "GCPeriod": "1h",
    "Spec": {
      "mounts": [
        {
          "child": {``
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "HashOnRead": false,
    "BloomFilterSize": 0
  },
  "Addresses": {
    "Swarm": [   ],
    "Announce": [""],
    "NoAnnounce": [],
    "API": "",
    "Gateway": ""
  },
  "Mounts": {
    "IPFS": "",
    "IPNS": "",
    "FuseAllowOther": false
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Routing": {
    "Type": "dht"
  },
  "Ipns": {
    "RepublishPeriod": "",
    "RecordLifetime": "",
    "ResolveCacheSize": 128
  },
  "Bootstrap": [],
  "Gateway": {
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "RootRedirect": "",
    "Writable": false,
    "PathPrefixes": [],
    "APICommands": [],
    "NoFetch": false,
    "NoDNSLink": false,
    "PublicGateways": null
  },
  "API": {
    "HTTPHeaders": {}
  },
  "Swarm": {
    "AddrFilters": null,
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "EnableRelayHop": false,
    "EnableAutoRelay": false,
    "Transports": {
      "Network": {},
      "Security": {},
      "Multiplexers": {}
    },
    "ConnMgr": {
      "Type": "basic",
      "LowWater": 600,
      "HighWater": 900,
      "GracePeriod": "20s"
    }
  },
  "AutoNAT": {},
  "Pubsub": {
    "Router": "",
    "DisableSigning": false
  },
  "Peering": {
    "Peers": null
  },
  "DNS": {
    "Resolvers": {}
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Provider": {
    "Strategy": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "UrlstoreEnabled": false,
    "ShardingEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "AcceleratedDHTClient": false
  },
  "Plugins": {
    "Plugins": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Internal": {}
}

Description

We are having IPFS run as pods in the Kubernetes. While starting the IPFS pod the memory consumption of IPFS container is under 1.5 GB but while doing the data addition (the data are being pinned as well) operation for 3 to 4 hours, the container memory is drastically increasing to nearly 4 GB. Would like to understand why the memory consumption is showing this drastic increase?

Is there an option with IPFS to limit the max memory consumption by the IPFS containers, otherwise this could crash Pod thus impacting the incoming new data?

[Jorropo]

You can do ipfs diag profile and post it here please.
You can also look at it yourself using go tool pprof -help.

[smvdn]

ipfs-profile.zip
@Jorropo Thanks for the quick response Attaching the performance profile.

IPFS binary in the dump is removed due to size constraints but we have not made any customization to docker Image ipfs/go-ipfs:v0.19.0

[Jorropo]

@smvdn thx, I've looked at your profile and 40% of it is spent generating the archive, the rest is spent doing network IO.
It totals 6s running over 30s recorded.

That is a 20% which is not great, but not that bad given this assume one core (and I hope you have more than one).
Ideally I would have time rewriting our yamux implementation to be goroutine free, but we have way worst performance cases than this elsewhere (such as when you get hit by lots of bitswap requests).

If this was just a lucky moment, you can run a way longer profile by passing --profile-time 300s.

About memory I see very little things allocated:

[smvdn]

ipfs-node-1-profile-300s.zip

@Jorropo we are seeing issues variation in utilization with memory during loads, CPU variation seems to be normal. Attaching the profile logs with 300s as well.

Is there any reference article you would suggest for understanding more about different threads happening on background (like Archive, network IO, etc.)

[Jorropo]

@smvdn in your last profile I see 23TB of memory allocated (but then freed) the heap is only 70MB big tho, so idk about the 4GB.
The 23TB of allocations is extra GC pressure that should be dealt with, I'll chat with the go-libp2p maintainers and open the required issues in go-libp2p.
If you want to cap out usage you can checkout the GOMEMLIMIT option too (will trade off CPU for more controled memory usage).

[smvdn]

@smvdn in your last profile I see 23TB of memory allocated (but then freed) the heap is only 70MB big tho, so idk about the 4GB. The 23TB of allocations is extra GC pressure that should be dealt with, I'll chat with the go-libp2p maintainers and open the required issues in go-libp2p. If you want to cap out usage you can checkout the GOMEMLIMIT option too (will trade off CPU for more controled memory usage).

As mentioned earlier we are running the service as Pods in Kubernetes. The Kubernetes nodes are having 16GB memory and IPFS containers are having a max of 8GB. Not clear how the tool is showing 23TB (which is way too high) memory available on the VM.

I believe 70 MB of heap is on low side too. Should we look on optimizing heap as well?

[Jorropo]

@smvdn the 23TB is a flow over how long you recorded the profile not an instant mesure. That means memory is allocated then gargage collected then it repeats, for a total of 23TB.

[smvdn]

@smvdn in your last profile I see 23TB of memory allocated (but then freed) the heap is only 70MB big tho, so idk about the 4GB. The 23TB of allocations is extra GC pressure that should be dealt with, I'll chat with the go-libp2p maintainers and open the required issues in go-libp2p. If you want to cap out usage you can checkout the GOMEMLIMIT option too (will trade off CPU for more controled memory usage).

@Jorropo We are not able to limit the Memory usage GOMEMLIMIT. We are setting the env GOMEMLIMIT=1024MiB in the IPFS containers, however the IPFS service memory is going beyond 2.5 GB post the env update.

Attaching the snapshot of GOMEMLIMIT env config for reference.

Need some help to see if I am missing some config here.

[Jorropo]

@smvdn GOMEMLIMIT is not magic, Kubo uses ~3GiB of memory at minimum (2.5 In your example), if you put GOMEMLIMIT bellow that it will run GC every second trying to get it down, but if Kubo still use the memory it can't be magic.

BTW v0.19.0 is buggy try v0.21.0 😉

[smvdn]

@Jorropo have tested with IPFS v0.21.0 but GOMEMLIMIT was not effective in limiting memory in our case. The memory still seems to increase with data operations beyond the limit set.

One more thing noticed is the memory seems to be persistent once it reached around 3 GB even when there are no data operations. If GC is running ideally this should reduce the memory post data operations correct ?

[Jorropo]

@smvdn you can find the docs for GOMEMLIMIT here:
https://pkg.go.dev/runtime

The GOMEMLIMIT variable sets a soft memory limit for the runtime. This memory limit includes the Go heap and all other memory managed by the runtime, and excludes external memory sources such as mappings of the binary itself, memory managed in other languages, and memory held by the operating system on behalf of the Go program. GOMEMLIMIT is a numeric value in bytes with an optional unit suffix. The supported suffixes include B, KiB, MiB, GiB, and TiB. These suffixes represent quantities of bytes as defined by the IEC 80000-13 standard. That is, they are based on powers of two: KiB means 2^10 bytes, MiB means 2^20 bytes, and so on. The default setting is math.MaxInt64, which effectively disables the memory limit. runtime/debug.SetMemoryLimit allows changing this limit at run time.

This is a soft not a hard limit, this is a feature of the go runtime, we don't have a whole lot of control on how it works.
Does #10066 helps ? You could try setting it 8GiB if it's what you have dedicated to your ipfs nodes.

[lidel]

We are setting the env GOMEMLIMIT=1024MiB in the IPFS containers, however the IPFS service memory is going beyond 2.5 GB post the env update.

Triage notes:

• There is not much here we can do for your specific case, as you are running with limits below recommended settings.
• Closing as we've updated minimal requirements in docs(readme): minimal reqs #10066
• Remaining problem with memory usage in ProvideMany is tracked in ProvideMany: high memory usage when providing tens of millions of CIDs boxo#354