High memory usage when uploading with S3ng backend

[AndreasSko]

Describe the bug

I'm migrating my files from Nextcloud to OCIS using the OwnCloud Client for Mac (v2.8.2, build 4246). I noticed high memory usage when doing this, which resulted in many OOM kills because on my small vServer (only 2GB). I see memory usage of more than 1GB immediately after I unpause the synchronization. This seems a bit too high for a single user trying to upload his files.

Steps to reproduce

Steps to reproduce the behavior:

1. Use S3ng backend for home and user storage
2. Use the OwnCloud Client and synchronize folders with many files (ranging between size of few kilobytes, to multiple GBs)

Expected behavior

I would expect OCIS to use less than 1GB of memory for only one user uploading files.

Actual behavior

After I unpause the synchronization, the memory usage immediately jumps to over 1GB

Setup

I run OCIS in a docker-compose setup with Traefik as a proxy. My vServer has 2GB of RAM and 8GB of SWAP enabled (had to increase it that high as otherwise OCIS would be OOM killed immediately). The latest version of OCIS (v1.9.0) is used. The setup details can also be found here.

STORAGE_HOME_DRIVER: s3ng
STORAGE_USERS_DRIVER: s3ng
STORAGE_METADATA_DRIVER: ocis # keep metadata on ocis storage since this are only small files atm
# s3ng specific settings
STORAGE_DRIVER_S3NG_ENDPOINT: https://s3.eu-central-003.backblazeb2.com
...
# Storage Frontend
STORAGE_FRONTEND_DEFAULT_UPLOAD_PROTOCOL: v1
STORAGE_FRONTEND_UPLOAD_MAX_CHUNK_SIZE: 100000000 # <100MB

Additional context

I already tried to set STORAGE_FRONTEND_UPLOAD_MAX_CHUNK_SIZE to 0 and tried different default upload protocols, but unfortunately didn't see much of a difference.

I created a heap profile, maybe this helps a bit: profile001.pdf

[AndreasSko]

I did some more digging and found minio/minio-go#1496. So maybe the problem lies with https://github.com/cs3org/reva/blob/7d251bffd9e699af55f811c1dbb54f4e9a14f377/pkg/storage/fs/s3ng/blobstore/blobstore.go#L63 because it's not defining the size upfront. Guess this looks more like an issue that I should have reported in Reva? Not sure 🤔

[AndreasSko]

I opened a new issue in the Reva repo: cs3org/reva#1933
Closing this one now.