etcdserver exceeds database space, preventing all further milvus operations

[NotRyan]

After a high number of collection creations, collection drops, successful inserts, and failed inserts, all milvus operations will begin to fail with the error message

"BaseException: <BaseException: (code=1, message=Drop collection failed: etcdserver: mvcc: database space exceeded)>"

despite the instance hosting the etcd node having plenty of free disk space and free memory.

Steps/Code to reproduce:

Inconsistent, has occurred on two separate occasions while performing collection and insertion operations. Have not found a reliable way to reproduce yet.

Expected result:

etcd host does not appear to be out of memory, so this should not happen.

Actual results:

"BaseException: <BaseException: (code=1, message=Drop collection failed: etcdserver: mvcc: database space exceeded)>"

Memory and Disk space on etcd node:

docker logs from etcd node:
etcd_logs.txt

docker system df:

ec2-user@ip-10-0-0-102 /]$ sudo docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         39.45MB   0B (0%)
Containers      1         1         2.532GB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

Environment:

• Milvus version(e.g. v2.0.0-RC2 or 8b23a93): v2.0.0-RC2 (Docker image v2.0.0-rc2-20210712-a8e5fd2)
• Deployment mode(standalone or cluster): Cluster
• SDK version(e.g. pymilvus v2.0.0rc2): PyMilvus v2.0.0rc3dev6
• OS(Ubuntu or CentOS): Amazon Linux 2
• CPU/Memory: 16 GB memory
• GPU:
• Others: 1 TB disk

Configuration file:

Deployed on AWS using Terraform/Ansible, setup procedure described in https://zilliverse.feishu.cn/docs/doccnuxzvQVhPqzityx2sO6kUfc

Additional context:

[filip-halt]

I am running into the same issue, for me it occurred when inserting and indexing. @xiaofan-luan is this something that anyone else has been running into on the team?

[godchen0212]

Can you provide etcd metrics for a etcd debug？
The method of exporting metrics varies according to the etcd version, you can refer to this document
Metrics | etcd

[LoveEachDay]

The default etcd storage size limit is 2GB, you can check the etcd db size with the following command:
etcdctl endpoint status -w table
Check the DB SIZE cell.

You can run etcdctl compact <rev> to decrease the db size. After the db size is less than 2GB, you must run etcdctl alarm disarm before running any etcd writes.

[xiaofan-luan]

@godchen0212 @jeffoverflow we may need to change default etcd settings to maintain less snapshots, we hit the similar issue when we use zookeeper

[xiaofan-luan]

/assign @jeffoverflow

[jeffoverflow]

/assign @LoveEachDay

[LoveEachDay]

We've enabled auto compaction for etcd in docker-compose and helm charts.