Write amplification with v3.3 transactional writes

[gyuho]

#8510 added txn stresser, and tester now fails with mvcc: database space exceeded error...

Easily reproducible as below

Update test script

diff --git a/test b/test
index 01dfe64aa..e16dfbaa2 100755
--- a/test
+++ b/test
@@ -132,10 +132,10 @@ function functional_pass {
                -client-ports 12379,22379,32379 \
                -peer-ports 12380,22380,32380 \
                -limit 1 \
-               -schedule-cases "0 1 2 3 4 5" \
-               -stress-qps 1000 \
-               -stress-key-txn-count 100 \
-               -stress-key-txn-ops 10 \
+               -schedule-cases "5" \
+               -stress-qps 2500 \
+               -stress-key-txn-count 1 \
+               -stress-key-txn-ops 1 \
                -exit-on-failure && echo "'etcd-tester' succeeded"
        ETCD_TESTER_EXIT_CODE=$?
        echo "ETCD_TESTER_EXIT_CODE:" ${ETCD_TESTER_EXIT_CODE}

Update etcd-agent

diff --git a/tools/functional-tester/etcd-agent/agent.go b/tools/functional-tester/etcd-agent/agent.go
index b7be29096..fb7f44ac5 100644
--- a/tools/functional-tester/etcd-agent/agent.go
+++ b/tools/functional-tester/etcd-agent/agent.go
@@ -169,10 +169,10 @@ func (a *Agent) terminate() error {
        if err != nil {
                return err
        }
-       err = os.RemoveAll(a.dataDir())
-       if err != nil {
-               return err
-       }
+       // err = os.RemoveAll(a.dataDir())
+       // if err != nil {
+       //      return err
+       // }
        a.state = stateTerminated
        return nil
 }

Run

PASSES="functional" ./test

Check db file size

$ ls -lah ./agent-1/etcd.data/member/snap/
total 181M
drwx------ 2 gyuho 4.0K Jan 11 02:59 .
drwx------ 4 gyuho 4.0K Jan 11 02:59 ..
-rw-rw-r-- 1 gyuho 8.8K Jan 11 02:59 0000000000000003-0000000000002714.snap
-rw-rw-r-- 1 gyuho 8.9K Jan 11 02:59 0000000000000003-0000000000004e42.snap
-rw------- 1 gyuho 181M Jan 11 02:59 db

Now update test script to disable transaction stresser

diff --git a/test b/test
index 01dfe64aa..e16dfbaa2 100755
--- a/test
+++ b/test
@@ -132,10 +132,10 @@ function functional_pass {
                -client-ports 12379,22379,32379 \
                -peer-ports 12380,22380,32380 \
                -limit 1 \
-               -schedule-cases "0 1 2 3 4 5" \
-               -stress-qps 1000 \
-               -stress-key-txn-count 100 \
-               -stress-key-txn-ops 10 \
+               -schedule-cases "5" \
+               -stress-qps 2500 \
+               -stress-key-txn-count 0 \
+               -stress-key-txn-ops 0 \
                -exit-on-failure && echo "'etcd-tester' succeeded"
        ETCD_TESTER_EXIT_CODE=$?
        echo "ETCD_TESTER_EXIT_CODE:" ${ETCD_TESTER_EXIT_CODE}

Check db size

$ ls -lah ./agent-1/etcd.data/member/snap/
total 6.8M
drwx------ 2 gyuho 4.0K Jan 11 03:02 .
drwx------ 4 gyuho 4.0K Jan 11 03:02 ..
-rw-rw-r-- 1 gyuho 8.9K Jan 11 03:02 0000000000000003-00000000000044c2.snap
-rw------- 1 gyuho 6.7M Jan 11 03:02 db

Compare db files with/without transactional writes:

$ ETCDCTL_API=3 etcdctl snapshot status db-no-txn --write-out table
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 5722414d |    16163 |      27639 |     7.0 MB |
+----------+----------+------------+------------+

$ ETCDCTL_API=3 etcdctl snapshot status db-txn --write-out table
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 846f2ed9 |    18490 |      26844 |     189 MB |
+----------+----------+------------+------------+

[xiang90]

does 3.2 have the same issue?

[gyuho]

@xiang90 Yes.

v3.2.14 with freelist sync is 215 MiB.

$ ETCDCTL_API=3 etcdctl snapshot status agent-2/etcd.data/member/snap/db --write-out table

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| e49c6d7b |    20368 |      28587 |     208 MB |
+----------+----------+------------+------------+

Trying to make this easily reproducible to verify further.

[gyuho]

I was setting up stresser weights wrong...

Confirmed that db size only with txn writes grows as expected.
Most of the data was coming from large puts with 33 KiB values.

This is snapshot from tester only with txn stresser (6-byte key, 5-byte value).

ETCDCTL_API=3 etcdctl snapshot status ./agent-1/etcd.data/member/snap/db --write-out table
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| cf5a299a |    11037 |      10009 |     639 kB |
+----------+----------+------------+------------+