Duplicated watch event detected in robustness test #17247
Comments
Change-Id: Ieed5f16ded627ac27f6acae99fea23211e6f718d
Change-Id: Ieed5f16ded627ac27f6acae99fea23211e6f718d
serathius
added
priority/important-soon
|
Got a repro on both https://github.com/etcd-io/etcd/actions/runs/7539634396?pr=17248 |
Change-Id: Ieed5f16ded627ac27f6acae99fea23211e6f718d Signed-off-by: Marek Siarkowicz <[email protected]>
|
Got a one local repro. |
serathius
changed the title
|
Watch request-response with a duplicated revision 882, interesting part: Full response
```
{
"Request": {
"Key": "/registry/pods/",
"Revision": 854,
"WithPrefix": true,
"WithProgressNotify": true
},
"Responses": [
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/zlqKy",
"Value": {
"Value": "980",
"Hash": 0
},
"Revision": 854
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/CPWLs",
"Value": {
"Value": "982",
"Hash": 0
},
"Revision": 855
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/n5T9S",
"Value": {
"Value": "983",
"Hash": 0
},
"Revision": 856
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/llCqr",
"Value": {
"Value": "984",
"Hash": 0
},
"Revision": 857
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/Gvvm7",
"Value": {
"Value": "985",
"Hash": 0
},
"Revision": 858
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/qalpx",
"Value": {
"Value": "986",
"Hash": 0
},
"Revision": 859
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/EQPt4",
"Value": {
"Value": "987",
"Hash": 0
},
"Revision": 860
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/158WY",
"Value": {
"Value": "988",
"Hash": 0
},
"Revision": 861
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/WAGPJ",
"Value": {
"Value": "991",
"Hash": 0
},
"Revision": 862
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/gqmYo",
"Value": {
"Value": "992",
"Hash": 0
},
"Revision": 863
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/lwMud",
"Value": {
"Value": "993",
"Hash": 0
},
"Revision": 864
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/bbfGl",
"Value": {
"Value": "994",
"Hash": 0
},
"Revision": 865
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/e9AyQ",
"Value": {
"Value": "996",
"Hash": 0
},
"Revision": 866
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/mpvvr",
"Value": {
"Value": "997",
"Hash": 0
},
"Revision": 867
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/F3TRD",
"Value": {
"Value": "998",
"Hash": 0
},
"Revision": 868
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/OVXTy",
"Value": {
"Value": "1000",
"Hash": 0
},
"Revision": 869
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/uZBI1",
"Value": {
"Value": "1002",
"Hash": 0
},
"Revision": 870
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/qbvoO",
"Value": {
"Value": "1003",
"Hash": 0
},
"Revision": 871
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/4b7V4",
"Value": {
"Value": "1005",
"Hash": 0
},
"Revision": 872
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/WPXbR",
"Value": {
"Value": "1006",
"Hash": 0
},
"Revision": 873
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/F3Ro3",
"Value": {
"Value": "1007",
"Hash": 0
},
"Revision": 874
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/s2DPx",
"Value": {
"Value": "1008",
"Hash": 0
},
"Revision": 875
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/Gvvm7",
"Value": {
"Value": "1009",
"Hash": 0
},
"Revision": 876
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/XcZU8",
"Value": {
"Value": "1010",
"Hash": 0
},
"Revision": 877
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/553WP",
"Value": {
"Value": "1011",
"Hash": 0
},
"Revision": 878
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/Ft0M7",
"Value": {
"Value": "1012",
"Hash": 0
},
"Revision": 879
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/EQPt4",
"Value": {
"Value": "1013",
"Hash": 0
},
"Revision": 880
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/PSvk1",
"Value": {
"Value": "1014",
"Hash": 0
},
"Revision": 881
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/gqmYo",
"Value": {
"Value": "1015",
"Hash": 0
},
"Revision": 882
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/gqmYo",
"Value": {
"Value": "1015",
"Hash": 0
},
"Revision": 882
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/llCqr",
"Value": {
"Value": "1017",
"Hash": 0
},
"Revision": 883
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/vSBGf",
"Value": {
"Value": "1019",
"Hash": 0
},
"Revision": 884
},
{
"Type": "put-operation",
"Key": "/registry/pods/default/n5T9S",
"Value": {
"Value": "1020",
"Hash": 0
},
"Revision": 885
}
],
"IsProgressNotify": false,
"Revision": 885,
"Time": 6124853237
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/OVXTy",
"Value": {
"Value": "1021",
"Hash": 0
},
"Revision": 886
}
],
"IsProgressNotify": false,
"Revision": 886,
"Time": 6125394945
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/XcZU8",
"Value": {
"Value": "1023",
"Hash": 0
},
"Revision": 887
}
],
"IsProgressNotify": false,
"Revision": 887,
"Time": 6134684586
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/mxBK7",
"Value": {
"Value": "1024",
"Hash": 0
},
"Revision": 888
}
],
"IsProgressNotify": false,
"Revision": 888,
"Time": 6137375281
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/bbfGl",
"Value": {
"Value": "1025",
"Hash": 0
},
"Revision": 889
}
],
"IsProgressNotify": false,
"Revision": 889,
"Time": 6145422960
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/lRR85",
"Value": {
"Value": "1026",
"Hash": 0
},
"Revision": 890
}
],
"IsProgressNotify": false,
"Revision": 890,
"Time": 6147789266
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/v0Wg5",
"Value": {
"Value": "1027",
"Hash": 0
},
"Revision": 891
}
],
"IsProgressNotify": false,
"Revision": 891,
"Time": 6157548521
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/PI7s9",
"Value": {
"Value": "1028",
"Hash": 0
},
"Revision": 892
}
],
"IsProgressNotify": false,
"Revision": 892,
"Time": 6162857744
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/v0Wg5",
"Value": {
"Value": "1029",
"Hash": 0
},
"Revision": 893
}
],
"IsProgressNotify": false,
"Revision": 893,
"Time": 6173448149
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/ZsJkQ",
"Value": {
"Value": "1030",
"Hash": 0
},
"Revision": 894
}
],
"IsProgressNotify": false,
"Revision": 894,
"Time": 6177937264
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/eikdd",
"Value": {
"Value": "1031",
"Hash": 0
},
"Revision": 895
}
],
"IsProgressNotify": false,
"Revision": 895,
"Time": 6182406700
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/fNa1r",
"Value": {
"Value": "1032",
"Hash": 0
},
"Revision": 896
}
],
"IsProgressNotify": false,
"Revision": 896,
"Time": 6188104553
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/EQPt4",
"Value": {
"Value": "1033",
"Hash": 0
},
"Revision": 897
}
],
"IsProgressNotify": false,
"Revision": 897,
"Time": 6198847616
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/WAGPJ",
"Value": {
"Value": "1035",
"Hash": 0
},
"Revision": 898
}
],
"IsProgressNotify": false,
"Revision": 898,
"Time": 6212652771
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/HdoK9",
"Value": {
"Value": "1036",
"Hash": 0
},
"Revision": 899
}
],
"IsProgressNotify": false,
"Revision": 899,
"Time": 6219512628
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/bbfGl",
"Value": {
"Value": "1037",
"Hash": 0
},
"Revision": 900
}
],
"IsProgressNotify": false,
"Revision": 900,
"Time": 6225460171
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/s2DPx",
"Value": {
"Value": "1038",
"Hash": 0
},
"Revision": 901
}
],
"IsProgressNotify": false,
"Revision": 901,
"Time": 6231087364
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/4t7Cm",
"Value": {
"Value": "1039",
"Hash": 0
},
"Revision": 902
}
],
"IsProgressNotify": false,
"Revision": 902,
"Time": 6237653749
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/rwstr",
"Value": {
"Value": "1040",
"Hash": 0
},
"Revision": 903
}
],
"IsProgressNotify": false,
"Revision": 903,
"Time": 6242222081
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/vs80D",
"Value": {
"Value": "1041",
"Hash": 0
},
"Revision": 904
}
],
"IsProgressNotify": false,
"Revision": 904,
"Time": 6252938003
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/63GF1",
"Value": {
"Value": "1042",
"Hash": 0
},
"Revision": 905
}
],
"IsProgressNotify": false,
"Revision": 905,
"Time": 6259062598
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/zlqKy",
"Value": {
"Value": "1043",
"Hash": 0
},
"Revision": 906
}
],
"IsProgressNotify": false,
"Revision": 906,
"Time": 6279987619
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/PSvk1",
"Value": {
"Value": "1044",
"Hash": 0
},
"Revision": 907
}
],
"IsProgressNotify": false,
"Revision": 907,
"Time": 6280049124
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/7hGI0",
"Value": {
"Value": "1045",
"Hash": 0
},
"Revision": 908
}
],
"IsProgressNotify": false,
"Revision": 908,
"Time": 6284565610
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/9nMUa",
"Value": {
"Value": "1046",
"Hash": 0
},
"Revision": 909
}
],
"IsProgressNotify": false,
"Revision": 909,
"Time": 6287166957
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/ZVcyt",
"Value": {
"Value": "1047",
"Hash": 0
},
"Revision": 910
}
],
"IsProgressNotify": false,
"Revision": 910,
"Time": 6292679653
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/5uhDY",
"Value": {
"Value": "1048",
"Hash": 0
},
"Revision": 911
}
],
"IsProgressNotify": false,
"Revision": 911,
"Time": 6297197079
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/qalpx",
"Value": {
"Value": "1049",
"Hash": 0
},
"Revision": 912
}
],
"IsProgressNotify": false,
"Revision": 912,
"Time": 6307661469
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/PI7s9",
"Value": {
"Value": "1050",
"Hash": 0
},
"Revision": 913
}
],
"IsProgressNotify": false,
"Revision": 913,
"Time": 6313212397
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/jOxQ8",
"Value": {
"Value": "1051",
"Hash": 0
},
"Revision": 914
}
],
"IsProgressNotify": false,
"Revision": 914,
"Time": 6324605471
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/edTMh",
"Value": {
"Value": "1052",
"Hash": 0
},
"Revision": 915
}
],
"IsProgressNotify": false,
"Revision": 915,
"Time": 6328803116
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/rwZnc",
"Value": {
"Value": "1053",
"Hash": 0
},
"Revision": 916
}
],
"IsProgressNotify": false,
"Revision": 916,
"Time": 6333627710
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/UvmJr",
"Value": {
"Value": "1054",
"Hash": 0
},
"Revision": 917
}
],
"IsProgressNotify": false,
"Revision": 917,
"Time": 6340956298
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/qalpx",
"Value": {
"Value": "1055",
"Hash": 0
},
"Revision": 918
}
],
"IsProgressNotify": false,
"Revision": 918,
"Time": 6349607460
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/6Z4sD",
"Value": {
"Value": "1056",
"Hash": 0
},
"Revision": 919
}
],
"IsProgressNotify": false,
"Revision": 919,
"Time": 6352693006
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/qbvoO",
"Value": {
"Value": "1057",
"Hash": 0
},
"Revision": 920
}
],
"IsProgressNotify": false,
"Revision": 920,
"Time": 6363659529
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/F3TRD",
"Value": {
"Value": "1058",
"Hash": 0
},
"Revision": 921
}
],
"IsProgressNotify": false,
"Revision": 921,
"Time": 6371889239
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/pMgix",
"Value": {
"Value": "1059",
"Hash": 0
},
"Revision": 922
}
],
"IsProgressNotify": false,
"Revision": 922,
"Time": 6375499923
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/rwZnc",
"Value": {
"Value": "1060",
"Hash": 0
},
"Revision": 923
}
],
"IsProgressNotify": false,
"Revision": 923,
"Time": 6386067716
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/v0Wg5",
"Value": {
"Value": "1061",
"Hash": 0
},
"Revision": 924
}
],
"IsProgressNotify": false,
"Revision": 924,
"Time": 6388870782
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/0AUzn",
"Value": {
"Value": "1062",
"Hash": 0
},
"Revision": 925
}
],
"IsProgressNotify": false,
"Revision": 925,
"Time": 6392359146
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/lwMud",
"Value": {
"Value": "1063",
"Hash": 0
},
"Revision": 926
}
],
"IsProgressNotify": false,
"Revision": 926,
"Time": 6407253095
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/6Z4sD",
"Value": {
"Value": "1064",
"Hash": 0
},
"Revision": 927
}
],
"IsProgressNotify": false,
"Revision": 927,
"Time": 6417900820
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/Ft0M7",
"Value": {
"Value": "1066",
"Hash": 0
},
"Revision": 928
}
],
"IsProgressNotify": false,
"Revision": 928,
"Time": 6438074579
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/OVXTy",
"Value": {
"Value": "1068",
"Hash": 0
},
"Revision": 929
}
],
"IsProgressNotify": false,
"Revision": 929,
"Time": 6452060313
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/SOhlK",
"Value": {
"Value": "1070",
"Hash": 0
},
"Revision": 930
}
],
"IsProgressNotify": false,
"Revision": 930,
"Time": 6469929313
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/pUZix",
"Value": {
"Value": "1071",
"Hash": 0
},
"Revision": 931
}
],
"IsProgressNotify": false,
"Revision": 931,
"Time": 6477969346
},
{
"Events": [
{
"Type": "put-operation",
"Key": "/registry/pods/default/HdoK9",
"Value": {
"Value": "1072",
"Hash": 0
},
"Revision": 932
}
],
"IsProgressNotify": false,
"Revision": 932,
"Time": 6482229480
}
]
}
```
|
|
Trying to bisect in the PR responsible. Last passing nightly https://github.com/etcd-io/etcd/actions/runs/7486757727 with commit b3bf59a as a bisect start. However it's not guaranteed corrct commit, due to flakes in #17246 we cannot be 100% sure about error rate of this issue. |
|
We should also investigate why this issue only reproduces on arm64 in CI, even though locally it also appears on amd. Issues independent of platform should be detected on both. Expect this is because of two reasons:
I think we should consider increasing number of runs of exploratory tests ( |
|
Bisection pointed to ed99424 as first broken commit, due to not 100% reproducibility we need to more testing to confirm. Will continue to work on confirming the results, but in the mean time we need to block backports. |
|
Reproduced the issue enough time to make me sure that this is #17228 @siyuanfoundation Do you have time look into this soon or would you prefer to revert and fix later? |
|
I will take a look today |
might be the cause of etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
might be the cause of etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
might be the cause of etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
|
#17263 (comment) is a valid point. The key point is etcd/server/storage/backend/tx_buffer.go Line 220 in e3c70c8 Actually there shouldn't be any duplicated items,
Proposed actions:
|
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
to fix etcd-io#17247. Signed-off-by: Siyuan Zhang <[email protected]>
I disagree with the proposal.
|
Where is the assumption? I didn't make any assumption. Always remember to simplify the repo as much as possible! In this case, the
Anyway, let's revert #17228 firstly to not to fail the workflow |
"there shouldn't be any duplicated items" is an assumption. Or if you prefer, a constraint. This might be true on the etcd level because there is yet no use cases of putting duplicate keys between lock and unlock. But at the tx/buffer level, if Yes, adding verification could safeguard it. But it is just strange to have such a constraint. In addition, it does not make the code simpler, because
#17263 will fix the workflow. |
Actually pull/17228 was fixing a non-exist issue, because there will never have duplicated items. Please refer to etcd-io#17247 (comment). The PR also introduces a bug (issues/17247). Signed-off-by: Benjamin Wang <[email protected]>
It isn't an assumption, nor constraint. It's an invariant property, at least for the key space/bucket. Note etcd applys MVCC on the key space, as I mentioned above multiple times, the keys are monotonically growing revisions. There is no overwrite on any existing keys in the Key bucket. It's the key point to support concurrent & consistent read. We should add a verification to ensure this invariant property. Refer to the verification added in the second commit in #17281. And it successfully identified some test issues so far,
|
It's the most important part of etcd. Please let's do not blindly make any decision (e.g. just because it can fix the workflow failure) before it's well understood and discussed. Reverting the PR #17228 firstly, afterwards delivering a clean PR to resolve it is the most prudential approach. |
|
IMO there are two potential problems:
I think #17263 can be consided to fix first one. We don't need to introduce map for duplicate key check. For second one, if there are overlap between buffer cache and read transacton, we should panic. Just my two cents. |
For the second one, etcd/server/storage/backend/read_tx.go Line 86 in dd0b052 |
|
Solution 1: #17281 (comment) We need to run benchmark to compare the performance among: Please anyone feel free to continue to work on this issue. |
|
Is there a standard benchmark process? |
Long story short we don't currently have a standard or easily democratised way for benchmarking. We have two tools that are available however, For a quick comparison I would suggest running benchmark with something like: # Server
GOMAXPROCS=10 ./bin/etcd --quota-backend-bytes=17179869184 --listen-client-urls http://0.0.0.0:23790 --advertise-client-urls http://127.0.0.1:23790
# Benchmark
GOMAXPROCS=6 go run ./tools/benchmark put --conns=64 --clients=64 --key-size=256 --val-size=4096 --key-space-size=50000 --total=2000000 --endpoints "http://127.0.0.1:23790"
# make clean && make build between each runFor results to have any validity please run several times and ensure all runs for each branch being tested happen on the same hardware and as much as possible system load state. |
Hi @siyuanfoundation sorry for late reply. This issue is caused by committed key/values in buffer cache. |
|
I have run benchmark 6 times as @jmhbnz suggested # Server
GOMAXPROCS=10 ./bin/etcd --quota-backend-bytes=17179869184 --listen-client-urls http://0.0.0.0:23790 --advertise-client-urls http://127.0.0.1:23790
# Benchmark
GOMAXPROCS=6 go run ./tools/benchmark put --conns=64 --clients=64 --key-size=256 --val-size=4096 --key-space-size=50000 --total=2000000 --endpoints "http://127.0.0.1:23790"
# make clean && make build between each runThis is the result of the total time of each benchmark run: It is pretty noisy. But there does not seem to be much difference between #17281 and #17290. |
|
Just to double check. Have we expected lack of performance difference between #17281 and #17290 or is it a surprise? If that was unexpected can you double check the benchmark method? For example you used |
In theory, #17281 should have the best performance, and #17290 have the worst performance, and So previous test result provided by @siyuanfoundation doesn't comply with our understanding. Probably there indeed isn't much performance difference. It's still better if we can double confirm,
|
is only called when the bucket does not exist yet in the read buffer, which is pretty rare. |
Why it's rare? Please read #17247 (comment) and try the two proposed ways to double confirm the performance impact. |
|
Can we close the issue? |
|
Meant to ask if there is anything else that needs to be addressed before I close the issue, but that works too :P. |
Bug report criteria
What happened?
2 robustness test failures:
Common elements:
Differences:
raftAfterSaveSnap=panic()andMemberReplaceBoth failures on have duplicated watch event:
What did you expect to happen?
No robustness test failure
How can we reproduce it (as minimally and precisely as possible)?
No repro yet, I don't have access to arm64 machine to debug
cc @ahrtr @jmhbnz
Anything else we need to know?
No response
Etcd version (please run commands below)
Etcd configuration (command line flags or environment variables)
paste your configuration here
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
No response
The text was updated successfully, but these errors were encountered: