Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move KEP 2340 to Beta #4134

Merged
merged 1 commit into from
Feb 7, 2024
Merged

Conversation

serathius
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 23, 2023
@k8s-ci-robot k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 23, 2023
@serathius
Copy link
Contributor Author

ping @wojtek-t

user can reconfigure kube-apiserver to always pass request to etcd by
disabling the feature flag. Metric `apiserver_watch_cache_read_wait`
will measure wait time experienced by reads for watch cache to become fresh.
Per request override will be done via a HTTP request header `k8s.io/apiserver/skip-watch-cache`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deads2k - I would like to get a feedback about it from you

@serathius
Copy link
Contributor Author

serathius commented Aug 28, 2023

Results from testing manual progress notification on broken etcd versions v3.4.24:

Full report
count type revision
84 progress_notify 1
60 progress_notify 2
4 progress_notify 3
5 progress_notify 4
4 progress_notify 5
1 progress_notify 6
32 progress_notify 7
52 progress_notify 9
3 progress_notify 10
6 progress_notify 11
5 progress_notify 13
5 progress_notify 14
3 progress_notify 15
1 progress_notify 16
7 progress_notify 17
1 progress_notify 18
37 progress_notify 19
71 progress_notify 20
3 progress_notify 21
1 progress_notify 22
11 progress_notify 23
264 progress_notify 24
5 progress_notify 25
8 progress_notify 26
13 progress_notify 27
6 progress_notify 28
34 progress_notify 29
56 progress_notify 30
6 progress_notify 31
3 progress_notify 32
4 progress_notify 33
13 progress_notify 34
10 progress_notify 35
29 progress_notify 36
65 progress_notify 37
1 progress_notify 38
14 progress_notify 39
4 progress_notify 40
3 progress_notify 41
6 progress_notify 42
3 progress_notify 43
5 progress_notify 44
4 progress_notify 45
3 progress_notify 46
28 progress_notify 47
228 progress_notify 48
6 progress_notify 49
3 progress_notify 50
8 progress_notify 51
4 progress_notify 52
76 progress_notify 54
9 progress_notify 55
54 progress_notify 56
9 progress_notify 57
1 progress_notify 58
4 progress_notify 59
1 progress_notify 60
3 progress_notify 61
55 progress_notify 62
12 progress_notify 63
244 progress_notify 64
10 progress_notify 65
48 progress_notify 66
57 progress_notify 67
9 progress_notify 68
6 progress_notify 69
58 progress_notify 70
217 progress_notify 71
1 progress_notify 72
38 progress_notify 73
15 progress_notify 74
49 progress_notify 75
3 progress_notify 76
15 progress_notify 77
58 progress_notify 80
66 progress_notify 81
20 progress_notify 82
191 progress_notify 84
61 progress_notify 85
52 progress_notify 86
28 progress_notify 87
1 progress_notify 88
1 progress_notify 89
190 progress_notify 90
19 progress_notify 91
4 progress_notify 92
1 progress_notify 93
24 progress_notify 94
216 progress_notify 95
1 event 2
1 event 3
1 event 4
1 event 5
1 event 6
1 event 7
1 event 8
1 event 9
1 event 10
1 event 11
1 event 12
1 event 13
1 event 14
1 event 15
1 event 16
1 event 17
1 event 18
1 event 19
1 event 20
1 event 21
1 event 22
1 event 23
1 event 24
1 event 25
1 event 26
1 event 27
1 event 28
1 event 29
1 event 30
1 event 31
1 event 32
1 event 33
1 event 34
1 event 35
1 event 36
1 event 37
1 event 38
1 event 39
1 event 40
1 event 41
1 event 42
1 event 43
1 event 44
1 event 45
1 event 46
1 event 47
1 event 48
1 event 49
1 event 50
1 event 51
1 event 52
1 event 53
1 event 54
1 event 55
1 event 56
1 event 57
1 event 58
1 event 59
1 event 60
1 event 61
1 event 62
1 event 63
1 event 64
1 event 65
1 event 66
1 event 67
1 event 68
1 event 69
1 event 70
1 event 71
1 event 72
1 event 73
1 event 74
1 event 75
1 event 76
1 event 77
1 event 78
1 event 79
1 event 80
1 event 81
1 event 82
1 event 83
1 event 84
1 event 85
1 event 86
1 event 87
1 event 88
1 event 89
1 event 90
1 event 91
1 event 92
1 event 93
1 event 94
1 event 95
16 progress_notify 95
1 event 96
30 progress_notify 96
1 event 97
5 progress_notify 97
1 event 98
2 progress_notify 98
1 event 99
5 progress_notify 99
1 event 100
1 event 101
1 event 102
10 progress_notify 102
1 event 103
5 progress_notify 103
1 event 104
1 event 105
8 progress_notify 105
1 event 106
1 progress_notify 105
11 progress_notify 106
1 event 107
1 progress_notify 106
2 progress_notify 107
1 event 108
1 progress_notify 107
289 progress_notify 108
1 event 109
4 progress_notify 109
1 event 110
1 event 111
14 progress_notify 111
1 event 112
5 progress_notify 112
1 event 113
4 progress_notify 113
1 event 114
1 progress_notify 114
1 event 115
2 progress_notify 114
11 progress_notify 115
1 event 116
15 progress_notify 116
1 event 117
1 progress_notify 116
43 progress_notify 117
1 event 118
15 progress_notify 118
1 event 119
1 event 120
1 progress_notify 119
1 progress_notify 120
1 event 121
9 progress_notify 120
10 progress_notify 121
1 event 122
7 progress_notify 121
9 progress_notify 122
1 event 123
2 progress_notify 123
1 event 124
17 progress_notify 123
1 progress_notify 124
1 event 125
1 progress_notify 124
16 progress_notify 125
1 event 126
1 event 127
44 progress_notify 127
1 event 128
7 progress_notify 128
1 event 129
5 progress_notify 129
1 event 130
225 progress_notify 130
1 event 131
6 progress_notify 131
1 event 132
3 progress_notify 132
1 event 133
4 progress_notify 133
1 event 134
3 progress_notify 134
1 event 135
8 progress_notify 134
42 progress_notify 135
1 event 136
7 progress_notify 136
1 event 137
1 progress_notify 136
5 progress_notify 137
2 progress_notify 138
1 event 138
7 progress_notify 138
1 event 139
1 progress_notify 138
3 progress_notify 139
1 event 140
3 progress_notify 140
1 event 141
5 progress_notify 141
1 event 142
5 progress_notify 142
1 event 143
1 progress_notify 142
11 progress_notify 143
1 event 144
1 event 145
1 progress_notify 144
22 progress_notify 145
1 event 146
18 progress_notify 146
1 event 147
21 progress_notify 147
1 event 148
2 progress_notify 147
5 progress_notify 148
1 event 149
2 progress_notify 148
2 progress_notify 149
1 event 150
4 progress_notify 150
1 event 151
1 progress_notify 150
3 progress_notify 151
1 event 152
264 progress_notify 152
1 event 153
4 progress_notify 153
1 event 154
2 progress_notify 153
66 progress_notify 154
1 event 155
6 progress_notify 155
1 event 156
7 progress_notify 156
1 event 157
1 event 158
1 progress_notify 157
4 progress_notify 158
1 event 159
5 progress_notify 158
1 event 160
1 progress_notify 160
1 event 161
5 progress_notify 160
38 progress_notify 161
1 event 162
1 progress_notify 162
1 event 163
4 progress_notify 163
1 event 164
1 progress_notify 163
58 progress_notify 164
1 event 165
3 progress_notify 164
6 progress_notify 165
1 event 166
1 event 167
18 progress_notify 167
1 event 168
9 progress_notify 167
5 progress_notify 168
1 event 169
1 progress_notify 169
1 event 170
10 progress_notify 169
182 progress_notify 170
1 event 171
5 progress_notify 171
1 event 172
2 progress_notify 172
1 event 173
1 event 174
64 progress_notify 174
1 event 175
3 progress_notify 175
1 event 176
12 progress_notify 176
1 event 177
4 progress_notify 177
1 event 178
13 progress_notify 178
1 event 179
3 progress_notify 179
1 event 180
22 progress_notify 180
1 event 181
7 progress_notify 181
1 event 182
1 event 183
16 progress_notify 183
1 event 184
1 event 185
1 progress_notify 185
1 event 186
1 event 187
1 progress_notify 186
11 progress_notify 187
1 event 188
55 progress_notify 188
1 event 189
1 progress_notify 188
7 progress_notify 189
1 event 190
230 progress_notify 190
1 event 191
1 progress_notify 190
6 progress_notify 191
1 event 192
5 progress_notify 192
1 event 193
7 progress_notify 193
1 event 194
1 event 195
14 progress_notify 195
1 event 196
6 progress_notify 196
1 event 197
23 progress_notify 197
1 event 198
6 progress_notify 198
1 event 199
66 progress_notify 199
1 event 200
6 progress_notify 200
1 event 201
2 progress_notify 200
6 progress_notify 201
1 event 202
1 event 203
1 progress_notify 202
4 progress_notify 203
1 event 204
2 progress_notify 204
1 event 205
1 event 206
56 progress_notify 206
1 event 207
254 progress_notify 207
1 event 208
4 progress_notify 208
1 event 209
1 event 210
1 progress_notify 210
1 event 211
10 progress_notify 211
1 event 212
14 progress_notify 211
5 progress_notify 212
1 event 213
6 progress_notify 212
1 event 214
1 progress_notify 212
14 progress_notify 213
1 progress_notify 214
1 event 215
3 progress_notify 214
1 event 216
9 progress_notify 214
5 progress_notify 215
26 progress_notify 216
1 event 217
11 progress_notify 216
1 event 218
1 progress_notify 217
37 progress_notify 218
1 event 219
2 progress_notify 219
1 event 220
1 event 221
70 progress_notify 221
1 event 222
6 progress_notify 222
1 event 223
4 progress_notify 223
1 event 224
1 event 225
1 progress_notify 224
11 progress_notify 225
1 event 226
6 progress_notify 225
9 progress_notify 226
1 event 227
33 progress_notify 227
1 event 228
1 progress_notify 227
2 progress_notify 228
1 event 229
1 progress_notify 228
6 progress_notify 229
1 event 230
56 progress_notify 230
1 event 231
7 progress_notify 230
31 progress_notify 231
1 event 232
4 progress_notify 231
18 progress_notify 232
1 event 233
1 progress_notify 232
28 progress_notify 233
1 event 234
5 progress_notify 233
18 progress_notify 234
1 event 235
3 progress_notify 234
13 progress_notify 235
1 event 236
16 progress_notify 235
19 progress_notify 236
1 event 237
3 progress_notify 236
15 progress_notify 237
1 event 238
14 progress_notify 237
69 progress_notify 238
1 event 239
10 progress_notify 238
16 progress_notify 239
1 event 240
13 progress_notify 239
2 progress_notify 240
1 event 241
16 progress_notify 240
28 progress_notify 241
1 event 242
2 progress_notify 241
89 progress_notify 242
1 event 243
5 progress_notify 243
1 event 244
1 progress_notify 243
8 progress_notify 244
1 event 245
7 progress_notify 245
1 event 246
2 progress_notify 246
1 event 247
205 progress_notify 247
1 event 248
10 progress_notify 248
1 event 249
1 progress_notify 249
1 event 250
1 progress_notify 249
62 progress_notify 250
1 event 251
2 progress_notify 251
1 event 252
3 progress_notify 251
4 progress_notify 252
1 event 253
7 progress_notify 253
1 event 254
1 progress_notify 254
1 event 255
47 progress_notify 255
1 event 256
7 progress_notify 256
1 event 257
3 progress_notify 257
1 event 258
1 event 259
1 event 260
1 progress_notify 259
39 progress_notify 260
1 event 261
2 progress_notify 261
1 event 262
2 progress_notify 262
1 event 263
1 progress_notify 263
1 event 264
1 progress_notify 263
285 progress_notify 264
1 event 265
1 progress_notify 264
8 progress_notify 265
1 event 266
3 progress_notify 265
6 progress_notify 266
1 event 267
3 progress_notify 267
1 event 268
1 event 269
1 progress_notify 268
1 progress_notify 269
1 event 270
1 event 271
1 progress_notify 270
48 progress_notify 271
1 event 272
60 progress_notify 272
1 event 273
4 progress_notify 273
1 event 274
1 progress_notify 274
1 event 275
1 progress_notify 274
8 progress_notify 275
1 event 276
2 progress_notify 276
1 event 277
1 event 278
1 progress_notify 277
4 progress_notify 278
1 event 279
4 progress_notify 279
1 event 280
3 progress_notify 280
1 event 281
3 progress_notify 280
34 progress_notify 281
1 event 282
2 progress_notify 282
1 event 283
4 progress_notify 282
226 progress_notify 283
1 event 284
1 progress_notify 283
3 progress_notify 284
1 event 285
21 progress_notify 285
1 event 286
1 progress_notify 286
1 event 287
3 progress_notify 287
1 event 288
34 progress_notify 288
1 event 289
1 progress_notify 288
2 progress_notify 289
1 event 290
5 progress_notify 290
1 event 291
69 progress_notify 291
1 event 292
5 progress_notify 292
1 event 293
1 progress_notify 292
3 progress_notify 293
1 event 294
3 progress_notify 294
1 event 295
12 progress_notify 295
1 event 296
42 progress_notify 296
1 event 297
2 progress_notify 297
1 event 298
1 progress_notify 297
7 progress_notify 298
1 event 299
50 progress_notify 299
1 event 300
61 progress_notify 300
1 event 301
5 progress_notify 300
51 progress_notify 301
1 event 302
199 progress_notify 302
1 event 303
1 progress_notify 302
66 progress_notify 303
1 event 304
14 progress_notify 304
1 event 305
6 progress_notify 305
1 event 306
6 progress_notify 306
1 event 307
7 progress_notify 306
1 progress_notify 307
1 event 308
1 progress_notify 307
25 progress_notify 308
1 event 309
8 progress_notify 308
48 progress_notify 309
1 event 310
74 progress_notify 310
1 event 311
3 progress_notify 311
1 event 312
5 progress_notify 312
1 event 313
51 progress_notify 313
1 event 314
4 progress_notify 314
1 event 315
65 progress_notify 315
1 event 316
1 progress_notify 315
6 progress_notify 316
1 event 317
61 progress_notify 317
1 event 318
5 progress_notify 318
1 event 319
1 progress_notify 318
67 progress_notify 319
1 event 320
1 progress_notify 319
75 progress_notify 320
1 event 321
1 progress_notify 320
4 progress_notify 321
1 event 322
1 progress_notify 321
229 progress_notify 322
1 event 323
63 progress_notify 323
1 event 324
1 event 325
3 progress_notify 325
1 event 326
4 progress_notify 326
1 event 327
11 progress_notify 327
1 event 328
28 progress_notify 328
1 event 329
6 progress_notify 329
1 event 330
68 progress_notify 330
1 event 331
5 progress_notify 331
1 event 332
36 progress_notify 332
1 event 333
5 progress_notify 333
1 event 334
1 progress_notify 333
59 progress_notify 334
1 event 335
1 progress_notify 334
81 progress_notify 335
1 event 336
1 progress_notify 335
3 progress_notify 336
1 event 337
1 progress_notify 336
1 event 338
62 progress_notify 338
1 event 339
67 progress_notify 339
1 event 340
3 progress_notify 340
1 event 341
69 progress_notify 341
1 event 342
4 progress_notify 342
1 event 343
172 progress_notify 343
1 event 344
3 progress_notify 343
17 progress_notify 344
1 event 345
1 progress_notify 344
19 progress_notify 345
1 event 346
8 progress_notify 345
15 progress_notify 346
1 event 347
12 progress_notify 346
71 progress_notify 347
1 event 348
1 progress_notify 347
19 progress_notify 348
1 event 349
11 progress_notify 348
68 progress_notify 349
1 event 350
6 progress_notify 349
29 progress_notify 350
1 event 351
3 progress_notify 350
24 progress_notify 351
1 event 352
69 progress_notify 352
1 event 353
1 event 354
1 progress_notify 353
1 event 355
7 progress_notify 355
1 event 356
40 progress_notify 356
1 event 357
66 progress_notify 357
1 event 358
1 progress_notify 357
4 progress_notify 358
1 event 359
64 progress_notify 359
1 event 360
275 progress_notify 360
1 event 361
71 progress_notify 361
1 event 362
4 progress_notify 362
1 event 363
6 progress_notify 363
1 event 364
5 progress_notify 364
1 event 365
4 progress_notify 364
9 progress_notify 365
1 event 366
6 progress_notify 366
1 event 367
28 progress_notify 367
1 event 368
6 progress_notify 368
1 event 369
127 progress_notify 369
1 event 370
5 progress_notify 370
1 event 371
60 progress_notify 371
1 event 372
1 progress_notify 372
1 event 373
60 progress_notify 373
1 event 374
62 progress_notify 374
1 event 375
2 progress_notify 375
1 event 376
21 progress_notify 376
1 event 377
2 progress_notify 376
5 progress_notify 377
1 event 378
55 progress_notify 378
1 event 379
247 progress_notify 379
1 event 380
3 progress_notify 380
1 event 381
66 progress_notify 381
1 event 382
3 progress_notify 382
1 event 383
1 progress_notify 382
12 progress_notify 383
1 event 384
2 progress_notify 384
1 event 385
6 progress_notify 385
1 event 386
38 progress_notify 386
1 event 387
77 progress_notify 387
1 event 388
57 progress_notify 388
1 event 389
4 progress_notify 389
1 event 390
1 progress_notify 389
67 progress_notify 390
1 event 391
6 progress_notify 391
1 event 392
2 progress_notify 391
151 progress_notify 392
1 event 393
5 progress_notify 393
1 event 394
68 progress_notify 394
1 event 395
4 progress_notify 395
1 event 396
59 progress_notify 396
1 event 397
251 progress_notify 397
1 event 398
4 progress_notify 398
1 event 399
57 progress_notify 399
1 event 400
1 progress_notify 399
12 progress_notify 400
1 event 401
4 progress_notify 401
1 event 402
1 progress_notify 402
1 event 403
1 progress_notify 403
1 event 404
13 progress_notify 404
1 event 405
12 progress_notify 404
1 progress_notify 405
1 event 406
8 progress_notify 405
164 progress_notify 406
1 event 407
4 progress_notify 407
1 event 408
1 progress_notify 407
5 progress_notify 408
1 event 409
57 progress_notify 409
1 event 410
3 progress_notify 410
1 event 411
3 progress_notify 410
62 progress_notify 411
1 event 412
1 progress_notify 411
4 progress_notify 412
1 event 413
58 progress_notify 413
1 event 414
6 progress_notify 414
1 event 415
51 progress_notify 415
1 event 416
72 progress_notify 416
1 event 417
2 progress_notify 417
1 event 418
1 progress_notify 417
247 progress_notify 418
1 event 419
65 progress_notify 419
1 event 420
5 progress_notify 420
1 event 421
1 progress_notify 420
6 progress_notify 421
1 event 422
1 progress_notify 421
7 progress_notify 422
1 event 423
7 progress_notify 423
1 event 424
2 progress_notify 423
9 progress_notify 424
1 event 425
26 progress_notify 425
1 event 426
152 progress_notify 426
1 event 427
1 event 428
6 progress_notify 427
48 progress_notify 428
1 event 429
2 progress_notify 428
5 progress_notify 429
1 event 430
46 progress_notify 430
1 event 431
2 progress_notify 431
1 event 432
5 progress_notify 431
105 progress_notify 432
1 event 433
14 progress_notify 432
21 progress_notify 433
1 event 434
3 progress_notify 433
5 progress_notify 434
1 event 435
13 progress_notify 434
10 progress_notify 435
1 event 436
8 progress_notify 435
13 progress_notify 436
1 event 437
5 progress_notify 436
17 progress_notify 437
1 event 438
10 progress_notify 437
110 progress_notify 438
1 event 439
3 progress_notify 439
1 event 440
1 progress_notify 439
252 progress_notify 440
1 event 441
65 progress_notify 441
1 event 442
7 progress_notify 442
1 event 443
58 progress_notify 443
1 event 444
71 progress_notify 444
1 event 445
7 progress_notify 445
1 event 446
53 progress_notify 446
1 event 447
1 progress_notify 446
57 progress_notify 447
1 event 448
79 progress_notify 448
1 event 449
1 progress_notify 448
242 progress_notify 449
1 event 450
1 progress_notify 449
65 progress_notify 450
1 event 451
1 progress_notify 450
287 progress_notify 451
1 event 452
1 progress_notify 451
72 progress_notify 452
1 event 453
1 progress_notify 453
1 event 454
1 progress_notify 453
218 progress_notify 454
1 event 455
217 progress_notify 455
1 event 456
62 progress_notify 456
1 event 457
136 progress_notify 457
1 event 458
134 progress_notify 458
1 event 459
356 progress_notify 459
1 event 460
324 progress_notify 460
1 event 461
6 progress_notify 461
1 event 462
230 progress_notify 462
1 event 463
75 progress_notify 463
1 event 464
69 progress_notify 464
1 event 465
34 progress_notify 465
1 event 466
9 progress_notify 466
1 event 467
53 progress_notify 467
1 event 468
7 progress_notify 468
1 event 469
59 progress_notify 469
1 event 470
3 progress_notify 470
1 event 471
57 progress_notify 471
1 event 472
7 progress_notify 472
1 event 473
55 progress_notify 473
1 event 474
112 progress_notify 474
1 event 475
4 progress_notify 474
62 progress_notify 475
1 event 476
28 progress_notify 476
1 event 477
5 progress_notify 476
24 progress_notify 477
1 event 478
3 progress_notify 477
2 progress_notify 478
1 event 479
4 progress_notify 478
19 progress_notify 479
1 event 480
1 progress_notify 479
24 progress_notify 480
1 event 481
9 progress_notify 480
17 progress_notify 481
1 event 482
9 progress_notify 481
1 event 483
1 progress_notify 482
242 progress_notify 483
1 event 484
1 progress_notify 484
1 event 485
1 progress_notify 484
12 progress_notify 485
1 event 486
1 progress_notify 485
66 progress_notify 486
1 event 487
1 progress_notify 486
6 progress_notify 487
1 event 488
1 progress_notify 487
1 event 489
66 progress_notify 489
1 event 490
1 progress_notify 490
1 event 491
76 progress_notify 491
1 event 492
5 progress_notify 492
1 event 493
237 progress_notify 493
1 event 494
6 progress_notify 494
1 event 495
39 progress_notify 495
1 event 496
1 event 497
1 event 498
1 event 499
49 progress_notify 499
1 progress_notify 500
1 event 500
4 progress_notify 500
1 event 501
60 progress_notify 501
1 event 502
3 progress_notify 502
1 event 503
71 progress_notify 503
1 event 504
1 progress_notify 503
6 progress_notify 504
1 event 505
63 progress_notify 505
1 event 506
1 progress_notify 505
5 progress_notify 506
1 event 507
61 progress_notify 507
1 event 508
1 progress_notify 507
1 event 509
57 progress_notify 509
1 event 510
12 progress_notify 510
1 event 511
53 progress_notify 511
1 event 512
66 progress_notify 512
1 event 513
8 progress_notify 513
1 event 514
59 progress_notify 514
1 event 515
1 event 516
1 progress_notify 515
235 progress_notify 516
1 event 517
1 progress_notify 517
1 event 518
68 progress_notify 518
1 event 519
1 progress_notify 518
5 progress_notify 519
1 event 520
4 progress_notify 520
1 event 521
11 progress_notify 521
1 event 522
2 progress_notify 522
1 event 523
4 progress_notify 523
1 event 524
34 progress_notify 524
1 event 525
75 progress_notify 525
1 event 526
48 progress_notify 526
2 progress_notify 527
1 event 527
2 progress_notify 527
1 event 528
1 progress_notify 527
70 progress_notify 528
1 event 529
2 progress_notify 528
1 progress_notify 529
1 event 530
75 progress_notify 530
1 event 531
1 event 532
2 progress_notify 531
50 progress_notify 532
1 event 533
6 progress_notify 533
1 event 534
69 progress_notify 534
1 event 535
279 progress_notify 535
1 event 536
3 progress_notify 536
1 event 537
1 progress_notify 537
2 progress_notify 538
1 event 538
61 progress_notify 538
1 event 539
13 progress_notify 539
1 event 540
2 progress_notify 539
6 progress_notify 540
1 event 541
3 progress_notify 541
1 event 542
8 progress_notify 542
1 event 543
93 progress_notify 543
1 event 544
5 progress_notify 544
1 event 545
60 progress_notify 545
1 event 546
5 progress_notify 546
1 event 547
57 progress_notify 547
1 event 548
3 progress_notify 548
1 event 549
4 progress_notify 548
78 progress_notify 549
1 event 550
1 progress_notify 549
4 progress_notify 550
1 event 551
62 progress_notify 551
1 event 552
77 progress_notify 552
1 event 553
9 progress_notify 553
1 event 554
63 progress_notify 554
1 event 555
3 progress_notify 555
1 event 556
238 progress_notify 556
1 event 557
64 progress_notify 557
1 event 558
6 progress_notify 558
1 event 559
7 progress_notify 559
1 event 560
1 event 561
1 progress_notify 560
2 progress_notify 561
1 event 562
2 progress_notify 561
12 progress_notify 562
1 event 563
1 event 564
39 progress_notify 564
1 event 565
49 progress_notify 565
1 event 566
1 progress_notify 566
1 event 567
2 progress_notify 566
37 progress_notify 567
1 event 568
64 progress_notify 568
1 event 569
2 progress_notify 568
6 progress_notify 569
1 event 570
58 progress_notify 570
1 event 571
6 progress_notify 571
1 event 572
31 progress_notify 572
1 event 573
26 progress_notify 573
1 event 574
5 progress_notify 573
103 progress_notify 574
1 event 575
16 progress_notify 574
2 progress_notify 575
1 event 576
16 progress_notify 575
368 progress_notify 576
1 event 577
7 progress_notify 577
1 event 578
61 progress_notify 578
1 event 579
2 progress_notify 579
1 event 580
6 progress_notify 580
1 event 581
4 progress_notify 581
1 event 582
5 progress_notify 582
1 event 583
2 progress_notify 583
1 event 584
7 progress_notify 584
1 event 585
8 progress_notify 584
45 progress_notify 585
1 event 586
1 progress_notify 585
23 progress_notify 586
1 event 587
1 progress_notify 586
77 progress_notify 587
1 event 588
5 progress_notify 588
1 event 589
1 progress_notify 589
4 progress_notify 590
1 event 590
53 progress_notify 590
1 event 591
1 progress_notify 590
76 progress_notify 591
1 event 592
1 progress_notify 592
1 event 593
73 progress_notify 593
1 event 594
3 progress_notify 594
1 event 595
2 progress_notify 594
74 progress_notify 595
1 event 596
2 progress_notify 596
1 event 597
226 progress_notify 597
1 event 598
2 progress_notify 598
1 event 599
61 progress_notify 599
1 event 600
4 progress_notify 600
1 event 601
6 progress_notify 601
1 event 602
7 progress_notify 602
1 event 603
2 progress_notify 602
8 progress_notify 603
1 event 604
28 progress_notify 604
1 event 605
2 progress_notify 604
6 progress_notify 605
1 event 606
63 progress_notify 606
1 event 607
52 progress_notify 607
1 event 608
5 progress_notify 608
1 event 609
64 progress_notify 609
1 event 610
3 progress_notify 610
1 event 611
1 progress_notify 610
63 progress_notify 611
1 event 612
1 event 613
1 progress_notify 612
30 progress_notify 613
1 event 614
7 progress_notify 613
18 progress_notify 614
1 event 615
39 progress_notify 615
1 event 616
44 progress_notify 616
1 event 617
1 progress_notify 616
6 progress_notify 617
1 event 618
252 progress_notify 618
1 event 619
1 event 620
71 progress_notify 620
1 event 621
1 progress_notify 620
5 progress_notify 621
1 event 622
3 progress_notify 622
1 event 623
5 progress_notify 623
1 event 624
1 event 625
1 progress_notify 624
5 progress_notify 625
1 event 626
44 progress_notify 626
1 event 627
7 progress_notify 627
1 event 628
64 progress_notify 628
1 event 629
1 progress_notify 628
69 progress_notify 629
1 event 630
68 progress_notify 630
1 event 631
4 progress_notify 631
1 event 632
63 progress_notify 632
1 event 633
1 progress_notify 632
3 progress_notify 633
1 event 634
68 progress_notify 634
1 event 635
1 progress_notify 634
2 progress_notify 635
2 progress_notify 636
1 event 636
67 progress_notify 636
1 event 637
4 progress_notify 637
1 event 638
67 progress_notify 638
1 event 639
1 event 640
254 progress_notify 640
1 event 641
1 progress_notify 640
78 progress_notify 641
1 event 642
3 progress_notify 641
4 progress_notify 642
1 event 643
4 progress_notify 643
1 event 644
2 progress_notify 643
8 progress_notify 644
1 event 645
1 progress_notify 645
1 event 646
5 progress_notify 646
1 event 647
26 progress_notify 647
1 event 648
4 progress_notify 647
3 progress_notify 648
1 event 649
69 progress_notify 649
1 event 650
75 progress_notify 650
1 event 651
1 event 652
75 progress_notify 652
1 event 653
66 progress_notify 653
1 event 654
4 progress_notify 654
1 event 655
69 progress_notify 655
1 event 656
1 event 657
71 progress_notify 657
1 event 658
1 progress_notify 657
91 progress_notify 658
1 event 659
235 progress_notify 659
1 event 660
1 progress_notify 659
5 progress_notify 660
1 event 661
1 event 662
36 progress_notify 662
1 event 663
11 progress_notify 662
1 event 664
1 progress_notify 663
1 event 665
1 progress_notify 664
16 progress_notify 665
1 event 666
1 progress_notify 665
9 progress_notify 666
1 event 667
9 progress_notify 666
19 progress_notify 667
1 event 668
6 progress_notify 667
19 progress_notify 668
1 event 669
6 progress_notify 668
20 progress_notify 669
1 event 670
6 progress_notify 669
13 progress_notify 670
1 event 671
7 progress_notify 670
11 progress_notify 671
1 event 672
13 progress_notify 671
26 progress_notify 672
1 event 673
10 progress_notify 672
34 progress_notify 673
1 event 674
1 progress_notify 673
138 progress_notify 674
1 event 675
4 progress_notify 674
121 progress_notify 675
1 event 676
5 progress_notify 676
1 event 677
5 progress_notify 677
1 event 678
1 progress_notify 677
4 progress_notify 678
1 event 679
2 progress_notify 679
1 event 680
26 progress_notify 680
1 event 681
3 progress_notify 681
1 event 682
70 progress_notify 682
1 event 683
263 progress_notify 683
1 event 684
7 progress_notify 684
1 event 685
72 progress_notify 685
1 event 686
6 progress_notify 686
1 event 687
6 progress_notify 687
1 event 688
1 event 689
8 progress_notify 689
1 event 690
42 progress_notify 690
1 event 691
4 progress_notify 691
1 event 692
62 progress_notify 692
1 event 693
5 progress_notify 693
1 event 694
67 progress_notify 694
1 event 695
6 progress_notify 695
1 event 696
56 progress_notify 696
1 event 697
4 progress_notify 697
1 event 698
97 progress_notify 698
1 event 699
81 progress_notify 699
1 event 700
5 progress_notify 700
1 event 701
62 progress_notify 701
1 event 702
3 progress_notify 702
1 event 703
1 progress_notify 702
249 progress_notify 703
1 event 704
1 progress_notify 703
68 progress_notify 704
1 event 705
4 progress_notify 705
1 event 706
6 progress_notify 706
1 event 707
4 progress_notify 706
6 progress_notify 707
1 event 708
4 progress_notify 708
1 event 709
4 progress_notify 708
6 progress_notify 709
1 event 710
3 progress_notify 710
1 event 711
1 progress_notify 710
21 progress_notify 711
1 event 712
64 progress_notify 712
1 event 713
5 progress_notify 713
1 event 714
67 progress_notify 714
1 event 715
9 progress_notify 715
1 event 716
69 progress_notify 716
1 event 717
7 progress_notify 717
1 event 718
54 progress_notify 718
1 event 719
6 progress_notify 719
1 event 720
59 progress_notify 720
1 event 721
1 progress_notify 720
7 progress_notify 721
1 event 722
4 progress_notify 721
65 progress_notify 722
1 event 723
3 progress_notify 723
1 event 724
56 progress_notify 724
1 event 725
1 progress_notify 724
2 progress_notify 725
1 event 726
3 progress_notify 725
234 progress_notify 726
1 event 727
75 progress_notify 727
1 event 728
1 progress_notify 727
4 progress_notify 728
1 event 729
6 progress_notify 729
1 event 730
7 progress_notify 730
1 event 731
4 progress_notify 731
1 event 732
2 progress_notify 731
5 progress_notify 732
1 event 733
25 progress_notify 733
1 event 734
3 progress_notify 734
1 event 735
2 progress_notify 734
70 progress_notify 735
1 event 736
69 progress_notify 736
1 event 737
6 progress_notify 737
1 event 738
59 progress_notify 738
1 event 739
1 event 740
5 progress_notify 739
67 progress_notify 740
1 event 741
4 progress_notify 741
1 event 742
2 progress_notify 741
73 progress_notify 742
1 event 743
9 progress_notify 743
1 event 744
54 progress_notify 744
1 event 745
8 progress_notify 745
1 event 746
68 progress_notify 746
1 event 747
1 progress_notify 746
1 event 748
1 progress_notify 747
236 progress_notify 748
1 event 749
66 progress_notify 749
1 event 750
5 progress_notify 750
1 event 751
1 event 752
3 progress_notify 752
1 event 753
9 progress_notify 753
1 event 754
11 progress_notify 753
72 progress_notify 754
1 event 755
77 progress_notify 755
1 event 756
3 progress_notify 756
1 event 757
1 event 758
98 progress_notify 758
1 event 759
10 progress_notify 759
1 event 760
65 progress_notify 760
1 event 761
1 progress_notify 760
7 progress_notify 761
1 event 762
1 event 763
1 progress_notify 762
13 progress_notify 763
1 event 764
24 progress_notify 764
1 event 765
1 progress_notify 764
1 event 766
1 progress_notify 765
49 progress_notify 766
1 event 767
11 progress_notify 766
45 progress_notify 767
1 event 768
4 progress_notify 767
10 progress_notify 768
1 event 769
8 progress_notify 768
30 progress_notify 769
1 event 770
8 progress_notify 769
26 progress_notify 770
1 event 771
14 progress_notify 770
193 progress_notify 771
1 event 772
4 progress_notify 772
1 event 773
44 progress_notify 773
1 event 774
4 progress_notify 774
1 event 775
64 progress_notify 775
1 event 776
67 progress_notify 776
1 event 777
4 progress_notify 777
1 progress_notify 778
1 event 778
74 progress_notify 778
1 event 779
4 progress_notify 779
1 event 780
73 progress_notify 780
1 event 781
2 progress_notify 780
5 progress_notify 781
1 event 782
1 progress_notify 781
35 progress_notify 782
1 event 783
1 progress_notify 782
77 progress_notify 783
1 event 784
5 progress_notify 784
1 event 785
69 progress_notify 785
1 event 786
3 progress_notify 786
1 event 787
72 progress_notify 787
1 event 788
249 progress_notify 788
1 event 789
72 progress_notify 789
1 event 790
6 progress_notify 790
1 event 791
1 progress_notify 790
60 progress_notify 791
1 event 792
68 progress_notify 792
1 event 793
4 progress_notify 793
1 event 794
1 event 795
70 progress_notify 795
1 event 796
81 progress_notify 796
1 event 797
70 progress_notify 797
1 event 798
1 progress_notify 797
194 progress_notify 798
1 event 799
197 progress_notify 799
1 event 800
4 progress_notify 800
1 event 801
3 progress_notify 801
1 event 802
1 progress_notify 801
193 progress_notify 802
1 event 803
7 progress_notify 803
1 event 804
7 progress_notify 804
2 progress_notify 805
1 event 805
162 progress_notify 805
1 event 806
4 progress_notify 806
1 event 807
2 progress_notify 807
1 event 808
2 progress_notify 808
1 event 809
722 progress_notify 809
1 event 810
4 progress_notify 810
1 event 811
192 progress_notify 811
1 event 812
3 progress_notify 812
1 event 813
2 progress_notify 812
1 progress_notify 813
1 event 814
195 progress_notify 814
1 event 815
207 progress_notify 815
1 event 816
214 progress_notify 816
1 event 817
5 progress_notify 816
15 progress_notify 817
1 event 818
3 progress_notify 817
11 progress_notify 818
1 event 819
19 progress_notify 819
1 event 820
1 progress_notify 819
18 progress_notify 820
1 event 821
1049 progress_notify 821
1 event 822
1 progress_notify 822
1 event 823
1 progress_notify 823
1 event 824
2 progress_notify 823
6 progress_notify 824
1 event 825
72 progress_notify 825
1 event 826
1 progress_notify 825
80 progress_notify 826
1 event 827
18 progress_notify 827
1 event 828
4 progress_notify 828
1 event 829
181 progress_notify 829
1 event 830
127 progress_notify 830
1 event 831
68 progress_notify 831
1 event 832
437 progress_notify 832
1 event 833
101 progress_notify 833
1 event 834
82 progress_notify 834
1 event 835
3 progress_notify 835
1 event 836
1 progress_notify 835
7 progress_notify 836
1 event 837
3 progress_notify 836
1 progress_notify 837
1 event 838
5 progress_notify 838
1 event 839
64 progress_notify 839
1 event 840
71 progress_notify 840
1 event 841
1 event 842
1 progress_notify 841
11 progress_notify 842
1 event 843
4 progress_notify 843
1 event 844
1 event 845
1 progress_notify 844
18 progress_notify 845
1 event 846
1 progress_notify 845
37 progress_notify 846
1 event 847
75 progress_notify 847
1 event 848
9 progress_notify 848
1 event 849
6 progress_notify 849
1 event 850
64 progress_notify 850
1 event 851
206 progress_notify 851
1 event 852
4 progress_notify 852
1 event 853
5 progress_notify 853
1 event 854
1 progress_notify 854
1 event 855
2 progress_notify 854
2 progress_notify 855
1 event 856
4 progress_notify 856
1 event 857
12 progress_notify 857
1 event 858
34 progress_notify 858
1 event 859
58 progress_notify 859
1 event 860
5 progress_notify 860
1 event 861
10 progress_notify 861
1 event 862
2 progress_notify 861
6 progress_notify 862
1 event 863
18 progress_notify 863
1 event 864
24 progress_notify 864
1 event 865
1 progress_notify 864
8 progress_notify 865
1 event 866
224 progress_notify 866
1 event 867
3 progress_notify 867
1 event 868
8 progress_notify 868
1 event 869
13 progress_notify 869
1 event 870
8 progress_notify 870
1 event 871
51 progress_notify 871
1 event 872
6 progress_notify 871
4 progress_notify 872
1 event 873
12 progress_notify 872
71 progress_notify 873
1 event 874
1 progress_notify 873
9 progress_notify 874
1 event 875
9 progress_notify 875
1 event 876
6 progress_notify 876
1 event 877
1 progress_notify 877
1 event 878
1 progress_notify 877
29 progress_notify 878
1 event 879
10 progress_notify 879
1 event 880
6 progress_notify 880
1 event 881
222 progress_notify 881
1 event 882
15 progress_notify 882
1 event 883
2 progress_notify 882
10 progress_notify 883
1 event 884
37 progress_notify 884
1 event 885
12 progress_notify 885
1 event 886
76 progress_notify 886
1 event 887
6 progress_notify 887
1 event 888
41 progress_notify 888
2 progress_notify 889
1 event 889
9 progress_notify 889
1 event 890
4 progress_notify 890
1 event 891
1 progress_notify 890
207 progress_notify 891
1 event 892
4 progress_notify 892
1 event 893
71 progress_notify 893
1 event 894
60 progress_notify 894
1 event 895
16 progress_notify 895
3 progress_notify 896
1 event 896
60 progress_notify 896
1 event 897
229 progress_notify 897
1 event 898
3 progress_notify 898
1 event 899
1 progress_notify 898
3 progress_notify 899
1 event 900
1 progress_notify 900
1 event 901
7 progress_notify 901
1 event 902
66 progress_notify 902
1 event 903
2 progress_notify 903
1 event 904
2 progress_notify 903
67 progress_notify 904
1 event 905
6 progress_notify 904
10 progress_notify 905
1 event 906
6 progress_notify 906
1 event 907
48 progress_notify 907
1 event 908
8 progress_notify 908
1 event 909
88 progress_notify 909
1 event 910
6 progress_notify 909
4 progress_notify 910
1 event 911
7 progress_notify 910
68 progress_notify 911
1 event 912
10 progress_notify 911
78 progress_notify 912
1 event 913
6 progress_notify 912
127 progress_notify 913
1 event 914
3 progress_notify 913
41 progress_notify 914
1 event 915
2 progress_notify 914
25 progress_notify 915
1 event 916
8 progress_notify 915
21 progress_notify 916
1 event 917
3 progress_notify 916
132 progress_notify 917
1 event 918
52 progress_notify 918
1 event 919
5 progress_notify 919
1 event 920
7 progress_notify 920
1 event 921
1 progress_notify 921
1 event 922
5 progress_notify 921
2 progress_notify 922
1 event 923
6 progress_notify 923
1 event 924
1 progress_notify 924
1 event 925
7 progress_notify 925
1 event 926
2 progress_notify 926
1 event 927
1 event 928
1 progress_notify 927
242 progress_notify 928
1 event 929
1 event 930
1 event 931
5 progress_notify 931
1 event 932
9 progress_notify 931
36 progress_notify 932
1 event 933
9 progress_notify 933
1 event 934
1 event 935
1 event 936
1 progress_notify 937
1 event 937
1 event 938
3 progress_notify 937
2 progress_notify 938
1 event 939
25 progress_notify 939
1 event 940
1 event 941
1 event 942
58 progress_notify 942
1 event 943
2 progress_notify 943
1 event 944
6 progress_notify 944
1 event 945
3 progress_notify 944
1 event 946
1 progress_notify 946
1 event 947
1 event 948
1 progress_notify 947
2 progress_notify 948
1 event 949
1 event 950
43 progress_notify 950
1 event 951
1 progress_notify 950
6 progress_notify 951
1 event 952
4 progress_notify 952
1 event 953
246 progress_notify 953
1 event 954
2 progress_notify 954
1 event 955
10 progress_notify 955
1 event 956
4 progress_notify 956
1 event 957
4 progress_notify 957
1 event 958
9 progress_notify 958
1 event 959
1 event 960
30 progress_notify 960
1 event 961
6 progress_notify 961
1 event 962
9 progress_notify 962
1 event 963
5 progress_notify 963
1 event 964
43 progress_notify 964
1 event 965
3 progress_notify 965
1 event 966
4 progress_notify 965
1 event 967
1 event 968
13 progress_notify 967
3 progress_notify 968
1 event 969
13 progress_notify 968
9 progress_notify 969
1 event 970
14 progress_notify 970
1 event 971
15 progress_notify 970
1 event 972
1 progress_notify 971
10 progress_notify 972
1 event 973
1 progress_notify 972
6 progress_notify 973
1 event 974
4 progress_notify 974
1 event 975
29 progress_notify 975
1 event 976
283 progress_notify 976
1 event 977
5 progress_notify 977
1 event 978
77 progress_notify 978
1 event 979
1 event 980
1 progress_notify 979
11 progress_notify 980
1 event 981
7 progress_notify 981
1 event 982
2 progress_notify 981
5 progress_notify 982
1 event 983
3 progress_notify 982
8 progress_notify 983
1 event 984
4 progress_notify 984
1 event 985
3 progress_notify 984
28 progress_notify 985
1 event 986
1 progress_notify 985
3 progress_notify 986
1 event 987
67 progress_notify 987
1 event 988
1 event 989
72 progress_notify 989
1 event 990
5 progress_notify 990
1 event 991
1 progress_notify 990
60 progress_notify 991
1 event 992
2 progress_notify 991
4 progress_notify 992
1 event 993
56 progress_notify 993
1 event 994
72 progress_notify 994
1 event 995
7 progress_notify 995
1 event 996
2 progress_notify 995
47 progress_notify 996
1 event 997
3 progress_notify 997
1 event 998
243 progress_notify 998
1 event 999
1 event 1000
1 progress_notify 999
54 progress_notify 1000
1 event 1001
10 progress_notify 1000
16 progress_notify 1001
1 event 1002
1 progress_notify 1002
1 event 1003
7 progress_notify 1003
1 event 1004
5 progress_notify 1004
1 event 1005
1 progress_notify 1005
1 event 1006
1 progress_notify 1005
38 progress_notify 1006
1 event 1007
2 progress_notify 1007
1 event 1008
60 progress_notify 1008
1 event 1009
6 progress_notify 1009
1 event 1010
62 progress_notify 1010
1 event 1011
2 progress_notify 1011
1 event 1012
55 progress_notify 1012
1 event 1013
1 progress_notify 1013
1 event 1014
1 progress_notify 1013
75 progress_notify 1014
1 event 1015
3 progress_notify 1015
1 event 1016
68 progress_notify 1016
1 event 1017
4 progress_notify 1017
1 event 1018
69 progress_notify 1018
1 event 1019
2 progress_notify 1019
1 event 1020
73 progress_notify 1020
1 event 1021
233 progress_notify 1021
1 event 1022
4 progress_notify 1022
1 event 1023
61 progress_notify 1023
1 event 1024
5 progress_notify 1024
1 event 1025
8 progress_notify 1025
1 event 1026
2 progress_notify 1025
2 progress_notify 1026
1 event 1027
1 event 1028
1 progress_notify 1027
21 progress_notify 1028
1 event 1029
4 progress_notify 1029
1 event 1030
17 progress_notify 1029
2 progress_notify 1030
1 event 1031
9 progress_notify 1030
19 progress_notify 1031
1 event 1032
7 progress_notify 1031
60 progress_notify 1032
1 event 1033
71 progress_notify 1033
1 event 1034
12 progress_notify 1034
1 event 1035
59 progress_notify 1035
1 event 1036
1 progress_notify 1036
1 event 1037
71 progress_notify 1037
1 event 1038
1 event 1039
1 progress_notify 1038
72 progress_notify 1039
1 event 1040
4 progress_notify 1040
1 event 1041
24 progress_notify 1041
1 event 1042
1 progress_notify 1041
2 progress_notify 1042
1 event 1043
16 progress_notify 1042
18 progress_notify 1043
1 event 1044
37 progress_notify 1044
1 event 1045
10 progress_notify 1044
34 progress_notify 1045
1 event 1046
1 progress_notify 1045
22 progress_notify 1046
1 event 1047
43 progress_notify 1047
1 event 1048
1 progress_notify 1047
217 progress_notify 1048
1 event 1049
4 progress_notify 1048
35 progress_notify 1049
1 event 1050
1 event 1051
1 event 1052
14 progress_notify 1052
1 event 1053
2 progress_notify 1053
1 event 1054
1 progress_notify 1053
39 progress_notify 1054
1 event 1055
1 progress_notify 1054
1 progress_notify 1055
1 event 1056
7 progress_notify 1055
34 progress_notify 1056
1 event 1057
3 progress_notify 1057
1 event 1058
1 progress_notify 1057
71 progress_notify 1058
1 event 1059
62 progress_notify 1059
1 event 1060
6 progress_notify 1060
1 event 1061
69 progress_notify 1061
1 event 1062
1 event 1063
73 progress_notify 1063
1 event 1064
1 progress_notify 1063
4 progress_notify 1064
1 event 1065
72 progress_notify 1065
1 event 1066
3 progress_notify 1066
1 event 1067
244 progress_notify 1067
1 event 1068
4 progress_notify 1068
1 event 1069
65 progress_notify 1069
1 event 1070
3 progress_notify 1070
1 event 1071
3 progress_notify 1070
2 progress_notify 1071
1 event 1072
1 progress_notify 1072
1 event 1073
1 progress_notify 1073
1 event 1074
5 progress_notify 1073
1 event 1075
63 progress_notify 1075
1 event 1076
1 progress_notify 1075
65 progress_notify 1076
1 event 1077
6 progress_notify 1077
1 event 1078
1 progress_notify 1077
59 progress_notify 1078
1 event 1079
1 progress_notify 1078
4 progress_notify 1079
1 event 1080
64 progress_notify 1080
1 event 1081
1 progress_notify 1080
2 progress_notify 1081
1 event 1082
67 progress_notify 1082
1 event 1083
69 progress_notify 1083
1 event 1084
68 progress_notify 1084
1 event 1085
1 progress_notify 1085
1 event 1086
71 progress_notify 1086
1 event 1087
2 progress_notify 1087
1 event 1088
229 progress_notify 1088
1 event 1089
9 progress_notify 1089
1 event 1090
62 progress_notify 1090
1 event 1091
6 progress_notify 1091
1 event 1092
3 progress_notify 1092
1 event 1093
1 progress_notify 1092
13 progress_notify 1093
1 event 1094
10 progress_notify 1094
1 event 1095
28 progress_notify 1095
1 event 1096
1 event 1097
1 progress_notify 1096
73 progress_notify 1097
1 event 1098
4 progress_notify 1098
1 event 1099
61 progress_notify 1099
1 event 1100
3 progress_notify 1099
1 progress_notify 1100
1 event 1101
5 progress_notify 1100
64 progress_notify 1101
1 event 1102
1 progress_notify 1101
3 progress_notify 1102
1 event 1103
85 progress_notify 1103
1 event 1104
2 progress_notify 1103
3 progress_notify 1104
1 event 1105
64 progress_notify 1105
1 event 1106
2 progress_notify 1105
60 progress_notify 1106
1 event 1107
5 progress_notify 1107
1 event 1108
1 progress_notify 1107
59 progress_notify 1108
1 event 1109
236 progress_notify 1109
1 event 1110
1 progress_notify 1110
1 event 1111
65 progress_notify 1111
1 event 1112
1 progress_notify 1111
4 progress_notify 1112
1 event 1113
2 progress_notify 1113
1 progress_notify 1114
1 event 1114
3 progress_notify 1114
1 event 1115
1 progress_notify 1115
1 event 1116
1 progress_notify 1116
1 event 1117
1 event 1118
1 progress_notify 1117
26 progress_notify 1118
1 event 1119
8 progress_notify 1118
95 progress_notify 1119
1 event 1120
68 progress_notify 1120
1 event 1121
2 progress_notify 1121
1 event 1122
1 event 1123
64 progress_notify 1123
1 event 1124
5 progress_notify 1124
1 event 1125
72 progress_notify 1125
1 event 1126
49 progress_notify 1126
1 event 1127
68 progress_notify 1127
1 event 1128
1 progress_notify 1127
6 progress_notify 1128
1 event 1129
59 progress_notify 1129
1 event 1130
2 progress_notify 1129
6 progress_notify 1130
1 event 1131
249 progress_notify 1131
1 event 1132
4 progress_notify 1132
1 event 1133
58 progress_notify 1133
1 event 1134
1 progress_notify 1134
1 event 1135
2 progress_notify 1134
4 progress_notify 1135
1 event 1136
1 progress_notify 1136
1 event 1137
6 progress_notify 1137
1 event 1138
54 progress_notify 1138
1 event 1139
3 progress_notify 1138
6 progress_notify 1139
1 event 1140
66 progress_notify 1140
1 event 1141
8 progress_notify 1141
1 event 1142
58 progress_notify 1142
1 event 1143
20 progress_notify 1143
1 event 1144
10 progress_notify 1143
25 progress_notify 1144
1 event 1145
11 progress_notify 1144
10 progress_notify 1145
1 event 1146
17 progress_notify 1145
8 progress_notify 1146
1 event 1147
1 event 1148
8 progress_notify 1146
1 progress_notify 1147
31 progress_notify 1148
1 event 1149
1 progress_notify 1148
13 progress_notify 1149
1 event 1150
5 progress_notify 1149
1 progress_notify 1150
1 event 1151
15 progress_notify 1150
3 progress_notify 1151
1 event 1152
4 progress_notify 1151
11 progress_notify 1152
1 event 1153
3 progress_notify 1152
369 progress_notify 1153
1 event 1154
4 progress_notify 1154
1 event 1155
67 progress_notify 1155
1 event 1156
1 progress_notify 1155
13 progress_notify 1156
1 event 1157
8 progress_notify 1157
1 event 1158
6 progress_notify 1158
1 event 1159
1 event 1160
10 progress_notify 1160
1 event 1161
1 progress_notify 1160
31 progress_notify 1161
1 event 1162
1 progress_notify 1162
1 event 1163
5 progress_notify 1162
74 progress_notify 1163
1 event 1164
1 event 1165
1 progress_notify 1164
70 progress_notify 1165
1 event 1166
69 progress_notify 1166
1 event 1167
3 progress_notify 1167
1 event 1168
66 progress_notify 1168
1 event 1169
5 progress_notify 1169
1 event 1170
56 progress_notify 1170
1 event 1171
63 progress_notify 1171
1 event 1172
1 progress_notify 1172
1 event 1173
60 progress_notify 1173
1 event 1174
2 progress_notify 1174
1 event 1175
217 progress_notify 1175
1 event 1176
6 progress_notify 1176
1 event 1177
41 progress_notify 1177
1 event 1178
1 progress_notify 1177
7 progress_notify 1178
1 event 1179
2 progress_notify 1179
1 event 1180
3 progress_notify 1180
1 event 1181
13 progress_notify 1181
1 event 1182
36 progress_notify 1182
1 event 1183
1 progress_notify 1182
4 progress_notify 1183
1 event 1184
66 progress_notify 1184
1 event 1185
1 progress_notify 1185
1 event 1186
62 progress_notify 1186
1 event 1187
2 progress_notify 1186
4 progress_notify 1187
1 event 1188
69 progress_notify 1188
1 event 1189
5 progress_notify 1189
1 event 1190
58 progress_notify 1190
1 event 1191
1 progress_notify 1191
1 event 1192
5 progress_notify 1191
65 progress_notify 1192
1 event 1193
77 progress_notify 1193
1 event 1194
6 progress_notify 1194
1 event 1195
53 progress_notify 1195
1 event 1196
236 progress_notify 1196
1 event 1197
5 progress_notify 1197
1 event 1198
61 progress_notify 1198
1 event 1199
2 progress_notify 1199
1 event 1200
3 progress_notify 1200
1 event 1201
1 event 1202
1 progress_notify 1201
15 progress_notify 1202
1 event 1203
2 progress_notify 1203
1 event 1204
26 progress_notify 1204
1 event 1205
6 progress_notify 1205
1 event 1206
58 progress_notify 1206
1 event 1207
85 progress_notify 1207
1 event 1208
6 progress_notify 1208
1 event 1209
64 progress_notify 1209
1 event 1210
5 progress_notify 1210
1 event 1211
66 progress_notify 1211
1 event 1212
3 progress_notify 1212
1 event 1213
69 progress_notify 1213
1 event 1214
1 event 1215
1 progress_notify 1214
76 progress_notify 1215
1 event 1216
1 event 1217
67 progress_notify 1217
1 event 1218
5 progress_notify 1218
1 event 1219
224 progress_notify 1219
1 event 1220
70 progress_notify 1220
1 event 1221
3 progress_notify 1221
1 event 1222
1 progress_notify 1221
8 progress_notify 1222
1 event 1223
5 progress_notify 1223
1 event 1224
5 progress_notify 1224
1 event 1225
8 progress_notify 1225
1 event 1226
2 progress_notify 1226
1 event 1227
5 progress_notify 1226
20 progress_notify 1227
1 event 1228
63 progress_notify 1228
1 event 1229
2 progress_notify 1229
1 event 1230
55 progress_notify 1230
1 event 1231
4 progress_notify 1231
1 event 1232
1 progress_notify 1231
61 progress_notify 1232
1 event 1233
2 progress_notify 1232
2 progress_notify 1233
1 event 1234
52 progress_notify 1234
1 event 1235
11 progress_notify 1235
1 event 1236
65 progress_notify 1236
1 event 1237
8 progress_notify 1237
1 event 1238
63 progress_notify 1238
1 event 1239
68 progress_notify 1239
1 event 1240
1 event 1241
165 progress_notify 1241
1 event 1242
2 progress_notify 1241
1 progress_notify 1242
1 event 1243
12 progress_notify 1242
36 progress_notify 1243
1 event 1244
6 progress_notify 1243
36 progress_notify 1244
1 event 1245
2 progress_notify 1244
75 progress_notify 1245
1 event 1246
13 progress_notify 1245
18 progress_notify 1246
1 event 1247
19 progress_notify 1247
1 event 1248
13 progress_notify 1248
1 event 1249
5 progress_notify 1248
10 progress_notify 1249
1 event 1250
8 progress_notify 1249
177 progress_notify 1250
1 event 1251
70 progress_notify 1251
1 event 1252
1 progress_notify 1251
139 progress_notify 1252
1 event 1253
1 progress_notify 1252
218 progress_notify 1253
1 event 1254
3 progress_notify 1254
1 event 1255
7 progress_notify 1255
1 event 1256
63 progress_notify 1256
1 event 1257
4 progress_notify 1257
1 event 1258
3 progress_notify 1258
1 event 1259
2 progress_notify 1259
1 event 1260
1 progress_notify 1260
1 event 1261
58 progress_notify 1261
1 event 1262
289 progress_notify 1262
1 event 1263
398 progress_notify 1263
1 event 1264
68 progress_notify 1264
1 event 1265
4 progress_notify 1265
1 event 1266
5 progress_notify 1266
1 event 1267
3 progress_notify 1267
1 event 1268
4 progress_notify 1268
1 event 1269
1 event 1270
1 progress_notify 1269
93 progress_notify 1270
1 event 1271
240 progress_notify 1271
1 event 1272
79 progress_notify 1272
1 event 1273
159 progress_notify 1273
1 progress_notify 1274
1 event 1274
186 progress_notify 1274
1 event 1275
151 progress_notify 1275
1 event 1276
1 progress_notify 1275
63 progress_notify 1276
1 event 1277
617 progress_notify 1277
1 event 1278
81 progress_notify 1278
1 event 1279
57 progress_notify 1279
1 event 1280
8 progress_notify 1280
1 event 1281
55 progress_notify 1281
1 event 1282
64 progress_notify 1282
1 event 1283
263 progress_notify 1283
1 event 1284
72 progress_notify 1284
1 event 1285
1 progress_notify 1284
46 progress_notify 1285
1 event 1286
2 progress_notify 1285
318 progress_notify 1286
1 event 1287
1 progress_notify 1286
4 progress_notify 1287
1 event 1288
3 progress_notify 1288
1 event 1289
64 progress_notify 1289
1 event 1290
58 progress_notify 1290
1 event 1291
531 progress_notify 1291
1 event 1292
290 progress_notify 1292
1 event 1293
1 progress_notify 1292
76 progress_notify 1293
1 event 1294
4 progress_notify 1294
1 event 1295
432 progress_notify 1295
1 event 1296
72 progress_notify 1296
1 event 1297
163 progress_notify 1297
1 event 1298
78 progress_notify 1298
1 event 1299
303 progress_notify 1299
1 event 1300
1 progress_notify 1299
55 progress_notify 1300
1 event 1301
158 progress_notify 1301
1 event 1302
353 progress_notify 1302
1 event 1303
1 progress_notify 1302
153 progress_notify 1303
1 event 1304
1 progress_notify 1303
6 progress_notify 1304
1 event 1305
645 progress_notify 1305
1 event 1306
227 progress_notify 1306
1 event 1307
14 progress_notify 1306
1 progress_notify 1307
1 event 1308
2 progress_notify 1307
58 progress_notify 1308
1 event 1309
47 progress_notify 1309
1 event 1310
1 progress_notify 1309
106 progress_notify 1310
1 event 1311
1 progress_notify 1310
19 progress_notify 1311
1 event 1312
12 progress_notify 1311
352 progress_notify 1312
1 event 1313
69 progress_notify 1313
1 event 1314
1 progress_notify 1313
470 progress_notify 1314
1 event 1315
46 progress_notify 1315
1 event 1316
273 progress_notify 1316
1 event 1317
241 progress_notify 1317
1 event 1318
236 progress_notify 1318
1 event 1319
1 progress_notify 1318
77 progress_notify 1319
1 event 1320
12 progress_notify 1320
1 event 1321
8 progress_notify 1321
1 event 1322
46 progress_notify 1322
1 event 1323
6 progress_notify 1322
274 progress_notify 1323
1 event 1324
133 progress_notify 1324
1 event 1325
76 progress_notify 1325
1 event 1326
292 progress_notify 1326
1 event 1327
1 progress_notify 1326
7 progress_notify 1327
1 event 1328
61 progress_notify 1328
1 event 1329
7 progress_notify 1329
1 event 1330
1 event 1331
4 progress_notify 1330
4 progress_notify 1331
1 event 1332
5 progress_notify 1332
1 event 1333
35 progress_notify 1333
1 event 1334
6 progress_notify 1334
1 event 1335
67 progress_notify 1335
1 event 1336
1 progress_notify 1336
1 event 1337
64 progress_notify 1337
1 event 1338
4 progress_notify 1338
1 event 1339
79 progress_notify 1339
1 event 1340
89 progress_notify 1340
1 event 1341
4 progress_notify 1341
1 event 1342
72 progress_notify 1342
1 event 1343
5 progress_notify 1343
1 event 1344
3 progress_notify 1343
59 progress_notify 1344
1 event 1345
1 progress_notify 1344
5 progress_notify 1345
1 event 1346
64 progress_notify 1346
1 event 1347
8 progress_notify 1347
1 event 1348
233 progress_notify 1348
1 event 1349
1 progress_notify 1349
1 event 1350
2 progress_notify 1349
71 progress_notify 1350
1 event 1351
2 progress_notify 1351
1 event 1352
1 event 1353
1 progress_notify 1353
1 event 1354
7 progress_notify 1353
16 progress_notify 1354
1 event 1355
3 progress_notify 1355
1 event 1356
34 progress_notify 1356
1 event 1357
4 progress_notify 1357
1 event 1358
1 progress_notify 1357
60 progress_notify 1358
1 event 1359
8 progress_notify 1359
1 event 1360
52 progress_notify 1360
1 event 1361
9 progress_notify 1361
1 event 1362
1 progress_notify 1361
33 progress_notify 1362
1 event 1363
3 progress_notify 1363
1 event 1364
44 progress_notify 1364
1 event 1365
14 progress_notify 1364
20 progress_notify 1365
1 event 1366
11 progress_notify 1365
1 progress_notify 1366
1 event 1367
17 progress_notify 1366
28 progress_notify 1367
1 event 1368
14 progress_notify 1367
27 progress_notify 1368
1 event 1369
21 progress_notify 1369
1 event 1370
1 event 1371
12 progress_notify 1369
2 progress_notify 1370
306 progress_notify 1371
1 event 1372
3 progress_notify 1372
1 event 1373
56 progress_notify 1373
1 event 1374
4 progress_notify 1374
1 event 1375
1 progress_notify 1375
1 event 1376
4 progress_notify 1375
4 progress_notify 1376
1 event 1377
1 event 1378
1 progress_notify 1377
10 progress_notify 1378
1 event 1379
1 progress_notify 1378
4 progress_notify 1379
1 event 1380
1 progress_notify 1379
1 event 1381
34 progress_notify 1381
1 event 1382
1 progress_notify 1381
4 progress_notify 1382
1 event 1383
66 progress_notify 1383
1 event 1384
6 progress_notify 1384
1 event 1385
2 progress_notify 1384
60 progress_notify 1385
1 event 1386
5 progress_notify 1386
1 event 1387
62 progress_notify 1387
1 event 1388
7 progress_notify 1388
1 event 1389
1 progress_notify 1388
66 progress_notify 1389
1 event 1390
61 progress_notify 1390
1 event 1391
2 progress_notify 1391
1 event 1392
66 progress_notify 1392
1 event 1393
3 progress_notify 1392
4 progress_notify 1393
1 event 1394
219 progress_notify 1394
1 event 1395
2 progress_notify 1395
1 event 1396
55 progress_notify 1396
1 event 1397
1 progress_notify 1397
1 event 1398
9 progress_notify 1398
1 event 1399
1 progress_notify 1398
6 progress_notify 1399
1 event 1400
1 progress_notify 1399
2 progress_notify 1400
1 event 1401
2 progress_notify 1401
1 event 1402
1 progress_notify 1401
31 progress_notify 1402
1 event 1403
3 progress_notify 1403
1 event 1404
72 progress_notify 1404
1 event 1405
63 progress_notify 1405
1 event 1406
5 progress_notify 1406
1 event 1407
52 progress_notify 1407
1 event 1408
1 progress_notify 1407
4 progress_notify 1408
1 event 1409
1 progress_notify 1408
64 progress_notify 1409
1 event 1410
79 progress_notify 1410
1 event 1411
5 progress_notify 1411
1 event 1412
66 progress_notify 1412
1 event 1413
6 progress_notify 1413
1 event 1414
57 progress_notify 1414
1 event 1415
2 progress_notify 1415
1 event 1416
229 progress_notify 1416
1 event 1417
6 progress_notify 1417
1 event 1418
63 progress_notify 1418
1 event 1419
2 progress_notify 1419
1 event 1420
1 progress_notify 1420
1 event 1421
4 progress_notify 1420
6 progress_notify 1421
1 event 1422
8 progress_notify 1422
1 event 1423
1 event 1424
43 progress_notify 1424
1 event 1425
60 progress_notify 1425
1 event 1426
4 progress_notify 1426
1 event 1427
61 progress_notify 1427
1 event 1428
1 progress_notify 1427
5 progress_notify 1428
1 event 1429
53 progress_notify 1429
1 event 1430
73 progress_notify 1430
1 event 1431
1 progress_notify 1430
6 progress_notify 1431
1 event 1432
9 progress_notify 1432
1 event 1433
51 progress_notify 1433
1 event 1434
1 progress_notify 1433
126 progress_notify 1434
1 event 1435
6 progress_notify 1435
1 event 1436
220 progress_notify 1436
1 event 1437
1 event 1438
1 progress_notify 1437
16 progress_notify 1438
1 event 1439
7 progress_notify 1438
20 progress_notify 1439
1 event 1440
6 progress_notify 1439
36 progress_notify 1440
1 event 1441
10 progress_notify 1441
1 event 1442
1 progress_notify 1441
7 progress_notify 1442
1 event 1443
4 progress_notify 1443
1 event 1444
3 progress_notify 1444
1 event 1445
4 progress_notify 1445
1 event 1446
29 progress_notify 1446
1 event 1447
7 progress_notify 1447
1 event 1448
2 progress_notify 1447
5 progress_notify 1448
1 event 1449
52 progress_notify 1449
1 event 1450
70 progress_notify 1450
1 event 1451
1 progress_notify 1450
5 progress_notify 1451
1 event 1452
57 progress_notify 1452
1 event 1453
3 progress_notify 1453
1 event 1454
63 progress_notify 1454
1 event 1455
2 progress_notify 1455
1 event 1456
54 progress_notify 1456
1 event 1457
2 progress_notify 1457
1 event 1458
1 progress_notify 1457
72 progress_notify 1458
1 event 1459
3 progress_notify 1459
1 event 1460
67 progress_notify 1460
1 event 1461
250 progress_notify 1461
1 event 1462
6 progress_notify 1462
1 event 1463
55 progress_notify 1463
1 event 1464
5 progress_notify 1464
1 event 1465
1 progress_notify 1464
6 progress_notify 1465
1 event 1466
5 progress_notify 1466
1 event 1467
1 event 1468
1 progress_notify 1467
10 progress_notify 1468
1 event 1469
2 progress_notify 1468
53 progress_notify 1469
1 event 1470
1 progress_notify 1469
35 progress_notify 1470
1 event 1471
1 progress_notify 1470
11 progress_notify 1471
1 event 1472
7 progress_notify 1471
13 progress_notify 1472
1 event 1473
1 event 1474
1 progress_notify 1473
16 progress_notify 1474
1 event 1475
10 progress_notify 1474
14 progress_notify 1475
1 event 1476
3 progress_notify 1475
1 event 1477
1 progress_notify 1476
3 progress_notify 1477
1 event 1478
15 progress_notify 1477
1 event 1479
1 event 1480
1 progress_notify 1478
16 progress_notify 1479
107 progress_notify 1480
1 event 1481
10 progress_notify 1480
148 progress_notify 1481
1 event 1482
6 progress_notify 1482
1 event 1483
7 progress_notify 1483
1 event 1484
232 progress_notify 1484
1 event 1485
4 progress_notify 1485
1 event 1486
69 progress_notify 1486
1 event 1487
2 progress_notify 1487
1 event 1488
6 progress_notify 1488
1 event 1489
3 progress_notify 1489
1 event 1490
5 progress_notify 1490
1 event 1491
12 progress_notify 1491
1 event 1492
36 progress_notify 1492
1 event 1493
7 progress_notify 1493
1 event 1494
59 progress_notify 1494
1 event 1495
12 progress_notify 1495
1 event 1496
56 progress_notify 1496
1 event 1497
1 progress_notify 1496
67 progress_notify 1497
1 event 1498
4 progress_notify 1498
1 event 1499
58 progress_notify 1499
1 event 1500
3 progress_notify 1500
1 event 1501
62 progress_notify 1501
1 event 1502
3 progress_notify 1501
5 progress_notify 1502
1 event 1503
61 progress_notify 1503
1 event 1504
4 progress_notify 1504
1 event 1505
61 progress_notify 1505
1 event 1506
2 progress_notify 1505
2 progress_notify 1506
1 event 1507
64 progress_notify 1507
1 event 1508
1 event 1509
1 progress_notify 1508
229 progress_notify 1509
1 event 1510
3 progress_notify 1510
1 event 1511
62 progress_notify 1511
1 event 1512
4 progress_notify 1512
1 event 1513
1 event 1514
1 progress_notify 1513
12 progress_notify 1514
1 event 1515
5 progress_notify 1515
1 event 1516
1 progress_notify 1516
1 event 1517
31 progress_notify 1517
1 event 1518
3 progress_notify 1518
1 event 1519
72 progress_notify 1519
1 event 1520
1 progress_notify 1520
1 event 1521
1 progress_notify 1520
70 progress_notify 1521
1 event 1522
4 progress_notify 1522
1 event 1523
62 progress_notify 1523
1 event 1524
1 progress_notify 1524
1 event 1525
1 progress_notify 1524
68 progress_notify 1525
1 event 1526
4 progress_notify 1526
1 event 1527
71 progress_notify 1527
1 event 1528
65 progress_notify 1528
1 event 1529
5 progress_notify 1529
1 event 1530
65 progress_notify 1530
1 event 1531
1 progress_notify 1531
1 event 1532
202 progress_notify 1532
1 event 1533
6 progress_notify 1533
1 event 1534
65 progress_notify 1534
1 event 1535
4 progress_notify 1535
1 event 1536
1 progress_notify 1535
9 progress_notify 1536
1 event 1537
14 progress_notify 1537
1 event 1538
1 progress_notify 1537
6 progress_notify 1538
1 event 1539
17 progress_notify 1539
1 event 1540
5 progress_notify 1540
1 event 1541
44 progress_notify 1541
1 event 1542
5 progress_notify 1542
1 event 1543
2 progress_notify 1542
55 progress_notify 1543
1 event 1544
5 progress_notify 1544
1 event 1545
74 progress_notify 1545
1 event 1546
6 progress_notify 1546
1 event 1547
1 progress_notify 1546
56 progress_notify 1547
1 event 1548
4 progress_notify 1548
1 event 1549
73 progress_notify 1549
1 event 1550
84 progress_notify 1550
1 event 1551
1 progress_notify 1551
1 event 1552
78 progress_notify 1552
1 event 1553
3 progress_notify 1553
3 progress_notify 1554
1 event 1554
214 progress_notify 1554
1 event 1555
4 progress_notify 1554
5 progress_notify 1555
1 event 1556
68 progress_notify 1556
1 event 1557
2 progress_notify 1558
1 event 1558
3 progress_notify 1558
1 event 1559
4 progress_notify 1558
1 event 1560
1 event 1561
4 progress_notify 1561
1 event 1562
42 progress_notify 1562
1 event 1563
69 progress_notify 1563
1 event 1564
1 progress_notify 1563
3 progress_notify 1564
1 event 1565
56 progress_notify 1565
1 event 1566
6 progress_notify 1566
1 event 1567
65 progress_notify 1567
1 event 1568
1 event 1569
69 progress_notify 1569
1 event 1570
1 progress_notify 1570
1 event 1571
84 progress_notify 1571
1 event 1572
4 progress_notify 1572
1 event 1573
31 progress_notify 1573
1 event 1574
13 progress_notify 1573
85 progress_notify 1574
1 event 1575
2 progress_notify 1574
11 progress_notify 1575
1 event 1576
8 progress_notify 1575
2 progress_notify 1576
1 event 1577
12 progress_notify 1576
38 progress_notify 1577
1 event 1578
10 progress_notify 1577
57 progress_notify 1578
1 event 1579
2 progress_notify 1578
34 progress_notify 1579
1 event 1580
18 progress_notify 1580
1 event 1581
4 progress_notify 1580
15 progress_notify 1581
1 event 1582
8 progress_notify 1581
9 progress_notify 1582
1 event 1583
11 progress_notify 1582
27 progress_notify 1583
1 event 1584
5 progress_notify 1583
20 progress_notify 1584
1 event 1585
5 progress_notify 1584
13 progress_notify 1585
1 event 1586
5 progress_notify 1585
111 progress_notify 1586
1 event 1587
6 progress_notify 1586
217 progress_notify 1587
1 event 1588
2 progress_notify 1588
1 event 1589
3 progress_notify 1589
1 event 1590
7 progress_notify 1590
1 event 1591
42 progress_notify 1591
1 event 1592
7 progress_notify 1592
1 event 1593
43 progress_notify 1593
1 event 1594
4 progress_notify 1594
1 event 1595
70 progress_notify 1595
1 event 1596
3 progress_notify 1596
1 event 1597
81 progress_notify 1597
1 event 1598
5 progress_notify 1598
1 event 1599
235 progress_notify 1599
1 event 1600
1 progress_notify 1599
5 progress_notify 1600
1 event 1601
62 progress_notify 1601
1 event 1602
5 progress_notify 1602
1 event 1603
6 progress_notify 1603
1 event 1604
2 progress_notify 1604
1 event 1605
3 progress_notify 1605
1 event 1606
2 progress_notify 1605
7 progress_notify 1606
1 event 1607
34 progress_notify 1607
1 event 1608
4 progress_notify 1608
1 event 1609
60 progress_notify 1609
1 event 1610
3 progress_notify 1609
4 progress_notify 1610
1 event 1611
55 progress_notify 1611
1 event 1612
1 progress_notify 1611
5 progress_notify 1612
1 event 1613
55 progress_notify 1613
1 event 1614
82 progress_notify 1614
1 event 1615
4 progress_notify 1615
1 event 1616
56 progress_notify 1616
1 event 1617
6 progress_notify 1617
1 event 1618
40800 progress_notify 1618
1 event 1619
17 progress_notify 1619

@serathius
Copy link
Contributor Author

Conclusion, manual watch notification are totally broken on unpatched etcd versions. We cannot depend on them at all, meaning that for this feature cannot be enabled by default without confirming that administrator runs etcd that has been patched.

Options for making ConsistentListFromCache default:

  • Wait for etcd v3.5 to get out of support window. ETA 2-4 years.
  • Automatically detect etcd version using /version endpoint. Problematic because it will be hard to confirm if watch and http request are sent do the same etcd instance.
  • Require administrators to manually provide etcd version to apiserver binary. Would only work if flag would be required, but that would be a breaking change.
  • Have etcd expose minimal binary version in API. There is a precedence for that, etcd downgrade API already exposes etcd cluster version defined as minimal minor etcd version. We could expand cluster version to include patch version and add another Method to grpc API to get cluster version via grpc. When opening a watch client could verify cluster version to decide whether it can trust watch progress notification.

@serathius
Copy link
Contributor Author

cc @ahrtr

@logicalhan
Copy link
Member

Conclusion, manual watch notification are totally broken on unpatched etcd versions. We cannot depend on them at all, meaning that for this feature cannot be enabled by default without confirming that administrator runs etcd that has been patched.

Options for making ConsistentListFromCache default:

  • Wait for etcd v3.5 to get out of support window. ETA 2-4 years.
  • Automatically detect etcd version using /version endpoint. Problematic because it will be hard to confirm if watch and http request are sent do the same etcd instance.
  • Require administrators to manually provide etcd version to apiserver binary. Would only work if flag would be required, but that would be a breaking change.
  • Have etcd expose minimal binary version in API. There is a precedence for that, etcd downgrade API already exposes etcd cluster version defined as minimal minor etcd version. We could expand cluster version to include patch version and add another Method to grpc API to get cluster version via grpc. When opening a watch client could verify cluster version to decide whether it can trust watch progress notification.

Option 5, @jpbetz is working on a component resource API, specifically so that we can support mixed version states of the apiserver, but it should be generic enough for component registration for any control-plane component. There could be a path where etcd registers itself to the apiserver along with the version.

@serathius
Copy link
Contributor Author

Option 6, we don't enable the feature by default for Beta in 1.29 and wait

@serathius serathius force-pushed the kep2340beta branch 3 times, most recently from 49686b6 to eb7c1ac Compare August 30, 2023 07:44
@jpbetz
Copy link
Contributor

jpbetz commented Oct 3, 2023

Are we still wanting to bump this to beta in 1.29? I just reviewed the list of options, it looks like the viable ones are down to either the etcd version flag or modifying etcd to make the version available in the API?

@serathius
Copy link
Contributor Author

Don't have capacity to finish the design nor to implement for 1.29

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 22, 2024
@serathius
Copy link
Contributor Author

Want to resurrect the KEP and move it to Beta before enhancement freeze on Friday 9th February.

@serathius serathius force-pushed the kep2340beta branch 3 times, most recently from b5b97eb to d950ae5 Compare February 6, 2024 14:12
@serathius
Copy link
Contributor Author

Based on my discussion with @wojtek-t, we want to move forward with making the feature enabled by default, with a safeguard that will disable the feature if users runs etcd version with broken progress notification.

Pushed the update to KEP, @jpbetz @deads2k @wojtek-t, please take another look.

endpoints have etcd version that includes the fix for [etcd-io/etcd#15220].

As we cannot break users we propose that during Beta, the failed condition will
cause kube-apiserver to fallback to reading to etcd (as before) and write a warning
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expanding a couple possible cases here

  1. version return not recognized or not etcd - in this case we should warn, but continue with whatever the current value of the featuregate is
  2. etcd too old, featuregate not explicitly specified, featuregate not locked to true - disable the featuregate
  3. etcd too old, featuregate not explicitly specified, featuregate locked to true - apiserver should fail to start
  4. etcd too old, featuregate explicitly enabled - apiserver should fail to start because the user wanted this featuregate and the intent cannot be fulfilled.
  5. etcd new enough - use whatever the current value of the featuregate is
  6. etcd does not respond (recall we don't have an ordering requirement) - in this case we should warn, but continue with whatever the current value of the featuregate is

I think that handles all the known cases, prevents the most common failure modes, and doesn't restrict cases where we aren't sure that there is a correctness problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How the first and sixth case differ. In both cases we cannot confirm that feature works. Would it be safer to disable the feature gate?

@serathius
Copy link
Contributor Author

@deads2k addressed the comments, please take another look.


Per request override should allow user to compare request results without
impacting other requests or requiring to redeploy whole cluster. The exact
details of override API will be clarified during API review. In healthy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC @deads2k wants the ListOptions field, but I would leave it to him to approve during API review.

@wojtek-t
Copy link
Member

wojtek-t commented Feb 7, 2024

Thanks - that all looks reasonable to me. The PRR looks good too.

/lgtm
/approve PRR

Still @deads2k approval from SIG level is needed.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 7, 2024
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 7, 2024
@serathius serathius force-pushed the kep2340beta branch 3 times, most recently from eb62839 to 9fb245d Compare February 7, 2024 15:57
Co-authored-by: David Eads <[email protected]>
@deads2k
Copy link
Contributor

deads2k commented Feb 7, 2024

Thanks for spending time digging into the details of this one with me.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 7, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, serathius, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 7, 2024
@k8s-ci-robot k8s-ci-robot merged commit 7329d8b into kubernetes:master Feb 7, 2024
4 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.30 milestone Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory lgtm "Looks good to me", indicates that a PR is ready to be merged. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants