Skip to content

Commit

Permalink
storage: fork-lift proposer-evaluated KV
Browse files Browse the repository at this point in the history 
Fixes #6290.

Add experimental proposer-evaluated KV gated behind the environment variable
`COCKROACH_PROPOSER_EVALUATED_KV`. When set to a truthy value, Raft proposals
are evaluated and the resulting RocksDB `WriteBatch` submitted to Raft along
with some auxiliary metadata. The result of the evaluation is only stored in
the pending command on the proposer, and returned to the waiting client after
the `WriteBatch` has been applied.

Introduce a natural failfast path for (most) proposals returning an error.
Instead of proposing, waiting for Raft, and only then receiving an error,
proposals which do not lead to a state change receive their error directly
when the proposal is evaluated, upstream of Raft. Only errors which still
want to persist data (for instance, `*TransactionRetryError` when intents
were laid down) go through the whole proposal, with the client receiving
the error after the associated `Batch` commits.

While proposer-evaluated KV is now ready for review, preliminary testing and
benchmarking, the current implementation is incomplete and incorrect:
- `Lease` acquisition is not special-cased, meaning that lease state may be
  clobbered freely when non-leaseholders propose a lease request based on stale
  data. This needs to be fixed but it also shows that we don't stress that
  scenario sufficiently in testing yet.
- Similarly, we don't check that commands can only apply under the lease that
  they were proposed (which is necessary).
- `CommandQueue` does not account for internal keys accessed by overlapping
  commands correctly (this is tracked in #10084), which in principle also lead
  to anomalies which should be exposed by testing and addressed.
  Instead, **every** command inserts a span that covers everything. Horrible
  for performance, great for correctness; #10084 needs to address this.
- `TestingCommandFilter` needs to be refactored to be an explicit interceptor
  for the pre-Raft stage of commands. Tests were fixed up enough to pass with
  proposer-evaluated KV as well, but it's possible that some tests don't test
  what they used to.

Benchmark results for non-proposer-evaluated-KV against `master` below.

**Note that all of them have the "correctness hack" in `(*Replica).beginCmds`
disabled; not doing that should give much worse results but we don't care about
those in practice**

We pay a small fee in allocations, but nothing appears outrageous. So from
a performance perspective, this PR seems safe to merge:

```
$ for br in exp-prop-kv masterdo git checkout $br && make bench COCKROACH_PROPOSER_EVALUATED_KV=false PKG=./sql TESTS='(Select|Insert|Update|Delete)1[0-9]*_Cockroach' TESTFLAGS='-benchmem -count 10' BENCHTIMEOUT=1h > $(git rev-parse --abbrev-ref HEAD); done

$ benchstat master exp-prop-kv
name                                old time/op    new time/op    delta
Select1_Cockroach-24                  79.8µs ± 2%    79.7µs ± 3%    ~     (p=0.912 n=10+10)
Insert1_Cockroach-24                   500µs ± 3%     520µs ± 5%  +4.12%  (p=0.001 n=10+10)
Insert10_Cockroach-24                  665µs ± 3%     667µs ± 3%    ~     (p=0.631 n=10+10)
Insert100_Cockroach-24                1.77ms ± 5%    1.78ms ± 4%    ~     (p=0.529 n=10+10)
Insert1000_Cockroach-24               11.9ms ± 3%    11.9ms ± 2%    ~     (p=1.000 n=10+10)
Update1_Cockroach-24                   727µs ± 2%     732µs ± 6%    ~     (p=0.971 n=10+10)
Update10_Cockroach-24                 1.14ms ± 4%    1.15ms ± 3%    ~     (p=0.211 n=9+10)
Update100_Cockroach-24                4.53ms ±12%    4.46ms ± 6%    ~     (p=0.796 n=10+10)
Update1000_Cockroach-24               33.8ms ± 4%    32.9ms ± 3%  -2.60%  (p=0.019 n=10+10)
Delete1_Cockroach-24                   674µs ± 2%     671µs ± 2%    ~     (p=0.796 n=10+10)
Delete10_Cockroach-24                  812µs ± 2%     828µs ± 2%  +2.01%  (p=0.003 n=9+10)
Delete100_Cockroach-24                2.35ms ± 1%    2.39ms ± 5%    ~     (p=0.094 n=9+9)
Delete1000_Cockroach-24               17.0ms ± 4%    17.0ms ± 2%    ~     (p=0.853 n=10+10)
InterleavedSelect1000_Cockroach-24    52.7ms ± 4%    52.8ms ± 5%    ~     (p=0.549 n=10+9)

name                                old alloc/op   new alloc/op   delta
Select1_Cockroach-24                  7.31kB ± 0%    7.31kB ± 0%  +0.03%  (p=0.000 n=10+9)
Insert1_Cockroach-24                  36.6kB ± 0%    40.0kB ± 0%  +9.25%  (p=0.000 n=10+10)
Insert10_Cockroach-24                 85.2kB ± 0%    90.5kB ± 0%  +6.14%  (p=0.000 n=10+10)
Insert100_Cockroach-24                 538kB ± 0%     543kB ± 0%  +0.96%  (p=0.000 n=10+9)
Insert1000_Cockroach-24               4.63MB ± 0%    4.64MB ± 0%  +0.09%  (p=0.002 n=10+10)
Update1_Cockroach-24                  62.2kB ± 0%    65.5kB ± 0%  +5.37%  (p=0.000 n=8+10)
Update10_Cockroach-24                  136kB ± 0%     142kB ± 0%  +3.95%  (p=0.000 n=10+9)
Update100_Cockroach-24                 863kB ± 0%     865kB ± 0%  +0.29%  (p=0.000 n=9+9)
Update1000_Cockroach-24               7.15MB ± 0%    7.15MB ± 0%    ~     (p=0.065 n=9+10)
Delete1_Cockroach-24                  54.0kB ± 0%    57.3kB ± 0%  +6.15%  (p=0.000 n=10+10)
Delete10_Cockroach-24                 64.4kB ± 0%    67.9kB ± 0%  +5.49%  (p=0.000 n=10+10)
Delete100_Cockroach-24                 173kB ± 0%     178kB ± 0%  +2.87%  (p=0.000 n=10+9)
Delete1000_Cockroach-24               1.19MB ± 0%    1.14MB ± 0%  -3.72%  (p=0.000 n=9+10)
InterleavedSelect1000_Cockroach-24     797kB ± 0%     797kB ± 0%    ~     (p=0.811 n=10+10)

name                                old allocs/op  new allocs/op  delta
Select1_Cockroach-24                     111 ± 0%       111 ± 0%    ~     (all equal)
Insert1_Cockroach-24                     386 ± 0%       395 ± 0%  +2.33%  (p=0.000 n=9+8)
Insert10_Cockroach-24                    612 ± 0%       622 ± 0%  +1.63%  (p=0.000 n=8+8)
Insert100_Cockroach-24                 2.63k ± 0%     2.64k ± 0%  +0.36%  (p=0.000 n=10+7)
Insert1000_Cockroach-24                22.6k ± 0%     22.6k ± 0%  +0.05%  (p=0.006 n=10+10)
Update1_Cockroach-24                     682 ± 0%       691 ± 0%  +1.32%  (p=0.000 n=10+10)
Update10_Cockroach-24                  1.01k ± 0%     1.02k ± 0%  +0.96%  (p=0.000 n=10+9)
Update100_Cockroach-24                 3.99k ± 0%     4.00k ± 0%  +0.22%  (p=0.000 n=9+9)
Update1000_Cockroach-24                31.6k ± 0%     31.6k ± 0%  +0.03%  (p=0.000 n=9+9)
Delete1_Cockroach-24                     568 ± 0%       577 ± 0%  +1.58%  (p=0.000 n=10+8)
Delete10_Cockroach-24                    693 ± 0%       703 ± 0%  +1.46%  (p=0.000 n=10+10)
Delete100_Cockroach-24                 1.83k ± 0%     1.84k ± 0%  +0.55%  (p=0.000 n=9+10)
Delete1000_Cockroach-24                12.7k ± 0%     12.7k ± 0%    ~     (p=0.954 n=10+10)
InterleavedSelect1000_Cockroach-24     3.11k ± 0%     3.11k ± 0%    ~     (p=0.586 n=10+7)
```

`master` vs proposer-evaluated KV. Takes a hit, as expected (prop-eval KV does
more work as it first preps a batch, serializes and unserializes it, then
commits (this is optimizable):
```
name                                old time/op    new time/op    delta
Select1_Cockroach-24                  78.1µs ± 2%    80.7µs ± 3%   +3.26%  (p=0.000 n=10+10)
Insert1_Cockroach-24                   507µs ± 5%     525µs ± 5%   +3.41%  (p=0.019 n=10+10)
Insert10_Cockroach-24                  662µs ± 2%     674µs ± 4%     ~     (p=0.075 n=10+10)
Insert100_Cockroach-24                1.74ms ± 5%    1.80ms ± 4%   +3.69%  (p=0.013 n=9+10)
Insert1000_Cockroach-24               11.7ms ± 3%    11.6ms ± 3%     ~     (p=0.436 n=10+10)
Update1_Cockroach-24                   727µs ± 4%     693µs ± 1%   -4.67%  (p=0.000 n=10+8)
Update10_Cockroach-24                 1.15ms ± 5%    1.14ms ± 6%     ~     (p=0.579 n=10+10)
Update100_Cockroach-24                4.42ms ± 6%    4.52ms ± 7%     ~     (p=0.190 n=10+10)
Update1000_Cockroach-24               32.9ms ± 3%    34.3ms ± 5%   +4.04%  (p=0.000 n=10+10)
Delete1_Cockroach-24                   675µs ± 3%     672µs ± 2%     ~     (p=0.579 n=10+10)
Delete10_Cockroach-24                  799µs ± 2%     827µs ± 3%   +3.46%  (p=0.000 n=8+10)
Delete100_Cockroach-24                2.38ms ± 3%    2.52ms ± 4%   +6.16%  (p=0.000 n=10+10)
Delete1000_Cockroach-24               16.7ms ± 4%    17.7ms ± 3%   +5.82%  (p=0.000 n=10+10)
InterleavedSelect1000_Cockroach-24    52.0ms ± 1%    51.9ms ± 2%     ~     (p=0.274 n=10+8)

name                                old alloc/op   new alloc/op   delta
Select1_Cockroach-24                  7.31kB ± 0%    7.31kB ± 0%   +0.03%  (p=0.000 n=8+9)
Insert1_Cockroach-24                  36.6kB ± 0%    39.5kB ± 0%   +7.89%  (p=0.000 n=9+10)
Insert10_Cockroach-24                 85.2kB ± 0%    91.2kB ± 0%   +7.04%  (p=0.000 n=10+10)
Insert100_Cockroach-24                 538kB ± 0%     559kB ± 0%   +3.91%  (p=0.000 n=10+10)
Insert1000_Cockroach-24               4.63MB ± 0%    4.80MB ± 0%   +3.54%  (p=0.000 n=10+9)
Update1_Cockroach-24                  62.2kB ± 0%    65.4kB ± 0%   +5.12%  (p=0.000 n=10+10)
Update10_Cockroach-24                  136kB ± 0%     143kB ± 0%   +5.09%  (p=0.000 n=9+8)
Update100_Cockroach-24                 863kB ± 0%     883kB ± 0%   +2.31%  (p=0.000 n=8+10)
Update1000_Cockroach-24               7.15MB ± 0%    7.33MB ± 0%   +2.53%  (p=0.000 n=10+10)
Delete1_Cockroach-24                  54.0kB ± 0%    56.8kB ± 0%   +5.15%  (p=0.000 n=10+10)
Delete10_Cockroach-24                 64.3kB ± 0%    68.7kB ± 0%   +6.74%  (p=0.000 n=10+10)
Delete100_Cockroach-24                 173kB ± 0%     194kB ± 0%  +11.72%  (p=0.000 n=10+10)
Delete1000_Cockroach-24               1.19MB ± 0%    1.26MB ± 0%   +5.98%  (p=0.000 n=9+10)
InterleavedSelect1000_Cockroach-24     797kB ± 0%     796kB ± 0%     ~     (p=0.095 n=10+9)

name                                old allocs/op  new allocs/op  delta
Select1_Cockroach-24                     111 ± 0%       111 ± 0%     ~     (all equal)
Insert1_Cockroach-24                     386 ± 0%       389 ± 0%   +0.78%  (p=0.000 n=8+9)
Insert10_Cockroach-24                    612 ± 0%       616 ± 0%   +0.65%  (p=0.000 n=8+8)
Insert100_Cockroach-24                 2.63k ± 0%     2.63k ± 0%   +0.13%  (p=0.000 n=9+6)
Insert1000_Cockroach-24                22.6k ± 0%     22.6k ± 0%   -0.17%  (p=0.000 n=10+10)
Update1_Cockroach-24                     682 ± 0%       685 ± 0%   +0.44%  (p=0.000 n=10+10)
Update10_Cockroach-24                  1.01k ± 0%     1.02k ± 0%   +0.39%  (p=0.000 n=9+8)
Update100_Cockroach-24                 3.99k ± 0%     3.99k ± 0%   +0.12%  (p=0.000 n=9+10)
Update1000_Cockroach-24                31.6k ± 0%     31.6k ± 0%   +0.04%  (p=0.000 n=8+7)
Delete1_Cockroach-24                     568 ± 0%       571 ± 0%   +0.65%  (p=0.000 n=10+10)
Delete10_Cockroach-24                    692 ± 0%       697 ± 0%   +0.64%  (p=0.000 n=10+10)
Delete100_Cockroach-24                 1.83k ± 0%     1.83k ± 0%   +0.31%  (p=0.000 n=10+9)
Delete1000_Cockroach-24                12.7k ± 0%     12.7k ± 0%     ~     (p=0.195 n=10+10)
InterleavedSelect1000_Cockroach-24     3.11k ± 0%     3.11k ± 0%     ~     (p=0.190 n=10+9)
```

Finally (just for completeness) this PR without proposer-eval'ed KV against the
PR with proposer-eval'ed KV.

```
name                                old time/op    new time/op    delta
Select1_Cockroach-24                  80.7µs ± 3%    80.9µs ± 3%    ~     (p=1.000 n=10+9)
Insert1_Cockroach-24                   525µs ± 5%     522µs ± 3%    ~     (p=0.436 n=10+10)
Insert10_Cockroach-24                  674µs ± 4%     672µs ± 5%    ~     (p=0.631 n=10+10)
Insert100_Cockroach-24                1.80ms ± 4%    1.82ms ± 6%    ~     (p=0.796 n=10+10)
Insert1000_Cockroach-24               11.6ms ± 3%    11.9ms ± 1%  +1.97%  (p=0.008 n=10+9)
Update1_Cockroach-24                   693µs ± 1%     743µs ± 2%  +7.29%  (p=0.000 n=8+10)
Update10_Cockroach-24                 1.14ms ± 6%    1.18ms ± 5%    ~     (p=0.075 n=10+10)
Update100_Cockroach-24                4.52ms ± 7%    4.61ms ± 8%    ~     (p=0.393 n=10+10)
Update1000_Cockroach-24               34.3ms ± 5%    33.3ms ± 6%    ~     (p=0.105 n=10+10)
Delete1_Cockroach-24                   672µs ± 2%     671µs ± 4%    ~     (p=0.631 n=10+10)
Delete10_Cockroach-24                  827µs ± 3%     814µs ± 1%  -1.59%  (p=0.016 n=10+8)
Delete100_Cockroach-24                2.52ms ± 4%    2.40ms ± 2%  -4.94%  (p=0.000 n=10+9)
Delete1000_Cockroach-24               17.7ms ± 3%    17.2ms ± 3%  -3.14%  (p=0.009 n=10+10)
InterleavedSelect1000_Cockroach-24    51.9ms ± 2%    51.8ms ± 3%    ~     (p=0.633 n=8+10)

name                                old alloc/op   new alloc/op   delta
Select1_Cockroach-24                  7.31kB ± 0%    7.31kB ± 0%    ~     (p=0.246 n=9+10)
Insert1_Cockroach-24                  39.5kB ± 0%    40.0kB ± 0%  +1.28%  (p=0.000 n=10+10)
Insert10_Cockroach-24                 91.2kB ± 0%    90.4kB ± 0%  -0.85%  (p=0.000 n=10+10)
Insert100_Cockroach-24                 559kB ± 0%     543kB ± 0%  -2.84%  (p=0.000 n=10+9)
Insert1000_Cockroach-24               4.80MB ± 0%    4.64MB ± 0%  -3.36%  (p=0.000 n=9+10)
Update1_Cockroach-24                  65.4kB ± 0%    65.5kB ± 0%  +0.24%  (p=0.000 n=10+10)
Update10_Cockroach-24                  143kB ± 0%     142kB ± 0%  -0.93%  (p=0.000 n=8+8)
Update100_Cockroach-24                 883kB ± 0%     865kB ± 0%  -1.97%  (p=0.000 n=10+8)
Update1000_Cockroach-24               7.33MB ± 0%    7.15MB ± 0%  -2.45%  (p=0.000 n=10+10)
Delete1_Cockroach-24                  56.8kB ± 0%    57.4kB ± 0%  +1.11%  (p=0.000 n=10+10)
Delete10_Cockroach-24                 68.7kB ± 0%    67.9kB ± 0%  -1.08%  (p=0.000 n=10+10)
Delete100_Cockroach-24                 194kB ± 0%     178kB ± 0%  -7.87%  (p=0.000 n=10+10)
Delete1000_Cockroach-24               1.26MB ± 0%    1.14MB ± 0%  -9.15%  (p=0.000 n=10+10)
InterleavedSelect1000_Cockroach-24     796kB ± 0%     797kB ± 0%  +0.05%  (p=0.002 n=9+8)

name                                old allocs/op  new allocs/op  delta
Select1_Cockroach-24                     111 ± 0%       111 ± 0%    ~     (all equal)
Insert1_Cockroach-24                     389 ± 0%       395 ± 0%  +1.54%  (p=0.000 n=9+8)
Insert10_Cockroach-24                    616 ± 0%       622 ± 0%  +0.97%  (p=0.000 n=8+8)
Insert100_Cockroach-24                 2.63k ± 0%     2.64k ± 0%  +0.24%  (p=0.000 n=6+9)
Insert1000_Cockroach-24                22.6k ± 0%     22.6k ± 0%  +0.22%  (p=0.000 n=10+10)
Update1_Cockroach-24                     685 ± 0%       691 ± 0%  +0.88%  (p=0.000 n=10+10)
Update10_Cockroach-24                  1.02k ± 0%     1.02k ± 0%  +0.69%  (p=0.000 n=8+7)
Update100_Cockroach-24                 3.99k ± 0%     4.00k ± 0%  +0.13%  (p=0.000 n=10+8)
Update1000_Cockroach-24                31.6k ± 0%     31.6k ± 0%  -0.02%  (p=0.002 n=7+8)
Delete1_Cockroach-24                     571 ± 0%       578 ± 0%  +1.10%  (p=0.000 n=10+10)
Delete10_Cockroach-24                    697 ± 0%       704 ± 0%  +0.98%  (p=0.000 n=10+10)
Delete100_Cockroach-24                 1.83k ± 0%     1.84k ± 0%  +0.28%  (p=0.000 n=9+10)
Delete1000_Cockroach-24                12.7k ± 0%     12.7k ± 0%    ~     (p=1.000 n=10+10)
InterleavedSelect1000_Cockroach-24     3.11k ± 0%     3.11k ± 0%    ~     (p=0.079 n=9+10)
```

Block writer `master`, `PR, false` `PR, true`:
```
master BenchmarkBlockWriter   	 1435603       	    208981.4 ns/op
PR wo BenchmarkBlockWriter   	 1508092       	    198933.8 ns/op
with BenchmarkBlockWriter   	 1374589       	    218254.9 ns/op

    1s: 4587.2/sec  4587.1/sec | 4349.7/sec  4349.4/sec | 4001.9/sec  4001.8/sec
    2s: 4224.0/sec  4406.8/sec | 4330.0/sec  4339.8/sec | 3967.9/sec  3984.9/sec
    3s: 4137.1/sec  4317.3/sec | 4185.0/sec  4288.4/sec | 3588.0/sec  3852.9/sec
    4s: 4015.0/sec  4242.0/sec | 4204.7/sec  4267.5/sec | 3695.2/sec  3813.5/sec
    5s: 4151.0/sec  4223.8/sec | 3983.2/sec  4210.7/sec | 3879.0/sec  3826.6/sec
    6s: 4155.8/sec  4212.5/sec | 3851.0/sec  4150.9/sec | 3914.0/sec  3841.2/sec
    7s: 4020.1/sec  4185.1/sec | 3895.2/sec  4114.4/sec | 3747.0/sec  3827.7/sec
    8s: 3831.0/sec  4140.9/sec | 3807.9/sec  4076.1/sec | 3632.0/sec  3803.3/sec
    9s: 3875.8/sec  4111.5/sec | 3914.8/sec  4058.2/sec | 3553.8/sec  3775.6/sec
   10s: 3884.2/sec  4088.8/sec | 3987.2/sec  4051.1/sec | 3604.2/sec  3758.4/sec
   11s: 4089.9/sec  4088.9/sec | 3911.0/sec  4038.4/sec | 3656.9/sec  3749.2/sec
   12s: 3985.2/sec  4080.3/sec | 3901.9/sec  4027.0/sec | 3707.0/sec  3745.7/sec
   13s: 4018.8/sec  4075.5/sec | 3804.0/sec  4009.9/sec | 3584.0/sec  3733.3/sec
   14s: 3984.1/sec  4069.0/sec | 4010.9/sec  4010.0/sec | 3439.9/sec  3712.3/sec
   15s: 3699.1/sec  4044.4/sec | 3665.1/sec  3987.0/sec | 3478.2/sec  3696.7/sec
   16s: 3912.7/sec  4036.2/sec | 3764.0/sec  3973.1/sec | 3616.9/sec  3691.7/sec
   17s: 3770.3/sec  4020.5/sec | 3747.0/sec  3959.8/sec | 3639.0/sec  3688.6/sec
   18s: 3997.9/sec  4019.3/sec | 3903.9/sec  3956.7/sec | 3588.1/sec  3683.1/sec
   19s: 3630.0/sec  3998.8/sec | 4115.8/sec  3965.0/sec | 3508.8/sec  3673.9/sec
   20s: 4005.9/sec  3999.2/sec | 3870.2/sec  3960.3/sec | 3759.1/sec  3678.1/sec
   21s: 3719.1/sec  3985.8/sec | 3618.1/sec  3944.0/sec | 3682.1/sec  3678.3/sec
   22s: 3722.0/sec  3973.8/sec | 3970.1/sec  3945.2/sec | 3674.9/sec  3678.2/sec
   23s: 3902.0/sec  3970.7/sec | 3441.9/sec  3923.3/sec | 3468.0/sec  3669.0/sec
   24s: 3638.0/sec  3956.9/sec | 3684.9/sec  3913.4/sec | 3631.0/sec  3667.5/sec
   25s: 6620.0/sec  4063.3/sec | 5308.0/sec  3969.2/sec | 3617.7/sec  3665.5/sec
   26s: 6123.1/sec  4142.5/sec | 6115.1/sec  4051.7/sec | 3225.5/sec  3648.6/sec
   27s: 6660.7/sec  4235.7/sec | 5974.1/sec  4122.8/sec | 5913.4/sec  3732.4/sec
   28s: 6209.2/sec  4306.2/sec | 6431.0/sec  4205.2/sec | 5624.0/sec  3800.0/sec
   29s: 6580.9/sec  4384.6/sec | 6189.8/sec  4273.7/sec | 6057.5/sec  3877.8/sec
   30s: 6016.1/sec  4438.9/sec | 6489.3/sec  4347.5/sec | 5491.4/sec  3931.6/sec
   31s: 6454.1/sec  4503.9/sec | 5094.9/sec  4371.6/sec | 6233.2/sec  4005.8/sec
   32s: 5101.9/sec  4522.6/sec | 6427.0/sec  4435.8/sec | 5396.7/sec  4049.3/sec
   33s: 6572.9/sec  4584.7/sec | 5962.2/sec  4482.0/sec | 5990.1/sec  4108.1/sec
   34s: 4667.2/sec  4587.1/sec | 6078.0/sec  4529.0/sec | 5562.0/sec  4150.8/sec
   35s: 6594.7/sec  4644.5/sec | 6301.0/sec  4579.6/sec | 5663.0/sec  4194.0/sec
   36s: 6504.2/sec  4696.1/sec | 5936.9/sec  4617.3/sec | 5877.0/sec  4240.8/sec
   37s: 6061.0/sec  4733.0/sec | 6466.9/sec  4667.3/sec | 5790.9/sec  4282.6/sec
   38s: 6444.1/sec  4778.0/sec | 6123.9/sec  4705.6/sec | 5912.2/sec  4325.5/sec
   39s: 6294.0/sec  4816.8/sec | 6248.1/sec  4745.1/sec | 5657.1/sec  4359.7/sec
   40s: 6782.5/sec  4866.0/sec | 5872.2/sec  4773.3/sec | 5854.9/sec  4397.0/sec
   41s: 9043.7/sec  4967.8/sec | 8606.9/sec  4866.8/sec | 5819.9/sec  4431.7/sec
   42s: 8676.5/sec  5056.1/sec | 8308.8/sec  4948.7/sec | 5291.8/sec  4452.2/sec
   43s: 8344.1/sec  5132.5/sec | 8517.3/sec  5031.7/sec | 6688.5/sec  4504.2/sec
   44s: 8777.1/sec  5215.4/sec | 8169.6/sec  5103.0/sec | 8453.8/sec  4594.0/sec
   45s: 8428.1/sec  5286.7/sec | 8810.7/sec  5185.4/sec | 7550.9/sec  4659.7/sec
   46s: 8874.7/sec  5364.7/sec | 8286.6/sec  5252.8/sec | 8502.2/sec  4743.2/sec
   47s: 8416.3/sec  5429.6/sec | 8596.9/sec  5323.9/sec | 6749.1/sec  4785.9/sec
   48s: 8860.8/sec  5501.1/sec | 8173.3/sec  5383.2/sec | 8306.8/sec  4859.2/sec
   49s: 8766.2/sec  5567.7/sec | 8589.9/sec  5448.7/sec | 8556.9/sec  4934.7/sec
   50s: 8214.9/sec  5620.6/sec | 8577.9/sec  5511.3/sec | 6479.5/sec  4965.5/sec
   51s: 8893.0/sec  5684.8/sec | 7346.0/sec  5547.2/sec | 8454.6/sec  5033.9/sec
   52s: 8326.0/sec  5735.5/sec | 8664.9/sec  5607.2/sec | 6760.0/sec  5067.1/sec
   53s: 8646.5/sec  5790.5/sec | 8677.1/sec  5665.1/sec | 8539.9/sec  5132.7/sec
   54s: 8873.8/sec  5847.5/sec | 7655.2/sec  5701.9/sec | 6050.0/sec  5149.6/sec
   55s: 8763.5/sec  5900.5/sec | 8749.0/sec  5757.3/sec | 8429.9/sec  5209.3/sec
   56s: 6465.9/sec  5910.6/sec | 8723.9/sec  5810.3/sec | 8601.0/sec  5269.8/sec
   57s: 3879.7/sec  5875.0/sec | 8659.0/sec  5860.3/sec | 3968.7/sec  5247.0/sec
   58s: 2531.1/sec  5817.4/sec | 1449.9/sec  5784.2/sec | 6217.5/sec  5263.7/sec
   59s: 4186.3/sec  5789.7/sec | 3661.2/sec  5748.2/sec | 8284.1/sec  5314.9/sec
  1m0s: 7975.0/sec  5826.2/sec | 1211.1/sec  5672.6/sec | 2871.8/sec  5274.2/sec
  1m1s: 6711.0/sec  5840.7/sec | 1431.9/sec  5603.1/sec | 4990.3/sec  5269.6/sec
  1m2s: 4748.7/sec  5823.0/sec | 7576.7/sec  5635.0/sec | 6227.2/sec  5285.0/sec
  1m3s: 5278.5/sec  5814.4/sec | 6698.6/sec  5651.8/sec | 8392.9/sec  5334.3/sec
  1m4s: 2612.9/sec  5764.4/sec | 8542.8/sec  5697.0/sec | 4390.2/sec  5319.6/sec
  1m5s: 4458.8/sec  5744.3/sec | 3849.8/sec  5668.6/sec |   15.0/sec  5238.0/sec
  1m6s:  778.0/sec  5669.1/sec |  973.0/sec  5597.5/sec | 6057.5/sec  5250.4/sec
  1m7s: 7397.8/sec  5694.9/sec | 6525.1/sec  5611.3/sec | 1583.9/sec  5195.7/sec
  1m8s: 4206.3/sec  5673.0/sec | 5286.7/sec  5606.5/sec | 1670.1/sec  5143.8/sec
  1m9s: 9329.0/sec  5726.0/sec |  145.0/sec  5527.4/sec | 7968.2/sec  5184.8/sec
 1m10s: 4207.0/sec  5704.3/sec | 8178.6/sec  5565.2/sec | 6377.1/sec  5201.8/sec
 1m11s:  467.1/sec  5630.5/sec | 5808.1/sec  5568.7/sec | 8555.2/sec  5249.0/sec
 1m12s: 3249.5/sec  5597.5/sec | 3945.1/sec  5546.1/sec | 1160.3/sec  5192.3/sec
 1m13s: 7547.9/sec  5624.2/sec |10219.6/sec  5610.1/sec |  753.0/sec  5131.5/sec
 1m14s:10479.6/sec  5689.8/sec | 8955.5/sec  5655.3/sec | 4614.0/sec  5124.5/sec
 1m15s: 2402.0/sec  5645.9/sec | 2536.8/sec  5613.8/sec |  818.0/sec  5067.1/sec
 1m16s:  438.0/sec  5577.4/sec | 6229.0/sec  5621.8/sec | 1130.0/sec  5015.3/sec
 1m17s: 3561.1/sec  5551.2/sec |   13.0/sec  5549.0/sec | 6255.3/sec  5031.4/sec
 1m18s: 3588.8/sec  5526.1/sec | 9045.8/sec  5593.8/sec | 9972.2/sec  5094.7/sec
 1m19s:10131.5/sec  5584.4/sec | 2488.7/sec  5554.5/sec | 7508.0/sec  5125.2/sec
 1m20s: 1915.0/sec  5538.5/sec | 4666.1/sec  5543.4/sec | 9947.6/sec  5185.5/sec
 1m21s: 1530.6/sec  5489.0/sec | 5259.1/sec  5539.9/sec | 8631.2/sec  5228.1/sec
 1m22s: 5218.2/sec  5485.7/sec | 8594.8/sec  5577.2/sec | 2574.8/sec  5195.7/sec
 1m23s: 1790.0/sec  5441.2/sec | 5812.0/sec  5580.0/sec | 6606.1/sec  5212.7/sec
 1m24s: 1077.0/sec  5389.3/sec | 2105.3/sec  5538.6/sec | 2788.3/sec  5183.8/sec
 1m25s: 9192.3/sec  5434.0/sec | 9297.8/sec  5582.9/sec | 7046.2/sec  5205.7/sec
 1m26s: 2206.0/sec  5396.5/sec | 6761.8/sec  5596.6/sec | 2829.2/sec  5178.1/sec
 1m27s: 8510.9/sec  5432.3/sec | 2130.1/sec  5556.7/sec | 7195.1/sec  5201.3/sec
 1m28s: 7004.2/sec  5450.1/sec | 8941.0/sec  5595.2/sec | 4929.9/sec  5198.2/sec
 1m29s:10563.6/sec  5507.6/sec | 6855.6/sec  5609.3/sec | 9913.1/sec  5251.2/sec
 1m30s: 1207.0/sec  5459.8/sec | 2984.9/sec  5580.2/sec | 5823.7/sec  5257.5/sec
 1m31s: 1888.0/sec  5420.5/sec | 9592.9/sec  5624.3/sec | 9037.5/sec  5299.1/sec
 1m32s: 6090.1/sec  5427.8/sec | 8368.9/sec  5654.1/sec | 1075.9/sec  5253.2/sec
 1m33s: 1273.0/sec  5383.1/sec | 9538.1/sec  5695.9/sec | 2864.3/sec  5227.5/sec
 1m34s: 1743.9/sec  5344.4/sec | 5547.4/sec  5694.3/sec | 5787.0/sec  5233.4/sec
 1m35s: 6407.9/sec  5355.6/sec | 1070.1/sec  5645.6/sec | 5361.6/sec  5234.8/sec
 1m36s: 9923.0/sec  5403.2/sec | 9663.3/sec  5687.5/sec | 4910.0/sec  5231.4/sec
 1m37s: 4074.8/sec  5389.5/sec | 6670.1/sec  5697.6/sec | 5113.5/sec  5230.2/sec
 1m38s: 9495.4/sec  5431.4/sec | 5261.2/sec  5693.1/sec | 8203.4/sec  5260.5/sec
 1m39s: 4153.8/sec  5418.5/sec |   22.0/sec  5635.9/sec | 4737.9/sec  5255.2/sec
 1m40s:  103.0/sec  5365.3/sec | 6668.1/sec  5646.2/sec | 7005.8/sec  5272.8/sec
 1m41s:  286.0/sec  5315.1/sec | 1530.9/sec  5605.4/sec | 5857.7/sec  5278.5/sec
 1m42s: 7923.5/sec  5340.6/sec | 9456.7/sec  5643.2/sec | 8831.6/sec  5313.4/sec
 1m43s:10307.9/sec  5388.8/sec | 8832.2/sec  5674.1/sec |  884.0/sec  5270.4/sec
 1m44s: 4427.9/sec  5379.6/sec | 6880.4/sec  5685.7/sec |  935.0/sec  5228.7/sec
 1m45s: 1116.0/sec  5339.0/sec |  264.0/sec  5634.1/sec | 4876.8/sec  5225.3/sec
 1m46s: 9290.0/sec  5376.3/sec |   85.0/sec  5581.8/sec | 6945.1/sec  5241.6/sec
 1m47s:10097.9/sec  5420.4/sec | 1739.3/sec  5545.9/sec | 4116.4/sec  5231.0/sec
 1m48s: 2953.8/sec  5397.6/sec | 7145.1/sec  5560.7/sec | 4073.2/sec  5220.3/sec
 1m49s: 2462.0/sec  5370.6/sec | 3741.0/sec  5544.0/sec | 6090.8/sec  5228.3/sec
 1m50s: 6493.2/sec  5380.8/sec | 6756.6/sec  5555.0/sec | 1162.9/sec  5191.4/sec
 1m51s: 5867.9/sec  5385.2/sec | 9214.1/sec  5588.0/sec | 9444.3/sec  5229.7/sec
 1m52s:   80.0/sec  5337.9/sec | 7630.3/sec  5606.2/sec | 8124.0/sec  5255.5/sec
 1m53s:   66.0/sec  5291.2/sec |   22.0/sec  5556.8/sec | 1259.9/sec  5220.1/sec
 1m54s:5144.2/sec  5289.9/sec | 8677.7/sec  5584.1/sec | 7879.3/sec  5243.5/sec
 1m55s: 10304.2/sec  5333.5/sec | 5515.0/sec  5583.5/sec |   15.0/sec  5198.0/sec
 1m56s: 5323.0/sec  5333.4/sec | 4051.9/sec  5570.3/sec | 8838.8/sec  5229.4/sec
 1m57s:   48.0/sec  5288.3/sec | 6286.7/sec  5576.5/sec |   35.0/sec  5185.0/sec
 1m58s: 5770.6/sec  5292.3/sec | 9040.7/sec  5605.8/sec |  126.0/sec  5142.1/sec
 1m59s: 3523.4/sec  5277.5/sec | 9035.0/sec  5634.6/sec | 2184.4/sec  5117.3/sec
  2m0s: 8119.7/sec  5301.2/sec | 7646.5/sec  5651.4/sec | 7637.9/sec  5138.3/sec
  2m1s: 4755.7/sec  5296.7/sec | 3334.8/sec  5632.3/sec | 1518.0/sec  5108.4/sec
  2m2s:10283.7/sec  5337.5/sec |  755.0/sec  5592.3/sec | 4609.2/sec  5104.3/sec
  2m3s: 5964.9/sec  5342.6/sec | 6245.0/sec  5597.6/sec | 5944.8/sec  5111.1/sec
  2m4s: 2230.7/sec  5317.5/sec | 7813.5/sec  5615.5/sec | 5123.2/sec  5111.2/sec
  2m5s: 5116.7/sec  5315.9/sec | 4813.5/sec  5609.0/sec | 6269.3/sec  5120.5/sec
  2m6s: 3105.0/sec  5298.4/sec | 5171.3/sec  5605.6/sec | 4086.5/sec  5112.3/sec
  2m7s: 6890.8/sec  5310.9/sec | 4732.9/sec  5598.7/sec | 3453.0/sec  5099.2/sec
  2m8s: 1377.0/sec  5280.2/sec | 2662.3/sec  5575.8/sec |   42.0/sec  5059.7/sec
  2m9s:   48.0/sec  5239.6/sec | 6369.8/sec  5581.9/sec | 5075.4/sec  5059.8/sec
 2m10s: 5518.2/sec  5241.8/sec | 5810.2/sec  5583.7/sec | 5711.2/sec  5064.8/sec
 2m11s: 9196.1/sec  5271.9/sec | 4373.2/sec  5574.4/sec | 1813.0/sec  5040.0/sec
 2m12s: 1666.9/sec  5244.6/sec | 7006.9/sec  5585.3/sec | 3148.4/sec  5025.7/sec
 2m13s: 6160.3/sec  5251.5/sec |    0.0/sec  5543.3/sec | 8122.3/sec  5048.9/sec
 2m14s: 7106.3/sec  5265.4/sec |    0.0/sec  5501.9/sec | 4414.4/sec  5044.2/sec
 2m15s: 4772.5/sec  5261.7/sec |    2.0/sec  5461.2/sec | 8962.5/sec  5073.2/sec
 2m16s: 4069.8/sec  5253.0/sec | 1009.0/sec  5428.5/sec |   33.0/sec  5036.2/sec
 2m17s: 5765.1/sec  5256.7/sec | 3482.2/sec  5414.3/sec |   41.0/sec  4999.7/sec
 2m18s: 8875.2/sec  5282.9/sec | 9462.9/sec  5443.6/sec | 2424.8/sec  4981.1/sec
 2m19s: 5182.5/sec  5282.2/sec | 9167.0/sec  5470.4/sec | 7051.9/sec  4996.0/sec
 2m20s: 1634.1/sec  5256.1/sec |10603.1/sec  5507.0/sec | 5391.7/sec  4998.8/sec
 2m21s:  323.0/sec  5221.2/sec | 1704.8/sec  5480.1/sec | 1377.0/sec  4973.1/sec
 2m22s: 5411.2/sec  5222.5/sec | 8832.5/sec  5503.7/sec | 2668.0/sec  4956.9/sec
 2m23s: 1981.0/sec  5199.8/sec |  712.0/sec  5470.2/sec | 6169.5/sec  4965.3/sec
 2m24s: 3647.6/sec  5189.0/sec |11296.1/sec  5510.6/sec | 3489.7/sec  4955.1/sec
 2m25s:  166.0/sec  5154.4/sec | 2499.8/sec  5489.9/sec | 5114.0/sec  4956.2/sec
 2m26s: 5974.4/sec  5160.0/sec | 6934.7/sec  5499.8/sec | 6929.5/sec  4969.7/sec
 2m27s: 1527.1/sec  5135.3/sec | 2523.4/sec  5479.5/sec |  731.9/sec  4940.9/sec
 2m28s:  359.0/sec  5103.0/sec |   54.0/sec  5442.8/sec |    0.0/sec  4907.5/sec
 2m29s: 8377.1/sec  5125.0/sec | 6178.9/sec  5447.8/sec |  680.0/sec  4879.1/sec
 2m30s: 4546.7/sec  5121.2/sec | 6312.5/sec  5453.5/sec | 8254.5/sec  4901.6/sec
 2m31s: 3130.9/sec  5108.0/sec | 9287.2/sec  5478.9/sec | 1600.0/sec  4879.8/sec
 2m32s: 1238.9/sec  5082.5/sec |   10.0/sec  5443.0/sec | 2549.9/sec  4864.4/sec
 2m33s: 8487.0/sec  5104.8/sec | 2116.2/sec  5421.2/sec | 9079.3/sec  4892.0/sec
 2m34s:   42.0/sec  5071.9/sec | 5653.3/sec  5422.7/sec | 1373.9/sec  4869.1/sec
 2m35s:   73.0/sec  5039.6/sec |10985.9/sec  5458.6/sec | 1715.9/sec  4848.8/sec
 2m36s: 8468.5/sec  5061.6/sec | 5165.2/sec  5456.7/sec | 3184.1/sec  4838.1/sec
 2m37s: 7946.5/sec  5080.0/sec | 1527.3/sec  5431.7/sec | 8996.7/sec  4864.6/sec
 2m38s: 2291.8/sec  5062.3/sec | 5349.0/sec  5431.2/sec | 5937.5/sec  4871.4/sec
 2m39s:10515.7/sec  5096.6/sec | 4281.7/sec  5424.0/sec | 7707.2/sec  4889.2/sec
 2m40s: 7624.0/sec  5112.4/sec | 3941.0/sec  5414.7/sec |   41.0/sec  4858.9/sec
 2m41s: 6627.3/sec  5121.8/sec | 7552.6/sec  5428.0/sec | 5506.6/sec  4862.9/sec
 2m42s:  779.0/sec  5095.0/sec | 6501.1/sec  5434.6/sec | 2935.9/sec  4851.1/sec
 2m43s: 6369.6/sec  5102.9/sec | 7616.9/sec  5448.0/sec |10298.3/sec  4884.5/sec
 2m44s: 1653.0/sec  5081.8/sec | 2503.4/sec  5430.0/sec | 5839.0/sec  4890.3/sec
 2m45s: 2058.1/sec  5063.5/sec |   33.0/sec  5397.3/sec |10947.2/sec  4927.0/sec
 2m46s: 9075.9/sec  5087.7/sec | 9044.7/sec  5419.3/sec | 4132.5/sec  4922.2/sec
 2m47s:10425.3/sec  5119.6/sec | 3219.8/sec  5406.1/sec | 8988.0/sec  4946.6/sec
 2m48s: 3264.3/sec  5108.6/sec | 1196.1/sec  5381.1/sec |10646.2/sec  4980.5/sec
 2m49s: 2728.0/sec  5094.5/sec | 4809.8/sec  5377.7/sec | 2066.0/sec  4963.2/sec
 2m50s:10931.3/sec  5128.8/sec | 6443.3/sec  5383.9/sec | 1873.0/sec  4945.1/sec
 2m51s: 6323.5/sec  5135.8/sec |  164.0/sec  5353.4/sec | 4180.2/sec  4940.6/sec
 2m52s: 1607.0/sec  5115.3/sec | 6430.5/sec  5359.7/sec | 2776.9/sec  4928.0/sec
 2m53s:  982.0/sec  5091.4/sec | 9842.2/sec  5385.6/sec |11246.5/sec  4964.5/sec
 2m54s: 7885.9/sec  5107.5/sec | 1940.9/sec  5365.8/sec | 3676.7/sec  4957.1/sec
 2m55s: 1905.9/sec  5089.2/sec |10557.4/sec  5395.5/sec | 9840.8/sec  4985.0/sec
 2m56s: 1937.1/sec  5071.3/sec | 7563.9/sec  5407.8/sec | 5596.0/sec  4988.5/sec
 2m57s: 1679.0/sec  5052.1/sec | 1272.1/sec  5384.4/sec | 4720.2/sec  4987.0/sec
 2m58s: 4255.4/sec  5047.6/sec | 9745.7/sec  5408.9/sec | 1282.2/sec  4966.2/sec
 2m59s: 8426.2/sec  5066.5/sec | 7192.5/sec  5418.9/sec |    0.0/sec  4938.4/sec
  3m0s: 1167.0/sec  5044.8/sec | 1870.0/sec  5399.2/sec | 2833.1/sec  4926.7/sec
  3m1s: 3522.1/sec  5036.4/sec |   48.0/sec  5369.6/sec |   56.0/sec  4899.8/sec
  3m2s: 3810.0/sec  5029.7/sec | 3947.4/sec  5361.8/sec | 6804.3/sec  4910.3/sec
  3m3s: 3109.1/sec  5019.2/sec | 4846.9/sec  5359.0/sec | 5008.5/sec  4910.8/sec
  3m4s:   57.0/sec  4992.2/sec |   44.0/sec  5330.1/sec | 4704.1/sec  4909.7/sec
  3m5s: 4918.1/sec  4991.8/sec | 8990.2/sec  5349.9/sec | 5397.7/sec  4912.3/sec
  3m6s: 9671.3/sec  5017.0/sec | 3887.1/sec  5342.0/sec |   55.0/sec  4886.2/sec
  3m7s:  130.0/sec  4990.9/sec | 8050.1/sec  5356.5/sec | 5212.2/sec  4888.0/sec
  3m8s:  199.0/sec  4965.4/sec | 8238.1/sec  5371.8/sec | 3365.8/sec  4879.9/sec
  3m9s:  836.0/sec  4943.5/sec | 9291.2/sec  5392.6/sec | 7874.5/sec  4895.7/sec
 3m10s: 1827.0/sec  4927.1/sec |   19.0/sec  5364.3/sec | 2081.9/sec  4880.9/sec
 3m11s: 4937.3/sec  4927.2/sec | 4660.5/sec  5360.6/sec |   41.0/sec  4855.6/sec
 3m12s:  336.0/sec  4903.3/sec | 5277.5/sec  5360.2/sec | 6003.1/sec  4861.5/sec
 3m13s: 2059.3/sec  4888.5/sec | 1565.1/sec  5340.5/sec | 1630.1/sec  4844.8/sec
 3m14s:11245.9/sec  4921.3/sec | 6013.7/sec  5344.0/sec | 1080.9/sec  4825.4/sec
 3m15s: 2425.9/sec  4908.5/sec | 2092.2/sec  5327.3/sec | 1754.1/sec  4809.6/sec
 3m16s:10817.1/sec  4938.6/sec | 5876.5/sec  5330.1/sec | 7598.4/sec  4823.9/sec
 3m17s: 1429.7/sec  4920.8/sec | 6989.3/sec  5338.5/sec | 8318.6/sec  4841.6/sec
 3m18s: 4555.1/sec  4919.0/sec |10488.4/sec  5364.5/sec | 4193.9/sec  4838.3/sec
 3m19s: 2661.3/sec  4907.6/sec | 3623.0/sec  5355.8/sec | 3804.2/sec  4833.1/sec
 3m20s:   17.0/sec  4883.2/sec | 5813.5/sec  5358.1/sec | 4302.8/sec  4830.5/sec
 3m21s: 2624.1/sec  4872.0/sec | 8798.3/sec  5375.2/sec | 5396.1/sec  4833.3/sec
 3m22s: 2060.1/sec  4858.0/sec | 4410.4/sec  5370.4/sec | 3108.5/sec  4824.8/sec
 3m23s: 8427.1/sec  4875.6/sec | 9463.4/sec  5390.6/sec | 4465.0/sec  4823.0/sec
 3m24s: 5281.8/sec  4877.6/sec | 3938.9/sec  5383.4/sec |  122.0/sec  4799.9/sec
 3m25s: 1801.8/sec  4862.6/sec | 10357.4/sec  5407.7/sec| 8710.2/sec  4819.0/sec
 3m26s: 5574.2/sec  4866.1/sec | 9170.2/sec  5426.0/sec | 2832.1/sec  4809.4/sec
 3m27s: 6970.0/sec  4876.2/sec |   62.0/sec  5400.1/sec | 5623.2/sec  4813.3/sec
 3m28s: 4561.1/sec  4874.7/sec | 4262.5/sec  5394.6/sec | 3071.6/sec  4804.9/sec
 3m29s: 5857.8/sec  4879.4/sec |  828.0/sec  5372.7/sec |    0.0/sec  4781.9/sec
 3m30s: 1756.0/sec  4864.5/sec | 3004.9/sec  5361.5/sec |    0.0/sec  4759.2/sec
 3m31s: 3234.0/sec  4856.8/sec | 3952.0/sec  5354.8/sec |    0.0/sec  4736.6/sec
 3m32s: 3143.0/sec  4848.7/sec | 3447.0/sec  5345.8/sec |    0.0/sec  4714.3/sec
 3m33s: 2955.2/sec  4839.8/sec | 2388.0/sec  5331.9/sec |    0.0/sec  4692.1/sec
 3m34s: 2099.6/sec  4827.0/sec | 5861.5/sec  5334.4/sec | 4770.4/sec  4692.5/sec
 3m35s: 4698.0/sec  4826.4/sec | 7248.3/sec  5343.3/sec | 8676.7/sec  4711.0/sec
 3m36s: 7993.7/sec  4841.1/sec | 3046.1/sec  5332.6/sec |   88.0/sec  4689.6/sec
 3m37s: 7314.4/sec  4852.5/sec | 8254.4/sec  5346.1/sec | 5004.5/sec  4691.1/sec
 3m38s: 9046.5/sec  4871.7/sec | 4847.6/sec  5343.8/sec | 2257.1/sec  4679.9/sec
 3m39s: 9369.4/sec  4892.3/sec | 8348.0/sec  5357.5/sec | 2752.0/sec  4671.1/sec
 3m40s: 4950.4/sec  4892.5/sec |   62.0/sec  5333.5/sec | 6111.4/sec  4677.7/sec
 3m41s:   61.0/sec  4870.7/sec | 2671.0/sec  5321.4/sec |  407.1/sec  4658.3/sec
 3m42s: 7474.4/sec  4882.4/sec | 2607.3/sec  5309.2/sec |   57.0/sec  4637.6/sec
 3m43s: 7187.7/sec  4892.7/sec | 5913.5/sec  5311.9/sec | 4693.3/sec  4637.9/sec
 3m44s: 7027.2/sec  4902.2/sec | 4433.3/sec  5308.0/sec | 2793.8/sec  4629.6/sec
 3m45s: 3752.3/sec  4897.1/sec |10589.6/sec  5331.5/sec | 8715.7/sec  4647.8/sec
 3m46s: 9824.8/sec  4918.9/sec | 4915.1/sec  5329.6/sec | 4384.0/sec  4646.6/sec
 3m47s:10184.8/sec  4942.1/sec | 8602.4/sec  5344.0/sec |  106.0/sec  4626.6/sec
 3m48s: 2577.9/sec  4931.8/sec | 1374.0/sec  5326.6/sec | 9405.0/sec  4647.6/sec
 3m49s: 8359.3/sec  4946.7/sec | 2558.1/sec  5314.5/sec | 3944.3/sec  4644.5/sec
 3m50s: 8608.3/sec  4962.7/sec |  966.0/sec  5295.6/sec | 5170.7/sec  4646.8/sec
 3m51s: 3402.7/sec  4955.9/sec | 5807.4/sec  5297.8/sec | 3573.7/sec  4642.1/sec
 3m52s: 5969.5/sec  4960.3/sec | 6293.8/sec  5302.1/sec | 9580.1/sec  4663.4/sec
 3m53s: 4655.6/sec  4959.0/sec |  330.1/sec  5280.8/sec |10790.9/sec  4689.7/sec
 3m54s: 4637.9/sec  4957.6/sec | 1899.0/sec  5266.4/sec | 5911.4/sec  4694.9/sec
 3m55s: 7502.2/sec  4968.4/sec | 5942.4/sec  5269.2/sec | 3858.4/sec  4691.4/sec
 3m56s: 3467.5/sec  4962.1/sec | 8172.9/sec  5281.5/sec | 8306.7/sec  4706.7/sec
 3m57s: 1986.1/sec  4949.5/sec |  500.0/sec  5261.4/sec | 2844.8/sec  4698.8/sec
 3m58s: 6150.3/sec  4954.5/sec | 6793.5/sec  5267.8/sec |    0.0/sec  4679.1/sec
 3m59s: 1767.9/sec  4941.2/sec | 9447.6/sec  5285.3/sec |    0.0/sec  4659.5/sec
  4m0s:    0.0/sec  4920.6/sec | 4337.1/sec  5281.3/sec | 5272.7/sec  4662.1/sec
  4m1s:    0.0/sec  4900.2/sec | 1792.2/sec  5266.9/sec | 2052.0/sec  4651.3/sec
  4m2s:    0.0/sec  4880.0/sec | 4982.0/sec  5265.7/sec |   40.0/sec  4632.2/sec
  4m3s:    0.0/sec  4859.9/sec | 8689.0/sec  5279.8/sec | 7462.2/sec  4643.8/sec
  4m4s: 5126.8/sec  4861.0/sec | 3963.1/sec  5274.4/sec | 6461.0/sec  4651.3/sec
  4m5s: 9416.8/sec  4879.6/sec | 4915.1/sec  5272.9/sec | 3797.6/sec  4647.8/sec
  4m6s: 5596.7/sec  4882.5/sec | 5439.8/sec  5273.6/sec | 6717.2/sec  4656.2/sec
  4m7s: 7137.0/sec  4891.6/sec | 2062.2/sec  5260.6/sec | 5986.5/sec  4661.6/sec
  4m8s: 2900.1/sec  4883.6/sec | 4199.6/sec  5256.3/sec | 5423.3/sec  4664.7/sec
  4m9s: 2352.7/sec  4873.4/sec | 5207.5/sec  5256.1/sec | 5621.3/sec  4668.5/sec
 4m10s: 7146.7/sec  4882.5/sec |   49.0/sec  5235.3/sec |   59.0/sec  4650.1/sec
 4m11s: 7591.3/sec  4893.3/sec | 3108.5/sec  5226.8/sec | 4744.5/sec  4650.5/sec
 4m12s: 3887.8/sec  4889.3/sec | 2972.8/sec  5217.9/sec | 5453.8/sec  4653.6/sec
 4m13s:    0.0/sec  4870.0/sec | 5755.6/sec  5220.0/sec | 5422.1/sec  4656.7/sec
 4m14s: 1462.2/sec  4856.6/sec | 1625.7/sec  5205.9/sec | 3522.7/sec  4652.2/sec
 4m15s: 4982.9/sec  4857.1/sec |  368.0/sec  5186.9/sec |10206.9/sec  4674.1/sec
 4m16s:10435.8/sec  4878.9/sec |11150.1/sec  5210.2/sec |  617.9/sec  4658.3/sec
 4m17s: 5842.5/sec  4882.6/sec | 2585.7/sec  5200.0/sec | 3725.3/sec  4654.7/sec
 4m18s: 3380.2/sec  4876.8/sec |10330.0/sec  5219.9/sec | 5500.1/sec  4658.0/sec
 4m19s: 6659.0/sec  4883.7/sec | 7819.7/sec  5229.9/sec | 6104.0/sec  4663.6/sec
 4m20s: 8776.2/sec  4898.6/sec |    0.0/sec  5209.8/sec | 4216.6/sec  4661.8/sec
 4m21s: 6155.8/sec  4903.5/sec | 6136.3/sec  5213.3/sec | 7253.9/sec  4671.8/sec
 4m22s: 3866.0/sec  4899.5/sec | 3681.5/sec  5207.5/sec |   29.0/sec  4654.1/sec
 4m23s: 8734.5/sec  4914.1/sec | 8508.6/sec  5220.0/sec | 2136.9/sec  4644.5/sec
 4m24s: 3044.3/sec  4907.0/sec |10632.0/sec  5240.5/sec |10642.7/sec  4667.2/sec
 4m25s: 5469.2/sec  4909.1/sec | 3237.3/sec  5233.0/sec | 3578.8/sec  4663.1/sec
 4m26s:  141.0/sec  4891.2/sec | 2676.2/sec  5223.3/sec |   84.0/sec  4645.9/sec
 4m27s: 9001.3/sec  4906.6/sec | 5368.3/sec  5223.9/sec | 9260.7/sec  4663.2/sec
 4m28s: 2234.9/sec  4896.6/sec | 2853.3/sec  5215.0/sec | 2918.7/sec  4656.6/sec
 4m29s: 9095.2/sec  4912.2/sec | 5955.1/sec  5217.8/sec | 2949.2/sec  4650.3/sec
 4m30s: 1032.0/sec  4897.9/sec | 2187.2/sec  5206.6/sec |10807.0/sec  4673.1/sec
 4m31s: 4271.4/sec  4895.5/sec | 4461.0/sec  5203.8/sec | 5457.1/sec  4676.0/sec
 4m32s: 1673.0/sec  4883.7/sec | 4818.8/sec  5202.4/sec | 6558.1/sec  4682.9/sec
 4m33s: 3209.4/sec  4877.6/sec | 5941.2/sec  5205.1/sec | 5019.6/sec  4684.1/sec
 4m34s:  288.0/sec  4860.8/sec | 2781.2/sec  5196.3/sec | 2193.2/sec  4675.1/sec
 4m35s: 3475.2/sec  4855.8/sec | 1115.3/sec  5181.4/sec | 7251.0/sec  4684.4/sec
 4m36s: 8060.5/sec  4867.4/sec | 1429.7/sec  5167.8/sec |  296.0/sec  4668.5/sec
 4m37s:   85.0/sec  4850.1/sec | 4989.5/sec  5167.2/sec | 5162.4/sec  4670.3/sec
 4m38s: 6838.7/sec  4857.3/sec | 1986.1/sec  5155.7/sec | 3189.4/sec  4665.0/sec
 4m39s: 3861.7/sec  4853.7/sec |   98.0/sec  5137.6/sec | 8847.6/sec  4680.0/sec
 4m40s: 8261.0/sec  4865.9/sec | 3867.6/sec  5133.1/sec | 2022.0/sec  4670.5/sec
 4m41s: 3799.3/sec  4862.1/sec | 4147.5/sec  5129.6/sec | 1072.0/sec  4657.7/sec
 4m42s: 1449.2/sec  4850.0/sec | 8232.9/sec  5140.6/sec | 8564.0/sec  4671.5/sec
 4m43s: 3237.0/sec  4844.3/sec | 2193.8/sec  5130.2/sec | 2522.2/sec  4663.9/sec
 4m44s: 2437.2/sec  4835.8/sec |    0.0/sec  5112.1/sec | 2106.9/sec  4654.9/sec
 4m45s: 5582.3/sec  4838.4/sec |    0.0/sec  5094.2/sec | 4277.9/sec  4653.6/sec
 4m46s: 4077.4/sec  4835.8/sec | 1699.9/sec  5082.3/sec | 6279.3/sec  4659.3/sec
 4m47s: 1518.0/sec  4824.2/sec | 9092.4/sec  5096.2/sec | 5723.4/sec  4663.0/sec
 4m48s: 5973.9/sec  4828.2/sec | 1382.1/sec  5083.4/sec | 2328.0/sec  4654.9/sec
 4m49s: 1477.9/sec  4816.6/sec | 3623.8/sec  5078.3/sec |10018.7/sec  4673.5/sec
 4m50s:   19.0/sec  4800.1/sec | 7984.6/sec  5088.3/sec | 6391.7/sec  4679.4/sec
 4m51s: 1320.0/sec  4788.1/sec | 1828.9/sec  5077.1/sec | 7006.3/sec  4687.4/sec
 4m52s: 9922.4/sec  4805.7/sec | 1778.0/sec  5065.8/sec |  507.0/sec  4673.1/sec
 4m53s: 5176.3/sec  4806.9/sec |   39.0/sec  5048.7/sec |  646.0/sec  4659.3/sec
 4m54s:  197.0/sec  4791.3/sec | 1036.1/sec  5035.0/sec | 1559.8/sec  4648.8/sec
 4m55s:   22.0/sec  4775.1/sec | 8240.7/sec  5045.9/sec |  196.0/sec  4633.7/sec
 4m56s:10154.1/sec  4793.3/sec | 9409.3/sec  5060.6/sec | 5429.9/sec  4636.4/sec
 4m57s: 2641.1/sec  4786.0/sec | 8583.7/sec  5072.5/sec |  880.0/sec  4623.7/sec
 4m58s:10420.8/sec  4804.9/sec |  692.0/sec  5057.8/sec |  935.0/sec  4611.3/sec
 4m59s:  599.9/sec  4790.9/sec |  789.0/sec  5043.5/sec |    0.0/sec  4595.9/sec
  5m0s: 3067.3/sec  4785.1/sec |   34.0/sec  5026.8/sec |  329.1/sec  4581.7/sec
  • Loading branch information
@tbg
tbg committed Nov 3, 2016
1 parent 7851548 commit 7ecb613
Show file tree
Hide file tree
Showing 19 changed files with 1,414 additions and 421 deletions.
6 changes: 5 additions & 1 deletion build/teamcity-acceptance.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
set -euxo pipefail
build/builder.sh make build
build/builder.sh make install

build/builder.sh go test -v -c -tags acceptance ./pkg/acceptance
cd pkg/acceptance
../../acceptance.test -nodes 3 -l ../../artifacts/acceptance -test.v -test.timeout 10m 2>&1 | go-test-teamcity

for prop_eval_kv in false true; do
COCKROACH_PROPOSER_EVALUATED_KV=${prop_eval_kv} ../../acceptance.test -nodes 3 -l ../../artifacts/acceptance -test.v -test.timeout 10m 2>&1 | go-test-teamcity
done
15 changes: 9 additions & 6 deletions build/teamcity-stress.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,14 @@ set -euxo pipefail

mkdir artifacts

exit_status=0
build/builder.sh make stress PKG="$PKG" GOFLAGS="${GOFLAGS-}" TAGS="${TAGS-}" TESTTIMEOUT=0 TESTFLAGS='-test.v' STRESSFLAGS='-maxtime 15m -maxfails 1 -stderr' 2>&1 | tee artifacts/stress.log || exit_status=$?
for prop_eval_kv in false true; do
exit_status=0
build/builder.sh make stress COCKROACH_PROPOSER_EVALUATED_KV="${prop_eval_kv}" PKG="$PKG" GOFLAGS="${GOFLAGS-}" TAGS="${TAGS-}" TESTTIMEOUT=0 TESTFLAGS='-test.v' STRESSFLAGS='-maxtime 15m -maxfails 1 -stderr' 2>&1 | tee "artifacts/stress.${prop_eval_kv}.log" || exit_status=$?

if [ $exit_status -ne 0 ]; then
build/builder.sh env GITHUB_API_TOKEN="$GITHUB_API_TOKEN" BUILD_VCS_NUMBER="$BUILD_VCS_NUMBER" github-post < artifacts/stress.log
fi;
if [ $exit_status -ne 0 ]; then
build/builder.sh env GITHUB_API_TOKEN="$GITHUB_API_TOKEN" BUILD_VCS_NUMBER="$BUILD_VCS_NUMBER" COCKROACH_PROPOSER_EVALUATED_KV="${prop_eval_kv}" github-post < "artifacts/stress.${prop_eval_kv}.log"
exit $exit_status
fi;
done

exit $exit_status
exit 0
4 changes: 3 additions & 1 deletion build/teamcity-test.sh
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/usr/bin/env bash
set -euxo pipefail
build/builder.sh make test TESTFLAGS='-v' 2>&1 | go-test-teamcity
for prop_eval_kv in false true; do
build/builder.sh make test COCKROACH_PROPOSER_EVALUATED_KV="${prop_eval_kv}" TESTFLAGS='-v' 2>&1 | go-test-teamcity
done

build/builder.sh env BUILD_VCS_NUMBER="$BUILD_VCS_NUMBER" TARGET=stress github-pull-request-make
6 changes: 4 additions & 2 deletions build/teamcity-testrace.sh
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/usr/bin/env bash
set -euxo pipefail
build/builder.sh make testrace TESTFLAGS='-v' 2>&1 | go-test-teamcity

build/builder.sh env BUILD_VCS_NUMBER="$BUILD_VCS_NUMBER" TARGET=stressrace github-pull-request-make
for prop_eval_kv in false true; do
build/builder.sh make testrace COCKROACH_PROPOSER_EVALUATED_KV="${prop_eval_kv}" TESTFLAGS='-v' 2>&1 | go-test-teamcity
build/builder.sh env COCKROACH_PROPOSER_EVALUATED_KV="${prop_eval_kv}" BUILD_VCS_NUMBER="$BUILD_VCS_NUMBER" TARGET=stressrace github-pull-request-make
done
16 changes: 14 additions & 2 deletions pkg/kv/txn_correctness_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -750,12 +750,18 @@ func (hv *historyVerifier) runHistory(
var wg sync.WaitGroup
wg.Add(len(txnMap))
retryErrs := make(chan *retryError, len(txnMap))
errs := make(chan error, 1) // only populated while buffer available

for i, txnCmds := range txnMap {
go func(i int, txnCmds []*cmd) {
if err := hv.runTxn(i, priorities[i], isolations[i], txnCmds, db, t); err != nil {
if re, ok := err.(*retryError); !ok {
t.Errorf("(%s): unexpected failure: %s", cmds, err)
reportErr := errors.Wrapf(err, "(%s): unexpected failure", cmds)
select {
case errs <- reportErr:
default:
t.Error(reportErr)
}
} else {
retryErrs <- re
}
Expand All @@ -765,7 +771,13 @@ func (hv *historyVerifier) runHistory(
}
wg.Wait()

// If we received a retry error, propagate the first one now.
// For serious errors, report the first one.
select {
case err := <-errs:
return err
default:
}
// In the absence of serious errors, report the first retry error, if any.
select {
case re := <-retryErrs:
return re
Expand Down
23 changes: 16 additions & 7 deletions pkg/storage/client_raft_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,11 @@ func TestRestoreReplicas(t *testing.T) {
}
}

// TODO(bdarnell): more aggressive testing here; especially with
// proposer-evaluated KV, what this test does is much less as it doesn't
// exercise the path in which the replica change fails at *apply* time (we only
// test the failfast path), in which case the replica change isn't even
// proposed.
func TestFailedReplicaChange(t *testing.T) {
defer leaktest.AfterTest(t)()

Expand Down Expand Up @@ -939,18 +944,17 @@ func TestStoreRangeCorruptionChangeReplicas(t *testing.T) {
syncutil.Mutex
store *storage.Store
}
sc.TestingKnobs.TestingCommandFilter = func(filterArgs storagebase.FilterArgs) *roachpb.Error {
sc.TestingKnobs.TestingApplyFilter = func(filterArgs storagebase.ApplyFilterArgs) *roachpb.Error {
corrupt.Lock()
defer corrupt.Unlock()

if corrupt.store == nil || filterArgs.Sid != corrupt.store.StoreID() {
if corrupt.store == nil || filterArgs.StoreID != corrupt.store.StoreID() {
return nil
}

if filterArgs.Req.Header().Key.Equal(roachpb.Key("boom")) {
return roachpb.NewError(storage.NewReplicaCorruptionError(errors.New("test")))
}
return nil
return roachpb.NewError(
storage.NewReplicaCorruptionError(errors.New("boom")),
)
}

// Don't timeout raft leader.
Expand Down Expand Up @@ -998,7 +1002,7 @@ func TestStoreRangeCorruptionChangeReplicas(t *testing.T) {
return err
})

args := putArgs(roachpb.Key("boom"), []byte("value"))
args := putArgs(roachpb.Key("any write"), []byte("should mark as corrupted"))
if _, err := client.SendWrapped(context.Background(), rg1(store0), &args); err != nil {
t.Fatal(err)
}
Expand Down Expand Up @@ -1173,6 +1177,11 @@ func TestStoreRangeDownReplicate(t *testing.T) {

// TestChangeReplicasDescriptorInvariant tests that a replica change aborts if
// another change has been made to the RangeDescriptor since it was initiated.
//
// TODO(tschottdorf): If this test is flaky because the snapshot count does not
// increase, it's likely because with proposer-evaluated KV, less gets proposed
// and so sometimes Raft discards the preemptive snapshot (though we count that
// case in stats already) or doesn't produce a Ready.
func TestChangeReplicasDescriptorInvariant(t *testing.T) {
defer leaktest.AfterTest(t)()
mtc := startMultiTestContext(t, 3)
Expand Down
12 changes: 11 additions & 1 deletion pkg/storage/client_split_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1082,6 +1082,9 @@ func TestSplitSnapshotRace_SnapshotWins(t *testing.T) {
// non-atomically with respect to the reads (and in particular their update of
// the timestamp cache), then some of them may not be reflected in the
// timestamp cache of the new range, in which case this test would fail.
//
// TODO(tschottdorf): hacks around #10084, see usage of
// ProposerEvaluatedKVEnabled() within.
func TestStoreSplitTimestampCacheReadRace(t *testing.T) {
defer leaktest.AfterTest(t)()
splitKey := roachpb.Key("a")
Expand All @@ -1091,6 +1094,11 @@ func TestStoreSplitTimestampCacheReadRace(t *testing.T) {
}

getContinues := make(chan struct{})
if storage.ProposerEvaluatedKVEnabled() {
// TODO(tschottdorf): because of command queue hack (would deadlock
// otherwise); see #10084.
close(getContinues)
}
var getStarted sync.WaitGroup
storeCfg := storage.TestStoreConfig(nil)
storeCfg.TestingKnobs.DisableSplitQueue = true
Expand All @@ -1101,7 +1109,9 @@ func TestStoreSplitTimestampCacheReadRace(t *testing.T) {
if st == nil || !st.LeftDesc.EndKey.Equal(splitKey) {
return nil
}
close(getContinues)
if !storage.ProposerEvaluatedKVEnabled() {
close(getContinues)
}
} else if filterArgs.Req.Method() == roachpb.Get &&
bytes.HasPrefix(filterArgs.Req.Header().Key, splitKey.Next()) {
getStarted.Done()
Expand Down
4 changes: 4 additions & 0 deletions pkg/storage/helpers_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -227,3 +227,7 @@ func GetGCQueueTxnCleanupThreshold() time.Duration {
func (nl *NodeLiveness) StopHeartbeat() {
close(nl.stopHeartbeat)
}

func ProposerEvaluatedKVEnabled() bool {
return propEvalKV
}
Loading

0 comments on commit 7ecb613

Please sign in to comment.