[chain] Warm Key Path

[patrick-ogrady]

Closes: #491
Blocked by: ava-labs/avalanchego#2167

We wait for 64 keys to be updated and then trigger a path prefetch async.

TODO

• handle rollbacks in tstate
• finish comment around PrefetchPath
• review naming

[patrick-ogrady]

TODO: better name?

[patrick-ogrady]

Maybe "Prefetcher"?

[patrick-ogrady]

🤔

[patrick-ogrady]

🤔 🤔

[patrick-ogrady]

Prefetch time can vary quite a bit by key:

prefetch completed [0 86 99 8 81 207 68 179 170 252 253 42 33 151 214 77 55 124 44 55 106 16 37 204 222 137 184 211 228 139 140 218 11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 7.126792ms
prefetch completed [0 253 124 230 245 115 96 144 162 235 87 31 144 249 93 162 236 212 130 15 56 246 84 197 64 115 69 25 10 147 246 39 243 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 62.5µs
prefetch completed [0 240 46 238 19 241 148 71 137 180 163 178 22 139 91 112 26 93 29 39 121 131 126 236 99 52 20 94 130 48 50 134 115 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 3.146417ms
prefetch completed [0 63 223 253 176 35 216 197 45 221 205 68 34 61 231 26 183 140 30 24 2 208 174 160 240 38 64 18 59 37 185 124 171 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 3.078708ms
prefetch completed [0 8 214 35 250 112 36 253 83 117 202 213 221 88 14 139 191 109 5 119 163 153 173 36 81 144 49 19 92 231 35 137 105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 31.917µs
prefetch completed [0 137 191 166 149 207 95 194 249 147 250 27 70 118 125 243 7 23 32 151 175 70 90 132 57 97 166 57 170 97 12 243 207 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 962.709µs
prefetch completed [0 45 209 233 224 120 161 216 173 117 73 205 158 31 246 134 251 34 75 165 188 247 67 208 187 171 58 209 6 188 90 213 54 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 12.167µs
prefetch completed [0 200 104 88 119 101 209 119 5 71 114 242 144 45 114 26 99 142 168 212 155 44 229 152 30 85 105 160 86 121 147 48 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 17.583µs
prefetch completed [0 222 129 87 21 87 149 230 14 39 229 134 106 212 32 122 64 95 111 41 79 193 115 28 182 125 173 88 21 225 54 116 210 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 3.184542ms
prefetch completed [0 221 172 35 189 161 91 248 19 22 250 59 250 226 16 57 50 23 126 215 169 201 188 125 137 78 82 123 150 188 215 76 31 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 2.151916ms
prefetch completed [0 65 74 53 84 202 195 230 253 96 48 139 8 75 35 209 37 94 254 50 201 24 7 227 25 132 182 50 42 10 50 23 139 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 12.583µs
prefetch completed [0 21 159 112 197 29 55 197 94 61 218 19 56 49 162 207 9 109 93 243 65 203 192 42 131 228 14 21 64 230 253 92 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 2.215208ms
prefetch completed [0 88 200 241 125 82 145 96 77 157 93 200 92 169 97 184 36 199 151 234 179 189 183 121 83 89 184 110 239 78 116 152 51 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 4.103541ms
prefetch completed [0 119 168 132 251 116 151 161 185 7 86 151 204 229 193 228 191 15 6 155 244 232 210 82 225 224 148 195 126 181 136 45 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] 3.132375ms

[patrick-ogrady]

Performance improves a bunch when doing synchronously (no goroutine -> when async, may end up conflicting with rootgen):

[patrick-ogrady]

Prefetch paths greatly conflicts with root generation:

[patrick-ogrady]

Best guess for slowdown...frequently updating cache is causing contention with prefetch or tons of extra memory allocations?

[github-actions]

This PR has become stale because it has been open for 30 days with no activity. Adding the lifecycle/frozen label will exempt this PR from future lifecycle events..

[github-actions]

This PR has become stale because it has been open for 30 days with no activity. Adding the lifecycle/frozen label will exempt this PR from future lifecycle events..

[patrick-ogrady]

Prefetch cache optimization was reverted here: https://github.com/ava-labs/avalanchego/pull/2576/files

[github-actions]

This PR has become stale because it has been open for 30 days with no activity. Adding the lifecycle/frozen label will exempt this PR from future lifecycle events..