Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store transactions in nanoBlocks #136

Closed
wants to merge 105 commits into from
Closed

Store transactions in nanoBlocks #136

wants to merge 105 commits into from

Conversation

patrick-ogrady
Copy link
Contributor

@patrick-ogrady patrick-ogrady commented Apr 6, 2023

Resolves: #130

  • start broadcasting chunk ownership before block building is done
  • use expiring map to clear chunks no longer being used (where accept height is the key)
  • add metrics for chunks produced, fetched, and chunk fetch errors
  • add chunk cache when persisting to disk
  • Must be able to respond with ShouldVerifyWithContext in ParseBlock, GetBlock, BuildBlock: https://github.com/ava-labs/avalanchego/blob/7d73b59cb4838d304387ea680b9cc4053b72620c/vms/rpcchainvm/vm_server.go#L457-L464
  • Make chunk push dynamic
  • push chunks after creation to some number of hosts (to avoid surge of requests on a single node) -> only keep in memory if send by preferred producer
  • send processing gossip immediately after building (instead of waiting normal 100ms)
  • add pruning mechanism
    • convert txID storage to a cache (avoid persisting on disk)
    • remove old chunks
  • if bootstrapping, store chunks on disk
  • consider generating root async (as soon as verify is called the first time) so engine just really "polls" -> should have no processing consensus messages but should still bubble votes
    • will need to handle case where root underlying verification changes
    • may be an easier stopgap to just generate root async and include state hash of block we are building on
  • allow for configurable gossip frequency
  • only cache chunk if it was non-empty
  • don't append handler to chunks using append (will change ID)
  • allow for only spinning up a single E2E network
  • if don't have chunk, return list of people that do
  • exit handle AppGossip as soon as possible (does handling block? If so, tx handling in app gossip should also be separated) -> async hardcoded at 3 per VM
    • so, we are handling 3 concurrently and the whole thing will end up dead-locking on handling
    • consider just allowing network router to set own concurrency limit? tx app gossip is probably almost always fully used...
  • run builder async -> can end up being a long-time offline per second in a small network (stuck verifying)
  • add metric for time waiting on chunks
  • increase async threads allowed in avalanche-ops config
  • advertise optimistically fetched chunks
  • fetch optimistic chunks concurrently
  • gossip chunk info to all connected peers
  • delete chunks from optimistic if accepted block (then add to just an ID cache to avoid re-fetching optimistic chunks)
  • store last X blocks of chunks in a map in storage instead of using LRU
  • put chunk fetch wait in block instead of on requester
  • what is throughput if just make build/async with 2MB blocks?
  • make sure mempool lock is called before submit (needed to allow block failure to be re-added)
  • gossip unconfirmed data with block hash/number so can easily identify info no longer useful
  • prefetch data concurrently
  • specify custom value for: throttler-outbound-at-large-alloc-size and throttler-inbound-at-large-alloc-size
  • Consider adding a nanoBlock height to make chunk manager simpler
  • txBlock and checkpointBlock

Alternative: only generate root every X blocks

@patrick-ogrady patrick-ogrady marked this pull request as draft April 6, 2023 02:04
@@ -182,6 +182,14 @@ func (vm *VM) processAcceptedBlocks() {
vm.warpManager.GatherSignatures(context.TODO(), tx.ID(), result.WarpMessage.Bytes())
}

// Update chunk store
for _, chunk := range b.FetchedChunks {
if err := vm.StoreChunk(chunk); err != nil {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: add chunk cache

return k
}

func (vm *VM) StoreChunk(chunk []byte) error {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: add async clearing of chunks (store block heights)

vm/vm.go Outdated
@@ -468,6 +478,7 @@ func (vm *VM) Shutdown(ctx context.Context) error {
}

// Shutdown other async VM mechanisms
vm.chunkManager.Done()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: stop chunk manager last

@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 19, 2023 23:58 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 19, 2023 23:58 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 19, 2023 23:58 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 19, 2023 23:58 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 19, 2023 23:58 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 03:37 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 03:37 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 03:37 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 03:37 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 03:37 — with GitHub Actions Inactive
@patrick-ogrady
Copy link
Contributor Author

TONS of SetPreference changes:
image

@patrick-ogrady
Copy link
Contributor Author

Add metric for innerVerify time and compare this to ParseToVerify

@patrick-ogrady
Copy link
Contributor Author

Don't re-trigger notify build until async done (disable timer)

@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 19:31 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 19:31 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 19:31 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 19:31 — with GitHub Actions Inactive
@patrick-ogrady patrick-ogrady temporarily deployed to long-ci April 20, 2023 19:31 — with GitHub Actions Inactive
@patrick-ogrady
Copy link
Contributor Author

patrick-ogrady commented Apr 20, 2023

Async overhead (~25% + spiky):
image

@patrick-ogrady
Copy link
Contributor Author

Simple example of fast and slow accept:

[04-20|21:58:31.972] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:609 parsed block {"id": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS", "height": 1733, "chunks": 16}
[04-20|21:58:32.028] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "DjWTVsFRKhARp4iGfR4epJxkh2aNzoyckfzJvSrL2BAU6MTsF", "size": 524146, "t": "75.109348ms"}
[04-20|21:58:32.034] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "cNGvxS86mbeLTZkzSYHh9nHuZmaxLkm3ahnYSysGpkq9dEh85", "size": 524146, "t": "80.990559ms"}
[04-20|21:58:32.038] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "tygoeorqtb7zgJsKq8eApY5oKN6Ro8a2s3mmVYTBzNqoNk7Ea", "size": 524146, "t": "84.961477ms"}
[04-20|21:58:32.043] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "GoswwoHEzALXqEjmP4sH1uK3hAoAoSRwauoJgN3ZbfdDLA5Wp", "size": 524146, "t": "90.339395ms"}
[04-20|21:58:32.048] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "UoQkm6ye88FNu1XvgxtNxNSufTuv5eTqunVB3jfa3omofs3Jd", "size": 524146, "t": "94.72755ms"}
[04-20|21:58:32.052] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "GtTB6AJNsZhuqaXkmYBY9gJvmCe6MaX36Gybnxo2BV7dyaV3u", "size": 524146, "t": "98.795897ms"}
[04-20|21:58:32.065] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "TYcGrHo4XCYqqRox2qPsne5Y81vMnxZ45CpWZnb5nb1xNwP3N", "size": 524146, "t": "112.247715ms"}
[04-20|21:58:32.070] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "2W873pPe2Zy1RwA46UfBwCDFwoxpkZNridBP9v3KCNRbQE7RDA", "size": 524146, "t": "117.067249ms"}
[04-20|21:58:32.076] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "qxvPT7xJ7JcgcjYXmM7PgJRsRfSVcoy7RdoHiRiC8gXubHASz", "size": 524146, "t": "122.766243ms"}
[04-20|21:58:32.081] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "ib4JX1Ay2T9Bj8eZiAsNXEMBg6HSyiufwpzbinZjCK6hj4fRZ", "size": 524146, "t": "128.328241ms"}
[04-20|21:58:32.087] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "25koSgVcnmSSt3RRoxD7K21AvzRSqAk6CFcDB3LGD73LsJGsyL", "size": 524146, "t": "134.000247ms"}
[04-20|21:58:32.093] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "TzVjbzDj82Ro9twBicdqmHpQhJW1c9FUosDMqhS3VrtHBvpaC", "size": 524146, "t": "139.840609ms"}
[04-20|21:58:32.102] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "xywLUhKmAhha3FQeo9m9pTy24xAcSkLrsLa6fiYW9GRPjEKdc", "size": 524146, "t": "149.058483ms"}
[04-20|21:58:32.105] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "yQAvnd7Nv1r8njoo4zF1kDh6M7uhBkRXg41Epa4WPFNRioX3E", "size": 524146, "t": "152.279838ms"}
[04-20|21:58:32.111] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "bVdWLjHHgZV884cBh8kVstHZGxwn4DroK7PkMLg9VbK1gkKgL", "size": 524146, "t": "157.917367ms"}
[04-20|21:58:32.116] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "XJYBMdxEsS4SVk9LeyGZ6qYfCENQ4V3ZuvQtxHJaDjC4sF26c", "size": 524146, "t": "162.810319ms"}
[04-20|21:58:32.116] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:185 fetched chunks {"height": 1733, "chunks": 16, "t": "143.439199ms"}
[04-20|21:58:32.162] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:657 verify context {"height": 1733, "unit price": 1, "block cost": 0}
[04-20|21:58:32.460] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/processor.go:61 block prefetch finished {"t": "295.406649ms"}
[04-20|21:58:32.524] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:817 inner verify completed {"height": 1733, "blkID": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS", "chunks": 16, "t": "367.225288ms", "root wait": "7.52498ms", "sig wait": "53.248245ms"}
[04-20|21:58:32.524] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:516 verified async {"height": 1733, "blkID": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS", "t": "367.280157ms"}
[04-20|21:58:32.536] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:107 verified block {"blkID": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS", "height": 1733, "txs": 37776, "state ready": true}
[04-20|21:58:32.537] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:648 starting async build {"parent": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS"}
[04-20|21:58:32.995] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:257 accepted block {"blkID": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS", "height": 1733, "txs": 37776, "size": 797, "units": 17830272, "dropped mempool txs": 0, "state ready": true}
[04-20|21:58:33.021] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:278 evicted chunks from memory {"n": 0, "processing": 0}
[04-20|21:58:33.054] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:198 block processed {"blkID": "iTqEwft37Bi2WKBTSCiE8mpd7buZipabcL5NYheyc2cvP5NKS", "height": 1733}
[04-20|21:58:33.055] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/builder.go:334 built block {"hght": 1734, "attempted": 37777, "added": 37776, "chunks": 16, "mempool size": 87192, "mempool lock wait": "7.439µs", "context": true, "state changes": 1000, "state operations": 113328, "prefetch wait": "207.801556ms", "repeat wait": "40.470908ms", "t": "511.625686ms"}
[04-20|21:58:33.055] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:669 built block async {"t": "517.506851ms"}
[04-20|21:58:33.055] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:635 found previously built block {"blkID": "ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py"}
{"level":"info","timestamp":"2023-04-20T21:58:33.076Z","logger":"cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain","caller":"proposervm/block.go:269","msg":"built block","blkID":"2VpbSzrDNfJsWsgAQsmFFEF4tLcjVFrF1hHpe27BR6qKsAPrxY","innerBlkID":"ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py","height":1734,"parentTimestamp":"2023-04-20T21:58:31.000Z","blockTimestamp":"2023-04-20T21:58:33.000Z"}
[04-20|21:58:33.076] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:481 skipping verification, already processed {"height": 1734, "blkID": "ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py"}
[04-20|21:58:33.097] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:107 verified block {"blkID": "ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py", "height": 1734, "txs": 37776, "state ready": true}
[04-20|21:58:33.098] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:648 starting async build {"parent": "ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py"}
[04-20|21:58:33.826] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:257 accepted block {"blkID": "ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py", "height": 1734, "txs": 37776, "size": 797, "units": 17830272, "dropped mempool txs": 0, "state ready": true}
[04-20|21:58:33.866] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:278 evicted chunks from memory {"n": 16, "processing": 0}
[04-20|21:58:33.869] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/builder.go:334 built block {"hght": 1735, "attempted": 37777, "added": 37776, "chunks": 16, "mempool size": 50360, "mempool lock wait": "9.104µs", "context": true, "state changes": 1000, "state operations": 113328, "prefetch wait": "367.691294ms", "repeat wait": "76.035307ms", "t": "763.603691ms"}
[04-20|21:58:33.869] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:669 built block async {"t": "770.888585ms"}
[04-20|21:58:33.870] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:635 found previously built block {"blkID": "26UpBk6NrdRo5CbX6Ag8BxdP8AtuiexukKK4YuzucZYUosj4CP"}
{"level":"info","timestamp":"2023-04-20T21:58:33.906Z","logger":"cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain","caller":"proposervm/block.go:269","msg":"built block","blkID":"J2LY3bGfrsRKTzcVcsTxUDXmBT3fvyFJP2zGSpqdqQ65AyXNJ","innerBlkID":"26UpBk6NrdRo5CbX6Ag8BxdP8AtuiexukKK4YuzucZYUosj4CP","height":1735,"parentTimestamp":"2023-04-20T21:58:33.000Z","blockTimestamp":"2023-04-20T21:58:33.000Z"}
[04-20|21:58:33.907] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:481 skipping verification, already processed {"height": 1735, "blkID": "26UpBk6NrdRo5CbX6Ag8BxdP8AtuiexukKK4YuzucZYUosj4CP"}
[04-20|21:58:33.927] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:107 verified block {"blkID": "26UpBk6NrdRo5CbX6Ag8BxdP8AtuiexukKK4YuzucZYUosj4CP", "height": 1735, "txs": 37776, "state ready": true}
[04-20|21:58:33.931] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:198 block processed {"blkID": "ygQBH98SoXZtEoRxnSNNnZivAiSJXPVG4WiDQASc1zvmuc1Py", "height": 1734}
[04-20|21:58:34.960] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:257 accepted block {"blkID": "26UpBk6NrdRo5CbX6Ag8BxdP8AtuiexukKK4YuzucZYUosj4CP", "height": 1735, "txs": 37776, "size": 797, "units": 17830272, "dropped mempool txs": 0, "state ready": true}
[04-20|21:58:34.991] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:278 evicted chunks from memory {"n": 16, "processing": 0}
[04-20|21:58:35.014] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:609 parsed block {"id": "Ep1o4dsgGzqZf9QArZT5BCkH4mKuzVEQqdUuAr7jdrMkwhpLj", "height": 1736, "chunks": 16}
[04-20|21:58:35.040] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:198 block processed {"blkID": "26UpBk6NrdRo5CbX6Ag8BxdP8AtuiexukKK4YuzucZYUosj4CP", "height": 1735}
[04-20|21:58:35.043] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "AuxGZqkzq9Bdyef3Ac3EVLdcWimnrmqAwnsCh9GotoezYMGNb", "size": 524146, "t": "48.979478ms"}
[04-20|21:58:35.047] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "xB6G7UoiMfAZE7e7VPjZn4LkLzxDgKF2mckaAQArMUMo4huWb", "size": 524146, "t": "52.684733ms"}
[04-20|21:58:35.121] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "2sCAtrHDAm7dhwbcSCAQZWE9rExWBf6eeCMBKgx4Qua8FkiER4", "size": 524146, "t": "126.874631ms"}
[04-20|21:58:35.128] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "QsGB6Eze9puAj1DDzCP9CXKP6aPg2WViHyVBQMS86Mqs3PSp4", "size": 524146, "t": "133.605252ms"}
[04-20|21:58:35.131] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "vNSxGLaNoaVvgCpCWjqDCrG91fDYENU6aho3rD35oCh5ijPvW", "size": 524146, "t": "136.23867ms"}
[04-20|21:58:35.136] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "DPn8323es9eK2DjaMnnoAtBPQk5PAzTxeBPRLaU2natz6QhyV", "size": 524146, "t": "142.091905ms"}
[04-20|21:58:35.143] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "2DhrA7pSYKGYGtZn9Rjo56Y1bvj6sCo4gdjEG1tAskeQnLiMJE", "size": 524146, "t": "147.987015ms"}
[04-20|21:58:35.148] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "LgsWTQzNuCTQJNRvRh2qkfzQST1W3uHEVPFpRUzawH6AzL6Gq", "size": 524146, "t": "153.905937ms"}
[04-20|21:58:35.155] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "e1RL9ucsTWTQCrvovVsBVtENfUh2bJMufUXDTQwXsDeZE3HSm", "size": 524146, "t": "160.0227ms"}
[04-20|21:58:35.161] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "2umz5W9iwV5vGS7DsN52ty1BG3xudn4NhYpmNWTf7HF69GJ2EG", "size": 524146, "t": "166.707938ms"}
[04-20|21:58:35.167] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "aiAkdHcQXm9JK6pZATZR1UTric4cB7TJpdg9MU3cyvTZ41FtG", "size": 524146, "t": "172.653532ms"}
[04-20|21:58:35.174] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "vb8821SWj74NHLFGy12JXSX1b1gwS2GKdUYfWevY9AHzkzP2i", "size": 524146, "t": "179.1604ms"}
[04-20|21:58:35.189] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "22bUGwmvuUUqu2wXAS1qTcMEaCykhqCHVURcWf8jvSFKU6x6jY", "size": 524146, "t": "194.719348ms"}
[04-20|21:58:35.195] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "2VS3JD7B4pF2ZM7JzzRHcbrfwHCxHjzCKifq4U7RNHJRjznMVm", "size": 524146, "t": "200.077803ms"}
[04-20|21:58:35.199] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "z1PSj7gVB5mZibutyo6odfqmUEioWjrogvWkjkXusTv3pTqSd", "size": 524146, "t": "204.17625ms"}
[04-20|21:58:35.204] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/chunk_manager.go:437 optimistically fetched chunk {"chunkID": "2JF2VS3hUMzkAwGfwuY4a8zoc6VjoKznYDDa6p1e6UgJ9g1wZT", "size": 524146, "t": "209.908255ms"}
[04-20|21:58:35.204] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:185 fetched chunks {"height": 1736, "chunks": 16, "t": "189.875581ms"}
[04-20|21:58:35.357] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:657 verify context {"height": 1736, "unit price": 1, "block cost": 0}
[04-20|21:58:35.663] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/processor.go:61 block prefetch finished {"t": "303.556336ms"}
[04-20|21:58:35.671] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:817 inner verify completed {"height": 1736, "blkID": "Ep1o4dsgGzqZf9QArZT5BCkH4mKuzVEQqdUuAr7jdrMkwhpLj", "chunks": 16, "t": "321.031369ms", "root wait": "6.07286ms", "sig wait": "5.171µs"}
[04-20|21:58:35.671] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> chain/block.go:516 verified async {"height": 1736, "blkID": "Ep1o4dsgGzqZf9QArZT5BCkH4mKuzVEQqdUuAr7jdrMkwhpLj", "t": "321.076717ms"}
[04-20|21:58:35.703] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:107 verified block {"blkID": "Ep1o4dsgGzqZf9QArZT5BCkH4mKuzVEQqdUuAr7jdrMkwhpLj", "height": 1736, "txs": 37776, "state ready": true}
[04-20|21:58:35.704] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/vm.go:648 starting async build {"parent": "Ep1o4dsgGzqZf9QArZT5BCkH4mKuzVEQqdUuAr7jdrMkwhpLj"}
[04-20|21:58:36.174] INFO <cBjTJsX7GbpMrTtxxnwLdLvSdJN3WJZieAUzMaY7ZD3NQeWVU Chain> vm/resolutions.go:257 accepted block {"blkID": "Ep1o4dsgGzqZf9QArZT5BCkH4mKuzVEQqdUuAr7jdrMkwhpLj", "height": 1736, "txs": 37776, "size": 797, "units": 17830272, "dropped mempool txs": 4963, "state ready": true}

@patrick-ogrady patrick-ogrady changed the title Store transactions in block.Chunks Store transactions in nanoBlocks Apr 25, 2023
patrick-ogrady added a commit that referenced this pull request May 11, 2023
* fix last computation in window

* add genesis block to cache

* nit

* init seen data structures earlier

* log proposer monitor refresh time

* track mempool size

* size-based gossip

* make NetworkSizeLimit a const

* add mode single

* nits

* port 159

* update DEVNET instructions

* add prometheus cmd

* update watch TPS

* overhaul spam script

* don't add potential block to check

* check if should build more frequently

* lint

* add lll exemption

* add helpful comment to window calc

* fix spam cmd

* lint

* add more gossip controls

* add gossip settings to scripts run

* fix cluster shutdown

* ensure client does not write concurrently to ws conn

* fix flaky startup

* avoid submitting duplicate txs

* update with new HYPERSDK_VERSION
@patrick-ogrady patrick-ogrady mentioned this pull request May 12, 2023
19 tasks
@github-actions
Copy link

This PR has become stale because it has been open for 30 days with no activity. Adding the lifecycle/frozen label will exempt this PR from future lifecycle events..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[storage + vm] Allow VM to delete proposervm blocks (or make blocks smaller)
2 participants