Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip verify on build #126

Merged
merged 18 commits into from
Apr 3, 2023
Merged

Skip verify on build #126

merged 18 commits into from
Apr 3, 2023

Conversation

patrick-ogrady
Copy link
Contributor

@patrick-ogrady patrick-ogrady commented Apr 2, 2023 •
edited
Loading

TODO

@patrick-ogrady patrick-ogrady mentioned this pull request Apr 3, 2023
Merged
12 tasks
@patrick-ogrady
Copy link
Contributor Author

Results after processing 2M transactions:

Allocated Memory:

Showing nodes accounting for 13234.98MB, 49.05% of 26981.44MB total
Dropped 385 nodes (cum <= 134.91MB)
Showing top 10 nodes out of 186
      flat  flat%   sum%        cum   cum%
 1970.96MB  7.30%  7.30%  1970.96MB  7.30%  github.com/ava-labs/hypersdk/tstate.(*TState).Insert
 1778.64MB  6.59% 13.90%  1778.64MB  6.59%  github.com/ava-labs/hypersdk/examples/tokenvm/storage.PrefixBalanceKey (inline)
 1744.99MB  6.47% 20.36%  1744.99MB  6.47%  github.com/ava-labs/avalanchego/x/merkledb.(*trieView).invalidateChildrenExcept
 1566.52MB  5.81% 26.17%  1566.52MB  5.81%  github.com/ava-labs/avalanchego/utils/wrappers.(*Packer).expand
 1506.25MB  5.58% 31.75%  1506.25MB  5.58%  golang.org/x/exp/maps.Clone[...] (inline)
 1039.64MB  3.85% 35.61%  2545.90MB  9.44%  github.com/ava-labs/avalanchego/x/merkledb.(*node).clone
  962.60MB  3.57% 39.17%  2496.27MB  9.25%  github.com/ava-labs/avalanchego/x/merkledb.(*trieView).calculateNodeIDsHelper
  958.12MB  3.55% 42.72%   958.12MB  3.55%  github.com/ava-labs/avalanchego/x/merkledb.newPath (inline)
  856.16MB  3.17% 45.90%  2049.57MB  7.60%  github.com/ava-labs/hypersdk/chain.(*Processor).Prefetch.func1
  851.10MB  3.15% 49.05%   851.10MB  3.15%  google.golang.org/grpc.(*parser).recvMsg

In Use Memory:

Showing nodes accounting for 1487.69MB, 85.90% of 1731.96MB total
Dropped 165 nodes (cum <= 8.66MB)
Showing top 10 nodes out of 91
      flat  flat%   sum%        cum   cum%
  521.27MB 30.10% 30.10%   521.27MB 30.10%  golang.org/x/exp/maps.Clone[...] (inline)
  336.05MB 19.40% 49.50%   857.32MB 49.50%  github.com/ava-labs/avalanchego/x/merkledb.(*node).clone (inline)
  131.02MB  7.56% 57.06%   131.02MB  7.56%  github.com/ava-labs/avalanchego/x/merkledb.newPath
  128.32MB  7.41% 64.47%   128.32MB  7.41%  github.com/ava-labs/avalanchego/x/merkledb.newChangeSummary
   80.17MB  4.63% 69.10%    80.17MB  4.63%  google.golang.org/protobuf/internal/impl.consumeBytesNoZero
   76.30MB  4.41% 73.51%    76.30MB  4.41%  github.com/ava-labs/hypersdk/vm.(*decisionRPCServer).Run
   64.55MB  3.73% 77.23%   947.71MB 54.72%  github.com/ava-labs/avalanchego/x/merkledb.(*trieView).applyChangedValuesToTrie
   61.01MB  3.52% 80.76%   137.02MB  7.91%  github.com/ava-labs/hypersdk/chain.UnmarshalTx
      47MB  2.71% 83.47%       47MB  2.71%  github.com/ava-labs/avalanchego/x/merkledb.(*trieView).recordValueChange
      42MB  2.43% 85.90%    43.02MB  2.48%  bytes.(*Buffer).grow

CPU:

Showing nodes accounting for 27.42s, 72.04% of 38.06s total
Dropped 581 nodes (cum <= 0.19s)
Showing top 10 nodes out of 208
      flat  flat%   sum%        cum   cum%
     5.35s 14.06% 14.06%      5.35s 14.06%  crypto/internal/edwards25519/field.addMul64 (inline)
     4.51s 11.85% 25.91%      4.51s 11.85%  syscall.syscall
     3.46s  9.09% 35.00%      3.46s  9.09%  runtime.usleep
     3.20s  8.41% 43.41%      3.20s  8.41%  runtime.kevent
     3.14s  8.25% 51.66%      6.85s 18.00%  crypto/internal/edwards25519/field.feSquareGeneric
     2.74s  7.20% 58.85%      6.64s 17.45%  crypto/internal/edwards25519/field.feMulGeneric
     1.69s  4.44% 63.29%      1.69s  4.44%  crypto/internal/edwards25519/field.carryPropagate
     1.26s  3.31% 66.61%      1.97s  5.18%  crypto/internal/edwards25519/field.(*Element).Add
     1.05s  2.76% 69.36%      1.05s  2.76%  runtime.pthread_cond_signal
     1.02s  2.68% 72.04%      1.47s  3.86%  crypto/internal/edwards25519/field.(*Element).Subtract

@patrick-ogrady patrick-ogrady linked an issue Apr 3, 2023 that may be closed by this pull request
[memory] Make tstate.Insert more efficient and use a sync.Pool in tokenvm.storage #127
Open
@patrick-ogrady patrick-ogrady merged commit 52f5a9c into testing-warp-with-avalanche-ops Apr 3, 2023
@patrick-ogrady patrick-ogrady deleted the skip-verify-on-build branch April 3, 2023 15:57
patrick-ogrady added a commit that referenced this pull request Apr 6, 2023
@gyuho @patrick-ogrady
automated devent deploy + skip verify on build (#94)
dc7eb5e 
* avalanche-ops.aws.md: initial commit for TokenVM

Signed-off-by: Gyuho Lee <[email protected]>

* specify --subnet-validate-period-in-days

Signed-off-by: Gyuho Lee <[email protected]>

* fix typo

Signed-off-by: Gyuho Lee <[email protected]>

* default

Signed-off-by: Gyuho Lee <[email protected]>

* update

Signed-off-by: Gyuho Lee <[email protected]>

* just use 16 day

Signed-off-by: Gyuho Lee <[email protected]>

* clarify aws credential requirements + node provision steps

Signed-off-by: Gyuho Lee <[email protected]>

* Read URIs from avalanche-ops (#116)

* read from avalanche-ops

* fix error

* linter

* change to ops

* progress

* remove install from source option on rust

* save progress

* more cleanup

* first pass at downloads

* one more tidbit

* tmp nit

* cleanup commands

* missing quote

* more command cleanup

* more path cleanup

* update instructions

* add correct input for import-ops command

* more tweaks to simplify instructions

* make second deploy optional

* add more clarify to balance check

* add improved default listner port

* more logging

* fix static lint

* add log viewing to DEVNETS

* add TODO to optimize unmarshaling

* remove unnecessary todo

* add prometheus instructions to devnets file

* wait for 10 runs to start issuing txs

* add more configs to custom genesis

* add metrics file auto-generation

* add useful queries

* make distribution faster

* make returns much faster

* add directions for rate limit

* add directions for RL override

* add instructions for CloudWatch metrics

* nits on README

* add data folder exemption

* add instruction for SSH in nodes

* stop running fio

* store full query

* add consensus isolation

* add additional tuning to config

* typo

* add support for continuous profiling

* add continuous profiler to config

* fix profiler dir

* add nolint

* DEVNETS: comment "instance-types" flags

Signed-off-by: Gyuho Lee <[email protected]>

* Skip verify on build (#126)

* insert height into state

* add merkledb metrics

* cleanup more metrics handling

* add additional arguments for config

* add HeightKey

* log lock wait

* use NodeID as random

* fix profiles

* gofumpt

* gofmt

* add more logs

* perform proper verify

* cache built block

* reduce memory usage of tstate

* improve memory efficiency of storage

* align devnets and scripts

* re-add proposer delay

* use default peer gossip size

* add note about custom instance types

* revert mainnet gossip changes

* more doc tweaks

* fixing incorrect filepath

* add avalanchego profiling

* add star history to readme

* fix duplicate image

* add warning about memory size

* ensure avalanchego has allocated cores

* track received txs

* disable compression an profiling

* Add metrics for root calculation and signature wait (#133)

* add verify metrics

* cleanup wording

* add more comments

* update resolution

* [mempool] make balance verification optional (#134)

* make balance checking optional

* config nit

---------

Signed-off-by: Gyuho Lee <[email protected]>
Co-authored-by: Patrick O'Grady <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[memory] Make tstate.Insert more efficient and use a sync.Pool in tokenvm.storage
1 participant
@patrick-ogrady