Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

Disk Usage fast growth #11713

Open
jinglics opened this issue May 15, 2020 · 10 comments
Open

Disk Usage fast growth #11713

jinglics opened this issue May 15, 2020 · 10 comments

Comments

@jinglics
Copy link

jinglics commented May 15, 2020 •
edited
Loading

  • OpenEthereum version: Parity Ethereum Client.
    version Parity-Ethereum/v2.7.2-stable-2662d19-20200206/x86_64-unknown-linux-gnu/rustc1.41.0

  • Operating system: Linux

  • Installation: homebrew / one-line installer / built from source

  • Fully synchronized: yes

  • Network: ethereum

  • Restarted: yes

I run parity to get the ethereum mainnet and it synced in more one day. Now it caught up with the mainnet. However, after I start to use the node, running json rpc request and so on, the node cost disk rapidly, almost grows 1GB in 5 mins.

Here is my command to run parity:
parity --cache-size 4096 --db-compaction ssd --base-path "/.eth" --jsonrpc-apis "web3,eth,pubsub,net,parity,private,parity_pubsub,traces,rpc,parity_transactions_pool,personal" --jsonrpc-interface 0.0.0.0 --jsonrpc-cors * --enable-signing-queue true

Here is the log:
May 15 13:10:22: 2020-05-15 13:10:22 UTC #3110666 18/25 peers 221 MiB chain 138 MiB db 0 bytes queue 36 MiB sync RPC: 0 conn, 4 req/s, 25 µs
May 15 13:10:26: 2020-05-15 13:10:26 UTC 37 -> GetNodeData: item 192/384 – slow state fetch for hash 0xb8904769acb3fc0902eb03496c9329c5c4c319185754b2ecf3b7004cc0b4ed76; took Instant { tv_sec: 887827, tv_nsec: 236457208 }
May 15 13:10:31: 2020-05-15 13:10:31 UTC Imported #10070924 0xd45c…22f8 (184 txs, 9.96 Mgas, 3307 ms, 35.58 KiB)
May 15 13:10:33: 2020-05-15 13:10:33 UTC 37 -> GetNodeData: item 10/384 – slow state fetch for hash 0xda4f4ed2b1bb8db5d0e9bb55e8a6e70258e64fe8021228c5e8f295896f576402; took Instant { tv_sec: 887834, tv_nsec: 232453585 }
May 15 13:10:42: 2020-05-15 13:10:42 UTC Imported #10070926 0x08e5…6965 (0 txs, 0.00 Mgas, 16 ms, 0.52 KiB)
May 15 13:10:51: 2020-05-15 13:10:51 UTC Imported #10070921 0x52b7…0dc5 (143 txs, 9.97 Mgas, 3419 ms, 22.28 KiB)
May 15 13:10:54: 2020-05-15 13:10:54 UTC #3116924 20/25 peers 221 MiB chain 138 MiB db 0 bytes queue 39 MiB sync RPC: 0 conn, 35 req/s, 37 µs
May 15 13:10:59: 2020-05-15 13:10:59 UTC 37 -> GetNodeData: item 69/384 – slow state fetch for hash 0x7227bb7e5ee96e544a38736163de276d232b6afc8be0b1b7f42381110596dbed; took Instant { tv_sec: 887860, tv_nsec: 352783122 }
May 15 13:11:04: 2020-05-15 13:11:04 UTC 37 -> GetNodeData: item 3/384 – slow state fetch for hash 0xb70290b58b5e1e8b80ca3ce87a65ba915e66273c190ba41b51e77d07d2d42e41; took Instant { tv_sec: 887865, tv_nsec: 124825721 }
May 15 13:11:06: 2020-05-15 13:11:06 UTC Imported #10070927 0xe264…a28c (169 txs, 9.99 Mgas, 4350 ms, 25.15 KiB)
May 15 13:11:13: 2020-05-15 13:11:13 UTC 37 -> GetNodeData: item 29/384 – slow state fetch for hash 0xd3caa42ae5b83d3b609f688f386648d4f5b441eae64865ebd2f45da52de6eff2; took Instant { tv_sec: 887874, tv_nsec: 680094813 }
May 15 13:11:30: 2020-05-15 13:11:30 UTC Imported #10070928 0xbca1…98b2 (172 txs, 9.96 Mgas, 4374 ms, 25.99 KiB)
May 15 13:11:31: 2020-05-15 13:11:31 UTC #3123816 17/25 peers 222 MiB chain 139 MiB db 0 bytes queue 36 MiB sync RPC: 0 conn, 4 req/s, 22 µs
May 15 13:11:43: 2020-05-15 13:11:43 UTC Imported #10070929 0x5c6b…6fdc (137 txs, 9.98 Mgas, 4732 ms, 23.64 KiB)
May 15 13:11:54: 2020-05-15 13:11:54 UTC Imported #10070930 0xcade…975f (100 txs, 9.91 Mgas, 6032 ms, 21.55 KiB)
May 15 13:12:05: 2020-05-15 13:12:05 UTC Reorg to #10070932 0x08a0…0049 (0xb411…88b7 #10070930 0xcade…975f 0x1017…909b)
May 15 13:12:13: 2020-05-15 13:12:13 UTC Imported #10070932 0x08a0…0049 (0 txs, 0.00 Mgas, 2731 ms, 0.52 KiB) + another 1 block(s) containing 94 tx(s)
May 15 13:12:20: 2020-05-15 13:12:20 UTC #3129173 20/25 peers 226 MiB chain 138 MiB db 0 bytes queue 44 MiB sync RPC: 0 conn, 18 req/s, 19 µs
May 15 13:12:21: 2020-05-15 13:12:21 UTC Imported #10070933 0xcd68…77e1 (170 txs, 10.00 Mgas, 4908 ms, 29.98 KiB)
May 15 13:12:31: 2020-05-15 13:12:31 UTC Imported #10070934 0x6765…2b3f (147 txs, 9.97 Mgas, 3966 ms, 25.80 KiB)
May 15 13:12:50: 2020-05-15 13:12:50 UTC #3136348 21/25 peers 188 MiB chain 138 MiB db 0 bytes queue 53 MiB sync RPC: 0 conn, 19 req/s, 28 µs
May 15 13:13:02: 2020-05-15 13:13:02 UTC Imported #10070935 0x1941…b56c (98 txs, 9.99 Mgas, 6063 ms, 15.50 KiB)
May 15 13:13:26: 2020-05-15 13:13:26 UTC #3153829 21/25 peers 189 MiB chain 138 MiB db 0 bytes queue 47 MiB sync RPC: 0 conn, 5 req/s, 20 µs
May 15 13:13:35: 2020-05-15 13:13:35 UTC 37 -> GetNodeData: item 5/384 – slow state fetch for hash 0x3d9ae468bd18c2bce5b81904843be444c38a4d55724a01e7b04cc6c59ee7d34f; took Instant { tv_sec: 888016, tv_nsec: 789367584 }
May 15 13:14:03: 2020-05-15 13:14:03 UTC #3182026 18/25 peers 193 MiB chain 138 MiB db 0 bytes queue 35 MiB sync RPC: 0 conn, 16 req/s, 22 µs
May 15 13:14:03: 2020-05-15 13:14:03 UTC Imported #10070936 0x03bb…1478 (255 txs, 9.96 Mgas, 3328 ms, 41.42 KiB)
@jinglics jinglics changed the title Disk Usage growth fast Disk Usage fast growt May 15, 2020
@jinglics jinglics changed the title Disk Usage fast growt Disk Usage fast growth May 15, 2020
@jinglics
Copy link
Author

What's the meaning of this log
2020-05-15 13:13:26 UTC #3153829 21/25 peers 189 MiB chain 138 MiB db 0 bytes queue 47 MiB sync RPC: 0 conn, 5 req/s, 20 µs.
I'm doing request to the node.

A lot of files generated at overlayrecent, like 012493.sst

@SomethingUseful
Copy link

SomethingUseful commented Jun 10, 2020 •
edited
Loading

Same issue but only started growing rapidly on openeth 3.0. No change in config on upgrade from 2.7 to 3. Running latest 3.0.1... eating 4-5GB per day! Now up to 400+GB

OpenEthereum version: OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1

Operating system: Linux

Installation: https://github.com/openethereum/openethereum/releases/download/v3.0.1/openethereum-linux-v3.0.1.zip

Fully synchronized: yes
Network: ethereum mainnet
Restarted: yes

Setup: 6CPU/16GB of ram. SSD.

EXAMPLE RUN: openethereum --chain mainnet --config .../config.toml

SNIPPET OF CONFIG:
`[parity]
mode = "last"
no_persistent_txqueue = true
chain = "mainnet"
base_path = "/not/default/.ethereumparity/share/io.parity.ethereum"
db_path = "/not/default/ethereum/chains"
light = false

[network]
discovery = true
warp = true

tx_queue_size = 8192
tx_queue_per_sender = 8192

[footprint]
tracing = "off"
fat_db = "off"
pruning = "fast"
db_compaction = "ssd"

[snapshots]
disable_periodic = true`

CHAIN DATA: 400+GB and growing fast at 5/GB/DAY
6700+ sst files...

ethereum/chains/ethereum/db/906a34e69aec8c0d/overlayrecent/db
/..
2.4 GiB [##########] /blooms
91.9 MiB [ ] 436793.sst
67.6 MiB [ ] 434030.sst
67.6 MiB [ ] 390882.sst
67.5 MiB [ ] 419861.sst
67.4 MiB [ ] 400297.sst
67.4 MiB [ ] 421432.sst
67.3 MiB [ ] 424576.sst
67.2 MiB [ ] 411739.sst
67.1 MiB [ ] 381813.sst
67.1 MiB [ ] 435272.sst
67.1 MiB [ ] 424770.sst
67.0 MiB [ ] 387992.sst

/ethereum/chains/ethereum/db/906a34e69aec8c0d
/..
402.5 GiB [##########] /overlayrecent
14.3 GiB [ ] /snapshot

@denisgranha
Copy link

thank you for your reports.

Can I ask what’s the content of /not/default/ethereum/chains ?

please paste:

ls /not/default/ethereum/chains

@SomethingUseful
Copy link

^ added to original comment above

@jinglics
Copy link
Author

jinglics commented Jun 11, 2020 •
edited
Loading

My node now is running normally, but the blockchain data keeps growing about 1GB per hour now.
However, when I restart, the storage size will decrease to 290GB around and then grow again.

parity --cache-size 12000 --db-compaction ssd --base-path /root/.eth --jsonrpc-apis "web3,eth,pubsub,net,parity,private,parity_pubsub,traces,rpc,parity_transactions_pool,personal" --jsonrpc-interface 0.0.0.0 --jsonrpc-cors * --enable-signing-queue true

65M 176025.sst
27M 176026.sst
421M 176027.log
175M 176029.log
418M 176030.log
578M 176031.sst
172M 176032.log
247M 176033.log
16M 176034.sst
404M 176035.log
150M 176036.log
177M 176037.log
536K 176039.sst

@SomethingUseful
Copy link

mine has 6700 sst files and only 94 log files. all the log files are less than 2 days old and less than 65MB. still increasing in size at 4 to 5GB per day.

@kael-shipman
Copy link

Possibly related: #11705

@DubbaThony
Copy link

@kael-shipman
I am fairly convinced it's related or same issue, but for me it apeared in 2.7, and upgrade to 3.0 simply didn't fix it. We still have crippling issues with that node, resyncing from scratch dosen't help, I tried many times. Only solution I am aware of is to fallback to some version around 2.4

@kael-shipman
Copy link

I just realized I linked that other issue but never actually mentioned that I'm having the same problem I wrote a stack exchange question about it here, but haven't done anything else to make progress.

I don't think I really have anything to add here, other than that I'm also experiencing the issue in all versions past 2.5.

@claberus
Copy link

The next release of OpenEthereum will be based on 2.5.13-stable and will include all of the bug fixes and support for hard forks since 2.7 and 3.0.

@DubbaThony DubbaThony mentioned this issue Sep 26, 2020
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@kael-shipman @jinglics @denisgranha @DubbaThony @claberus @SomethingUseful