Light node using too much storage, doing fast sync? (after 1.9.17) #22151
Labels
Comments
|
I synced a les node this week with the current master in 8 minutes and 300MB. |
|
You said "completely omremoved ~/.ethereum folder", but your args uses ".ethereum-mainnet", so maybe it didn't start from scratch?Otherwise, please provide your startup logs.
|
|
Sorry for the confusion holiman, but yes I have been wiping the correct folder Thanks for testing Marius, that's good to know. You are right, those logs appear in both the old and new versions. Never mind those logs then, I was mistaken. It still seems to me that v1.9.17 is using less disk space than v1.9.25. In fact the disk usage sometimes shrinks over time. Could it be that v1.9.17 is collecting garbage more efficiently than v1.9.25 is? I will run against the github version 89030ec now, and see how that goes. Startup with 89030ec looks like: Update: I have not been getting any problem with the recent github version 89030ec. 👍 Still testing to confirm whether 1.9.25 really does have a problem. |
|
My colleague says his v1.9.25 grows to about 2.1GB and then stabilises (but I suspect it will slowly grow in future). My v1.9.25 reaches 1.3GB. (And then stops, because my disk is full. The geth build used up some space.) Also, syncing seems to take longer with this version. Logfile:
The good news is that my build from master 89030ec doesn't have this problem! The Although those files don't appear to be much larger. So I guess there must be more of them. It's actually growing and shrinking (but luckily it never exceeds 1.3GB). |
|
Could you try a from the "large" run |
|
Sure thing @holiman. Sorry it took a while, had to start it from scratch again. At this point I stopped the process, so my disk would not fill up. Edit: Wow, after restarting it, the usage eventually dropped to 266MB! So it can discard the unused data, it just isn't doing it often enough. (At least not for my needs.) Limiting the bandwidth the node can use seems to help. Since restarting with I monitor the usage over time with: |
|
can you run
|
|
Bit more context about that: we noticed that your |
|
Good news (sort of). For some reason, the space grew to 1.3GB again (and then my disk filled). So it's the right conditions to do an inspection! |
It's not a normal log, but it's the WAL file of the leveldb. So you definitely lose a lot of data. The real log is |
|
Also you are using the |
|
Is your application doing a log history search? That seems like the most obvious reason that can justify downloading a lot of block bodies. Syncinc does not do that. |
|
@zsfelfoldi For each block, we are reading every ETH transaction and every USDT 'Transfer' event. We do this not at the head block, but 10 blocks behind the head (for confirmation). (We also regularly fetch a bunch of balances, again 10 blocks behind the head.) Based on what you said, I am guessing that the events could indeed be the cause. Thanks. It had not occurred to me that the disk usage might come from the way we use the node, rather than the syncing. @rjl493456442 Yes we are using 1.9.25. If it is indeed the LevelDB cache eating the disk space, then it seems to be pruned more often than the sync data, but perhaps not regularly enough for our purposes. However, the current implementation might be fine for most users. If we are causing this growth by spamming the node, then we should probably just invest in more disk space! (We do have a fast node as well. Our light nodes are running as backups, in case the fast node breaks, and also for devs to use, which is why they are receiving requests.) Thank you all for your info. If you think this is not a concern for most users, please feel free to close the issue. |
|
@joeytwiddle Yup, I will close it very soon. Just a quick question, the pruner is triggered by every 12 hours by default. It will scan all the data(receipts, bodies, etc) that belong to the previous sections and delete them. Do you think it's useful that the trigger can somehow be configured from a user's perspective? Yes, I think we can make it configurable. Close it now. |
|
@rjl493456442 Yes I think that would be useful for my case. Either by time, or number of blocks, or space used. Less important, but perhaps useful for recovery/testing/debugging, might be a command to trigger the pruner. (Or just run it automatically on startup.) Or perhaps that exists already? As |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Related: #22128
Situation
Starting from scratch (completely removed
~/.ethereum*folders)Command:
geth --syncmode lightThe full setup I am using (probably not relevant):
I'm actually limiting the process somewhat, to reduce my AWS bill.
Behaviour with geth v1.9.17
The node syncs up in under 3 hours. Disk usage is 500MB. 👍
(9 hours later, the usage has dropped to 324 MB.)
Behaviour with geth v1.9.25
Sync takes longer. After many hours (maybe 24) the disk usage is 1.9GB. 👎
I see in the logsEdit: Disregard. As Marius pointed out, the logs are expected for light nodes too.Imported new block headers. But we should only see that for fast/full nodes, right?(I see those same logs with v1.9.22 and v1.9.23, but I haven't checked to see if they also consume a lot of disk space.)Possible regression?
Has there been a regression, whereby the light node now uses more disk space than it used to, more than it needs?
Is it doing a full or fast sync?If so, I suspect the regression was introduced
> v1.9.19 and <= v1.9.22.(I am guessing that v1.9.19 is also working fine, because I recall I used to use that version happily. But I haven't actually tested to confirm that.)
The text was updated successfully, but these errors were encountered: