Export block makes extremely high cost of memory which usually cause system killed on Linux.

[superboyiii]

Export block makes extremely high cost of memory which usually cause system killed on Linux.
On mainnet export, our server which has 16G memory, is only able to hold on about 2 million blocks' export. After that, it's been killed by ubuntu. Here's the line graph. I think ImportBlocks plugin is necessary to improve algorithm for export.

[shargon]

Thanks for this tests @superboyiii , this will helps to find performance issues

[lock9]

@superboyiii Hi! Is this solved?

[shargon]

@superboyiii we have the same issue with import?

[superboyiii]

@shargon No, it's because of that exporting block will write data straightly into memory which will cause overflow if your memory isn't large enough. But Import doesn't have this mechanism. Tom has made some effective fixes on this issue.

[shargon]

I think that the problem is here

https://github.com/neo-project/neo-plugins/blob/1c91346338c78537bec2760adfb363f3eb1619f5/ImportBlocks/ImportBlocks.cs#L84

Snapshot is a cache, and these blocks still on memory until the snapshot is disposed.
The best solution is create a snapshot for every block. Or add a new method, GetWithoutCache

[shargon]

@superboyiii could you told Tom to make a Pull request? also if he can join the two logics will be great!

https://github.com/neo-project/neo-plugins/blob/1c91346338c78537bec2760adfb363f3eb1619f5/ImportBlocks/ImportBlocks.cs#L63

[shargon]

I see that is already done! :)
#135

[ZhangTao1596]

Will do