Is it possible to recover parsing after initial process was killed?

[mribbons]

So, it's taking a long time to process the entire BTC blockchain. I had to reboot the machine I'm using.

When I try to run the parser again I get these errors. Should I perhaps delete some partially written block files in order to resume processing? I really don't want to start from scratch again.

I did start with a completely new parser data folder after the recent release, I have also just updated to edb6b46 and the issue still occurs.

Is this the same issue as #31?

mribbons@toastname:~$ blocksci_parser --coin-directory ~/.bitcoin --output-directory ~/blocksciparse2
0.00% done, Block 375436/498166terminate called after throwing an instance of 'NextQueueFinishedEarlyException'
  what():  Next queue finished early
Aborted (core dumped)
mribbons@toastname:~$ blocksci_parser --coin-directory ~/.bitcoin --output-directory ~/blocksciparse2
100.0% done fetching block headers
Starting with chain of 375466 blocks
Removing 0 blocks
Adding 122700 blocks
terminate called after throwing an instance of 'NextQueueFinishedEarlyException'
  what():  Next queue finished early
Aborted (core dumped)
mribbons@toastname:~$ blocksci_parser --coin-directory ~/.bitcoin --output-directory ~/blocksciparse2
100.0% done fetching block headers
Starting with chain of 375488 blocks
Removing 0 blocks
Adding 122678 blocks
0.00% done, Block 375489/498166terminate called after throwing an instance of 'NextQueueFinishedEarlyException'
  what():  Next queue finished early
Aborted (core dumped)

[mribbons]

I have also tried with the updated command syntax:

$ blocksci_parser --output-directory ~/blocksciparse2 update disk --coin-directory ~/.bitcoin

[hkalodner]

Sorry but the parser currently doesn't have any way of cleanly exciting mid run so the produced data is likely corrupted. I've added Issue #33 which would greatly improve this situation, but for now there's not a ton to be done.

The issue that you referenced has to do only with when there is a blockchain reorganization that has occurred that affects blocks that have already been parsed and so it is only possible in fully up to date data.

The new command syntax was really just to remove the dependency on boost program options and didn't reflect any other changes.

If you'd like to be able to play around with BlockSci on a portion of the chain, the parser has an --max-block argument which you can set when invoking the parser to limit what block height it will process up to. This can also be used to run the parsing in multiple chunks.

On another note, your computer may not be quite up to the task of running the parser. Running it through block 400000 should be relatively quick since the most of blocks before that point were not full and thus the vast majority in the blockchain comes from blocks that were added after that point. On an 8 core machine with 61 GB of memory (r4.2xlarge EC2 machine) the whole parser currently runs in a bit under 24 hours. If I remember correctly running up to where you interrupted doesn't take more than a couple hours with that setup.

[mribbons]

Hi @hkalodner,
Thanks for the feedback, using --max-blocks I can at least do a backup every now and then.

Things seem to really slow down for me around 300000.

The computer is decent in terms of CPU, but only has 16GB of ram and not enough SSD space, so have had to run of a HDD.

Will try to free up some SSD space and go from there. Thanks again.