bad number of block parsed with blocksci_parser

[CrunchyFanch]

Hi!
Your tools seems very great but i have a problem when i uses the standard blocksci_parser command
blocksci_parser --output-directory bitcoin-data update disk --coin-directory .bitcoin
It seems to parse only 499291 blocks but there is currently 504087 saved by bitcoind on my hdd.
bitcoin-cli getblockcount
504087
I tried to reforce the parse but the number of block didn't increase...
This is the message returned by the parser:

`blocksci_parser --output-directory bitcoin-data update disk --coin-directory .bitcoin
100.0% done fetching block headers
Starting with chain of 499291 blocks
Removing 0 blocks
Adding 0 blocks
Updating hash index
Updating index with scripts of type nonstandard_script

Updating index with scripts of type pubkey_script

Updating index with scripts of type scripthash_script

Updating index with scripts of type multisig_script

Updating index with scripts of type null_data_script

Updating address index
Updating index with scripts of type nonstandard_script

Updating index with scripts of type pubkey_script

Updating index with scripts of type scripthash_script

Updating index with scripts of type multisig_script

Updating index with scripts of type null_data_script`

Have an idea
Thanks in advance
Crunchy

[hkalodner]

This has me a bit stumped. It's probably something specific to your configuration since this bug hasn't appeared in other places. I'll whip up some code to dump out more debug output and hopefully get to the bottom of this.

The parse occurs in multiple phases. During the "fetching block headers" chunk, the all of the blk.dat files in .bitcoin/blocks are parsed and block headers are extracted from all of them. Then the longest chain is built from the parsed blocks.

The bug is likely to be either due to the parser not reading all headers or somehow not being able to construct a longer chain, but I'm not sure why either would be true.

[CrunchyFanch]

Ok thank you for your answer.
Do not hesitate if you want, i run specific commands. or even create an temporary user ssh for u.

More info... the blocks seems continuously parsed 1 to 499291 because when i execute this python code :
chain = blocksci.Blockchain('/home/xxxx/bitcoin-data/')
block_height = 499290
print(chain[block_height])
block_height = 499292
print(chain[block_height])

this is the output on a jupyter notebook :

Block(numTxes=3139, height=499290, header_hash=00000000000000000023b12b9e46bffb32031434bd09e66b4a081c26f659e332, version=536870912, timestamp=1513273240, bits=402698477, nonce=3408706076)

IndexError Traceback (most recent call last)
<ipython-input-12-ed57a120f5f0> in <module>()
2 print(chain[block_height])
3 block_height = 499292
----> 4 print(chain[block_height])
5
6

IndexError:

The hash 00000000000000000023b12b9e46bffb32031434bd09e66b4a081c26f659e332 is the true hash of the block's header 499290 (see on the blockchain.info website)

Crunchy

[CrunchyFanch]

Maybe it can also help you : if i try to update with rpc command it show me the following error message
blocksci_parser --output-directory bitcoin-data update rpc --username xxx --password xxx --address 127.0.0.1 --port 8332
terminate called after throwing an instance of std::length_error' what(): vector::reserve
Abandon (core dumped)`

[abirke]

I am seeing the exact same behavior using both the update disk variant as well as the update rpc one. The former stops parsing at 235,278 blocks while bitcoin-cli knows about 504,687 blocks. The latter stops with said error.

terminate called after throwing an instance of 'std::length_error'
  what():  vector::reserve
Aborted (core dumped)

I'd be happy to provide any debugging help, if possible.

[CrunchyFanch]

I found a temporary solution to avoid this issue.

• stop bitcoin core
• Upgrade Bitcoin Core to v0.15.1.0-g7b57bc998f (64-bit)
• delete /home/xxx/.bitcoin (or dir where are yours blocks)
• redownload blocks without forget to specify (txindex=1) in bitcoin.conf (to include txs)
• reparse with blocksci update block command

I was able to parse the 504000 blocks and more but in 5 days of download and parse...

[hkalodner]

I think this was the same bug as described in #69. This should now be resolved. It was caused by some strangeness in how bitcoin core serializes blocks. Apparently sometimes when bitcoin core is halted it can leave the data in an odd state that I had never run into myself.