Not all blocks parsed

[nomeata]

It seems that BlockSci does not reliably index all transactions. For example, this transaction is found using

$ bitcoin-cli b83f3c690163517aa32d8ff3524acf87cc6f344595f642674c0733876f079642
{
  "txid": "b83f3c690163517aa32d8ff3524acf87cc6f344595f642674c0733876f079642",
  "hash": "b83f3c690163517aa32d8ff3524acf87cc6f344595f642674c0733876f079642",
  "version": 1,
…

but not using

>>> import blocksci
>>> chain = blocksci.Blockchain("…")
>>> chain.tx_with_hash('b83f3c690163517aa32d8ff3524acf87cc6f344595f642674c0733876f079642')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: No Match for hash

I created the index using

./blocksci_parser --output-directory …/blocksci update disk --coin-directory …/bitcoin/datadir

and the parser finished successfully.

There are more transactions affected, but I could not yet detect a pattern. It is not the case that it simply ignores all transaction with nulldata output, as transactions like this are found just fine.

[hkalodner]

Looking up that particular transaction works fine on my installation. I'll do a full test and see if there are any txids missing. Is anyone else hitting this problem?

I'm not sure what would be causing this. The code to generate the tx hash index is pretty straightforward. I'm guessing a reinstallation would fix your problem, but that's not a very satisfying answer.

[nomeata]

Thanks for checking so quickly. So it is a local problem of sorts? That’s weird.

Does it matter to BlockSci whether the bitcoind whose data directory BlockSci parses, is called with -txindex?

[nomeata]

A stab in the dark: I am re-building the BlockSci index using the rpc rather than the disk method. Once that is done I will report back if it made a difference.

[nomeata]

Hmm, loading the whole bitcoin chain using rpc will take a few days, it seems…

Is there anything else I can try to debug? I tried to load elements related to the missing transaction, but they are all missing:

• chain.address_from_string('1BjxLkPF1biMgBdwyzwvaW5ZNUaUYQjWWD') is None
• the previous transaction, chain.tx_with_hash('41c8460b28e30a2546eccbeb896e9470aa3d7c3b69c2f337689aa284eb2d1617'), is also None
• and chain[367578] throws an IndexError. 367578 is the height of the block containing this transaction, according to https://blockexplorer.com/block/0000000000000000004ac6df6a44bae938d717b55f205f4266d59e98c0d21a11. Is this the correct way of looking up a block with BlockSci? If so, it looks like the whole block is missing… any guess why that would happen?

Indeed, when I run update again, i.e.

$ ./blocksci_parser --output-directory …/blocksci update disk --coin-directory …/bitcoin/datadir 
100.0% done fetching block headers
Starting with chain of 364308 blocks
Removing 0 blocks
Adding 0 blocks
Updating hash index
Updating address index
Updating index with scripts of type multisig_script

it says only 364308 blocks are found, when we should be at 515707 blocks now, and my bitcoind knows about these blocks:

$ bitcoin-cli getblockchaininfo 
{
  "chain": "main",
  "blocks": 515707,
  "headers": 515707,
…

Maybe my bitcoin data directory is somehow corrupt? Or what else would make ./blocksci_parser not look at all blocks?

[nomeata]

Ok, reparsing fixed it. Sorry for the noise!

[Haaroon]

i had the same issue, for me it was when the txes were not all properly indexed on the machine, pretty annoying but hey the solution was to reindex the blockchain then to reparse the entire chain,

[nomeata]

I am reparsing again, but I am running into #100 it seems :(

[joequant]

FYI by removing parser/blockList.dat I was able to get blocksci to reload the blocks.