Not detecting latest chain length in disk mode #100
Comments
|
Did you run the parser to update on the same maschine with which you initial parsed your 517549 blocks? |
|
The parser is being run on the same machine. However, the initial parsing job pointed to a mounted filesystem and this new parsing job is pointing to a different one. Thanks. |
|
Hmm, so the problem is resolved? I'm not sure I understand exactly what went wrong. |
|
Thanks. I am afraid the problem is not yet resolved. To clarify, when I run the blocksci parser using the command above, it finds the chain length to be 517549. However, when I run bitcoin-cli getblockcount on the same bitcoin folder, it finds block length of 520k+. Is there a debug or deeper logging mode in blocksci? |
|
I just updated the parser to finally resolve #69 which might be the same issue here. Can you update to the latest commit on the master branch and try running the parser again? |
|
Thanks for trying to fix the issue. Will do.
…Sent from my iPhone
On May 5, 2018, at 5:33 PM, hkalodner ***@***.***> wrote:
I just updated the parser to finally resolve #69 which might be the same issue here. Can you update to the latest commit on the master branch and try running the parser again?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
I'm keeping my fingers crossed! This highlights the hazards of depending on parsing the |
|
Thanks. In the long run do you think disk and RPC mode have equal
likelihood of being maintained or do you think the RPC mode is more likely?
…On Sat, May 5, 2018 at 10:19 PM, hkalodner ***@***.***> wrote:
I'm keeping my fingers crossed! This highlights the hazards of depending
on parsing the blkfiles. Unfortunately it's the only option I can think
of that allows the parser to run concurrently with an unmodified full node
and keep in sync with it. The other option would be modifying a full node
to incorporate BlockSci code into it, but from a maintenance perspective
that would be infeasible.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#100 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/Aha7cyDtEkQIq64kKmx6XDUCxVWZ8UFFks5tvl2agaJpZM4Trnzt>
.
|
|
It's hard to talk about the long run currently since I'm not entirely sure what's going to happen to BlockSci. I'm currently spearheading the project and will be for at least the next year, but I then I'll graduate with my Phd and after that it's a bit of an unknown, though I'd like to keep working on BlockSci. RPC is definitely the easiest to maintain and the least likely to break over the long term, but it suffers from comparably terrible performance to the disk parser which is why it's great for altcoins, but not so great for Bitcoin. I'd like to keep the disk parser working as long as possible. It depends on a lot of unknowns about how the Bitcoin protocol and Bitcoin Core will change over time. If they don't make any drastic changes, there hopefully won't be any major problems. That section of the code is pretty easy to edit so it's mostly a matter of how difficult it is to understanding how the data is stored on disk since it's undocumented and unsupported (Though currently an exact match for the network protocol which is documented). There's a compromise solution that I could implement which would have avoided the issue I just fixed and would probably be easier to maintain, but would be pretty bad in terms of usability. Bitcoin Core stores block headers in a LevelDB database which would eliminate the need to perform the first phase of parsing where block headers are scanned. However, accessing this database requires shutting down Bitcoin Core since LevelDB is single reader only. Having to shut down Bitcoin Core every time you want to run the parser to update new blocks wouldn't be great. |
|
Thanks for the insights.
I am updating to the latest commit on master now. Will let you know.
… On 2018-05-06, at 10:21, hkalodner ***@***.***> wrote:
It's hard to talk about the long run currently since I'm not entirely sure what's going to happen to BlockSci. I'm currently spearheading the project and will be for at least the next year, but I then I'll graduate with my Phd and after that it's a bit of an unknown, though I'd like to keep working on BlockSci.
RPC is definitely the easiest to maintain and the least likely to break over the long term, but it suffers from comparably terrible performance to the disk parser which is why it's great for altcoins, but not so great for Bitcoin.
I'd like to keep the disk parser working as long as possible. It depends on a lot of unknowns about how the Bitcoin protocol and Bitcoin Core will change over time. If they don't make any drastic changes, there hopefully won't be any major problems. That section of the code is pretty easy to edit so it's mostly a matter of how difficult it is to understanding how the data is stored on disk since it's undocumented and unsupported (Though currently an exact match for the network protocol which is documented).
There's a compromise solution that I could implement which would have avoided the issue I just fixed and would probably be easier to maintain, but would be pretty bad in terms of usability. Bitcoin Core stores block headers in a LevelDB database which would eliminate the need to perform the first phase of parsing where block headers are scanned. However, accessing this database requires shutting down Bitcoin Core since LevelDB is single reader only. Having to shut down Bitcoin Core every time you want to run the parser to update new blocks wouldn't be great.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
It looks like the local compile instructions don't have steps to make rocksdb among other things that was required when I first installed BlockSci. Does using the latest commit on master mean that the indexing will start from scratch? Or will it pick up where I last left it? Thanks. DB. |
|
v0.5 should be backwards compatible with v0.4.5. I got rid of a bunch of external dependencies by pulling them into the build process itself so now rocksdb will be built with blocksci and then compiled into the library rather than requiring an external installation. |
|
Tried to compile. The last step (pip3) is failing for me. Please attached log file. Am I doing something incorrectly? |
|
Hmm. I thought I had solved the issues related to previous installations interfering, but I guess not. I refactored a lot in this release and it's mixing new files and old files. The quick fix should be deleting |
|
Sounds good. Will give that a shot.
… On 2018-05-06, at 16:25, hkalodner ***@***.***> wrote:
Hmm. I thought I had solved the issues related to previous installations interfering, but I guess not. I refactored a lot in this release and it's mixing new files and old files. The quick fix should be deleting /usr/include/blocksci/ and installing BlockSci again.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Deleting /usr/local/include/blocksci. I don’t have a /usr/include/blocksci directory.
… On 2018-05-06, at 16:26, Brick In Edifice ***@***.***> wrote:
Sounds good. Will give that a shot.
> On 2018-05-06, at 16:25, hkalodner ***@***.***> wrote:
>
> Hmm. I thought I had solved the issues related to previous installations interfering, but I guess not. I refactored a lot in this release and it's mixing new files and old files. The quick fix should be deleting /usr/include/blocksci/ and installing BlockSci again.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
|
|
I am afraid it doesn't look as if it worked. It's giving me the same message as what is at the very top of this issue. However, I noticed that the blocksci_parser in /usr/local/bin did not change. Should this have changed? At the same time, the latest compilation process put includes in /usr/include/blocksci and not in /usr/local/include/blocksci. Does this matter? Thanks. |
|
Ah whoops. Try deleting the blocksci stuff from I'm going to add instructions on how to delete the old installation to the compilation instructions. It could also totally be that I didn't fix the bug you're hitting and just found a different bug related to block parsing. |
|
Some progress. Now I am getting this error.
Thoughts on possible paths forward? |
|
Same issue (parser not updating even though blockchain is) on v0.5 installed brand-new in the beginning of June. I am not sure #69 was completely resolved yet. |
|
Same issue. Use BlockSci v0.5, delete diego@big:~/.bitcoin$ bitcoin-cli getblockcount diego@big:~/parsedata$ blocksci_parser -o btc update disk -c /home/diego/.bitcoin |
Hi: I am running the parser in disk mode. The last time (and first time) I ran the parser, it updated to a chain of 517549 blocks. Right now the latest block file in
bitcoin_core/blocksfolder isblk01246.datcorresponding to chain length of 520xxx. When I run the parser to update, it is not detecting the latest chain length. I am running the following command to update.sudo nohup blocksci_parser -o bitcoin_data update disk -c bitcoin_core >> log &The log file states this:
100.0% done fetching block headersStarting with chain of 517549 blocksRemoving 0 blocksAdding 0 blocksUpdating hash indexUpdating address indexUpdating index with scripts of type multisig_scriptAny thoughts on how I can move forward?
Thanks.
The text was updated successfully, but these errors were encountered: