Write witness caches when writing the best block #1904

str4d · 2016-11-30T01:22:31Z

For steady-state operation, this reduces the average time between wallet disk
writes from once per block to once per hour.

On -rescan, witness caches are only written out at the end along with the best
block, increasing speed while ensuring that on-disk state is kept consistent.

Witness caches are now never recreated during a -reindex, on the assumption that
the blocks themselves are not changing (the chain is just being reconstructed),
and so the witnesses will remain valid.

Part of #1749.

For steady-state operation, this reduces the average time between wallet disk writes from once per block to once per hour. On -rescan, witness caches are only written out at the end along with the best block, increasing speed while ensuring that on-disk state is kept consistent. Witness caches are now never recreated during a -reindex, on the assumption that the blocks themselves are not changing (the chain is just being reconstructed), and so the witnesses will remain valid. Part of zcash#1749.

str4d · 2016-11-30T01:23:12Z

Running tests to check for anything missed.

@zkbot try

zkbot · 2016-11-30T01:23:16Z

⌛ Trying commit 03f83b9 with merge 03f83b9...

zkbot · 2016-11-30T02:10:42Z

☀️ Test successful - zcash

bitcartel

Comments based on PR from pairing. Now that user has confirmed speed-up, these comments may become stale as PR is updated.

BerkeleyDB does support nested transactions e.g dbenv->txn_begin(parentTxn... , so we could create a parent transaction with two child transactions, one for best block, the other for witness cache. However, upstream API does not support parent transactions, so current approach works fine.

Document that SetBestChain() now does two things:

Saves location of best block to disk
Saves witness cache to disk

Do we still need the template for testing, can we move WriteWitnessCache to .cpp?

Can we have a better (easier to read) name than SetBestChainINTERNAL e.g. SetBestChain_internal or something else entirely?

Document why ClearNoteWitnessCache() has been removed.

Also confirm and document what happens if -reindex and -rescan are interrupted, whether cleanly via ctrl-c, or kill -9. Will the witness cache still be consistent?

str4d · 2016-12-07T06:48:48Z

Document that SetBestChain() now does two things:

Saves location of best block to disk

Saves witness cache to disk

Done.

Do we still need the template for testing, can we move WriteWitnessCache to .cpp?

Still necessary (for the now-named SetBestChainINTERNAL).

Can we have a better (easier to read) name than SetBestChainINTERNAL e.g. SetBestChain_internal or something else entirely?

I was following existing conventions (CMerkleTx::GetDepthInMainChainINTERNAL in the same file), but if you feel strongly about it I can change it.

Document why ClearNoteWitnessCache() has been removed.

Is the documentation in the commit message sufficient?

Also confirm and document what happens if -reindex and -rescan are interrupted, whether cleanly via ctrl-c, or kill -9. Will the witness cache still be consistent?

If -reindex is interrupted, the witness cache will not be , regardless of how. The chain index will need to continue being rebuilt (and rerunning the command as-is would obviously restart the rescan).

With -rescan, there are two cases:

Ctrl+C: doesn't work yet so rescan will continue to completion, see Enable Ctrl+C during rescan #1908.
kill -9: the witness cache in-memory was cleared at the start of the rescan, but not on-disk, so after restart the on-disk witness cache will be reloaded into memory.

ebfull · 2016-12-07T17:05:51Z

ACK

bitcartel · 2016-12-07T21:07:11Z

^-- restart the rescan) typo? You mean 'restart the reindex`?

str4d · 2016-12-07T22:41:29Z

Yes, typo.

bitcartel · 2016-12-07T23:46:24Z

ACK (note: i'm currently in the middle of yet another -reindex which should take ~45-60 mins to complete as it doesn't have the verifyjoinsplit checkpoint patch applied)

bitcartel · 2016-12-08T01:50:44Z

Temporary NACK - investigating an issue.
I rebased this PR on #1892 and on invoking zcashd -reindex I came across this issue:
wallet/wallet.cpp:762: void CWallet::DecrementNoteWitnesses(const CBlockIndex*): Assertion `(nd->witnessHeight == -1) || (nd->witnessHeight == pindex->nHeight)' failed.

bitcartel · 2016-12-08T03:52:54Z

ACK

This PR on its own does not cause the assertion above.
After a sucessful reindex, node was shutdown and relaunched at which point it said "Rescanning last 9366 blocks (from block 14397)...". That's a lot of blocks to rescan and will take a long time. As part of this PR, should a clean shutdown e.g. zcash-cli stop involve writing the best block so that rescan will only involve the usual 288 blocks?

bitcartel · 2016-12-09T08:02:18Z

@str4d @ebfull Temporary NACK.
I rebased this PR on master (which has now merged #1919). I get the assertion error on reindex.

zcashd: wallet/wallet.cpp:762: void CWallet::DecrementNoteWitnesses(const CBlockIndex*): Assertion `(nd->witnessHeight == -1) || (nd->witnessHeight == pindex->nHeight)' failed.

debug.log:

2016-12-09 07:59:10 UpdateTip: new best=00000d8812e5f4021ee3cbd11163b2aed9d8a49d5dbc4a6a4b062ff6e0c1509b  height=244  log2_work=25.032341  tx=245  date=2016-10-28 16:40:21 progress=0.000534  cache=0.1MiB(244tx)
2016-12-09 07:59:10 UpdateTip: new best=0000023d58e6c3f0665c53fbb30a34eb799910798552c8d3ab951241a27b97db  height=245  log2_work=25.059861  tx=246  date=2016-10-28 16:40:24 progress=0.000536  cache=0.1MiB(245tx)
2016-12-09 07:59:10 UpdateTip: new best=000007ab6f8f736d9b697c6383fa6f009de256836903ec17be1e247ad8fcfa1d  height=246  log2_work=25.087376  tx=247  date=2016-10-28 16:40:34 progress=0.000539  cache=0.1MiB(246tx)
2016-12-09 07:59:10 LoadExternalBlockFile: Processing out of order child 0000013b550f8c390f64a04d5e1c2f4ca835218e16ce07f86eb84d60983cb164 of 00001691bb16540dfea2e0e49bc74bda6f21e0d6bafc316c6b2b6b55b66fd361
2016-12-09 07:59:10 UpdateTip: new best=0000023d58e6c3f0665c53fbb30a34eb799910798552c8d3ab951241a27b97db  height=245  log2_work=25.059861  tx=246  date=2016-10-28 16:40:24 progress=0.000536  cache=0.1MiB(245tx)

str4d · 2016-12-09T11:29:01Z

It looks like your local blockchain captured an orphan - neat! It then gets replayed as you reindex:

Blocks 00001691bb16540dfea2e0e49bc74bda6f21e0d6bafc316c6b2b6b55b66fd361 and 000007ab6f8f736d9b697c6383fa6f009de256836903ec17be1e247ad8fcfa1d are read in, and as the latter is more work, it becomes the chain tip.
Block 0000013b550f8c390f64a04d5e1c2f4ca835218e16ce07f86eb84d60983cb164 is read in, which is the child of 00001691bb16540dfea2e0e49bc74bda6f21e0d6bafc316c6b2b6b55b66fd361, causing that to have more work and forcing a reorg.
During the reorg, your node calls DecrementNoteWitnesses() and triggers the assertion.

I've also figured out the bug: when I added support for incrementing new notes on rescan (in #1400), I conditionally ignored notes that had already been witnessed above the current height (for efficiency). I neglected to add a similar conditional for decrementing, which at the time was fine, because:

On rescan, the chain is unchanged, and rescanning only involves incrementing witnesses.
On reindex, the witnesses were cleared and rebuilt from scratch.

With this PR, witnesses are no longer cleared on reindex, and combined with the fact that orphans get replayed, this means that I need to implement the corresponding conditional in DecrementNoteWitnesses().

…s caches Ref: zcash#1904 (comment)

bitcartel · 2016-12-09T18:33:25Z

Checked out PR, -reindex works, does not trigger assertion.
Checked out PR, rebased on master 6575df, -reindex works, does not trigger assertion.
ACK.

str4d · 2016-12-09T20:18:52Z

@ebfull could you please re-review to update your ACK?

ebfull · 2016-12-09T21:10:55Z

reACK, thanks @str4d and @bitcartel!

@zkbot r+

zkbot · 2016-12-09T21:10:57Z

📌 Commit 9d2cc3a has been approved by ebfull

zkbot · 2016-12-09T21:11:00Z

⌛ Testing commit 9d2cc3a with merge 9f7bc6c...

… r=ebfull Write witness caches when writing the best block For steady-state operation, this reduces the average time between wallet disk writes from once per block to once per hour. On -rescan, witness caches are only written out at the end along with the best block, increasing speed while ensuring that on-disk state is kept consistent. Witness caches are now never recreated during a -reindex, on the assumption that the blocks themselves are not changing (the chain is just being reconstructed), and so the witnesses will remain valid. Part of #1749.

zkbot · 2016-12-09T21:59:57Z

☀️ Test successful - zcash

bitcartel · 2016-12-09T23:17:15Z

@str4d @ebfull We may need to rollback this merge as it is still triggering an assertion.

str4d · 2016-12-10T01:35:31Z

The second assertion being triggered is fixed in #1933.

Fixes the bug resulting from zcash#1904.

Only increment new notes on reindex Addresses another issue in #1904. When an existing one of our notes was found again, its cache was reset and it was re-witnessed. This would cause encountered notes to get out-of-sync with the otherwise-ignored newer notes, which could be a problem if the wallet data happens to be written out during a reindex.

* Add getlocalsolps and getnetworksolps RPC calls, show them in getmininginfo * Add benchmark for attempting decryption of notes * Add benchmark for incrementing note witnesses * Add -metricsui flag to toggle between persistent screen and rolling metrics Defaults to true if stdout is a TTY, else false. * Add -metricsrefreshtime option * Only show metrics by default if stdout is a TTY * Document metrics screen options * Fix stale comment referencing upstream block interval * Add checkpoint at block height 15000 * Added mainnet, testnet, and onion nodes * Make command line option to show all debugging consistent with similar options Most people expect a value of 1 to enable all for command line arguments. However to do this for the -debug option you must type "-debug=". This has been changed to allow "-debug=1" as well as "-debug=" to enable all debug logging * Update documentation to match the zcash#4219 change * Update help message to match the zcash#4219 change * Clarify that metrics options are only useful without -daemon and -printtoconsole * Increase length of metrics divider * Closes zcash#1857. Fixes bug where tx spending only notes had priority of 0. * Closes zcash#1901. Increase default settings for the max block size when mining and the amount of space available for priority transactions. * Write witness caches when writing the best block For steady-state operation, this reduces the average time between wallet disk writes from once per block to once per hour. On -rescan, witness caches are only written out at the end along with the best block, increasing speed while ensuring that on-disk state is kept consistent. Witness caches are now never recreated during a -reindex, on the assumption that the blocks themselves are not changing (the chain is just being reconstructed), and so the witnesses will remain valid. Part of zcash#1749. * Add porter dev overrides for CC, CXX, MAKE, BUILD, HOST * Apply miniupnpc patches to enable compilation on Solaris 11 These can be removed after the next MiniUPnP release. Closes zcash#1835. * Closes zcash#1903. Add fee parameter to z_sendmany. * Add an upstream miniupnpc patch revision * Metrics - Don't exclaim unless > 1 "You have validated 0 transactions!" sounds a little less enthusiastic that intended. Also, only says "1 transaction". * Address review comments, tweak strings * bash-completion: Adapt for 0.12 and 0.13 * separate completion for bitcoind and bitcoin-cli * remove RPC support from bitcoind completion * add completion for bitcoin-tx and bitcoin-qt * rely on autoloading of completions * Change function names to not clash with Bitcoin, apply to correct binaries * Add bash completion files to Debian package * Always bash-complete the default account * Add Zcash RPC commands to CLI argument completion * Fixes zcash#1823. Witness anchors for input notes no longer cross block boundaries. * Edit for grammar: "block chain" At this point, I believe it is universally accepted that "blockchain" is one word, and should not be separated into two. * Increase timeout as laptops on battery power have cpu throttling. * Isolate verification to a `ProofVerifier` context object that allows verification behavior to be tuned by the caller. * Regression test. * Ensure cache contains valid entry when anchor is popped. * Ensure ProofVerifier cannot be accidentally copied. * Document behaviour of CWallet::SetBestChain * WitnessAnchorData only needs to store one witness per JSOutPoint. * Rename Dummy to Disabled. * Add more tests for ProofVerifier. * Fix indentation * Generate JS for trydecryptnotes, make number of addresses a variable * Add JS to second block to ensure witnesses are incremented * ASSERT_TRUE -> ASSERT_FALSE * Skip JoinSplit verification before the last checkpoint Part of zcash#1749 * Gather release notes from previous release to HEAD Also update release-process.md to replace git shortlog command with release-notes.py script. * Add a reindex test that fails because of a bug in decrementing witness caches Ref: zcash#1904 (comment) * Make the test pass by fixing the bug! * Only check cache validity for witnesses being incremented or decremented Fixes the bug resulting from zcash#1904. * Check that E' points are actually in G2 by ensuring they are of order r. * Fix bug in wallet tests * Extract block-generation wallet test code into a function * Rewrite reindex test to check beyond the max witness cache size * Fix bug in IncrementNoteWitness() * Update payment API docs to recommend -rescan for fixing witness errors * Update version to 1.0.4 * Update man pages * Release notes, authors, changelog * Update seed nodes * Bugfix #14 - getblocksubsidy RPC command is incorrect

* Add getlocalsolps and getnetworksolps RPC calls, show them in getmininginfo * Add benchmark for attempting decryption of notes * Add benchmark for incrementing note witnesses * Add -metricsui flag to toggle between persistent screen and rolling metrics Defaults to true if stdout is a TTY, else false. * Add -metricsrefreshtime option * Only show metrics by default if stdout is a TTY * Document metrics screen options * Fix stale comment referencing upstream block interval * Add checkpoint at block height 15000 * Make command line option to show all debugging consistent with similar options Most people expect a value of 1 to enable all for command line arguments. However to do this for the -debug option you must type "-debug=". This has been changed to allow "-debug=1" as well as "-debug=" to enable all debug logging * Update documentation to match the zcash#4219 change * Update help message to match the zcash#4219 change * Clarify that metrics options are only useful without -daemon and -printtoconsole * Increase length of metrics divider * Closes zcash#1857. Fixes bug where tx spending only notes had priority of 0. * Closes zcash#1901. Increase default settings for the max block size when mining and the amount of space available for priority transactions. * Write witness caches when writing the best block For steady-state operation, this reduces the average time between wallet disk writes from once per block to once per hour. On -rescan, witness caches are only written out at the end along with the best block, increasing speed while ensuring that on-disk state is kept consistent. Witness caches are now never recreated during a -reindex, on the assumption that the blocks themselves are not changing (the chain is just being reconstructed), and so the witnesses will remain valid. Part of zcash#1749. * Add porter dev overrides for CC, CXX, MAKE, BUILD, HOST * Apply miniupnpc patches to enable compilation on Solaris 11 These can be removed after the next MiniUPnP release. Closes zcash#1835. * Closes zcash#1903. Add fee parameter to z_sendmany. * Add an upstream miniupnpc patch revision * Metrics - Don't exclaim unless > 1 "You have validated 0 transactions!" sounds a little less enthusiastic that intended. Also, only says "1 transaction". * Address review comments, tweak strings * bash-completion: Adapt for 0.12 and 0.13 * separate completion for bitcoind and bitcoin-cli * remove RPC support from bitcoind completion * add completion for bitcoin-tx and bitcoin-qt * rely on autoloading of completions * Change function names to not clash with Bitcoin, apply to correct binaries * Add bash completion files to Debian package * Always bash-complete the default account * Add Zcash RPC commands to CLI argument completion * Fixes zcash#1823. Witness anchors for input notes no longer cross block boundaries. * Edit for grammar: "block chain" At this point, I believe it is universally accepted that "blockchain" is one word, and should not be separated into two. * Increase timeout as laptops on battery power have cpu throttling. * Isolate verification to a `ProofVerifier` context object that allows verification behavior to be tuned by the caller. * Regression test. * Ensure cache contains valid entry when anchor is popped. * Ensure ProofVerifier cannot be accidentally copied. * Document behaviour of CWallet::SetBestChain * WitnessAnchorData only needs to store one witness per JSOutPoint. * Rename Dummy to Disabled. * Add more tests for ProofVerifier. * Fix indentation * Generate JS for trydecryptnotes, make number of addresses a variable * Add JS to second block to ensure witnesses are incremented * ASSERT_TRUE -> ASSERT_FALSE * Skip JoinSplit verification before the last checkpoint Part of zcash#1749 * Gather release notes from previous release to HEAD Also update release-process.md to replace git shortlog command with release-notes.py script. * Add a reindex test that fails because of a bug in decrementing witness caches Ref: zcash#1904 (comment) * Make the test pass by fixing the bug! * Only check cache validity for witnesses being incremented or decremented Fixes the bug resulting from zcash#1904. * Update release process to check in with users who opened resolved issues * Check that E' points are actually in G2 by ensuring they are of order r. * Fix bug in wallet tests * Extract block-generation wallet test code into a function * Rewrite reindex test to check beyond the max witness cache size * Fix bug in IncrementNoteWitness() * Extend createjoinsplit to benchmark parallel JoinSplits Closes zcash#1940 * Update payment API docs to recommend -rescan for fixing witness errors * Add total number of commitments to getblockchaininfo * Update version to 1.0.4 * Update man pages * Release notes, authors, changelog * Only enable getblocktemplate when wallet is enabled * Only run wallet tests when wallet is enabled * Add a tool for profiling the creation of JoinSplits * Add test for IncrementalMerkleTree::size(). * Exclude test binaries from make install Closes zcash#1943. * Fixes zcash#1964 to catch general exception in z_sendmany and catch exceptions as reference-to-const. * Fixes zcash#1967 by adding age of note to z_sendmany logging. * Scan the whole chain whenever a z-key is imported Closes zcash#1941. * Instruct users to run zcash-fetch-params if network params aren't available Closes zcash#1786. * Fixes a bug where the unsigned transaction was logged by z_sendmany after a successful sign and send, meaning that the logged hash fragment would be different from the txid logged by "AddToWallet". This issue occured when sending from transparent addresses, as utxo inputs must be signed. It did not occur when sending from shielded addresses. * Trigger metrics UI refresh on new messages * Strip out the SECURE flag in metrics UI so message style is detected * Add 'CreateJoinSplit' standalone utility to gitignore. * Handle newlines in UI messages * Suggest ./zcutil/fetch-params.sh as well Once we improve the from-source installation docs to use 'make install', we can revert this commit. * Update debug categories Closes zcash#1954. * CreateJoinSplit: add start_profiling() call This solves the problem of profiling output displaying nonsensical large time values. * rpc: Implement random-cookie based authentication When no `-rpcpassword` is specified, use a special 'cookie' file for authentication. This file is generated with random content when the daemon starts, and deleted when it exits. Read access to this file controls who can access through RPC. By default this file is stored in the data directory but it be overriden with `-rpccookiefile`. This is similar to Tor CookieAuthentication: see https://www.torproject.org/docs/tor-manual.html.en Alternative to zcash#6258. Like that pull, this allows running bitcoind without any manual configuration. However, daemons should ideally never write to their configuration files, so I prefer this solution. * Rename build-aux/m4/bitcoin_find_bdb48.m4 to remove version Closes zcash#1622. * Bump COPYRIGHT_YEAR from 2016 to 2017. * Throw an error if zcash.conf is missing An empty zcash.conf is sufficient to bypass this error. * Show a friendly message explaining why zcashd needs a zcash.conf * Closes zcash#1780. Result of z_getoperationstatus now sorted by creation time of operation * Create ISSUE_TEMPLATE.md * move template to subdirectory, fix typo, include prompt under describing issue section, include uploading file directly to github ticket as option for sharing logs * Remove UTF-8 BOM efbbbf from zcash.conf to avoid problems with command line tools * Closes zcash#1097 so zcash-cli now displays license info like zcashd. LicenseInfo is refactored from init.cpp to util.cpp so that the bitcoin-cli makefile target does not need to be modified. * Fixes zcash#1497 ZCA-009 by restricting data exporting to user defined folder. Previously the RPC interface allowed z_exportwallet, backupwallet and dumpwallet to write data to an arbitrary filename. ZCA-009 demonstrates how this is vulnerable. The resolution is to only allow data to written when the -exportdir has been configured. Also filenames are restricted to alphanumeric characters. * Closes zcash#1957 by adding tx serialization size to listtransactions output. * Fix gtest ordering broken by zcash#1949 Part of zcash#1539 * Fixes zcash#1960: z_getoperationstatus/result now includes operation details. * Debian package lint - Tweak description synopsis to make Debian happy - Put bash completion files in correct directory - Add a manpage for zcash-fetch-params * Generate Debian control file to fix shlibs lint * Create empty zcash.conf during performance measurements * Create empty zcash.conf during coverage checks Fixes regression caused by zcash#2013. * Coverage build system tweaks * Update walletbackup.py qa test to use -exportdir option * Add missing header required by std::accumulate * Increase timeout for z_sendmany transaction in wallet.py qa test * Add test for z_importkey rescanning from beginning of chain. * Bump version to 1.0.5. * Update release notes and Debian package. * V1.0.4 mac (#51) * initial mac version of zclassic Work in progress - 15JAN2017 more refactoring linux refactoring fixes osx refactoring fixes initial win64 commit fixup! initial win64 commit compile libsnark with posix threads build gtest and gmock with posix Working build fixup! Working build * Windows and Linux builds ok * fixup! Merge tag 'v1.0.5' into v1.0.5-multios * fixup! fixup! Merge tag 'v1.0.5' into v1.0.5-multios * fixup! fixup! fixup! Merge tag 'v1.0.5' into v1.0.5-multios * Fix OSX compatibility with depends * OSX Compat - Fix site_t ambiguity in json * fixup! OSX Compat - Fix site_t ambiguity in json

str4d added I-performance Problems and improvements with respect to performance S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. A-wallet Area: Wallet labels Nov 30, 2016

str4d added this to the 1.0.4 milestone Nov 30, 2016

str4d mentioned this pull request Nov 30, 2016

Do not flush wallet when incrementing or decrementing note witnesses #1891

Closed

str4d mentioned this pull request Nov 30, 2016

daemon with lots of z transactions in larger wallet.dat get out of sync and resyncs extremely slow #1749

Closed

bitcartel reviewed Nov 30, 2016

View reviewed changes

Document behaviour of CWallet::SetBestChain

d85758f

bitcartel mentioned this pull request Dec 8, 2016

Isolate verification to a ProofVerifier context object that allows verification behavior to be tuned by the caller. #1919

Merged

bitcartel mentioned this pull request Dec 8, 2016

Skip JoinSplit verification before the last checkpoint #1892

Merged

str4d added 2 commits December 10, 2016 00:51

Add a reindex test that fails because of a bug in decrementing witnes…

eeee6d5

…s caches Ref: zcash#1904 (comment)

Make the test pass by fixing the bug!

9d2cc3a

zkbot merged commit 9d2cc3a into zcash:master Dec 9, 2016

str4d deleted the 1749-write-witness-cache-with-best-block branch December 10, 2016 01:33

str4d mentioned this pull request Dec 10, 2016

Only check cache validity for witnesses being decremented #1933

Merged

str4d added a commit to str4d/zcash that referenced this pull request Dec 12, 2016

Only check cache validity for witnesses being incremented or decremented

6f78f4f

Fixes the bug resulting from zcash#1904.

str4d mentioned this pull request Dec 13, 2016

Only increment new notes on reindex #1939

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write witness caches when writing the best block #1904

Write witness caches when writing the best block #1904

str4d commented Nov 30, 2016

str4d commented Nov 30, 2016

zkbot commented Nov 30, 2016

zkbot commented Nov 30, 2016

bitcartel left a comment •

edited

Loading

str4d commented Dec 7, 2016

ebfull commented Dec 7, 2016

bitcartel commented Dec 7, 2016 •

edited

Loading

str4d commented Dec 7, 2016

bitcartel commented Dec 7, 2016

bitcartel commented Dec 8, 2016 •

edited

Loading

bitcartel commented Dec 8, 2016

bitcartel commented Dec 9, 2016 •

edited

Loading

str4d commented Dec 9, 2016

bitcartel commented Dec 9, 2016

str4d commented Dec 9, 2016

ebfull commented Dec 9, 2016

zkbot commented Dec 9, 2016

zkbot commented Dec 9, 2016

zkbot commented Dec 9, 2016

bitcartel commented Dec 9, 2016 •

edited

Loading

str4d commented Dec 10, 2016

Write witness caches when writing the best block #1904

Write witness caches when writing the best block #1904

Conversation

str4d commented Nov 30, 2016

str4d commented Nov 30, 2016

zkbot commented Nov 30, 2016

zkbot commented Nov 30, 2016

bitcartel left a comment • edited Loading

Choose a reason for hiding this comment

str4d commented Dec 7, 2016

ebfull commented Dec 7, 2016

bitcartel commented Dec 7, 2016 • edited Loading

str4d commented Dec 7, 2016

bitcartel commented Dec 7, 2016

bitcartel commented Dec 8, 2016 • edited Loading

bitcartel commented Dec 8, 2016

bitcartel commented Dec 9, 2016 • edited Loading

str4d commented Dec 9, 2016

bitcartel commented Dec 9, 2016

str4d commented Dec 9, 2016

ebfull commented Dec 9, 2016

zkbot commented Dec 9, 2016

zkbot commented Dec 9, 2016

zkbot commented Dec 9, 2016

bitcartel commented Dec 9, 2016 • edited Loading

str4d commented Dec 10, 2016

bitcartel left a comment •

edited

Loading

bitcartel commented Dec 7, 2016 •

edited

Loading

bitcartel commented Dec 8, 2016 •

edited

Loading

bitcartel commented Dec 9, 2016 •

edited

Loading

bitcartel commented Dec 9, 2016 •

edited

Loading