Add compact block filters blockchain feature using the Nakamoto client #207

thunderbiscuit · 2022-10-13T13:23:00Z

Work in Progress.

Fixes #118.

Description

This PR adds the compact block filters blockchain option.

Notes to the reviewers

Not ready for merge yet. Lots to be tested still.

Questions I still want to answer:

How much size does it add to the final binary we ship for Android/iOS?
Full testing on testnet/mainnet

Changelog notice

## Added Features
    - Compact Block Filters client ([Nakamoto]) [#207]

[Nakamoto](https://github.com/cloudhead/nakamoto)
[#207](https://github.com/bitcoindevkit/bdk-ffi/pull/207)

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

thunderbiscuit · 2022-10-13T14:38:54Z

Making progress!

This is the error I get when attempting to sync a testnet wallet on an Android emulator:

org.bitcoindevkit.BdkException$Cbf: Cbf(Nakamoto(Handle(Disconnected)))

2022-10-13 10:35:11.932 28544-28544 DevkitWallet            com.goldenraven.devkitwallet         I  Wallet is syncing
2022-10-13 10:35:11.965 28544-28544 AndroidRuntime          com.goldenraven.devkitwallet         E  FATAL EXCEPTION: main
                                                                                                    Process: com.goldenraven.devkitwallet, PID: 28544
                                                                                                    org.bitcoindevkit.BdkException$Cbf: Cbf(Nakamoto(Handle(Disconnected)))
                                                                                                    	at org.bitcoindevkit.FfiConverterTypeBdkError.read(bdk.kt:2511)
                                                                                                    	at org.bitcoindevkit.FfiConverterTypeBdkError.read(bdk.kt:2466)
                                                                                                    	at org.bitcoindevkit.FfiConverter$DefaultImpls.liftFromRustBuffer(bdk.kt:153)
                                                                                                    	at org.bitcoindevkit.FfiConverterRustBuffer$DefaultImpls.liftFromRustBuffer(bdk.kt:165)
                                                                                                    	at org.bitcoindevkit.FfiConverterTypeBdkError.liftFromRustBuffer(bdk.kt:2466)
                                                                                                    	at org.bitcoindevkit.FfiConverterTypeBdkError.liftFromRustBuffer(bdk.kt:2466)
                                                                                                    	at org.bitcoindevkit.FfiConverterRustBuffer$DefaultImpls.lift(bdk.kt:166)
                                                                                                    	at org.bitcoindevkit.FfiConverterTypeBdkError.lift(bdk.kt:2466)
                                                                                                    	at org.bitcoindevkit.FfiConverterTypeBdkError.lift(bdk.kt:2466)
                                                                                                    	at org.bitcoindevkit.BdkException$ErrorHandler.lift(bdk.kt:2462)
                                                                                                    	at org.bitcoindevkit.BdkException$ErrorHandler.lift(bdk.kt:2461)
                                                                                                    	at org.bitcoindevkit.Wallet.sync(bdk.kt:3163)
                                                                                                    	at com.goldenraven.devkitwallet.data.Wallet.sync(Wallet.kt:133)
                                                                                                    	at com.goldenraven.devkitwallet.ui.wallet.WalletViewModel.updateBalance(HomeScreen.kt:49)

thunderbiscuit · 2022-10-14T15:37:59Z

I decided to try it on my desktop using bdk-jvm and a small Kotlin cli wallet I made a while back. It seems to be syncing something so far, so that's good. But as a side note the progress float returned seem to be going up and down. Sometimes in the 60s sometimes in the 40s, back and forth.

thunderbiscuit · 2022-10-14T15:39:53Z

Oh darn it worked! Took about 3-4 minutes and the wallet is displaying the correct balance for the descriptor. 🔥 🔥 🔥

thunderbiscuit · 2022-10-14T16:00:19Z

Ok so so far here is what is happening:

On desktop and using bdk-jvm the new blockchain appears to be working 🚀
On Android not at all, throwing this error: Cbf(Nakamoto(Handle(Disconnected))) (both in the emulator and on a physical device, same error)

The odd thing is that these two libraries are pretty much duplicates of each other. So the issue is definitely coming from something on Android, and most likely not from the FFI layer or uniffi-rs. I could try testing with Python and see what happens, but I suspect it will work just as well as with the jvm library.

@cloudhead @rajarshimaitra any ideas? I tried looking through the Nakamoto codebase but couldn't quite pick out the Handle(Disconnected) error and what could be triggering it.

rajarshimaitra · 2022-10-15T05:14:14Z

Thanks @thunderbiscuit .. Thats super progress.. Although I am as clueless about the HandleDisconnect error.. Just making some wild guess.. The sync process starts by spawning a separate thread that runs the nakamoto client process in background.. The HandleDisconnect could mean that for some reason that thread is dropped or terminated and the main bdk thread cannot communicate with it anymore..

Is that something that happens on android? Does it require special consideration for spawning threads?

cloudhead · 2022-10-15T09:32:09Z

Thanks @thunderbiscuit .. Thats super progress.. Although I am as clueless about the HandleDisconnect error.. Just making some wild guess.. The sync process starts by spawning a separate thread that runs the nakamoto client process in background.. The HandleDisconnect could mean that for some reason that thread is dropped or terminated and the main bdk thread cannot communicate with it anymore..

Is that something that happens on android? Does it require special consideration for spawning threads?

This is likely what's going on.

thunderbiscuit · 2022-10-16T15:47:58Z

Ok good stuff guys, thanks for chiming in. I'll need to dig a bit more into this. It means that somehow the connection between the native binary and the glue code is handled differently on Android than it is server-side, and I don't know that we have encountered that before. My next step will be to make sure it works in the unit tests (which run on the dev machine and not in the emulator/device).

Side note, I tested with the Python library and it worked right off the bat.

thunderbiscuit · 2022-10-17T13:32:18Z

Folks it looks like we have liftoff!

The issue was that I had given a default value of ./nakamoto for the datadir field of the CBFConfig in the ffi layer, and in my tests on desktop/server/mobile I left that field null so it was using the default path. This worked on the server because I called my test wallet from a directory where I had write permission.

On Android however, the process is not started from a place where you have write permission (and I haven't quite figured out where this default path leads to the attempted directory creation but it doesn't really matter for now), and so that default path simply doesn't create the .nakamoto directory (I'd be nice to get a specific error for that maybe). Subsequent sync calls would then fail. The trick was to provide a valid path for the app to write the directory in. On Android this looks like

val path: String = applicationContext.filesDir.toString()
val nakamotoConfig = CbfConfig(
    Network.TESTNET,
    "$path/bdk-nakamoto",
)

I suspect the easiest way to deal with this in the short run is to make the datadir field required on the FFI layer (as per my latest commit), but there might be other ways as well.

A few notes for myself/future reference:

On the server a full sync of testnet was about 300-400 MB for the 3 files (filters.db, headers.db, peers.json)
The process times out on the UI thread (no surprise there), but that means we end up launching it in a coroutine, and that's where we can get into tricky situations (A few concurrency-related issues Block has faced while using bdk-ffi #205)
The size of the native binary file of bdk-android 0.9.0 for the arm64-v8a architecture is 9.8MB. When building Nakamoto in it becomes 21.0MB.

cloudhead · 2022-10-17T20:08:28Z

Great work!

In terms of catching the error, is there something nakamoto could have done better? I suspect there would have been an error logged, though I recently reworked all the logging because there were some issues in 0.3.0.

On your notes:

Mainnet should take about 100 MB total disk space, since there are about 3x less blocks compared to testnet.
Since cloudhead/nakamoto@1c5afc9, the block header/filter loading is interruptible, though I'm still working on the API
I'm curious about the binary size, since the nakamoto-node which contains everything that would be imported into bdk is only 5.7MB on my system (release mode).

andreasgriffin · 2023-04-13T19:13:51Z

I tried to get CBF working in python, and was successfull on mainnet (and regtest) ( see diff on my branch https://github.com/andreasgriffin/bdk-ffi/commits/compact_filters2 )

Even better: It safes the latest tip and does not redownload everything again

andreasgriffin · 2023-04-14T04:16:44Z

I really like CBF because it has good privacy, while also offering great speed.

@thunderbiscuit : Does it makes sense to develop https://github.com/andreasgriffin/bdk-ffi/commits/compact_filters2 into a PR right now, even though bdk has a major restructuring ahead?

thunderbiscuit · 2023-04-14T11:01:41Z

Hey thanks for taking a look!

Instead of making a new PR on this repo directly, I think it might make sense to keep this PR up to date (it's quite old at this point) so that it's easier to review and keep up to date in the future. Would you mind making your PR on my fork instead? Your contributions will still be shown because your commits will actually show up on this PR once I merge your PR on my branch.

I'll rebase this (it's about 500 commits behind) and everything will break, but would you be interested in opening a PR on my cbf branch with your working implementation?

To be transparent with you I will not actually consider merging this in the FFI layer until the 1.0 release of BDK, which will impact significantly how sync is done (and so you can expect sizeable changes to the compact block filter crate as well). But for anyone who would like to experiment with it and for testing the advances made by the Rust BDK team on CBF it would be great to have this PR fairly up-to-date and ready to play with.

thunderbiscuit · 2023-04-14T11:06:25Z

Ok I've rebased the changes. If you PR on my CBF branch I'll take the time to review and merge there, so we can get this PR in working order!

thunderbiscuit · 2023-06-22T17:28:43Z

I'm closing this in favour or #379.

thunderbiscuit self-assigned this Oct 13, 2022

thunderbiscuit force-pushed the cbf branch from 8d0092c to dca9209 Compare October 13, 2022 13:53

thunderbiscuit mentioned this pull request Oct 13, 2022

nakamoto as Compact Block Filter Blockchain bitcoindevkit/bdk#751

Closed

3 tasks

thunderbiscuit mentioned this pull request Oct 17, 2022

Mobile integration cloudhead/nakamoto#87

Open

This was referenced Oct 17, 2022

Add compact block filters blockchain support #118

Open

W42 BDK Language Bindings Dev Call bitcoindevkit/.github#28

Closed

thunderbiscuit force-pushed the cbf branch 2 times, most recently from 5b1f9b6 to 427e12b Compare April 14, 2023 11:04

This was referenced Apr 14, 2023

Working CBF with bdk master branch thunderbiscuit/bdk-ffi#5

Closed

Working CBF with bdk master branch thunderbiscuit/bdk-ffi#6

Closed

thunderbiscuit and others added 5 commits June 22, 2023 11:43

Add compact block filters blockchain feature using the Nakamoto client

62946b2

generating bindings OK with compact filters

19d71c8

addres 1 address

6b6d154

Add multiple peer addresses

2c35b4a

Added skip_blocks

3a33297

thunderbiscuit force-pushed the cbf branch from 427e12b to 3a33297 Compare June 22, 2023 15:49

andreasgriffin mentioned this pull request Jun 22, 2023

[DO NOT MERGE] CBF support #379

Closed

5 tasks

thunderbiscuit closed this Jun 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add compact block filters blockchain feature using the Nakamoto client #207

Add compact block filters blockchain feature using the Nakamoto client #207

thunderbiscuit commented Oct 13, 2022 •

edited

Loading

thunderbiscuit commented Oct 13, 2022 •

edited

Loading

thunderbiscuit commented Oct 14, 2022

thunderbiscuit commented Oct 14, 2022

thunderbiscuit commented Oct 14, 2022 •

edited

Loading

rajarshimaitra commented Oct 15, 2022

cloudhead commented Oct 15, 2022

thunderbiscuit commented Oct 16, 2022

thunderbiscuit commented Oct 17, 2022 •

edited

Loading

cloudhead commented Oct 17, 2022

andreasgriffin commented Apr 13, 2023 •

edited

Loading

andreasgriffin commented Apr 14, 2023

thunderbiscuit commented Apr 14, 2023

thunderbiscuit commented Apr 14, 2023

thunderbiscuit commented Jun 22, 2023

Add compact block filters blockchain feature using the Nakamoto client #207

Add compact block filters blockchain feature using the Nakamoto client #207

Conversation

thunderbiscuit commented Oct 13, 2022 • edited Loading

Description

Notes to the reviewers

Changelog notice

Checklists

All Submissions:

New Features:

thunderbiscuit commented Oct 13, 2022 • edited Loading

thunderbiscuit commented Oct 14, 2022

thunderbiscuit commented Oct 14, 2022

thunderbiscuit commented Oct 14, 2022 • edited Loading

rajarshimaitra commented Oct 15, 2022

cloudhead commented Oct 15, 2022

thunderbiscuit commented Oct 16, 2022

thunderbiscuit commented Oct 17, 2022 • edited Loading

cloudhead commented Oct 17, 2022

andreasgriffin commented Apr 13, 2023 • edited Loading

andreasgriffin commented Apr 14, 2023

thunderbiscuit commented Apr 14, 2023

thunderbiscuit commented Apr 14, 2023

thunderbiscuit commented Jun 22, 2023

thunderbiscuit commented Oct 13, 2022 •

edited

Loading

thunderbiscuit commented Oct 13, 2022 •

edited

Loading

thunderbiscuit commented Oct 14, 2022 •

edited

Loading

thunderbiscuit commented Oct 17, 2022 •

edited

Loading

andreasgriffin commented Apr 13, 2023 •

edited

Loading