Add `available` field to listwallets RPC #11485

meshcollider · 2017-10-11T11:24:45Z

Modifies listwallets to return a list of 'available' wallets in the wallet directory, by looking for the BDB magic bytes (0x00053162 source: https://github.com/file/file/blob/master/magic/Magdir/database) .

This is a breaking change to listwallets, but multiwallet RPC calls are experimental in 0.15 so shouldn't be a big concern to change it. c.f. @jnewbery's comment here. Split from #11466.

Would be great if someone could confirm if this is endianness independent too :)

promag · 2017-10-11T13:40:36Z

As said in #11466 (comment), a different interface is:

{
  "wallets": [
    { "name": "foo", "loaded": true },
    { "name": "bar", "loaded": false }
  ]
}

Which has the advantage of allowing to add new attributes later.

Another option is to have a different RPC. So:

listwallets - returns a lists of active/loaded wallets
findwallets - returns a list of all possible/available wallets.

This has the advantage that each RPC has it's own purpose and implementation, which happen to be very different. For those that manage (keep record of) the wallets externally findwallets is of no interest, only listwallets.

jnewbery · 2017-10-11T14:14:43Z

concept ACK splitting this from #11466. And concept ACK using the BDB magic bytes rather than filename

meshcollider · 2017-10-13T09:10:22Z

Removed WIP, now uses BDB magic bytes, although I haven't confirmed that this works on big endian or also potentially on old versions of BDB wallets if the magic bytes changed

jnewbery

Looks good. Can you extend the checking to exclude symlinks from the available wallets list, and add a test to cover that?

I think it might also be a good idea to exclude currently loaded wallets from the available list (ie the loaded and available lists should be mutually exclusive)

jnewbery · 2017-10-13T21:17:01Z

src/wallet/rpcwallet.cpp

+        file.seekg(12, std::ios::beg); // Magic bytes start at offset 12
+        unsigned char fileBytes[4] = {0};
+        file.read((char*)fileBytes, sizeof(fileBytes)); // Read 4 bytes of file to compare against magic
+        if (is_regular_file(*it) && file.is_open() && memcmp(fileBytes, bdb_magic, sizeof(bdb_magic)) == 0) {


Please explicitly use fs::is_regular_file() (this only compiles because of Argument Dependant Lookups http://en.cppreference.com/w/cpp/language/adl which makes it difficult to understand where this function is coming from).

boost::filesystem::is_regular_file() will return true if called on a symlink to a regular file. We don't want those to show up in the available list since they can't be loaded as a wallet, so you should extend this if test with fs::is_symlink().

jnewbery · 2017-10-13T21:34:53Z

I also tested manually that files show up as wallets when bytes 12-15 match the magic bdb bytes:

→ bitcoin-cli listwallets
{
  "loaded": [
    "wallet.dat"
  ],
  "available": [
    "wallet.dat"
  ]
}
[ubuntu] /home/ubuntu/.bitcoin/regtest
→ bitcoin-cli stop
Bitcoin server stopping
[ubuntu] /home/ubuntu/.bitcoin/regtest
→ for i in {1..20}; do head wallet.dat -c $i > wallet_prefix_$i; done
[ubuntu] /home/ubuntu/.bitcoin/regtest
→ bitcoind
Bitcoin server starting
[ubuntu] /home/ubuntu/.bitcoin/regtest
→ bitcoin-cli listwallets
{
  "loaded": [
    "wallet.dat"
  ],
  "available": [
    "wallet_prefix_20",
    "wallet_prefix_19",
    "wallet.dat",
    "wallet_prefix_16",
    "wallet_prefix_17",
    "wallet_prefix_15",
    "wallet_prefix_18"
  ]
}

meshcollider · 2017-10-13T22:27:03Z

Addressed @jnewbery nits and made available list mutually exclusive to loaded list. Thanks for the review :-)

promag · 2017-10-13T22:40:40Z

src/wallet/rpcwallet.cpp

@@ -2645,10 +2645,38 @@ UniValue listwallets(const JSONRPCRequest& request)

        LOCK(pwallet->cs_wallet);

-        obj.push_back(pwallet->GetName());
+        loaded.push_back(pwallet->GetName());
    }


Suggestion to remove loop below:

std::set<std::string> loaded_set; ... loaded.push_back(pwallet->GetName()); loaded_set.insert(pwallet->GetName()) } ... if (loaded_set.count(it->path().filename().string()) == 0) { available.push_back(it->path().filename().string()); }

meshcollider · 2017-10-13T22:54:19Z

Good idea @promag, thanks :-)

promag · 2017-10-22T00:59:14Z

src/wallet/rpcwallet.cpp

+
+        file.read((char*)fileBytes, sizeof(fileBytes)); // Read 4 bytes of file to compare against magic
+
+        if (fs::is_regular_file(*it) && !fs::is_symlink(*it) && file.is_open() && memcmp(fileBytes, bdb_magic, sizeof(bdb_magic)) == 0) {


Some of these conditions can be moved before seekg (just continue. Only the compare is needed.

Could use 64 bit int instead?

Fixed the conditional, but IMO the char array is clearer and perhaps safer than using an int :)

meshcollider · 2017-10-22T06:41:25Z

Rebased on master to fix travis failure due to multiwallet.py being modified

promag · 2017-10-23T20:58:24Z

src/wallet/rpcwallet.cpp

+            }
+        }
+
+        file.close();


Remove, see https://stackoverflow.com/a/748059.

promag · 2017-10-23T21:02:20Z

src/wallet/rpcwallet.cpp

+        unsigned char fileBytes[4] = {0};
+        file.read((char*)fileBytes, sizeof(fileBytes)); // Read 4 bytes of file to compare against magic
+
+        if (memcmp(fileBytes, bdb_magic, sizeof(bdb_magic)) == 0) {


Alternative, avoid nesting:

if (memcmp(...) != 0) continue; if (loaded_set.count(filename) > 0) continue; available.push_back(filename);

Note that close is automatically called, see comment below.

Nit, add:

const std::string filename = it->path().filename().string();

promag · 2017-10-23T21:04:47Z

src/wallet/rpcwallet.cpp

+        if(!file.is_open()) continue;
+
+        file.seekg(12, std::ios::beg); // Magic bytes start at offset 12
+        unsigned char fileBytes[4] = {0};


Nit: file_bytes.

promag · 2017-10-23T21:04:57Z

src/wallet/rpcwallet.cpp

+    const unsigned char bdb_magic[4] = {0x62, 0x31, 0x05, 0x00}; // Berkeley DB Btree magic bytes
+
+    for (fs::directory_iterator it(walletdir); it != fs::directory_iterator(); ++it) {
+


Nit, remove empty line.

promag · 2017-10-23T21:10:03Z

test/functional/multiwallet.py

@@ -19,7 +19,7 @@ def set_test_params(self):
        self.extra_args = [['-wallet=w1', '-wallet=w2', '-wallet=w3']]

    def run_test(self):
-        assert_equal(set(self.nodes[0].listwallets()), {"w1", "w2", "w3"})
+        assert_equal(set(self.nodes[0].listwallets()['loaded']), {"w1", "w2", "w3"})



Assert available is empty array?

meshcollider · 2017-10-24T00:31:00Z

Fixed @promag's nits, thanks :)

BTW it would be good to have #11466 reviewed first because this is only really useful with that IMO

meshcollider · 2017-11-19T02:49:55Z

Rebased and updated to work with new wallets directory from #11466

maflcko · 2017-11-27T16:55:03Z

Needs rebase, but I assume this won't work after #11687 anyway?

laanwj · 2017-11-28T08:10:34Z

Needs rebase, but I assume this won't work after #11687 anyway?

It won't work if someone uses external wallets, but it should still work if they have all wallets in the wallet directory? (which is the default)

meshcollider · 2017-11-29T00:21:41Z

I've rebased. If #11687 defaults to creating directories for each wallet rather than individual BDB files, then this may need to be modified to list directories as well as BDB files? But yes should still work

jnewbery

Tested ACK 946b2ce. One suggested change for the new test.

jnewbery · 2017-11-29T19:21:54Z

test/functional/multiwallet.py

@@ -63,6 +64,10 @@ def run_test(self):

        self.start_node(0, self.extra_args[0])

+        # ensure listwallets only names BDB files, not symlinks, other files or directories


I like this new test, but I think it can be improved in a couple of ways:

test that a non-empty file without the magic bytes is excluded

there's an implicit assumption that the symlink and folder created earlier in the test are still around. You could explicitly assert that. If the earlier parts of this test get change, eg the file names are changed, then this check will continue to succeed but won't be testing anything.

What do you think about:

--- a/test/functional/multiwallet.py +++ b/test/functional/multiwallet.py @@ -65,8 +65,20 @@ class MultiWalletTest(BitcoinTestFramework): self.start_node(0, self.extra_args[0]) # ensure listwallets only names BDB files, not symlinks, other files or directories - open(os.path.join(wallet_dir, 'w01.dat'), 'a').close() - assert_equal(set(self.nodes[0].listwallets()['available']), {"w22"}) + with open(os.path.join(wallet_dir, 'w01.dat'), 'a') as wallet_file: + # Create a file containing a long string + print("a" * 2000, file=wallet_file) + + available_wallets = self.nodes[0].listwallets()['available'] + + assert os.path.islink(os.path.join(wallet_dir, 'w12')) + assert 'w12' not in available_wallets + + assert os.path.isdir(os.path.join(wallet_dir, 'w11')) + assert 'w11' not in available_wallets + + assert os.path.isfile(os.path.join(wallet_dir, 'w01.dat')) + assert 'w01.dat' not in available_wallets

TheBlueMatt

Code looks good, RPC docs could be more descriptive (and are out-of-date after the change).

TheBlueMatt · 2017-12-04T18:09:58Z

src/wallet/rpcwallet.cpp

-            "  \"walletname\"            (string) the wallet name\n"
-            "   ...\n"
-            "]\n"
+            "{\n"


Need to update the docs above. Maybe also note that false-positives may include files from exceedingly old (pre-0.8, I believe) nodes as well as duplicates of wallets which cannot be opened simultaneously.

meshcollider · 2017-12-12T22:57:47Z

Update the documentation as per @TheBlueMatt suggestion

ryanofsky

utACK 3dac78f

ryanofsky · 2017-12-18T15:05:13Z

doc/release-notes.md

@@ -91,6 +91,10 @@ Low-level RPC changes
 - The wallet RPC `getreceivedbyaddress` will return an error if called with an address not in the wallet.


+- `listwallets` now returns two arrays of wallet names, one listing the `loaded` wallets


I don't think it's a big deal to break backwards compatibility here, but I also don't see any benefit in doing so. I'd think that a better approach to overloading the listwallets function would be to add a new listwalletdir function that would return a list of wallet files inside -walletdir. Advantages to this approach:

Backwards compatible

Better performance for listwallets. It could continue to just return the loaded wallets from memory instead of having now having to do disk io.

More descriptive function name. listwalletdir naming is consistent with recently added walletdir parameter, and tells you what the function actually does.

ryanofsky · 2017-12-18T15:05:44Z

src/wallet/rpcwallet.cpp

-            "]\n"
+            "{\n"
+            "  \"loaded\":     [ \"walletname\" ... ] (json array of strings) Names of loaded wallets\n"
+            "  \"available\":  [ \"walletname\" ... ] (json array of strings) Names of BDB files in wallet directory (may not always actually be wallets)\n"


Would suggest returning an array of objects rather than an array of strings to future proof this. This would allow including other information like sizes, last modification times, whether databases are flushed, or locked by another process. (Latter two would be meaningful after #11687 when wallet files can be opened in their own berkelydb environments instead of all using the same shared environment).

Partially agree with @ryanofsky. Still think one array is enough, see #11485 (comment). And listwallets could have an optional parameter to filter available, loaded or both.

ryanofsky · 2017-12-18T15:25:26Z

src/wallet/rpcwallet.cpp

+    for (fs::directory_iterator it(walletdir); it != fs::directory_iterator(); ++it) {
+        if (!fs::is_regular_file(*it) || fs::is_symlink(*it)) continue;
+
+        std::ifstream file(it->path().string(), std::ios::binary);


Note: to be compatible with #11687, you would drop the continue above and change this line to something like:

std::ifstream file(fs::is_regular_file(*it) ? it->path() : (it->path() / "wallet.dat")).string(), std::ios::binary);

ryanofsky · 2018-01-08T20:50:45Z

Might be ready for merge. Unclear if "Code looks good" from @TheBlueMatt above is equivalent to a code ack.

meshcollider · 2018-01-08T22:11:50Z

@ryanofsky sorry I haven't had time to work on your suggested changes above, I'm happy to let this wait until I've had time to address them

laanwj · 2018-08-31T12:28:27Z

Closing for now w/ up for grabs, let me know when you start working on this and it needs to be reopened.

jnewbery · 2018-08-31T14:20:53Z

Shame this has gone stale. I think it's useful new functionality.

I'd be very happy to re-review if someone picked this up again.

promag · 2018-09-21T22:02:53Z

I'm tempted to pick this as this is also useful in the UI — must be exposed in interfaces::Nodes.

As suggested by @Sjors in #13100 (review)

For opening an existing wallet I have a light preference for just scanning the default directory for wallets and showing a list of wallets to choose from. This also prevents confusion around wallets that are files vs. wallets that are directories. Offering multiple wallets and offering the ability to put wallets anywhere in the filesystem are two different features.

Thoughts?

ListWalletDir returns all available wallets in the current wallet directory. Based on MeshCollider work in pull bitcoin#11485.

d56a068 docs: Add release notes for listwalletdir RPC (João Barbosa) 0cb3cad qa: Add tests for listwalletdir RPC (João Barbosa) cc33773 rpc: Add listwalletdir RPC (João Barbosa) d1b03b8 interfaces: Add getWalletDir and listWalletDir to Node (João Barbosa) fc4db35 wallet: Add ListWalletDir utility (João Barbosa) Pull request description: `ListWalletDir` returns all available wallets in the current wallet directory. Based on MeshCollider work in pull #11485. Tree-SHA512: 5843e3dbd1e0449f55bb8ea7c241a536078ff6ffcaad88ce5fcf8963971d48c78600fbc4f44919523b8a92329d5d8a5f567a3e0ccb0270fdd27366e19603a716

ListWalletDir returns all available wallets in the current wallet directory. Based on MeshCollider work in pull bitcoin#11485.

jarolrod · 2021-05-08T02:41:36Z

@fanquake This shouldn't be Up for grabs anymore. The main idea behind this PR was implemented in #14291.

d56a068 docs: Add release notes for listwalletdir RPC (João Barbosa) 0cb3cad qa: Add tests for listwalletdir RPC (João Barbosa) cc33773 rpc: Add listwalletdir RPC (João Barbosa) d1b03b8 interfaces: Add getWalletDir and listWalletDir to Node (João Barbosa) fc4db35 wallet: Add ListWalletDir utility (João Barbosa) Pull request description: `ListWalletDir` returns all available wallets in the current wallet directory. Based on MeshCollider work in pull bitcoin#11485. Tree-SHA512: 5843e3dbd1e0449f55bb8ea7c241a536078ff6ffcaad88ce5fcf8963971d48c78600fbc4f44919523b8a92329d5d8a5f567a3e0ccb0270fdd27366e19603a716

fanquake added the RPC/REST/ZMQ label Oct 11, 2017

meshcollider mentioned this pull request Oct 11, 2017

Specify custom wallet directory with -walletdir param #11466

Merged

meshcollider changed the title ~~[WIP] Add available field to listwallets RPC~~ Add available field to listwallets RPC Oct 13, 2017

jnewbery reviewed Oct 13, 2017

View reviewed changes

promag reviewed Oct 13, 2017

View reviewed changes

promag reviewed Oct 22, 2017

View reviewed changes

promag reviewed Oct 23, 2017

View reviewed changes

meshcollider added 2 commits November 29, 2017 13:16

Add release notes for listwallets RPC change

87e3923

Add available BDB list to listwallets RPC

946b2ce

laanwj mentioned this pull request Nov 29, 2017

Add -debuglogfile option #11781

Merged

jnewbery reviewed Nov 29, 2017

View reviewed changes

TheBlueMatt reviewed Dec 4, 2017

View reviewed changes

Fix listwallets documentation

3dac78f

ryanofsky reviewed Dec 18, 2017

View reviewed changes

maflcko added the Needs rebase label Jun 6, 2018

laanwj added the Up for grabs label Aug 31, 2018

laanwj closed this Aug 31, 2018

promag mentioned this pull request Sep 21, 2018

gui: Add dynamic wallets support #13100

Closed

promag mentioned this pull request Sep 21, 2018

wallet: Add ListWalletDir utility function #14291

Merged

promag added a commit to promag/bitcoin that referenced this pull request Oct 18, 2018

wallet: Add ListWalletDir utility

fc4db35

ListWalletDir returns all available wallets in the current wallet directory. Based on MeshCollider work in pull bitcoin#11485.

jfhk pushed a commit to jfhk/bitcoin that referenced this pull request Nov 14, 2018

wallet: Add ListWalletDir utility

53a3443

ListWalletDir returns all available wallets in the current wallet directory. Based on MeshCollider work in pull bitcoin#11485.

HashUnlimited pushed a commit to HashUnlimited/chaincoin that referenced this pull request Nov 26, 2018

wallet: Add ListWalletDir utility

7275b87

ListWalletDir returns all available wallets in the current wallet directory. Based on MeshCollider work in pull bitcoin#11485.

laanwj removed the Needs rebase label Oct 24, 2019

fanquake removed the Up for grabs label May 8, 2021

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022


		file.read((char*)fileBytes, sizeof(fileBytes)); // Read 4 bytes of file to compare against magic

		if (fs::is_regular_file(it) && !fs::is_symlink(it) && file.is_open() && memcmp(fileBytes, bdb_magic, sizeof(bdb_magic)) == 0) {

		const unsigned char bdb_magic[4] = {0x62, 0x31, 0x05, 0x00}; // Berkeley DB Btree magic bytes

		for (fs::directory_iterator it(walletdir); it != fs::directory_iterator(); ++it) {

		@@ -63,6 +64,10 @@ def run_test(self):

		self.start_node(0, self.extra_args[0])

		# ensure listwallets only names BDB files, not symlinks, other files or directories

		@@ -91,6 +91,10 @@ Low-level RPC changes
		- The wallet RPC `getreceivedbyaddress` will return an error if called with an address not in the wallet.


		- `listwallets` now returns two arrays of wallet names, one listing the `loaded` wallets

Add available field to listwallets RPC #11485

Add available field to listwallets RPC #11485

Conversation

meshcollider commented Oct 11, 2017 • edited Loading

promag commented Oct 11, 2017 • edited Loading

jnewbery commented Oct 11, 2017

meshcollider commented Oct 13, 2017 • edited Loading

jnewbery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnewbery commented Oct 13, 2017

meshcollider commented Oct 13, 2017

Choose a reason for hiding this comment

meshcollider commented Oct 13, 2017

Choose a reason for hiding this comment

meshcollider Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

meshcollider commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meshcollider commented Oct 24, 2017 • edited Loading

meshcollider commented Nov 19, 2017

maflcko commented Nov 27, 2017

laanwj commented Nov 28, 2017 • edited Loading

meshcollider commented Nov 29, 2017

jnewbery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meshcollider commented Dec 12, 2017

ryanofsky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

promag Dec 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky commented Jan 8, 2018

meshcollider commented Jan 8, 2018

laanwj commented Aug 31, 2018

jnewbery commented Aug 31, 2018

promag commented Sep 21, 2018

jarolrod commented May 8, 2021

Add `available` field to listwallets RPC #11485

Add `available` field to listwallets RPC #11485

meshcollider commented Oct 11, 2017 •

edited

Loading

promag commented Oct 11, 2017 •

edited

Loading

meshcollider commented Oct 13, 2017 •

edited

Loading

meshcollider Oct 22, 2017 •

edited

Loading

meshcollider commented Oct 24, 2017 •

edited

Loading

laanwj commented Nov 28, 2017 •

edited

Loading

promag Dec 18, 2017 •

edited

Loading