Speed-up account iteration

[azero-live]

Dear subxt team,

I'm currently using latest subxt v0.34.0 and I would like to iterate through all existing accounts of the Aleph Zero network. I'm following this example: https://github.com/paritytech/subxt/blob/master/subxt/examples/storage_iterating.rs

Unfortunately, it seems like iterating the accounts is extremely slow.
I know from earlier subxt versions, there was the possibility to specify an additional page_size param to the iter(). Is there another option I'm missing that allows me to do a fast iteration through all the accounts?

Thanks for all the help and efforts.

[jsdw]

Good question!

For a bit of background, Subxt supports using the "old" RPC methods (the default still) or the "new" V2 RPC methods (https://github.com/paritytech/json-rpc-interface-spec/). You can instantiate an OnlineClient with either a LegacyBackend or UnstableBackend to pick which one (LegacyBackend is the default).

With that in mind, the page_size option disappeared because it doesn't make sense when using the new backend (that one delivers storage results in a streaming fashion to Subxt, so there is no concept of a page size, just a continuous stream of results).

That said, I think that we should put a bit of time into testing the speed of the current (legacy) storage iteration and see what we can do to improve it. One option might be to allow the storage_iteration_page_size to be configured as a property of the LegacyBackend itself, for instance. The way we fetch items is also fairly naive and could probably be improved.

If you'd like to be a guinea pig, you could fork subxt and increase the number here and see if it makes things much faster for you:

subxt/subxt/src/backend/legacy/mod.rs

Line 336 in 14b796a

const STORAGE_PAGE_SIZE: u32 = 32;

[azero-live]

Thanks @jsdw for the detailed answer. I think it would be really helpful if a storage_iteration_page_size could be configured for the LegacyBackend until the UnstableBackend is the default. At the moment I have created a subxt fork and customized the STORAGE_PAGE_SIZE as you suggested. Which now appears to make things much faster for me. Thanks again for the help.

[jsdw]

Ah great, it's good to hear that helps a bunch; adding that configuration option to a LegacyBackendBuilder looks like the way to go then here!

[jsdw]

@azero-live I added a config option as suggested above; see #1458 for example usage :)

[azero-live]

Perfect, thank you @jsdw jsdw for the fast response and integration! 😄