Subscribe-able runtime apis

[Ank4n]

Inspired by https://forum.polkadot.network/t/wasm-view-functions/1045/6

Most frontend clients depend on storage based apis for reading state of the chain/pallet. We have runtime-apis that solves this problem partially but they are pull based and there is no nice way for frontend to know when to query the runtime-api again. A common way to implement it would be to still manually subscribe to underlying storage changes (by looking at the source code) that the runtime-api depends on. But this is error prone.

Can we make runtime apis subscription based? On high level, this could work as follows: 1) record all the storage that was touched (via host function) when a runtime api was called, 2) subscribe to all those storage keys, 3) return a response every time any of those storage changes.

This also makes storage just an implementation detail (which is ideally how it should be) and frontends don't have to depend on them. Specifically storage migrations would not break bunch of clients as long as runtime apis are backward compatible.

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/wasm-view-functions/1045/8

[kianenigma]

Can we make runtime apis subscription based? On high level, this could work as follows: 1) record all the storage that was touched (via host function) when a runtime api was called, 2) subscribe to all those storage keys, 3) return a response every time any of those storage changes.

Not familiar with the code but I think this can work. Obviously there are security constraints, but that is up to the developer to know what runtime APIs they are exposing.

It. can be something like state_watch, analogous to state_call which is the one-off version.

[ntn-x2]

+1 to see this happening. I think we have faced numerous times that depending on raw storage elements is very bad across migrations, hence we are considering moving more to versioned runtime APIs, but the lack of subscriptions is one of the downsides of this approach. Workarounds are possible but nothing with first-class support.

[rossbulat]

Would be welcome to see, but a couple of concerns / questions with implementation:

• Afaik runtime APIs are not currently exposed to metadata - is this correct and if so is this problematic and should they be exposed so applications can be made aware of them without hardcoding. This would truly bring them on par with using storage item subscriptions.
• How trivial of an amendment would it be to implement subscriptions on top of the runtime APIs, and do any of the core developers have bandwidth for it given the focus on agile coretime and so on.

[Ank4n]

• Afaik runtime APIs are not currently exposed to metadata - is this correct and if so is this problematic and should they be exposed so applications can be made aware of them without hardcoding. This would truly bring them on par with using storage item subscriptions.

Great point. There is another related issue where this should be tackled.

• How trivial of an amendment would it be to implement subscriptions on top of the runtime APIs, and do any of the core developers have bandwidth for it given the focus on agile coretime and so on.

I don't know. But I hope if we make this issue visible enough for the community and it gets enough support, that is a signal for us to prioritise this.

[jsdw]

Afaik runtime APIs are not currently exposed to metadata

Just to say; V15 metadata does expose the necessary information to work with runtime APIs

[bkontur]

We also have another scenario where a subscribe-able runtime API would be preferable over reading raw storage - see comment.

[xlc]

I am pretty sure it is not hard to collect all the accessed storages and prefixes while executing a runtime API and then we just need to subscribe to all of those storages and prefixes. This requires a new RPC call so I think will require an update to the spec https://github.com/paritytech/json-rpc-interface-spec

[jsdw]

When there is some agreement on what we want to see, perhaps one could follow an approach like this to get it implemented and stabilised:

1. Implement underlying code in Substrate.
2. Expose some new RPC method (nothing specified, just something to allow people to start playing with it eg state_watch as Kian suggested).
3. When happy with it, open an issue in https://github.com/paritytech/json-rpc-interface-spec to add a specified API endpoint. (If we got here quickly, the newmethod could be part of the v1 stabilised API, but otherwise we'd bump to v2)
4. We'd want to update metadata too; we know about runtime APIs but would want to know which of these can/are subscription based (maybe some/many calls can be both?) so that we can generate nice interfaces for it.
   • First step here could be to add some information to the "custom metadata" field. This can be done easily and is a good way of trialing things in general.
   • Next we can add it to the set of changes we'll want in V16 metadata when we start thinking about that (this might be in Q3-Q4 offhand). This step will take longer.

Me and the product eng rust devs can probably help out once we get to step 3 and get the RPC/metadata stuff done :)

[xlc]

If done correctly, all runtime API can be subscripted. The only consideration is resource usages. But that's a different issue #1628

[bkchr]

We'd want to update metadata too; we know about runtime APIs but would want to know which of these can/are subscription based (maybe some/many calls can be both?) so that we can generate nice interfaces for it.

* First step here could be to add some information to the ["custom metadata" field](https://github.com/paritytech/frame-metadata/blob/main/frame-metadata/src/v15.rs#L62). This can be done easily and is a good way of trialing things in general.

* Next we can add it to the set of changes we'll want in V16 metadata when we start thinking about that (this might be in Q3-Q4 offhand). This step will take longer.

I don't see why this should be added to the metadata? Basically every runtime api could be subscription based.

Generally for the feature, I'm not sure we should add it. At least in the current form I don't see how it could work for light clients without overloading full nodes. Or does someone has an idea on how to implement this for light clients? Reasoning behind this is that we should not invest time into stuff that will not work with light clients.

[xlc]

This is how it should be implemented, which I think can be done by light clients but let me know otherwise

• Run the runtime API
• Record all the accessed storages
• Subscribe to all those storages
• Rerun the runtime API if any of the subscribed storages are updated
• Check if the return value of the runtime API is changed, and notify clients if needed
• Update storage subscriptions if changed
• Repeat

[jsdw]

I don't see why this should be added to the metadata? Basically every runtime api could be subscription based.

Ah yup, if every API is subscription based then no need to change anything on the metadata side! I'd assumed that it would be opt-in whether a runtime API would be subscribable or not

[bkchr]

This is how it should be implemented, which I think can be done by light clients but let me know otherwise

I mean this is the way I would also have done it. However, what you propose requires that the light client re-executes the runtime api for every imported block. Which means that it needs to send the request to a full node to execute the runtime api, that needs to send a proof etc. This sounds quite expensive to me?

[xlc]

it only re-execute it if any accessed storages are modified so for simple use case like query balances, it will rarely re-executed
if the runtime API is for example access block number, then it will be executed every single block but what prevents light clients spamming such requests to full node currently? the same prevention mechanism should be involved

[bkchr]

it only re-execute it if any accessed storages are modified so for simple use case like query balances, it will rarely re-executed

But the light client doesn't know which storage items are modified without querying them.

the same prevention mechanism should be involved

Good point, that I also wanted to bring up. This requires some proper design. Currently we prevent spam by sequentially processing the request, but this is clearly not a really good protection.

[xlc]

can light client subscribe for storage modification from fullnode?
I guess we need another issue to discuss DoS prevention mechanism for fullnode with p2p protocol?

[bkchr]

can light client subscribe for storage modification from fullnode?

No. Back in the early days there was the changes trie, but it was removed because it was never really working or something along these lines.

I guess we need another issue to discuss DoS prevention mechanism for fullnode with p2p protocol?

Yes. I will create one.

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/wasm-view-functions/1045/19

[kianenigma]

re. light client risks, I think a full node should be able to turn off its ability to respond to subscribe-able runtime apis if it wishes to, or allow some kind of rate limiting on them.

I would then leave state_call as-is, and add a new state_subscribe, and implemented exactly as @xlc proposed above.

[jsdw]

I would then leave state_call as-is, and add a new state_subscribe

When settled on the approach we should also add an issue to https://github.com/paritytech/json-rpc-interface-spec to propose the addition of chainHead_v1_subscribe for Smoldot to implement so that the feature is accessible through the light client.

[athei]

I would then leave state_call as-is, and add a new state_subscribe

When settled on the approach we should also add an issue to https://github.com/paritytech/json-rpc-interface-spec to propose the addition of chainHead_v1_subscribe for Smoldot to implement so that the feature is accessible through the light client.

With the removal of the changes trie light clients will essentially just poll the view function every block when using this RPC? Since they cannot subscribe to storage items.

[bkchr]

With the removal of the changes trie light clients will essentially just poll the view function every block when using this RPC? Since they cannot subscribe to storage items.

Yeah exactly. I mean we could probably use the closest merkle value feature. However, it would still require that the light client is requesting proofs for every of these nodes.

[athei]

Thanks for clarifying. Shouldn't influence the design of how we subscribe to runtime APIS though. It will work on full nodes as described by @xlc and on light clients we can't do much anyways besides polling. But we probably should fix this later down the road. With block times decreasing it doesn't seem viable to poll every block?

[bkchr]

With block times decreasing it doesn't seem viable to poll every block?

Even without low block times, I would say that the operation is already quite expensive. For a single client probably not, but the more connections you have potentially more of these subscriptions.

[athei]

I would imagine a single Dapp connected to a single chain could have a lot of view function subscriptions already. The light client should be able to at least batch the storage requests of different view functions together in this case. But yeah it's still bad.

Shouldn't change the plan for view functions though. When we implement a solution to allow light clients to subscribe to storage it should just improve view functions without breaking user code.

But I think at least we have higher conviction now that we want subscribeable view functions as a way to update your Dapp. Before we kinda thought maybe events are a way to monitor the chain. This seems to be settled now.

So we should maybe start looking into solutions to allow light clients to better monitor storage items? Maybe just adding a big bloom filter over the touched storage keys to the block header?

[bkchr]

I would imagine a single Dapp connected to a single chain could have a lot of view function subscriptions already. The light client should be able to at least batch the storage requests of different view functions together in this case. But yeah it's still bad.

I actually wanted to bring the point that for normal full nodes this is probably already quite expensive.

So we should maybe start looking into solutions to allow light clients to better monitor storage items? Maybe just adding a big bloom filter over the touched storage keys to the block header?

Probably needs to be relative big? I remember some discussion around transaction inclusion bloom filter to speed up the checking of light clients if a transaction got included and that was not that small. The amount of changed keys is much higher than the number of transactions, which should lead to a much bigger bloom filter.