state_getKeysPaged Returns duplicate keys

[lexnv]

Using smoldot 0.16 and smoldot-light 0.14 (Updating to latest blocked by #1638 since subxt uses currently the legacy RPC methods)

Repro case:

• Using subxt from this branch: https://github.com/paritytech/subxt/tree/lexnv/repro-duplicate-keys
• cd subxt && RUST_LOG="trace,polling=info,async_io=info,sync-service-polkadot=info,soketto=info" cargo run --example light_client_parachains --features="unstable-light-client"

logs.txt

cc paritytech/subxt#1453

[tomaka]

I'm not sure I understand your logs, but as far as I understand:

• You call state_getKeysPaged with a count parameter equal to 10
• It returns 10 keys
• You then call state_getKeysPaged a second time and pass for the start_key parameter a value equal to the last key that was yielded
• The first key that this second call to state_getKeysPaged yields is identical to the start_key, in other words equal to the last key of the previous call

If so, this behavior is actually "intended" by smoldot.
I put "intended" in quotes because I do this on purpose, but since state_getKeysPaged doesn't have a spec it's not clear to me what it should actually do.

See TODO:

smoldot/light-base/src/json_rpc_service/background/state_chain.rs

Line 716 in 7b63a69

.filter(|k| start_key.as_ref().map_or(true, |start| *k >= start.0)) // TODO: not sure if start should be in the set or not?

[jsdw]

FWIW Substrate ignores the start_key provided for this API; I (somewhat painstakingly) traced the logic of the Substrate impl to some code where it constructs some KeysIter thing and configures the args there explicitly to ignore it.

I made a small fix in Subxt to ignore such a key if it's returned, since we had a couple of issues raised about this. Would you be up for a PR in Smoldot to skip over the start_key to line up with Substrate?

[tomaka]

I was simply waiting for someone's confirmation that Substrate indeed skips the first key in order to open the PR. You could have simply commented here instead of adding a hack in SubXT (paritytech/subxt#1534).

[jsdw]

Awesome, thank you for the quick fix!

You could have simply commented here instead of adding a hack in SubXT

Since the fix was super small in Subxt I figured I may as well apply it there since we are a little out of date on Smoldot version anyway, so it'll give us a little breathing room to update it!