Fetching events from blocks previous than the runtime upgrade Kusama/9320 is broken

[paulormart]

After the last runtime upgrade on Kusama/9320 is no longer possible to fetch events from blocks before the runtime upgrade.

When fetching for the events at block 15426015 where the runtime upgrade was enacted we get the following error:

Error: Codec(Error { cause: None, desc: "out of range decoding Compact<u32>" })

For any block number smaller than 15426015 we get the following error:

Error: DecodeValue(VariantNotFound(161, TypeDefVariant { variants: [Variant { name: "Normal", fields: [], index: 0, docs: [] }, Variant { name: "Operational", fields: [], index: 1, docs: [] }, Variant { name: "Mandatory", fields: [], index: 2, docs: [] }] }))

Below is just the example to demonstrate the error when fetching a block:

use futures::StreamExt;
use sp_keyring::AccountKeyring;
use std::time::Duration;
use subxt::{rpc::BlockNumber, tx::PairSigner, OnlineClient, PolkadotConfig};

#[subxt::subxt(
    runtime_metadata_url = "wss://kusama-rpc.polkadot.io:443",
    derive_for_all_types = "PartialEq, Clone"
)]
pub mod polkadot {}

/// Subscribe to all events, and then manually look through them and
/// pluck out the events that we care about.
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Create a client to use:
    // let api = OnlineClient::<PolkadotConfig>::new().await?;
    let api =
        OnlineClient::<PolkadotConfig>::from_url("wss://kusama-rpc.polkadot.io:443")
            .await?;

    let block_number: u32 = 15426014;
    let block_hash = api.rpc().block_hash(Some(block_number.into())).await?;
    let events = api.events().at(block_hash).await?;
    if let Some(event) =
    events.find_first::<polkadot::system::events::ExtrinsicSuccess>()?
    {
        println!("{:?}", event);
    }

    Ok(())
}

Output:

Error: DecodeValue(VariantNotFound(161, TypeDefVariant { variants: [Variant { name: "Normal", fields: [], index: 0, docs: [] }, Variant { name: "Operational", fields: [], index: 1, docs: [] }, Variant { name: "Mandatory", fields: [], index: 2, docs: [] }] }))

Is there a workaround for this? Or do you think this could be fixed within subxt?

Thank you

[lexnv]

Hi,

The decoding error from your example is coming from the find_first.

The metadata changed between the block number 15426014(hash 0x925eea1b3a1944fb592aa26b4e41c0926921d2e289a932942d6267a038cbcbce) and the latest chain head.

Running the example with trace logs enabled:

2022-11-22T11:09:17.466776Z DEBUG subxt::events::events_type: Decoding Event 'System::ExtrinsicSuccess'

The old metadata contains:

id 20: DispatchInfo
 id 21: DispatchClass

While the latest metadata:

id 21: DispatchInfo
  id 22: DispatchClass

Adding some debug logs the decoding error originates from:

EventFieldMetadata { name: Some("dispatch_info"), type_name: Some("DispatchInfo"), type_id: 21 }

subxt/subxt/src/events/events_type.rs

Lines 191 to 200 in 15ffbb6

	// Skip over the bytes belonging to this event.
	for field_metadata in event_metadata.fields() {
	// Skip over the bytes for this field:
	scale_decode::decode(
	input,
	field_metadata.type_id(),
	&metadata.runtime_metadata().types,
	scale_decode::visitor::IgnoreVisitor,
	)?;
	}

[lexnv]

From the event documentation, the events at older blocked are not entirely supported

subxt/subxt/src/events/events_client.rs

Lines 43 to 48 in 15ffbb6

	/// # Warning
	///
	/// This call only supports blocks produced since the most recent
	/// runtime upgrade. You can attempt to retrieve events from older blocks,
	/// but may run into errors attempting to work with them.
	pub fn at(

To unblock this, have created patch #727 to sync the metadata with the given block hash.

With the prior patch the dynamic events are properly decoded and can be inspected as:

for event in events.iter() {
  let event = event?;
  println!(
    "pallet_name: {} variant_name: {}",
    event.pallet_name(),
    event.variant_name()
  );

However you will have to make sense of the EventDetails bytes and filter manually for
the interesting event, as the polkadot::system::events::ExtrinsicSuccess was generated
with an incompatible metadata from the decoding standpoint.

Let us know if we could offer further help here.

[paulormart]

@lexnv many thanks for the help, the patch is working great!

[jsdw]

One reason I've been pretty hesitant about introducing this sort of change (and instead just warning users) is that we don't have a sane way (afaik) to find out which metadata version would apply for some range of blocks and download it. Instead, we have to do what the linked PR does and download metadata doe every single block requested, when an explicit hash is given, just incase it is different. This of course causes a huge amount of extra overhead.

I'd rather that for now, we think of a way to help facilitate this use case without adding that sort of overhead per block fetched. This might simply mean that, to do this sort of thing, you have to manually obtain older metadata and pass it in to the events stuff or something, and perhaps we need to tweak some APIs to make that possible :)

However you will have to make sense of the EventDetails bytes and filter manually for
the interesting event, as the polkadot::system::events::ExtrinsicSuccess was generated
with an incompatible metadata from the decoding standpoint.

And this sort of limitation is the sort of thing that shouldn't exist in a public API that we provide either, I reckon. So let's make it possible but not necessarily "easy" to do this (to avoid making such tradeoffs in the "common path")

[paulormart]

Sorry, but actually when tested earlier all blocks before the runtime upgrade are fetched correctly but it still fails on the block 15426015 which is the one that contains the Democracy::PreimageUsed event and not Democracy::Seconded as per subscan.

current error at block 15426015:

block_number: 15426015
block_hash: Some(0xb75d774aa07914df01283dc8d1604aee58d704124f5413651ffafb3a25c379ec)
2022-11-22T16:21:05.281377Z DEBUG subxt::events::events_type: Decoding Event 'Democracy::Seconded'
Error: Codec(Error { cause: None, desc: "out of range decoding Compact<u32>" })
[Finished running. Exit status: 1]

example:

use sp_keyring::AccountKeyring;
use std::time::Duration;
use subxt::{rpc::BlockNumber, tx::PairSigner, OnlineClient, PolkadotConfig};

// runtime_metadata_url = "wss://kusama-rpc.polkadot.io:443",

#[subxt::subxt(
    runtime_metadata_path = "../artifacts/kusama_metadata_9320.scale",
    derive_for_all_types = "PartialEq, Clone"
)]
pub mod polkadot {}

/// Subscribe to all events, and then manually look through them and
/// pluck out the events that we care about.
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Create a client to use:
    // let api = OnlineClient::<PolkadotConfig>::new().await?;
    let api = OnlineClient::<PolkadotConfig>::from_url("wss://kusama-rpc.polkadot.io:443").await?;

    // polkadot::validate_codegen(&api)?;
    let mut optional: Option<u32> = Some(15426014);
    while let Some(block_number) = optional {
        if block_number > 15440452 {
            optional = None;
        } else {
            println!("block_number {block_number}");
            let block_hash = api.rpc().block_hash(Some(block_number.into())).await?;
            let events = api.events().at(block_hash).await?;
            for event in events.iter() {
                let event = event?;
                println!(
                  "pallet_name: {} variant_name: {}",
                  event.pallet_name(),
                  event.variant_name()
                );
            }
            optional = Some(block_number + 1);
        }
    }

    Ok(())
}

[lexnv]

That behavior is strange indeed.

I believe for that exact block # 15426015 decoding should work with an immediate earlier hash:

let block_hash_md = api.rpc().block_hash(Some((block_number - 1).into())).await?.expect("valid num; qed");
let metadata = api.rpc().metadata(Some(block_hash_md)).await?

let events = Events::new_from_client(metadata, block_hash, api.clone()).await?;

The previous example is updated to reflect the changes in #727.

[paulormart]

That's great! Thanks again for the quick support.

For me this new API works well, it will be nice to leave it available in future subxt releases.

output:

...
block_number: 15048375
block_hash: 0x4342991c16f9582321e43b8b4f80b89572ff894c7c9351cc857593f265d4e8c2
2022-11-22T21:32:47.196728Z DEBUG subxt::events::events_type: Decoding Event 'Democracy::PreimageUsed'
pallet_name: Democracy variant_name: PreimageUsed
2022-11-22T21:32:47.196775Z DEBUG subxt::events::events_type: Decoding Event 'System::CodeUpdated'
pallet_name: System variant_name: CodeUpdated
2022-11-22T21:32:47.196784Z DEBUG subxt::events::events_type: Decoding Event 'Democracy::Executed'
...

example:

use futures::StreamExt;
use sp_keyring::AccountKeyring;
use std::time::Duration;
use subxt::{
    events::Events, rpc::BlockNumber, tx::PairSigner, OnlineClient, PolkadotConfig,
};

// runtime_metadata_url = "wss://kusama-rpc.polkadot.io:443",

#[subxt::subxt(
    runtime_metadata_path = "../artifacts/kusama_metadata_9320.scale",
    derive_for_all_types = "PartialEq, Clone"
)]
pub mod polkadot {}

/// Subscribe to all events, and then manually look through them and
/// pluck out the events that we care about.
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Create a client to use:
    // let api = OnlineClient::<PolkadotConfig>::new().await?;
    let api =
        OnlineClient::<PolkadotConfig>::from_url("wss://kusama-rpc.polkadot.io:443")
            .await?;

    // polkadot::validate_codegen(&api)?;
    let mut optional: Option<u32> = Some(15048374);
    while let Some(block_number) = optional {
        if block_number > 15048375 {
            optional = None;
        } else {
            if let Some(block_hash) =
                api.rpc().block_hash(Some(block_number.into())).await?
            {
                println!("block_number: {block_number}");
                println!("block_hash: {:?}", block_hash);
                let metadata = api.rpc().metadata(Some(block_hash)).await?;
                let events =
                    Events::new_from_client(metadata, block_hash, api.clone()).await?;
                for event in events.iter() {
                    let event = event?;
                    println!(
                        "pallet_name: {} variant_name: {}",
                        event.pallet_name(),
                        event.variant_name()
                    );
                }
            }
            optional = Some(block_number + 1);
        }
    }

    Ok(())
}

[paulormart]

I was clearly looking into a different block in my previous comment. The exception seems to be only with block 15426015 where events are only decoded correctly if metadata is previously fetched from the previous block, i'll creation an exception for this case in my end.

Thank you

[lexnv]

Glad to hear that solves your problem!

Let us know if there is anything else we could help with!