Help implementing data format with different null representations

[lukesteensen]

I'm trying to implement a data format for the Kafka protocol but getting stuck on nullable strings and arrays:

bytes, string - These types consist of a signed integer giving a length N followed by N bytes of content. A length of -1 indicates null. string uses an int16 for its size, and bytes uses an int32.

This seems simple enough, but implies that deserialize_option and serialize_none need to know about the inner type in order to know whether length should be i16 or i32. Is there some way to get this information that I'm missing?

Thanks!

[dtolnay]

I would recommend not using the default Serde impls for Option<T>. Instead you could provide serialize_with and deserialize_with functions for Kafka nullable types.

#[derive(Serialize, Deserialize)]
struct LukeSteensen {
    /* ... */

    #[serde(with = "kafka::nullable")]
    maybe_string: Option<String>,

    #[serde(with = "kafka::nullable")]
    maybe_bytes: Option<Vec<u8>>,
}

The implementation would be something like:

pub mod nullable {
    pub trait KafkaNullable {
        fn nullable_newtype() -> &'static str;
    }

    impl KafkaNullable for String {
        fn nullable_newtype() -> &'static str { "kafka::string" }
    }
    // and other impls

    pub fn serialize<T, S>(option: &Option<T>, serializer: S) -> Result<S::Ok, S::Error>
    where
        T: KafkaNullable + Serialize,
        S: Serializer,
    {
        serializer.serialize_newtype_struct(T::nullable_newtype(), option)
    }

    pub fn deserialize<'de, T, D>(deserializer: D) -> Result<Option<T>, D::Error>
    where
        T: KafkaNullable + Deserialize<'de>,
        D: Deserializer<'de>,
    {
        deserializer.deserialize_newtype_struct(T::nullable_newtype(), ...)
    }
}

The serialize_none / serialize_some / deserialize_option methods of your data format would return error unless invoked through one of these recognized kafka::nullable newtype structs.

[lukesteensen]

Got it working, thank you!

To complete your example code, I added an Option<&'static str> field to my serializer/deserializer, set it in {,de}serialize_newtype_struct, and required that it be present in deserialize_option and serialize_{none,some}. For the deserialization side, I also needed to implement a NullableVisitor<T>.

The visitor ended up a little funky looking since it needed to "use" both the deserializer lifetime and the inner type of the optional:

struct NullableVisitor<'de, T: Deserialize<'de>> {
    _foo: ::std::marker::PhantomData<T>,
    _bar: &'de ::std::marker::PhantomData<u8>,
}

Figuring out that I needed a new Visitor and how it should work was the trickiest part, partially because there's not much background on visitors in the docs. There's probably a cleaner way to implement it to, but I'm just glad it works 😄

Thanks again!