WIP: add functions for converting Nats and Bytes with multiple word sizes

[stew]

Overview

What does this change accomplish and why? i.e. How does it change the user experience?

Fixes: #2277

Implementation notes

I added encode and decode functions to Bytes.hs and calling them as foreign functions

Interesting/controversial decisions

I would love for someone that knows what they are actually doing to check my encode functions on Bytes, and see if what I'm doing with a mutable block is an efficient / correct way of creating Bytes

Test coverage

TODO

[dolio]

So, something occurred to me. Tagging @pchiusano too.

If encoding is single values, like you encode one Nat and then concatenate, that's going to lead to pretty fragmented Bytes values, isn't it? The chunks will be extremely small, or if they aren't, it will be a bunch of expensive (due to copying) ByteString concatenation.

It's probably better to have something like Haskell's binary where you 'write' to an intermediate type and then finalize to get a Bytes with reasonable chunk sizes.

[dolio]

I think instead of implementing this manually, it'd be better to use on of the binary encoding libraries we already depend on. E.G. we use bytes which has a bunch of endian-specific functions:

https://hackage.haskell.org/package/bytes-0.17.1/docs/Data-Bytes-Put.html

And can spit out a ByteString afterward.

[stew]

@dolio so I think we are doing much better now in this respect?

[dolio]

Yeah, that looks better. I just default to preferring something in a more widely used library if we're already using it. That way there's hopefully less question whether some fiddly detail is mistaken (because more people would have noticed by now).

[pchiusano]

If encoding is single values, like you encode one Nat and then concatenate, that's going to lead to pretty fragmented Bytes values, isn't it? The chunks will be extremely small, or if they aren't, it will be a bunch of expensive (due to copying) ByteString concatenation.

So, the idea is that you can just call Bytes.flatten to convert a fragmented Bytes to a single-chunk one, and that function is quite efficient and does everything in one pass. You can even control the chunk size, like if you want 1024-sized chunks, you can take 1024, flatten, and concatenate with the remaining recursively chunked Bytes.

Basically, Bytes can function as a buffer (unlike ByteString). Maybe the constant factors aren't quite as good as a separate Builder type, but OTOH having a separate Builder type leads to both more plumbing code and also inefficiencies with converting back and forth at various API boundaries.

Also on that note, I wonder if going through Put/Get is overkill and possibly less efficient. Or maybe it gets optimized and is fine. But more problematic with those libraries is they work only with ByteString, which uses pinned memory (and this is never getting fixed it seems, due to ByteString also being used for FFI interchange), so to avoid needless copying we'd need to have Bytes also use ByteString for its chunk type. I think using pinned memory is the wrong choice and don't want to change Bytes to use that.

That said, the implementation of the decoders could be much more efficient @stew to avoid going through Maybe for all those bytes getting pulled out. Will leave a note on how to improve that.

[dolio]

I see. I guess flattening the bytes afterwards works fine.

The stuff Stew wrote is also producing ByteString, though, so then that needs to be implemented using something else if you don't want to use ByteString.

[pchiusano]

The stuff Stew wrote is also producing ByteString, though, so then that needs to be implemented using something else if you don't want to use ByteString.

Ah yeah, good point.

[pchiusano]

Great! Seems like this is pretty much there.

What's going on with the CI though?

[pchiusano]

Great!