Consider using store for serialization?

[mgsloan]

Awesome project! One of my first published haskell projects was an attempt at doing protobufs well: https://github.com/mgsloan/sproto

Anyway, if making this fast is a priority, then you might be interested in porting it to store, as it's almost certainly faster than attoparsec.

It's still new, mgsloan/store#36 probably ought to be implemented before it's a responsible choice for this lib.

[judah]

Hey, thanks for the suggestion! Once we've added some benchmarks we can try comparing the performance of different serialization libs. Though I suspect that currently, the main bottleneck is the actual proto-lens code that does the encoding/decoding and the intermediate data structures that it uses.

There's a couple other features we'd probably need in order to try this out; I couldn't immediately tell from the docs if they're supported by store:

• Detect the end of input (e.g., Data.Attoparsec.ByteString.endOfInput), since the proto encoding doesn't specify its length.
• Get a substring of a specific length, e.g., Data.Attoparsec.ByteString.take; or, even more convenient, isolate a parse to a specific subsequence of the data, like Data.Binary.Get.isolate. This helps to parse sub-messages.

[mgsloan]

Detect the end of input (e.g., Data.Attoparsec.ByteString.endOfInput), since the proto encoding doesn't specify its length.

The decode family of functions require that the the Peek action consume exactly the whole input, no more, no less.

Get a substring of a specific length, e.g., Data.Attoparsec.ByteString.take; or, even more convenient, isolate a parse to a specific subsequence of the data, like Data.Binary.Get.isolate. This helps to parse sub-messages.

Having known-size extraction of sequences of bytes is covered by mgsloan/store#40

We do have isolate, but it's mostly just a way to skip forward those bytes and ensure the inner peek didn't go beyond. It's still the responsibility of the inner Peek to not advance beyond the isolated region (but that gets checked).

[judah]

I'm closing this ticket since we ended up using our own custom parser monad.

Its implementation is fairly similar to that of store. Compare:
http://hackage.haskell.org/package/store-core-0.4.4/docs/Data-Store-Core.html#t:Peek
https://github.com/google/proto-lens/blob/master/proto-lens/src/Data/ProtoLens/Encoding/Parser/Internal.hs#L14

In addition to the issues mentioned above, the main blocker was the lack of support for endian-specific numeric types (mgsloan/store#31).