feature: CBOR sequences

[espoal]

Is your feature request related to a problem? Please describe.
I would like to append multiple CBOR messages to a file

Describe the solution you'd like
Ideally I would like CBOR sequences, or a practical way to decode multiple messages to a file.

Describe alternatives you've considered
At the moment I'm using NDJSON, but it's a pain

Additional context
Spec

[x448]

Looks like you can do this by creating an encoder using NewEncoder(w) with io.Writer. Same with decoder created by NewDecoder(r) with io.Reader.

Easiest way is to use package level encoder := cbor.NewEncoder(w) if default options are acceptable.

encoder := cbor.NewEncoder(w)
err := encoder.Encode(v) 

If you need custom options, you can create EncMode and DecMode at startup and reuse them.

opts := cbor.CoreDetEncOptions()  // Use modern options.  You can customize the returned opts.
em, err := opts.EncMode()         // This creates reusable encoding mode (safe for concurrency).
encoder := em.NewEncoder(w)       // Create encoder with io.Writer.
err := encoder.Encode(msg1)       // Use encoder to send data to underlying io.Writer.
err = encoder.Encode(msg2)        // Use encoder to send data to underlying io.Writer.

For security, you can use io.LimitReader to limit size when decoding very large or indefinite size data.

There's more info and examples at https://github.com/fxamacker/cbor/#quick-start

[espoal]

Ok thank you very much, I made it work both with reader and marshall:

package main

import (
	"bytes"
	"github.com/fxamacker/cbor/v2"
	"log"
)

func main() {

	type Record struct {
		Payload string
		Counter int
	}

	r1 := Record{
		"hello", 1}
	r2 := Record{
		" world", 2}

	var buff []byte

	b, err := cbor.Marshal(r1)
	if err != nil {
		log.Fatal(err)
	}
	buff = append(buff, b...)

	b, err = cbor.Marshal(r2)
	if err != nil {
		log.Fatal(err)
	}
	buff = append(buff, b...)

	log.Println(buff)

	var out1, out2 Record
	decoder := cbor.NewDecoder(bytes.NewReader(buff))

	err = decoder.Decode(&out1)
	if err != nil {
		log.Fatal(err)
	}
	log.Println(out1)

	err = decoder.Decode(&out2)
	if err != nil {
		log.Fatal(err)
	}
	log.Println(out2)

}

One last use case covered by NDJSON and CBOR sequences, which is stream parsing: let's say I have 5 messages in my file, and I want to read the 4th without parsing the first 3, is it possible with this approach?

Should I maybe build a custom decoder, read only the length at the beginning, and then skip to the next position in the buffer? Does CBOR prepend the length of the full message?

[x448]

CBOR doesn't define "full message". If you meant "data item" as defined in CBOR RFC 8949, then some data items like CBOR Array will have number of elements (but not size of entire array in bytes).

@fxamacker what do you think of adding a Skip feature to Decoder that allows skipping next data item?

[fxamacker]

feature: CBOR sequences

At a glance, CBOR Sequence (RFC 8742) is just a concatenation of zero or more encoded CBOR data items, without markers in between nor at the end. CBOR data items are defined in Section 1.2 of RFC 8949.

Should I maybe build a custom decoder, read only the length at the beginning, and then skip to the next position in the buffer? Does CBOR prepend the length of the full message?

As @x448 mentioned, if by "full message" you meant CBOR "data item" as defined in RFC 8949, then I can add a Skip feature to the decoder.

CBOR doesn't prepend length (size in bytes) of data item.

One last use case covered by NDJSON and CBOR sequences, which is stream parsing: let's say I have 5 messages in my file, and I want to read the 4th without parsing the first 3, is it possible with this approach?

It sounds like you want to "skip" the first 3 messages and then parse the 4th message. Skip isn't currently supported by Decoder. Until I add a Skip feature, a less efficient way is to decode the messages you want to skip to cbor.RawMessage. cbor.RawMessage copies data item into a byte slice without parsing the data item.

var throwAway cbor.RawMessage
err := decodoer.Decode(&throwAway)

One more thing, I noticed cbor.Marshal is used in your code snippet. For CBOR sequence, cbor.Encoder is more efficient and easier to use.

[x448]

@espoal @fxamacker

I think this issue can be closed because the original request is already supported by this CBOR codec. The Skip() feature request is in the newly opened issue #366.

[espoal]

@x448 @fxamacker I actually meant to skip a message (data item?) in a sequence. I thought it was covered by this part of the spec. Will Skip() allow me to skip a message in a sequence?