proposal: encoding/xml: add iterators Decoder.Tokens, Decoder.RawTokens

[dolmen]

Proposal Details

Add iter.Seq2 iterator methods on encoding/xml.Decoder to iter over tokens and raw tokens.

func (d *Decoder) Tokens() iter.Seq2[Token, error] {
	return func(yield func(Token, error) bool) {
		for {
			token, err := d.Token()
			if err == io.EOF {
				break
			}
			if !yield(token, err) {
				break
			}
			// What should we do here if err != nil?
		}
	}
}

func (d *Decoder) RawTokens() iter.Seq2[Token, error] {
	return func(yield func(Token, error) bool) {
		for {
			token, err := d.RawToken()
			if err == io.EOF {
				break
			}
			if !yield(token, err) {
				break
			}
			// What should we do here if err != nil?
		}
	}
}

Note: I'm not yet sure to like an iterator that handles a stream error this way.

Related:

• go/types: add iterator methods #66626

[gabyhelp]

Related Issues

• encoding/xml: add decode from TokenReader, to enable stream transformers #19480 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dolmen]

Note that generic functions can also be written outside the package so the Decoder methods would be purely to allow "native" range-over-func support:

func IterTokens[Token any, Tokenizer interface{ Token() (Token, error) }](tok Tokenizer) iter.Seq2[Token, error] {
	return func(yield func(Token, error) bool) {
		for {
			token, err := tok.Token()
			if err == io.EOF {
				break
			}
			if !yield(token, err) {
				break
			}
		}
	}
}

func IterRawTokens[Token any, Tokenizer interface{ RawToken() (Token, error) }](tok Tokenizer) iter.Seq2[Token, error] {
	return func(yield func(Token, error) bool) {
		for {
			token, err := tok.RawToken()
			if err == io.EOF {
				break
			}
			if !yield(token, err) {
				break
			}
		}
	}
}