sortedTokenBytes doesn't seem necessary

[stoneflying]

Thank you for your efforts. I found that in the NewCoreBPE function, the result of the following code does not seem to be used anywhere,

        sortedTokenBytes := make([][]byte, 0, len(encoder))
	for k := range encoder {
		sortedTokenBytes = append(sortedTokenBytes, []byte(k))
	}
	sort.Slice(sortedTokenBytes, func(i, j int) bool {
		return bytes.Compare(sortedTokenBytes[i], sortedTokenBytes[j]) < 0
	})

	return &CoreBPE{
		......
		sortedTokenBytes:     sortedTokenBytes,
	}, nil

but this sorting operation seems to be very expensive. Is there any consideration here?

[pkoukk]

Yes, this code currently is not necessary.
This variable is mainly used when multi-threaded encoding is needed, but this feature has not been implemented yet.
I will implement it later, maybe this week.