fmt: add Append, Appendf, Appendln

[seebs]

What version of Go are you using (go version)?

1.16

Does this issue reproduce with the latest release?

Sure.

What operating system and processor architecture are you using (go env)?

N/A

What did you do?

Profiled something that was using Sprintf in a minor way.

What did you expect to see?

Less Sprintf in the profile.

What did you see instead?

So much Sprintf.

The issue here is that most of what's happening is WAY cheaper than allocations. strconv has AppendInt, but that can't do formatting.

It makes sense, for most uses, that fmt.Sprintf is an allocator that produces a string, but there are times when you want formatted-output, but want to write into a []byte. And you can do that with bytes.Buffer and fmt.Fprintf, but creating a bytes.Buffer around a []byte is... an allocation. And fmt.Fprintf does an allocation. (Curiously, if I do fmt.Fprintf into a bytes.Buffer I just made for that purpose, I only get the one allocation.)

What I want: Something like Sprintf, but that can write into a []byte, and can fail gracefully if there's not enough space to write things.

Proposed name: fmt.Snprintf, because snprintf is what you call when you already have a buffer you want written to and you have a length limit in mind.

So, fmt.Snprintf(dest []byte, fmt string, args ...interface{}) (int, error), perhaps. The C standard's answer to "what if n isn't big enough" is "you report the n you would have used if n had been big enough", which exists to allow a single-pass process to figure out how much space you actually need. Alternatively, it could return number of bytes actually written, and if it didn't fit, an error with a concrete type that indicates space needed.

I note, browsing source, that fmt already has fmtBytes, although this doesn't do quite the thing this would need.

[ianlancetaylor]

Edited to add: this doesn't work, as the fmt package pools buffers and will reuse the byte slice.

You could do this instead of adding new API;

package x_test

import (
	"bytes"
	"fmt"
	"testing"
)

type sliceStealer struct {
	b []byte
}

func (ss *sliceStealer) Write(s []byte) (int, error) {
	if len(ss.b) > 0 {
		panic("multiple calls to Write")
	}
	ss.b = s
	return len(s), nil
}

func FprintfToBytes(format string, a ...interface{}) []byte {
	var ss sliceStealer
	fmt.Fprintf(&ss, format, a...)
	return ss.b
}

func TestFprintf(t *testing.T) {
	s := FprintfToBytes("%d", 10)
	if !bytes.Equal(s, []byte("10")) {
		t.Errorf(`FprintfToBytes("%%d", 10) = %q, want %q`, s, "10")
	}
}

func BenchmarkFprintf(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		FprintfToBytes("%d", 10)
	}
}

[DeedleFake]

~~It looks like an allocation can be eliminated by creating a simpler type locally instead of using bytes.Buffer. Does the allocation with bytes.Buffer happen because of the return of a pointer from inside of bytes.NewBuffer()?

Edit: Ah, darn. @ianlancetaylor beat me to it.~~

Edit 2: Never mind. See #47579 (comment) below.

[seebs]

Is there any guarantee that Fprintf does only one write? Also, while that allows me to steal the buffer fprintf used, it doesn't let me write into an existing buffer that I have, which is the really interesting case. For instance, to put a 0-padded value in filename[5:9] or something. The distinction being, with that, BenchmarkFprintf could report 0 allocs/op, theoretically.

So right now, I don't think it's possible to loop on Fprintf without at least one allocation per call...

[josharian]

Also, while that allows me to steal the buffer fprintf used, it doesn't let me write into an existing buffer that I have, which is the really interesting case.

The Write method of your io.Writer could copy the buffer's contents into your existing buffer.

It does mean an extra copy of the data, which is cheap but not free. In any case that should let you avoid the allocation.

Also, I'm not sure the slice stealer is safe. IIRC package fmt pools its buffers, so the buffer passed to Write will be used again by subsequent fmt work.

[ianlancetaylor]

Also, I'm not sure the slice stealer is safe. IIRC package fmt pools its buffers, so the buffer passed to Write will be used again by subsequent fmt work.

Good point.

[ianlancetaylor]

fmt.Snprintf(dest []byte, fmt string, args ...interface{}) (int, error)

I think we can simplify the issues about exceeding the size of the buffer by making this

fmt.Snprintf(dest []byte, fmt string, args ...interface{}) ([]byte, error)

That is, just return the byte slice, which will be a new slice if the original cap is not large enough. People who need to limit the size of the output, which I suspect is less common, can use Fprintf with a custom Writer. (But with this modification Snprintf is not the right name.)

As it happens, I think this might be fairly simple to implement: replace the pp.buf field with the passed in buffer, and then remove it and restore the original pp.buf field before releasing the pp.

[josharian]

If Sprintf is string-printf, then it seems to me that bytes-printf should be Bprintf.

[seebs]

Bprintf would also make sense to me. Hmm. "if cap isn't large enough, then allocate" seems like a reasonable choice, but I don't see a pretty/clean way to express "if this didn't fit in the space I provided". I guess something like if &provided[0] != &returned[0] maybe? (This isn't quite safe in all theoretical cases, because the returned len could be 0, but if the returned len is 0, I don't have to worry about the allocation, do I?)

I'd thought about creating a thing which just copies into an existing buffer, but then fmt's still allocating -- but if it's pooling those, I guess I care less about that than I thought I did.

got it! fmt.Apprendf

[josharian]

I don’t think the function would return an error; fmt functions only return errors when writes fail. So the signature would be

func Bprintf(dest []byte, format string, args ...interface{}) []byte

[josharian]

I don't see a pretty/clean way to express "if this didn't fit in the space I provided".

How about cap(provided) != cap(returned)? Can’t grow without modifying cap.

[seebs]

Yeah. I just dislike the lack of a way to express "is this a different thing" separately from "does this thing have different characteristics". And I suppose I could imagine a case where the desired outcome would be "if for some reason it doesn't fit, fail rather than allocating", but that feels like a different thing.

[tdakkota]

@DeedleFake I think your benchmark is not correct

        var buf [128]byte
	allocs := testing.AllocsPerRun(128, func() {
		fmt.Fprintf(bytes.NewBuffer(buf[:0]), "This is a test.") // buf[:] -> buf[:0]
	})
	fmt.Println(allocs)

makes only one allocation (bytes.Buffer leaks to heap).

fmt.Fprintf(bytes.NewBuffer(buf[:]), "This is a test.")

does not re-use buf .
It creates a new 128-byte buffer and appends This is a test..

[DeedleFake]

@tdakkota

You're right, I don't know how I missed that. Whoops.

[bcmills]

@ianlancetaylor

fmt.Snprintf(dest []byte, fmt string, args ...interface{}) ([]byte, error)

I'm confused by this signature. Does the proposed function append to dest (as the strconv.Append* functions do) up to cap(dest), or does it write to the slice from 0 to len(dest) and truncate output if the length is too small (as copy does), or does it do a third thing (like writing from 0 to cap(dest) and returning the final length)?

If it is like copy or strconv.Append*, how is that different from using a bytes.Buffer today, perhaps with a LimitedWriter as discussed in #17688 (comment)?

If it is identical except for allocation behavior, would it be better to improve the compiler's inlining and escape analysis instead? Then the only new API surface we would need is io.LimitedWriter, which seems much more generally useful than just a new fmt function.

(https://play.golang.org/p/aBpCGXDSmmw)

[bcmills]

@josharian

If Sprintf is string-printf, then it seems to me that bytes-printf should be Bprintf.

Sprintf is “print to a string”, which must return the string because Go strings are immutable.

Since slices are mutable, I would expect a function named Bprintf to print to the passed-in slice and return the number of bytes “printed”, in the same way that Printf and Fprintf do:

// Bprintf prints up to len(dst) bytes to dst, returning the number of bytes printed.
// If the output does not fit within len(dst), Bprintf returns len(dst), io.ErrShortBuffer.
func Bprintf(dst []byte, format string, args ...interface{}) (int, error)

For a function with the meaning “append the formatted bytes to the passed-in slice”, I would expect the name Appendf, because “appending” is not “printing”.

// Appendf appends the formatted args to dst and returns the extended buffer.
func Appendf(dst []byte, format string, args ...interface{}) []byte

[rsc]

fmt.Fprintf guarantees to do a single write to the writer.
Why is it not good enough to pass a bytes.Buffer (or your own custom writer) to Fprintf?

[rsc]

/cc @robpike

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[robpike]

I agree with @rsc's comment that Fprintf already provides enough flexibility.

[rsc]

It seems like we are still waiting on an answer to why fmt.Fprintf into an appropriate appending writer is not good enough.

[tdakkota]

Passing a byte slice using bytes.Buffer as io.Writer causes allocation due to type-to-interface conversion, but fmt.Appendf([]byte, ...) []byte will not.

Also Appendf does not need fmt pools buffers at all.

[clausecker]

Independently of the escape analysis issue, I like the idea of a Bprintf function for convenience.

[josharian]

It'd also help with buffer re-use. Right now there's no way to get the slice back out of a bytes.Buffer when you're done with it, which makes it hard to re-use.

[as]

What is the problem with just calling this:

func Append(dst []byte, a ...interface{}) []byte
func Appendf(dst []byte, format string, a ...interface{}) []byte
func Appendln(dst []byte, a ...interface{}) []byte

Or

func Bprint(dst []byte, a ...interface{}) []byte
func Bprintf(dst []byte, format string, a ...interface{}) []byte
func Bprintln(dst []byte, a ...interface{}) []byte

Are the words Print and Append really adding that much clarity that they must be used together in the name?

[akavel]

Personally, fmt.Append and fmt.Appendf do indeed look lovingly sweet to me: the fmt. prefix seems to be clarifying enough already, and the name seems to also fit with the other "modern" addition already in the package, fmt.Errorf. And on the other hand it's as clearly as possible alluding to the builtin append — basically, what it means to me is: "like builtin append, but with fmt magic on top"... which seems to perfectly encapsulate exactly what it does.

edit: On top of this, in fact fmt.AppendPrint is sincerely harder for me to understand. More meanings clumped together, more to decode, analyze and understand. I was somewhat confused as to whether I really get what's going on in this PR, until I finally landed on the fmt.Append suggestion, which was the a-ha moment for me: it's just like builtin append + fmt "magic", nothing more, nothing less.

[rsc]

I do think, though, that I would probably prefer that AppendPrintf make an explicit statement about what if anything will have been done to the previous buffer if it does in fact allocate. Even if the statement is "it is unspecified".

I'm quite confident that's what the statement will be - it is unspecified.

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[mvdan]

Was the point about shorter func names missed, perhaps? #47579 (comment)

[robpike]

I am a fan of Append, Appendf, and Appendln.

[rsc]

Spoke to @robpike about this and I agree that Append, Appendf, Appendln fit better with fmt's overall API.
Rob is going to look into this for Go 1.19.

[gopherbot]

Change https://go.dev/cl/406177 mentions this issue: fmt: add Append, Appenln, Appendf

[gopherbot]

Change https://go.dev/cl/406357 mentions this issue: go/gcexportdata: don't assume that fmt names never change