Why do Response.AppendBodyString use so much memory?

[PaulRaUnite]

I'm just trying to do effective backend using Iris and groupcache. But when I test downloading of 400kB file by 100 parallel connections, fasthttp produce about 40 or more MB of garbage.
Attach test server + client + pprof heap info

example_and_info.zip

[PaulRaUnite]

Well, I tried to solve problem and when I delete pprof code from program, it start work exellent.
But why? pprof handler worked on another server.

[alexio777]

I have same there:
6757.69MB 21.89% 21.89% 6757.69MB 21.89% github.com/valyala/fasthttp.(*Response).AppendBodyString

[erikdubbelboer]

Looking at the code I see that the actual content is served from a bytes.Reader from a byte slice.
oop/filer.go line 52:

ctx.ServeContent(bytes.NewReader(data), file, t, false)

I have quickly written a program which also serves a bytes.Reader from a byte slice but this time without all the other libraries that might be causing your issue. The program prints the total allocated heap memory every second.

package main
  
import (
  "bytes"
  "fmt"
  "io/ioutil"
  "runtime"
  "time"

  "github.com/valyala/fasthttp"
)

var (
  units = []string{"B", "KB", "MB", "GB", "TB"}
)

func formatSize(size uint64) string {
  s := float64(size)
  for i := 0; i < len(units); i++ {
    if s < 1000 {
      return fmt.Sprintf("%.2f", s) + units[i]
    }
    s /= 1024
  }
  return fmt.Sprintf("%.0fTB", s)
}

func main() {
  runtime.MemProfileRate = 1

  data, err := ioutil.ReadFile("server/files/perform.html")
  if err != nil {
    panic(err)
  }

  go func() {
    for {
      time.Sleep(time.Second * 1)

      s := new(runtime.MemStats)
      runtime.ReadMemStats(s)

      fmt.Println(formatSize(s.TotalAlloc))
    }
  }()

  if err := fasthttp.ListenAndServe("localhost:8080", func(ctx *fasthttp.RequestCtx) {
    ctx.SetBodyStream(bytes.NewReader(data), len(data))
  }); err != nil {
    panic(err)
  }
}

Now this is the result I'm getting. I'm using your exact unmodified client program.

1.01MB
1.02MB
1.02MB
1.02MB
1.02MB <-- this was before I started your client
2.04MB <-- this was after your client
2.04MB
2.04MB
2.16MB <-- this is after I started your client for the second time
2.16MB

This was the output of the client:

client erik$ go run main.go 
100 http://localhost:8080/static/perform.html 2
average dur: 12.294892ms
all duration:17.655556ms from:2018-08-29 23:18:02.107297 +0800 +08 m=+0.001256558
client erik$ go run main.go 
100 http://localhost:8080/static/perform.html 2
average dur: 12.71475ms
all duration:16.757898ms from:2018-08-29 23:18:05.597363 +0800 +08 m=+0.001392350

Judging from this result I'm guessing one of your other libraries such as iris or groupcache is causing this issue. Can you try and reproduce your issue without those libraries?

[PaulRaUnite]

Hello, thank you for your attention to the issue after all this time! It's nice to see the repository is actively maintained now.
But unfortunately, I haven't time and enthusiasm to investigate the problem, because it is probably caused by other libraries, not fasthttp. So, if nobody wants to investigate, the issue can be closed, I guess.