Too many open files

[stefdoerr]

I'm facing a similar issue with the minio-go client as here: minio/minio#5093

If I spam PresignedPutObject requests I get:
Get http://127.0.0.1:9000/inputdd2ca3fa-de29-4915-9db7-7de1a5287d25/?location=: dial tcp 127.0.0.1:9000: socket: too many open files

Is seems to have been fixed in the js client, is it possible that the fix has not been ported to the go client?

[stefdoerr]

As a workaround I fixed it locally by setting my file ulimit -n to 65k. Before it was 1024

[vadmeste]

@stefdoerr are you using one Minio client instance or initiating multiple instances ?

[stefdoerr]

Just one, inside a singularity container built from the minio/minio docker

[vadmeste]

This works fine for me:

package main

import (
        "log"
        "time"

        "github.com/minio/minio-go/v6"
)

func main() {

        s3Client, err := minio.New("play.min.io", "Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG", false)
        if err != nil {
                log.Fatalln(err)
        }

        for {
                presignedURL, err := s3Client.PresignedPutObject("vadmeste-testbucket", "my-objectname", time.Duration(1000)*time.Second)
                if err != nil {
                        log.Fatalln(err)
                }
                log.Println(presignedURL)
        }
}

Can you paste the code that reproduces your issue so we can inspect it ?

[harshavardhana]

I think this your application issue, you are not closing http connections used with presigned URL

[stefdoerr]

You are probably right, but I cannot figure it out. I made a small example that can reproduce it. I have a net/http server which sends the requests for bucket creation. I would appreciate if you could tell me what is wrong.

package main

import (
	"flag"
	"fmt"
	"os"
	"strings"

	stdlog "log"
	log "github.com/inconshreveable/log15"
	"net/http"
	"time"
	"io/ioutil"
	"encoding/json"

	"github.com/gorilla/mux"
	_ "github.com/lib/pq"
	minio "github.com/minio/minio-go"
)


func CreateUrlBucket(bucketName, direction string) string {
	endpoint := "127.0.0.1:9000"
	accessKeyID := "xxx"
	secretAccessKey := "xxx"
	useSSL := false

	// Initialize minio client object.
	minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		stdlog.Fatalln(err)
	}

	expiry := time.Second * 24 * 60 * 60 * 7 // 1000 day.
	presignedURL, err := minioClient.PresignedPutObject(bucketName, direction, expiry)
	if err != nil {
		logger.Error("Failed at creating presigned URL for bucket", "err", err)
	} else {
		logger.Info("Successfully created presigned URL for bucket", "bucketName", bucketName)
	}
	
	return presignedURL.String()
}

type Execution struct {
	InputID        			string    `gorm:"type:text" json:"inputid"`
}

func DoStuff(w http.ResponseWriter, r *http.Request) {
	defer r.Body.Close()

	var e Execution
	b, _ := ioutil.ReadAll(r.Body)
	json.Unmarshal(b, &e)
	
	input := "input" + e.InputID
	_ = CreateUrlBucket(input, "input")
}

type Runner struct {
	Router *mux.Router
}

func (r *Runner) Initialize() {
	r.Router = mux.NewRouter()
	r.initializeRoutes()
}

func (r *Runner) initializeRoutes() {
	r.Router.HandleFunc("/test", DoStuff).Methods("POST")
}

func (r *Runner) Run(addr string, update bool) {
	stdlog.Fatal(http.ListenAndServe(addr, r.Router))
}


var (
	logger = log.New("module", "test/main")

	flagUpdate     = flag.Bool("update", false, "Update")
	flagLogLevel   = flag.String("l", "info", "Log level")
)

func init() {
	flag.Parse()

	handler := log.StreamHandler(os.Stderr, log.TerminalFormat())
	level, err := log.LvlFromString(strings.ToLower(*flagLogLevel))
	if err != nil {
		stdlog.Fatal(err)
	}
	filteredHandler := log.LvlFilterHandler(level, handler)
	log.Root().SetHandler(filteredHandler)
}

func main() {
	r := Runner{}
	r.Initialize()
	addr := fmt.Sprintf(":%s", "9997")
	r.Run(addr, *flagUpdate)
}

And then I can break it after a few hundred requests by using python requests:

import requests
import uuid

headers = {"Content-type": "application/json", "Accept": "text/plain" }
for i in range(50000):
    r = requests.post("http://127.0.0.1:9997/test", headers, json={"inputid": uuid.uuid4(), "outputid": uuid.uuid4(), "intermediateid": uuid.uuid4()})
    print(r)

with log:

[...]
INFO[01-10|11:02:45] Successfully created presigned URL for bucket module=test/main bucketName=input
2020/01/10 11:02:45 http: Accept error: accept tcp [::]:9997: accept4: too many open files; retrying in 5ms
EROR[01-10|11:02:45] Failed at creating presigned URL for bucket module=test/main err="Get http://127.0.0.1:9000/input/?location=: dial tcp 127.0.0.1:9000: socket: too many open files"
2020/01/10 11:02:45 http: panic serving 127.0.0.1:39956: runtime error: invalid memory address or nil pointer dereference
goroutine 6129 [running]:
net/http.(*conn).serve.func1(0xc0000d2e60)
	/usr/local/go/src/net/http/server.go:1767 +0x139
panic(0x87ee60, 0xca66b0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
net/url.(*URL).String(0x0, 0x913462, 0x2b)
	/usr/local/go/src/net/url/url.go:775 +0x4a
main.CreateUrlBucket(0x900d1f, 0x5, 0x900d1f, 0x5, 0x0, 0x900d1f)
	/home/sdoerr/Work/test/main.go:42 +0x274
main.DoStuff(0x9b4060, 0xc0002ec0e0, 0xc000204d00)
	/home/sdoerr/Work/test/main.go:57 +0x178
net/http.HandlerFunc.ServeHTTP(0x929698, 0x9b4060, 0xc0002ec0e0, 0xc000204d00)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00017a000, 0x9b4060, 0xc0002ec0e0, 0xc000204b00)
	/home/sdoerr/Software/golib/src/github.com/gorilla/mux/mux.go:210 +0xe2
net/http.serverHandler.ServeHTTP(0xc00017c0e0, 0x9b4060, 0xc0002ec0e0, 0xc000204b00)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0000d2e60, 0x9b52a0, 0xc001964840)
	/usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

[harshavardhana]

First bug is creating minio.New() for each new requests @stefdoerr

[stefdoerr]

Yup, that solved it. Thanks a bunch! I inherited that code from someone else so I was not that sure on the details of how to use the Minio client correctly.

[Jon-Murray]

First bug is creating minio.New() for each new requests @stefdoerr

I think the documentation just needs a line saying that you should have one client shared across context. I landed here after having the same issue with too many open files using the golang client. Similar issue #598

All of the examples kind of look like it should be one client per operation (though i will admit of course that they need to be like that for brevity... but then you get dumb people like me misusing it..)

Thanks