proposal: cmd/vet: detect defer rows.Close()

[zachgersh]

I'd like to suggest, based on the prior art of #17780, that we also add a check to make sure that rows.Close() is always deferred for any SQL call that returns rows). I struggle to think of a false positive especially due to the fact that rows.Close() can be safely deferred in every case.

Consider this code:

        _, err := db.Exec(`INSERT into (age) VALUES (30)`)
        if err != nil {
                return err
        }

Let's say I've done this in some test setup, I'd like to clean up the DB for another set of tests, I am now unable to truncate the database due to the fact that I have left this connection open. Production code won't run into this truncation issue (I hope) but it could potentially put pressure on the database due to the number of open connections instead.

A False positive could potentially look like this:

        rows, err := db.QueryContext(ctx, "SELECT name FROM users WHERE age=?", 10)
	if err != nil {
		log.Fatal(err)
	}

	for rows.Next() {
	}

My for loop will have consumed all the rows and I therefore get a close call automatically. We still show a deferred close here for users just in case they decide to break out of their for loop early thus turning this back into a problem.

[rsc]

/cc @robpike @kardianos but this does seem OK given #17780.
It would be nice to have some numbers about how often this happens in the wild.
(See cmd/vet/README.)

[nemith]

I have some people on my team at work new to Go and we had two outages due to rows.Close() missing. What makes this worse is with the mysql driver this results in leaked goroutines and fails in random ways and it's not very clear from the failures or the goroutine leak that the code was missing rows.Close().

Now some of this could be fixed by some changes in the way packets and errors are handled in the mysql driver but having a clear vet would make it easier to discover the problem. We ended up just manually scanning the changed code looking for the missing calls which is a lot more error prone.

[robpike]

Seems OK to me based on the story here, but I am not a user of that package.

[kardianos]

Any helper function that wraps Query and returns Rows would probably be a false positive. I don't know how common those are.

QueryContext will close Rows when ctx is canceled.

I typically buffer the entire result set and close the Query by default in part to avoid this type of issue, so it can be a problem.

In short, it would probably be okay, but I'm guessing for ~1% of users it will display a false positive, though that may be high. (defering rows.Close wouldn't work in a helper function that calls Query and returns Rows.)

[zachgersh]

(defering rows.Close wouldn't work in a helper function that calls Query and returns Rows.)

How does this work now if one returns a resp from a helper function in the eyes of vet? Seems like pretty much the same issue as this one to me.

[rsc]

Vet checks aim to have zero false positives, or else false positives with an easy rewrite to silence the error. I don't see a rewrite here for wrappers other than something like if false { rows.Close() }. That would be a problem.

At the least we need more data here. But we also need a story for why there won't be unfixable (or only fixable with ugly code) false positives. Maybe the check would have to look for rows being returned and understand that close need not be called. But then that would apply to rows being put in data structures too. Or sent on channels. It's unclear where it ends.

Maybe one could define a clear "rows does not escape" signal, like only methods on rows are invoked and it is never copied anywhere (not into other functions, channels, data structures, etc). We'd then need to check and see if that catches useful bugs.

(I guess the http.Response Body doesn't get wrapped or copied around as much?)

[kardianos]

@rsc I agree this needs more information. I'll see what I can dig up this week for data.

I picked a random "used by" on pkg.go.dev and the first one I got was this:
https://github.com/readystock/goqu/blob/v5.0.0/database.go#L216

func (me *Database) Query(query string, args ...interface{}) (*sql.Rows, error) {
	me.Trace("QUERY", query, args...)
	return me.Db.Query(query, args...)
}

Which I think would fail the vet check proposed.

---

One possibility is to add this check ONLY if rows.Scan is also called in the same method. I think this should be safe 100% of the time. But I should probably do a corpus analysis before making that judgement.

[rsc]

We can wait a little longer for data but it's really starting to sound like this is a no-go. The false positive with no clear workaround really sinks this from a vet perspective. We'd need not just data but also a design without that problem.

[rsc]

The discussion above has identified that there are common patterns that will cause false positives in this check that will be quite difficult (or at least ugly and non-obvious) to work around.

Based on that, this seems like a likely decline.

[rsc]

No change in consensus, so declined.