storage: stream error: stream ID 3319; INTERNAL_ERROR

[DrRibosome]

Client

golang google cloud storage

Environment

Alpine Docker on GKE

golang:1.12.5-alpine3.9

Code

func (w *Worker) handleCheckExists(_ *CheckExist, req *Request) (exists bool, err error) {
	newCtx, cancel := context.WithDeadline(req.ctx, time.Now().Add(60 * time.Second))
	defer cancel()
	obj := w.client.Bucket(req.gsPath.Bucket).Object(req.gsPath.Key)
	var reader *storage.Reader
	reader, err = obj.NewReader(newCtx)

	if err != nil {
		if err == storage.ErrObjectNotExist {
			exists = false
			err = nil
			return
		}

		// else, an actual error
		return
	}
	defer reader.Close()

	exists = true
	return
}

Expected behavior

No errors or hanging

Actual behavior

After a little while of relatively high load, existence checks hang seemingly indefinitely (without the WithDeadline context that is). Restarting the existence checks can occasionally yields errors like:

`Get https://storage.googleapis.com/...: stream error: stream ID 3319; INTERNAL_ERROR`

Once requests for a worker starts yield INTERNAL_ERROR, it looks like all subsequent requests yield the same error

Additional context

This appears to start occurring after fairly high load when checking existence of objects at a high rate.

go.mod file:

require (
	cloud.google.com/go v0.52.0
	cloud.google.com/go/bigquery v1.4.0 // indirect
	cloud.google.com/go/storage v1.5.0

[tbpg]

The pattern I've seen in other places is using obj.Attrs() to check if the object exists. Might be better than creating (then closing) a reader unnecessarily. Seems like some sort of underlying bug here, though.

[tritone]

I would also advise trying obj.Attrs() instead-- could you let us know if the issue occurs if you do this?

Also, this issue looks superficially similar to #987 (although that involved writes rather than reads). It looks like that issue didn't have much of a resolution aside from the user opting to use a context timeout as a workaround.

[DrRibosome]

Thanks - I did see #987, but didnt see a fix there.

In the meantime, will switch to obj.Attrs() for existence checks

[DrRibosome]

Reporting back for anyone who might view this later - I switched to using obj.Attrs() instead to check existence and havent hit the error in a while now

[odeke-em]

Thank you for filing this issue @DrRibosome, and thank you for the remedy @tbpg and @tritone!

To me this issue seems to allude to us perhaps having more guidance e.g. a package example for Object.Exists. I mention this because Object.Attrs while matching a method on Client, isn't intuitive for customers who haven't spent time staring at these libraries.

Perhaps let's mail 2 examples:

• ObjectExists
• BucketExists

which will use the appropriate Attrs() methods, can be indexed by search engines and serve as guidance so that folks don't have to try to create the reader.

However, I acknowledge that the HTTP/2 stream error issue is indicative of a bug as @tbpg mentioned. Still though I think adding guidance can help resolve this issue until we get to tackling the underlying stream error after creating a reader on a non-existent file.

[tritone]

Thanks very much for adding these examples, @odeke-em !