auth: 'GOOGLE_CLOUD_QUOTA_PROJECT' env variable should override quota project setting from credential file

[michalskalski]

Client

Client which works with new auth library

Environment

% go version
go version go1.22.6 darwin/arm64

Code and Dependencies

package main

import (
	"context"
	"os"

	pubsub "cloud.google.com/go/pubsub/apiv1"
	"cloud.google.com/go/pubsub/apiv1/pubsubpb"

	"google.golang.org/api/iterator"
)

func main() {
	// enable new auth lib
	os.Setenv("GOOGLE_API_GO_EXPERIMENTAL_ENABLE_NEW_AUTH_LIB", "true")

	projectID := "my-gcp-project"
	// override quota project
	os.Setenv("GOOGLE_CLOUD_QUOTA_PROJECT", projectID)

	ctx := context.Background()

	pubsubClient, err := pubsub.NewPublisherClient(ctx)
	if err != nil {
		panic(err)
	}

	it := pubsubClient.ListTopics(ctx, &pubsubpb.ListTopicsRequest{
		Project: "projects/" + projectID,
	})

	for {
		resp, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			panic(err)
		}
		println(resp.Name)
	}

}

go.mod

module github.com/michalskalski/gcp-quota-project

go 1.22.6

require (
	cloud.google.com/go/pubsub v1.42.0
	google.golang.org/api v0.195.0
)

require (
	cloud.google.com/go/auth v0.9.2 // indirect
	cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
	cloud.google.com/go/compute/metadata v0.5.0 // indirect
	cloud.google.com/go/iam v1.1.13 // indirect
	github.com/felixge/httpsnoop v1.0.4 // indirect
	github.com/go-logr/logr v1.4.2 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/google/s2a-go v0.1.8 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.3 // indirect
	github.com/googleapis/gax-go/v2 v2.13.0 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
	go.opentelemetry.io/otel v1.28.0 // indirect
	go.opentelemetry.io/otel/metric v1.28.0 // indirect
	go.opentelemetry.io/otel/trace v1.28.0 // indirect
	golang.org/x/crypto v0.26.0 // indirect
	golang.org/x/net v0.28.0 // indirect
	golang.org/x/oauth2 v0.22.0 // indirect
	golang.org/x/sync v0.8.0 // indirect
	golang.org/x/sys v0.24.0 // indirect
	golang.org/x/text v0.17.0 // indirect
	golang.org/x/time v0.6.0 // indirect
	google.golang.org/genproto v0.0.0-20240823204242-4ba0660f739c // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240827150818-7e3bb234dfed // indirect
	google.golang.org/grpc v1.66.0 // indirect
	google.golang.org/protobuf v1.34.2 // indirect
)        

Expected behavior

When I set "GOOGLE_CLOUD_QUOTA_PROJECT" environment variable I would expect that quota project defined by that variable will be used instead of what is defined in credential file. This is a behavior of previous (current) auth code: https://github.com/googleapis/google-api-go-client/blob/main/internal/creds.go#L242-L261

Actual behavior

% jq .quota_project_id ~/.config/gcloud/application_default_credentials.json
"this-project-not-exist"

./gcp-quota-project 
panic: rpc error: code = InvalidArgument desc = Project 'project:this-project-not-exist' not found or deleted.
error details: name = ErrorInfo reason = USER_PROJECT_DENIED domain = googleapis.com metadata = map[consumer:projects/this-project-not-exist service:pubsub.googleapis.com]

goroutine 1 [running]:
main.main()
        /Users/michal/projects/code/quota/main.go:38 +0x1a8

Additional context

#10797 fixed similar issue for overriding through client option, but env variable will not be take into consideration because file credentials found in default location will populate credential struct with QuotaProjectID: https://github.com/googleapis/google-cloud-go/blob/main/auth/grpctransport/grpctransport.go#L262 and https://github.com/googleapis/google-cloud-go/blob/main/auth/auth.go#L176-L180 which will cause not looking into env variables https://github.com/googleapis/google-cloud-go/blob/main/auth/internal/internal.go#L98-L100.

[codyoss]

Thank you for the report. I believe I have a PR to fix this linked above, please lmk if that works for your usecase. The solution you purposed is something we do want to do in the future, but there is more to it that requires work in the transport packages to feed some values in quota project fields as well. I filed #10808 to track this explicit feature.

[michalskalski]

Thank you @codyoss, what you proposed unlocks path to check env variable before conclude on quota project variable. My use case will work with this.