BigQuery LoadJobConfiguration doesn't respect destination dataset projectId

[pbzdyl]

• OS: MacOS Mojave
• Java version: 9.0.1
• google-cloud-java version(s): 1.62.0

Steps to reproduce

gcloud auth application-default login using an account from some ProjectA ([email protected] in the code below). That account has BigQuery admin role in ProjectB and no BigQuery permissions in ProjectA.

Running the code below where ProjectB is some arbitrary GCP project different from the above ProjectA:

    BigQuery bigQuery = BigQueryOptions.getDefaultInstance().getService();

    TableId destinationTableId = TableId.of("ProjectB", "test_dataset", "test_table");
    String sourceUri = "gs://some/test/bucket/data.json";

    LoadJobConfiguration loadJobConfiguration =
            LoadJobConfiguration
                    .builder(destinationTableId, sourceUri)
                    .setFormatOptions(FormatOptions.json())
                    .setAutodetect(true)
                    .build();

    JobInfo jobInfo = JobInfo.of(loadJobConfiguration);
    Job job = bigQuery.create(jobInfo);
    job.waitFor();

results in the exception included below as the project from the default application credentials is used instead of the one specified for destination table.

Stacktrace

Exception in thread "main" com.google.cloud.bigquery.BigQueryException: Access Denied: Project ProjectA: The user [email protected] does not have bigquery.jobs.create permission in project ProjectA.
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:100)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:185)
	at com.google.cloud.bigquery.BigQueryImpl$4.call(BigQueryImpl.java:241)
	at com.google.cloud.bigquery.BigQueryImpl$4.call(BigQueryImpl.java:238)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:237)
	at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:212)
	at oc_perf_metrics.Test.main(Test.java:24)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Access Denied: Project ProjectA: The user [email protected] does not have bigquery.jobs.create permission in project ProjectA.",
    "reason" : "accessDenied"
  } ],
  "message" : "Access Denied: Project ProjectA: The user [email protected] does not have bigquery.jobs.create permission in project ProjectA."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:401)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1097)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:183)
	... 8 more

External references such as API reference guides used

I think there was a similar issue #3924 reported recently that got fixed in #4183 but it still doesn't work for me.

Workaround

Instead of using a default instance of BigQuery that would use ProjectA from the default credentials:

BigQuery bigQuery = BigQueryOptions.getDefaultInstance().getService();

use an instance that sets the default projectID to ProjectB explicitly:

BigQuery bigQuery = BigQueryOptions.newBuilder().setProjectId("ProjectB").build().getService();

Thanks!

[ajaaym]

@pbzdyl please set the project id to the id in which you want to run the job like below.

BigQuery bigQuery = BigQueryOptions.getDefaultInstance().toBuilder().setProjectId("ProjectB").build().getService();

[pbzdyl]

It seems I misunderstood how the BigQuery Java API should be used. It seems that I need to configure the BigQuery service instance with the projectId where the calling account has proper API permissions (e.g. ProjectA) and TableId's projectId might need to be specified if the specific job needs to access a table from a different GCP project than the BigQuery API project (e.g. a table shared from another project). Thanks @ajaaym!