fetchIdToken fails with a 400 error when using the Compute client on GKE with Workload Identity

[sjaq]

Environment details

• OS: Alpine Linux v3.11
• Node.js version: v14.18.1
• npm version: 6.14.15
• google-auth-library version: 7.9.2

Steps to reproduce

1. Run a container in GKE with Workload Identity configured
2. Attempt to generate an ID token using the following code:

const google = require('google-auth-library');
const client = await new google.GoogleAuth().getClient()
console.log(await client.fetchIdToken('IAP_CLIENT_ID'))

Uncaught:
GaxiosError: Could not fetch ID token: Unsuccessful response status code. Request failed with status code 400
    at Gaxios._request (/app/node_modules/gaxios/build/src/gaxios.js:129:23)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async metadataAccessor (/app/node_modules/gcp-metadata/build/src/index.js:68:21)
    at async Compute.fetchIdToken (/app/node_modules/google-auth-library/build/src/auth/computeclient.js:80:23)
    at async REPL4:1:36 {
  response: {
    config: {
      url: 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity?format=full&[SNIP].apps.googleusercontent.com',
      headers: [Object],
      retryConfig: [Object],
      responseType: 'text',
      timeout: 3000,
      paramsSerializer: [Function: paramsSerializer],
      validateStatus: [Function: validateStatus],
      method: 'GET'
    },
    data: 'non-empty audience parameter required\n',
    headers: {
      connection: 'close',
      'content-length': '38',
      'content-type': 'text/plain; charset=utf-8',
      date: 'Tue, 19 Oct 2021 10:50:12 GMT',
      'x-content-type-options': 'nosniff'
    },
    status: 400,
    statusText: 'Bad Request',
    request: {
      responseURL: 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity?format=full&audience=[SNIP].apps.googleusercontent.com'
    }
  },
  config: {
    url: 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity?format=full&audience=[SNIP].apps.googleusercontent.com',
    headers: { 'Metadata-Flavor': 'Google' },
    retryConfig: {
      noResponseRetries: 3,
      currentRetryAttempt: 0,
      retry: 3,
      httpMethodsToRetry: [Array],
      statusCodesToRetry: [Array]
    },
    responseType: 'text',
    timeout: 3000,
    paramsSerializer: [Function: paramsSerializer],
    validateStatus: [Function: validateStatus],
    method: 'GET'
  },
  code: '400'
}

Cause of issue

After some debugging what is going wrong it seems the Workload Identity Metadata server only accepts the audience query parameter if it is the first argument.

# Returns a valid token
$ wget -qO- --header="Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity?audience=[snip].apps.googleusercontent.com&format=full
eyJhbGciOiJSUzI1NiIs[...snip...]

# Returns the same 400 error, note that only the order of the query params changed
$ wget -qO- --header="Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity?format=full&audience=[snip].apps.googleusercontent.com
wget: server returned error: HTTP/1.1 400 Bad Request

[bcoe]

@sjaq thanks for digging into this, this is a somewhat fascinating bug.

@bojeil-google is this a known issue with the workload identify server, I'm wondering if we should open an internal bug.

[bojeil-google]

This is a known limitation. We have filed an internal feature request for this (@bcoe FYI: b/196280129)
You can go around it for now by calling https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${serviceAccountEmail}:generateIdToken with the impersonated service account email and a workload identity pool token.

[sjaq]

Thanks for looking into this!

Is this something that I need to work around in my implementation or something I can expect an update of the library for? Just want to prevent any unnecessary workaround in my codebase... Seems like an easy fix to reorder the query arguments in the library itself?

[bojeil-google]

Hey @sjaq, we will implement it the same way I am describing above (it is not a temporary fix or hack). So you can do it on your own if you can't wait. We want to eventually do it but we have to align across languages (we need to support in several languages and not just Node.js), so it may take time to get to it given the many projects in our pipeline at the moment.

[sjaq]

Thanks @bojeil-google, I'll implement a workaround for the time being then. Not entirely sure how to get the service account email properly when running with workload identity though, could you point me in the right direction?

[bojeil-google]

There is a getServiceAccountEmail() API. You can use that.
I think this should work:

const client = await new google.GoogleAuth().getClient();
const serviceAccountEmail = client.getServiceAccountEmail();

[sjaq]

Thanks @bojeil-google, but that method seems to be unavailable for the Compute client I receive as the default client.

I've used the following implementation for now, which is far from ideal considering it has to do multiple requests to collect all the information and the service account needs the roles/iam.serviceAccountTokenCreator permission for itself:

async function getImpersonatedToken(
  client: Client,
  { serviceAccount, audience }: TokenOptions
): Promise<string> {
  const response = await client.request<any>({
    method: 'POST',
    url: `https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${serviceAccount}:generateIdToken`,
    data: {
      audience,
      includeEmail: true
    }
  });

  const token = response?.data?.token;

  if (typeof token === 'string') {
    return token;
  } else {
    throw new Error('Failed to establish secure API communication, invalid response');
  }
}

export async function getIdToken(audience: string, serviceAccount?: string): Promise<string> {
  const client = await auth.getClient();
  if (client instanceof JWT || client instanceof Compute || client instanceof UserRefreshClient) {
    if (serviceAccount) {
      return getImpersonatedToken(client, { serviceAccount, audience: audience });
    } else if (client instanceof Compute || client instanceof JWT) {
      const token = await client.getAccessToken();
      if (!token.token) {
        throw new Error(
          'Failed to establish secure API communication, could not fetch access token from Compute instance'
        );
      }

      const info = await client.getTokenInfo(token.token);
      if (!info.email) {
        throw new Error(
          'Failed to establish secure API communication, could not fetch service account email from Compute instance'
        );
      }

      return getImpersonatedToken(client, { serviceAccount: info.email, audience: audience });
    }
  }

  throw new Error('Failed to establish secure API communication, invalid client type received');
}

[bcoe]

@sjaq I'm glad you found a workaround, but I agree this is ugly (sorry that you have to do this).

@bojeil-google and I will work together to get a real fix out to you as soon as we can, we'll keep this bug open until then.

[bojeil-google]

I am not sure why you are using a Compute client. I thought you were using workload identity federation (external account client). This doesn't match the initial premise of the issue. The suggested solution should work for the latter.

[sjaq]

@bojeil-google I'm using workload identity as described here to run workloads that have a provided Google Service Account, when I use the nodejs client I'm provided with a Compute client as the default authorization client, not sure what I could do differently on my end?

[bcoe]

@sjaq I'm glad you've found a workaround that's doing the trick for you.

It seems like a nice feature would be if you could somehow upgrade a Compute client to a WorkLoadIdentityCompute client? Leaving this open as a feature request.