Impossible to use signed url v4 and the Service Account Credential APIs

[guillaumeblaquiere]

Months ago, I solved an issue to a Stack overflow user by explaining how to generate a signed URL without a service account JSON fey file. I also release an article on this

Today, a comment of a user told me that is doesn't work. They open a new Stack Overflow question and in their question, there is the v4 mentioned.

I dug into the code and yes, the v2 and the v4 haven't the same behavior: the v4 check if the credential can generate a signed URL by itself, that implies the useless check later that allows to use Service Account Credential API, instead of the service account private key to generate the signature.

Environment details

• Linux (Cloud Function and Cloud Run)
• Python version: python --version 3.7 and 3.8
• google-cloud-storage version: latest (1.35.0)

Steps to reproduce

1. Working code on Cloud Run (for example) Replace YOUR_BUCKET by a valid bucket

import os
from flask import Flask, request

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def sign_url():
    from google.cloud import storage
    from datetime import datetime, timedelta

    import google.auth

    credentials, project_id = google.auth.default()

    # Perform a refresh request to get the access token of the current credentials (Else, it's None)
    from google.auth.transport import requests
    r = requests.Request()
    credentials.refresh(r)

    client = storage.Client()
    bucket = client.get_bucket('YOUR_BUCKET')
    blob = bucket.get_blob('name.csv')

    service_account_email = credentials.service_account_email

    url = blob.generate_signed_url(service_account_email=service_account_email, access_token=credentials.token,version="v2", method="PUT",expiration=timedelta(minutes=120),content_type="application/octet-stream")
    return url, 200

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=int(os.environ.get('PORT', 8080)))

2. Broken code on Cloud Run (for example) Replace YOUR_BUCKET by a valid bucket

import os
from flask import Flask, request

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def sign_url():
    from google.cloud import storage
    from datetime import datetime, timedelta

    import google.auth

    credentials, project_id = google.auth.default()

    # Perform a refresh request to get the access token of the current credentials (Else, it's None)
    from google.auth.transport import requests
    r = requests.Request()
    credentials.refresh(r)

    client = storage.Client()
    bucket = client.get_bucket('YOUR_BUCKET')
    blob = bucket.get_blob('name.csv')

    service_account_email = credentials.service_account_email

    url = blob.generate_signed_url(service_account_email=service_account_email, access_token=credentials.token,version="v4", method="PUT",expiration=timedelta(minutes=120),content_type="application/octet-stream")
    return url, 200

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=int(os.environ.get('PORT', 8080)))

Only the version (v2/v4) has changed

Fix proposal is coming.

[nagro-c]

I'm facing the same issue on cloud function env, and getting the error below:

AttributeError: you need a private key to sign credentials.the credentials you are currently using <class 'google.auth.compute_engine.credentials.Credentials'> just contains a token. see https://googleapis.dev/python/google-api-core/latest/auth.html#setting-up-a-service-account for more details.

[deepamgupta]

@guillaumeblaquiere
after applying the fix provided by you, I am getting this exception
<class 'google.auth.exceptions.RefreshError'>, RefreshError('invalid_scope: Invalid OAuth scope or ID token audience provided.', '{"error":"invalid_scope","error_description":"Invalid OAuth scope or ID token audience provided."}'), <traceback object at 0x7f4d783cc680>

at the line credentials.refresh(r)

Any idea?

[guillaumeblaquiere]

@deepamgupta can you describe your runtime context? Is it on your workstation? On Compute Engine? elsewhere?

[deepamgupta]

@guillaumeblaquiere Currently I am working on my Laptop. Using Pycharm debugger to run a flask application.

[marcusbevans]

I ran into the same problem trying to run a simple FastAPI app on Cloud Run just today. Just an FYI that the problem still exists.

Also, thank you for your article. It allowed me to build the work around.

[carlthome]

Same issue here with v4 and FastAPI.