read-only permissions don't throw an exception on an attempted write

[Taywee]

When a user with read-only credentials on a bucket connects and attempts to write a blob, it silently exits as if it succeeded even though nothing was written.

This service account has only read permissions on the bucket referenced (project and bucket name have been changed for this example).

credentials = ServiceAccountCredentials.from_json_keyfile_dict(credentials_dict)
client = storage.Client(credentials=credentials, project='testproject')
bucket = client.get_bucket('testbucket')
blob = bucket.blob('testblob')
blob.upload_from_string('this is a test blob\n')

Nothing gets uploaded, but the final function call exits with no indication of failure.

[dhermes]

@Taywee This sounds like either an error in the backend (I'd expect a 400 or 403) or an error in the credentials loaded. When you say "read-only credentials", you mean that your account only has read access to your bucket?

[Taywee]

Yes, that's what I mean. Only read-access to the bucket.

[Taywee]

When I update the permissions so the service user has write-permissions, it writes the object, but when the service account has only read permissions on the bucket, the code execution gives no indication.

[dhermes]

It's not really about the code execution, the backend should reject the request if you really didn't have write permissions, then we'd throw an exception for a non-200 response. By default read_write is one of the scopes used when creating auth tokens, so if the account does have write access it will.

Maybe the writer/reader permissions on that account aren't quite what they seem?

You saying the upload_from_string call fails to upload and there is no indication of the failure. Have you checked after the fact that nothing got uploaded?

[Taywee]

Yes. I've checked. It's certain that nothing gets uploaded, and after I switch the service account to allow write permissions, upload happens with no change to the python code.

[dhermes]

Interesting! I'd love to be able to reproduce to dig in deeper. Where in the cloud console have you set the read-only setting for your service account.

It seems Blob._check_response_error would catch the error if there was one?

[Taywee]

I set them in the bucket permissions.

The last entry is my service user there. When I switch the box to Writer, the blob can be written.

[dhermes]

I wonder what happens if your service account is also among owners and editors for your project. Can you confirm what group it's in?

[Taywee]

Looks like the service account is in neither of those groups.

[dhermes]

Digging in now to reproduce.

[dhermes]

I can't reproduce. I'm getting 403 Forbidden when I try to upload_from_string.

I have it as a reader on the bucket

and it is a viewer on the project

[Taywee]

Interesting. I'll make a new user and flow through the process now to see if I can replicate the issue, providing all the steps here (I'm trying to keep it as readable as possible, sorry if it's messy).

I go into API Library, then Credentials, and do create service account key.

---

I create it as such, and download the file absperfStandard-4e745b8c607d.json. I then go into cloud storage and create a bucket shoulddenybucket, altering it so the user should have read-only permission:

>>> from gcloud import storage
>>> from oauth2client.service_account import ServiceAccountCredentials
>>> credentials = ServiceAccountCredentials.from_json_keyfile_name('absperfStandard-4e745b8c607d.json')
>>> client = storage.Client(credentials=credentials, project='absperfstandard')
>>> bucket = client.get_bucket('shoulddenybucket')
>>> blob = bucket.blob('fooblob')
>>> blob.upload_from_string('this is a test string')
>>> 

---

And shoulddenybucket is still empty after a refresh:

---

But changing the user to a writer on shoulddenybucket and executing the same code with the same None returned from the upload_from_string method call:

---

The blob is there, but neither time raised any kind of exception. Looking in IAM and searching for the user, I similarly see nothing:

[Taywee]

Also, changing the user back to read-only, and trying to overwrite the named blob fooblob still exits without an exception being thrown, but does not actually overwrite the file.

Removing the user from bucket permissions entirely gives an expected exception:

>>> from gcloud import storage
>>> from oauth2client.service_account import ServiceAccountCredentials
>>> credentials = ServiceAccountCredentials.from_json_keyfile_name('absperfStandard-4e745b8c607d.json')
>>> client = storage.Client(credentials=credentials, project='absperfstandard')
>>> bucket = client.get_bucket('shoulddenybucket')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.4/site-packages/gcloud/storage/client.py", line 167, in get_bucket
    bucket.reload(client=self)
  File "/usr/lib64/python3.4/site-packages/gcloud/storage/_helpers.py", line 77, in reload
    _target_object=self)
  File "/usr/lib64/python3.4/site-packages/gcloud/connection.py", line 343, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.Forbidden: 403 Forbidden (GET https://www.googleapis.com/storage/v1/b/shoulddenybucket?projection=noAcl)
>>> 

[dhermes]

At the very beginning when you create the account, it is by default a project editor. Can you verify that?

[Taywee]

Where could I verify that? I don't see it anywhere in the IAM list.

[dhermes]

I used https://console.cloud.google.com/iam-admin/iam/project to check it.

[Taywee]

It's not in there:

[dhermes]

OK. I'll try to create an account and remove it from the list of project editors / viewers / etc.

[dhermes]

Still can't reproduce, I'm getting 403 forbiddens.

@thobrla can you take a look?

[thobrla]

@dhermes is there a way to make gcloud-python dump the HTTP request / response? I don't see anything obvious here, but as it stands have no indication of whether this is a client or service problem.

[dhermes]

Unfortunately no. I was hoping to reproduce so we didn't have to have @Taywee dig into the source.

@Taywee are you comfortable dropping into the source and adding a debug trace?

        if upload.strategy == RESUMABLE_UPLOAD:
            http_response = upload.stream_file(use_chunks=True)
        else:
            http_response = make_api_request(connection.http, request,
                                             retries=num_retries)
        import pdb  # ADDED
        pdb.set_trace()  # ADDED

        self._check_response_error(request, http_response)

and poking the request and http_response objects to get an idea of what's in them?

You can find the location of the file via

from gcloud.storage import blob
print(blob.__file__)

and if you installed with sudo, you'd need to use sudo to edit the file (alternatively you could install gcloud in a virtual env.)

[Taywee]

Ah, I found the issue. My gcloud was out of date (was on version 0.11.0). That version had no _check_response_error call. I thought I had updated it before testing this. That was entirely my my mistake. It works as expected now. Thanks for the help, and sorry for wasting your time.

[tseaver]

@Taywee no worries -- glad the latest release fixes the issue for you.

[dhermes]

Thanks @Taywee for the interaction throughout! Clearly it was a bug at one point in time!