GCP 403 error with PGHoard 2.2.1

[TheMonsido]

I am trying to upload my backup data to GCP. When running PGHoard i am getting status code 401.
This error comes with version 2.2.1 of PGHoard and not version 2.1.1. When i am using PGHoard 2.2.1 and saving the data local the problem will not happen. I have not been testing other cloud providers (AWS etc) so i can't tell if this only happens in GCP but i am pretty sure its only there.

Setup:

Pre install:

sudo apt-get update
sudo apt-get install python3-pip
sudo apt install libpq-dev python3-dev
pip3 install psycopg2-binary
pip3 install google-api-python-client
pip3 install oauth2client

PGHoard install:

git clone https://github.com/aiven/pghoard
python3 setup.py bdist_egg
sudo python3 -m easy_install pghoard-2.2.1.dev65-py3.6.egg

PSQL:

sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-10

PSQL config:

wal_level = archive
max_wal_senders = 10
host    replication  pghoard  127.0.0.1/32  md5
CREATE USER pghoard PASSWORD 'PASSWORD_FOR_THE_PGHOARD_PSQL_USER' REPLICATION;

pghoard.json

{
    "backup_location": "/var/lib/pghoard",
    "syslog": true,
    "compression": {
      "algorithm": "snappy"
    },
    "backup_sites": {
       "example-site": {
            "active": true,
            "version": 10,
            "active_backup_mode": "pg_receivexlog",
            "basebackup_count": 30,
            "basebackup_interval_hours": 24,
            "basebackup_mode": "pipe",
            "basebackup_chunks_in_progress": 2,
            "nodes": [
                {
                    "host": "127.0.0.1",
                    "password": "PASSWORD_FOR_THE_PGHOARD_PSQL_USER",
                    "port": 5432,
                    "user": "pghoard"
                }
            ],
            "object_storage": {
                "bucket_name": "NAME_OF_BUCKET",
                "credential_file": "PATH/TO/THE/KEY.JSON",
                "project_id": "PROJECT_ID",
                "storage_type": "google"
            },
            "pg_bin_directory": "/usr/lib/postgresql/10/bin",
            "pg_data_directory": "/var/lib/postgresql/10/main"
        }
    }
}

After making this setup i try to run PGHoard with the command:

pghoard pghoard.json

And gets the current error:

2021-09-10 12:21:53,638	pghoard	MainThread	ERROR	Unexpected exception in PGHoard main loop
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 739, in run
    self.handle_site(site, site_config)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 615, in handle_site
    self.refresh_backup_list_and_delete_old(site)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 452, in refresh_backup_list_and_delete_old
    basebackups = self.get_remote_basebackups_info(site)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 382, in get_remote_basebackups_info
    storage = self.get_or_create_site_storage(site=site)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 377, in get_or_create_site_storage
    storage = get_transfer(storage_config)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/__init__.py", line 40, in get_transfer
    return storage_class(**storage_config)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py", line 106, in __init__
    self.bucket_name = self.get_or_create_bucket(bucket_name)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py", line 411, in get_or_create_bucket
    self._retry_on_reset(request, request.execute)
  File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py", line 167, in _retry_on_reset
    return action()
  File "/usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/googleapiclient/http.py", line 937, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 401 when requesting https://storage.googleapis.com/storage/v1/b/BUCKET_NAME?alt=json returned "Invalid Credentials". Details: "[{'message': 'Invalid Credentials', 'domain': 'global', 'reason': 'authError', 'locationType': 'header', 'location': 'Authorization'}]">
2021-09-10 12:21:55,320	pghoard	MainThread	WARNING	Quitting, signal: 2

When running PGHoard versionn 2.1.1 this error will not happen and will instead do as intended and upload the PSQL backup to my bucket on GCP

"Fix":
A way i have found around this is to either use version 2.1.1 of PGHoard instead of version 2.2.1 or another small "fix". If i change the line in the google.py (My path is /usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py) and change line 132 which is
return build("storage", "v1", http=http)
to the line from PGHoard 2.1.1 which is on line 109
return build("storage", "v1", credentials=self.google_creds)
i get to upload the data to my GCP bucket. This will do that the HTTP object won't be used but the google credentials object. This means some features won't be able to be used so this fix won't be able to help everyone.

This is only a temporary fix and should not be the right way to do it but i have not been able to find the right way to do this.
I hope there is some smart people who are able to find the right way to fix this or just explain something i do wrong.