feat(ingest): Add config option to set Bigquery credential in source config #3786
Conversation
| "project_id": "project-id-1234567", | ||
| "private_key_id": "d0121d0000882411234e11166c6aaa23ed5d74e0", | ||
| "private_key": "-----BEGIN PRIVATE KEY-----\nMIIyourkey\n-----END PRIVATE KEY-----", | ||
| "client_email": "[email protected]", |
|
|
||
| 1. Setup a ServiceAccount (https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-console) | ||
| and assign the previously created role to this service account | ||
| 2. Download a service account JSON keyfile: |
There was a problem hiding this comment.
There are 2 ":" in this line :)
| "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%suppproject-id-1234567.iam.gserviceaccount.com" | ||
| } | ||
| ``` | ||
| 3. Here you have two options: |
There was a problem hiding this comment.
Let's rephrase as:
To provide credentials to the source, you can either
- Set an environment variable
- Provide credential fields in the source recipe
|
|
||
| *or* | ||
|
|
||
| Set credential config in your source based on the credential json file like this: |
There was a problem hiding this comment.
Set credential config in your source based on the credential json file. For example:
|
|
||
| Set credential config in your source based on the credential json file like this: | ||
| ```yml | ||
| credential: |
There was a problem hiding this comment.
Wondering if this should also be in the example recipe :)
There was a problem hiding this comment.
Great idea, I will add it.
| logger.debug( | ||
| f"Creating temporary credential file at {self.credentials_path}" | ||
| ) | ||
| os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.credentials_path |
There was a problem hiding this comment.
Is this strictly required? It is strange we need to set an env variable
There was a problem hiding this comment.
We are using various APIs (sqlalchemy, BigQueryClient (https://cloud.google.com/bigquery/docs/reference/libraries#setting_up_authentication) and GCPLoggingClient (https://cloud.google.com/logging/docs/reference/libraries#setting_up_authentication), and with this way, we can authenticate with all of these.
I think it is simple and safe as this environment variable only lives during the life of the python app but works for all the APIs.
|
|
||
| # We can't use close as it is not called if the ingestion is not successful | ||
| def __del__(self): | ||
| if self.config.credentials_path: |
|
|
||
| with open(config.credentials_path) as jsonFile: | ||
| json_credential = json.load(jsonFile) | ||
| jsonFile.close() |
There was a problem hiding this comment.
Last q: Did you try setting the credentials using the config method manually?
There was a problem hiding this comment.
Yes, I tried it out a couple of times on my computer.
| server: "http://localhost:8080" |
Checklist