-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Miscellaneous fixes to BigQuery connector #959
Commits on Jan 30, 2024
-
Configuration menu - View commit details
-
Copy full SHA for e63e403 - Browse repository at this point
Copy the full SHA e63e403View commit details -
Prefer not NoneType when inferring schema for Table load to BigQuery
If a Parsons Table column has values like `[None, None, True, False]`, the BigQuery connector will infer that the appropriate type for this column is NoneType, which it will translate into a STRING type. This change ensures that types returned by petl.typecheck() will choose the first available type that isn't 'NoneType' if that is available.
Configuration menu - View commit details
-
Copy full SHA for 9bb57cc - Browse repository at this point
Copy the full SHA 9bb57ccView commit details -
Source types ultimately come from `petl.typeset`, which calls `type(v).__name__`. This call does not include source module, but only the type name itself. e.g. `date` and not `datetime.date`
Configuration menu - View commit details
-
Copy full SHA for 3cc1f66 - Browse repository at this point
Copy the full SHA 3cc1f66View commit details -
Fix commented out row to use job_config passed as argument
It looks like this line was accidentally commented out
Configuration menu - View commit details
-
Copy full SHA for 9589593 - Browse repository at this point
Copy the full SHA 9589593View commit details -
Parse python datetime objects for BigQuery as datetime or timestamp
Python datetime objects may represent timestamps or datetimes in BigQuery, depending on whether they do or do not have a timezone attached.
Configuration menu - View commit details
-
Copy full SHA for e52bcd1 - Browse repository at this point
Copy the full SHA e52bcd1View commit details -
Only generate schema for BigQuery when table does not already exist
Always passing a schema to BigQuery is not necessary, and introduces situations for provided schema to mismatch actual schema. When table already exists in BigQuery, fetch the schema from BigQuery
Configuration menu - View commit details
-
Copy full SHA for 4331900 - Browse repository at this point
Copy the full SHA 4331900View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2118cab - Browse repository at this point
Copy the full SHA 2118cabView commit details
Commits on Jan 31, 2024
-
Configuration menu - View commit details
-
Copy full SHA for d87688f - Browse repository at this point
Copy the full SHA d87688fView commit details