Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miscellaneous fixes to BigQuery connector #959

Merged
merged 8 commits into from
Jan 31, 2024

Commits on Jan 30, 2024

  1. Configuration menu
    Copy the full SHA
    e63e403 View commit details
    Browse the repository at this point in the history
  2. Prefer not NoneType when inferring schema for Table load to BigQuery

    If a Parsons Table column has values like `[None, None, True, False]`,
    the BigQuery connector will infer that the appropriate type for this
    column is NoneType, which it will translate into a STRING type.
    
    This change ensures that types returned by petl.typecheck() will
    choose the first available type that isn't 'NoneType' if that is
    available.
    austinweisgrau committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    9bb57cc View commit details
    Browse the repository at this point in the history
  3. Fix BigQuery type map

    Source types ultimately come from `petl.typeset`, which calls
    `type(v).__name__`. This call does not include source module, but only
    the type name itself. e.g. `date` and not `datetime.date`
    austinweisgrau committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    3cc1f66 View commit details
    Browse the repository at this point in the history
  4. Fix commented out row to use job_config passed as argument

    It looks like this line was accidentally commented out
    austinweisgrau committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    9589593 View commit details
    Browse the repository at this point in the history
  5. Parse python datetime objects for BigQuery as datetime or timestamp

    Python datetime objects may represent timestamps or datetimes in
    BigQuery, depending on whether they do or do not have a timezone
    attached.
    austinweisgrau committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    e52bcd1 View commit details
    Browse the repository at this point in the history
  6. Only generate schema for BigQuery when table does not already exist

    Always passing a schema to BigQuery is not necessary, and introduces
    situations for provided schema to mismatch actual schema.
    
    When table already exists in BigQuery, fetch the schema from BigQuery
    austinweisgrau committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    4331900 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    2118cab View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Configuration menu
    Copy the full SHA
    d87688f View commit details
    Browse the repository at this point in the history