Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use name and type comparising when appending a dataframe into table #14

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/source/changelog.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
Changelog
=========

0.2.0 / 2017-?
0.2.0 / 2017-02-26
--------------

Fixed an issue with appending to a BigQuery table where fields have modes (NULLABLE,REQUIRED,REPEATED). The changes concern solely the comparision of the local (DataFrame) and remote (BQ) schema in GbqConnector.verify_schema function. The fix is to omit other field attributes than name and type.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add (:issue:`13`).


0.1.2 / 2017-02-23
------------------

Expand Down
6 changes: 5 additions & 1 deletion pandas_gbq/gbq.py
Original file line number Diff line number Diff line change
Expand Up @@ -563,8 +563,12 @@ def verify_schema(self, dataset_id, table_id, schema):
datasetId=dataset_id,
tableId=table_id).execute()['schema']

remote_fields = [{'name': field_remote['name'],
'type': field_remote['type']}
for field_remote in remote_schema['fields']]

fields_remote = set([json.dumps(field_remote)
for field_remote in remote_schema['fields']])
for field_remote in remote_fields])
fields_local = set(json.dumps(field_local)
for field_local in schema['fields'])

Expand Down
28 changes: 28 additions & 0 deletions pandas_gbq/tests/test_gbq.py
Original file line number Diff line number Diff line change
Expand Up @@ -1161,6 +1161,34 @@ def test_upload_data_flexible_column_order(self):
_get_project_id(), if_exists='append',
private_key=_get_private_key_path())

def test_verify_schema_ignores_field_mode(self):
test_id = "14"
test_schema_1 = {'fields': [{'name': 'A',
'type': 'FLOAT',
'mode': 'NULLABLE'},
{'name': 'B',
'type': 'FLOAT',
'mode': 'NULLABLE'},
{'name': 'C',
'type': 'STRING',
'mode': 'NULLABLE'},
{'name': 'D',
'type': 'TIMESTAMP',
'mode': 'REQUIRED'}]}
test_schema_2 = {'fields': [{'name': 'A',
'type': 'FLOAT'},
{'name': 'B',
'type': 'FLOAT'},
{'name': 'C',
'type': 'STRING'},
{'name': 'D',
'type': 'TIMESTAMP'}]}

self.table.create(TABLE_ID + test_id, test_schema_1)
self.assertTrue(self.sut.verify_schema(
Copy link
Contributor

@jreback jreback Feb 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just use

assert self.sut.verify_schema(.......), .....

the self.assertTrue was a nose convention, now using pytest so want to switch

Copy link
Author

@ghost ghost Feb 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I just looked the convention from the tests above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hah, I was going to change it on merge...but forgot...no worries

self.dataset_prefix + "1", TABLE_ID + test_id, test_schema_2),
'Expected schema to match')

def test_list_dataset(self):
dataset_id = self.dataset_prefix + "1"
self.assertTrue(dataset_id in self.dataset.datasets(),
Expand Down