Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record leaf nodes support #2943

Closed
vigbk opened this issue Jan 18, 2017 · 6 comments
Closed

Record leaf nodes support #2943

vigbk opened this issue Jan 18, 2017 · 6 comments
Assignees
Labels
api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@vigbk
Copy link

vigbk commented Jan 18, 2017

Python version - 2.7.13
Google-cloud-bigquery - 0.22.1

I am trying to insert data using the big query python client library. I am getting an error that record leaf types are not supported. The schema contains nested records that are repeatable. Is there a way using the client to insert nested repeatable records ?

my schema is as follows:
screen shot 2017-01-18 at 6 23 15 pm

@daspecster daspecster added the api: bigquery Issues related to the BigQuery API. label Jan 18, 2017
@daspecster
Copy link
Contributor

Thanks for reporting @vigbk!

Do you have some example code/stacktrace?

@tseaver is this kind of related to #2354?

@vigbk
Copy link
Author

vigbk commented Jan 18, 2017

@daspecster Here is an excerpt of the code I used

from google.cloud import bigquery
import os
import json
from datetime import datetime


def transform(item):
    newItem = {"insertId": getUUID(), "json": item}
    return newItem

def getTemplateSuffix():

    utcDate =  datetime.utcnow().strftime("%Y%m%d")
    templateSuffix = "_" + utcDate
    return templateSuffix

def uploadToBigQuery(data):

     rowData = json.loads(data)



     os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "credentials.json"

     # Instantiates a client
     bigquery_client = bigquery.Client("logs")

     dataset = bigquery_client.dataset("eventlogs")
     table = dataset.table("events")

     # Reload the table to get the schema.
     table.reload()

     rows = [rowData]
     print(rows)
     errors = table.insert_data(rows=rows, template_suffix=getTemplateSuffix())

     if not errors:
         print('Loaded 1 row into {}:{}'.format(dataset, table))
     else:
         print('Errors:')
         print(errors)


data = '{"user_dim":{"user_id":"12345","user_properties":[{"key":"subdomain_id","value":{"string_value":"custom.zendesk.com"}}]},"event_dim":{"date":"20170116","category":"ALERT","name":"NUMBER_NOT_A_MOBILE","timestamp_micros":1484580530384000,"params":[{"key":"network","value":{"string_value":"Singapore"}},{"key":"country","value":{"string_value":"Singapore"}}]}}'

uploadToBigQuery(data) 

@lukesneeringer
Copy link
Contributor

Is this a duplicate of #2951? (It seems like it is, but I am hesitant to close without being sure.)

@lukesneeringer lukesneeringer self-assigned this Feb 28, 2017
@lukesneeringer lukesneeringer added the priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. label Feb 28, 2017
@bjwatson bjwatson added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Feb 28, 2017
@dhermes
Copy link
Contributor

dhermes commented Mar 1, 2017

@tswast Can you confirm / deny the duplicateness?

@tswast
Copy link
Contributor

tswast commented Mar 1, 2017

These two issues look like duplicates to me, too.

@lukesneeringer
Copy link
Contributor

Okay. I marked the other one as important, and am going to close this one for personal sanity / tracking. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

6 participants