Feature/python model v1 #209
Conversation
| models__simple_python_model_v2 = """ | ||
| import pandas | ||
|
|
||
| def model(dbt): |
There was a problem hiding this comment.
What the model code looks like
| spark = SparkSession.builder.appName('smallTest').getOrCreate() | ||
|
|
||
| spark.conf.set("viewsEnabled","true") | ||
| spark.conf.set("temporaryGcsBucket","python-model-test") |
There was a problem hiding this comment.
TODO in the comment above
dbt/adapters/bigquery/impl.py
Outdated
| from google.cloud import dataproc_v1 | ||
| from google.cloud import storage |
There was a problem hiding this comment.
- Let's add these as an
extras_require— e.g.pip install dbt-bigquery[dataproc] - Let's raise a clearer error below if a user wants to use Python models, and hasn't installed the extra
dbt/adapters/bigquery/impl.py
Outdated
| matches = re.match("gs://(.*?)/(.*)", response.driver_output_resource_uri) | ||
| output = ( | ||
| storage.Client() | ||
| .get_bucket(matches.group(1)) | ||
| .blob(f"{matches.group(2)}.000000000") | ||
| .download_as_string() | ||
| ) |
There was a problem hiding this comment.
There's no way to pass a full GCS resource URL into storage.Client()? That's sorta surprising to me. This isn't the worst regex, so I'm not strictly opposed to taking this approach if it's necessary
There was a problem hiding this comment.
I am not sure! I just followed a tutorial google provided for this, we can look into it more if needed
dbt/adapters/bigquery/impl.py
Outdated
| # Dataproc job output is saved to the Cloud Storage bucket | ||
| # allocated to the job. Use regex to obtain the bucket and blob info. | ||
| matches = re.match("gs://(.*?)/(.*)", response.driver_output_resource_uri) | ||
| output = ( |
There was a problem hiding this comment.
I take it that these are Spark logs being streamed back to GCS: https://cloud.google.com/dataproc/docs/guides/dataproc-job-output
If we felt motivated, we could try parsing these logs to infer metadata for the result. Not a priority right now
There was a problem hiding this comment.
Yes, I looked into it briefly, didn't found much things to surface up right now. We can take another look later on
* Use same auth for Dataproc + GCS * remove fetch result Co-authored-by: Chenyu Li <[email protected]>
* fix test and add clear install instruction * rename and fix format
ChenyuLInx
requested review from
gshank,
stu-k,
VersusFacit,
McKnight-42 and
jtcohen6
This change currently includes table materialization.
Also super happy to hear any feedback and what you think we missed.
TODO: