From 3dd747c2d70fe243cba3a69f422dfb3b83990c12 Mon Sep 17 00:00:00 2001 From: Ernesto Ongaro Date: Thu, 11 Apr 2024 01:24:54 +0200 Subject: [PATCH 1/7] Adds basic documentation and example --- website/docs/docs/build/python-models.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index 399a7cedcb6..a574a9aa4a9 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -630,6 +630,26 @@ def model(dbt, session): ) ``` +**External Access Integrations and Secrets** +If you want to use Snowflake’s [external access]([url](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview)) together with [secrets]([url](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference)) to query external APIs within dbt Python models there are some additional configurations to use: + +``` +import pandas +import snowflake.snowpark as snowpark + +def model(dbt, session: snowpark.Session): + dbt.config( + materialized="table", + secrets={"secret_variable_name": "test_secret"}, + external_access_integrations=["test_external_access_integration"], + ) + import _snowflake + return session.create_dataframe( + pandas.DataFrame( + [{"secret_value": _snowflake.get_generic_secret_string('secret_variable_name')}] + ) + ) +``` **About "sprocs":** dbt submits Python models to run as _stored procedures_, which some people call _sprocs_ for short. By default, dbt will create a named sproc containing your model's compiled Python code, and then _call_ it to execute. Snowpark has an Open Preview feature for _temporary_ or _anonymous_ stored procedures ([docs](https://docs.snowflake.com/en/sql-reference/sql/call-with.html)), which are faster and leave a cleaner query history. You can switch this feature on for your models by configuring `use_anonymous_sproc: True`. We plan to switch this on for all dbt + Snowpark Python models starting with the release of dbt Core version 1.4. From 46ab35425347c8786b0e0ebd83a00ac0448b55e4 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 11 Apr 2024 13:24:15 +0100 Subject: [PATCH 2/7] Update website/docs/docs/build/python-models.md --- website/docs/docs/build/python-models.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index a574a9aa4a9..fc6f5cefbaf 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -631,7 +631,7 @@ def model(dbt, session): ``` **External Access Integrations and Secrets** -If you want to use Snowflake’s [external access]([url](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview)) together with [secrets]([url](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference)) to query external APIs within dbt Python models there are some additional configurations to use: +To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: ``` import pandas From e603ada9d4ae7e0f989891b3d8683e9b8b9c67ea Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 11 Apr 2024 13:25:14 +0100 Subject: [PATCH 3/7] Update website/docs/docs/build/python-models.md --- website/docs/docs/build/python-models.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index fc6f5cefbaf..eff5e228e44 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -630,7 +630,7 @@ def model(dbt, session): ) ``` -**External Access Integrations and Secrets** +**External access integrations and secrets** To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: ``` From 99934d71c967cc3d3730874a9366bd40b190ae71 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 12 Apr 2024 16:16:18 +0100 Subject: [PATCH 4/7] Update python-models.md add colon --- website/docs/docs/build/python-models.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index eff5e228e44..93718f43995 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -630,8 +630,7 @@ def model(dbt, session): ) ``` -**External access integrations and secrets** -To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: +**External access integrations and secrets**: To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: ``` import pandas From 899667d28767cccb07304bec27a7609f1746e6df Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 12 Apr 2024 16:19:48 +0100 Subject: [PATCH 5/7] Update python-models.md add vb --- website/docs/docs/build/python-models.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index 93718f43995..884b9bdd321 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -629,7 +629,8 @@ def model(dbt, session): python_version="3.11" ) ``` - + + **External access integrations and secrets**: To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: ``` @@ -649,6 +650,9 @@ def model(dbt, session: snowpark.Session): ) ) ``` + + + **About "sprocs":** dbt submits Python models to run as _stored procedures_, which some people call _sprocs_ for short. By default, dbt will create a named sproc containing your model's compiled Python code, and then _call_ it to execute. Snowpark has an Open Preview feature for _temporary_ or _anonymous_ stored procedures ([docs](https://docs.snowflake.com/en/sql-reference/sql/call-with.html)), which are faster and leave a cleaner query history. You can switch this feature on for your models by configuring `use_anonymous_sproc: True`. We plan to switch this on for all dbt + Snowpark Python models starting with the release of dbt Core version 1.4. From 31eba3faf6bcda9aec90aac66fca5e4f3f409265 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 12 Apr 2024 16:31:38 +0100 Subject: [PATCH 6/7] Update python-models.md fix rendering --- website/docs/docs/build/python-models.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index 884b9bdd321..41c71caf229 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -629,6 +629,7 @@ def model(dbt, session): python_version="3.11" ) ``` + **External access integrations and secrets**: To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: From 8af042b88b99c80f3f5496965c4166a178addc8a Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 12 Apr 2024 16:40:12 +0100 Subject: [PATCH 7/7] remove space --- website/docs/docs/build/python-models.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md index 41c71caf229..87191d74d8f 100644 --- a/website/docs/docs/build/python-models.md +++ b/website/docs/docs/build/python-models.md @@ -631,7 +631,7 @@ def model(dbt, session): ``` - + **External access integrations and secrets**: To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: ```