Skip to content

Commit

Permalink
added in secrets and external access integrations to python based udf
Browse files Browse the repository at this point in the history
  • Loading branch information
jonhopper-dataengineers committed Oct 26, 2023
1 parent 7dc49f5 commit c3f2df5
Show file tree
Hide file tree
Showing 4 changed files with 32 additions and 18 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# dbt_dataengineers_materializations Changelog
## 0.2.7.3

* Added in the ability to add `external_access_integrations` to user defined functions
* Added in the ability to add `secrets` to user defined functions

## 0.2.7.2

* Added in ability to set the timeout seconds for the task (`timeout`)
Expand Down
29 changes: 16 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This [dbt](https://github.com/dbt-labs/dbt) package contains materizations that
Add the following to your packages.yml file
```
- git: https://github.com/DataEngineersNZ/dbt-snowflake-datops-materilizations.git
revision: "0.2.7.2"
revision: "0.2.7.3"
```
----

Expand Down Expand Up @@ -437,23 +437,26 @@ To create a user defined function using Python, you need to add the following co
immutable=false,
runtime_version = '3.8',
packages = "('numpy','pandas','xgboost==1.5.0')",
external_access_integrations = "(your_access_integration)",
secrets = "('cred' = oauth_token )"
handler_name = 'udf',
return_type = 'variant')
}}
```

| property | description | required | default |
| -------------------- | ------------------------------------------------------- | -------- | ----------------------- |
| `materialized` | specifies the type of materialisation to run | yes | `user_defined_function` |
| `preferred_language` | specifies the landuage for the UDF function | yes | `python` |
| `is_secure` | specifies the function whether it is secure or not? | no | `false` |
| `immutable` | specifies the function is mutable or immutable | no | `false` |
| `return_type` | specifies the datatype for the return value | yes | |
| `parameters` | specifies the parameter for the function | no | |
| `runtime_version` | specifies the version of python | yes | |
| `packages` | specifies the packages required for the python function | yes | |
| `handler_name` | specifies the handler name for the function | yes | |

| property | description | required | default |
| ------------------------------ | ---------------------------------------------------------------- | -------- | ----------------------- |
| `materialized` | specifies the type of materialisation to run | yes | `user_defined_function` |
| `preferred_language` | specifies the landuage for the UDF function | yes | `python` |
| `is_secure` | specifies the function whether it is secure or not? | no | `false` |
| `immutable` | specifies the function is mutable or immutable | no | `false` |
| `return_type` | specifies the datatype for the return value | yes | |
| `parameters` | specifies the parameter for the function | no | |
| `runtime_version` | specifies the version of python | yes | |
| `packages` | specifies the packages required for the python function | yes | |
| `handler_name` | specifies the handler name for the function | yes | |
| `external_access_integrations` | specifies the name of the external access integration to be used | no | |
| `secrets` | specifies the secrets that are to be used by the function | no | |

## Materialized View
To create a Materialized View, you need to add the following config to the top of your model:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
{%- set imports = config.get('imports', default=null) -%}
{%- set target_path = config.get('target_path', default=null) -%}
{%- set runtime_version = config.get('runtime_version', default=null) -%}

{%- set external_access_integrations = config.get('external_access_integrations', default=none) %}
{%- set secrets = config.get('secrets', default=none) %}
{%- set identifier = config.get('override_name', default=model['alias'] ) -%}
{%- set return_type = config.get('return_type', default='varchar' ) -%}

Expand All @@ -45,7 +46,7 @@
{% if is_external %}
{{ dbt_dataengineers_materializations.snowflake_create_external_user_defined_functions_statement(target_relation, is_secure, immutable, parameters, return_type, api_integration, api_uri) }}
{% else %}
{{ dbt_dataengineers_materializations.snowflake_create_user_defined_functions_statement(target_relation, is_secure, preferred_language, immutable, parameters, return_type, sdk_version, import_Path, packages, handler_name, imports, target_path, runtime_version, sql) }}
{{ dbt_dataengineers_materializations.snowflake_create_user_defined_functions_statement(target_relation, is_secure, preferred_language, immutable, parameters, return_type, sdk_version, import_Path, packages, external_access_integrations, secrets, handler_name, imports, target_path, runtime_version, sql) }}
{% endif %}
{%- endcall %}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{%- macro snowflake_create_user_defined_functions_statement(relation, is_secure, preferred_language, immutable, parameters, return_type, sdk_version, import_Path, packages, handler_name, imports, target_path,runtime_version, sql) -%}
{%- macro snowflake_create_user_defined_functions_statement(relation, is_secure, preferred_language, immutable, parameters, return_type, sdk_version, import_Path, packages, external_access_integrations, secrets, handler_name, imports, target_path,runtime_version, sql) -%}

{% if is_secure %}
CREATE OR REPLACE SECURE FUNCTION {{ relation.include(database=(not temporary), schema=(not temporary)) }}({{ parameters }})
Expand All @@ -19,7 +19,12 @@ LANGUAGE {{ preferred_language }}
RUNTIME_VERSION = '{{ runtime_version }}'
HANDLER = '{{ handler_name }}'
PACKAGES = {{ packages }}

{% if external_access_integrations is not none %}
EXTERNAL_ACCESS_INTEGRATIONS = {{ external_access_integrations }}
{% endif %}
{% if secrets is not none %}
SECRETS = {{ secrets }}
{% endif %}
{% elif preferred_language == 'java' %}
handler = {{ handler_name }}
target_path = {{ target_path }}
Expand All @@ -31,4 +36,4 @@ $$
{{ sql }}
$$
;
{%- endmacro -%}
{%- endmacro -%}

0 comments on commit c3f2df5

Please sign in to comment.