Skip to content

Commit

Permalink
User defined function (#1)
Browse files Browse the repository at this point in the history
* added user defined functions

* added description for readme.md

* fixed as per PR comment

* formatting

* fixed readme
  • Loading branch information
adhakal321 authored Aug 9, 2022
1 parent b2712b9 commit 615465a
Show file tree
Hide file tree
Showing 4 changed files with 201 additions and 0 deletions.
107 changes: 107 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Conatins the following materializations for Snowflake:
* Streams
* Tables
* Generic
* User Defined Functions

Adds the ability to create the raw tables based on the yml file

Expand Down Expand Up @@ -187,3 +188,109 @@ CREATE OR REPLACE api integration SnowWatch_Prod_API
API_KEY = 'xxxxxxxxxxxxxxxxxxxx'
enabled = true;
```

## User Defined Functions

When creating a user defined function, you can use a number of different languages. The following are the supported languages:

### SQL

To create a user defined function using SQL, you need to add the following config to the top of your model:

```sql
{{
config(materialized='user_defined_function',
preferred_language = 'sql',
is_secure= false,
immutable=false,
return_type = 'float')
}}
```

| property | description | required | default |
| -------------------- | --------------------------------------------------- | -------- | ----------------------- |
| `materialized` | specifies the type of materialisation to run | yes | `user_defined_function` |
| `preferred_language` | specifies the landuage for the UDF function | no | `sql` |
| `is_secure` | specifies the function whether it is secure or not? | no | `false` |
| `immutable` | specifies the function is mutable or immutable | no | `false` |
| `return_type` | specifies the datatype for the return value | yes | |
| `parameters` | specifies the parameter for the function | no | |

### Javascript

To create a user defined function using Javascript, you need to add the following config to the top of your model:

```sql
{{
config(materialized='user_defined_function',
preferred_language = 'javascript',
is_secure = True,
immutable = false,
return_type = 'float')
}}
```

| property | description | required | default |
| -------------------- | --------------------------------------------------- | -------- | ----------------------- |
| `materialized` | specifies the type of materialisation to run | yes | `user_defined_function` |
| `preferred_language` | specifies the landuage for the UDF function | yes | `javascript` |
| `is_secure` | specifies the function whether it is secure or not? | no | `false` |
| `immutable` | specifies the function is mutable or immutable | no | `false` |
| `return_type` | specifies the datatype for the return value | yes | |
| `parameters` | specifies the parameter for the function | no | |

### Java

To create a user defined function using Java, you need to add the following config to the top of your model:

```sql
{{
config(materialized='user_defined_function',
preferred_language = 'java',
is_secure = false,
handler_name = "'testfunction.echoVarchar'",
target_path = "'@~/testfunction.jar'",
return_type = 'varchar',
parameters = 'my_string varchar')
}}
```

| property | description | required | default |
| -------------------- | -------------------------------------------------------- | -------- | ----------------------- |
| `materialized` | specifies the type of materialisation to run | yes | `user_defined_function` |
| `preferred_language` | specifies the landuage for the UDF function | yes | `java` |
| `is_secure` | specifies the function whether it is secure or not? | no | `false` |
| `immutable` | specifies the function is mutable or immutable | no | `false` |
| `handler_name` | specifies the combination of class and the function name | yes | |
| `target_path` | specifies the path for the jar file | yes | |
| `return_type` | specifies the datatype for the return value | yes | |
| `parameters` | specifies the parameter for the function | no | |

### Python

To create a user defined function using Python, you need to add the following config to the top of your model:

```sql
{{
config(materialized='user_defined_function',
preferred_language = 'python',
is_secure= false,
immutable=false,
runtime_version = '3.8',
packages = "('numpy','pandas','xgboost==1.5.0')",
handler_name = 'udf',
return_type = 'variant')
}}
```

| property | description | required | default |
| -------------------- | ------------------------------------------------------- | -------- | ----------------------- |
| `materialized` | specifies the type of materialisation to run | yes | `user_defined_function` |
| `preferred_language` | specifies the landuage for the UDF function | yes | `python` |
| `is_secure` | specifies the function whether it is secure or not? | no | `false` |
| `immutable` | specifies the function is mutable or immutable | no | `false` |
| `return_type` | specifies the datatype for the return value | yes | |
| `parameters` | specifies the parameter for the function | no | |
| `runtime_version` | specifies the version of python | yes | |
| `packages` | specifies the packages required for the python function | yes | |
| `handler_name` | specifies the handler name for the function | yes | |
55 changes: 55 additions & 0 deletions macros/user_defined_functions/snowflake__user_defined_function.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
This materialization is used for creating user defined function objects.
The idea behind this materialization is for ability to define CREATE user defined function statements and have dbt use the necessary logic
of deploying the user defined function in a consistent manner and logic.
*/
{%- materialization user_defined_function, adapter='snowflake' -%}
{%- set preferred_language = config.get('preferred_language', default=SQL) -%}
{%- set parameters = config.get('parameters', default='') -%}
{%- set is_secure = config.get('is_secure', default=false) -%}
{%- set immutable = config.get('immutable', default=false) -%}

{%- set sdk_version = config.get('sdk_version', default=null) -%}
{%- set import_Path = config.get('import_Path', default=null) -%}
{%- set packages = config.get('packages', default=null) -%}
{%- set handler_name = config.get('handler_name', default=null) -%}
{%- set imports = config.get('imports', default=null) -%}
{%- set target_path = config.get('target_path', default=null) -%}
{%- set runtime_version = config.get('runtime_version', default=null) -%}

{%- set identifier = config.get('override_name', default=model['alias'] ) -%}
{%- set return_type = config.get('return_type', default='varchar' ) -%}


{%- set target_relation = api.Relation.create( identifier=identifier, schema=schema, database=database) -%}

{%- set has_transactional_hooks = (hooks | selectattr('transaction', 'equalto', True) | list | length) > 0 %}

-- setup
{{ run_hooks(pre_hooks, inside_transaction=False) }}

-- BEGIN happens here:
{{ run_hooks(pre_hooks, inside_transaction=True) }}

--------------------------------------------------------------------------------------------------------------------
-- build model

{% call statement('main') -%}
{{ dbt_dataengineers_materilizations.snowflake_create_user_defined_functions_statement(target_relation, is_secure, preferred_language, immutable, parameters, return_type, sdk_version, import_Path, packages, handler_name, imports, target_path, runtime_version, sql) }}

{%- endcall %}

--------------------------------------------------------------------------------------------------------------------
-- build model
{{ run_hooks(post_hooks, inside_transaction=True) }}

-- `COMMIT` happens here
{{ adapter.commit() }}
{{ run_hooks(post_hooks, inside_transaction=False) }}

-- return
{{ return({'relations': [target_relation]}) }}

{%- endmaterialization -%}
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{%- macro snowflake_create_user_defined_functions_statement(relation, is_secure, preferred_language, immutable, parameters, return_type, sdk_version, import_Path, packages, handler_name, imports, target_path,runtime_version, sql) -%}

{% if is_secure %}
CREATE OR REPLACE SECURE FUNCTION {{ relation.include(database=(not temporary), schema=(not temporary)) }}({{ parameters }})
{% else %}
CREATE OR REPLACE FUNCTION {{ relation.include(database=(not temporary), schema=(not temporary)) }}({{ parameters }})
{% endif %}
RETURNS {{ return_type }}
{% if preferred_language != 'sql' %}
LANGUAGE {{ preferred_language }}
{% endif %}

{% if preferred_language == 'python' %}
RUNTIME_VERSION = '{{ runtime_version }}'
HANDLER = '{{ handler_name }}'
PACKAGES = {{ packages }}

{% elif preferred_language == 'java' %}
handler = {{ handler_name }}
target_path = {{ target_path }}
{% endif %}

AS

{% if preferred_language != 'python' %}
'{{ sql }}'
{% else %}
{{ sql }}
{% endif %}
;
{%- endmacro -%}
8 changes: 8 additions & 0 deletions macros/user_defined_functions/user_defined_functions.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: 2

macros:

- name: snowflake_user_defined_functions
docs:
show: false

0 comments on commit 615465a

Please sign in to comment.