Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OK-602 Päivitetty valintaperusteiden lataus #111

Merged
merged 1 commit into from
Feb 6, 2025
Merged

OK-602 Päivitetty valintaperusteiden lataus #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 4 additions & 52 deletions dbt/models/dw/_dw.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,79 +213,31 @@ models:
- name: dw_metadata_dw_stored_at
data_type: timestamp with time zone
- name: dw_valintaperusteet_hakukohde
description: >-
Valintaperusteiden tietoja

lähdejärjestelmästä ei löydy muokkausaikaleimaa, joten tässä käytetään tiedoston luontihetkeä muokattu-aikaleimana. Tämä arvo voi olla duplikaatti jos sama tiedosto luetaan useaan kertaan, joten tämä DW-malli on erilainen.
description: Valintaperusteiden tietoja
columns:
- name: valinnanvaihe_id
description: ""
data_type: text
- name: hakukohde_oid
description: ""
data_type: text
- name: haku_oid
description: ""
data_type: text
- name: muokattu
data_type: timestamp with time zone
- name: tarjoaja_oid
description: ""
data_type: text
- name: viimeinenvalinnanvaihe
description: ""
data_type: integer
- name: hakukohteenvalintaperuste
description: ""
data_type: jsonb
- name: valinnanvaihe_nimi
description: ""
data_type: text
- name: valinnanvaihe_jarjestysluku
description: ""
data_type: integer
- name: valinnanvaihetyyppi
description: ""
data_type: text
- name: valintatapajono
description: ""
data_type: jsonb
- name: valintakoe
description: ""
data_type: jsonb
- name: jonot
description: ""
- name: valinnanvaiheet
data_type: jsonb
- name: aktiivinen
description: ""
data_type: boolean
- name: muokattu
description: muokattu on tiedoston luontihetki, koska kannassa ei ole muutostietoja. Tämä tarkoittaa sitä että tällä voin olla duplikaattiarvoja (sama tiedosto luettu useampaan kertaan)
data_type: timestamp with time zone
- name: dw_metadata_source_timestamp_at
description: ""
data_type: timestamp with time zone
- name: dw_metadata_stg_stored_at
description: ""
data_type: timestamp with time zone
- name: dw_metadata_dbt_copied_at
description: ""
data_type: timestamp with time zone
- name: dw_metadata_filename
description: ""
data_type: character varying
- name: dw_metadata_file_row_number
description: ""
data_type: integer
- name: dw_metadata_hash
description: ""
data_type: text
- name: dw_metadata_key
description: ""
data_type: text
- name: dw_metadata_timestamp
description: ""
data_type: timestamp with time zone
- name: dw_metadata_dw_stored_at
description: ""
data_type: timestamp with time zone
- name: dw_sure_suoritus
description: Sisältää Suoritusrekisterin suoritus tietoja
Expand Down
2 changes: 1 addition & 1 deletion dbt/models/dw/dw_valintaperusteet_hakukohde.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{%- set stage_model = ref('stg_valintaperusteet_hakukohde') -%}
{%- set key_columns_list = ['valinnanvaihe_id','muokattu'] -%}
{%- set key_columns_list = ['hakukohde_oid','muokattu'] -%}

{{ generate_dw_model_muokattu(stage_model, key_columns_list) }}
1 change: 1 addition & 0 deletions dbt/models/int/int_valintaperusteet_valintatapajono.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{{
config(
enabled=false,
materialized='incremental',
unique_key ='valinnanvaihe_id',
full_refresh = false,
Expand Down
43 changes: 4 additions & 39 deletions dbt/models/stg/_stg.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,65 +30,30 @@ models:
- name: dw_metadata_file_row_number
data_type: integer
- name: stg_valintaperusteet_hakukohde
description: ""
columns:
- name: valinnanvaihe_id
description: ""
data_type: text
- name: hakukohde_oid
description: ""
data_type: text
- name: haku_oid
description: ""
data_type: text
- name: muokattu
data_type: timestamp with time zone
- name: tarjoaja_oid
description: ""
data_type: text
- name: viimeinenvalinnanvaihe
description: ""
data_type: integer
- name: hakukohteenvalintaperuste
description: ""
data_type: jsonb
- name: valinnanvaihe_nimi
description: ""
data_type: text
- name: valinnanvaihe_jarjestysluku
description: ""
data_type: integer
- name: valinnanvaihetyyppi
description: ""
data_type: text
- name: valintatapajono
description: ""
data_type: jsonb
- name: valintakoe
description: ""
data_type: jsonb
- name: jonot
description: ""
- name: valinnanvaiheet
data_type: jsonb
- name: aktiivinen
description: ""
data_type: boolean
- name: muokattu
description: Hetki jolloin tiedot poimittiin lähdejärjestelmästä. Tässä tiedossa ei ole mukana datan muokkaushetkeä
data_type: timestamp with time zone
- name: dw_metadata_source_timestamp_at
description: ""
data_type: timestamp with time zone
- name: dw_metadata_stg_stored_at
description: ""
data_type: timestamp with time zone
- name: dw_metadata_dbt_copied_at
description: ""
data_type: timestamp with time zone
- name: dw_metadata_filename
description: ""
data_type: character varying
- name: dw_metadata_file_row_number
description: ""
data_type: integer

- name: stg_sure_suoritus
description: Sisältää Suoritusrekisterin suoritustietoja
columns:
Expand Down
12 changes: 2 additions & 10 deletions dbt/models/stg/stg_valintaperusteet_hakukohde.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,12 @@ with source as (

final as (
select
coalesce(data -> 'valinnanVaihe' ->> 'valinnanVaiheOid'::varchar, 'puuttuu') as valinnanvaihe_id,
data ->> 'hakukohdeOid'::varchar as hakukohde_oid,
data ->> 'hakuOid'::varchar as haku_oid,
(data ->> 'lastModified')::timestamptz as muokattu,
data ->> 'tarjoajaOid'::varchar as tarjoaja_oid,
data -> 'valinnanVaihe' ->> 'nimi'::varchar as valinnanvaihe_nimi,
(data -> 'valinnanVaihe' ->> 'valinnanVaiheJarjestysluku')::int as valinnanvaihe_jarjestysluku,
(data ->> 'viimeinenValinnanvaihe')::int as viimeinenValinnanvaihe,
(data -> 'valinnanVaihe' ->> 'aktiivinen')::boolean as aktiivinen,
data -> 'valinnanVaihe' ->> 'valinnanVaiheTyyppi'::varchar as valinnanvaihe_tyyppi,
(data -> 'hakukohteenValintaperuste')::jsonb as hakukohteenValintaperuste,
(data -> 'valinnanVaihe' -> 'valintatapajono')::jsonb as valintatapajono,
(data -> 'valinnanVaihe' -> 'valintakoe')::jsonb as valintakoe,
(data -> 'valinnanVaihe' ->> 'jonot')::jsonb as jonot,
dw_metadata_source_timestamp_at as muokattu,
(data -> 'valinnanVaiheet')::jsonb as valinnanvaiheet,
{{ metadata_columns() }}
from source
)
Expand Down
Loading