Daily and night fishing events incremental implementation by Chris#104

Merged

smpiano merged 36 commits intodevelopfrom feature/PIPELINE-2170

Dec 17, 2024

+1,875-379

Commits on Oct 18, 2024

Rename fishing events queries and split processing steps

Chr96er
authored and
smpiano
committed
Modify incremental processing steps; 1 and 2 tested; 3 subject to change

Chr96er
authored and
smpiano
committed
Add closed events to overall merged table

Chr96er
authored and
smpiano
committed
Wrap up incremental fishing events; tested on staging pipeline

Chr96er
authored and
smpiano
committed
Use placeholder instead of hard coded table name

Chr96er
authored and
smpiano
committed
Don't reload backfill closed events during merge

Chr96er
authored and
smpiano
committed
Remove obsolete injected code; rename parameter; add event_end_date

Chr96er
authored and
smpiano
committed
Fix filters by adding quotes; rename parameter; filter delta table for oldest event only; set correct column order

Chr96er
authored and
smpiano
committed
Rename parameters; add CREATE TABLE statement; add event_end_date column

Chr96er
authored and
smpiano
committed
Add code that orchestrates delta and backfill of incremental load

Chr96er
authored and
smpiano
committed
Add readme containing challenges and solutions of incremental fishing events

Chr96er
authored and
smpiano
committed
Add requirements.txt

Chr96er
authored and
smpiano
committed
Fix PIPELINE-2192 - bug in splitting gaps in fishing events

Chr96er
authored and
smpiano
committed
Move filter for earliest delta event to correct CTE; remove overlapping fishing events

Chr96er
authored and
smpiano
committed
Further parameterise orchestration

Chr96er
authored and
smpiano
committed
Remove comment

Chr96er
authored and
smpiano
committed
Document processing steps: original and incremental

Chr96er
authored and
smpiano
committed
Properly indent stages

Chr96er
authored and
smpiano
committed
Finalize fishing events: move cleanup to filter; apply less restrictive fishing vessel filter; include and combine night loitering events; finish overall orchestration pipeline

Chr96er
authored and
smpiano
committed
Remove obsolete and add new parameters

Chr96er
authored and
smpiano
committed
Update description for incremental setup

Chr96er
authored and
smpiano
committed
Update description for incremental setup

Chr96er
authored and
smpiano
committed
Adds utilities: bigqueryhelper and parser of commands
smpiano
committed
Adds merge filter and auth schemas
smpiano
committed
Add client and harness to run the queries with their schemas
smpiano
committed
Setup the flake8 and runs black over the py code
smpiano
committed
parametrize the max_fishing_event_gap_hours and removes the creation in ddl move to the py code
smpiano
committed

Commits on Oct 21, 2024

Commits on Nov 19, 2024

Commits on Nov 22, 2024

Commits on Dec 17, 2024

Merge pull request #105 from GlobalFishingWatch/feature/PIPELINE-2170-restricted-view
smpiano
authored