Docs for triggered_dataset_event #34410
Conversation
…heck the templates page on the datasets page
|
Good to go 👍 |
| WHERE "updated_at" >= '{{ (triggering_dataset_events.values() | first | first).source_dag_run.data_interval_start }}' | ||
| AND "updated_at" < '{{ (triggering_dataset_events.values() | first | first).source_dag_run.data_interval_end }}'; |
There was a problem hiding this comment.
to use your words this is ugly... maybe it's weird to use a user-defined macro in the example dag, but it would make this easier to follow i think, and it's probably a better thing to do
do these sort properly? are we actually getting the first dataset event by time (for lower bound), and the last one by time for upper bound?
There was a problem hiding this comment.
😂 hoisted by my own petard. I feel like, maybe future work could be an Airflow-wide macro to make this less bend-over-backward?
I'm hesitant to add a DAG-specific macro to the example DAG as that might be "a step too far" if people are unfamiliar with multiple concepts being presented at once, and then don't know why things are breaking if they just copypaste the jinja template, but don't know they also need the special macro.
I think the | first also make the structure of it obvious. So, though ugly, I vaguely prefer being explicit with the ugliness.
re: sorting - I have no idea how the list sorts. Luckily this is an example for a single-dataset schedule, so it doesn't matter 😄
There was a problem hiding this comment.
as you wish, as you wish
There was a problem hiding this comment.
I'm hesitant to add a DAG-specific macro to the example DAG as that might be "a step too far" if people are unfamiliar with multiple concepts being presented at once
I agree, it's better to use a standard/official thing, even if it's ugly
There was a problem hiding this comment.
I've bet that someone might be confused what actually first, not users all familiar with Jinja filters and even don't know which are builtin, for example it might be a good idea to add this link in code example comments https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.first
There was a problem hiding this comment.
I will just quietly sneak in a change to macro later ;)
There was a problem hiding this comment.
Added the link @Taragolis - that also gives an opportunity to explain what the heck is happening there 😅
| WHERE "updated_at" >= '{{ (triggering_dataset_events.values() | first | first).source_dag_run.data_interval_start }}' | ||
| AND "updated_at" < '{{ (triggering_dataset_events.values() | first | first).source_dag_run.data_interval_end }}'; |
There was a problem hiding this comment.
I'm hesitant to add a DAG-specific macro to the example DAG as that might be "a step too far" if people are unfamiliar with multiple concepts being presented at once
I agree, it's better to use a standard/official thing, even if it's ugly
* add templates reference for triggering_dataset_events and a note to check the templates page on the datasets page * add working example, correct type of triggering_dataset_events * explain | first | first (cherry picked from commit 21610c1)
templates.rstreference for triggering_dataset_eventsValidated with breeze locally :) Still can't get stuff like
:class:`~airflow.models.dataset.DatasetEvent`to link correctly, but that's outside the scope of this