Devlop an example Airflow DAG based on a typical pipeline

[taktsutsumi]

Objective

Develop a simple Airflow workflow based on the structure outlined in Figure 1 of the An example RADPS Workflow Decomposition. Summarize key learning in a presentation.

Prerequisite

• Select and complete Apache Airflow Tutorial #12

Requirements

Develop an Airflow DAG replicating the structure of Figure 1 in the An example RADPS Workflow Decomposition.

• Tasks can be no-ops or simple sleep operations.
• The DAG should be run locally to verify execution.
• Include at least one conditional loop or branching task to demonstrate dynamic behavior.

Definition of Done

• An Airflow DAG mimicking the structure in Figure 1 is created.
• Tasks in the DAG use placeholders (e.g., no-op or sleep tasks) to demonstrate flow structure.
• At least one conditional loop or branching task is added to the DAG to illustrate dynamic behavior.
• The DAG is successfully executed in a local environment.
• The DAG includes basic reliability and fault tolerance features, such as retries and failure notifications.

Key Decision Points

1. Figure 1 Workflow Interpretation:
   • Agree on the structure and task definitions for the DAG based on Figure 1.
2. Conditional Task Design:
   • Decide how and where conditional tasks or task branching should be integrated into the DAG.

Artifacts

• A functional Airflow DAG script with reliability and fault tolerance features.
• Logs and outputs from local DAG execution.
• Notes on the design and implementation

Success Criteria

• The team has a foundational understanding of Airflow's core features, including reliability and fault tolerance.
• A working Airflow DAG, including conditional loops or task branching and failure handling, is successfully run locally.
• The team is equipped to design and implement more complex workflows in the future.

Potential Challenges

• Correctly interpreting and implementing Figure 1's DAG structure.
• Troubleshooting environment or execution issues during local DAG testing.
• Implementing conditional task branching correctly and ensuring its functionality in the DAG.