fix(ingestion/airflow-plugin): updated the document for developers

datahub-project · Jun 2, 2024 · d28caac · d28caac
1 parent bc0c3ef
commit d28caac
Show file tree

Hide file tree

Showing 2 changed files with 22 additions and 1 deletion.
diff --git a/docs/lineage/airflow.md b/docs/lineage/airflow.md
@@ -69,7 +69,7 @@ enabled = True  # default
 | -------------------------- | -------------------- | ---------------------------------------------------------------------------------------- |
 | enabled                    | true                 | If the plugin should be enabled.                                                         |
 | conn_id                    | datahub_rest_default | The name of the datahub rest connection.                                                 |
-| cluster                    | prod                 | name of the airflow cluster                                                              |
+| cluster                    | prod                 | name of the airflow cluster, this is equivalent to the `env` of the instance                                                            |
 | capture_ownership_info     | true                 | Extract DAG ownership.                                                                   |
 | capture_tags_info          | true                 | Extract DAG tags.                                                                        |
 | capture_executions         | true                 | Extract task runs and success/failure statuses. This will show up in DataHub "Runs" tab. |

diff --git a/metadata-ingestion/developing.md b/metadata-ingestion/developing.md
@@ -34,6 +34,27 @@ cd metadata-ingestion-modules/airflow-plugin
 ../../gradlew :metadata-ingestion-modules:airflow-plugin:installDev
 source venv/bin/activate
 datahub version  # should print "DataHub CLI version: unavailable (installed in develop mode)"
+
+# start the airflow web server
+export AIRFLOW_HOME=~/airflow
+airflow webserver --port 8090 -d
+
+# start the airflow scheduler
+airflow scheduler
+
+# access the airflow service and run any of the DAG
+open http://localhost:8090/
+select any DAG and click on the `play arrow` button to start the DAG
+
+# add the debug lines in the codebase, i.e. in ./src/datahub_airflow_plugin/datahub_listener.py
+logger.debug("this is the sample debug line")
+
+# run the DAG again and you can see the debug lines in the task_run log at,
+1. click on the `timestamp` in the `Last Run` column
+2. select the task
+3. click on the `log` option
+
+P.S. if you are not able to see the log lines, then restart the `airflow scheduler` and rerun the DAG
 ```
 ### (Optional) Set up your Python environment for developing on Dagster Plugin