Fix system test for dataform operators (#44729)

docs/apache-airflow-providers-google/operators/cloud/dataform.rst

@@ -25,32 +25,35 @@ Dataform lets you manage data transformation in the Extraction, Loading, and Tra
 for data integration. After raw data is extracted from source systems and loaded into BigQuery, Dataform
 helps you to transform it into a well-defined, tested, and documented suite of data tables.
 
-For more information about the task visit `Dataform production documentation <Product documentation <https://cloud.google.com/dataform/docs>`__
+For more information about the task visit `Dataform documentation <https://cloud.google.com/dataform/docs>`__
 
 
 Configuration
 -------------
 
 Before you can use the Dataform operators you need to initialize repository and workspace, for more information
-about this visit `Dataform Production documentation <Product documentation <https://cloud.google.com/dataform/docs>`__
+about this visit `Dataform documentation <https://cloud.google.com/dataform/docs>`__
 
 Create Repository
 -----------------
-Create repository for tracking your code in Dataform service. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateRepositoryOperator`
+To create a repository for tracking your code in Dataform service use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateRepositoryOperator`.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
     :start-after: [START howto_operator_create_repository]
     :end-before: [END howto_operator_create_repository]
 
+
 Create Workspace
 ----------------
-Create workspace for storing your code in Dataform service. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkspaceOperator`
+To create a workspace for storing your code in Dataform service use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkspaceOperator`.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
@@ -61,192 +64,205 @@ Create workspace for storing your code in Dataform service. Example of usage can
 
 Create Compilation Result
 -------------------------
-A simple configuration to create Compilation Result can look as followed:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateCompilationResultOperator`
+To create a Compilation Result use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateCompilationResultOperator`.
+A simple configuration can look as followed:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
     :start-after: [START howto_operator_create_compilation_result]
     :end-before: [END howto_operator_create_compilation_result]
 
+
 Get Compilation Result
 ----------------------
 
-To get a Compilation Result you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetCompilationResultOperator`
+To get a Compilation Result you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetCompilationResultOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_get_compilation_result]
     :end-before: [END howto_operator_get_compilation_result]
 
+
 Create Workflow Invocation
 --------------------------
 
-To create a Workflow Invocation you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkflowInvocationOperator`
-
-We have possibility to run this operation in the sync mode and async, for async operation we also have
-a sensor:
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationStateSensor`
-
-We also have a sensor to check the status of a particular action for a workflow invocation triggered
-asynchronously.
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationActionStateSensor`
-
+To create a Workflow Invocation you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformCreateWorkflowInvocationOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_create_workflow_invocation]
     :end-before: [END howto_operator_create_workflow_invocation]
 
+
+We have possibility to run this operation in the sync mode and async, for async operation we also have
+a sensor
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationStateSensor`.
+
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_create_workflow_invocation_async]
     :end-before: [END howto_operator_create_workflow_invocation_async]
 
+
+We also have a sensor
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformWorkflowInvocationActionStateSensor`
+to check the status of a particular action for a workflow invocation triggered asynchronously.
+
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_create_workflow_invocation_action_async]
     :end-before: [END howto_operator_create_workflow_invocation_action_async]
 
+
 Get Workflow Invocation
 -----------------------
 
-To get a Workflow Invocation you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetWorkflowInvocationOperator`
+To get a Workflow Invocation you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformGetWorkflowInvocationOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_get_workflow_invocation]
     :end-before: [END howto_operator_get_workflow_invocation]
 
+
 Query Workflow Invocation Action
 --------------------------------
 
-To query Workflow Invocation Actions you can use:
-
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformQueryWorkflowInvocationActionsOperator`
+To query Workflow Invocation Actions you can use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformQueryWorkflowInvocationActionsOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_query_workflow_invocation_actions]
     :end-before: [END howto_operator_query_workflow_invocation_actions]
 
+
 Cancel Workflow Invocation
 --------------------------
 
-To cancel a Workflow Invocation you can use:
-
-:class:`~airflow.providers.google.cloud.sensors.dataform.DataformCancelWorkflowInvocationOperator`
+To cancel a Workflow Invocation you can use
+:class:`~airflow.providers.google.cloud.sensors.dataform.DataformCancelWorkflowInvocationOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_cancel_workflow_invocation]
     :end-before: [END howto_operator_cancel_workflow_invocation]
 
+
 Delete Repository
 -----------------
-Deletes repository. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator`
+To delete a repository use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator`.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
-    :start-after: [START howto_operator_delete_workspace]
-    :end-before: [END howto_operator_delete_workspace]
+    :start-after: [START howto_operator_delete_repository]
+    :end-before: [END howto_operator_delete_repository]
+
 
 Delete Workspace
 ----------------
-Deletes workspace. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator`
+To delete a workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformDeleteWorkspaceOperator`.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
-    :start-after: [START howto_operator_delete_repository]
-    :end-before: [END howto_operator_delete_repository]
+    :start-after: [START howto_operator_delete_workspace]
+    :end-before: [END howto_operator_delete_workspace]
+
 
 Remove file
 -----------
-Removes file. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveFileOperator`
+To remove a file use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveFileOperator`.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
     :start-after: [START howto_operator_remove_file]
     :end-before: [END howto_operator_remove_file]
 
+
 Remove directory
 ----------------
-Removes directory. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveDirectoryOperator`
+To remove a directory use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformRemoveDirectoryOperator`.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
     :start-after: [START howto_operator_remove_directory]
     :end-before: [END howto_operator_remove_directory]
 
+
 Initialize workspace
 --------------------
-Creates default projects structure for provided workspace. Before it can be done workspace and repository should be created. Example of usage can be seen below:
 
-:class:`~airflow.providers.google.cloud.utils.dataform.make_initialization_workspace_flow`
+Creates default projects structure for provided workspace. Before it can be done workspace and repository should be created.
+Example of usage can be seen below:
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 0
     :start-after: [START howto_initialize_workspace]
     :end-before: [END howto_initialize_workspace]
 
+
 Write file to workspace
 -----------------------
-Writes file with given content to specified workspace.
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformWriteFileOperator`
+To write a file with given content to specified workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformWriteFileOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_write_file]
     :end-before: [END howto_operator_write_file]
 
+
 Make directory in workspace
 ---------------------------
-Make directory with given path in specified workspace.
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformMakeDirectoryOperator`
+To make a directory with given path in specified workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformMakeDirectoryOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_make_directory]
     :end-before: [END howto_operator_make_directory]
 
+
 Install NPM packages
 --------------------
-Installs npm packages for specified workspace
 
-:class:`~airflow.providers.google.cloud.operators.dataform.DataformInstallNpmPackagesOperator`
+To install npm packages for specified workspace use
+:class:`~airflow.providers.google.cloud.operators.dataform.DataformInstallNpmPackagesOperator`.
 
 .. exampleinclude:: /../../providers/tests/system/google/cloud/dataform/example_dataform.py
     :language: python

providers/tests/system/google/cloud/dataform/example_dataform.py

@@ -171,33 +171,22 @@
         region=REGION,
         repository_id=REPOSITORY_ID,
         workflow_invocation_id=(
-            "{{ task_instance.xcom_pull('create-workflow-invocation')['name'].split('/')[-1] }}"
+            "{{ task_instance.xcom_pull('create-workflow-invocation-async')['name'].split('/')[-1] }}"
         ),
         expected_statuses={WorkflowInvocation.State.SUCCEEDED},
     )
     # [END howto_operator_create_workflow_invocation_async]
 
     # [START howto_operator_create_workflow_invocation_action_async]
-    create_workflow_invocation_async_action = DataformCreateWorkflowInvocationOperator(
-        task_id="create-workflow-invocation-async-action",
-        project_id=PROJECT_ID,
-        region=REGION,
-        repository_id=REPOSITORY_ID,
-        asynchronous=True,
-        workflow_invocation={
-            "compilation_result": "{{ task_instance.xcom_pull('create-compilation-result')['name'] }}"
-        },
-    )
-
     is_workflow_invocation_action_done = DataformWorkflowInvocationActionStateSensor(
         task_id="is-workflow-invocation-action-done",
         project_id=PROJECT_ID,
         region=REGION,
         repository_id=REPOSITORY_ID,
         workflow_invocation_id=(
-            "{{ task_instance.xcom_pull('create-workflow-invocation')['name'].split('/')[-1] }}"
+            "{{ task_instance.xcom_pull('create-workflow-invocation-async')['name'].split('/')[-1] }}"
         ),
-        target_name="YOUR_TARGET_HERE",
+        target_name="first_view",
         expected_statuses={WorkflowInvocationAction.State.SUCCEEDED},
         failure_statuses={
             WorkflowInvocationAction.State.SKIPPED,
@@ -340,26 +329,37 @@
     )
     (
         last_initialization_step
-        >> install_npm_packages
-        >> create_compilation_result
-        >> get_compilation_result
-        >> create_workflow_invocation
-        >> get_workflow_invocation
-        >> query_workflow_invocation_actions
-        >> create_workflow_invocation_async
-        >> is_workflow_invocation_done
-        >> create_workflow_invocation_async_action
-        >> is_workflow_invocation_action_done
-        >> create_workflow_invocation_for_cancel
-        >> cancel_workflow_invocation
         >> make_test_directory
         >> write_test_file
-        # TEST TEARDOWN
         >> remove_test_file
         >> remove_test_directory
         >> delete_dataset
-        >> delete_workspace
-        >> delete_repository
+    )
+    (
+        last_initialization_step
+        >> install_npm_packages
+        >> create_compilation_result
+        >> [
+            get_compilation_result,
+            create_workflow_invocation,
+            create_workflow_invocation_async,
+            create_workflow_invocation_for_cancel,
+        ]
+    )
+    (
+        create_workflow_invocation
+        >> [get_workflow_invocation, query_workflow_invocation_actions]
+        >> delete_dataset
+    )
+    (
+        create_workflow_invocation_async
+        >> [is_workflow_invocation_done, is_workflow_invocation_action_done]
+        >> delete_dataset
+    )
+    create_workflow_invocation_for_cancel >> cancel_workflow_invocation >> delete_dataset
+    (
+        # TEST TEARDOWN
+        delete_dataset >> delete_workspace >> delete_repository
     )
 
     # ### Everything below this line is not part of example ###