From 9359c3cdafde939d90af83b9180ca9f2afb4bbe8 Mon Sep 17 00:00:00 2001 From: Laiba Zaman Date: Fri, 31 Mar 2023 14:20:47 +0000 Subject: [PATCH 1/2] added dataflow folder and guide --- en/google/dataflow/dataflow-hanged-jobs.md | 47 ++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 en/google/dataflow/dataflow-hanged-jobs.md diff --git a/en/google/dataflow/dataflow-hanged-jobs.md b/en/google/dataflow/dataflow-hanged-jobs.md new file mode 100644 index 000000000..2231e6834 --- /dev/null +++ b/en/google/dataflow/dataflow-hanged-jobs.md @@ -0,0 +1,47 @@ +[![CloudSploit](https://cloudsploit.com/img/logo-new-big-text-100.png "CloudSploit")](https://cloudsploit.com) + +# GOOGLE / Dataflow / Dataflow Hanged Jobs +## Quick Info + +| | | +|-|-| +| **Plugin Title** | Dataflow Hanged Jobs | +| **Cloud** | GOOGLE | +| **Category** | Datflow | +| **Description** | Ensure that Cloud Dataflow jobs are not in same state for more than defined amount of time. | +| **More Info** | Cloud Dataflow jobs transit between different states and normally reach terminal state. If they stay in same state for abnormal amount of time, job administrator should stop such jobs to save unnecessary cost. | +| **GOOGLE Link** | https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel | +| **Recommended Action** | Cancel/stop Dataflow jobs which are in same state for more than set amount of time. | + +## Detailed Remediation Steps +### Name + +gcloud dataflow jobs cancel - cancels all jobs that match the command line arguments + +### Synopsis + +`gcloud dataflow jobs cancel` `*[JOB_ID](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#JOB_ID)*` [`*[JOB_ID](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#JOB_ID)*` ...] [`[--force](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#--force)`] [`[--region](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#--region)`=`*REGION_ID*`] [`*[GCLOUD_WIDE_FLAG](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#GCLOUD-WIDE-FLAGS) ...*`] + +### Flags + +`--force` + +Forcibly cancels a Dataflow job, leaking any VMs the Dataflow job created. Regular cancel must have been attempted at least 30 minutes ago for a job to be force cancelled. + +`--region`=`*REGION_ID*` + +The region ID of the jobs' regional endpoint. Defaults to 'us-central1'. + +### GCloud Wide Flags + +These flags are available to all commands: `[--access-token-file](https://cloud.google.com/sdk/gcloud/reference#--access-token-file)`, `[--account](https://cloud.google.com/sdk/gcloud/reference#--account)`, `[--billing-project](https://cloud.google.com/sdk/gcloud/reference#--billing-project)`, `[--configuration](https://cloud.google.com/sdk/gcloud/reference#--configuration)`, `[--flags-file](https://cloud.google.com/sdk/gcloud/reference#--flags-file)`, `[--flatten](https://cloud.google.com/sdk/gcloud/reference#--flatten)`, `[--format](https://cloud.google.com/sdk/gcloud/reference#--format)`, `[--help](https://cloud.google.com/sdk/gcloud/reference#--help)`, `[--impersonate-service-account](https://cloud.google.com/sdk/gcloud/reference#--impersonate-service-account)`, `[--log-http](https://cloud.google.com/sdk/gcloud/reference#--log-http)`, `[--project](https://cloud.google.com/sdk/gcloud/reference#--project)`, `[--quiet](https://cloud.google.com/sdk/gcloud/reference#--quiet)`, `[--trace-token](https://cloud.google.com/sdk/gcloud/reference#--trace-token)`, `[--user-output-enabled](https://cloud.google.com/sdk/gcloud/reference#--user-output-enabled)`, `[--verbosity](https://cloud.google.com/sdk/gcloud/reference#--verbosity)`. + +Run `$ [gcloud help](https://cloud.google.com/sdk/gcloud/reference)` for details. + +### Notes + +This variant is also available: + +``` +gcloud beta dataflow jobs cancel +``` \ No newline at end of file From 5a1d2e8be320863cd814e6030a3bfcea8edc195f Mon Sep 17 00:00:00 2001 From: Amanda Reed <96201528+areed42@users.noreply.github.com> Date: Tue, 4 Apr 2023 14:28:23 -0400 Subject: [PATCH 2/2] Update dataflow-hanged-jobs.md Made some changes for clarity --- en/google/dataflow/dataflow-hanged-jobs.md | 38 ++++++---------------- 1 file changed, 10 insertions(+), 28 deletions(-) diff --git a/en/google/dataflow/dataflow-hanged-jobs.md b/en/google/dataflow/dataflow-hanged-jobs.md index 2231e6834..809daea13 100644 --- a/en/google/dataflow/dataflow-hanged-jobs.md +++ b/en/google/dataflow/dataflow-hanged-jobs.md @@ -14,34 +14,16 @@ | **Recommended Action** | Cancel/stop Dataflow jobs which are in same state for more than set amount of time. | ## Detailed Remediation Steps -### Name +1. The 'gcloud dataflow jobs cancel' commandlet cancels all jobs that match the command line arguments. -gcloud dataflow jobs cancel - cancels all jobs that match the command line arguments +2. To cancel jobs, log into the Google Cloud Console and open a cloud shell instance. See [Using Cloud Shell](#https://cloud.google.com/shell/docs/using-cloud-shell) for more details on access. -### Synopsis +3. Run the command in the following manner, replacing the variables with values corresponding to your organization: + ``` + gcloud dataflow jobs cancel JOB_ID [JOB_ID …] [--force] [--region=REGION_ID] [GCLOUD_WIDE_FLAG …] + ``` -`gcloud dataflow jobs cancel` `*[JOB_ID](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#JOB_ID)*` [`*[JOB_ID](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#JOB_ID)*` ...] [`[--force](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#--force)`] [`[--region](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#--region)`=`*REGION_ID*`] [`*[GCLOUD_WIDE_FLAG](https://cloud.google.com/sdk/gcloud/reference/dataflow/jobs/cancel#GCLOUD-WIDE-FLAGS) ...*`] - -### Flags - -`--force` - -Forcibly cancels a Dataflow job, leaking any VMs the Dataflow job created. Regular cancel must have been attempted at least 30 minutes ago for a job to be force cancelled. - -`--region`=`*REGION_ID*` - -The region ID of the jobs' regional endpoint. Defaults to 'us-central1'. - -### GCloud Wide Flags - -These flags are available to all commands: `[--access-token-file](https://cloud.google.com/sdk/gcloud/reference#--access-token-file)`, `[--account](https://cloud.google.com/sdk/gcloud/reference#--account)`, `[--billing-project](https://cloud.google.com/sdk/gcloud/reference#--billing-project)`, `[--configuration](https://cloud.google.com/sdk/gcloud/reference#--configuration)`, `[--flags-file](https://cloud.google.com/sdk/gcloud/reference#--flags-file)`, `[--flatten](https://cloud.google.com/sdk/gcloud/reference#--flatten)`, `[--format](https://cloud.google.com/sdk/gcloud/reference#--format)`, `[--help](https://cloud.google.com/sdk/gcloud/reference#--help)`, `[--impersonate-service-account](https://cloud.google.com/sdk/gcloud/reference#--impersonate-service-account)`, `[--log-http](https://cloud.google.com/sdk/gcloud/reference#--log-http)`, `[--project](https://cloud.google.com/sdk/gcloud/reference#--project)`, `[--quiet](https://cloud.google.com/sdk/gcloud/reference#--quiet)`, `[--trace-token](https://cloud.google.com/sdk/gcloud/reference#--trace-token)`, `[--user-output-enabled](https://cloud.google.com/sdk/gcloud/reference#--user-output-enabled)`, `[--verbosity](https://cloud.google.com/sdk/gcloud/reference#--verbosity)`. - -Run `$ [gcloud help](https://cloud.google.com/sdk/gcloud/reference)` for details. - -### Notes - -This variant is also available: - -``` -gcloud beta dataflow jobs cancel -``` \ No newline at end of file +4. This variant is also available: + ``` + gcloud beta dataflow jobs cancel + ```