Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

t - aws glue job #36770

Merged
merged 6 commits into from
Apr 16, 2024
Merged

t - aws glue job #36770

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3380d20
r/aws_glue_job: update test and documentation
nikhil-goenka Apr 6, 2024
2b8e32e
r/aws_glue_job: update test and documentation
nikhil-goenka Apr 6, 2024
a9d38a8
r/aws_glue_job: update test and documentation
nikhil-goenka Apr 6, 2024
495798d
Merge branch 'main' into r/aws_glue_job
jar-b Apr 16, 2024
811aef3
r/aws_glue_job: number_of_workers and worker_type are optional/computed
jar-b Apr 16, 2024
5b4d557
chore: changelog
jar-b Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .changelog/36770.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:bug
resource/aws_glue_job: Mark `number_of_workers` and `worker_type` as optional/computed, preventing persistent differences when `max_capacity` is set.
```
2 changes: 2 additions & 0 deletions internal/service/glue/job.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ func ResourceJob() *schema.Resource {
"number_of_workers": {
Type: schema.TypeInt,
Optional: true,
Computed: true,
ConflictsWith: []string{"max_capacity"},
ValidateFunc: validation.IntAtLeast(1),
},
Expand All @@ -176,6 +177,7 @@ func ResourceJob() *schema.Resource {
"worker_type": {
Type: schema.TypeString,
Optional: true,
Computed: true,
ConflictsWith: []string{"max_capacity"},
ValidateFunc: validation.StringInSlice(glue.WorkerType_Values(), false),
},
Expand Down
14 changes: 14 additions & 0 deletions internal/service/glue/job_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -652,6 +652,20 @@ func TestAccGlueJob_workerType(t *testing.T) {
resource.TestCheckResourceAttr(resourceName, "worker_type", "G.2X"),
),
},
{
Config: testAccJobConfig_workerType(rName, "G.4X"),
Check: resource.ComposeTestCheckFunc(
testAccCheckJobExists(ctx, resourceName, &job),
resource.TestCheckResourceAttr(resourceName, "worker_type", "G.4X"),
),
},
{
Config: testAccJobConfig_workerType(rName, "G.8X"),
Check: resource.ComposeTestCheckFunc(
testAccCheckJobExists(ctx, resourceName, &job),
resource.TestCheckResourceAttr(resourceName, "worker_type", "G.8X"),
),
},
{
ResourceName: resourceName,
ImportState: true,
Expand Down
2 changes: 2 additions & 0 deletions website/docs/r/glue_job.html.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ This resource supports the following arguments:
* For the Standard worker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker.
* For the G.1X worker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. Recommended for memory-intensive jobs.
* For the G.2X worker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. Recommended for memory-intensive jobs.
* For the G.4X worker type, each worker maps to 4 DPU (16 vCPUs, 64 GB of memory) with 256GB disk (approximately 235GB free), and provides 1 executor per worker. Recommended for memory-intensive jobs. Only available for Glue version 3.0. Available AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm).
* For the G.8X worker type, each worker maps to 8 DPU (32 vCPUs, 128 GB of memory) with 512GB disk (approximately 487GB free), and provides 1 executor per worker. Recommended for memory-intensive jobs. Only available for Glue version 3.0. Available AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm).
* For the G.025X worker type, each worker maps to 0.25 DPU (2 vCPU, 4GB of memory, 64 GB disk), and provides 1 executor per worker. Recommended for low volume streaming jobs. Only available for Glue version 3.0.
* For the Z.2X worker type, each worker maps to 2 M-DPU (8vCPU, 64 GB of m emory, 128 GB disk), and provides up to 8 Ray workers based on the autoscaler.
* `number_of_workers` - (Optional) The number of workers of a defined workerType that are allocated when a job runs.
Expand Down
Loading