fix: race condition with parallel + enable_sharing #2024

snakster · 2024-12-30T16:43:04Z

What this PR does / why we need it:

This PR fixes a race condition when using the --parallel flag in combination with outputs sharing.

Trace:

fatal error: concurrent map writes

goroutine 58 [running]:
github.com/terramate-io/terramate/cmd/terramate/cli.(*cli).runAll.func7({0x140084ac9a0, {0x14008970480, 0x1, 0x1}, 0xffffffffffffffff})
github.com/terramate-io/terramate/cmd/terramate/cli/run.go:481 +0x19b8
github.com/terramate-io/terramate/scheduler.(*Parallel[...]).visitNode.func1()
github.com/terramate-io/terramate/scheduler/parallel.go:87 +0xb0
github.com/terramate-io/terramate/scheduler.(*Parallel[...]).forkTask.func1()
github.com/terramate-io/terramate/scheduler/parallel.go:111 +0x58
created by github.com/terramate-io/terramate/scheduler.(*Parallel[...]).forkTask in goroutine 56
github.com/terramate-io/terramate/scheduler/parallel.go:109 +0x94

Additional chore commits:

Bump dev version
Upgrade setup-opentofu action because the previous version can no longer install opentofu 1.6.x, see No matching version found when used version 1.6.0 opentofu/setup-opentofu#43

Special notes for your reviewer:

Does this PR introduce a user-facing change?

no

github-actions · 2024-12-30T16:45:09Z

2025/01/02 13:33:47 Terraform detected version: 1.10.2
toolsetPath: /tmp/cmd-terramate-test-2215080536
=== RUN   TestInteropCloudSyncPreview
=== RUN   TestInteropCloudSyncPreview/preview:_basic-drift
    interoperability_test.go:46: using GITHUB_EVENT_FILE=/home/runner/work/_temp/_github_workflow/event.json
=== RUN   TestInteropCloudSyncPreview/preview:_basic-drift-uppercase-id
    interoperability_test.go:46: using GITHUB_EVENT_FILE=/home/runner/work/_temp/_github_workflow/event.json
--- PASS: TestInteropCloudSyncPreview (13.86s)
    --- PASS: TestInteropCloudSyncPreview/preview:_basic-drift (7.33s)
    --- PASS: TestInteropCloudSyncPreview/preview:_basic-drift-uppercase-id (6.53s)
=== RUN   TestInteropSyncDeployment
=== RUN   TestInteropSyncDeployment/deployment:_empty
=== RUN   TestInteropSyncDeployment/deployment:_empty-uppercase-id
--- PASS: TestInteropSyncDeployment (46.31s)
    --- PASS: TestInteropSyncDeployment/deployment:_empty (23.49s)
    --- PASS: TestInteropSyncDeployment/deployment:_empty-uppercase-id (22.82s)
=== RUN   TestInteropDrift
=== RUN   TestInteropDrift/drift:_basic-drift
=== RUN   TestInteropDrift/drift:_basic-drift-uppercase-id
--- PASS: TestInteropDrift (74.50s)
    --- PASS: TestInteropDrift/drift:_basic-drift (37.65s)
    --- PASS: TestInteropDrift/drift:_basic-drift-uppercase-id (36.86s)
PASS
ok  	github.com/terramate-io/terramate/e2etests/cloud/interop	144.618s

github-actions · 2024-12-30T17:00:23Z

metric: time/op
ChangeDetection-4: old 4.18ms ± 6%: new 4.07ms ± 4%: delta: -2.54%
ChangeDetectionTFAndTG-4: old 13.8ms ± 2%: new 13.6ms ± 4%: delta: -1.52%
CloudReadLines-4: old 1.13ms ± 4%: new 1.01ms ± 2%: delta: -11.11%
CloudReadLine-4: old 7.52ms ± 1%: new 7.32ms ± 2%: delta: -2.63%
ListFiles-4: old 88.6µs ± 1%: new 88.3µs ± 1%: delta: 0.00%
Generate-4: old 1.60s ± 2%: new 1.56s ± 3%: delta: -2.53%
GenerateRegex-4: old 1.10s ± 3%: new 1.07s ± 3%: delta: -2.71%
TokensForExpressionComplex-4: old 1.28ms ± 1%: new 1.27ms ± 1%: delta: -0.97%
TokensForExpressionPlainStringNoNewline-4: old 805ns ± 1%: new 798ns ± 1%: delta: -0.92%
TokensForExpressionStringWith100Newlines-4: old 24.5µs ± 1%: new 24.2µs ± 1%: delta: -1.16%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 1.46ms ± 0%: new 1.46ms ± 1%: delta: 0.00%
TokensForExpression-4: old 1.27ms ± 0%: new 1.27ms ± 1%: delta: 0.00%
PartialEvalComplex-4: old 548µs ± 2%: new 549µs ± 3%: delta: 0.00%
PartialEvalSmallString-4: old 3.95µs ± 1%: new 3.99µs ± 2%: delta: 1.14%
PartialEvalHugeString-4: old 1.92ms ± 0%: new 1.92ms ± 0%: delta: -0.40%
PartialEvalHugeInterpolatedString-4: old 5.04ms ± 2%: new 5.05ms ± 2%: delta: 0.00%
PartialEvalObject-4: old 27.5µs ± 1%: new 27.7µs ± 2%: delta: 0.78%
TmAllTrueLiteralList-4: old 625µs ± 1%: new 627µs ± 1%: delta: 0.00%
TmAllTrueFuncall-4: old 20.7µs ± 1%: new 20.8µs ± 1%: delta: 0.71%
TmAnyTrueLiteralList-4: old 4.93ms ± 1%: new 4.97ms ± 1%: delta: 0.77%
TmAnyTrueFuncall-4: old 21.0µs ± 1%: new 21.2µs ± 1%: delta: 0.66%
TmTernary-4: old 2.50µs ± 0%: new 2.54µs ± 1%: delta: 1.50%
TmTryUnknownFunc-4: old 2.28µs ± 1%: new 2.30µs ± 1%: delta: 1.08%
TmTryUnknownVariable-4: old 2.18µs ± 1%: new 2.22µs ± 1%: delta: 1.94%
TmTryUnknownObjectKey-4: old 2.50µs ± 1%: new 2.53µs ± 1%: delta: 1.18%
metric: alloc/op
ChangeDetection-4: old 357kB ± 0%: new 357kB ± 0%: delta: 0.00%
ChangeDetectionTFAndTG-4: old 4.81MB ± 0%: new 4.81MB ± 0%: delta: 0.00%
CloudReadLines-4: old 3.12MB ± 0%: new 3.12MB ± 0%: delta: 0.00%
CloudReadLine-4: old 3.37MB ± 0%: new 3.37MB ± 0%: delta: 0.00%
ListFiles-4: old 27.6kB ± 0%: new 27.6kB ± 0%: delta: 0.00%
Generate-4: old 2.24GB ± 0%: new 2.24GB ± 0%: delta: 0.00%
GenerateRegex-4: old 926MB ± 0%: new 926MB ± 0%: delta: 0.00%
TokensForExpressionComplex-4: old 395kB ± 0%: new 394kB ± 0%: delta: -0.00%
TokensForExpressionPlainStringNoNewline-4: old 512B ± 0%: new 512B ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-4: old 12.5kB ± 0%: new 12.5kB ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 395kB ± 0%: new 395kB ± 0%: delta: 0.00%
TokensForExpression-4: old 395kB ± 0%: new 394kB ± 0%: delta: 0.00%
PartialEvalComplex-4: old 361kB ± 0%: new 361kB ± 0%: delta: 0.00%
PartialEvalSmallString-4: old 1.95kB ± 0%: new 1.95kB ± 0%: delta: 0.00%
PartialEvalHugeString-4: old 196kB ± 0%: new 196kB ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 4.31MB ± 0%: new 4.31MB ± 0%: delta: 0.00%
PartialEvalObject-4: old 19.4kB ± 0%: new 19.4kB ± 0%: delta: 0.00%
TmAllTrueLiteralList-4: old 321kB ± 0%: new 321kB ± 0%: delta: 0.00%
TmAllTrueFuncall-4: old 10.6kB ± 0%: new 10.6kB ± 0%: delta: 0.00%
TmAnyTrueLiteralList-4: old 2.09MB ± 0%: new 2.09MB ± 0%: delta: 0.00%
TmAnyTrueFuncall-4: old 10.7kB ± 0%: new 10.7kB ± 0%: delta: 0.00%
TmTernary-4: old 1.18kB ± 0%: new 1.18kB ± 0%: delta: 0.00%
TmTryUnknownFunc-4: old 784B ± 0%: new 784B ± 0%: delta: 0.00%
TmTryUnknownVariable-4: old 768B ± 0%: new 768B ± 0%: delta: 0.00%
TmTryUnknownObjectKey-4: old 952B ± 0%: new 952B ± 0%: delta: 0.00%
metric: allocs/op
ChangeDetection-4: old 2.48k ± 0%: new 2.48k ± 0%: delta: 0.00%
ChangeDetectionTFAndTG-4: old 33.8k ± 0%: new 33.8k ± 0%: delta: 0.00%
CloudReadLines-4: old 5.54k ± 0%: new 5.54k ± 0%: delta: 0.00%
CloudReadLine-4: old 60.0k ± 0%: new 60.0k ± 0%: delta: 0.00%
ListFiles-4: old 335 ± 0%: new 335 ± 0%: delta: 0.00%
Generate-4: old 25.7M ± 0%: new 25.7M ± 0%: delta: 0.00%
GenerateRegex-4: old 18.4M ± 0%: new 18.4M ± 0%: delta: 0.00%
TokensForExpressionComplex-4: old 4.83k ± 0%: new 4.83k ± 0%: delta: 0.00%
TokensForExpressionPlainStringNoNewline-4: old 20.0 ± 0%: new 20.0 ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-4: old 227 ± 0%: new 227 ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-4: old 3.19k ± 0%: new 3.19k ± 0%: delta: 0.00%
TokensForExpression-4: old 4.83k ± 0%: new 4.83k ± 0%: delta: 0.00%
PartialEvalComplex-4: old 3.78k ± 0%: new 3.78k ± 0%: delta: 0.00%
PartialEvalSmallString-4: old 26.0 ± 0%: new 26.0 ± 0%: delta: 0.00%
PartialEvalHugeString-4: old 38.0 ± 0%: new 38.0 ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-4: old 26.1k ± 0%: new 26.1k ± 0%: delta: 0.00%
PartialEvalObject-4: old 183 ± 0%: new 183 ± 0%: delta: 0.00%
TmAllTrueLiteralList-4: old 5.94k ± 0%: new 5.94k ± 0%: delta: 0.00%
TmAllTrueFuncall-4: old 275 ± 0%: new 275 ± 0%: delta: 0.00%
TmAnyTrueLiteralList-4: old 59.6k ± 0%: new 59.6k ± 0%: delta: 0.00%
TmAnyTrueFuncall-4: old 277 ± 0%: new 277 ± 0%: delta: 0.00%
TmTernary-4: old 27.0 ± 0%: new 27.0 ± 0%: delta: 0.00%
TmTryUnknownFunc-4: old 21.0 ± 0%: new 21.0 ± 0%: delta: 0.00%
TmTryUnknownVariable-4: old 20.0 ± 0%: new 20.0 ± 0%: delta: 0.00%
TmTryUnknownObjectKey-4: old 23.0 ± 0%: new 23.0 ± 0%: delta: 0.00%

github-actions · 2025-01-02T10:27:18Z

Preview of ubuntu-focal/go1.21 tests in `ae19148`

🔍 View Details on Terramate Cloud

.
cmd/terramate/cli
run
e2etests/core

github-actions · 2025-01-02T10:32:55Z

Preview of macos-ventura/go1.21 tests in `ae19148`

🔍 View Details on Terramate Cloud

.
cmd/terramate/cli
run
e2etests/core

sixstone-qq

Great work, just a few comments :)

e2etests/core/run_sharing_test.go

run/oncemap.go

snakster force-pushed the snk-fix-sharing-panic branch from 8bb52e6 to bd54112 Compare December 30, 2024 16:52

snakster force-pushed the snk-fix-sharing-panic branch from bd54112 to a42042a Compare January 2, 2025 10:04

snakster marked this pull request as ready for review January 2, 2025 11:07

snakster requested a review from a team as a code owner January 2, 2025 11:07

sixstone-qq reviewed Jan 2, 2025

View reviewed changes

e2etests/core/run_sharing_test.go Show resolved Hide resolved

run/oncemap.go Outdated Show resolved Hide resolved

run/oncemap.go Show resolved Hide resolved

run/oncemap.go Outdated Show resolved Hide resolved

snakster added 3 commits January 2, 2025 14:29

fix: race condition with parallel + enable_sharing

9e11aeb

chore: bump dev version

d5c08ea

chore: upgrade to opentofu/setup-opentofu v1.0.5

ae19148

snakster force-pushed the snk-fix-sharing-panic branch from 468a0e7 to ae19148 Compare January 2, 2025 13:33

sixstone-qq approved these changes Jan 2, 2025

View reviewed changes

mariux approved these changes Jan 2, 2025

View reviewed changes

snakster merged commit d9056d7 into main Jan 2, 2025
7 checks passed

snakster deleted the snk-fix-sharing-panic branch January 2, 2025 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: race condition with parallel + enable_sharing #2024

fix: race condition with parallel + enable_sharing #2024

snakster commented Dec 30, 2024 •

edited

Loading

github-actions bot commented Dec 30, 2024 •

edited

Loading

github-actions bot commented Dec 30, 2024 •

edited

Loading

github-actions bot commented Jan 2, 2025 •

edited

Loading

github-actions bot commented Jan 2, 2025 •

edited

Loading

sixstone-qq left a comment

fix: race condition with parallel + enable_sharing #2024

fix: race condition with parallel + enable_sharing #2024

Conversation

snakster commented Dec 30, 2024 • edited Loading

What this PR does / why we need it:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

github-actions bot commented Dec 30, 2024 • edited Loading

github-actions bot commented Dec 30, 2024 • edited Loading

github-actions bot commented Jan 2, 2025 • edited Loading

Preview of ubuntu-focal/go1.21 tests in ae19148

github-actions bot commented Jan 2, 2025 • edited Loading

Preview of macos-ventura/go1.21 tests in ae19148

sixstone-qq left a comment

Choose a reason for hiding this comment

snakster commented Dec 30, 2024 •

edited

Loading

github-actions bot commented Dec 30, 2024 •

edited

Loading

github-actions bot commented Dec 30, 2024 •

edited

Loading

github-actions bot commented Jan 2, 2025 •

edited

Loading

Preview of ubuntu-focal/go1.21 tests in `ae19148`

github-actions bot commented Jan 2, 2025 •

edited

Loading

Preview of macos-ventura/go1.21 tests in `ae19148`