Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e2e flake: fatal error: concurrent map read and map write #1571

Closed
nirs opened this issue Sep 23, 2024 · 0 comments · Fixed by #1591
Closed

e2e flake: fatal error: concurrent map read and map write #1571

nirs opened this issue Sep 23, 2024 · 0 comments · Fixed by #1591
Assignees
@nirs
Labels
bug Something isn't working test Testing related issue

Comments

@nirs
Copy link
Member

nirs commented Sep 23, 2024

Looks like a random failure - maybe we are missing locking on a map?

2024-09-23T18:24:15.804+0530	INFO	util/config.go:48	No configuration file specified, using default value config.yaml
2024-09-23T18:24:15.816+0530	INFO	e2e/main_test.go:41	Global setting	{"Timeout": 600}
2024-09-23T18:24:15.816+0530	INFO	e2e/main_test.go:42	Global setting	{"Retry Interval": 30}
=== RUN   TestSuites
2024-09-23T18:24:15.817+0530	INFO	e2e/main_test.go:57	TestSuites
=== RUN   TestSuites/Validate
=== RUN   TestSuites/Validate/CheckRamenHubOperatorStatus
2024-09-23T18:24:15.817+0530	INFO	e2e/validation_suite_test.go:25	enter CheckRamenHubOperatorStatus
2024-09-23T18:24:15.860+0530	INFO	e2e/validation_suite_test.go:33	Ramen Hub Operator is running	{"pod": "ramen-hub-operator-84d7dc89bd-dv7xh"}
=== RUN   TestSuites/Validate/CheckRamenSpokeOperatorStatus
2024-09-23T18:24:15.860+0530	INFO	e2e/validation_suite_test.go:40	enter CheckRamenSpokeOperatorStatus
2024-09-23T18:24:15.886+0530	INFO	e2e/validation_suite_test.go:48	Ramen Spoke Operator is running on cluster 1	{"pod": "ramen-dr-cluster-operator-896d8c9f6-bj742"}
2024-09-23T18:24:15.911+0530	INFO	e2e/validation_suite_test.go:59	Ramen Spoke Operator is running on cluster 2	{"pod": "ramen-dr-cluster-operator-896d8c9f6-6jp9r"}
=== RUN   TestSuites/Exhaustive
=== PAUSE TestSuites/Exhaustive
=== CONT  TestSuites/Exhaustive
2024-09-23T18:24:15.970+0530	INFO	util/crud.go:77	channel ramen-gitops is created
=== RUN   TestSuites/Exhaustive/Deploy-cephfs
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs
=== RUN   TestSuites/Exhaustive/Deploy-cephfs#01
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs#01
=== RUN   TestSuites/Exhaustive/Deploy-cephfs#02
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs#02
=== RUN   TestSuites/Exhaustive/Deploy-rbd
=== PAUSE TestSuites/Exhaustive/Deploy-rbd
=== RUN   TestSuites/Exhaustive/Deploy-rbd#01
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#01
=== RUN   TestSuites/Exhaustive/Deploy-rbd#02
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#02
=== CONT  TestSuites/Exhaustive/Deploy-cephfs
=== RUN   TestSuites/Exhaustive/Deploy-cephfs/Subscr
=== CONT  TestSuites/Exhaustive/Deploy-rbd#01
=== RUN   TestSuites/Exhaustive/Deploy-rbd#01/Appset
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#01/Appset
=== CONT  TestSuites/Exhaustive/Deploy-rbd#01/Appset
=== CONT  TestSuites/Exhaustive/Deploy-rbd
=== RUN   TestSuites/Exhaustive/Deploy-rbd#01/Appset/Deploy
=== RUN   TestSuites/Exhaustive/Deploy-rbd/Subscr
=== CONT  TestSuites/Exhaustive/Deploy-rbd#02
=== PAUSE TestSuites/Exhaustive/Deploy-rbd/Subscr
2024-09-23T18:24:15.971+0530	INFO	deployers/applicationset.go:17	enter Deploy appset-deploy-rbd-busybox
=== CONT  TestSuites/Exhaustive/Deploy-cephfs#01
=== CONT  TestSuites/Exhaustive/Deploy-cephfs#02
=== RUN   TestSuites/Exhaustive/Deploy-cephfs#02/Disapp
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs#02/Disapp
=== RUN   TestSuites/Exhaustive/Deploy-rbd#02/Disapp
=== CONT  TestSuites/Exhaustive/Deploy-rbd/Subscr
=== CONT  TestSuites/Exhaustive/Deploy-cephfs#02/Disapp
fatal error: concurrent map read and map write
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#02/Disapp

goroutine 25 [running]:
github.com/ramendr/ramen/e2e/testcontext.GetTestContext({0xc0004b4240, 0x27})
	/home/github/actions-runner/_work/ramen/ramen/e2e/testcontext/testcontext.go:36 +0x38
github.com/ramendr/ramen/e2e_test.runTestFlow(0xc00069e000)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:104 +0x45
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc00069e000)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:93 +0xcc
testing.tRunner(0xc00069e000, 0xc000234000)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1[14](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:15)
	/usr/local/go/src/testing/testing.go:1743 +0x390

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00040b380, {0x187514f?, 0x0?}, 0x1969ce0)
	/usr/local/go/src/testing/testing.go:1751 +0x3ab
testing.runTests.func1(0xc00040b380)
	/usr/local/go/src/testing/testing.go:2168 +0x37
testing.tRunner(0xc00040b380, 0xc00005db30)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
testing.runTests(0xc00049a330, {0x26dd020, 0x1, 0x1}, {0xe?, 0x1?, 0x0?})
	/usr/local/go/src/testing/testing.go:2166 +0x43d
testing.(*M).Run(0xc0004bfea0)
	/usr/local/go/src/testing/testing.go:2034 +0x64a
github.com/ramendr/ramen/e2e_test.TestMain(0xc0004bfea0)
	/home/github/actions-runner/_work/ramen/ramen/e2e/main_test.go:44 +0x2c5
main.main()
	_testmain.go:47 +0xa8

goroutine 51 [chan receive]:
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1651 +0x49b
testing.tRunner(0xc00040b6c0, 0x1969ce0)
	/usr/local/go/src/testing/testing.go:1696 +0x12d
created by testing.(*T).Run in goroutine 1
	/usr/local/go/src/testing/testing.go:1743 +0x390

	/usr/local/go/src/testing/testing.go:562 +0xa5
testing.(*T).Parallel(0xc000626820)
	/usr/local/go/src/testing/testing.go:1479 +0x1a5
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc000626820)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:91 +0x45
testing.tRunner(0xc000626820, 0xc000493cb0)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 79
	/usr/local/go/src/testing/testing.go:1743 +0x390

goroutine 26 [runnable]:
testing.(*common).frameSkip(0xc00069e340, 0x4)
	/usr/local/go/src/testing/testing.go:755 +0x593
testing.(*common).decorate(0x50?, {0xc0006a6050, 0x42}, 0xc00050[15](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:16)08?)
	/usr/local/go/src/testing/testing.go:771 +0x35
testing.(*common).logDepth(0xc00069e340, {0xc0006a6050, 0x42}, 0x3)
	/usr/local/go/src/testing/testing.go:1037 +0x14a
testing.(*common).log(...)
	/usr/local/go/src/testing/testing.go:1011
testing.(*common).Skipf(0xc00069e340, {0x18b3fdb?, 0xc000413200?}, {0xc000629ef0?, 0x[16](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:17)aeb00?, 0x473db0?})
	/usr/local/go/src/testing/testing.go:1103 +0x54
github.com/ramendr/ramen/e2e_test.runTestFlow(0xc00069e340)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:110 +0x16e
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc00069e340)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:93 +0xcc
testing.tRunner(0xc00069e340, 0xc000234b40)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 81
	/usr/local/go/src/testing/testing.go:[17](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:18)43 +0x390

goroutine 27 [runnable]:
sync.(*Map).LoadAndDelete(0x26ff9a0, {0x15d0ca0, 0xc00062fed8})
	/usr/local/go/src/sync/map.go:296 +0x[19](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:20)2
sync.(*Map).Delete(...)
	/usr/local/go/src/sync/map.go:321
testing.(*T).Parallel(0xc00070a000)
	/usr/local/go/src/testing/testing.go:1481 +0x1dd
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc00070a000)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:91 +0x45
testing.tRunner(0xc00070a000, 0xc0004408d0)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 116
	/usr/local/go/src/testing/testing.go:1743 +0x390
make: *** [Makefile:198: e2e-rdr] Error 1
exit status 2
FAIL	github.com/ramendr/ramen/e2e	0.[22](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:23)3s

Build: https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349
@nirs nirs added bug Something isn't working test Testing related issue labels Sep 23, 2024
nirs added a commit to nirs/ramen that referenced this issue Oct 21, 2024
@nirs
Fix testcontext map thread safety
dbe428e 
We modify the textcontext map form multiple tests running in parallel,
so we must synchronize access to the map.

Fixes RamenDR#1571

Signed-off-by: Nir Soffer <[email protected]>
@nirs nirs self-assigned this Oct 21, 2024
@nirs nirs mentioned this issue Oct 21, 2024
Merged
nirs added a commit to nirs/ramen that referenced this issue Oct 22, 2024
@nirs
Simplify test context management
fbc3ece 
We had 2 places using a test context - test suites, and action helpers.
To work with both, we store the test context in a map and had
complicated search mechanism to locate the context in both the parent
and child sub-tests. This failed randomly since the map was not
protected with a mutex.

Simplify the design by implementing the actions (deploy, enable, ..) in
the test context. With this we can create a test context instance and
pass it to the code running a test flow, and we don't need to manage any
global state.

Fixes: RamenDR#1571
Signed-off-by: Nir Soffer <[email protected]>
@nirs nirs mentioned this issue Oct 22, 2024
Merged
nirs added a commit to nirs/ramen that referenced this issue Oct 22, 2024
@nirs
Simplify test context management
9c5c307 
We had 2 places using a test context - test suites, and action helpers.
To work with both, we store the test context in a map and had
complicated search mechanism to locate the context in both the parent
and child sub-tests. This failed randomly since the map was not
protected with a mutex.

Simplify the design by implementing the actions (deploy, enable, ..) in
the test context. With this we can create a test context instance and
pass it to the code running a test flow, and we don't need to manage any
global state.

Fixes: RamenDR#1571
Signed-off-by: Nir Soffer <[email protected]>
@raghavendra-talur raghavendra-talur mentioned this issue Oct 25, 2024
Closed
nirs added a commit to nirs/ramen that referenced this issue Oct 30, 2024
@nirs
Simplify test context management
1664ba6 
We had 2 places using a test context - test suites, and action helpers.
To work with both, we store the test context in a map and had
complicated search mechanism to locate the context in both the parent
and child sub-tests. This failed randomly since the map was not
protected with a mutex.

Simplify the design by implementing the actions (deploy, enable, ..) in
the test context. With this we can create a test context instance and
pass it to the code running a test flow, and we don't need to manage any
global state.

Fixes: RamenDR#1571
Signed-off-by: Nir Soffer <[email protected]>
ShyamsundarR pushed a commit that referenced this issue Oct 31, 2024
@nirs @ShyamsundarR
Fix testcontext map thread safety
b4f58e9 
We modify the textcontext map form multiple tests running in parallel,
so we must synchronize access to the map.

Fixes #1571

Signed-off-by: Nir Soffer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nirs nirs

Labels
bug Something isn't working test Testing related issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

e2e: Fix thread unsafe map access nirs/ramen
1 participant
@nirs