Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky CI due server panic #7518

Closed
ScharfViktor opened this issue Oct 17, 2023 · 9 comments
Closed

Flaky CI due server panic #7518

ScharfViktor opened this issue Oct 17, 2023 · 9 comments
Labels
Type:Bug

Comments

@ScharfViktor
Copy link
Contributor

ScharfViktor commented Oct 17, 2023 •
edited
Loading

related #7309

sometimes during the test run server crashes
see log here https://drone.owncloud.com/owncloud/ocis/27811/67/3

we ignore it and restart server using ocis wrapper. As result we have flaky test

example: https://drone.owncloud.com/owncloud/ocis/27811/67/5
Screenshot 2023-10-17 at 17 43 45

https://drone.owncloud.com/owncloud/ocis/27812/58/5
@saw-jan
Copy link
Member

saw-jan commented Oct 18, 2023 •
edited
Loading

ocis crash log:

crash 1 
2023/10/17 20:16:45 http: panic serving 127.0.0.1:37982: runtime error: invalid memory address or nil pointer dereference
goroutine 1764 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1854 +0xbf
panic({0x2cb5320, 0x5db5360})
	/usr/local/go/src/runtime/panic.go:890 +0x263
github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav.(*svc).sspReferenceIsChildOf(0xc000166690, {0x43e61d0, 0xc0077e7a10}, {0x43c3f60?, 0xc0007481e0?}, 0xc002b57a00, 0xc002b579c0)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav/ocdav.go:342 +0x113
github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav.(*svc).referenceIsChildOf(0x2?, {0x43e61d0, 0xc0077e7a10}, {0x43c3f60?, 0xc0007481e0?}, 0xc002b57a00, 0xc002b579c0)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav/ocdav.go:390 +0x250
github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav.(*svc).handleMove(0xc000166690, {0x43e61d0, 0xc0077e7a10}, {0x7f923e0fa538?, 0xc0077e7170?}, 0xc001b5cb00, 0xc002b579c0, 0xc002b57a00, {{0x43d8780, 0xc000631dd0}, ...})
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav/move.go:144 +0x8f
github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav.(*svc).handlePathMove(0xc000166690, {0x7f923e0fa538, 0xc0077e7170}, 0xc001b5cb00, {0xc005aa57a0, 0x2b})
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav/move.go:102 +0x8d3
github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav.(*WebDavHandler).Handler.func1({0x7f923e0fa538, 0xc0077e7170}, 0xc001b5cb00)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav/webdav.go:86 +0x299
net/http.HandlerFunc.ServeHTTP(0xc0005c6480?, {0x7f923e0fa538?, 0xc0077e7170?}, 0x43b6df0?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav.(*svc).Handler.func1({0x7f923e0fa538, 0xc0077e7170}, 0xc001b5ca00)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/services/owncloud/ocdav/ocdav.go:218 +0x907
net/http.HandlerFunc.ServeHTTP(0x2c82a00?, {0x7f923e0fa538?, 0xc0077e7170?}, 0xc002b99285?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc001c73140, {0x7f923e0fa538, 0xc0077e7170}, 0xc001b5ca00)
	/drone/src/vendor/github.com/go-chi/chi/v5/mux.go:444 +0x236
net/http.HandlerFunc.ServeHTTP(0xc0020ba300?, {0x7f923e0fa538?, 0xc0077e7170?}, 0xc001b5ca00?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/rs/cors.(*Cors).Handler.func1({0x7f923e0fa538, 0xc0077e7170}, 0xc001b5ca00)
	/drone/src/vendor/github.com/rs/cors/cors.go:236 +0x1c4
net/http.HandlerFunc.ServeHTTP(0x43e61d0?, {0x7f923e0fa538?, 0xc0077e7170?}, 0x60880b8?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/cs3org/reva/v2/internal/http/interceptors/appctx.handler.func1({0x7f923e0fa538, 0xc0077e7170}, 0xc001b5c900)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/interceptors/appctx/appctx.go:58 +0x706
net/http.HandlerFunc.ServeHTTP(0x43e61d0?, {0x7f923e0fa538?, 0xc0077e7170?}, 0x43b6df0?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/go-chi/chi/v5/middleware.RequestID.func1({0x7f923e0fa538, 0xc0077e7170}, 0xc001b5c800)
	/drone/src/vendor/github.com/go-chi/chi/v5/middleware/request_id.go:76 +0x22d
net/http.HandlerFunc.ServeHTTP(0xc0077e7620?, {0x7f923e0fa538?, 0xc0077e7170?}, 0x32f5c00?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/cs3org/reva/v2/internal/http/interceptors/auth.New.func2.1({0x7f923e0fa538, 0xc0077e7170}, 0xc001b5c700)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/interceptors/auth/auth.go:212 +0x7e2
net/http.HandlerFunc.ServeHTTP(0x43e3380?, {0x7f923e0fa538?, 0xc0077e7170?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/cs3org/reva/v2/internal/http/interceptors/log.loggingHandler.ServeHTTP({{0x43cf400?, 0xc0005e02d0?}}, {0x43e3380, 0xc00698c460}, 0xc001b5c700)
	/drone/src/vendor/github.com/cs3org/reva/v2/internal/http/interceptors/log/log.go:57 +0x151
github.com/cs3org/reva/v2/pkg/micro/ocdav.traceHandler.func1.1({0x43e3380, 0xc00698c460}, 0xc001b5c600)
	/drone/src/vendor/github.com/cs3org/reva/v2/pkg/micro/ocdav/service.go:242 +0x296
net/http.HandlerFunc.ServeHTTP(0x7f923e252e50?, {0x43e3380?, 0xc00698c460?}, 0xc000600400?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/cs3org/reva/v2/pkg/micro/ocdav.useMiddlewares.func3.1({0x43e3380?, 0xc00698c460?}, 0x5db3301?)
	/drone/src/vendor/github.com/cs3org/reva/v2/pkg/micro/ocdav/service.go:216 +0x4a
net/http.HandlerFunc.ServeHTTP(0x43e6128?, {0x43e3380?, 0xc00698c460?}, 0x5db33e0?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc001c73140, {0x43e3380, 0xc00698c460}, 0xc001b5c500)
	/drone/src/vendor/github.com/go-chi/chi/v5/mux.go:90 +0x310
net/http.serverHandler.ServeHTTP({0xc006a53740?}, {0x43e3380, 0xc00698c460}, 0xc001b5c500)
	/usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc002626bd0, {0x43e61d0, 0xc0017f8720})
	/usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x5ed
crash 2 
fatal error: concurrent map iteration and map write

goroutine 80969 [running]:
github.com/cs3org/reva/v2/pkg/share/manager/jsoncs3.(*Manager).ListReceivedShares.func2()
	/drone/src/vendor/github.com/cs3org/reva/v2/pkg/share/manager/jsoncs3/jsoncs3.go:865 +0x247
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/drone/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/drone/src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 1 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7fd46dff48d8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000731880?, 0xc001839448?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000731880)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000731880)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc001445290)
	/usr/local/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc001445290)
	/usr/local/go/src/net/tcpsock.go:297 +0x3d
net/http.(*Server).Serve(0xc0002574a0, {0x43d75b0, 0xc001445290})
	/usr/local/go/src/net/http/server.go:3059 +0x385
net/http.Serve(...)
	/usr/local/go/src/net/http/server.go:2581
github.com/owncloud/ocis/v2/ocis/pkg/runtime/service.Start({0xc00109d8c0?, 0x18?, 0x603c0c0?})
	/drone/src/ocis/pkg/runtime/service/service.go:411 +0x8e5
github.com/owncloud/ocis/v2/ocis/pkg/runtime.(*Runtime).Start(0x2bb5560?)
	/drone/src/ocis/pkg/runtime/runtime.go:22 +0x53
github.com/owncloud/ocis/v2/ocis/pkg/command.Server.func2(0xc0010cdce0?)
	/drone/src/ocis/pkg/command/server.go:26 +0x69
github.com/urfave/cli/v2.(*Command).Run(0xc0010cdce0, 0xc001454000, {0xc00061da30, 0x1, 0x1})
	/drone/src/vendor/github.com/urfave/cli/v2/command.go:274 +0x9eb
github.com/urfave/cli/v2.(*Command).Run(0xc001452580, 0xc000d2df40, {0xc000052040, 0x2, 0x2})
	/drone/src/vendor/github.com/urfave/cli/v2/command.go:267 +0xc4d
github.com/urfave/cli/v2.(*App).RunContext(0xc0001fef00, {0x43da5e0?, 0xc0000620b0}, {0xc000052040, 0x2, 0x2})
	/drone/src/vendor/github.com/urfave/cli/v2/app.go:332 +0x616
github.com/urfave/cli/v2.(*App).Run(...)
	/drone/src/vendor/github.com/urfave/cli/v2/app.go:309
github.com/owncloud/ocis/v2/ocis/pkg/command.Execute()
	/drone/src/ocis/pkg/command/root.go:28 +0x17a
main.main()
	/drone/src/ocis/cmd/ocis/main.go:11 +0x19

goroutine 10 [select]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc0001b29f0, 0x0?)
	/drone/src/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0x85
created by github.com/patrickmn/go-cache.runJanitor
	/drone/src/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xed
...

@saw-jan
Copy link
Member

saw-jan commented Oct 18, 2023 •
edited
Loading

Seen first in:

After it got merged:

The server related changes in PR #7477 were:

  • FRONTEND_PASSWORD_POLICY_BANNED_PASSWORDS_LIST=banned-password.txt (in .drone.star)
  • AutoAcceptShares: true (in services/frontend/pkg/config/defaults/defaultconfig.go)

@ScharfViktor
Copy link
Contributor Author

ScharfViktor commented Oct 18, 2023 •
edited
Loading

crash 1

panic in CI: Core-API-Tests-ocis-storage-3 -> ocis-server https://drone.owncloud.com/owncloud/ocis/27845/67/4
I could reproduce it localy:

  • run test make test-acceptance-from-core-api TEST_SERVER_URL=https://localhost:9200 TEST_WITH_GRAPH_API=true TEST_OCIS=true BEHAT_FEATURE=tests/acceptance/features/coreApiShareOperationsToShares1/changingFilesShare.feature
  • see server log

Actual: failed step: When user "Brian" moves file "textfile0.txt" to "/Shares/shared/shared_file.txt" using the WebDAV API 502 error

@ScharfViktor
Copy link
Contributor Author

crash 2
fixed here #7526

@kobergj
Copy link
Collaborator

kobergj commented Oct 26, 2023

Does that mean all crashes have been fixed now? Or is there still a panic?

@saw-jan
Copy link
Member

saw-jan commented Oct 26, 2023

PR cs3org/reva#4283 was merged in reva which I think was for crash 1. so, if the reva is bumped in ocis after that PR then I guess all crashes are fixed now.

@ScharfViktor
Copy link
Contributor Author

ScharfViktor commented Oct 26, 2023 •
edited
Loading

cs3org/reva#4283

this PR #7581 bumped reva I guess?

but panic still exist Core-API-Tests-ocis-storage-3 -> ocis-server
https://drone.owncloud.com/owncloud/ocis/28158/28/3

@kobergj
Copy link
Collaborator

kobergj commented Oct 26, 2023

It is another panic to be precise 😄 cs3org/reva#4290

@ScharfViktor
Copy link
Contributor Author

Checked after #7598 ocis bump. No panic 😄
@kobergj thank you 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type:Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kobergj @saw-jan @ScharfViktor