Starting oCIS can fail with nats connection error

[nirajacharya2]

Describe the bug

While starting ocis server, some services fail with Failed to connect to NATS Server: nats: no servers available for connection error and ocis exists. ociswrapper then tries to restart the ocis but the search service could not start properly (error: unexpected end of JSON input)

+ /drone/src/tests/ociswrapper/bin/ociswrapper serve --bin ocis/bin/ocis --url https://ocis-server:9200
2023/11/30 00:57:32 [ociswrapper] Starting ociswrapper on port 5200...
{"level":"info","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"Migrating spaces directory structure..."}
{"level":"info","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"Migrating space types indexes..."}
{"level":"info","root":"/root/.ocis/storage/users","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"Migrating to messagepack metadata backend..."}
{"level":"info","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"done."}
{"level":"warn","error":"open /root/.ocis/storage/users/indexes/by-user-id: no such file or directory","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"error listing user indexes"}
{"level":"warn","error":"open /root/.ocis/storage/users/indexes/by-group-id: no such file or directory","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"error listing group indexes"}
{"level":"info","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"done."}
{"level":"info","time":"2023-11-30T00:57:34Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"done."}
{"level":"error","service":"invitations","error":"Failed to connect to NATS Server: nats: no servers available for connection","transport":"http","time":"2023-11-30T00:57:34Z","message":"Shutting down server"}
2023/11/30 00:57:34 [ociswrapper] oCIS server exited with code 1
2023/11/30 00:57:34 [ociswrapper] Retry starting oCIS server... (retry 1)
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:36Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:36Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:36Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:36Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:36Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:36Z","message":"Error initializing search service"}
{"level":"info","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"Migrating spaces directory structure..."}
{"level":"info","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"Migrating space types indexes..."}
{"level":"info","root":"/root/.ocis/storage/metadata","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"Migrating to messagepack metadata backend..."}
{"level":"info","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"done."}
{"level":"warn","error":"open /root/.ocis/storage/metadata/indexes/by-user-id: no such file or directory","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"error listing user indexes"}
{"level":"warn","error":"open /root/.ocis/storage/metadata/indexes/by-group-id: no such file or directory","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"error listing group indexes"}
{"level":"info","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"done."}
{"level":"info","time":"2023-11-30T00:57:36Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:170","message":"done."}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:39Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:40Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:40Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:40Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:40Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:40Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2023-11-30T00:57:43Z","message":"Error initializing search service"}

Builds:
https://drone.owncloud.com/owncloud/ocis/29404/37/6
https://drone.owncloud.com/owncloud/ocis/29423/58/9
https://drone.owncloud.com/owncloud/ocis/29427/36/3
https://drone.owncloud.com/owncloud/ocis/29460/31/3

[saw-jan]

ociswrapper is failing to restart. if it fails it should try to restart it self 5 times but dose not.

looks like there are some error logs but ocis didn't exit so the wrapper won't restart.

[saw-jan]

Builds:

• https://drone.owncloud.com/owncloud/ocis/29512/25/8
• https://drone.owncloud.com/owncloud/ocis/30023/8/3

[saw-jan]

Similar error: "error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n"
Build: https://drone.owncloud.com/owncloud/ocis/31448/11/5

{"level":"error","service":"app-provider","error":"unable to register services: rgrpc: grpc service appprovider could not be started,: Get \"http://fakeoffice:8080/hosting/discovery\": readLoopPeekFailLocked: %!w(<nil>)","time":"2024-01-31T01:22:07Z","message":"error starting the grpc server"}
2024/01/31 01:22:07 [ociswrapper] oCIS server exited with code 1
2024/01/31 01:22:07 [ociswrapper] Retry starting oCIS server... (retry 1)
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"Migrating spaces directory structure..."}
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"Migrating space types indexes..."}
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"Migrating spaces directory structure..."}
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"Migrating space types indexes..."}
{"level":"info","root":"/root/.ocis/storage/metadata","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"Migrating to messagepack metadata backend..."}
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"done."}
{"level":"warn","error":"open /root/.ocis/storage/metadata/indexes/by-user-id: no such file or directory","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"error listing user indexes"}
{"level":"warn","error":"open /root/.ocis/storage/metadata/indexes/by-group-id: no such file or directory","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"error listing group indexes"}
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"done."}
{"level":"info","root":"/root/.ocis/storage/users","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"Migrating to messagepack metadata backend..."}
{"level":"info","time":"2024-01-31T01:22:14Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go:195","message":"done."}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2024-01-31T01:22:14Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2024-01-31T01:22:14Z","message":"Error initializing search service"}
{"level":"error","service":"search","error":"error parsing mapping JSON: unexpected end of JSON input\nmapping contents:\n","time":"2024-01-31T01:22:14Z","message":"Error initializing search service"}

[saw-jan]

Haven't seen similar failures for months. There were a couple of nats fixes.
So closing for now