Helm charts failed to deploy some pods from on-prem build OCIS image

[dejavus]

Description

This issue might be related to either building OCIS docker image or ocis helm charts.

I build my own OCIS docker image using v5.0.0-alpha.5 without changing any code and Dockerfile.
I pushed the image to my own registry.

I tried to deploy my own OCIS using ocis helm charts (master branch commit ef81b57) by just changing image location.
Some pods were failed to deploy.

So I cannot replay the entire build and deploy process from the scratch.

Step to reproduce

1. Build docker image using docker file (v5.0.0-alpha.5)
   docker build -t MY_ORG/REPO:MY_TAG .
2. Push docker image into a repository
3. Modify charts/ocis/values.yaml to indicate the repo where my own OCIS image

# Image for oCIS services
image:
  # -- Image repository
  repository: MY_ORG/REPO
  # -- Image tag. Defaults to the chart's appVersion.
  tag: "MY_TAG"
  # -- Image sha / digest (optional).
  sha: ""

4. helm install ocis charts/ocis -f charts/ocis/values.yaml

Expected behavior

All pods running

Actual behavior

Some pods are not started

❯ kubectl get pods                   
NAME                                 READY   STATUS             RESTARTS        AGE
appregistry-77b689747b-n74mt         1/1     Running            0               5m39s
audit-758b89857b-hqw2z               1/1     Running            3 (13s ago)     5m36s
authmachine-78586dbbcc-gxnqb         1/1     Running            0               5m38s
eventhistory-954b44895-j69cx         1/1     Running            3 (22s ago)     5m43s
frontend-d45b74ffb-vf5k5             1/1     Running            0               5m39s
gateway-8fb496868-28vqc              1/1     Running            0               5m39s
graph-7cb6bd9765-jsh26               1/1     Running            3 (18s ago)     5m39s
groups-9b7986f84-jvhp6               1/1     Running            0               5m43s
idm-d9cf79846-gkd8f                  0/1     CrashLoopBackOff   5 (2m28s ago)   5m36s
idp-7c85b8f85f-r5tj4                 0/1     CrashLoopBackOff   5 (2m30s ago)   5m42s
nats-669d4946dd-g2kzv                0/1     Error              6 (2m47s ago)   5m39s
ocdav-6f7c8b7d86-trbtl               1/1     Running            0               5m39s
ocs-6fb7c567bf-zdnhn                 1/1     Running            0               5m36s
postprocessing-868f4f4c64-d4qdt      1/1     Running            3 (13s ago)     5m36s
proxy-95c5c49fb-nr4lp                1/1     Running            0               5m37s
search-77486b474f-qh7zk              0/1     CrashLoopBackOff   5 (2m39s ago)   5m43s
settings-84ffd7c559-lf4tn            1/1     Running            0               5m40s
sharing-5fcffb6674-6p8kj             1/1     Running            3 (5m4s ago)    5m41s
storagepubliclink-65c647df66-vdzdx   1/1     Running            0               5m43s
storageshares-797bfc694d-hmfrp       1/1     Running            0               5m40s
storagesystem-5575658976-7nh98       0/1     CrashLoopBackOff   5 (2m9s ago)    5m39s
storageusers-856d6bbf7c-wgvgq        1/1     Running            3 (20s ago)     5m41s
store-6477fbd94f-2xspg               0/1     CrashLoopBackOff   5 (2m32s ago)   5m41s
thumbnails-7b8fd846b-sxhhk           1/1     Running            0               5m39s
userlog-d547dcc5d-cjkx2              1/1     Running            3 (19s ago)     5m40s
users-64bb7699d-xtk9g                1/1     Running            0               5m37s
web-54cf889545-qjw7w                 1/1     Running            0               5m38s
webdav-5f8864c55c-thccj              1/1     Running            0               5m38s
webfinger-c6db9bf98-wkzdk            1/1     Running            0               5m43s

Details

❯ kubectl logs idm-d9cf79846-gkd8f        
Defaulted container "idm" out of: idm, init-dir (init)
{"level":"info","service":"idm","time":"2023-12-07T13:58:04Z","line":"/ocis/ocis-pkg/crypto/gencert.go:38","message":"ldap.crt certificate / key pair already present. skipping acme certificate generation"}
{"level":"error","service":"idm","db":"/.ocis/idm/ocis.boltdb","error":"open /.ocis/idm/ocis.boltdb: no such file or directory","time":"2023-12-07T13:58:04Z","line":"/ocis/ocis-pkg/log/logrus_wrapper.go:50","message":"Error opening database"}
{"level":"error","service":"idm","error":"open /.ocis/idm/ocis.boltdb: no such file or directory","time":"2023-12-07T13:58:04Z","line":"/ocis/services/idm/pkg/command/server.go:75","message":"failed to bootstrap idm database"}
{"level":"error","service":"idm","db":"/.ocis/idm/ocis.boltdb","error":"open /.ocis/idm/ocis.boltdb: no such file or directory","time":"2023-12-07T13:58:04Z","line":"/ocis/ocis-pkg/log/logrus_wrapper.go:50","message":"Error opening database"}
failed to create BoltDB handler: open /.ocis/idm/ocis.boltdb: no such file or directory


❯ kubectl logs idp-7c85b8f85f-r5tj4                   
{"level":"fatal","service":"idp","error":"mkdir /.ocis: read-only file system","time":"2023-12-07T13:57:54Z","line":"/ocis/services/idp/pkg/service/v0/service.go:54","message":"could not create default config"}


❯ kubectl logs nats-669d4946dd-g2kzv                    
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"Starting nats-server"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"  Version:  2.10.5"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"  Git:      [not set]"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"  Cluster:  ocis-cluster"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"  Name:     NCKYUOC5R7DP7U573FVIQZP2SWXK6PYAURAGTXST4K6TBB3GI2HT666C"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"  Node:     LFCJo4ge"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"  ID:       NCKYUOC5R7DP7U573FVIQZP2SWXK6PYAURAGTXST4K6TBB3GI2HT666C"}
{"level":"info","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:21","message":"Starting JetStream"}
{"level":"fatal","service":"nats","time":"2023-12-07T13:59:08Z","line":"/ocis/services/nats/pkg/logging/nats.go:33","message":"Can't start JetStream: could not create storage directory - mkdir /.ocis: read-only file system"

❯ kubectl logs search-77486b474f-qh7zk                     
{"level":"error","service":"search","error":"mkdir /.ocis: read-only file system","time":"2023-12-07T13:59:16Z","line":"/ocis/services/search/pkg/server/grpc/server.go:43","message":"Error initializing search service"}
{"level":"info","service":"search","error":"mkdir /.ocis: read-only file system","transport":"grpc","time":"2023-12-07T13:59:16Z","line":"/ocis/services/search/pkg/command/server.go:66","message":"Failed to initialize server"}
mkdir /.ocis: read-only file system

❯ kubectl logs storagesystem-5575658976-7nh98                       
{"level":"info","service":"storage-system","time":"2023-12-07T13:59:46Z","line":"/ocis/ocis-pkg/registry/register.go:17","message":"registering external service com.owncloud.api.storage-system-c267b8ff-8e72-48f2-8522-e717fae6b242@10.2.1.135:9215"}
{"level":"info","service":"storage-system","time":"2023-12-07T13:59:46Z","line":"/ocis/ocis-pkg/registry/register.go:17","message":"registering external service com.owncloud.web.storage-system-5cd748a9-a649-4874-9d69-c5fb5435fe0e@10.2.1.135:9216"}
{"level":"info","service":"storage-system","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/cmd/revad/runtime/runtime.go:85","message":"host info: storagesystem-5575658976-7nh98"}
{"level":"info","service":"storage-system","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/cmd/revad/runtime/runtime.go:178","message":"running on 1 cpus"}
{"level":"info","service":"storage-system","pkg":"grace","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/cmd/revad/internal/grace/grace.go:187","message":"pidfile saved at: /tmp/revad-storage-system-2afd520f-c7eb-4fe2-a03b-66ce348918ea.pid"}
{"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: gateway"}
{"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: userprovider"}
{"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: authregistry"}
{"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: authprovider"}
{"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: permissions"}
{"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: storageregistry"}
{"level":"error","service":"storage-system","error":"unable to register services: rgrpc: grpc service storageprovider could not be started,: mkdir /.ocis: read-only file system","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/cmd/revad/runtime/runtime.go:206","message":"error starting the grpc server"}
{"level":"info","service":"storage-system","pkg":"grace","time":"2023-12-07T13:59:46Z","line":"/ocis/vendor/github.com/cs3org/reva/v2/cmd/revad/internal/grace/grace.go:95","message":"pid file \"/tmp/revad-storage-system-2afd520f-c7eb-4fe2-a03b-66ce348918ea.pid\" got removed"}


❯ kubectl logs store-6477fbd94f-2xspg
{"level":"fatal","service":"store","error":"mkdir /.ocis: read-only file system","time":"2023-12-07T13:59:23Z","line":"/ocis/services/store/pkg/server/grpc/server.go:39","message":"could not initialize service handler"}

❯ kubectl exec -it ocs-6fb7c567bf-zdnhn -- /bin/sh
~ $ whoami
whoami: unknown uid 1000

owncloud/ocis:5.0.0-alpha.5 shows ocis-user for uid 1000.

❯ kubectl exec -it ocs-694459459f-mtps4 -- /bin/sh
~ $ whoami
ocis-user

I think this is expected from my own image?

[wkloucek]

I build my own OCIS docker image using v5.0.0-alpha.5 without changing any code and Dockerfile. I pushed the image to my own registry.

I tried to deploy my own OCIS using ocis helm charts (master branch commit ef81b57) by just changing image location. Some pods were failed to deploy.

I'm sorry to respond that late.

As stated yesterday:

master branch is only compatible to oCIS 4
next branch is only compatible to oCIS 5, but I didn't test 5.0.0-beta.2 so far