Wired flipt.db init

[ognif]

flipt:latest
kubernets cluster

If I start flipt in our cluster without any modification, the flipt service starts without any problem. After looking into /var/opt/flipt there is a file flipt.db.
Doing the same with a mounted pvc/volume, flipt is starting with an error.

You are currently running the latest version of Flipt [1.9.0]!
time="2022-07-21T23:24:57Z" level=info msg="shutting down..."
time="2022-07-21T23:24:58Z" level=error msg="getting db driver for: sqlite3: unable to open database file: no such file or directory"

Using an init container I see hat the flipt.db is not existing while startup the container. It looks like the flipt.db file is created on startup.

So I assume flipt is creating the flipt.db on startup. BUT it will not create this file inside the mounted volume.

Maybe flipt expect a mounted volume as an external database and will not create flipt.db?

Because the file flipt.db is not existing on pod startup, I also cannot use an init container as workaround to copy the file to the mounted volume.

[markphelps]

@ognif thanks for the issue! i'll try to reproduce and see if I can come up with a fix. Can you share part of your pvc/volume k8s config so that I could potentially reproduce?

Are you using the helm chart or just normal k8s config? (the helm chart would likely have the same problem but just wanted to double check)

[ognif]

@markphelps I have added a zip file with my version of the helm flipt chart.
You may have to change some settings from fliptvalues.yaml
The fliptvalues.yaml contain a part of persitent volume, if I disable this, flipt works well. If I enable it, the reported error message will pop up. The way I used the pvc/volume is how I use in many helm charts here, so I assume it have to work.
flipt_ingo.zip

[markphelps]

@ognif I have a PR up in the helm-charts repo to support PV/C. Let me know if you think it would work for you flipt-io/helm-charts#2

[ognif]

@markphelps thank you for your pr but please explain what this helm pr will solve the software bug?
Your helm chart will do the same like mine. Create a longhorn pvc storage but the final error keep the same:

You are currently running the latest version of Flipt [1.9.0]!
time="2022-07-26T22:28:53Z" level=info msg="shutting down..."
time="2022-07-26T22:28:54Z" level=error msg="getting db driver for: sqlite3: unable to open database file: no such file or directory"

As written before, I think it is a bug inside the software because the software will not write the flipt.db to the mounted path and so finaly the software cannot find the not existing database.
All work well with local file inside the docker container but not in mounted path.

[markphelps]

My apologies. Previously the Flipt helm chart was not even accounting for persistent volumes because it was not mounting them to the correct location in the container /var/opt/flipt. This is where the default flipt.db goes if running with SQLite.

I will try to use the latest version of the chart tonight and use the longhorn storage class like yours. Then I will be able to more easily see why it can't create the file.

[ognif]

@markphelps I have the exact same problem with Azure managed-premium storage class.

[markphelps]

@ognif what are the permissions on your mounted volume/path?

I tried running flipt and mounting a volume on my local machine with the following:

docker run -d -p 8080:8080 -p 9000:9000 -v $HOME/flipt:/var/opt/flipt markphelps/flipt:latest

And it ran fine. The $HOME/flipt directory did not exist before me running this.

Here are the permissions of that folder and contents:

~ » ls -la | grep flipt
drwxr-xr-x   3 markphelps  staff     96 Jul 27 08:59 flipt

~ » ls -la flipt/
total 152
drwxr-xr-x   3 markphelps  staff     96 Jul 27 08:59 .
drwxr-x---+ 71 markphelps  staff   2272 Jul 27 09:00 ..
-rw-r--r--   1 markphelps  staff  77824 Jul 27 08:59 flipt.db

I wonder if you may be running into permissions issues creating this file because the user that runs Flipt in the container is a non-root user, although this user should have r/w permissions to the mounted folder.

[ognif]

The daily discussion "But it works local". After all, Kubernetes is not local. So I ask you to test your software in a Kubernetes cluster. Persistence is inevitable inside a cluster.

[markphelps]

@ognif could you do me a favor and run with the log.level set to DEBUG and attach the log output? You can either set this in the configmap like:

log:
      level: DEBUG

or set it as an environment variable for the container using FLIPT_LOG_LEVEL=DEBUG. docs

When flipt starts up it checks to see if any DB migrations need to be performed, and on first run (or if flipt.db doesnt exist when running with SQLite) it will try to perform a migration.

I think what may be going on in your case is that flipt is trying to perform the migrations before the volume is ready/writeable?

Its hard for me to reproduce this currently as I dont have a cluster running with longhorn or azure storage class.

[markphelps]

@ognif can you try downgrading to v1.8.3 to see if it resolves your problem temporarily? Seems something changed in 1.9.0 that I need to debug. See #993

[markphelps]

@ognif can you try upgrading to either v1.9.1 or v1.10.1 and see if this resolves the issue?