Podman support #995

amnonh · 2020-07-19T11:52:49Z

podman should be a docker drop-in replacement, so technically, it should be enough to alias podman to docker
and the monitoring stack should work.

This is almost true.
We do not use root users inside or outside the containers, so for the monitoring solution to work fine it should use the same user id and group.

In docker, it's enough to add the running user group to Docker and change the running user roll, in podman it's different, you need to add the groups to the podman command and tell it to keep the user id.

There is also a bug in podman containers/podman#7087 that prevents rootless container to communicate using a pod.
As a workaround, the containers will use a host address

Fixes #994

amnonh · 2020-07-20T11:56:16Z

@gleb-cloudius can you take the PR for a spin?

avikivity · 2020-07-20T12:01:30Z

Not sure why you need any of it.

gleb-cloudius · 2020-07-20T12:10:35Z

It starts but generated grafana/provisioning/datasources/datasource.yaml is broken:

# config file version
apiVersion: 1
datasources:
- name: prometheus
  type: prometheus
  url: http://:9090
  access: proxy
  basicAuth: false

- name: alertmanager
  type: camptocamp-prometheus-alertmanager-datasource
  orgId: 1
  typeLogoUrl: public/img/icn-datasource.svg
  access: proxy
  url: http://:9093
  password:
  user:
  database:
  basicAuth:
  isDefault:
  jsonData:
    severity_critical: '4'
    severity_high: '3'
    severity_warning: '2'
    severity_info: '1'

amnonh · 2020-07-20T12:11:04Z

@avikivity
becuase of the user prometheus is using internally, prometheus needs external access to files on an external volume (so the data will be kept outside of the container)

Sometimes ago, prometheus stopped using root as their internal user, we move to use that the local user would become the user running the container which give the server inside the container permissions to change files on the local directory.

While it worked for Docker, it's not working for podman.

amnonh · 2020-07-20T12:11:52Z

@gleb-cloudius it's not broken, it's a container annoying kind of thing that there is no IP but the containers can find one another,
do you see any issues, or does it work correctly?

gleb-cloudius · 2020-07-20T12:15:01Z

@gleb-cloudius it's not broken, it's a container annoying kind of thing that there is no IP but the containers can find one another,
do you see any issues, or does it work correctly?

If it would have worked correctly I would not notice it. How this http://:9093 can work correctly?

avikivity · 2020-07-20T12:16:50Z

Is there no way to disable this non-root user? it's not common for containers.

amnonh · 2020-07-20T12:18:08Z

@gleb-cloudius
it should work with conainers, they are all on the same internal network and find one another. what error do you see?
and what command did you use to start?

gleb-cloudius · 2020-07-20T12:22:33Z

I do not think the url is even valid, so I have my doubds about it. The command is ./start-all.sh -d /data/gleb/scylla_prom -s prometheus/scylla_servers.example.yml the error is Template init failed. Bad Getaway when I try to access dashboards.

amnonh · 2020-07-20T12:22:35Z

Is there no way to disable this non-root user? it's not common for containers.

I think the idea is for security reasons, it prevents the container from gaining root access by mistake

Also, see here:
prometheus/prometheus#3441

amnonh · 2020-07-20T12:25:50Z

I do not think the url is even valid, so I have my doubds about it. The command is ./start-all.sh -d /data/gleb/scylla_prom -s prometheus/scylla_servers.example.yml the error is Template init failed. Bad Getaway when I try to access dashboards.

Can you run it with an actually running instance? template errors can be a result of no scylla instances (so some parts of the dashboards that are based on labels read from scylla are not initialized)

gleb-cloudius · 2020-07-20T12:30:11Z

On Mon, Jul 20, 2020 at 05:26:06AM -0700, Amnon Heiman wrote: > I do not think the url is even valid, so I have my doubds about it. The command is `./start-all.sh -d /data/gleb/scylla_prom -s prometheus/scylla_servers.example.yml` the error is `Template init failed. Bad Getaway` when I try to access dashboards. Can you run it with an actually running instance? template errors can be a result of no scylla instances (so some parts of the dashboards that are based on labels read from scylla are not initialized)

First of all the command above should work. Second I already know that the problem is in the url. If I fix it to have my local IP (and comment out line 123 in start-grafana.sh to not break it again) everything works.

…

-- Gleb.

amnonh · 2020-07-20T12:36:54Z

On Mon, Jul 20, 2020 at 05:26:06AM -0700, Amnon Heiman wrote: > I do not think the url is even valid, so I have my doubds about it. The command is ./start-all.sh -d /data/gleb/scylla_prom -s prometheus/scylla_servers.example.yml the error is Template init failed. Bad Getaway when I try to access dashboards. Can you run it with an actually running instance? template errors can be a result of no scylla instances (so some parts of the dashboards that are based on labels read from scylla are not initialized)
First of all the command above should work. Second I already know that the problem is in the url. If I fix it to have my local IP (and comment out line 123 in start-grafana.sh to not break it again) everything works.
…
-- Gleb.

Then it's another difference between docker and podman, in docker by default you don't use the local IP address, I'll try to see why it worked for me and how it should be fixed

amnonh · 2020-07-23T14:58:25Z

So far I've identified the issue as a podman network limitation on rootless containers, working on a resolution

nyh · 2020-07-30T17:51:48Z

@amnonh I see this patch is sitting here for 11 days reviewed by two people and still not committed. I think the problem is that it looks like a random collection of unrelated changes without any explanation what each of those changes fixes. It would have helped if each patch had a commit message saying what problem this specific patch solves.
I myself am failing now to run "start-all.sh -d ..." on podman, and I'll try whether your patch helps, but I sill won't commit it because probably just a small part of the these changes are related to the problem I'm seeing, and I can't understand what the rest does. But I'll report back once I test this more and try to understand your patches...

nyh · 2020-07-30T18:00:16Z

start-all.sh

+        group_args+=(--group-add "$gid")
+    done
+    group_args+=(--userns=keep-id)
+fi


Ok, I'm happy to report that this change (plus the one line below using group_args) is what solves the problem I had with the "-d" option, where the container didn't have permision to actual use the directory I gave it because it belonged to my user. Please split this as a separate patch, explain why it's needed, and then I'll commit this.
But, to be honest, while this works, I don't understand what that "group-add" is supposed to be doing... Isn't "--userns=keep-id" all the magic we need? What is this group-add thing doing?

nyh · 2020-07-30T18:03:49Z

start-all.sh

+if [ ! -z "$is_podman" ] && [ "$DB_ADDRESS" = ":9090" ]; then
+    HOST_IP=`hostname -I | awk '{print $1}'`
+    DB_ADDRESS="$HOST_IP:9090"
+fi


What does this change fix? It needs to be explained either in the code as a comment, or in a commit message.
Also, what does "hostname -I | awk '{print $1'}" even trying to do? The hostname manual page clearly says: "Do not make any assumptions about the order of the output" - so what's the meaning of taking the first one?

nyh · 2020-07-30T18:06:05Z

start-grafana.sh

+        group_args+=(--group-add "$gid")
+    done
+    group_args+=(--userns=keep-id)
+fi


start-grafana.sh worked well for me even without this patch. When is this change needed? Perhaps for the "-G" option to work? Or something else?
And again I understand the userns thing, but not the purpose of the groups thing.

nyh · 2020-07-30T18:07:07Z

start-alertmanager.sh

@@ -85,4 +86,8 @@ then
 fi

 AM_ADDRESS="$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $ALERTMANAGER_NAME):9093"
+if [ ! -z "$is_podman" ] && [ "$AM_ADDRESS" = ":9093" ]; then
+    HOST_IP=`hostname -I | awk '{print $1}'`


Again, I don't understand the meaning of taking the first response in "hostname -I", which is a list of IP address whose order is undefined.

nyh · 2020-07-30T18:14:18Z

I can confirm that after merging the entirety of this patch, start-all.sh -d somedatadir worked well for me on Fedora 32 and podman. The one line start-all.sh patch that adds "-namens" wasn't quite enough - everything started but I still had grafana errors on the screen. So I guess something else in the start-grafana.sh also helped?
That's the good news. But I still can't commit this myself without understanding what these changes are supposed to be fdoing.

amnonh · 2020-07-31T13:56:15Z

@nyh Thanks for giving it a try and confirm that it works.
I'm not a podman expert, and give it a trial-and-error approach until I got something that works.
There is currently a bug in Podman (containers/podman#7087) that made me use that ugly hack with some arbitrary host IP address.

Luckily while Podman is important for developer working with Fedora, it's not so important for users who uses other OS.

I'll keep updating the Podman support as new fixes will be available.

prometheus and grafana can use external directories and files, using a rootless containers requires that the user inside the container would have the same user and group id as the owner of that external directory.

rootless containers in podman can comunicate in two ways, the prefered (but currently have bug and not working) is with a shared pod. The alternative is using a host ip. Until the bug in podman is solved and in a release, this patch uses an arbitrary ip address of the host (the host can have multiple interfaces anyone would do).

dyasny · 2020-08-27T02:21:37Z

There is a compatibility package called podman-docker, present in centos 8. With it installed, the stack simply worked for me.

Rhel and centos 8 are native for podman and docker-ce on them is hard to install and harder to maintain, so making sure we support podman is important as the enterprise world dumps docker

amnonh · 2020-08-27T06:30:02Z

@dyasny the question is which stack? the one with commit f6165c9 or before that?

dyasny · 2020-08-27T13:22:08Z

@amnonh I used 3.4.1 and 3.4.3. My playbook was set to install docker-ce but as soon as I switched to CentOS8 it started failing. The easy way to install docker-ce on EL8 is by using the dnf --nobest option, and that opens a huge can of worms in terms of the ability to update the OS later on. You can also build it or manually install RPMs but that's even worse. You can also remove the container-tools group, but that would pull down docker-selinux, which is required (and installing it manually is again a no-go).

Docker officially doesn't support EL8 and so far it looks like they don't intend to. There's a long conversation around both github and the RH mailing list about this.

However, when instead of trying to install docker-ce, I installed podman-docker (just a thin wrapper above podman to enable docker commands) the stack came up nicely.

changed: [10.142.0.2] => {
  "changed": true, 
  "cmd": "cd /opt/scylla-monitoring/scylla-monitoring-scylla-monitoring-3.4.3/\n./start-all.sh  -d /opt/scylla-monitoring/data  -v latest,2020.1  -s /opt/scylla-monitoring/scylla-monitoring-scylla-monitoring-3.4.3//prometheus/scylla_servers.yml  -N /opt/scylla-monitoring/scylla-monitoring-scylla-monitoring-3.4.3//prometheus/scylla_manager_servers.yml  -b -storage.tsdb.retention.time=14d  -M 2.1 -a P@ssw0rd123\n", 
  "delta": "0:00:47.445981", 
  "end": "2020-08-26 20:01:53.746018", 
  "rc": 0, 
  "start": "2020-08-26 20:01:06.300037", 
  "stderr": "", 
  "stderr_lines": [], 
  "stdout": "Wait for alert manager container to start
                 Loading prometheus data from /opt/scylla-monitoring/data
                 Wait for Prometheus container to start..
                 Wait for Grafana container to start........
                 Start completed successfully, check http://localhost:3000", 
  "stdout_lines": ["Wait for alert manager container to start", 
                          "Loading prometheus data from /opt/scylla-monitoring/data", 
                          "Wait for Prometheus container to start..", 
                          "Wait for Grafana container to start........", 
                          "Start completed successfully, check http://localhost:3000"]
}

nyh · 2020-08-27T13:29:27Z

@dyasny indeed te point of this patch, and others, is to get podman and its podman-docker compatibility to work. The problem is that although podman is mostly compatible with docker, there are a lot of small problems or differences, and in this patch Amnon worked around some of them. So now things are working for you, because you are using the version with this patch.
I agree with you that if Redhat chose podman over docker, we should support it, without playing games like recommending people to get docker from some other repositories, or telling people to use weird kernel parameters to allow the docker to run. This was the point of this (and similar) patches.

dyasny · 2020-08-27T13:40:15Z

@nyh before I joined Scylla I was working on openshift/kubevirt, and I've been testing RH's container engine vs docker for heavyweight containers. CRI-O was bringing those up ~30% faster than docker, on average. The missing parts to CRI-O were all the other functions docker has built in, e.g. docker pull etc. Not very unixway-ish, packing it all under a single utility, but people got used to it. Podman pretty much is there to solve that problem (and a bunch of others of course). In short, I think docker is not going to be around for much longer as the de-facto container engine, and it's a great thing we adapt quickly. I'd even suggest taking these workarounds as material for a potential blog post by Amnon.

Me posting here is basically some additional confirmation that this stuff works, and works well. Kudos to you all :)

amnonh requested a review from gleb-cloudius July 20, 2020 11:56

avikivity changed the title ~~Podmon support~~ Podman support Jul 26, 2020

amnonh force-pushed the podmon branch 2 times, most recently from f69c70e to faaa0ea Compare July 26, 2020 17:47

nyh requested changes Jul 30, 2020

View reviewed changes

amnonh added 2 commits August 3, 2020 10:38

podman support rootless containers

1a4d31f

prometheus and grafana can use external directories and files, using a rootless containers requires that the user inside the container would have the same user and group id as the owner of that external directory.

amnonh force-pushed the podmon branch from faaa0ea to 45ebfdc Compare August 3, 2020 07:41

amnonh merged commit f6165c9 into scylladb:master Aug 3, 2020

amnonh deleted the podmon branch August 3, 2020 07:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman support #995

Podman support #995

amnonh commented Jul 19, 2020 •

edited

Loading

amnonh commented Jul 20, 2020

avikivity commented Jul 20, 2020

gleb-cloudius commented Jul 20, 2020

amnonh commented Jul 20, 2020

amnonh commented Jul 20, 2020 •

edited

Loading

gleb-cloudius commented Jul 20, 2020

avikivity commented Jul 20, 2020

amnonh commented Jul 20, 2020

gleb-cloudius commented Jul 20, 2020

amnonh commented Jul 20, 2020

amnonh commented Jul 20, 2020

gleb-cloudius commented Jul 20, 2020 via email

amnonh commented Jul 20, 2020

amnonh commented Jul 23, 2020

nyh commented Jul 30, 2020

nyh Jul 30, 2020

nyh Jul 30, 2020

nyh Jul 30, 2020

nyh Jul 30, 2020

nyh commented Jul 30, 2020

amnonh commented Jul 31, 2020

dyasny commented Aug 27, 2020

amnonh commented Aug 27, 2020

dyasny commented Aug 27, 2020

nyh commented Aug 27, 2020

dyasny commented Aug 27, 2020

Podman support #995

Podman support #995

Conversation

amnonh commented Jul 19, 2020 • edited Loading

amnonh commented Jul 20, 2020

avikivity commented Jul 20, 2020

gleb-cloudius commented Jul 20, 2020

amnonh commented Jul 20, 2020

amnonh commented Jul 20, 2020 • edited Loading

gleb-cloudius commented Jul 20, 2020

avikivity commented Jul 20, 2020

amnonh commented Jul 20, 2020

gleb-cloudius commented Jul 20, 2020

amnonh commented Jul 20, 2020

amnonh commented Jul 20, 2020

gleb-cloudius commented Jul 20, 2020 via email

amnonh commented Jul 20, 2020

amnonh commented Jul 23, 2020

nyh commented Jul 30, 2020

nyh Jul 30, 2020

Choose a reason for hiding this comment

nyh Jul 30, 2020

Choose a reason for hiding this comment

nyh Jul 30, 2020

Choose a reason for hiding this comment

nyh Jul 30, 2020

Choose a reason for hiding this comment

nyh commented Jul 30, 2020

amnonh commented Jul 31, 2020

dyasny commented Aug 27, 2020

amnonh commented Aug 27, 2020

dyasny commented Aug 27, 2020

nyh commented Aug 27, 2020

dyasny commented Aug 27, 2020

amnonh commented Jul 19, 2020 •

edited

Loading

amnonh commented Jul 20, 2020 •

edited

Loading