generate kube with containers having volumes

[ikke-t]

/kind feature

Description

I was trying to create Ansible AWX pod with few containers in it and export it as kube yaml. I succeeded putting containers into pod, but "podman generate kube awx" fails due not implemented volume export. So I can't export such pod into yaml.

Steps to reproduce the issue:

1. Generate pod ( --name awx)

2. add containers into pod with -v /tmp/volume:/volume:z

3. do podman generate kube awx

Describe the results you received:

$ sudo podman generate kube awx
volume names: not yet implemented

Describe the results you expected:

I would have expected awx.yaml file with info about 4 different containers.

Additional information you deem important (e.g. issue happens only occasionally):

Happens always.

Output of podman version:

This is on Fedora 29

$ podman version
Version:       1.0.0
Go Version:    go1.11.4
Git Commit:    "49780a1cf10d572edc4e1ea3b8a8429ce391d47d"
Built:         Mon Jan 14 22:38:17 2019
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-1.git82e8011.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 49780a1cf10d572edc4e1ea3b8a8429ce391d47d'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 214802432
  MemTotal: 8052531200
  OCIRuntime:
    package: runc-1.0.0-67.dev.git12f6a99.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: d164d9b08bf7fc96a931403507dd16bced11b865
      spec: 1.0.1-dev
  SwapFree: 6857945088
  SwapTotal: 8199860224
  arch: amd64
  cpus: 4
  hostname: ohuska.localdomain
  kernel: 4.19.8-300.fc29.x86_64
  os: linux
  rootless: false
  uptime: 400h 54m 19.58s (Approximately 16.67 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 8
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 11
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora 29 up to date on laptop.

[baude]

@ikke-t thanks for the issue, as you can see, we haven't implemented volumes for generate yet. perhaps you can help with some feedback ... can you describe your use case and what are your expectations on the Kubernetes side (besides saying it just works). should the yaml describe the volume, create it, etc?

[ikke-t]

Damned this became long story. TL;DR: I wanted to save pod+containers as yaml to create ansible template out of AWX. Totally personal selfish reason, scratching my own itch.

---

I'm happy to do so :) I'll explain my use case. I have been running Ansible AWX at home for practising at the beginning, and later on just keeping up my personal systems at home and some demo environments. I was also super curious about podman. AWX runs on spare laptop, so no kubernetes environment.

So after changing some containers already to podman, it was AWX turn. That is of a kind that has 4 containers communicating with each other, and sharing volumes. So ideal for a single pod from my mind. Now that podman finally could do the networking parts also, I decided to give it a go. But AWX exports one port from containers, so I can't build it using podman commands because podman can expose ports only once, and I couldn't figure out how to do that with podman pod create + putting the containers it. So I need the yaml in order to get AWX started with podman.

After I saw this blog: https://developers.redhat.com/blog/2019/01/29/podman-kubernetes-yaml/ I thought that's supercool way to get them all into yaml. I don't need to figure out all the options what to write, and format, I just start the pod manually and shovel the containers in, and export the yaml and that way get the whole pod+containers as one yaml. Then I can add the networking parts to the template.

At this phase I run into this shortcoming, as generate kube couldn't export settings as yaml while containers had mounts exported.

So it's not biggie, I can add the volume mounts manually afterwards. And will do, once the other issue (bug #2304 ) get's fixed and I can export anything at all to begin with.

Damned this became long story. Point being, I needed the yaml to create ansible template out of that, which I could've contributed back to AWX for such single node servers. I suppose this is very handy path from docker containers to podman, at least for me it would be. Creating yaml from scratch isn't fun.

All in all, this is not for kubernetes at all. One just needs the yaml in case containers within the pod export ports. As stated in that blog, they need to be described to podman at once, not one at the time in separate commands: https://developers.redhat.com/blog/2019/01/15/podman-managing-containers-pods/

For example, if you create a pod and then later decide you want to add a container that binds new ports, Podman will not be able to do this. You would need to recreate the pod with the additional port bindings before adding the new container.

Or then, I just misunderstood, which is very likely.

[baude]

@ikke-t thanks for the information. it is helpful that you took the time as it helps me understand what folks are trying to do and why. Would you mind clarifying with a little verbosity this one statement in your excellent response?

But AWX exports one port from containers, so I can't build it using podman commands because podman can expose ports only once, and I couldn't figure out how to do that with podman pod create + putting the containers it.

[ikke-t]

With that one I need guidance in deed :)

This is my temp hack to see what AWX installer does with docker install. I modified this:

https://github.com/ansible/awx/blob/devel/installer/roles/local_docker/tasks/standalone.yml

to be run with podman, until I couldn't figure out how to get the exposed port to get visible from host:

---
- name: create awx_pod
  command: >
    podman pod create --name awx

#    -p "{{ host_port }}:8052"


#    -v "{{ postgres_data_dir }}:/var/lib/postgresql/data:Z"
- name: Activate postgres container
  command: >
    podman run -d --pod awx --name postgres
    -e POSTGRES_USER="{{ pg_username }}"
    -e POSTGRES_PASSWORD="{{ pg_password }}"
    -e POSTGRES_DB="{{ pg_database }}"
    -e PGDATA="/var/lib/postgresql/data/pgdata"
    "{{ postgresql_image }}"
  register: postgres_container_activate

- name: Activate rabbitmq container
  command: >
    podman run -d --pod awx --name rabbitmq
    -e RABBITMQ_DEFAULT_VHOST="{{ rabbitmq_default_vhost }}"
    -e RABBITMQ_ERLANG_COOKIE="{{ rabbitmq_erlang_cookie }}"
    -e RABBITMQ_DEFAULT_USER="{{ rabbitmq_default_username }}"
    -e RABBITMQ_DEFAULT_PASS="{{ rabbitmq_default_password }}"
    "{{ rabbitmq_image }}"
  register: rabbitmq_container_activate

- name: Activate memcached container
  command: podman run -d --pod awx --name memcached memcached:alpine

- name: Wait for postgres and rabbitmq to activate
  pause:
    seconds: 15
  when: postgres_container_activate.changed or rabbitmq_container_activate.changed

- name: Set properties without postgres for awx_web
  set_fact:
    pg_hostname_actual: "{{ pg_hostname }}"
    awx_web_container_links:
      - rabbitmq
      - memcached
  when: pg_hostname is defined

- name: Set properties with postgres for awx_web
  set_fact:
    pg_hostname_actual: postgres
    awx_web_container_links:
      - rabbitmq
      - memcached
      - postgres
  when: pg_hostname is not defined or pg_hostname == ''

- name: Set properties without postgres for awx_task
  set_fact:
    pg_hostname_actual: "{{ pg_hostname }}"
    awx_task_container_links:
      - rabbitmq
      - memcached
      - awx_web:awxweb
  when: pg_hostname is defined

- name: Set properties with postgres for awx_task
  set_fact:
    pg_hostname_actual: postgres
    awx_task_container_links:
      - rabbitmq
      - memcached
      - awx_web:awxweb
      - postgres
  when: pg_hostname is not defined or pg_hostname == ''

- name: Activate AWX Web Container
  tags: test
    # links="{{ awx_web_container_links|list }}"
    # --dns-search "{{ awx_container_search_domains.split(',') if awx_container_search_domains is defined else omit }}"
    # --dns "{{ awx_alternate_dns_servers.split(',') if awx_alternate_dns_servers is defined else omit }}
    # -v {{
    #      ([project_data_dir + ':/var/lib/awx/projects:z'] if project_data_dir is defined else []) +
    #      ([ca_trust_dir + ':/etc/pki/ca-trust/source/anchors:ro'] if ca_trust_dir is defined else [])
    #    }}
    # -e DATABASE_SSLMODE: "{{ pg_sslmode | default(omit) }}"
    #     -p "{{ host_port }}:8052"
    #-v {{ project_data_dir + ':/var/lib/awx/projects:z' }}
  command: >
    podman run -d --pod awx --name awx_web
    -u root
    --hostname "{{ awx_web_hostname }}"
    -p "{{ host_port }}:8052"
    -e http_proxy="{{ http_proxy | default('') }}"
    -e https_proxy="{{ https_proxy | default('') }}"
    -e no_proxy="{{ no_proxy | default('') }}"
    -e SECRET_KEY="{{ secret_key }}"
    -e DATABASE_NAME="{{ pg_database }}"
    -e DATABASE_USER="{{ pg_username }}"
    -e DATABASE_PASSWORD="{{ pg_password }}"
    -e DATABASE_PORT="{{ pg_port }}"
    -e DATABASE_HOST="{{ pg_hostname_actual }}"
    -e RABBITMQ_USER="{{ rabbitmq_default_username }}"
    -e RABBITMQ_PASSWORD="{{ rabbitmq_default_password }}"
    -e RABBITMQ_HOST="rabbitmq"
    -e RABBITMQ_PORT="{{ rabbitmq_port }}"
    -e RABBITMQ_VHOST="{{ rabbitmq_default_vhost }}"
    -e MEMCACHED_HOST="memcached"
    -e MEMCACHED_PORT="11211"
    -e AWX_ADMIN_USER="{{ admin_user|default('admin') }}"
    -e AWX_ADMIN_PASSWORD="{{ admin_password|default('password') }}"
    ansible/awx_web
  register: awx_web_container

- name: Update CA trust in awx_web container
  command: podman exec awx_web '/usr/bin/update-ca-trust'
  when: awx_web_container.changed

- name: Activate AWX Task Container
    #-v {{
    #    ([project_data_dir + ':/var/lib/awx/projects:z'] if project_data_dir is defined else [])
    #    + ([ca_trust_dir + ':/etc/pki/ca-trust/source/anchors:ro'] if ca_trust_dir is defined else [])
    #  }}
    # links="{{ awx_task_container_links|list }}"
    # --dns-search "{{ awx_container_search_domains.split(',') if awx_container_search_domains is defined else omit }}"
    # --dns "{{ awx_alternate_dns_servers.split(',') if awx_alternate_dns_servers is defined else omit }}"
    # -e DATABASE_SSLMODE: "{{ pg_sslmode | default(omit) }}"
    #-v {{ project_data_dir + ':/var/lib/awx/projects:z' }}
  command: >
    podman run -d --pod awx --name awx_task
    -u root
    --hostname "{{ awx_task_hostname }}"
    -v {{ project_data_dir + ':/var/lib/awx/projects:z' }}
    -e http_proxy="{{ http_proxy | default('') }}"
    -e https_proxy="{{ https_proxy | default('') }}"
    -e no_proxy="{{ no_proxy | default('') }}"
    -e SECRET_KEY="{{ secret_key }}"
    -e DATABASE_NAME="{{ pg_database }}"
    -e DATABASE_USER="{{ pg_username }}"
    -e DATABASE_PASSWORD="{{ pg_password }}"
    -e DATABASE_HOST="{{ pg_hostname_actual }}"
    -e DATABASE_PORT="{{ pg_port }}"
    -e RABBITMQ_USER="{{ rabbitmq_default_username }}"
    -e RABBITMQ_PASSWORD="{{ rabbitmq_default_password }}"
    -e RABBITMQ_HOST="rabbitmq"
    -e RABBITMQ_PORT="{{ rabbitmq_port }}"
    -e RABBITMQ_VHOST="{{ rabbitmq_default_vhost }}"
    -e MEMCACHED_HOST="memcached"
    -e MEMCACHED_PORT="11211"
    -e AWX_ADMIN_USER="{{ admin_user|default('admin') }}"
    -e AWX_ADMIN_PASSWORD="{{ admin_password|default('password') }}"
    ansible/awx_task

Never mind the playbook being ugly, I just try to get the containers running so I can get the yaml out of it. I removed the volumes to get past this issue we are discussing.

There is port expose for ansible_web. It fails with this:

cannot set port bindings on an existing container network namespace

And reading the blog, it seems the expose would need to be done at the time of setting the infra container. Which gets created at the time of pod create. Like blog says, it fails if tried exposing port after infra container is created.

And if I move that port expose to infra container, for pod create, it won't pass the traffic to ansible_web container.

So my thought was, that it probably works putting all that to yaml, so podman can do the right thing at once while setting up the pod from yaml file, instead of separate commands. Is my logic right?

[ikke-t]

btw, the container and pod inspect are printed out in case #2304

[rhatdan]

@baude @ikke-t is this fixed now?

[ikke-t]

I'm not aware that anything would have been done to fix this. I just explained my use case. Could be that it gets closed/won't fix if I'm alone with my use case :)

[ikke-t]

do'h, I feel dumb now. I only realised now that podman can't read back the yamls itself. All the time I was thinking I create yaml file for awx containers for single pod. And that could be stored to e.g. /etc/podman/awx.yml, and somehow the pod could have been started by systemctl like podman pod create -f /etc/podman/awx.yml.

I don't see such -f option in podman create, which kills the use case. I don't know where I got that into my mind. Too optimistic thinking and too willd imagination, I guess. Would be super cool feature, although. Separating the pod config from podman start command. It would simplify systemctl files a lot, as parameters would not need to be there. Nor in external shell script.

E.g, this would be the systemd file:

[Unit]
Description=AWX Podman Container Pod
After=network.target

[Service]
Type=simple
TimeoutStartSec=2
ExecStartPre=-/usr/bin/podman pod rm awx

ExecStart=/usr/bin/podman pod run -f /etc/podman/awx.yml

ExecReload=-/usr/bin/podman pod stop "awx"
ExecReload=-/usr/bin/podman pod rm -f "awx"
ExecStop=-/usr/bin/podman pod stop "awx"
Restart=on-failure
RestartSec=30

[Install]
After=network.target

and podman config for the pod would be like this:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: awx
  name: awx
spec:
  containers:
  - command:
    - docker-entrypoint.sh
    - postgres
    env:
    - name: POSTGRES_DB
      value: awx
    - name: PGDATA
      value: /var/lib/postgresql/data/pgdata
    - name: POSTGRES_USER
      value: awx
    - name: POSTGRES_PASSWORD
      value: awxpass
    image: docker.io/library/postgres:9.6
    name: postgres
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
    workingDir: /
  - command:
    - docker-entrypoint.sh
    - /bin/sh
    - -c
    - /launch.sh
    env:
    - name: RABBITMQ_DEFAULT_VHOST
      value: awx
    - name: RABBITMQ_ERLANG_COOKIE
      value: cookiemonster
    - name: RABBITMQ_DEFAULT_USER
      value: guest
    - name: RABBITMQ_DEFAULT_PASS
      value: guest
    image: docker.io/ansible/awx_rabbitmq:3.7.4
    name: rabbitmq
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
    workingDir: /
  - command:
    - /tini
    - --
    - /bin/sh
    - -c
    - /usr/bin/launch_awx_task.sh
    env:
    - name: HOSTNAME
      value: awx
    - name: http_proxy
    - name: RABBITMQ_USER
      value: guest
    - name: MEMCACHED_PORT
      value: "11211"
    - name: AWX_ADMIN_USER
      value: admin
    - name: SECRET_KEY
      value: awxsecret
    - name: DATABASE_PASSWORD
      value: awxpass
    - name: MEMCACHED_HOST
      value: memcached
    - name: AWX_ADMIN_PASSWORD
      value: password
    - name: no_proxy
    - name: DATABASE_NAME
      value: awx
    - name: DATABASE_USER
      value: awx
    - name: RABBITMQ_PASSWORD
      value: guest
    - name: RABBITMQ_VHOST
      value: awx
    - name: https_proxy
    - name: DATABASE_HOST
      value: postgres
    - name: DATABASE_PORT
      value: "5432"
    - name: RABBITMQ_HOST
      value: rabbitmq
    - name: RABBITMQ_PORT
      value: "5672"
    image: docker.io/ansible/awx_task:latest
    name: awxtask
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      runAsUser: 0
    workingDir: /var/lib/awx
  - command:
    - /tini
    - --
    - /bin/sh
    - -c
    - /usr/bin/launch_awx.sh
    env:
    - name: HOSTNAME
      value: awxweb
    - name: container
      value: podman
    - name: no_proxy
    - name: DATABASE_PASSWORD
      value: awxpass
    - name: https_proxy
    - name: DATABASE_HOST
      value: postgres
    - name: RABBITMQ_PASSWORD
      value: guest
    - name: RABBITMQ_VHOST
      value: awx
    - name: MEMCACHED_HOST
      value: memcached
    - name: http_proxy
    - name: DATABASE_NAME
      value: awx
    - name: DATABASE_USER
      value: awx
    - name: RABBITMQ_USER
      value: guest
    - name: RABBITMQ_HOST
      value: rabbitmq
    - name: AWX_ADMIN_PASSWORD
      value: password
    - name: SECRET_KEY
      value: awxsecret
    - name: DATABASE_PORT
      value: "5432"
    - name: RABBITMQ_PORT
      value: "5672"
    - name: MEMCACHED_PORT
      value: "11211"
    - name: AWX_ADMIN_USER
      value: admin
    image: docker.io/ansible/awx_web:latest
    name: awxweb
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      runAsUser: 0
    workingDir: /var/lib/awx
    ports:
      - containerPort: 8052
        hostPort: 8052
        protocol: TCP
  - command:
    - docker-entrypoint.sh
    - memcached
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: HOSTNAME
      value: memcached
    - name: container
      value: podman
    - name: MEMCACHED_SHA1
      value: f67096ba64b0c47668bcad5b680010c4f8987d4c
    - name: MEMCACHED_VERSION
      value: 1.5.12
    image: docker.io/library/memcached:alpine
    name: memcached
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      runAsUser: 11211
    workingDir: /
status: {}

That would be nice way to configure pods. Compare to putting all those options to systemd file. This is BTW missing the volume mounts. This would be something in between k8s and docker compose for simple hosts only having podman.

[mheon]

That sounds like podman play kube which we do ship? (Minus the volume mounts, again)

[ikke-t]

thanks, I thought I saw somewhere that I could run the yaml with podman. Doesn't work on my fedora though, no such option in

$ rpm -q podman
podman-1.0.1-35.dev.gitb223d4e.fc31.x86_64

that version doesn't have "play" sub command. Is it in some other package?

[mheon]

...Huh. The code's been merged since December, so I'm rather curious as to why it's not there...

[ikke-t]

and for the mount, in this case it should be (from ansible), so it saves both AWX stuff and posgres data:

([project_data_dir + ':/var/lib/awx/projects:z'] if project_data_dir is defined else [])
{{ postgres_data_dir }}:/var/lib/postgresql/data:Z

This is the use case for this particular issue I was looking for.

[ikke-t]

I downloaded that version today from Koji: https://koji.fedoraproject.org/koji/packageinfo?packageID=26289, the latest. This is Fedora 29.

[ikke-t]

not there:

$ podman --help |grep -i play
  info        Display information pertaining to the host, current storage stats, and build of podman. Useful for the user and when reporting issues.
  inspect     Display the configuration of a container or image
  kube        Play a pod based on Kubernetes YAML
  stats       Display percentage of CPU, memory, network I/O, block I/O and PIDs for one or more containers
  top         Display the running processes of a container
  version     Display the Podman Version Information

[mheon]

Just built from master, it's missing there too.
@baude Looks like it might have accidentally been removed when we migrated to Cobra?

[mheon]

Also, how did our tests miss an entire command going missing?

[rhatdan]

podman play --help works.
Just listing it in podman --help is not.

[ikke-t]

@rhatdan, it is not there, no matter how you put the help switch:

$ podman play --help
manage pods and images

Usage:
  podman [flags]
  podman [command]

Available Commands:
....

The play just doesn't seem to exist:

$ sudo podman play kube awx.yml
manage pods and images

Usage:
  podman [flags]
  podman [command]

Available Commands:
  attach      Attach to a running container
....

[ikke-t]

Or, am I missing some dependency lib or something? Like said, this is Fedora 29 with podman downloaded from Koji by dnf. Would it need some forgotten dependency library for play to become active?

[rhatdan]

Well we are cutting the release of podman 1.1 today. So it will be in there.

[rhatdan]

I think we are partially there. PR #2575 should complete.

[haircommander]

we're blocked on generate by some backend work by @mheon . I will come back once that comes down the pike

[rhatdan]

@haircommander Didn't you fix this?

[haircommander]

No that was on play kube. @mheon's volume stuff landed though so I can take a stab at this now

[kbaegis]

+1 :)

[baude]

@haircommander updates on this?

[haircommander]

@baude nope I have yet to "take a stab at it"

[rhatdan]

We can open this up for an intern?

[haircommander]

SGTM

[fatherlinux]

@ikke-t thanks for the issue, as you can see, we haven't implemented volumes for generate yet. perhaps you can help with some feedback ... can you describe your use case and what are your expectations on the Kubernetes side (besides saying it just works). should the yaml describe the volume, create it, etc?

@baude bumping this a hair, and jumping back to one of your original questions. In my mind, the output yaml would include a PVC for /mnt. Then, it would let Kubernetes handle satisfying that need at runtime. Make sense? For example, if I typed:

podman run -dt -v /mnt/fred:/mnt/data quay.io/fatherlinux/two-pizza

podman generate kube -l (for shorthand)

I would expect it to add the following to the Deployment or Replication Controller:

...
volumeMounts:
- name: mnt-data
mountPath: /mnt/data
volumes:
- name: mnt-data
persistentVolumeClaim:
claimName: mnt-data

And, the following persistent volume claim:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mnt-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

I wouldn't feel bad about setting size to 10G and letting the user customize from there (manually in an editor).

[rhatdan]

WDYT @haircommander ^^

[haircommander]

@fatherlinux does the 10G default originate from somewhere or is it just a chosen value? Seems a bit steep for a default, unless it's well defined for k8s PVC