Converting Sensu Workshop docker-compose into an acornfile

[jspaleta]

Ok, I'm digging in and I'm gonna kick the tires on acorn with a real world project.
We've got a docker-compose based Sensu workshop environment that we spun up as educational collateral for people to be able to use locally. I'm going to see how far I can get to translating that beast of a docker-compose into an operational acornfile as a way to learn up acorn.

Here's what I'm starting with: https://github.com/sensu/sensu-go-workshop

I'm just going to try to convert the simplest workshop environment the docker-compose-default

I'll touch base in this discussion thread as I go, with successes/failures/questions. I don't think I can get it all done in one night so this is gonna be sort of an async adventure over multiple days.

Pardon me if I end up bringing up existing topics. I want this to be sort of a full journal of trying to translate a working docker-compose stack into an acorn. I think might be valuable in context.

So here we go.....

[jspaleta]

Acorn Question: environment value inheritance?

We're relying on the docker-compose understanding of the .env file to pre-seed several common environment variable values in the defined containers. Does acorn have an understanding of an environment file to read in and inherit values from? or do i need to set the env variable value explicitly?

[jspaleta]

Close I think. That definitely solves the bulk of the operational issues.

We're also using envvars to define the image and tag, its an abstraction so workshop users can pull the specific version of Sensu they want to test against.

Let me show you what I'm starting with in that docker-compose file:

services:
 sensu-backend:
   image: ${WORKSHOP_DOCKER_IMAGE}:${WORKSHOP_DOCKER_TAG}
   deploy:
     resources:
       limits:
         cpus: "1.0"
         memory: "2048M"
   labels:
   - sensu.io/workshop/role=sensu-backend
   ports:
   - 3000:3000
   - 8080:8080
   - 8081:8081
   environment:
   - SENSU_INTERNAL_ENVIRONMENT
   - SENSU_BACKEND_CLUSTER_ADMIN_USERNAME
   - SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD
   - SENSU_BACKEND_CLUSTER_ADMIN_API_KEY
   - SENSU_WORKSHOP_ENV_SECRET
   - SUMOLOGIC_HTTP_SOURCE_URL
   volumes:
   - type: volume
     source: sensu_data
     target: /var/lib/sensu
   healthcheck:
     test: wget -q -O- http://127.0.0.1:8080/health
     interval: 10s
     timeout: 5s
     retries: 6
   command: >-
     sensu-backend start
     --log-level=warn
     --debug=false

The arg.env construct solves the environment: pass-through in most cases. Though I do have a question about that. If you need a container to override a specific value in that args.env map can you do that? In docker-compose you can provide overrides values on a per container basis.

[cloudnautique]

You can do something like this:
structure the env.yaml

global:
  WORKSHOP_VERSION: "003"
  WORKSHOP_SENSU_VERSION: "6.7.2"
  WORKSHOP_SENSU_BUILD: "6283"
  WORKSHOP_DOCKER_IMAGE: "sensu/sensu"
  WORKSHOP_DOCKER_TAG: "${WORKSHOP_SENSU_VERSION}"
  WORKSHOP_HOSTNAME: "127.0.0.1"
  WORKSHOP_DEFAULT_TRAINEE_PASSWORD: "password"
  WORKSHOP_DEFAULT_ADMIN_USERNAME: "sensu"
  WORKSHOP_DEFAULT_ADMIN_PASSWORD: "sensu"
  WORKSHOP_DEFAULT_API_KEY: "4f3b7f9a-ba24-4580-a028-d927b63ea54f"
  WORKSHOP_SETUP_RETRIES: "10"
nodes:
  web:
    WORKSHOP_DOCKER_IMAGE: "test"

Then change the Acornfile

args: envs: { 
    global: {}
    nodes: {}
}

containers: {
    web: {
        image: "nginx"
        env: std.ifelse(args.envs.nodes["web"] != _|_, std.merge(args.envs.global, args.envs.nodes["web"]), args.envs.global)
    }
}

The syntax of | might change in the future, it is the bottom error in cue, and in this case it's testing if "web" exists. If it doesn't it will cause that | error value.

[cloudnautique]

Also, in your env.yaml file, you could reference secrets like:

# ...
WORKSHOP_ADMIN_SECRET: "secret://sensu-admin-token/token" 
# ...

Acornfile:

// ...
secrets: "sensu-admin-token": type: "token"
// ...

In that case Acorn would generate a random token for you and it would be set in that env var.

[cloudnautique]

On the image/tag issue, those should also be args, but used on the build.

acorn build . --image "sensu/sensu" --tag v1.1.1
or acorn build . --image sensu/sensu:v1.1.1

Acornfile

args: image: "sensu/sensu"
args: tag: "v1.1.1"

containers: {
  sensu: {
     // ...
     image: "\(args.image):\(args.tag)"
    // ...
  }
}

If you do it in a single string you can drop the tag bit.

The images get built into the Acorn so if you want to change them, you'll need to rebuild the Acorn.

[jspaleta]

that makes sense

[jspaleta]

Okay putting this to bed for the night. I have the first service in the docker-compose-default file converted and it looks like the sensu-backend is up in running and Healthy in minikube, but its not clear the ports are published as intended.

Success: the sensu-backend appears to be running now as an acorn

Here's what I started with in the docker-compose file:

  # Sensu Backend
  sensu-backend:
    image: ${WORKSHOP_DOCKER_IMAGE}:${WORKSHOP_DOCKER_TAG}
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: "2048M"
    labels:
    - sensu.io/workshop/role=sensu-backend
    ports:
    - 3000:3000
    - 8080:8080
    - 8081:8081
    environment:
    - SENSU_INTERNAL_ENVIRONMENT
    - SENSU_BACKEND_CLUSTER_ADMIN_USERNAME
    - SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD
    - SENSU_BACKEND_CLUSTER_ADMIN_API_KEY
    - SENSU_WORKSHOP_ENV_SECRET
    - SUMOLOGIC_HTTP_SOURCE_URL
    volumes:
    - type: volume
      source: sensu_data
      target: /var/lib/sensu
    healthcheck:
      test: wget -q -O- http://127.0.0.1:8080/health
      interval: 10s
      timeout: 5s
      retries: 6
    command: >-
      sensu-backend start
      --log-level=warn
      --debug=false

Here's the Acornfile equivalent I have right now:

// Definition of containers to run
containers: {
  "sensu-backend": {
    image: "sensu/sensu:6.7.2"
    cmd: [
           "sensu-backend", "start",
          "--log-level", "debug",
          "--debug", "true",
         ]
    ports: {
      publish: [
        "3000/http",
        "8080/http",
        "8081/http",
      ]
    }
    env: [
      "SENSU_INTERNAL_ENVIRONMENT=workshop",
      "SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=sensu", 
      "SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=sensu", 
      "SENSU_BACKEND_CLUSTER_ADMIN_API_KEY=4f3b7f9a-ba24-4580-a028-d927b63ea54f", 
      "SENSU_WORKSHOP_ENV_SECRET=helloworld", 
      "SUMOLOGIC_HTTP_SOURCE_URL=https://endpointX.collection.sumologic.com/receiver/v1/http/xxxxxxxxxxxxxxxxxxxx", 
    ]
    dirs: {
      "/var/lib/sensu": "volume://sensudata?subpath=backend"
    }
    probe: "http://localhost:8080/health"
  }
}


// Definition of volumes that this acorn needs to run
volumes: {
  "sensudata": {
    accessModes: "readWriteOnce"
  }
}

I need to iterate a bit so I can use an env.yaml as an arg, instead of hard-coding but I'm not too worried about that.

Proof it's alive:

acorn app workshop
NAME       IMAGE          HEALTHY   UP-TO-DATE   CREATED   ENDPOINTS                                                                                                                                                                                                                       MESSAGE
workshop   78e1fb021b3b   1         1            12m ago   http://sensu-backend.workshop.local.on-acorn.io => sensu-backend:3000, http://sensu-backend-8080.workshop.local.on-acorn.io => sensu-backend:8080, http://sensu-backend-8081.workshop.local.on-acorn.io => sensu-backend:8081   OK

kubectl get namespaces
NAME                    STATUS   AGE
acorn                   Active   3h52m
acorn-system            Active   3h52m
default                 Active   4h48m
ingress-nginx           Active   3h55m
kube-node-lease         Active   4h48m
kube-public             Active   4h48m
kube-system             Active   4h48m
kubernetes-dashboard    Active   3h57m
workshop-2d73fda0-c37   Active   15m

kubectl get pods -n workshop-2d73fda0-c37
NAME                             READY   STATUS    RESTARTS   AGE
sensu-backend-6dd85d45f6-xmfvh   1/1     Running   0          16m

kubectl get deploy -n workshop-2d73fda0-c37
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
sensu-backend   1/1     1            1           16m

Problem: No evidence that the port publishing is working as expected

Ingress addon is confirmed enabled

$ minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
| ingress                     | minikube | enabled ✅   | Kubernetes                     |

I ask minikube what the url should be for the sensu-backend service and its just blank

$ minikube service -n workshop-2d73fda0-c37 --all
|-----------------------|---------------|-------------|--------------|
|       NAMESPACE       |     NAME      | TARGET PORT |     URL      |
|-----------------------|---------------|-------------|--------------|
| workshop-2d73fda0-c37 | sensu-backend |             | No node port |
|-----------------------|---------------|-------------|--------------|

Yes this is minikube so the port stuff is weird anyways because of the minikube's semi-documented tunnel thing to cross the minikube vm to host boundary. But even without the tunnel if the port publishing was working I'd expect to get something 'fun' when I ask minikube for the service URL. I'm doing something wrong in that Acornfile in defining the ports to publish.. I have to be. But I'm wrapping up for tonight.

[cloudnautique]

All of the ports are HTTP and would go through the ingress. So if you do kubectl get ingress -n workshop-… you should see them. The other thing you can can publish them as TCP, and then Acorn will try and create svc-lbs for them. According to Acorn in your acorn apps output the endpoints all have on-acorn.io domains through the ingress.

[jspaleta]

yeah understanding ingress is gonna be a gotcha for a lot of people.

[jspaleta]

changing those ports to "tcp" got me over the hump

$ minikube service -n workshop-8096dd62-372 --all
|-----------------------|---------------|-------------|--------------|
|       NAMESPACE       |     NAME      | TARGET PORT |     URL      |
|-----------------------|---------------|-------------|--------------|
| workshop-8096dd62-372 | sensu-backend |             | No node port |
|-----------------------|---------------|-------------|--------------|
😿  service workshop-8096dd62-372/sensu-backend has no node port
|-----------------------|------------------------------------|-------------|---------------------------|
|       NAMESPACE       |                NAME                | TARGET PORT |            URL            |
|-----------------------|------------------------------------|-------------|---------------------------|
| workshop-8096dd62-372 | sensu-backend-publish-8096dd62-372 | 3000/3000   | http://192.168.49.2:31277 |
|                       |                                    | 8080/8080   | http://192.168.49.2:31303 |
|                       |                                    | 8081/8081   | http://192.168.49.2:32641 |
|-----------------------|------------------------------------|-------------|---------------------------|

those reported minikube service urls work from the host as I expect for this local development environment. So I'm going with that for now so I can move on. But I think people coming from a docker-compose are probably going to consistently stumble on this point. But its clear this is a great development versus production conditional config situation. In production I'll want the dns resolved urls. For local dev..its probably an unneeded complication for a lot of people.

[jspaleta]

Okay so a few days (and a brand new hot water heater later) I've got a chance to pick this back up.

So before going back and refactoring the sensu-backend logic to make it customizable with arguments and trying to figure out which secrets can be auto generated... i wanted to finish getting a baseline version of the workshop up.... so next up was the sensu-agent container.

There's a reason I'm not refactoring the backend yet. There is an extra configuration container in the workshop that seeds the sensu-backend with state that also needs to be refactored so I need to work my way back to that. So for now I'm using as much of the defaults as I can just to get things up and running.

Here's the docker-compose container definition with its bells and whistles that I started with.

  # Sensu Agent
  #
  # Includes monitoring agent, local API, and local StatsD socket.
  #
  # See: https://docs.sensu.io/sensu-go/latest/reference/agent/
  sensu-agent:
    image: ${WORKSHOP_DOCKER_IMAGE}:${WORKSHOP_DOCKER_TAG}
    deploy:
      resources:
        limits:
          memory: 256M
    labels:
    - sensu.io/workshop/role=sensu-agent
    ports:
    - ${SENSU_API_PORT}/tcp
    - ${SENSU_STATSD_METRICS_PORT}/udp
    environment:
    - SENSU_BACKEND_URL=ws://sensu-backend:8081
    - SENSU_AGENT_PASSWORD
    - SENSU_NAMESPACE
    - SENSU_SUBSCRIPTIONS
    - SENSU_KEEPALIVE_INTERVAL=5
    - SENSU_KEEPALIVE_WARNING_THRESHOLD=10
    - SENSU_KEEPALIVE_CRITICAL_THRESHOLD=20
    - SENSU_API_HOST
    - SENSU_API_PORT
    - SENSU_STATSD_METRICS_HOST
    - SENSU_STATSD_METRICS_PORT
    volumes:
    - type: tmpfs
      target: /var/lib/sensu
    healthcheck:
      test: wget -q -O- http://127.0.0.1:3031/healthz
      interval: 10s
      timeout: 5s
      retries: 6
    command: >-
      sensu-agent start
      --log-level=warn
      --deregister=true
      --detect-cloud-provider
      --labels="environment=training,workshop_version=${WORKSHOP_SENSU_VERSION}"
      --statsd-event-handlers sumologic-metrics
      --password ${SENSU_AGENT_PASSWORD}

Here's the stripped down acornfile definition

"sensu-agent": {
    image: "sensu/sensu:6.7.2"
    cmd: [
           "sensu-agent", "start",
           "--name", "acorn-agent",
           "--log-level=warn",
           "--deregister=true",
           "--detect-cloud-provider",
           "--labels", "build=acorn,environment=training,workshop_version=003"
         ]
    env: [
      "SENSU_BACKEND_URL=ws://sensu-backend:8081",
      "SENSU_SUBSCRIPTIONS=workshop autodiscovery probes poller",
    ]
    ports: {
      internal: [
        "3031/tcp",
        "8125/udp",
      ]
    }
    dirs: {
      "/var/lib/sensu": "volume://sensudata?subpath=agent"
    }
    probe: "http://localhost:3031/healthz"

Still a little mystified by the minikube ingress magic but using the url listed from minikube service command I'm able to get to the sensu-backend dashboard and I'm able to see the sensu-agent up and running with the labels and name I expect.
Screenshot:

So getting there, I now have a sensu-backend I can interact with from the host, and a sensu-agent that is only available in the scope of the k8s deployment. Now its just a matter of adding the auxiliary services like the workshop provided chat and influxdb..and then running the configuration container to seed state and its basically a functioning Sensu workshop environment.

[jspaleta]

Okay before I go further, one big take away from this is the largest refactoring hurdle for docker-compose users will be the loss of environment variable value inheritance from the external environment and/or .env file that docker-compose supports. This workshop uses the environment variable value inheritance pattern a lot.

I should be able to construct an arguments yaml map that works as a sufficient replacement, but this hurdle might be worth talking about in detail when aiming content at docker-compose converts. Especially if there's a "good reason"{tm} to not allow the inheritance pattern. Off the top of my head, supporting value inheritance like that might make some of the conditional logic for deploying to testing/production more brittle or error-prone. Being explicit about where you're grabbing the env variable values passed to containers means less surprises.

[cloudnautique]

There are two concepts we have that might address these issues, but they might be insufficient or need improved UX.

You could use an Acornfile of Acorns, but this path is probably overkill for simpler apps.
Profiles in the main Acornfile could help here, they change the default values when running the app. So you could do:

acorn run --profile workstation <image>
#or
acorn run --profile prod <image>

Each profile would set different variables and Acorn would do the right thing configuring it. You can also do the variable.yaml file per env.

acorn run <image> --input-vars @prod.yaml
# or
acorn run <image> --input-vars @workstation.yaml

Is the friction in converting the format from VAR_NAME=<value> to YAML? Or is it that you would want:

ONE_VALUE="one"
OTHER_ONE_VALUE="${ONE_VALUE}" 

[cjellick]

is --input-vars a generic acorn thing or something the acorn author would have to setup? I do not see it in any of the cli help

[cloudnautique]

You would have to define it as an Arg in the Acornfile.

[cjellick]

Interesting. Yeah, I don't love either of those solutions for what's being asked here.

[jspaleta]

Is the friction in converting the format from VAR_NAME=<value> to YAML? Or is it that you would want:

Having to convert from shell readable envvar format to yaml will have non-zero friction for some people, but its not conceptually or implementation difficult. In fact I think there are a couple of "good reasons"{tm} to talk about to justifying the need for this once you start using acorns to deploy into different environments and you need conditional logic for choosing the right nested map of values from argument maps.

But that is a detail. I'm more concerned about the loss of inheritance for envvar with no values defined in the container definition as a starting point for people who want to move quickly from docker-compose into acorn with one of their workloads.

in my docker-compose right now I have things like:

sensu-agent:
    environment:
    - SENSU_BACKEND_URL=ws://sensu-backend:8081
    - SENSU_NAMESPACE
    - SENSU_SUBSCRIPTIONS
    - SENSU_KEEPALIVE_INTERVAL=5
    - SENSU_KEEPALIVE_WARNING_THRESHOLD=10
    - SENSU_KEEPALIVE_CRITICAL_THRESHOLD=20

The values for SENSU_NAMESPACE and SENSU_SUBSCRIPTIONS get inherited from the docker-compose running environment and optionally from the .env file which will override the running environment. I don't actually need the .env I just need to do
export SENSU_NAMESPACE=fun and then run docker-compose start in the shell. All of those bare SENSU_NAMESPACE envvar references inherit the running shell environment's value for that variable and we are off to the races.

My current understanding is that the Acornfile requires an explicitly set value for these, there's no inheritance from the running environment. There's no example of this sort of inheritance pattern in the docs yet, so I'm assuming its not gonna work. Or am I wrong about that? I still to fuzz test some of my understanding and see how things break.