Skip to content

Latest commit

 

History

History
213 lines (162 loc) · 6.56 KB

README.md

File metadata and controls

213 lines (162 loc) · 6.56 KB

End to End testing

ToC

How it works

There are 3 main directories:

  • argo-workflows Specification of the end to end testing

  • containers Custom container images needed within the workflows

  • scripts

    • setup-instance Entrypoint and scripts dedicated for environments (locally, AWS dev, AWS gitlab)
    • run-instance Scripts executed in the argo-machine (locally, AWS instance)

setup-instance

run-instance

You need pupernetes:

$ curl -LO https://github.com/DataDog/pupernetes/releases/download/v${VERSION}/pupernetes
$ chmod +x ./pupernetes

Command line

AWS development

$ cd ${GOPATH}/src/github.com/DataDog/datadog-agent 
$ aws-vault exec ${DEV} -- inv -e e2e-tests -t dev --image datadog/agent-dev:master
$ echo $?

Locally

$ sudo ./pupernetes daemon run /opt/sandbox --job-type systemd 
$ cd ${GOPATH}/src/github.com/DataDog/datadog-agent 
$ inv -e e2e-tests -t local --image datadog/agent-dev:master
$ echo $?

Argo workflow

The argo documentation is available here, there are a lot of examples here too.

Argo assertion

To assert something in an argo workflow, you need to create a mongodb query:

name: find-kubernetes-state-deployments
activeDeadlineSeconds: 200
script:
  image: mongo:3.6.3
  command: [mongo, "fake-datadog.default.svc.cluster.local/datadog"]
  source: |
    while (1) {
      var nb = db.series.find({
      metric: "kubernetes_state.deployment.replicas_available", 
      tags: {$all: ["namespace:default", "deployment:fake-datadog"] }, 
      "points.0.1": { $eq: 1} });      
      print("find: " + nb)
      if (nb != 0) {
        break;
      }
      prevNb = nb;
      sleep(2000);
    }

This is an infinite loop with a timeout set by activeDeadlineSeconds: 200. The source is EOF to the command, equivalent to:

mongo "fake-datadog.default.svc.cluster.local/datadog" << EOF
while (1)
[...]
EOF

Try to maximise the usage of MongoDB query system without rewriting too much logic in JavaScript.

See some examples here

To discover more MongoDB capabilities:

Argo container

If you need to add a non existing public container in the workflow, create it in the container directory.

But, keep in mind this become an additional piece of software to maintain.

Upgrade - bump

This section helps you to upgrade any part of the end to end testing.

The current end to end testing pipeline relies on:

Bump hyperkube version

Read the command lines docs of pupernetes

In the Ignition systemd.units list, find the pupernetes.service one.

In its content, the systemd unit has a [service] directive where there is a single ExecStart= field: you need to add / edit the --hyperkube-version flag in the command line.

The value of this flag must be a valid release such as 1.10.1: --hyperkube-version=1.10.1

Concrete example, I want to bump the hyperkube version from 1.9.3 to 1.10.1:

{
  "systemd": {
    "units": [
      {
        "enabled": true,
        "name": "pupernetes.service",
        "contents": "[Unit]\nDescription=Run pupernetes\nRequires=setup-pupernetes.service docker.service\nAfter=setup-pupernetes.service docker.service\n\n[Service]\nEnvironment=SUDO_USER=core\nExecStart=/opt/bin/pupernetes daemon run /opt/sandbox --hyperkube-version 1.9.3 --kubectl-link /opt/bin/kubectl -v 5 --timeout 48h\nRestart=on-failure\nRestartSec=5\n\n[Install]\nWantedBy=multi-user.target\n"
      }
    ]
  }
}

Become:

{
  "systemd": {
    "units": [
      {
        "enabled": true,
        "name": "pupernetes.service",
        "contents": "[Unit]\nDescription=Run pupernetes\nRequires=setup-pupernetes.service docker.service\nAfter=setup-pupernetes.service docker.service\n\n[Service]\nEnvironment=SUDO_USER=core\nExecStart=/opt/bin/pupernetes daemon run /opt/sandbox --hyperkube-version 1.10.1 --kubectl-link /opt/bin/kubectl -v 5 --timeout 48h\nRestart=on-failure\nRestartSec=5\n\n[Install]\nWantedBy=multi-user.target\n"
      }
    ]
  }
}

Bump pupernetes - bump the version of p8s

Have a look at pupernetes.

Generate the ignition.json with the .py and the .yaml:

${GOPATH}/src/github.com/DataDog/pupernetes/environments/container-linux/ignition.py < ${GOPATH}/src/github.com/DataDog/pupernetes/environments/container-linux/ignition.yaml | jq .

Upgrade the relevant sections in the ignition script.

The ignition script must insert the core user with their ssh public key:

{
"passwd": {
    "users": [
      {
        "sshAuthorizedKeys": [
          "${SSH_RSA}"
        ],
        "name": "core"
      }
    ]
  }
}

If needed, use the ignition-linter to validate any changes.

Bump argo

Bump CoreOS Container Linux - Kinvolk Flatcar

Select any HVM AMIs from: