Run node-problem-detector e2e test continuously

[xueweiz]

What would you like to be added:

A few continuous test job to run NPD e2e tests.

Why is this needed:

This is useful for ensuring the healthiness of NPD project.
This will be part of kubernetes/node-problem-detector#296

[xueweiz]

/assign @xueweiz
/cc @krzyzacy
/cc @wangzhen127
/cc @Random-Liu
/cc @andyxning
/cc @dchen1107

[wangzhen127]

Shall we also add presubmits?

[xueweiz]

I'm not confident about the test right now. Once we can confirm that it's fast and non-flaky, then we can add it to presubmits. But for now, I'm not that confident :P I think it will just block the merge lol

[xueweiz]

Hum, seems I didn't set the environment variables correctly here:

  spec:
    containers:
    - image: gcr.io/k8s-testimages/kubekins-e2e:v20190917-d326d2a-master
      env:
      - name: SSH_USER
        value: $USER
      - name: SSH_KEY
        value: $JENKINS_GCE_SSH_PRIVATE_KEY_FILE

I wanted to set environment variable SSH_USER using the value of USER. However, it seem this didn't work from the test log:

GO111MODULE=on go test -mod vendor -timeout=10m -v -tags """ journald" \
./test/e2e/metriconly/... \
-project= -zone=us-central1-a \
-image=cos-73-11647-217-0 -image-project=cos-cloud \
-ssh-user=SER -ssh-key=ENKINS_GCE_SSH_PRIVATE_KEY_FILE \
-npd-build-tar=`pwd`/node-problem-detector-v0.7.1-15-g9828ab7.tar.gz \
-boskos-project-type=gce-project -job-name=ci-npd-e2e-test \
-artifacts-dir=/logs/artifacts

Clearly SSH_USER got set to "SER" lol

[xueweiz]

Thanks for the super fast reviews @krzyzacy !
The test is continuously running now. See test results at below links:
https://k8s-testgrid.appspot.com/sig-node-node-problem-detector#ci-npd-e2e-test
https://prow.k8s.io/job-history/kubernetes-jenkins/logs/ci-npd-e2e-test

I still see some occasional flakes (once in last 26 runs), like this one.

Trying to figure out what's going on.

[xueweiz]

This seems like a test issue. I will follow up on kubernetes/node-problem-detector#296

Closing this issue. And thanks for the help @krzyzacy !