[Chore] Add e2e test case for multi-cluster by exposing OTEL collector.

[IshwarKanse]

Adds e2e test case for multi-cluster by exposing OTEL collector.

[jaronoff97]

@IshwarKanse can you add this to the e2e.yml in the .github directory so this runs?

[IshwarKanse]

@jaronoff97 This test requires a OpenShift cluster to run, it depends on several operators like Jaeger and cert-manager. Its still in WIP cause there are several issues we detected exposing the route for OTEL. I'll update the test case once the issues are fixed and will check if we can get it running on Kind clusters.

[jaronoff97]

Thanks for letting me know, i tagged Pavol on this PR as he has much more context on OpenShift things :)

[IshwarKanse]

2023/09/07 11:47:05 kutt-test config testdirs is overridden with args: [ tests/e2e-openshift ]
=== RUN   kuttl
    harness.go:462: starting setup
    harness.go:252: running tests using configured kubeconfig.
    harness.go:275: Successful connection to cluster at:  https://REDACTED:6443
    harness.go:360: running tests
    harness.go:73: going to run test suite with timeout of 300 seconds for each step
    harness.go:372: testsuite: tests/e2e-openshift has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/multi-cluster
=== PAUSE kuttl/harness/multi-cluster
=== CONT  kuttl/harness/multi-cluster
    logger.go:42: 11:47:09 | multi-cluster | Ignoring check_traces.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Ignoring create_otlp_receiver.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Ignoring create_otlp_sender.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Ignoring generate_certs.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Creating namespace: kuttl-test-subtle-lobster
    logger.go:42: 11:47:09 | multi-cluster/0-create-namespaces | starting test step 0-create-namespaces
    logger.go:42: 11:47:10 | multi-cluster/0-create-namespaces | Namespace:/kuttl-multi-cluster-send created
    logger.go:42: 11:47:11 | multi-cluster/0-create-namespaces | Namespace:/kuttl-multi-cluster-receive created
    logger.go:42: 11:47:12 | multi-cluster/0-create-namespaces | test step completed 0-create-namespaces
    logger.go:42: 11:47:12 | multi-cluster/1-create-jaeger | starting test step 1-create-jaeger
    logger.go:42: 11:47:13 | multi-cluster/1-create-jaeger | Jaeger:kuttl-multi-cluster-receive/jaeger-allinone created
    logger.go:42: 11:47:23 | multi-cluster/1-create-jaeger | test step completed 1-create-jaeger
    logger.go:42: 11:47:23 | multi-cluster/2-otlp-receiver | starting test step 2-otlp-receiver
    logger.go:42: 11:47:23 | multi-cluster/2-otlp-receiver | running command: [sh -c ./generate_certs.sh]
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | ...+...........+...+...+...+.+...+..+.........+.+.........+.....+.......+..+...+....+...+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+..+...+.+......+..+.+..............+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.....+.......+..............+.+...+......+.....+.+.........+......+.....+...+...+.........+.........+......+................+..+.+......+.........+.....+.+........+......+......+.........+.+.....................+........+......+.+.....+.........+..........+...+..................+..............+...+.+.....+.+.....+.+...+..+.......+...+..+......+......+......+.........+.............+...............+...+.........+........+......+............+.......+...+..+...+.........+.+..............+...+.+......+.....+..........+...+.....+...............+.+............+..+.+..+...+......+.+..+...+.............+.........+...+.........+...+..+......+....+...+......+..............+.+...........+...+............+......+..........+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | ..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..+.........+...+.......+...+.....+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+...+......+..........+...+...+.....+.......+..+....+...........+...+...+.......+.....+.............+..+............+.+.....+...+....+........+...+.+.........+.....................+.......................+...+.......+...+...+......+........+...+...+......+.+.........+...........+...+....+...+..+...+....+........+......+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | Certificate request self-signature ok
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | subject=C = US, ST = California, L = San Francisco, O = My Organization, CN = *.apps.ikanse-99.qe.devcluster.openshift.com
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | Certificates generated successfully in /tmp/kuttl-certs directory.
    logger.go:42: 11:47:26 | multi-cluster/2-otlp-receiver | Error from server (NotFound): configmaps "kuttl-certs" not found
    logger.go:42: 11:47:27 | multi-cluster/2-otlp-receiver | Error from server (NotFound): configmaps "kuttl-certs" not found
    logger.go:42: 11:47:28 | multi-cluster/2-otlp-receiver | configmap/kuttl-certs created
    logger.go:42: 11:47:29 | multi-cluster/2-otlp-receiver | configmap/kuttl-certs created
    logger.go:42: 11:47:29 | multi-cluster/2-otlp-receiver | ConfigMaps created successfully.
    logger.go:42: 11:47:29 | multi-cluster/2-otlp-receiver | running command: [sh -c ./create_otlp_receiver.sh]
    logger.go:42: 11:47:31 | multi-cluster/2-otlp-receiver | opentelemetrycollector.opentelemetry.io/otlp-receiver created
    logger.go:42: 11:47:38 | multi-cluster/2-otlp-receiver | test step completed 2-otlp-receiver
    logger.go:42: 11:47:38 | multi-cluster/3-otlp-sender | starting test step 3-otlp-sender
    logger.go:42: 11:47:38 | multi-cluster/3-otlp-sender | running command: [sh -c ./create_otlp_sender.sh]
    logger.go:42: 11:47:41 | multi-cluster/3-otlp-sender | opentelemetrycollector.opentelemetry.io/otel-sender created
    logger.go:42: 11:47:43 | multi-cluster/3-otlp-sender | ServiceAccount:kuttl-multi-cluster-send/kuttl-multi-cluster created
    logger.go:42: 11:47:43 | multi-cluster/3-otlp-sender | ClusterRole:/kuttl-multi-cluster updated
    logger.go:42: 11:47:44 | multi-cluster/3-otlp-sender | ClusterRoleBinding:/kuttl-multi-cluster updated
    logger.go:42: 11:47:49 | multi-cluster/3-otlp-sender | test step completed 3-otlp-sender
    logger.go:42: 11:47:49 | multi-cluster/4-generate-traces | starting test step 4-generate-traces
    logger.go:42: 11:47:51 | multi-cluster/4-generate-traces | Job:kuttl-multi-cluster-send/generate-traces-http created
    logger.go:42: 11:47:51 | multi-cluster/4-generate-traces | Job:kuttl-multi-cluster-send/generate-traces-grpc created
    logger.go:42: 11:47:57 | multi-cluster/4-generate-traces | test step completed 4-generate-traces
    logger.go:42: 11:47:57 | multi-cluster/5- | starting test step 5-
    logger.go:42: 11:47:59 | multi-cluster/5- | running command: [sh -c ./tests/e2e-openshift/multi-cluster/check_traces.sh]
    logger.go:42: 11:48:01 | multi-cluster/5- | Traces for telemetrygen-http exist in Jaeger.
    logger.go:42: 11:48:02 | multi-cluster/5- | Traces for telemetrygen-grpc exist in Jaeger.
    logger.go:42: 11:48:02 | multi-cluster/5- | test step completed 5-
    logger.go:42: 11:48:02 | multi-cluster | multi-cluster events from ns kuttl-test-subtle-lobster:
    logger.go:42: 11:48:04 | multi-cluster | Deleting namespace: kuttl-test-subtle-lobster
=== CONT  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- PASS: kuttl (65.29s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/multi-cluster (61.95s)
PASS

[IshwarKanse]

@frzifus @pavolloffay Can you review this test case.

Since we cannot use two clusters, the test case simulates by creating the sender and receiver collector in two separate namespaces.

*Sends traces to both OTLP HTTP and GRPC receiver exposed via route.
*Uses mTLS in collector config and the route is using passthrough termination.
*Checks the presence of traces in for both HTTP and GRPC.

[frzifus]

Looks rly good to me @IshwarKanse!

Since we generate our own certificates and jaeger is deployed without the operator, what would be missing to run this test in this github ci too?

Primary the routes? The last time I checked, they have been installed for some tests too.

[IshwarKanse]

cc @frzifus @pavolloffay Can you help to reapprove and get this one merged.

[pavolloffay]

Could we use edge termination or insecure routes?

The recent version of the operator improved support for routes:

• hostname can be omitted
• the grpc route works for insecure and edge termination

[IshwarKanse]

@pavolloffay I have updated the test case to remove setting the hostname. But for setting custom certs, ran into this issue. https://issues.redhat.com/browse/TRACING-3609 For the time being, I've kept the termination as passthrough and using the custom certs.