Add multi-cluster kind setup for testing

[johscheuer]

Description

This is the initial setup script for running e2e tests across multiple kind clusters.

Type of change

Please select one of the options below.

• New feature (non-breaking change which adds functionality)

Discussion

Are there any design details that you would like to discuss further?

Testing

Locally created the setup:

$ kind get clusters
cluster1
cluster2
cluster3
cluster4

Tested with the multi-dc setup:

$ kubectl --kubeconfig $KUBECONFIG1 exec -ti test-cluster-dc1-storage-3  -- bash
[root@test-cluster-dc1-storage-3 /]# fdbcli
Using cluster file `/var/dynamic-conf/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/var/dynamic-conf/fdb.cluster'.

Configuration:
  Redundancy mode        - double
  Storage engine         - ssd-2
  Coordinators           - 9
  Desired Commit Proxies - 2
  Desired GRV Proxies    - 1
  Desired Resolvers      - 1
  Desired Logs           - 3
  Desired Remote Logs    - 3
  Desired Log Routers    - 3
  Usable Regions         - 2
  Regions:
    Primary -
        Datacenter                    - dc1
        Satellite datacenters         - dc3, dc2
        Satellite Logs                - 3
    Remote -
        Datacenter                    - dc2
        Satellite datacenters         - dc3, dc1
        Satellite Logs                - 3

Cluster:
  FoundationDB processes - 10
  Zones                  - 10
  Machines               - 10
  Memory availability    - 8.0 GB per process on machine with least available
  Fault Tolerance        - 1 machines
  Server time            - 11/21/22 14:10:01

I will do some more testing and bring up an FDB cluster in that setup.

Documentation

I will add the required documentation before merging.

Follow-up

Create the e2e tests :)

[johscheuer]

Thea idea is to convert those tests: https://github.com/FoundationDB/fdb-kubernetes-operator/tree/main/config/tests/multi_dc to use the new kind setup.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 431a37b
• Duration 1:50:28
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: ee88e36
• Duration 1:53:54
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: ec866d4
• Duration 4:06:25
• Result: ❌ FAILED
• Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 3b4214e
• Duration 4:06:13
• Result: ❌ FAILED
• Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 3b4214e
• Duration 2:37:27
• Result: ❌ FAILED
• Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[sbodagala]

I took a look, but probably this needs to be reviewed by people who know about kind and have a much deeper understanding of the workings of Kubernetes.